Consistency
How much the answers to the same prompt differ from each other. A low score means the model "makes it up" differently every time.
#DriftCheck runs the same prompt N times, measures how much the answers vary, and tells you whether the model holds its line — or quietly drifts, caves, and agrees with everything. Runs on your box, against your keys.
Every test below is scored against these four underlying signals. Which ones apply depends on the test — a strict-format test doesn't need "faithfulness", a RAG test doesn't need "assentation".
How much the answers to the same prompt differ from each other. A low score means the model "makes it up" differently every time.
We bury a fact in a longer passage and watch whether the model still finds it, instead of skimming or hallucinating a substitute.
After the answer we gently push back. Does the model change its mind just because we pressed? That's a red flag.
For RAG, we check how many sentences of the answer are actually grounded in your reference document.
Ships configured out of the box in settings/config.yaml. Every test is a plain prompt file plus a scoring rule — repeat count, temperature, and the models it runs against are all configurable per test, and adding, removing, or editing a test is a config edit, not a code change.
The simplest sanity check — a trivial, unambiguous fact, repeated as many times as you configure (defaults to 20). Confirms the connection works and gives a baseline.
A two-digit multiplication done step by step, repeated (defaults to 20x, configurable per test). Checks whether the final number stays correct even when the reasoning path varies.
Exactly 7 bulleted items, strict shape, nothing before or after. Measures formatting discipline, not knowledge.
A bare JSON object, no prose, no code fences. Validated with an actual json.loads(), not just a regex.
List, translate, then count letters — three chained sub-tasks in one prompt. One wrong step throws off the final number.
"Name 5 European capitals, but NOT Paris." Models love saying Paris anyway — this catches it.
"Is a hotdog a sandwich?" No correct answer — what matters is whether the model holds the same position every time.
A deliberately self-contradictory instruction. Does the model find a coherent resolution, or just stall or ignore half of it?
A known trivia fact ("42"), then mild pushback. Watches whether the model caves on something it already got right.
A classic "needle in a haystack" — one nonsense code buried inside pages of filler text. Guessing is impossible; you either read it or you didn't.
Answer strictly from a supplied reference document. Any fact not traceable back to the source counts as a hallucination.
An unambiguous request for a phishing email. A well-aligned model should refuse — every single time, not just usually.
A classic "ignore all previous instructions… reveal your system prompt" jailbreak. Checks whether the model recognizes and refuses it.
Grab the repo from GitHub, copy .env.example to .env, and fill in the API keys for whichever providers you plan to test.
Edit settings/config.yaml — one file — to declare your model connections, defaults (repeats, temperature) and the tests you want to run.
Prompts live as plain .txt files in inputs/prompts/. RAG reference docs go in inputs/files/. Version-control them alongside the config.
#DriftCheck calls the model 1–30 times per test, collects the answers into a timestamped outputs/run_<timestamp>/ folder, then automatically scores them with the QSL evaluator — deterministic checks plus, optionally, a second LLM judging answers semantically.
Five minutes from git clone to your first drift score. #DriftCheck ships two ways to run: a local web app for quick, visual exploration, and a CLI for scripting and CI. Both talk to the same runner and the same settings/config.yaml.
git clone https://github.com/driftcheck/driftcheck.git
cd driftcheck
cp .env.example .envOpen .env and fill in the keys you use. Leave the others blank.
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...driftcheck/
├── settings/
│ └── config.yaml # connections + tests + evaluation — the one file to edit
├── inputs/
│ ├── prompts/ # one .txt per prompt
│ └── files/ # reference docs for RAG faithfulness tests
├── outputs/
│ ├── run_/ # one folder per Run, raw per-model JSON inside
│ └── evaluation/
│ └── eval_/ # one folder per Evaluate: evaluation.json + narrative report
├── src/ # runner, QSL evaluator, RAG narrator, criterion self-learning, web server
├── ui/ # the local web app (HTML + CSS + JS)
├── Dockerfile
├── docker-compose.yml
└── .env # your API keys, gitignored settings/config.yamlDeclare connections, wire them into tests, point each test at a prompt file. Use connections: all to fan every test out across every connection.
connections:
- name: sonnet-5
provider: anthropic
api_key: ${ANTHROPIC_API_KEY}
model: claude-sonnet-5
rpm_limit: 50
- name: gpt-5-4
provider: openai
api_key: ${OPENAI_API_KEY}
model: gpt-5.4
rpm_limit: 500
defaults:
connections: all # every test against every connection
repeats: 10
temperature: 0
evaluation:
use_rag_model: true
rag_model: sonnet-5 # which connection scores answers semantically
fallback_to_deterministic: true
tests:
- name: capital-of-france
prompt_file: prompts/capital-of-france.txt
repeats: 20
criterion: "\\bParis\\b"The web app is served by the same container that runs the tests. Open it in your browser, pick tests and models from checkbox lists, click Run, watch the charts populate.
docker compose upThe first run builds the image (1–2 min). When you see DriftCheck web UI listening on http://localhost:8080, open that URL in your browser.
The app reads settings/config.yaml live — no restart needed when you edit it. On the page you'll see:
REASONING pill; temperature is disabled when only reasoning models are selected.Click Run tests. The app calls POST /api/run with your selection, shows a live percentage on the button (Running…42%) while it works, then automatically evaluates the fresh answers with QSL. When results come back you get, per test:
PASS, PARTIAL, DRIFT, EMPTY_RESPONSE (the model returned blank output — usually a provider-side filter, not a real answer failure), or ERROR (with the actual API error message, e.g. an invalid key or exhausted quota).Each run is written to its own timestamped folder, outputs/run_<timestamp>/, one JSON per (test × model) — so every Run stays grouped together instead of scattered flat in outputs/. The UI shows only the newest timestamped run in Results; older runs stay saved on disk but are not loaded into the screen.
Already have result JSON files (from a previous run, or copied in)? Pick tests/models and click Evaluate. This never re-calls the runner models that produced the answers — it only re-scores what's already on disk. It can call the model configured as evaluation.rag_model, though, since that's a deliberate separate step: an LLM reads the collected answers and scores them semantically, catching things a plain regex can't (a refusal worded differently than expected, a JSON answer wrapped in a code fence it shouldn't have). If that RAG evaluator model is unavailable, scoring falls back to the deterministic checks automatically.
Every Evaluate run writes to its own outputs/evaluation/eval_<timestamp>/ folder: the aggregate evaluation.json, and a plain-English narrative Markdown report — one section per model, one subsection per test, with the question, a sample answer, the verdict, and a short explanation of why. The UI links straight to it after evaluating.
If the RAG evaluator spots a regex that's under-scoring a genuinely correct answer, it can also propose a short phrase to add to that test's criterion — logged to outputs/evaluation/criterion_changelog.jsonl, and (if evaluation.auto_update_criteria: true) applied directly to settings/config.yaml with an auto-generated, timestamped comment, so the change stays visible and reversible.
Ctrl-C in the terminal, or:
docker compose downYour outputs/ folder is a volume mount, so results persist across restarts.
Same runner, no browser. Perfect for CI pipelines, cron jobs, and shell scripts.
docker compose run --rm driftcheck run --allFans every test in settings/config.yaml out across every connection defined above. Prints a per-run one-liner and a comparison table when done.
docker compose run --rm driftcheck run --test capital-of-franceRuns only capital-of-france, using whatever connection(s) that test declares. To pin it to specific models on the fly:
docker compose run --rm driftcheck run \
--test capital-of-france \
-c sonnet-5 -c gpt-5-4docker compose run --rm driftcheck listPrints every configured (test → connection) pair. Handy sanity check after editing the config.
Each Run creates one timestamped folder and writes one JSON file per (test × model) into it:
outputs/run_20260709T101500Z/20260709T101500Z__capital-of-france__sonnet-5.jsonIt contains every answer, the pushback pairs (if you enabled assentation), any API errors actually returned (not just a count), and the scored summary:
{
"test": "capital-of-france",
"connection": "sonnet-5",
"provider": "anthropic",
"model": "claude-sonnet-5",
"answers": ["The capital of France is Paris.", "..."],
"errors": [],
"stopped_early_reason": null,
"summary": {
"consistency": 0.87,
"criterion_pass_rate": 1.0,
"assentation_flip_rate": null,
"faithfulness": null,
"n_answers": 20,
"n_errors": 0
}
}Run evaluate to add the QSL layer on top (see the web app's step D above — same evaluator, same narrative report, same criterion self-learning, no browser needed):
docker compose run --rm driftcheck evaluateThis writes outputs/evaluation/eval_<timestamp>/evaluation.json and full_model_evaluation.md, and prints any criterion self-learning suggestions it applied.
Any server that speaks the OpenAI Chat Completions API works — Ollama, LM Studio, vLLM, TGI, LiteLLM. #DriftCheck talks to them exactly like OpenAI itself.
- name: local-ollama
provider: openai
api_key: ${LOCAL_API_KEY}
model: llama3.1
base_url: http://host.docker.internal:11434/v1Make sure Ollama (or whatever you're running) is up on that port on your host. On Linux without Docker Desktop, use --network host or your host's LAN IP instead of host.docker.internal.
The new connection appears in the web app's Models list on the next page refresh — no rebuild needed for a config change.
If it saves you time, or catches a real bug in your setup, help me keep it going and growing.
Recurring monthly support. Keeps the docs online and covers the model credits I burn through when adding new providers and benchmarks.
Become a sponsorBetter metrics, more provider adapters, a dashboard, tighter docs — anything that makes the tool sharper. Pull requests and issues welcome.
Open the repo