Methodology

Numbers publish here when the full proof run completes

This page exists before the numbers do, on purpose. Writing down how we'll measure, before we know what the measurements say, is the only version of "trust our benchmarks" worth anything. When results appear here, they'll be scored exactly as described below, and this page won't be quietly rewritten to fit them.

Exact match is the integrity floor

The strictest scorer runs first: did the system produce the exact expected answer, byte for byte where the benchmark defines one? Exact match understates real accuracy (a correct answer phrased differently scores zero), and that's fine. Its job isn't to flatter. It's to make inflation impossible at the floor.

Judged answer accuracy is the primary number

Above the floor, an independent judge model scores whether each answer is actually correct against the gold answer, so correct-but-differently-phrased answers count. The judge is calibrated against human labels before it scores anything, and the calibration figures publish alongside the results. Answer accuracy, judged, is the headline number. Recall of relevant material is necessary but never sufficient; finding the right passage and still saying the wrong thing counts as wrong.

Abstention is scored, not skipped

Every benchmark reports coverage (how often the system answered), risk (how often an answer was wrong), and the curve between them. Declining is a scored decision: an abstention with the fact genuinely absent is correct, and an abstention with the answer sitting in the data is a miss, counted as one. Leaderboards that only score answered questions reward systems for going quiet on anything hard. Ours doesn't get that discount.

Determinism checks

Same input, same pack, same answer, same citations. Runs are repeated and diffed; any nondeterminism is a bug in the run, not noise to average away. A result that can't be reproduced exactly doesn't publish.

Pinned baselines

Every published run pins what produced it: dataset version, pack build, engine commit, flags. Comparisons happen against those pins, never against a moving target. If a baseline changes, it changes as a new pinned entry, with the old one kept.

No benchmark-fitted data

The knowledge packs are built from researched public sources, never from benchmark answer keys, and never tuned toward a test set. Where the packs lack a fact a benchmark needs, the honest outcomes are: derive it from clean sources, or abstain and eat the score. Fitting the data to the test would make every number on this page worthless, so it's banned outright.

That's the whole method: a floor that can't be inflated, a headline number a calibrated judge can defend, silences scored as decisions, runs that repeat exactly, baselines that hold still, and data that never saw the test. The numbers land here when the full proof run completes, and not a day before.