Browse the traps
Every one of the 68 corpus items, with the plausible wrong answer it was built to catch. Pick a domain, open an item, reveal the trap. The bars show how the three non-LLM baselines score per domain on factual items; none of them is a language model. Try answering an item yourself →
Baseline accuracy by domain select a domain to filter
Status
Stated before anything else on the page, because a benchmark is easy to mistake for a result.
Baselines exist - no language model scored yet
This repository contributes a question corpus, a scoring harness, and three non-LLM baselines that establish the items discriminate: random floor 52.1%, TF-IDF retrieval 27.0%, oracle ceiling 100% accuracy. No language model has been evaluated on it yet.
- A real Anthropic Messages API adapter (
grounding_eval/adapters/anthropic_api.py) and a one-command run script (scripts/run_baselines.sh) now exist. Running them against an actual model is the immediate next step, and until that happens nothing here should be cited as a finding about any model. - The demonstration run under
synthetic/uses mock adapters and exists only to prove the harness executes end to end. It is not a result and the mock adapters do not represent any real system. - The preregistration below was written to make a later result auditable. It also makes this absence auditable.
Why this exists
Asked about overpressure protection for a chemical reactor, an assistant returned pressure relief valve data when rupture disks were the question.
Both are overpressure devices. They are not interchangeable. A rupture disk is a one-shot non-reclosing device with a nominal burst pressure and an operating margin requirement; a spring-loaded relief valve has a set pressure, a blowdown and a reseat behaviour. Quoting a blowdown percentage for a rupture disk is not a small error. It is an answer about a different device, delivered in the register of a correct one.
Read on
The substitution is invisible to anyone without the domain background to catch it, and the people most likely to ask the question are the people least likely to have that background. This is the structurally expected output of retrieval with no authoritative binding: the model returns the statistically likelier neighbour of the right answer.
So the corpus is built out of that failure mode on purpose. Alongside 68 questions it carries 23 minimal pairs: two questions whose correct answers diverge precisely because the device, or the regulatory trigger, differs. Answering one correctly while getting its partner wrong is the signal being measured, and it is why the primary outcome is paired accuracy rather than per-item accuracy.
State of the work
Read on
No language model has been evaluated on this corpus. What exists is the apparatus plus three non-LLM baselines (random floor 52.1%, TF-IDF retrieval 27.0%, oracle ceiling 100%) that show the items discriminate, and a real Anthropic Messages API adapter with a one-command run script, ready for the owner to run against an actual model.
That is a real limitation, not a phase of a rollout. Nothing here should be read as a finding about any language model until that run happens.
What this does and does not show
What this shows
- A corpus of 68 items across 8 domains, built around plausible-but-wrong adjacent answers, with 23 minimal pairs.
- Structural source verification: 68 of 68 checked items pass against the eCFR at a pinned edition date.
- That the items discriminate: random floor 52.1%, TF-IDF retrieval 27.0%, oracle ceiling 100%.
- A scoring harness and preregistration fixed before any system was run.
What this does not show
- No language model has been evaluated on this corpus. Nothing here should be cited as a finding about any model.
- The demonstration run under
synthetic/uses mock adapters and exists only to prove the harness executes end to end. It is not a result. - Verification is structural. It does not confirm that the answer key is a good answer.
- Scoring is lexical. A lexical scorer can be gamed by a verbose answer and can miss a correct answer phrased unusually.
The corpus
68 items across 8 domains, 63 factual and 5 category-error, organised into 44 families and 23 minimal pairs.
Corpus items by domain and risk tier
Why
| Domain | Items | High | Medium | Low | Source family |
|---|---|---|---|---|---|
| Overpressure protection: rupture disk devices vs. spring-loaded safety and relief valves | 12 | 8 | 4 | 0 | 46 CFR part 54 (Pressure Vessels), subpart 54.15 (Pressure-Relief Devices), US Coast Guard |
| The control of hazardous energy (lockout/tagout) | 10 | 6 | 4 | 0 | 29 CFR 1910.147 |
| OSHA injury and illness recording and reporting | 10 | 5 | 5 | 0 | 29 CFR part 1904 |
| Permit-required confined spaces (general industry) | 8 | 8 | 0 | 0 | 29 CFR 1910.146 |
| Machine guarding, electrical safety-related work practices, flammable liquid classification, and portable fire extinguishers | 8 | 2 | 6 | 0 | 29 CFR 1910.212, 1910.219, 1910.333, 1910.335, 1910.106, 1910.157 |
| Process safety management and risk management program intervals and obligations | 8 | 4 | 4 | 0 | 29 CFR 1910.119 (OSHA PSM) and 40 CFR part 68 (EPA RMP) |
| Respiratory protection assigned protection factors, IDLH provisions, and hearing conservation thresholds | 7 | 4 | 3 | 0 | 29 CFR 1910.134 and 29 CFR 1910.95 |
| Hazard communication: labels, safety data sheets, and GHS elements | 5 | 0 | 4 | 1 | 29 CFR 1910.1200 |
Why
Source verification
An item is only usable if a reader can check its answer against a source the reader can also reach.
- Items checked
- 68 of 68
- Passed
- 68, with 0 failures, 0 unparsed citations and 0 fetch errors
- eCFR edition
- 2026-08-01
- Sections fetched
- 19: 29 CFR 1904.39, 29 CFR 1904.7, 29 CFR 1910.106, 29 CFR 1910.119, 29 CFR 1910.1200, 29 CFR 1910.134, 29 CFR 1910.146, 29 CFR 1910.147, 29 CFR 1910.157, 29 CFR 1910.212, 29 CFR 1910.219, 29 CFR 1910.333, 29 CFR 1910.335, 29 CFR 1910.95, 40 CFR 68.42, 40 CFR 68.79, 46 CFR 54.15-10, 46 CFR 54.15-13, 46 CFR 54.15-5
- What is checked
- Checks provenance only: that the cited section exists in the eCFR, that the item's verbatim anchor_text appears in it, and that the cited paragraph markers appear in it. It does not evaluate whether the answer key is a good answer. Copyrighted standards are never fetched.
Verification is structural. It confirms that the cited section exists in the eCFR at a pinned edition date, that the item's verbatim anchor text appears in it, and that the cited paragraph markers are present. It does not confirm that the answer key is a good answer, and it never fetches a copyrighted standard.
What is scored
Three measured quantities, and an honest account of where the measurement is weakest.
Method, limits and notes
Adjacent substitution
Did the system return a semantically close but operationally wrong entity. This is scored separately from ordinary wrongness and carries the heaviest penalty, because a confident answer about the wrong device is more dangerous than an obviously bad one.
Citation presence and correctness
Is a source given, and is it the right one. A citation that exists but does not support the claim is graded differently from no citation at all.
Abstention credit
A system that declines scores above one that confabulates. Category-error items exist specifically to test this: they carry a false premise, and the correct response is to reject the premise rather than to answer.
Notes
Why the scorer is the weak point
Scoring is lexical. Required and forbidden concepts are matched as term sets with a coverage threshold, and the contrast heuristic that decides whether a term was asserted or merely mentioned works over a fixed character window. The repository's own limitations document names this as the weakest link, and it is right to.
A lexical scorer can be gamed by a verbose answer and can miss a correct answer phrased unusually. Human adjudication is specified in the preregistration for exactly this reason, with the adjudication protocol fixed in advance rather than invented once disagreements appear.
Preregistration
Fixed before any system was run, so that a later result cannot be the product of choices made after seeing the data.
| Parameter | Value |
|---|---|
coverage_threshold | 0.75 |
require_full_numeric | true |
adjacent_min_terms | 2 |
adjacent_low_coverage | 0.5 |
policy | safety |
| Contrast window | 70 characters |
Why
| Choice | Value |
|---|---|
| Primary outcome | Paired accuracy over the 23 minimal pairs |
| Secondary outcomes | Adjacent-substitution rate; per-item accuracy; abstention rate on answerable items; decline rate on category-error items; citation grade distribution |
| Arm comparison test | McNemar's exact test on discordant items |
| Interval method | Percentile cluster bootstrap over families, 2000 resamples, seed 20260904 |
| Alpha | 0.05, two-sided |
| Multiplicity | Only the primary outcome is confirmatory. Every secondary outcome is exploratory and will be labelled as such. |
| Repeats | 3 per arm minimum, reported separately with across-repeat spread |
| Temperature | Fixed and recorded per arm in the run file |
Why
Notes
Three arms
- Ungrounded. Instruction or persona prompt over parametric memory, with no retrieval.
- Pseudo-grounded. Retrieval over a plausible but unauthoritative corpus.
- Grounded. Retrieval over a version-pinned authoritative corpus with device-type metadata filtering.
The primary comparison is grounded against ungrounded on paired accuracy. The direction of the pseudo-grounded comparison is deliberately not predicted: retrieval over unauthoritative sources may plausibly do worse than no retrieval at all.
Recorded so it cannot be reinterpreted later
The preregistration states in advance which outcomes would count against the benchmark's own premise. If grounded and ungrounded do not differ on paired accuracy, that is a real result and will be reported as one. If adjacent-substitution rates are near zero in every arm, the corpus failed to do its job.
Run it yourself
python3 -m grounding_eval.cli corpus # describe the corpus
python3 -m grounding_eval.cli validate # corpus + stored source verification
python3 -m grounding_eval.cli demo # labelled demonstration run
python3 -m unittest discover -s tests -vPython 3.9+, pandas and numpy. No other dependencies, no build step. Baselines: scripts/run_baselines.sh.