Grounded›ehs-ai-grounding-eval›Story
Story 3 of 7 · the adjacent clause
The adjacent clause
A wrong answer about the next regulated entity over reads exactly like a right one. It is fluent, internally consistent, and it carries a citation. This benchmark was built to catch that one failure.
68 questions. 23 minimal pairs. Three non-LLM baselines. No language model has been scored yet, and the page says so wherever that number would go.
The answer comes back about the wrong device
It is fluent, it cites a real clause, and every quantity in it belongs to a different piece of equipment.
1 clause governs the rupture disk question
A rupture disk protects an unfired pressure vessel. The question asks what margin must hold between operating pressure and burst pressure. One clause in 46 CFR subpart 54.15 answers it.
A relief valve sits 1 step away and shares the vocabulary
Spring-loaded relief valves cover the same vessel under the same overpressure language. Retrieval on the words in the question returns both devices.
0 of the quantities in the answer exist for a rupture disk
The answer gives set pressure, blowdown and reseating. A rupture disk is a one-shot non-reclosing device and has none of them. The sentence still reads as correct.
The clause sets 1.3 times operating pressure and ends it
46 CFR 54.15-13(b)(3) requires operating pressure times 1.3 to stay under the nominal burst pressure. Opening the clause catches the swap in seconds. Reading the answer for plausibility never will.
23 minimal pairs cover 46 of the 68 items
The corpus is built so that guessing well is not the same as reading the clause. A system has to get both members of a pair right.
68 items cover eight regulated domains
Confined space, hazard communication, lockout/tagout, machine guarding and electrical, overpressure protection, process safety, recordkeeping, respiratory protection and noise. Every item names a clause and a source URL.
23 minimal pairs cover 46 of the items
A minimal pair is two entities the law treats separately that share vocabulary. Both members get asked. A system can pass one by luck and cannot pass both without reading the right clause.
5 items have no correct answer
These ask something that does not apply to the entity, so the right response is to decline. Answering anyway shows how a system behaves when the question itself is malformed.
68 of 68 items trace back to a published source
Six drafted items whose provenance could not be confirmed went to quarantine rather than into the corpus. Verification checks provenance, not editorial quality. A benchmark nobody can trace is not evidence.
Corpus counts as a table
| Total items | 68 |
|---|---|
| Factual items | 63 |
| Category-error items | 5 |
| Complete minimal pairs | 23 pairs (46 items) |
| Domains | 8 |
| Items verified against source | 68 of 68 |
| Drafted items excluded, provenance unverified | 6 |
Before the results · answer one yourself
You go first
One real item from the corpus. Read it, settle on your answer, then open the reveal. The point is not whether you get it right: it is how reasonable the wrong answer sounds once you see it.
Question
Items read from docs/try-data.json. 63 factual items; the 5 category-error items are excluded here because they carry no adjacent answer to show.
Retrieval substitutes 0.0 percent of the time, guessing 47.3 percent
Accuracy is the number people quote. The substitution rate decides whether a wrong answer reaches the plant floor.
Guessing scores 52.1 percent and substitutes 47.3 percent
The random arm picks between the two answers an item carries. It lands at 52.1 percent accuracy and takes the adjacent wrong answer 47.3 percent of the time. That second number is what the corpus was built to measure.
TF-IDF scores 27.0 percent and substitutes 0.0 percent
TF-IDF over the authoritative text gets 27.0 percent of the factual items right, below the guessing floor, and never substitutes the adjacent entity. It quotes a clause or it says nothing. BM25 behaves the same way at 28.6 percent.
The retrievers cite the right clause 73.5 percent of the time
On the vertical axis, citation correctness: 73.5 percent for TF-IDF and 75.0 for BM25, against 5.0 percent for guessing. They are wrong often and misleading rarely. A system that reverses that trade is the risk this benchmark was built to catch.
0 language models have been scored on this corpus
The point at 100 percent is a fixture that replays the answer key, not a system. The region between the retrievers and that ceiling stays empty until an LLM arm is actually run. Until then this page reports no model number at all.
Baseline table
| Arm | Accuracy (n=63) | Adjacent substitution | Citation correct clause | Paired accuracy |
|---|---|---|---|---|
| random_floor | 52.1% | 47.3% | 5.0% | 28.7% |
| retrieval_tfidf | 27.0% | 0.0% | 73.5% | 13.0% |
| retrieval_bm25 | 28.6% | 0.0% | 75.0% | 13.0% |
| oracle | 100.0% | 0.0% | 100.0% | 100.0% |
The honesty scene
What this does not establish
No LLM API was available on the machine these runs were made on. There is no language model arm anywhere in this repository, and any demonstration run carrying mock output is labelled mock in the artifact and on the page.
68 items across 8 domains is a built object, not a sample of the questions people ask. It cannot support a claim about how often substitution happens in the field, only about whether a system falls for it here.
Scoring matches signature terms; it does not read for meaning. The contrast heuristic that decides whether an answer substituted the adjacent entity is the weakest link in the pipeline, and it is documented as such.
68 items is small. Intervals are cluster bootstraps over item families, so minimal pairs resample together. Per-domain claims are out of reach at this size.
The source clauses are public text. Whether a given model has seen them in training is unknown and untested here.
Preregistration and limitations are written and committed, not refereed. The full list is docs/limitations.md, twelve entries long.
Source artifacts: docs/limitations.md, docs/CORPUS_CARD.md, docs/results.md
What a leader does Monday
Ask for the clause, not the answer
Take the last five AI-assisted answers your team acted on in a regulated area. For each one, open the cited clause and check that it governs the entity in the question, not its neighbour. If an answer carries no clause, no one has checked it.
That is a one-hour exercise and it is the same test this benchmark runs. A tool that can never show you the clause cannot be audited, whatever it scores.
- Pull five AI-assisted answers your team acted on.
- Open each cited clause and read it.
- Confirm the clause governs the entity in the question.
- Count how many carried no clause at all.
Story 4 of 7 · next in the path
Why people err, as a graph
The performance influencing factors behind the error, as a network, with one rule derivation traced edge by edge.