GroundedPriyatham Chimmani ↗

Try an item

Pick an answer to a real question from the benchmark. One option is correct; the other is the adjacent wrong answer, the co-hyponym a retrieval system reaches for because it sits right next to the right answer in the source text. See how often you catch it, and how often plain retrieval does not.

Domain
Minimal pairs only
You: 0 of 0

Clause

Keyboard: ←/→ previous/next, 1/2 to answer

ArmAccuracy
Accuracy vs. adjacent substitution rate, with 95% intervals. Numbers are read from results/baselines_summary.csv.
Data table
ArmAccuracyAccuracy 95% CIAdjacent substitution rateAdjacent 95% CI
Random floor52.1%45.9%–58.9%47.3%40.4%–53.5%
TF-IDF retrieval27.0%15.4%–40.0%0.0%0.0%–0.0%
BM25 retrieval28.6%17.2%–40.9%0.0%0.0%–0.0%
Oracle ceiling100.0%94.3%–100.0%0.0%0.0%–0.0%

What adjacent substitution is

Every item in this corpus carries a reference answer and one plausible-but-wrong "adjacent" answer: a fact about a semantically close but operationally different entity, drawn from the same regulatory neighbourhood. A rupture disk answer where the question asked about a relief valve. A reclassification provision cited where the alternate-entry provision applies. The two are close enough that a system retrieving on term overlap alone regularly returns the wrong one with full confidence.

23 of the corpus's 68 items are built as minimal pairs: two questions whose correct answers diverge precisely because the device, or the regulatory trigger, differs. Answering one right while getting its partner wrong is the signal the benchmark is built to catch, which is why the primary outcome reported in the README is paired accuracy across those 23 pairs, not per-item accuracy.

No language model has been scored against this corpus yet. What exists are three non-LLM baselines (below) that show the items discriminate, and a real Anthropic adapter with a one-command run script. See "Running a real system" in the README to run one.