The Hours Denominator: Data Quality in OSHA Establishment Injury Filings and What It Does to Every Benchmark Built on Them

Abstract#

Establishment-level injury and illness summaries filed with OSHA's Injury Tracking Application are the only public source from which industry injury-rate benchmarks can be built at scale. We process 2,801,064 establishment filings covering calendar years 2016 to 2024 and show that the hours-worked field, the denominator of every rate computed from these data, is dominated by a small number of implausible entries. Under a plausibility screen of 120 to 4,500 hours per employee per year, 2.07% of filings fail, yet those filings carry 96.7% of all hours reported. The pooled aggregate total recordable incident rate reads 0.134 on the unscreened file and 3.983 after screening. The correction is not a constant: the year-level ratio between screened and unscreened aggregates ranges from 1.39x in 2018 to 249x in 2019, because the share of hours sitting in implausible filings varies from 29% to 99.6% between years. Screened aggregates, by contrast, stay within 3.1 to 4.6 across all 9 years while unscreened aggregates swing between 0.016 and 2.92. Linking establishments across years shows the defect is not random: an establishment flagged one year is flagged the next at an odds ratio of 39.1, against 1.01 under a permutation null that holds each year's flag count fixed, so implausible hours reporting is a property of filers rather than of filings. We characterise the failing records, evaluate the screen's sensitivity to its bounds, examine zero-inflation and overdispersion in recordable-case counts with Poisson, negative binomial and zero-inflated fits, and assess year-over-year stability of peer-group percentile bands. The practical conclusion is narrow and consequential: any benchmark computed from the raw file is wrong by a factor that cannot be known without screening, and no single multiplier repairs it.

1. Introduction#

Every EHS leader eventually asks whether their establishment's injury rate is good. The honest answer requires comparison against establishments of similar size in the same industry, and the data to make that comparison are public: OSHA publishes establishment-level Form 300A summaries each year. They are, however, close to unusable as downloaded. This paper documents why, quantifies the damage, and shows that the damage is not stable enough to correct with a rule of thumb.

The contribution is deliberately modest in scope. We do not propose a new injury model. We show that the denominator underneath every published aggregate is unreliable in a specific, measurable, and year-dependent way, and we publish the reproducible pipeline that establishes it.

2. Data and reproducibility#

Nine public files were processed: the annual ITA data files for CY2016 through CY2022 and the 300A summary extracts covering 2023 through 2025. After de-duplication of 4,703 repeated rows, 2,801,064 filings remained. The pipeline downloads the source files itself, fails loudly if they are unavailable rather than substituting fixtures, and regenerates every table and figure by script. The repository carries 136 tests.

Year-level summary#

3 The dependency this paper cannot argue around#

Sections 3 through 5 make a claim about architecture: that a system whose conclusions carry derivation traces back to identified source clauses is auditable in a way that a system conditioned on a persona and a pasted document is not. That claim is about the relationship between a conclusion and a record. It says nothing about the relationship between the record and the world.

The distinction is the one AWS draws explicitly for its own deployed reasoning system. Their documentation states that a VALID result "covers only the parts of the input captured through policy variables," and their own worked example is a claim resting on a forged doctor's note that will be scored valid because no variable captures forgery ^{[aws_arc_docs]}. A sound procedure over a false premise returns a sound conclusion about the premise. Nothing about the soundness reaches the world.

For safety-critical question answering in process industries this is not a philosophical footnote, because the records in question are self-reported administrative filings that nothing validates at intake. This section quantifies how bad that is, using the public OSHA Injury Tracking Application (ITA) Form 300A corpus, and then states precisely what follows for a grounded architecture. The short version is that grounding and data validation are separate obligations, and satisfying one does not discharge the other.

All figures in this section were computed by the pipeline in repos/ehs-osha-analysis from nine ITA Form 300A files (CY2016 through CY2024) downloaded from osha.gov, with URLs, byte counts and SHA-256 digests pinned in the repository and verified on 2026-09-03 ^[osha_ita]. They are transcribed from the generated summary.json and output tables, and regression tests parse the headline values back out of the manuscript's source repository and fail if the prose and the tables disagree. Where a figure comes from the author's earlier work rather than from this pipeline, it is labelled as such.

4 What the corpus is, and what it is not#

OSHA requires annual Form 300A summary submission from establishments with 250 or more employees, and from establishments with 20 to 249 employees in the higher-hazard industry groups listed in 29 CFR 1904 Subpart E ^{[osha_ita_users_guide]}. The result is a mandated administrative collection over a selected slice of US industry. It is not a probability sample of the US workforce, and any national rate computed from it is a rate for that slice.

The screened aggregate reported below (about 3.98 recordable cases per 200,000 hours) sits well above the BLS Survey of Occupational Injuries and Illnesses private-industry total recordable case rate of 2.3 per 100 full-time equivalent workers for 2024 ^[bls2026]. That direction is what a universe skewed toward larger establishments in higher-hazard industries would produce. The two figures are not directly comparable and are not compared here; OSHA publishes its own ITA-versus-SOII comparison and it should be read before anyone treats the gap as a finding ^{[osha_ita_bls_comparison]}.

Three schema facts matter for anyone building a retrieval or reasoning layer over these files, because each is a silent-wrong-answer trap rather than a loud failure. Column order differs between the 2016–2022 and 2023+ files, and naics_year appears only from

1. ITA Data CY 2018.csv contains bytes that are not valid UTF-8 (0x92, a Windows-1252 right single quote), so a strict decoder raises and a permissive one silently substitutes. And the size field is not comparable across years: OSHA's summary data dictionary documents codes 1 (<20), 2 (20–249), 21 (20–99), 22 (100–249) and 3 (250+), and states that "code 2 was split to 21 and 22 with the collection of 2023 data" ^{[osha_ita_summary_dict]}. The files agree - among plausible filings, code 2 falls from 244,231 in 2022 to 41,343 in 2024 while codes 21 and 22 rise to 224,266 combined - so a pooled panel mixes one 20–249 band with two narrower bands covering the same establishments. The codes are documented; the pooled semantics are not.

None of these is exotic. They are the ordinary condition of regulatory data, and they are invisible to a chunk-and-embed index, which will happily retrieve a 2019 size value and a 2024 size value as if they denoted the same population.

5 The denominator failure#

An incident rate is a ratio:

TRIR = 200,000 × recordable cases / hours worked

The numerator is bounded by how many people work at a site. The denominator is a free-text number on a form. When rates are aggregated across establishments the standard estimator is a ratio of sums, so one filing with an impossible hours value can dominate the denominator of an entire industry, state or national figure while contributing nothing to the numerator. The aggregate is then biased toward zero, and it reads as good news.

Pooled across 2,801,064 deduplicated filings, 2016–2024, under a screen that flags filings outside 120–4,500 hours per average employee per year:

The concentration is extreme even within the flagged set. A single filing holds 88.81% of all hours ever reported to the ITA and contributes zero recordable cases: establishment 90427, reporting year 2019, declaring 7 employees and 16,831,620,723,179 hours worked. That is roughly 2.4 trillion hours per employee, against a physical ceiling of 8,760. The ten largest filings by declared hours hold 94.5% of all hours and 0.0008% of all cases. One flag does the entire job: hours per employee > 4,500 fires on 24,617 filings (0.879% of the corpus) carrying 96.65% of all hours. The missing-value and low-hours flags matter for establishment-level rates and are irrelevant to the aggregate.

The screened aggregate is not an artefact of where the bounds are drawn. Across a 25-cell grid of lower bounds (100–400 h) and upper bounds (3,500–6,000 h), the screened aggregate stays within 3.963–3.993. The flagged share is not similarly stable, running from 1.79% to 4.49% across the same grid, and that instability is the point rather than a weakness: the filings that move in and out of the flagged set as bounds shift carry almost none of the hours, so the correction they make is nearly identical. The conclusion survives the threshold choice; the count of flagged filings does not.

6 What does not replicate, and why the non-replication is the finding#

An earlier public repository by the same author ran the same class of plausibility screen over a narrower panel of OSHA establishment filings and reported a headline correction multiplier ^{[chimmani2026]}. That multiplier is not reproduced by the pipeline described here, it is superseded by the figures in this section, and it is not restated here as a quantity, because the reason it cannot be reproduced is also the reason it should never have been quoted as one.

The flagged share is stable and does replicate: 2.07% pooled here, and 1.56% to 2.60% across individual years. The multiplier is not a property of the corpus at all. It is a property of whichever years happened to contain an extreme filing:

The screened rate is stable across nine years (3.15–4.60). The unscreened rate is not (0.016–2.92). The multiplier between them ranges from 1.39× to 249×, and no year subset tried here reproduces the earlier figure. The correct statement of the finding is therefore structural and holds in every single year: a small minority of filings is internally implausible, those filings hold a disproportionate share of the denominator, and removing them raises the aggregate substantially. The specific multiplier is a property of the tail of the hours distribution in whatever slice was taken, and should not be quoted as a constant. We state this here rather than in a limitations paragraph because the earlier framing circulated, and correcting one's own published number in the body of a paper is cheaper than having a reviewer do it.

The methodological point generalizes beyond this corpus. Hanecke and colleagues faced the same problem in 1998 with more than 1.2 million German accident records and no exposure data, and had to construct and compare estimated exposure models before any rate could be computed at all ^{[hanecke1998]}. Hopkins reached the same conclusion from the other direction without any statistics, arguing that what determines whether a safety indicator is meaningful is not whether it is labelled leading or lagging but whether, at the level of aggregation in use, there are enough countable events to form a rate - the "zoom effect" ^{[hopkins2009]}. Both are arguments about denominators. Neither has an analogue in the performance-shaping-factor literature, which presupposes a well-defined opportunity count throughout ^[nureg2198], ^[groth2012].

7 Zero-recordable filings: overdispersion, not a hidden non-reporting population#

Among plausible filings, 37.1% of all establishment-years report zero recordable cases. For NAICS 325 (chemical manufacturing) the pooled figure is 37.4%, rising from 34.1% in 2016 to 41.3% in 2024. The prior repository's often-quoted figure of roughly 38% for chemical establishments is consistent with this panel's 37.4% ^{[chimmani2026]}.

The obvious reading is that a zero-inflated count model is required - that there exists a distinct population of establishments producing structural zeros through non-reporting. Fitting the models does not support that reading as the main story.

Four intercept-only models with an exposure offset (Poisson, NB2, ZIP, ZINB) were fitted by maximum likelihood and compared by AIC, BIC and observed-versus-expected zeros over the 30 largest NAICS 3-digit groups in 2024, each with at least 500 establishments, covering 306,711 establishment-filings. Industry selection reads only group size, never outcome values or fit statistics.

Two results point in opposite directions. Overdispersion is universal: Poisson never wins in any of the 30 industries, and the variance-to-mean ratio of recordable counts runs from 3.0 to 756 against the value of 1 that Poisson assumes. ZIP is never selected either, so adding structural zeros to a Poisson does not rescue it - the problem is the spread of the whole distribution rather than the zeros alone. Meanwhile zero-inflation is optional and, where present, small: in 14 of the 30 industries the ZINB inflation parameter collapses to the boundary (on the order of 1e-14) and NB2 wins outright, and where ZINB is selected the inflation probability ranges from 0.001 to 0.069. That is a few percent of establishments, not the 37–41% zero share that motivated fitting it. NB2 frequently predicts slightly more zeros than are observed.

Refitting the four models with a log-hours offset, establishment size-band dummies and NAICS 4-digit fixed effects within each 3-digit group (count_model_covariates_summary.csv in the repository, method in docs/COUNT_MODELS.md) leaves overdispersion intact in all 30 industries (NB2 beats Poisson at boundary-corrected p < 0.001 in every one; NB2 alpha median 0.79, minimum 0.16) but reduces the ZINB wins from 15 of 30 to 10 of 30, with the ZINB-vs-NB2 boundary test below 0.01 in 9. The intercept-only table above therefore overstates zero inflation: part of the excess zeros was mean heterogeneity across size and sub-industry. The conclusion that holds under both specifications is that overdispersion is universal and zero inflation beyond NB2 is present in a minority of industries.

Establishment size accounts for most of the rest. The zero share falls from 0.751 in the 1–19 employee band to about 0.08 above 500 employees, and the median establishment TRIR rises from 0.00 to 3.61 across the same range; a ten-person site has so little exposure that zero is the modal outcome. (The fall is not strictly monotonic - 0.073 at 250–499 rises to 0.082 at 500–999 - and we note it rather than smoothing it.) Any benchmark comparing a site against an industry median without conditioning on size is comparing it against a number driven by how large the other sites are.

This is a statement about distributional shape, not about reporting behaviour. An overdispersed process and a mixture of compliant and non-compliant reporters can generate similar count distributions, and these data cannot separate them. Nothing here is evidence that under-reporting is absent. What the result does establish is that the bare zero share is not evidence that it is present, which is how that statistic is ordinarily used.

The distributional facts also bound what any downstream predictive layer can do. A variance-to-mean ratio of 241 in the largest size band, combined with a base rate that puts one recordable at roughly one per 10,870 eight-hour worker-shifts at the 2024 BLS private-industry rate ^[bls2026], is a rare-event regime in which maximum-likelihood logistic regression underestimates event probabilities ^{[king_zeng2001]} and in which the standard imbalance corrections - random over- and undersampling, SMOTE ^[chawla2002] - degrade calibration by strongly overestimating minority-class probability without improving discrimination ^{[goorbergh2022]}, ^{[carriero2025]}. Discrimination metrics are close to uninformative here; calibration is the property that determines whether a score can be acted on ^{[vancalster2019]}, ^{[vancalster2016]}.

8 Benchmark bands move on their own#

Peer groups defined as NAICS 3-digit × size band, compared across eight adjacent year pairs over roughly 440 matched cells per pair, are highly reproducible in ordering (median Spearman rho 0.862–0.933 across the p25, p50, p75 and p90 bands) and 95.7–98.5% of publishable cells persist between adjacent years. But the level of each band moves by a median of 9.5–14.3% per year. An establishment sitting exactly on last year's p75 would be roughly a tenth of the way off this year's p75 without any change in its own performance.

This bears directly on the question-answering task. A retrieval system asked "is this site's TRIR above the industry 75th percentile?" will return an answer that is approximately a coin flip near the band edge, and it will return it with the same fluency whether the site is clearly above, clearly below, or inside the year-to-year noise. The band is a real quantity with a real sampling distribution; the answer is presented as a fact.

9 Hour-of-shift: a claim withdrawn, and the conditions for ever making it again#

The author's earlier work circulated a shift-timing claim about when during a shift injuries peak ^{[chimmani2026]}. It is withdrawn here rather than restated, and its numbers are not reproduced anywhere in this manuscript, for a reason that is structural rather than a matter of degree: the pipeline described in this section could not produce such a figure at all. Form 300A is an annual summary. It carries case counts and total hours worked and no time-of-day, time-employee-began-work, or narrative field at all ^{[osha_ita_summary_dict]}. Hour-of-shift requires Form 300/301 case-detail data, which OSHA began publishing on a different and narrower establishment universe with the 2023/2024 collection cycle ^{[osha_ita_case_detail_dict_2026]}. Any future hour-of-shift figure must name its own dataset, year range and N, must not be read as a subset of the 300A panel analysed here, and must not inherit this panel's filing count.

Two further constraints apply, and both are decisive for how the finding may be phrased.

First, the phenomenon is named and roughly twenty-five years old. Tucker, Sytnik, Macdonald and Folkard called it the "2–4 h shift phenomenon" ^[tucker2000], and Folkard and Tucker report "a slightly heightened risk from the second to the fifth hour" in their review of shift work, safety and productivity ^{[folkard2003]}. A large-scale corroboration in a new sector would be a real contribution. A discovery claim is not available, and no corroboration is offered in this manuscript.

Second, and more seriously, a peak in the count distribution is not a peak in risk. Any statistic of the form "such-and-such a share of injuries occurs in the first four hours" is by construction a share of counts, and counts are governed by how many people are at work in each hour of shift. Folkard and Tucker found that once exposure is accounted for, risk rises approximately exponentially with time on shift, more than doubling by the twelfth hour relative to the first eight ^{[folkard2003]} - a pattern that an unadjusted count distribution, dominated by the hours in which most people are working, will not show. Hanecke and colleagues had to build estimated exposure models precisely because the hours-at-work denominator was unavailable in the German data ^{[hanecke1998]}, and no public US dataset supplies an hours-at-risk-by-shift-hour denominator either. MSHA's Accidents file carries SHIFT_BEGIN_TIME and an accident time on essentially all records ^{[msha_accidents_definition]}, and BLS publishes an "hours worked before event" dimension in its case-characteristics series ^{[bls_ca_documentation]}, so cross-source comparison of the distribution is feasible. The exposure denominator is not.

The defensible statement is therefore about the observed distribution of reported incidents, explicitly not about risk, with the denominator gap stated as an open problem rather than assumed away. We adopt that phrasing throughout and recommend it to anyone reusing the figure.

10 How the two failures compose#

The preceding subsections describe defects in a data corpus. Sections 4 and 5 describe defects in ungrounded language models: parametric recall that degrades on long-tail content ^{[kandpal2023]}, ^[mallen2023], training and evaluation regimes that reward a plausible guess over an abstention ^[kalai2025], and distributional representations that encode relatedness without encoding relation type, so that taxonomic siblings are the maximal-confusion class. These are independent failure modes with independent causes. They compose in three specific ways, and the composition is worse than either alone.

A retrieved number is not a validated number. Retrieval-augmented generation supplies non-parametric memory and marginalizes over retrieved documents ^[lewis2020]. That machinery grounds an assertion in a document. It carries no commitment about whether the document is internally consistent, and none of the standard RAG evaluation metrics test for it: attribution frameworks score whether a claim is attributable to its cited source ^{[rashkin2023]}, which a filing declaring 2.4 trillion hours per employee satisfies perfectly. A system that retrieves establishment 90427's filing and reports a TRIR of 0.00 has produced a fully attributable, fully traceable, entirely wrong answer. The derivation trace this paper advocates certifies that the conclusion follows from the record. Whether the record is admissible is a separate predicate, and it has to be computed.

Aggregation hides the defect from the reader and from the model. The single filing above is visible at establishment level - 7 employees, 16.8 trillion hours - and invisible in an industry aggregate, where it appears only as a suspiciously low rate. The failure mode is therefore worst at exactly the level of aggregation at which executives and benchmarking tools operate. A model asked for an industry rate has no signal in the retrieved aggregate telling it the aggregate is broken, and the resulting answer is low-variance across paraphrase and across sampling, so consistency-based hallucination detectors ^{[manakul2023]} and semantic-entropy methods ^[kuhn2023], ^{[farquhar2024]} will score it confident. Uncertainty estimation detects the model's uncertainty. It does not detect the corpus's.

Grounding narrows the error class without eliminating it, and the residual is measurable. The governing empirical result is Magesh and colleagues' preregistered evaluation of commercial legal research tools marketed as hallucination-free, which measured hallucination rates between 17% and 33% ^[magesh2025]. There is no basis for expecting a process-safety deployment to do better, and every reason - hierarchical, cross-referenced, edition-versioned, table-heavy source documents - to expect it to face harder retrieval conditions. Any claim in this paper is therefore a measured delta under stated conditions, never an elimination.

The constructive consequence is that plausibility screening belongs inside the architecture rather than upstream of it, as a checkable predicate over each retrieved record. The screen described in §6.3 is six deterministic predicates over declared hours, declared employees and case components; we call it that, and not automated reasoning, because it has no decision procedure over a formal semantics, no complexity characterization, and emits no proof object. Stating a filing's admissibility as a constraint problem - given declared hours, employee count and case counts, is this record consistent with the physical and regulatory constraints? - would produce an artifact that a formal-methods reviewer would recognize, namely an unsat core naming which constraints a specific filing violates ^{[barrett2021smt]}, ^{[demoura2008]}. We have not built that, and we flag it as future work rather than as a contribution. The precedent for formalizing a body of regulation as executable logic, with the formalization kept traceable to the statutory text clause by clause, is forty years old ^[sergot1986]; we found no OSHA successor to it.

A closing note on verification that checks the wrong invariant. The benchmark corpus described in Section 7 originally reported 68 of 68 items verified against live eCFR text with zero failures. An independent audit found that one item cited 29 CFR 1910.157(d) while its anchor text actually appears in paragraph (e)(2): the verifier confirmed only that the cited paragraph markers appeared somewhere in the retrieved section, so a clause-to-anchor mismatch was structurally invisible to it. The headline "68/68, 0 failures" was true and weaker evidence than it looked. The defect was repaired, an anchor-locality check was added, and the corpus re-verified. We report this because it is the same failure as the OSHA denominator, one level up: a validation procedure that returns a clean result on a broken record, because the property it tests is not the property that matters. Leakage between predictor construction and outcome labels has the same shape and is documented inside safety machine learning specifically - Baker, Hallowell and Tixier rebuilt an earlier construction-injury prediction study with independent human annotation to eliminate artificial correlation between predictors and predictands ^{[baker2020ai]} - and across 294 papers in 17 fields more broadly ^[kapoor2023]. In each case the pipeline ran, the tests passed, and the number was wrong.

The practical requirement that follows is unglamorous and is the precondition for everything else in this paper: state the dataset, the year range, the N, the screen, and the denominator, separately, for every number. Report calibration rather than discrimination where a score will trigger an action ^{[steyerberg2010]}, ^{[collins2015tripod]}. And treat the admissibility of a retrieved record as something the system computes and exhibits, rather than something the retriever's confidence score is assumed to have covered.

11 Why the hours fail: implausible reporting is a property of establishments, not of filings#

Sections 5 and 6 establish that the exposure denominator fails, and that the failure is not stable enough to correct with a constant. Neither explains why. Two mechanisms predict opposite things. If implausible hours are keying errors, they should strike establishments independently from one year to the next, and knowing that an establishment filed implausibly in one year should tell you nothing about the next. If instead certain establishments report hours in the wrong unit as a matter of routine, because of how their filing process or payroll export is configured, bad filings should concentrate in the same establishments year after year.

The Injury Tracking Application carries a stable establishment_id, so this is directly testable. Linking the 1,238,236 distinct establishments across CY2016 to CY2024 yields 614,928 that appear in two or more years and 16,307 present in all nine.

Conditioning on consecutive-year pairs, an establishment flagged in one year is flagged again the next with probability 27.2% , against 0.95% for establishments not previously flagged. The pooled persistence odds ratio is 39.1, and every one of the eight year-pairs lies between 24.7 and 63.1.

An odds ratio is only meaningful against a null. Holding each year's number of flagged filings fixed and reassigning them uniformly at random across the establishments filing that year, over 200 permutations at seed 20260909, gives a mean odds ratio of 1.012 with a 95% interval of [0.924, 1.092]. The observed value of 39.1 lies far outside that interval, and 0% of permutations reach it.

The 2019 file, where 1.60% of filers are flagged and almost all reported hours sit in flagged records, follows the same pattern rather than standing apart. Establishments flagged in 2019 and also present in 2018 were flagged in 2018 29.3% of the time against a 2018 baseline of 1.56%, an enrichment of 18.8x; the corresponding 2020 enrichment is 16.4x.

Two qualifications matter. Cohen's kappa for the pooled transition is 0.28, which is fair rather than strong agreement, so a persistent subset coexists with genuinely one-off errors rather than replacing them. And while establishment_id is populated on every row and stable in state for 99.7% of identifiers, 313,527 of them (25.3%) carry more than one EIN across years and 108,855 (8.8%) more than one company name, so the linkage is good but not perfect and some persistence may reflect corporate restructuring rather than a single continuing filer.

The practical consequence is narrow and useful. Because the defect attaches to establishments and persists, a screen is not the only available remedy: an establishment with a consistent unit error across years is in principle repairable rather than merely excludable, and a filer-level correction is a more promising direction than any single global multiplier of the kind Section 6 rules out.

Limitations#

This paper contributes four things: a structural account of why ungrounded retrieval substitutes semantically adjacent answers in safety-critical technical question answering; an auditable encoding of a performance-influencing-factor vocabulary with an explicit crosswalk to the established frameworks it is drawn from; a rule engine whose screening output carries a derivation trace; and an open, source-verified benchmark corpus for measuring grounding.

Three of the four are instruments rather than results. The benchmark has not been run against any real system. The ontology has never been fitted to outcome data. The structural-equation work is a set of simulation studies of an estimator, not an analysis of injuries. Only the OSHA denominator analysis reports empirical findings, and its findings are about the integrity of a reported quantity rather than about safety performance.

The corpus holds 68 items across eight regulatory domains, of which 63 are factual and 5 rest on a false premise, organized into 23 complete minimal pairs and 44 question families. Every item's citation was re-checked against the eCFR versioner API, and the committed verification report records 68 of 68 verified with zero failures across 19 CFR sections. That number establishes provenance and nothing else. It says the cited paragraph exists and contains the quoted anchor. It does not say the answer key is a good answer.

Scoring is lexical, and that has a price. A paraphrase that avoids every listed surface form of a required concept is scored as a miss. The choice was made because the benchmark's subject is whether a system distinguishes near-synonymous technical entities, and scoring that with an embedding model would use as the measuring instrument the very mechanism under study. But the contrast heuristic that separates "a rupture disk has no blowdown" from an assertion of blowdown operates on a fixed character window and misfires in both directions. Any adjacent-substitution rate reported from this harness is a screen. Automatic attribution evaluation is itself unreliable ^[yue2023], which is why the established practice is a human-adjudicated subsample with inter-rater agreement reported ^{[rashkin2023, honovich2022true]}. No human adjudication has been performed here.

The four contextual categories encoded here are adopted, not invented. IDHEAS-G organizes 20 performance-influencing factors into exactly four context categories, and states directly that "PIFs are classified according to the four types of context: environment and situation, system, personnel, and task" ^[nureg2198]. Task and System are identical labels; Operational Context and Human Context are synonyms for Environment and Situation, and Personnel. The correspondence is one to one, and it maps further onto CREAM's nine common performance conditions ^{[hollnagel1998]}, SPAR-H's eight PSFs ^{[gertman2005]}, NUREG-1792's fifteen, and HSE's job, person and organisation headings ^[hse_pifs]. Presenting this vocabulary as novel would be indefensible, and we do not. What is contributed is the encoding, the crosswalk with per-row citations and asserted absences, and the traceability. That is a smaller claim, and it is the one the artifact supports.

A latent score is not a probability. It has no link function, no exposure denominator and no time window, so no threshold on it can be justified. Separately, factor scores are not uniquely determined by a fitted model: on the worked example, two equally valid sets of scores for the same factor correlate as low as 0.54 to 0.59 ^{[steiger1979, grice2001]}. A per-crew or per-shift score inherits that indeterminacy however well the model fits.

References#