vadimbelsky/esi-triage-bert-v85

Model & Data Card — ESI Triage BERT v85 (decision support, research use only)

Status: current as of 2026-06-12 (Round 1547). Supersedes the stale huggingface_space/MODEL_CARD.md (v57) and docs/v49_data_card.md (v49). Combined model + data card (standard for clinical ML). Shipped model: vadimbelsky/esi-triage-bert-v85 (HF Hub) + Space vadimbelsky/esi-triage-demo. Provenance of every number below: the frozen eval-v1 suite (9,318 records, 4 corpora, configs/eval_suite_v1_manifest.json) through the golden-tested harness, computed on the true shipped stack by scripts/eval/b3_freeze_readiness_census.py → results/b3_freeze_readiness_v85.json (Round 1527). No figure here is from memory.

---

⚠️ One-paragraph honest summary

v85 is a safety-first decision-support classifier for the Emergency Severity Index (ESI 1–5), built as a research artifact. It is not a medical device, not FDA/CE cleared, and not for clinical use. Its labels are single-nurse ESI assignments (literature κ ≈ 0.5–0.7); we have not yet validated it against adjudicated multi-rater gold, so the clinically meaningful claim — "model under-triages no more than a single nurse on adjudicated gold" — is not yet supported or refuted. What we can say, on retrospective single-nurse-labelled corpora, is that the production stack keeps per-corpus ESI 1 (most acute) recall ≥ 75% on all four eval corpora and that its residual harmful-under-triage rate sits inside the measured (synthetic) label-noise band on every corpus. Treat the model's output as an ordering aid under clinician oversight, never as ground truth.

---

1. Intended use

Task	Predict ESI acuity (1 = most acute … 5 = least) from ED triage text (chief complaint + vitals + brief history), ≤ 512 BERT tokens.
Intended user	A triage nurse or physician, as one input among several, retaining final assignment.
Intended setting	Research, benchmarking, methods development. Not point-of-care clinical use.
Primary design objective	ESI 1 recall (the safety floor), not pooled exact accuracy. Under asymmetric cost, one-tier over-triage (e.g. ESI 3 → 2) is acceptable; under-triage of a high-acuity patient is the harm to avoid.

Out of scope / do not use for

• Autonomous triage without a clinician in the loop.
• Any setting where the user cannot pause to review the basis of the recommendation (this is exactly the FDA CDS-exclusion failure mode — see §6).
• Populations the eval cannot speak to: pediatrics (≈ 0 eval coverage), and any geography/language outside the training corpora (primarily US/Boston MIMIC English + Stanford + two narrative dialects).
• Interpreting the calibrated probability as a clinical risk score.

---

2. Performance (frozen eval-v1, true shipped stack, v85 e3)

Direction-disaggregated, per results/b3_freeze_readiness_v85.json (R1527). The primary safety metric is the per-corpus harmful-under-triage rate, not pooled exact accuracy.

Corpus	n	Exact	Adjacent	Harmful-under rate	Catastrophic (gold≤2→stack≥4)	Synthetic noise floor
MIMIC-IV-ED	7,917	40.8%	78.5%	16.2% [15.4–17.0]	128	42.9%
MC-MED	1,000	55.7%	91.6%	15.9%	12	42.9%
MIETIC (narrative)	200	61.5%	77.5%	8.0%	2	27.3% (v1) / 55.0% (v3, n=40)
Lukina (narrative)	201	42.8%	85.1%	6.0%	2	9.1% (v1) / 37.5% (v3, n=40)

• 3-corpus full-stack ESI 1 recall: 95.2% (per the v85 ship memo / README). All four per-corpus ESI 1 R floors (≥ 75%) PASS.
• Read the harmful-under rate against the noise floor, not against zero. On every corpus the residual harmful-under rate is well inside the measured (synthetic) single-nurse label-noise band — i.e. the remaining error is dominated by label disagreement, not model deficiency. This is why the tactical corpus-iteration loop was deliberately frozen (see §7).
• The catastrophic residuals are mostly label noise. Hand-audits (Rounds 1064/1460/1527) found the MIMIC
  • MC-MED catastrophic records are overwhelmingly mis-labeled (censored CCs, "TEST"/"MEDICAL CLEARANCE" admin records, pyxis-only resourcing). At ship time there were 4 narrative catastrophic (2 MIETIC + 2 Lukina: active GI bleed, symptomatic bradycardia, preterm labor, paraphimosis) — real residual under-triages. Lever B (Round 1570–1572, shipped) rescues the MIETIC GI-bleed catastrophic (mietic_narrative_00438, ESI 5→2) by routing the BERT active_hemorrhage flag through the existing canonical B1 rule — net harmful-under −3, catastrophic −1, zero new harmful over-triage, all ESI 1 R floors PASS (official numbers: results/b3_freeze_readiness_v85_leverB.json). Remaining narrative catastrophic: 3 (1 MIETIC symptomatic bradycardia + 2 Lukina); these point to the perception layer / an architecture fix (§8), not more corpus.
• Exact accuracy is intentionally traded for the safety floor, especially on narrative dialects (calibration biases borderline ESI 2 → ESI 1). Do not read low exact as "inaccurate" without the direction breakdown — the misses are predominantly safe over-triage.

What these numbers are NOT

They are accuracy against single-nurse labels on retrospective research corpora. They are the proxy we optimize; they are not the clinical claim. The binding number — performance vs adjudicated multi-rater gold, and vs the single-nurse baseline on that gold — does not exist yet (Workstream C, §9).

---

3. Subgroup / fairness (D3 bias audit)

Direction-aware harmful-under-triage parity, Wilson 95% CIs, CI-disjoint flagging (scripts/eval/d3_bias_audit.py).

• Age (real, disclosed gap): on MIMIC, adult harmful-under 27.2% [26.1–28.4] vs geriatric 16.3% [14.9–17.8] — CIs disjoint, ~11pp, n=7,917. Survives within-gold-class decomposition (isolated to gold=2: adult 33.9% vs geriatric 16.0%, 18pp CI-disjoint; gold=3: 11pp). Hand-audit of 30 adult gold=2 misses: ~40% are B1 concepts hidden by MIMIC's compact-CC + meds-prefix format. This is a genuine subgroup performance disclosure, mechanistically the same compact-CC perception gap seen elsewhere.
• Sex (clean): no CI-disjoint M-vs-F harmful-under gap in any corpus (sex recovered from text; coverage 100/100/99.5/93%). Disclosed as clean sex parity on the primary safety metric.
• Structurally blocked subgroups: race/ethnicity is absent from all source files; pediatric eval coverage ≈ 0. We therefore make no race or pediatric performance claim — these gaps are at the eval set, not yet measurable. Closing them requires an eval-v2 with the relevant strata.

---

4. Model architecture

• Encoder: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext (12-layer, 768-hidden).
• Heads: 22-head multi-task v44 architecture; esi_head is the sole inference output. Auxiliary heads (symptom, flag, airway/resus safety, resource/resource-count, vitals, NER, arrival, pain, age, gestalt, …) supervise the encoder. ~15 heads carry real signal; several (vitals regression, disposition, history heads) are degenerate/dead and are documented as such (do not present them as working).
• Deterministic ESI engine: an offline handbook-v4 rules implementation used as a QA cross-check and, in production, as a bounded safety override (Step A / B1 only).
• Production stack (5 stages): BERT esi_head argmax → per-dialect temperature (T ≥ 1.0, smoothing only) → per-dialect safety thresholds (Bayes-optimal, ESI 1 R floor) → safety-head OR-rule → dialect_aware engine override. Calibration is re-derived per checkpoint (SC#51); the canonical, reproducible derivation is scripts/eval/derive_calibration_from_dump.py (reads the checkpoint's own dump, clamps T ≥ 1.0, selects T by ECE).

---

5. Training data

Source	Role	Dialect	Licensing
MIMIC-IV-ED (Beth Israel Deaconess)	bulk + nurse-gold ESI + pyxis truth	compact CC	PhysioNet credentialed, research-only DUA
MC-MED (Stanford ED)	telegraphic; ICD-inferred resources	telegraphic	research-use restrictions
MIETIC	narrative paraphrase	narrative	research
Lukina v3	Russian-physician narrative style	narrative	research (primarily eval)
medgemma-grounded synth	sparse-concept/dialect coverage; ESI inherited from real parent, never LLM-decided	mixed	capped ≤ 5% of corpus

~400K records after 512-token filtering + eval-leakage guard. ER-REASON (UCSF discharge summaries) is retained in training (per "never drop narratives") but removed from the eval slice (most exceed 512 tokens; not ED triage). Labels are single-nurse throughout — the central data limitation.

---

6. Regulatory & licensing status (from D2 scoping — counsel review required)

• Not a cleared device. Time-critical triage likely fails the §3060(a) CDS-exclusion criterion 4 (the user cannot independently review a neural-net recommendation in the seconds available), so a clinical deployment would be a device — likely Class II SaMD (US, De Novo) / Class IIb (EU MDR) and high-risk under the EU AI Act. ~18–30 months and $150k–500k to clearance. This card asserts research use only; under that posture the burden is honest labeling + DUA compliance.
• Weights inherit the data licenses. MIMIC-IV-ED's research-only DUA structurally blocks embedding the current weights in a commercial clinical product. A clinical product would require retraining on licensable/site-local data using the (clean-IP) data-craft pipeline. **The durable asset is the pipeline
  • extraction tools + evaluation harness, not the weights.**
• Weight license: released as MIT for the code/architecture, but the effective use is constrained to research by the upstream data DUAs. Do not treat the MIT tag as removing the data constraint.

---

7. The binding constraint (why we measure what we measure)

After ~1,480 rounds and ~85 self-corrections, the project's binding constraint shifted from model quality to measurement quality: the two costliest bugs were eval bugs, and the costliest metric chase was a correct number (MC-MED ESI 5 recall = 20%) interpreted wrongly (every miss was a safe over-triage). The tactical corpus loop was therefore frozen on purpose (B3): v85 already sits at the measured label-noise ceiling on all four corpora, and boundary-targeted corpus batches on a label-ceilinged model proved net-negative (3/5 such forecasts were wrong-sign). Further accuracy gains now require raising the label ceiling (Workstream C), not more corpus.

Measured (synthetic) single-nurse label-noise floors (configs/label_noise_floors_v3.json; synthetic 3-persona golden-gated panel — an UPPER bound on agreement, never reported as human agreement): MCMED/MIMIC ≈ 43% (stable), MIETIC 55% [40–69] (n=40), Lukina 37.5% [24–53] (n=40). Per-class recall headroom is bounded by ~(1 − noise); small model-vs-model deltas inside the band are uninterpretable without a direction+label hand-audit.

---

8. Known limitations & caveats (honest, specific)

• Single-rater labels. No multi-rater κ validation yet → true accuracy is unknown; reported accuracy is vs single-nurse labels at a measured ~9–55% disagreement band.
• Ontology not clinician-signed. The 207 SYMPTOM_LABELS are agent-derived; a review package (data/eval/c5_ontology_review/) is built but ED-physician sign-off is pending. Contains known duplicates (dyspnea/shortness_of_breath, an AMS triplet) and admin/lab-artifact candidates.
• Engine input ≠ documentation. The production engine is fed the BERT sym_head (which fires ~0% on compact CCs), not the documented live medspaCy extraction — so the "engine rescue net" is structurally crippled on compact-CC dialects; the esi_head, vitals, and airway/resus safety heads are the real ESI 1/2 catchers there. Restoring the documented extraction path is net-favorable in ablation but is an un-shipped, un-calibrated architecture change.
• A perception over-fire exists. The sym_head fires paraphimosis ≥ 0.5 on ~26 routine MIETIC walk-ins with no text mention (clone-overfit on the "young walk-in" format), over-triaging them in the eval stack. Fix is a post-freeze data-layer down-weight.
• Pediatrics & demographics: ≈ 0 pediatric eval coverage; race/ethnicity absent from sources. No claim made.
• Calibration provenance: v85's shipped per-dialect calibration was partly hand-tuned and is not mechanically reproducible (documented); a canonical reproducible tool now exists for future checkpoints.
• Probabilities are smoothed orderings, not risk scores.

---

9. What would make the clinical claim real (the critical path)

1. C1/C2 — adjudicated multi-rater gold. Build (done: 500-record blinded rater package + synthetic-panel proxy) → recruit 3+ clinicians → Fleiss κ + majority gold → eval-v2-adjudicated.
2. C4 — re-baseline model AND single-nurse-label vs adjudicated gold. The second number is the nurse baseline — the clinically meaningful comparator. The core claim ("model harmful-under ≤ nurse baseline on adjudicated gold, with subgroup parity and ESI 1 sensitivity ≥ 90%") can only be made here.
3. C5 — clinician ontology sign-off (package built; sign-off pending).
4. D4 — prospective shadow-mode at ≥ 1 site vs adjudicated retro-review + hard outcomes (admission, ICU, 72h mortality). This is the study that makes the claim deployable.

Until 1–4 exist, every accuracy number in this card is a proxy, not the goal.

---

10. Citation & maintenance

@misc{esi_triage_v85_2026,
  title  = {ESI Triage BERT v85 — safety-first BiomedBERT decision-support classifier
            with per-dialect calibration and a deterministic engine safety floor},
  author = {Belski, Vadim},
  year   = {2026},
  url    = {https://huggingface.co/vadimbelsky/esi-triage-bert-v85},
  note   = {Research use only. Single-nurse labels; not validated vs adjudicated multi-rater gold.
            Evaluated on the frozen eval-v1 suite (MIMIC-IV-ED, MC-MED, MIETIC, Lukina; n=9,318)
            via a golden-tested harness with direction-aware safety metrics.}
}

• Refresh triggers: a new shipped checkpoint; adjudicated gold lands (re-baseline §2/§3/§9); a D1 deployment-posture decision; clinician ontology sign-off.
• Sources for this card (all in-repo): results/b3_freeze_readiness_v85.json, configs/eval_suite_v1_manifest.json, configs/label_noise_floors_v3.json, tasks/d2_regulatory_scoping.md, tasks/clinical_readiness_master_plan.md (D3), data/eval/c5_ontology_review/, and the project memories.