dilr/mira-3-v11

Mira-3 v11 — Clinical Structured Extraction (Qwen3-4B, QLoRA)

Mira-3 reads a clinical document (lab report, discharge summary, intake form, progress note, …) and returns a single schema-valid, source-grounded JSON object. It is a LoRA adapter over Qwen/Qwen3-4B-Instruct-2507, built to run on-prem / offline on a customer's own GPU — privacy is the point.

This repo is the v0 artifact: trained on synthetic + permissively-licensed public data only (no PHI, no DUA/research-only data). It proves the pipeline; v1 will retrain on a design partner's real documents.

⚠️ Read the honest eval below before using. v0 is excellent on validity and grounding but has a real-world recall blind spot on narrative diagnoses/procedures. See MIRA3_LEARNINGS.md for the full analysis and how to fix it.

Files

• adapter/ — the shipping adapter (epoch 3, best by eval)
• epochs/epoch_{1,2,3}/ — per-epoch adapters (for reproducibility / best-epoch verification)
• training/metrics.json, training/env_versions.json — training run + exact dependency stack
• eval/probe_scorecard_v11.json — real-doc probe validity + hallucination + leak
• eval/phase25_gate_v11.json — Phase-2.5 exit gate verdict (PASS)
• eval/realdoc_eval_v11.json — the honest labeled real-doc field-F1 (with adjudicated gold)
• MIRA3_LEARNINGS.md — detailed, reusable playbook (generalizes to other extraction/OCR verticals, e.g. résumé parsing)

Training

Base	Qwen/Qwen3-4B-Instruct-2507 (Unsloth 4-bit)
Method	QLoRA (4-bit + LoRA r16/α32, dropout 0.05), Unsloth, sequence packing
Data	21k rows: 15k ladder + 3k schema-variant + 3k PII-abstention (synthetic + public)
Schedule	3 epochs, lr 2e-4 cosine, warmup 3%, effective batch 16, max_seq 3072
Compute	Kaggle T4, 2 sessions (~20.7h) via lossless checkpoint/resume
Loss	1.363 → 0.133 (clean cosine decay; natural completion at 867/867 steps)

Evaluation (honest)

Phase-2.5 gate: PASS. v11 beats the prior production model (Mira-Q2) on all real-doc probes for both JSON validity (100%) and hallucination.

Probe (real docs)	JSON validity	Hallucination (↓ better)
mtsamples_282	1.00	0.28
extraction_relevant_150	1.00	0.52
synthetic_v2_150	1.00	0.13
identifier leak	—	0 everywhere

Labeled real-doc field-F1 (the honest generalization metric): 0.61 — on 24 diverse mtsamples docs with double-annotated + adjudicated gold. (The synthetic in-distribution test set scores 1.00 and is not a generalization signal — see learnings §4c.)

Per-field: high precision (0.89–0.98) everywhere, but uneven recall. Strong on allergies (F1 0.99), encounter date/dept (0.96/0.88), vitals (0.84), medications (P 0.98). Weak on diagnoses (recall 0.10) and procedures (recall 0.33).

Limitations

• Diagnoses/procedures recall on narrative text is poor. The model extracts these well from structured Diagnoses:-style blocks (as in the synthetic training data) but misses them when stated in prose (real clinical notes). This is a data-coverage issue, targeted for v1. Do not rely on v0 for complete diagnosis capture from free-text notes.
• Every output is a draft for a human and must be routed through the verifier (schema + grounding + identifier-leak) before use. Never an autonomous decision.
• v0 is trained on synthetic/public data; real-world distribution shift is expected until the v1 retrain.

Usage

import torch
from unsloth import FastLanguageModel  # load with Unsloth, NOT vanilla PeftModel (4-bit quant mismatch)

model, tok = FastLanguageModel.from_pretrained(
    "dilr/mira-3-v11/adapter", max_seq_length=4096, dtype=torch.float16, load_in_4bit=True)
FastLanguageModel.for_inference(model)

SYSTEM = ("You are a clinical information extraction system. Read the clinical document and "
          "output a single JSON object matching the schema. Extract ONLY information explicitly "
          "stated ... Output valid JSON only - no prose, no markdown.")  # full prompt in the training data

msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": document_text}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to("cuda"), max_new_tokens=2048, do_sample=False)
raw = tok.decode(out[0], skip_special_tokens=True)
# NOTE: Qwen3 prepends a <think>...</think> block — strip it before json.loads():
import re, json
pred = json.loads(re.sub(r"^\s*<think>.*?</think>\s*", "", raw.split("assistant")[-1], flags=re.DOTALL))

License & data

LoRA adapter over Qwen3-4B (base: Apache-2.0). Internal dilr.ai artifact; not for redistribution. Trained on synthetic + permissively-licensed public data; contains no PHI and no patient identifiers. Eval derived from the public MTSamples corpus.