felipe53/slm-deid-name-judgment

slm-deid — context-sensitive personal-name judgment (Qwen3-1.7B QLoRA)

A 4-bit QLoRA adapter over unsloth/Qwen3-1.7B-unsloth-bnb-4bit fine-tuned for one narrow, safety-critical behavior: deciding whether a name in educational text refers to a real person, and tagging only those spans. Given a short passage it returns the text byte-identical except that person-name spans are wrapped in ⟨NAME⟩…⟨/NAME⟩ — and nothing else is tagged.

This is the judgment core of de-identification — the part regex and constrained decoding structurally cannot do. Pattern PII (email / phone / SSN / IDs / URLs) and output format are out of scope here; they are owned by the deterministic pipeline that surrounds this model.

• Canonical run: sft-v3-gpt551 — trained on data from a live teacher + independent verifier (not in-session templates). This is the credible reference line.
• Repo: https://github.com/f15cubing/slm-deid — full eval harness, dataset, and methodology.

The behavior, by example

Passage	Correct call
"Newton was frustrated when the experiment failed."	tag Newton (person)
"We applied the Newton method to approximate the root."	don't tag (method)
"I visited Chelsea last summer and loved it."	don't tag (place)
"Chelsea helped me revise my thesis statement."	tag Chelsea (person)
"thanks, Sam — that explanation finally clicked"	tag Sam (first-name-only person)

Same token, opposite call, decided by context. Run the side-by-side yourself with python -m src.demo --adapter <this-adapter>.

Base vs. tuned — the headline result

Evaluated on 51 quarantined hard cases (ambiguous names in context). Base = the prompted 4-bit Qwen3-1.7B on the same backend; tuned = base + this adapter. The comparison is prompt vs. fine-tune, which is the assignment's baseline. Full tables + 95% bootstrap CIs: docs/results.md → gpt551.

Metric	Prompted base	Fine-tuned (gpt551)
F5 (recall-weighted F)	0.507	0.847
Recall	0.518	0.852
Over-tag rate ↓	0.549	0.157
Integrity violation ↓	0.588	0.020
Leakage rate ↓	0.255	0.078
Pass rate	0.353	0.824
Consistency	0.375	0.562

Per-category: person_vs_eponym recall 0.33 → 1.00, person_vs_place recall → 1.00, negative_trap pass 0.40 → 1.00. Generalization to names never seen in training was verified on a held-out-names probe (base→tuned recall 0.08 → 1.00) and an OOD probe (recall 0.05 → 0.89) — the judgment generalized, it did not memorize.

For context, a frontier API model (gpt-4.1) scored pass 0.88 on the same 51 hard cases; the 1.7B tunes are competitive with it and actually beat it on recall (the frontier model is precision-first and under-recalls). See docs/eval-engine-comparison.md.

Training setup

Base model	unsloth/Qwen3-1.7B-unsloth-bnb-4bit (4-bit, non-thinking mode)
Method	QLoRA (unsloth + bitsandbytes 4-bit), bfloat16 compute (A100/Ampere)
LoRA	r=32, α=32, dropout 0.0; q/k/v/o + gate/up/down proj
Optim	lr=2e-4, 3 epochs, eff. batch 16 (2×8), linear schedule, warmup 0.05, wd 0.01, seed 0
Masking / seq	completion-only masking; max seq 2048
Config	configs/train.yaml — frozen since Day 4 (data-only iteration, no HP tuning)
Run	156 steps / 3 epochs, loss 0.474 → ~2e-4, no NaN

Config was frozen deliberately: a project rule forbids touching hyperparameters to paper over data problems — failure modes were fixed by generating targeted data, not by tuning lr/r/epochs.

Training data

818 train / 90 val (sft-v3-gpt551 splits). Generated by a live OpenAI-compatible teacher (via the TrueFoundry LLM Gateway) writing passages, plus an independent verifier pass that rejects teacher/verifier disagreements. Built from matched minimal pairs (person_vs_common / _place / _eponym / possessive) so the model sees the same token used both ways. Dataset card: docs/dataset-card-v3.md.

Quality-gate + leakage drops: 134 verifier-disagreement, 98 eval-surface-leak, 48 negative-trap-has-name, plus a few label-integrity rejects.

Eval-leakage — a hard ceiling, verified clean

The quarantined evaluation sets are physically separate and were never fed to the teacher, augmentation, or training. Independently re-verified post-hoc: 0 exact and 0 substring overlaps between the 818/90 training splits and all 201 quarantined eval inputs (hardcases / adversarial / held-out-names / OOD). Enforced in CI by tests/test_no_eval_leakage.py.

Intended use

• De-identifying personal names in short English educational text (essays, chat, dialogue, narrative, exposition), as the judgment stage of a larger de-id pipeline.
• Research / educational demonstration of reliability engineered through data vs. capability accessed through prompting on a narrow behavior.

Limitations & out-of-scope

1. Names only. No address / DOB / MRN / other PII categories (those are pipeline-side).
2. Byte-identity on messy real text. On clean text integrity is ~0.98; on very messy real prose (collapsed whitespace, zero-width chars) the "regenerate the passage verbatim" output format can drift — the surrounding pipeline's tag-by-offset projection is the fix (unwrap(project(...)) == original by construction). Prefer the projected path in production.
3. Possessive contrast is the residual weak spot (over-tag/integrity on possessive, n=3).
4. Small eval (n=51), single seed, single teacher+verifier. Per-category cells are noisy.
5. English + educational register only. Not validated on clinical/legal text or other languages.
6. Not a certified de-identification system; do not deploy on regulated data without human review.

How to use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen3-1.7B"  # or the 4-bit unsloth base on CUDA
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE)
model = PeftModel.from_pretrained(model, "<this-repo-id>")

# Easiest path — the repo's Tagger contract (handles the non-thinking chat template):
#   from src.infer import load_hf_tagger
#   tagger = load_hf_tagger(adapter="<adapter-path>")
#   tagger.tag("Chelsea helped me revise my thesis.")
#   → "⟨NAME⟩Chelsea⟨/NAME⟩ helped me revise my thesis."

The model expects the system prompt in src/common/prompts.py and Qwen3's non-thinking chat template. See src/infer.py and src/demo.py in the repo.

Provenance & reproduction

• Adapter: sft-v3-gpt551 (r=32/α=32, checkpoint-156).
• Reproduce on Colab: notebooks/v3_colab_train_eval.ipynb — live teacher via --provider openai with OPENAI_BASE_URL/TEACHER_MODEL → frozen configs/train.yaml QLoRA → base-vs-tuned on eval/hardcases.
• Source model cards (per run): docs/model-card-gpt551.md, docs/model-card-v3.md.
• License: Apache-2.0 (inherits the Qwen3-1.7B base license).