Scribeski intake classifier (r3)
Reads a whole behavioral-health session transcript and answers 62 choice fields of an intake form in a single forward pass: housing and household, work and income, substances, medications, PHQ-9 and GAD-7 items, suicide and violence risk items, daily living, referrals and follow-up. Every answer is either a filled value or blank, meaning needs review. The social worker fills blanks, as they do today.
It is built to be safe, not complete. On a hand-labelled test of real counselling speech it was never confidently wrong in a risk field, and it leaves most facts for the clinician.
Use
pip install mlx mlx-lm transformers
python classify.py session.txt # filled fields + how many are left for review
python classify.py session.txt --json # {field: {value, confidence, status}}
Transcript format: [mm:ss] WORKER: … / [mm:ss] CLIENT: … lines; # header lines are ignored.
Everything the classifier needs (prompt, questions, decoding thresholds) is frozen in
classifier.json. Runs in about 2 s per session on a 32 GB Apple M6 (8-bit MLX, 1.7 GB).
How it works
- Prompt: rules, then the transcript, then one question per field (multi-select fields
become one yes/no question per option), each ending in
Answer:. - Scoring, not generation: the logits at every
Answer:position, restricted to that question's option letters, give the answer distribution. There is no decoding step. - Abstaining: answers below 0.9 confidence (temperature 1.25, both fit on validation) are left blank.
- App-supplied, not classified:
session_type. - Answers that fit no option are left blank for review. Example: active suicidal thoughts where the plan was never assessed.
Evaluation (sealed test: 30 real counselling sessions, AnnoMI, hand-labelled)
| result | |
|---|---|
| wrong answers in risk fields | 0 |
| wrong answers overall | 7 of 1,860 (0.38%) |
| established facts answered correctly | 14 of 77 (18%); the other 63 left for review |
| precision when it answers | 14 of 21 (67%); 3 of the 7 misses are arguable strict labels |
| synthetic validation | 98.1% correct, 0.5% wrong |
The facts it misses are mostly people and circumstances mentioned in passing: who the client
lives with, supports, and protective factors. Full reports:
eval/classifier-round3-2026-09-23.md (and rounds 1–2) in the Scribeski repository.
Training
Full fine-tune of Qwen3-1.7B at 32k context on 1,207 synthetic behavioral-health sessions: 2 epochs on 1× H100, epoch 2 selected by validation loss. The sessions were written by six open-weight models from sampled ground-truth sheets, and labels were kept only where a blind DeepSeek pass agreed with the sheet. No client data was used. The dataset is published separately as Scribeski Intake Dialogues (CC BY 4.0).
Authors
Kevin Loo, Columbia University · kevin@loo.ski
Limitations
- Low coverage on real speech. It leaves roughly 80% of established facts for review.
- Not validated by clinicians. Not for clinical decision-making. Risk fields must always be reviewed by the clinician.
Quantized