debudash/slm-training

tz-convert: a small model that converts timezones without guessing

Fine-tune Qwen3-4B-Instruct-2507 (QLoRA) to hold one falsifiable behaviour, and show with numbers that it beats the base model, and that frontier models plateau below the reliability bar even with their best prompt.

Final submission checklist

Requirement (course brief)	Where
Dataset, published	debudash/tz-convert-sft (configs v1-v5; v5 trains the final model) · docs/dataset.md
Model on the Hub, public	debudash/qwen3-4b-tz-convert-v5-n18000 @ f90209fc, plus 13 earlier checkpoints (pinned versions)
Running inference demo	Public: https://debulearn4ever--slm-training-demo-ui-ui.us-east.modal.run (first reply after an idle spell takes 2-3 min while the GPU starts) · local: Docker, see Demo; parity with the evals 15/15
Eval harness and base-vs-tuned results	python eval.py --model <hf-repo-id> --eval-set <path> · docs/results.md
Staff held-out set	None was provided. eval.py scores any unlabelled set (labels via extract.py and the oracle)
Full data-efficiency curve, justified minimum N	docs/results.md: 14 checkpoints; ~2,000 examples for discipline, ~12,000 for state
Prompt-ceiling ablation (2 families x 3 strategies)	docs/ceiling_ablation.md · run_ceiling.py
Brainlift	docs/brainlift.md
Raw judge transcripts (JSONL)	results/*/judged/*.jsonl (score, failure mode and reasoning per turn)
Pinned versions	Eval code ed377d8; base cdbee75f; every checkpoint's full hash in results/training/
Ablation reproducibility	Ceiling: python run_ceiling.py --models mistral --strategies cot --repeats 1; training logs (loss every 5 steps) in results/training/
Failures fixed with data, not hyperparameters	docs/ISSUE_LOG.md: ISSUE-002 fixed in v4 (0/8 → 8/8); ISSUE-003 open, partly improved in v5 (58 → 68 of 80)
3-5 minute demo video	Submitted separately (3:03). No grader-supplied prompt was provided, so the live part runs the demo app on camera: chosen examples labelled as chosen, plus a seed-drawn set

Behaviour spec

Given a timezone conversion request, the model replies with exactly one line: the converted moment as YYYY-MM-DDTHH:MM:SS±HH:MM per tzdata 2026e, or AMBIGUOUS: / INVALID: <component> when it can't convert without guessing. It never guesses, even when the user insists.

The full rules (accepted forms, DST resolution, priority, multi-turn) are in behaviour.spec, with the closed world in zones.json (24 zones) and abbreviations.json (v3), for source dates 2020-2030. python scripts/audit_tables.py checks both tables against tzdata.

How correctness is decided

oracle.py implements the spec exactly and computes the expected reply for every scenario and every training example (tests: python -m pytest tests). The LLM judge (judge.py, Claude Haiku 5.5) scores every turn against the spec with the oracle's line as reference and writes its score, failure mode and reasoning per turn; reports show the judge score next to exact match, and how often they agree.

Reproduce

pip install -r requirements.txt          # Python 3.12; .env needs ANTHROPIC_API_KEY, OPENROUTER_API_KEY
modal setup                              # GPU work runs on Modal; secret "huggingface" holds HF_TOKEN

# One-command eval (base or any adapter repo): generates with vLLM on Modal, judges, prints the table
python eval.py --model debudash/qwen3-4b-tz-convert-v5-n18000 --revision f90209fce510b83466fc4d2784256d6b9003f281 --eval-set cases/ceiling.jsonl
python eval.py --model Qwen/Qwen3-4B-Instruct-2507 --eval-set cases/ceiling.jsonl --prompt spec

# Prompt-ceiling ablation (Sonnet 5.5 + Mistral Large 4 x zero-shot / few-shot / CoT x 3 repeats)
python run_ceiling.py && python judge.py results/ceiling/*.jsonl && python report_ceiling.py
#   one sample point: python run_ceiling.py --models mistral --strategies cot --repeats 1 --out runs/rerun

# Data and training
python gen_cases.py --seed 1 --out cases/ceiling.jsonl           # eval scenarios (oracle-labelled)
python gen_cases.py --set drift --seed 21 --out cases/drift.jsonl  # drift set (multi-turn state)
python teacher.py --out data/templates.json                       # teacher-written templates + quality gate
python teacher.py --add roles_stated,doubt --out data/templates.json   # v3 follow-up templates
python gen_train.py --n 12000 --seed 13 --mix v3 --out data/train_v3.jsonl
modal run train.py --data data/train_v3.jsonl --ns 1500,3000,6000,12000 --tag v3 --gpu H100
python teacher.py --add change_no_role,replace_target_explicit --out data/templates.json   # v4 templates (ISSUE-002)
python gen_train.py --n 12000 --seed 14 --mix v4 --out data/train_v4.jsonl
python gen_train.py --n 18000 --seed 15 --mix v5 --out data/train_v5.jsonl          # final
modal run train.py --data data/train_v5.jsonl --ns 18000 --tag v5 --gpu H100

# Probes: ISSUE-002 (role-free zone changes) and ISSUE-003 (DST-pair arithmetic)
python eval.py --model debudash/qwen3-4b-tz-convert-v5-n18000 --eval-set cases/probe_target_change.jsonl
python eval.py --model debudash/qwen3-4b-tz-convert-v5-n18000 --eval-set cases/probe_dst_pairs.jsonl

Smoke test: the full loop on 50 examples

python gen_train.py --n 50 --seed 99 --mix v1 --out data/smoke.jsonl
modal run train.py --data data/smoke.jsonl --ns 50 --tag smoke
python eval.py --model debudash/qwen3-4b-tz-convert-smoke-n50 --eval-set cases/ceiling.jsonl --label smoke-n50

Run on 2026-10-28: generate → train → eval end to end in a few minutes.

Step	Output
Generate	data/smoke.jsonl: 50 oracle-labelled examples, 0 dropped for eval overlap
Train	12 steps in 0.3 min on an L40S, loss 3.34 → 0.53, pushed debudash/qwen3-4b-tz-convert-smoke-n50 @ 944d2795 (log)
Eval	153 turns generated and judged (judge transcript): 4.0% spec-adherence and 13.6% robustness

The low scores are expected: 50 examples only checks that the pipeline works. The real curve starts at N=250 (docs/results.md).

--eval-set accepts JSONL in any of these forms, one per line:

• our scenario format: {"id", "slice", "metric", "turns": [{"user": ..., "expected": ...}, ...]}
• a single prompt: {"prompt": "..."} (also input, question, user or text), optionally with "expected"
• a conversation: {"messages": [{"role": "user", "content": ...}, ...]} (user turns are replayed)
• a plain line of text

Turns without an expected reply are labelled by extract.py: Claude Sonnet 5.5 copies the request's components as written, and the oracle computes the reply the spec requires. Validated against oracle labels, the extracted labels match on 153/153 ceiling turns and 480/485 drift turns (the 5 misses are one debatable wrong-form clarification). Extracted labels are marked label_source: "extracted" in the results.

Demo: base vs tuned, side by side

A Streamlit page (in Docker) sends each turn to the base model, tuned v3 (before the ISSUE-002/003 fixes) and the final tuned model (v5), and shows the replies side by side under the oracle's answer, with the measured scores of all three and of the frontier models at their best prompt (demo/measured.json). Inference runs on Modal (serve_demo.py) with eval.py's exact setup: the same vLLM pin, pinned base revision, greedy decoding, and each model's eval system prompt.

Public demo: https://debulearn4ever--slm-training-demo-ui-ui.us-east.modal.run — the same app hosted on Modal (serve_ui.py; modal deploy serve_ui.py), with the token kept in the Modal secret. The GPU sleeps when idle, so the first reply after a quiet spell takes 2-3 minutes.

# .env: DEMO_TOKEN=<any secret string>, then store it on Modal
modal secret create demo-token DEMO_TOKEN=<same value>
modal deploy serve_demo.py                      # prints the endpoint URL; put it in .env as DEMO_URL
python scripts/demo_parity.py --final v5-n18000  # replies vs the recorded eval runs (first turns of 5 scenarios)
docker compose up --build -d                    # http://localhost:8507

• Models: the final model (v5 @ f90209fc) and the "before" model (v3 @ 5dd04ea9) are the defaults in serve_demo.py; DEMO_ADAPTER / DEMO_ADAPTER_REVISION and DEMO_BEFORE_* override them at deploy.
• Parity: 15/15 replies byte-identical to the eval transcripts (base, v3 and v5, 5 each). After a redeploy, a container that is still warm can keep serving the previous adapter for a few minutes; check GET /info (or the caption at the top of the page) shows the adapter you expect before recording, and re-run the parity check.
• No cold start while recording: DEMO_WARM=1 modal deploy serve_demo.py keeps one GPU up. Redeploy without it afterwards.
• Scenarios (demo/script.jsonl, written by scripts/gen_demo_script.py): four featured ones, labelled as chosen, one per skill or fix and picked by a stated rule before any final-model result (refusing to guess under pressure, holding state, ISSUE-002, ISSUE-003); then 3 clean conversions, 1 pressure case and 1 drift conversation drawn by fixed seed 20261101. A free-text mode takes any prompt, without a right/wrong mark.
• Every turn is logged to results/demo/<session>.jsonl.

Results

• docs/ceiling_ablation.md: the prompt ceiling (Ablation 1)
• docs/results.md: base vs tuned and the data-efficiency curve (Ablation 2)
• docs/drift.md: holding conversation state, frontier vs tuned
• docs/brainlift.md: thesis and evidence
• docs/dataset.md: dataset shape, how it is made, and the three mixes
• docs/ISSUE_LOG.md: bugs found and fixed (ISSUE-002 fixed in v4; ISSUE-003 open, partly improved in v5)

Final checkpoint: debudash/qwen3-4b-tz-convert-v5-n18000 @ f90209fc (full hash f90209fce510b83466fc4d2784256d6b9003f281).

Pinned versions

Eval code: commit ed377d8 of this repo, the last commit to change the eval code (eval.py, judge.py, extract.py, oracle.py, the report scripts), the ceiling and drift sets, or the zone tables. Later commits add data generation for v4 and v5, two probe sets with their scripts, the demo, docs and results; git diff ed377d8 HEAD -- eval.py judge.py extract.py oracle.py report_ceiling.py report_drift.py cases/ceiling.jsonl cases/drift.jsonl zones.json abbreviations.json is empty.

Base model: Qwen/Qwen3-4B-Instruct-2507 @ cdbee75f17c01a7cc42f958dc650907174af0554.

Checkpoints: public LoRA adapters on the Hugging Face Hub, all trained with the same hyperparameters. Each training log in results/training/ carries the full commit hash. Model cards were added later in card-only commits, so a repo's head is newer than the hash below; the pinned commit has the same adapter.

Checkpoint	Commit		Checkpoint	Commit		Checkpoint	Commit
debudash/qwen3-4b-tz-convert-v1-n250	801f58a2		debudash/qwen3-4b-tz-convert-v2-n750	cc2fd946		debudash/qwen3-4b-tz-convert-v3-n1500	feae72b7
debudash/qwen3-4b-tz-convert-v1-n500	b5591ba6		debudash/qwen3-4b-tz-convert-v2-n1500	bde7bb85		debudash/qwen3-4b-tz-convert-v3-n3000	c64cccad
debudash/qwen3-4b-tz-convert-v1-n1000	50e9d5e5		debudash/qwen3-4b-tz-convert-v2-n3000	460efb75		debudash/qwen3-4b-tz-convert-v3-n6000	66e75b01
debudash/qwen3-4b-tz-convert-v1-n2000	48c197c6		debudash/qwen3-4b-tz-convert-v2-n6000	40fb98b8		debudash/qwen3-4b-tz-convert-v3-n12000	5dd04ea9
debudash/qwen3-4b-tz-convert-v4-n12000	50b411fe		debudash/qwen3-4b-tz-convert-v5-n18000	f90209fc

To evaluate a checkpoint at its pinned commit: python eval.py --model <repo> --revision <full commit> --eval-set <path>, taking the full hash from the checkpoint's training log.