francescortu/detectdistill-v5-lora-students

DetectDistill v5 — LoRA students distilled from gpt-oss-120b and GLM-5.3 reasoning traces

Research artifacts of the DetectDistill project, experiment exp_controlled_distill_easy_v5_2026-10-09 ("controlled-distill-easy-v5"). Each folder <run>/epochN/ is a LoRA adapter (r=32, alpha=64, dropout 0.05 on q/k/v/o/gate/up/down) plus two trained token rows (<think>, </think>) on a frozen instruct base model, saved after epoch N of a five-epoch SFT on filtered SuperGPQA teacher traces (assistant-only loss on <think>reasoning</think>answer<end-of-turn>). Every completed epoch of every v5 run is published, including runs whose students got worse; nothing is selected by outcome. Card generated 2026-10-19 10:15 CEST.

Data, evaluation sets, per-question evaluation outputs and the campaign records are in the dataset repo francescortu/detectdistill-v5-data. Code: https://github.com/francescortu/DetectDistill (branch v2-orfeo), src/exp_controlled_distill_easy_v5_2026-10-09.

Contents of a run folder

• epochN/adapter_model.safetensors, adapter_config.json: PEFT 0.21 LoRA + trainable_tokens_delta rows (tied models store one row set on embed_tokens, untied models also on lm_head).
• epochN/tokenizer.json, tokenizer_config.json, chat_template.jinja: the student tokenizer with <think>/</think> added (last two ids).
• epochN/training_state.pt: optimizer, scheduler, step and RNG state (resume with train.py --resume N); epoch.json, DONE.
• run_info/: base_model.json (exact base revision), train_timing.json (examples, dropped over-length ids, tokens, wall time, per-epoch loss), trainable_parameters.json, student_resolved.json, length statistics, over-length decision, and results/ (per-epoch evaluation summaries on both held-out sets, final reports, GPU ledger).
• UPLOAD_VERIFIED.json (per run) and UPLOAD_MANIFEST.json (repo): sha256/size inventory verified against the remote revision. Merged full-weight copies are not uploaded (regenerable from the pinned base + adapter).

Runs

Correct answers on the 200-question in-distribution SuperGPQA set and on the 100-question mixed held-out (34 MATH, 34 Codeforces, 32 SuperGPQA), base → epochs 1-5. Single sample per question at T=0.8: the same base model scored up to 9/200 apart in separate evaluations, so small differences are noise.

Run	Student	Teacher data	Trace text	In-dist 200	Mixed 100	Rephrasing eligibility
run_v1_2026-10-09	Llama-3.1-8B-Instruct	gpt-oss-120b (effort medium)	original	39 → 49/53/56/61/62	42 → 36/39/36/38/36	eligible
run_v3_2026-10-13	Llama-3.1-8B-Instruct	GLM-5.3 (effort high)	original	39 → 50/57/46/56/50	42 → 36/38/34/36/35	eligible
run_v4_2026-10-13	Mistral-7B-Instruct-v0.3	gpt-oss-120b (effort medium)	original	26 → 33/37/42/41/49	16 → 14/15/15/16/17	eligible
run_v6_2026-10-14	Mistral-7B-Instruct-v0.3	GLM-5.3 (effort high)	original	26 → 40/35/39/43/45	16 → 18/12/9/14/18	eligible
run_v5_2026-10-13	Qwen2.5-3B-Instruct	gpt-oss-120b (effort medium)	original	60 → 37/47/45/46/52	41 → 30/29/30/30/36	ineligible
run_v8_2026-10-14	Qwen2.5-3B-Instruct	GLM-5.3 (effort high)	original	60 → 41/33/36/33/33	35 → 29/28/30/34/32	ineligible
run_v7_2026-10-14	Qwen2.5-0.5B-Instruct	gpt-oss-120b (effort medium)	original	17 → 22/31/23/33/26	20 → 11/4/8/9/7	eligible
run_v11_2026-10-14	Qwen2.5-0.5B-Instruct	GLM-5.3 (effort high)	original	26 → 20/22/23/26/28	20 → 11/12/12/10/11	eligible
run_v9_2026-10-14	Llama-3.2-1B-Instruct	gpt-oss-120b (effort medium)	original	10 → 35/28/34/30/24	13 → 14/14/17/23/11	eligible
run_v10_2026-10-14	Llama-3.2-1B-Instruct	GLM-5.3 (effort high)	original	10 → 34/21/37/27/33	11 → 18/18/19/22/26	eligible
run_v12_2026-10-15	Falcon3-7B-Instruct	gpt-oss-120b (effort medium)	original	44 → 49/54/51/58/57	38 → 35/37/36/39/38	eligible
run_v13_2026-10-15	Falcon3-7B-Instruct	GLM-5.3 (effort high)	original	46 → 55/41/44/38/40	35 → 31/27/31/30/33	eligible
run_v14_2026-10-15	gemma-3-1b-it	gpt-oss-120b (effort medium)	original	3 → 20/17/28/22/34	28 → 9/8/8/8/9	excluded_by_user
run_v15_2026-10-15	gemma-3-1b-it	GLM-5.3 (effort high)	original	6 → 23/21/23/32/37	26 → 11/8/14/6/11	excluded_by_user
run_v16_2026-10-15	gemma-3-4b-it (text decoder)	gpt-oss-120b (effort medium)	original	33 → 27/43/35/44/41	52 → 32/37/36/41/38	excluded_by_user
run_v17_2026-10-15	gemma-3-4b-it (text decoder)	GLM-5.3 (effort high)	original	33 → 32/28/31/31/35	53 → 23/25/26/24/24	excluded_by_user
run_v18_2026-10-16	Llama-3.1-8B-Instruct	gpt-oss-120b (effort medium)	MiMo-V2.5 rephrased (of run_v1_2026-10-09)	base reused → 46/67/61/64/51	base reused → 33/37/37/37/32	new run
run_v19_2026-10-16	Llama-3.1-8B-Instruct	GLM-5.3 (effort high)	MiMo-V2.5 rephrased (of run_v3_2026-10-13)	base reused → 36/31/51/53/49	base reused → 34/39/40/35/33	new run
run_v20_2026-10-16	Mistral-7B-Instruct-v0.3	gpt-oss-120b (effort medium)	MiMo-V2.5 rephrased (of run_v4_2026-10-13)	base reused → 40/43/36/36/35	base reused → 18/14/16/12/15	new run
run_v21_2026-10-16	Mistral-7B-Instruct-v0.3	GLM-5.3 (effort high)	MiMo-V2.5 rephrased (of run_v6_2026-10-14)	base reused → 39/39/32/38/34	base reused → 13/14/15/11/8	new run
run_v22_2026-10-16	Qwen2.5-0.5B-Instruct	gpt-oss-120b (effort medium)	MiMo-V2.5 rephrased (of run_v7_2026-10-14)	base reused → 21/20/21/23/26	base reused → 12/7/15/11/14	new run
run_v23_2026-10-16	Qwen2.5-0.5B-Instruct	GLM-5.3 (effort high)	MiMo-V2.5 rephrased (of run_v11_2026-10-14)	base reused → 21/22/22/20/26	base reused → 14/14/12/10/11	new run
run_v24_2026-10-16	Llama-3.2-1B-Instruct	gpt-oss-120b (effort medium)	MiMo-V2.5 rephrased (of run_v9_2026-10-14)	base reused → 31/33/24/26/28	base reused → 17/11/15/24/14	new run
run_v25_2026-10-16	Llama-3.2-1B-Instruct	GLM-5.3 (effort high)	MiMo-V2.5 rephrased (of run_v10_2026-10-14)	base reused → 40/24/34/26/36	base reused → 18/22/19/15/17	new run
run_v26_2026-10-16	Falcon3-7B-Instruct	gpt-oss-120b (effort medium)	MiMo-V2.5 rephrased (of run_v12_2026-10-15)	base reused → 51/50/52/51/52	base reused → 29/40/37/38/37	new run
run_v27_2026-10-16	Falcon3-7B-Instruct	GLM-5.3 (effort high)	MiMo-V2.5 rephrased (of run_v13_2026-10-15)	base reused → 48/39/57/44/43	base reused → 29/31/38/30/33	new run
run_v28_2026-10-16	gemma-3-1b-it	gpt-oss-120b (effort medium)	MiMo-V2.5 rephrased (of run_v14_2026-10-15)	base reused → –/–/–/–/–	base reused → –/–/–/–/–	new run
run_v29_2026-10-16	gemma-3-1b-it	GLM-5.3 (effort high)	MiMo-V2.5 rephrased (of run_v15_2026-10-15)	base reused → –/–/–/–/–	base reused → –/–/–/–/–	new run
run_v30_2026-10-16	gemma-3-4b-it (text decoder)	gpt-oss-120b (effort medium)	MiMo-V2.5 rephrased (of run_v16_2026-10-15)	base reused → –/–/–/–/–	base reused → –/–/–/–/–	new run
run_v31_2026-10-16	gemma-3-4b-it (text decoder)	GLM-5.3 (effort high)	MiMo-V2.5 rephrased (of run_v17_2026-10-15)	base reused → –/–/–/–/–	base reused → –/–/–/–/–	new run

"Rephrasing eligibility" is the campaign's exploratory selection rule (max(E1..E5) > base on the in-distribution set), not a significance test.

Restore an adapter

import json, torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
REV = "<revision sha from UPLOAD_MANIFEST.json>"; RUN = "run_v4_2026-10-13"; EP = "epoch5"
d = snapshot_download("francescortu/detectdistill-v5-lora-students", revision=REV, allow_patterns=[f"{RUN}/{EP}/*", f"{RUN}/run_info/*"])
info = json.load(open(f"{d}/{RUN}/run_info/base_model.json"))
tok = AutoTokenizer.from_pretrained(f"{d}/{RUN}/{EP}")
base = AutoModelForCausalLM.from_pretrained(info["model"], revision=info["revision"], torch_dtype=torch.bfloat16)
if len(tok) > base.get_input_embeddings().weight.shape[0]:
    base.resize_token_embeddings(len(tok))      # NOT for Gemma 3 (see below)
model = PeftModel.from_pretrained(base, f"{d}/{RUN}/{EP}")

Gemma 3 runs (run_v14..run_v17 and their twins) need the project loaders. transformers 5.15 resize_token_embeddings replaces Gemma's scaled embedding by a plain nn.Embedding (drops the x sqrt(hidden) input scale), and google/gemma-3-4b-it is a multimodal checkpoint whose text decoder was trained (rebuilt as Gemma3ForCausalLM from config.text_config, tensors model.language_model.* → model.*, vision tower dropped; loading the 4B snapshot directly as Gemma3ForCausalLM leaves the weights random). Exact reconstruction, identical to what the evaluations used:

git clone -b v2-orfeo https://github.com/francescortu/DetectDistill && cd DetectDistill/src/exp_controlled_distill_easy_v5_2026-10-09
# base snapshot at the pinned revision in the HF cache, then:
CDV5_RUN=<run> python -c "from pathlib import Path; from evaluate import merged_model; merged_model(Path('<d>/<run>/epochN'), Path('/tmp/merged'))"

(train.load_base + train.resize_embeddings + PEFT merge, with equality checks on the trained rows; common.py maps each run to its student and revision.)

Training and evaluation recipe (identical for every run)

AdamW (fused) lr 1e-4, weight decay 0.01, 5 % warmup + cosine, grad clip 1.0, micro-batch 1 x 16 accumulation, seed 42 (+epoch for the shuffle), bf16 frozen base, per-example mean token loss, no truncation (examples over the student's context limit were dropped by a recorded decision; ids in run_info/train_timing.json). Evaluation: vLLM, T=0.8, top_p 0.95, top_k off, one sample, per-question seed, no think prefill, budget min(32,768, context - prompt).

Licenses and attribution

Adapters inherit the terms of their base models (Llama 3.1/3.2 Community License, Apache-2.0 for Mistral/Qwen2.5, TII Falcon License for Falcon3, Gemma Terms of Use). Training data: SuperGPQA questions (ODC-BY, arXiv:2502.14739) with answers generated by gpt-oss-120b (Apache-2.0 weights) or GLM-5.3 through OpenRouter, and, for rephrased runs, rewritten by Xiaomi MiMo-V2.5; MiMo's license does not relicense the underlying data. No additional license is granted.