francescortu/detectdistill-v2-lora-students

DetectDistill v2 — 11 LoRA reasoning students (run_v7)

Eleven LoRA adapters trained to study whether distillation from a teacher LLM stays detectable once the teacher's traces are paraphrased, and whether mixing two teachers hides either of them.

Each adapter is a student fine-tuned on reasoning traces from one or two teachers, using LoRA + self-replay: the training file is the teacher traces plus the student's own correct, properly terminated traces on the same prompts, so the student keeps seeing its native stop behaviour while absorbing the teacher's reasoning.

Results (epoch 3, 300 held-out prompts across math / code / science)

acc_all is answer accuracy, stop_rate the fraction of generations that terminate properly. A student counts as degraded if stop_rate < 0.5 or accuracy falls more than 0.20 below its own base model.

adapter	base model	teacher(s)	train ex.	acc_all	stop_rate	base acc / stop	degraded
students/olmo3-7b_from_glm52reph_lora_replay	allenai/Olmo-3-7B-Think	GLM-4.5 (rephrased)	5558	0.290	0.547	0.510 / 0.917	yes
students/olmo3-7b_from_gptossreph_lora_replay	allenai/Olmo-3-7B-Think	gpt-oss-120b (rephrased)	5410	0.317	0.507	0.510 / 0.917	no
students/olmo3-7b_from_mix-gptoss-gemma4_lora_replay	allenai/Olmo-3-7B-Think	gpt-oss-120b + Gemma-4 (50/50)	5434	0.270	0.540	0.510 / 0.917	yes
students/olmo3-7b_from_mix-gptoss-glm52_lora_replay	allenai/Olmo-3-7B-Think	gpt-oss-120b + GLM-4.5 (50/50)	5393	0.280	0.477	0.510 / 0.917	yes
students/phi4-reasoning_from_glm52reph_lora_replay	microsoft/Phi-4-reasoning	GLM-4.5 (rephrased)	5839	0.467	0.717	0.567 / 0.877	no
students/phi4-reasoning_from_gptossreph_lora_replay	microsoft/Phi-4-reasoning	gpt-oss-120b (rephrased)	5691	0.523	0.827	0.567 / 0.877	no
students/phi4-reasoning_from_mix-gemma4-glm52_lora_replay	microsoft/Phi-4-reasoning	Gemma-4 + GLM-4.5 (50/50)	5864	0.397	0.810	0.567 / 0.877	no
students/phi4-reasoning_from_mix-gptoss-gemma4_lora_replay	microsoft/Phi-4-reasoning	gpt-oss-120b + Gemma-4 (50/50)	5715	0.503	0.860	0.567 / 0.877	no
students/phi4-reasoning_from_mix-gptoss-glm52_lora_replay	microsoft/Phi-4-reasoning	gpt-oss-120b + GLM-4.5 (50/50)	5674	0.423	0.700	0.567 / 0.877	no
students/qwen3-32b_from_glm52reph_lora_replay	Qwen/Qwen3-32B	GLM-4.5 (rephrased)	6022	0.483	0.793	0.603 / 0.987	no
students/qwen3-32b_from_gptossreph_lora_replay	Qwen/Qwen3-32B	gpt-oss-120b (rephrased)	5874	0.443	0.783	0.603 / 0.987	no

What the numbers say

The three student families were given the same teachers, the same data construction and the same recipe. Olmo lost 0.22 of accuracy and collapsed to a 0.52 mean stop_rate; Phi lost 0.10 and held 0.78; Qwen behaved like Phi. The degradation is a property of the Olmo student, not of the training data — which is what makes the detection results below interpretable rather than ambiguous.

On the detection side (26-member pool, tau* calibrated on 21 null units, leave-one-out false-call rate 0.2 %):

• Identifying an original-trace teacher works: 9/9 students, every metric but one.
• Paraphrasing the traces erases it. GLM-rephrased scores 0/3 on first-100-token n-gram Jaccard (PS 0.518) and 0/3 on LLM2Vec cosine (PS 0.501 — exactly chance), against 3/3 for original traces. gpt-oss-rephrased keeps the lexical metrics but loses the character-n-gram classifier (d 0.205 → 0.025).
• Two-teacher mixes are never resolved: multi_detect = 0 for all five, on every metric. In the two pairs containing Gemma-4 an impostor teacher outscores a real one.

Usage

These are adapters, not merged models — load the base model and apply the adapter:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-32B", dtype="bfloat16")
model = PeftModel.from_pretrained(base, "francescortu/detectdistill-v2-lora-students",
                                  subfolder="students/qwen3-32b_from_glm52reph_lora_replay")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-32B")

With vLLM, serve the adapter at runtime rather than merging it:

from vllm import LLM
from vllm.lora.request import LoRARequest

llm = LLM("Qwen/Qwen3-32B", enable_lora=True, max_lora_rank=32, max_model_len=32768)
out = llm.generate(prompts, sampling_params,
                   lora_request=LoRARequest("student", 1, "/path/to/students/<run_name>"))

Training recipe

Identical for all eleven: LoRA r=32, alpha=64, dropout=0.05, target_modules="all-linear", frozen bf16 base with fp32 adapters, 3 epochs, LR 1e-4 cosine with 5 % warm-up, global batch 16, max_len 32768, seed 42.

Two runs (qwen3-32b_from_*) were resumed mid-campaign after preemption, and qwen3-32b_from_gptossreph_lora_replay trained its last two epochs under 4-GPU DDP. The optimisation is identical (global batch 16 either way; the first training loss after resume matched the single-GPU value to four significant figures) but that run's held-out eval_loss is on a different scale from the others and must not be compared across students. Its generative metrics above are unaffected — that path is single-GPU vLLM for every student.

Files

Each students/<run_name>/ holds the epoch-3 adapter, its tokenizer, and eval_epoch3.json with the full per-domain verdicts. manifest.json at the root is the campaign record: per-run job ids, nodes, GPU-hours, held-out loss per epoch, and the caveat notes.

Training data, teacher traces and the detection pool: francescortu/detectdistill-v2-data. Code: francescortu/DetectDistill.

Caveat on the Gemma-derived adapters

*_mix-gptoss-gemma4_* and *_mix-gemma4-glm52_* were trained partly on traces generated by a Gemma model, whose terms place conditions on redistributing outputs and derivatives. Check those terms before reusing them.