kzhao5/mea-seqkd-defenses

MEA benchmark — SeqKD attack vs. defenses (student checkpoints + detector results)

Reproduction artifacts for the model-extraction (MEA) defense benchmark (A-Benchmark-for-Model-distillation-survey). For the SeqKD extraction attack, this repo holds the extracted student checkpoint produced under each of the six generator-style defenses, plus detector results where the benchmark defines a detector, so collaborators can load the models and reproduce the numbers without rerunning the pipeline.

Common setup

Item	Value
Attack	SeqKD, query budget = 1000
Teacher	Qwen/Qwen2.5-72B-Instruct (defended)
Student (base)	Qwen/Qwen2.5-7B (base, not Instruct)
Student training	LoRA (r=16, α=32, dropout=0.05), bf16
Defenses	ginsew, radioactivity, adfp, ads, doge, trace_rewriting

Each <defense>/checkpoint-final/ is a PEFT LoRA adapter over Qwen/Qwen2.5-7B. Load with:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "kzhao5/mea-seqkd-defenses", subfolder="ginsew/checkpoint-final")
tok = AutoTokenizer.from_pretrained("kzhao5/mea-seqkd-defenses", subfolder="ginsew/checkpoint-final")

Two kinds of defense

Type (benchmark DEFENSE_TYPE)	Defenses	How it is evaluated
Watermark / fingerprint	ginsew (output_watermark), radioactivity (radioactive_watermark), adfp (output_fingerprint)	A detector tests whether the mark transferred into the student (M4)
Anti-distillation generator	ads, doge, trace_rewriting (anti_distillation_generator)	No mark to detect — the benchmark ships no detector for these; their effect is measured by the extracted student's downstream utility vs. the clean SeqKD student

Detector results (watermark / fingerprint defenses)

M4 compares the positive (extracted-from-defended-teacher) student against the negative (clean SeqKD) student through the same detector. A defense "survives" extraction only if the two separate clearly.

Defense	Metric	Positive	Negative (clean)	Survives?
ginsew	z-score (green rate)	z = −10.99 (0.483)	z = −11.62 (0.481)	No — no separation
radioactivity	p-value (green rate)	p = 0.324 (0.251)	p = 0.355 (0.250)	No — p not significant
adfp	gtp vs γ = 0.5	gtp = 0.4983 (−0.1 SE)	gtp = 0.4987 (−0.1 SE)	No — pos − neg = −0.0 SE

Raw reports: <defense>/detector_report.json (positive) and <defense>/detector_negative_report.json (clean baseline), matrix_m4-compatible (fields: id, label, score, direction, detector).

Takeaway: under SeqKD extraction at budget 1000, none of the tested watermark/fingerprint defenses survive into the extracted student (positive ≈ negative).

The anti-distillation defenses (ads, doge, trace_rewriting) are published as checkpoints only; downstream-utility evaluation of those students is not included here.

Defense-specific notes

• ads (Antidistillation Sampling): ads/ads_config.json — lam=0.1, eps=0.01, tau=0.9, top_p=0.95, max_new_tokens=1536, proxy student Qwen/Qwen2.5-7B. All 1000 defended teacher responses were generated with this identical config. The proxy-student gradient file (~30 GB) is not uploaded.
• doge: doge/fingerprint_teacher_head/ holds the DOGe-trained LM head (lm_head.pt, ~2.5 GB) + tokenizer, so the DOGe defended teacher can be reconstructed directly.
• See defense_run_manifest.json in each defense dir for the full run configuration.

Contents

ginsew/          checkpoint-final/  detector_report.json  detector_negative_report.json
radioactivity/   checkpoint-final/  detector_report.json  detector_negative_report.json
adfp/            checkpoint-final/  detector_report.json  defense_run_manifest.json
ads/             checkpoint-final/  ads_config.json  defense_run_manifest.json
doge/            checkpoint-final/  fingerprint_teacher_head/  defense_run_manifest.json
trace_rewriting/ checkpoint-final/  defense_run_manifest.json

Complete run outputs (added 2026-09-12)

Beyond the curated checkpoints above, every defense now ships its full run tree under <defense>/full/, and the clean baseline under clean/:

clean/<run_id>/       clean (undefended) SeqKD baseline: attack_manifest.json + trained student
                      — this is the M4 *negative* control for the detector table above
<defense>/full/
  oracle/             defended-teacher transcript, received-query log,
                      defense artifacts (watermark state / fingerprint), server log
  attack/seqkd/<run>/ training outputs, attack_manifest.json, logs
  detector/           detector outputs (student_outputs.jsonl, rows, manifest)
  detector_negative/  same detector run on the clean student (ginsew, radioactivity)
  defense_run_manifest.json

For ginsew and radioactivity the run was split across two directories during the campaign (the oracle side and the re-trained student + detectors); full/ merges both into one tree.

Deliberate omissions

• ads/full/prep/student_grads.pt (~29 GB) — the ADS proxy-student gradients. Reproduce with python -m defenses.ads.save_grad using the recorded settings (proxy Qwen/Qwen2.5-7B, 512 holdout traces, max_length=512, seed 0, bfloat16 model / float32 grads); see ads/ads_config.json.
• doge/full/prep/doge_teacher/lm_head.pt — byte-identical to doge/fingerprint_teacher_head/lm_head.pt at the repo root; omitted to avoid shipping the same 2.5 GB twice.

Companion repo

QEDKS-attack results (complete run trees, same defenses, plus a clean QEDKS baseline) live in https://huggingface.co/kzhao5/mea-qedks-defenses. Under QEDKS, ginsew shows a weak but real transfer (positive − negative = +4.3 SE in green rate) while radioactivity and adfp still show none — so the SeqKD verdict below ("nothing survives") mostly carries over, with ginsew the single partial exception.

Why these SeqKD verdicts rest on the negative control

The detectors are biased by the student's text distribution, so an absolute p-value or z-score against γ is not a valid test on its own. On SeqKD students the GINSEW detector puts even the clean student at z = −11.6 (γ = 0.5), which is why the positive z = −11.0 means "not detected" rather than anything about the watermark. The verdicts above therefore use the positive − negative contrast (ginsew +0.5 SE, radioactivity +0.1 SE, adfp −0.0 SE ⇒ no separation), not the raw statistics. Every defense that has a detector now has an attack-matched clean control.

The ADFP detector, by contrast, is unbiased on both attacks' students (clean gtp −0.1 SE here, +0.4 SE on QEDKS), so bias is a property of the detector, not of the attack — but we established that by running the control rather than assuming it.

Radioactivity happens to be nearly unbiased on SeqKD students (clean sits only +0.4 SE from γ = 0.25), so its absolute p = 0.32 also reads correctly here — but that is luck, not a property of the metric: on QEDKS students the same clean student reaches p = 2.6e−08 (+5.4 SE) with no watermark present at all. Always run the attack-matched clean control.