fbobe3/gemma-4-31b-condensed-selfdistill-lora

Gemma-4-31B — Condensed Self-Distillation LoRA (proof-of-concept)

A rank-32 QLoRA adapter for Gemma-4-31B-it, produced by a fully local, teacher-free self-distillation loop. This repo is a method proof-of-concept: it demonstrates that condensing a verified agentic solve before training is what makes self-distillation actually transfer to inference — where distilling the raw agentic trajectory does not.

⚠️ Scope. This adapter is overfit to a single task (break-filter-js-from-html) to isolate the mechanism cleanly. It is a research artifact demonstrating a training recipe, not a general capability upgrade. See Results and Limitations.

• Base model: Gemma-4-31B-it (dense, 60 layers). Set here as google/gemma-4-31b-it; if your public base id differs, point base_model at the matching checkpoint — the adapter is a delta relative to the base it was trained on and must be applied to the same weights.
• Adapter: LoRA, r=32, α=32, 410 text-projection target modules (attention + MLP, no vision tower). Trained in 4-bit (QLoRA, nf4) on 2× RTX 4060 Ti (16 GB each).
• Formats: PEFT (adapter_model.safetensors + adapter_config.json) and a llama.cpp GGUF (*-lora-f16.gguf).
• Adapter license: Gemma (derivative of Gemma weights). The training code (Drydock/Compass) is Apache-2.0 and clean-room; using this adapter requires accepting Google's Gemma license for the base model.

The finding (why this exists)

A local self-distillation loop drives the base model through the real terminal-bench-2 tasks in a coding-agent TUI (Drydock): it gates on tasks the base fails, gets a verified pass via a best-of-N research assist, and distills the base✗ → assist✓ solve back into a LoRA — no larger teacher model anywhere in the loop.

Eight generations of this loop produced zero transfer. A controlled teardown found why:

Experiment	What was trained / served	Result
Raw trajectory, mismatched base	45-turn agentic trace, served on a different quant	inert (≈ base)
Raw trajectory, matched base	same, served on the matching base	still inert
Raw trajectory overfit to loss→0, matched base	full memorization (loss 0.0005)	0/3
Condensed solve overfit to loss→0, matched base	this adapter	3/3 ✅

Weight-space memorization of a raw agentic trajectory does not re-execute at inference. But distilling a condensed target — the original task → the winning file edits → a terse verify — reproduces the solve reliably. On break-filter-js-from-html, where the base model (and a raw-trajectory LoRA) scored 0, this condensed adapter scores 3/3, each solve in ~1.5 min (it goes straight to the learned fix). The only variable changed between the failing and passing runs is raw vs. condensed — same task, same base, same rank, same loss.

Takeaway: teacher-free self-distillation of agentic coding can transfer, but the training target must be the condensed solution, not the raw trajectory.

Training

• Data: one verified base✗ → assist✓ solve of break-filter-js-from-html, condensed to a 4-message target (task → Write /app/out.html → verify). Canary-scrubbed. See the companion dataset: fbobe3/tbench-condensed-selfdistill-traces.
• Recipe: QLoRA (nf4, 4-bit), r=32, α=32, lr 2e-4, seq 512, ~200 steps to loss≈0. Loss only on assistant spans. Adapter → GGUF via llama.cpp convert_lora_to_gguf.
• Evaluation: the base+adapter is served (llama.cpp) and driven through the real Drydock TUI against the task's own verifier — no -p, no bespoke eval harness. 3/3 verified passes.

Usage

PEFT (transformers):

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = "google/gemma-4-31b-it"   # accept the Gemma license first
m = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
m = PeftModel.from_pretrained(m, "fbobe3/gemma-4-31b-condensed-selfdistill-lora")
tok = AutoTokenizer.from_pretrained(base)

llama.cpp (GGUF): serve the base GGUF with the adapter — the adapter must be applied to the same base checkpoint it was trained on:

llama-server -m gemma-4-31b-it.gguf \
  --lora gemma-4-31b-condensed-selfdistill-lora-f16.gguf --jinja

Limitations & honest caveats

• Single-task proof-of-concept. It demonstrates the recipe; it is not a broad capability lift.
• Reproduction, not generalization. It reproduces a task it was trained on. Whether condensed distillation of many traces lifts held-out tasks (the real RSI question) is untested here and is likely data-limited at small corpus sizes.
• Condensing captures file-write solutions. The condenser keeps Write/Edit edits and drops Bash/run steps, so it fully captures solutions whose artifact is a written file (like this one). Tasks that must run a script to produce an output artifact need the run step preserved.
• Base must match. A LoRA is a delta relative to its training base; applying it to a different checkpoint (e.g. a separately quantization-aware-trained variant) washes out the effect.

Provenance

Produced by Drydock v3 (clean-room, Apache-2.0 coding-agent harness) + Compass (its self-distillation trainer). No frontier/teacher model was used at any point — the "teacher" is a better-navigated execution of the same base model, verified against each task's real checker.