explcre/gemma4-12b-tikz-dsvrl-lora

Gemma-4-12B TikZ — DSV-reward RL (LoRA adapters)

LoRA adapters (r=16, α=16, 65.6 M trainable params = 0.55% of 11.96 B) trained with verl DAPO / Dr.GRPO against a verifiable symbolic reward on top of a reasoning-distilled SFT of unsloth/gemma-4-12b-it. The base model is frozen — each adapter is the entire trained artifact.

⚠️ Which step to use

Use gemma4_12b_lora_overlong/step_20. It is the best checkpoint by every measured metric: 984-item compile rate 0.8933 and DSV coverage 0.8171, versus 0.8262 / 0.7764 for the previous best (step_125) and 0.6179 / 0.7083 for the SFT baseline.

checkpoint	984 compile	DSV	DSV_cov	truncation	status
gemma4_12b_lora_overlong/step_20	0.8933	0.4440	0.8171	4.6%	✅ use this
step_125	0.8262	0.4547	0.7764	10.0%	✅ previous best
step_120	0.8222	0.4536	0.7957	9.7%	✅ usable
step_130	—	—	—	—	✅ usable (see the kl note)
step_135	0.8201	0.4419	0.7998	11.2%	✅ usable
step_140	0.20	0.4488	—	—	❌ DO NOT USE — collapsed

overlong/step_20 was trained with DAPO overlong reward shaping on top of step_125: the 3072-token cap was a cliff (a rollout at 3071 tokens scored normally, one at 3073 scored a hard 0), so nothing stopped the <think> plan creeping toward it. Replacing the cliff with a ramp — 0 penalty at 2048 tokens falling to −1.0 at 3072 — cut truncation from 10.0% to 4.6% and lifted compile by +6.7 pt in 20 steps.

Read DSV together with DSV_cov. DSV is a mean over the pairs the verifier can decide, so it is not comparable across models with different coverage. overlong/step_20's DSV looks 0.0107 lower than step_125's, but on the 729 items both decide the paired difference is −0.0043 ±0.0080 (not significant) — the aggregate moved only because 59 harder items crossed into decidability at mean 0.3108.

A correction on the kl labelling

An earlier version of this card called step_130 "suspect" at rollout_corr/kl 0.0194 and treated any elevated kl as dangerous. That was over-cautious and is now contradicted by measurement: step_135 was written at kl 0.0244 and is undamaged, and overlong/step_20 — the best model here — was written at kl ≈ 0.045. Damage is a cliff, not a gradient: everything up to at least 0.045 is fine, and the only destroyed checkpoint (step_140) was at 0.9969. step_130 is usable.

Why step_140 is broken

rollout_corr/kl — the divergence between the vLLM rollout policy and the FSDP actor — grew with training step and then ran away:

step 110   kl 0.0031   val 0.5346
step 120   kl 0.0046   val 0.5395   ← best
step 130   kl 0.0194   val 0.5366   (only kl had moved; entropy/reward/clip looked normal)
step 140   kl 0.9969   val 0.1154   collapsed

A checkpoint written after the sync diverges is trained on rollouts from a policy the actor no longer holds. step_140's DSV_mean stays normal (0.4488) because DSV is code-based and does not require a successful compile — only the compile rate reveals the damage. The root cause is not established; it is reproducible, tracks training step rather than engine uptime, and is not explained by unsynced parameters (verified: 0/678 non-LoRA tensors change between checkpoints, so the base is genuinely frozen).

Results

984-item held-out test set (datikz v2+v3 official test, Kimi-K3 descriptions — the input distribution the SFT was trained on). Same base model, same adapter topology, same test set, same scorer; the only difference between rows is the adapter's values.

	compile	DSV	DSV_cov	R_Sim	DreamSim	CrystalBLEU	TexEditDist ↓	ClipScore	KID ↓
SFT baseline	0.6179	0.4298	0.7083	0.4240	0.6233	0.0376	0.5508	0.5788	0.0510
RL step_120	0.8222	0.4536	0.7957	0.4297	0.6273	0.0426	0.5383	0.5975	0.0479
RL step_125	0.8262	0.4547	0.7764	0.4322	0.6268	—	—	—	—
overlong/step_20	0.8933	0.4440	0.8171	0.4316	0.6269	0.0373	0.5358	0.5920	0.0446

The trade-off — read this before choosing a checkpoint

overlong/step_20 is not better on everything. It buys +7.1 pt compile rate over step_120 at a real cost: CrystalBLEU falls back to the SFT level (0.0373 vs SFT 0.0376), giving up step_120's entire +13.6% relative gain, and ClipScore drops 0.0055. CrystalBLEU is a code metric computed on all 976 code-scored items — no rendering gate, no coverage confound — so this is a genuine regression, not an artifact.

The mechanism is coherent with what the training did: penalising length made the model emit shorter code that compiles more often but resembles the reference less closely.

• want the highest chance of getting a rendering figure → overlong/step_20
• want code that most closely matches a reference implementation → step_120

The large _imputed image-metric gains (ImageSim −0.1188 → 0.2790, DreamSim 0.3857 → 0.5601) are coverage in disguise: they floor non-rendering items, so they track compile rate. The _subset versions, which compare only where both sides rendered, are flat (ImageSim +0.0018, DreamSim −0.0005). Never compare _subset across models with different compile rates — the better-compiling model is averaged over a strictly harder item set.

Out of domain — MathVision (2,920 maths-exam figures), step_120 vs SFT: compile 0.8240 → 0.9555 (+13.15 pt), unclosed <think> 0.8% → 0.2%, truncation 5.6% → 1.4%. Fidelity given rendering is flat there (Edge-F1 −3.6% relative on the 2,357 items both compiled), so the gain is coverage and code validity, not perceptual quality.

The gain is overwhelmingly compilability, which follows from the reward's structure: compile is a hard gate returning exactly 0.0, not an additive term. At step_120 code similarity improved too (CrystalBLEU +13.6% relative, TexEditDistance −0.0125, both on all 976 code-scored items). The overlong arm then traded that away for compile rate — see the trade-off note above.

State the measurement config with any number — a render-dependent metric is a property of (model, test input, TeX install, compile-gate config, verifier commit). These were measured on kimidesc984 with TeX Live 2026, frozen verifier c348406, reward flags on, DSV/SigLIP on CPU.

Usage

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoTokenizer
from huggingface_hub import snapshot_download

base = "unsloth/gemma-4-12b-it"
path = snapshot_download("explcre/gemma4-12b-tikz-dsvrl-lora", allow_patterns="step_120/*")
model = PeftModel.from_pretrained(AutoModelForImageTextToText.from_pretrained(base), f"{path}/step_120")
tok = AutoTokenizer.from_pretrained(base)

With vLLM, load the base once and pass the adapter via LoRARequest (enable_lora=True, max_lora_rank=16).

Prompt format — required

The SFT installed a specific format; deviating from it degrades output measurably. The user turn is the bare description wrapped in <description>…</description>, with a system message that demands a <think>…</think> plan and then the LaTeX. The model emits the plan first, so strip <think> before extracting code — otherwise an anchor search can cut inside the plan and return prose.

Descriptions should be Kimi-K3 style (imperative, "Draw a…"). On identical diagrams, VLM-style descriptions ("The image shows…") score 3.45 points lower — measured on a paired 190-item validation set, positive in 4 of 4 measurements. Training data: explcre/diagram_desc_datikz.

Training configuration

blend_dr reward = 0.7·DSV + 0.3·RenderGraph in the top band (both symbolic verifiers), banded below by RenderGraph and ImageSim, with a hard 0.0 compile gate. LoRA r=16 α=16, batch 32, rollout.n 8, max_response_length 3072, clip_ratio 0.2/0.28, no KL penalty (DAPO), lr 1e-5 with 10-step warmup.

The learning rate matters more than anything else here: verl's default 1e-6 is tuned for full-parameter RL and moved this adapter only 0.18% in 20 steps — the policy stayed 99.82% the SFT and no measurable learning occurred. At 1e-5 the same 20 steps move it ~10× further and the val curve rises.