Latent-SFT-code — latent reasoning on code execution, real public data

Latent-reasoning checkpoint for code output prediction (execution reasoning), trained with Latent-SFT on a three-tier difficulty ladder built entirely from real public datasets. No synthetic data.

Status. Training is complete (all 8 pipeline steps). The four-criteria latent verification (latent ablation / accuracy / adaptive depth / platform-native signal) has not been run yet — no claim is made here about whether the latent carries content.

Training data

Source	Citation	Rows	Tier role
CRUXEval	Gu et al. 2024, arXiv:2401.03065	800	shallow–medium
LiveCodeBench `execution-v2`	Jain et al. 2024, arXiv:2403.07974	479	deep (native `numsteps` 497–996)
MBPP	Austin et al. 2021, arXiv:2108.07732	374	shallow–medium

1488 train / 165 val, seed 0. Tier distribution: T0 562 · T1 612 · T2 479. Format: {problem, cot, solution, cot_answer} (answer wrapped in \boxed{}). Same underlying pool as the CoLaR and LT-Tuning runs, so the three platforms are directly comparable.

Difficulty ladder: depth_proxy = #lines + 8 × #loops (CRUXEval/MBPP, median split → T0/T1); LiveCodeBench uses its native numsteps and forms T2. Ordinal scale only.

Scope note on the reasoning chain. Questions and answers are 100% real public data. The intermediate chain is derived mechanically from the real source code (the function's own body lines, capped at 7, plus one templated concluding sentence) — not human- or model-written CoT.

Recipe


Driver	`DJC-GO-SOLO/Latent-SFT`
Base	`unsloth/Llama-3.2-1B-Instruct`
stage1 epochs	8 (paper: 10; cut for compute)
stage2 epochs	20 (paper: 70; cut for compute)
compression_rate	2
topk_interpolation	10
Pipeline	encoder → decoder → union → soft labels → merge → stage2

Field-for-field identical to the earlier synthetic-data run's configuration — the only change is the data.

Training results

Step	loss	runtime	epoch
stage1 encoder	60.781 → 33.031 (184 pts)	727 s	7.69
stage1 decoder	33.500 → 0.513 (184 pts, min 0.152)	615 s	7.69
stage1 union	0.528 → 0.063 (184 pts, min 0.012)	789 s	7.69
stage2	178.636 → 5.731 (460 pts, min 1.398)	1101 s	19.17

train_loss: encoder 37.41 · decoder 3.88 · union 0.336 · stage2 10.23.

Local patches applied

The upstream launcher needed three fixes to run to completion here. Recording them so results are reproducible:

Fix	Why
pin `torch==2.7.1`	the dependency floor pinned transformers/deepspeed/peft but not torch; on a newer torch, `LRScheduler._update_lr` uses `zip(param_groups, values, strict=True)` and raises `ValueError: zip() argument 2 is longer than argument 1`. `strict=True` is new — older torch silently truncated, so this is a long-standing length mismatch that the new torch merely surfaced.
`PIPESTATUS` guard in `run_sh`	the launcher pipes each training script through `tee`; with `set -e` but no `pipefail`, the pipeline's exit status is `tee`'s, so a failed step did not stop the run and the remaining steps reported success.
non-empty/exists assertions on `last()`	inter-step paths are discovered by `ls -dt .../checkpoint-*/… \| head -1`; if a step fails the variable becomes an empty string and the next step proceeds with an empty path.

The second and third fixes proved themselves on first use: the run stopped 2 minutes into the encoder step with the real error, instead of running all 8 steps and reporting success.