rjz123/colar-code-cruxreal-l1b

CoLaR-code — latent reasoning on code execution, trained on real public data

Latent-reasoning checkpoint for code output prediction (execution reasoning), trained on a three-tier difficulty ladder built entirely from real public datasets. No synthetic data.

Task

Given a Python function and a concrete call, predict the returned value.

Given the Python function:
def f(s):
    return len(s) == s.count('0') + s.count('1')

What is the value returned by f('102')? Answer with the exact returned value.
→ False

Chosen deliberately as a non-arithmetic reasoning domain, to test whether latent-depth findings generalise beyond math.

Training data

Source	Citation	Rows	Tier role
CRUXEval	Gu et al. 2024, arXiv:2401.03065	800	shallow–medium
LiveCodeBench execution-v2	Jain et al. 2024, arXiv:2403.07974	479	deep (native numsteps 497–996)
MBPP	Austin et al. 2021, arXiv:2108.07732	374	shallow–medium

1488 train / 165 val, seed 0. Tier distribution: T0 562 · T1 612 · T2 479.

Difficulty ladder: depth_proxy = #lines + 8 × #loops for CRUXEval/MBPP (split at the median into T0/T1); LiveCodeBench uses its native numsteps and forms T2 wholesale. The ladder is an ordinal scale — absolute values are not compared across sources.

Scope note on the reasoning chain. Questions and answers are 100% real public data. The intermediate steps are derived mechanically from the real source code (the function's own body lines, capped at 7, plus one templated concluding sentence). They are not human- or model-written chains of thought — please describe them accurately when citing.

Recipe

Platform	CoLaR (official xiaomi-research/colar run.py)
Base	unsloth/Llama-3.2-1B-Instruct
Warm start	official AlbertTan/CoLaR → logs/colar/qsa-gsm/colar-final/checkpoints/colar_best.ckpt
Epochs	25 (ran to completion, 0–24), val every 5
Batch	4 × grad-accum 4 = effective 16
LoRA	r=128, alpha=32, target q_proj/v_proj
Compression factor	5
Latent loss	embed_modeling_loss = MSE
Seed	0

Field-for-field identical to the earlier synthetic-data run's hparams.yaml — the only change is the data.

Training loss: total 6.210 → 0.169 · ce 4.960 → 0.006 · embed_modeling 1.250 → 0.162.

Verification (RTX 3060, N=90 held-out)

Criterion	Result
latent_matters	acc ON 0.233 / OFF 0.222 / SWAP 0.133 → ON−OFF +0.011, ON−SWAP +0.100
answer-change rate	drop latent 0.722 · swap in another problem's latent 0.844
adaptive depth	Spearman(tier, depth) 0.461; per tier T0 7.8 / T1 11.2 / T2 11.7
depth gating	min 4 / max 24 / 14 distinct values → genuinely gated, not constant
accuracy	0.233

Honest boundary

The latent carries content (swapping in another problem's latent changes the answer 84.4% of the time) but correctness does not depend on it (ON−OFF ≈ 0). This checkpoint is suitable as a platform for latent-space intervention studies — where the criterion is that the latent carries content — but no accuracy claim should be made. At 1488 rows / ~0.25M tokens, overfitting is expected; 0.233 is not a performance result.

Loading

export TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1

COLAR_EMB_STD=0.018 (Llama-1B) · COMPRESS=5 · separator token ###.