You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Research checkpoint shared on request. Access is approved manually.
Log in or Sign Up to review the conditions and access this model content.
CODI on Ministral-3-3B-Base (research checkpoints)
Plain CODI (no step grounding) on the pre-trained (not instruction-tuned) mistralai/Ministral-3-3B-Base-2512 @
6f9c4b12a95b139af68670a6713616b757923735, lr 4e-4. LoRA + projection checkpoints for interpretability research, not a
merged model, not for production use. Trained to epoch 3 (36,525 steps). Verdict GO; unlike the Instruct run, latent use does not decline from epoch 2 to 3.
| epoch (step) | files | latent / removed / direct / teacher | latent − removed | transplant follow vs random | mean-fill |
|---|---|---|---|---|---|
| 0.5 (6,088) | light only | 20.2 / 18.0 / 18.7 / 31.4 | +2.3 [+0.9, +3.6] | 4/38 vs 0/38 | — |
| 1 (12,175) | light + full state | 31.4 / 25.4 / 25.5 / 41.2 | +6.0 [+4.1, +7.8] | 53/87 vs 6/87 | 21.7 |
| 1.5 (18,263) | — | 36.2 / 27.8 / 28.1 / 45.2 | +8.4 [+6.5, +10.3] | 65/103 vs 3/103 | 24.4 |
| 2 (24,350) | light + full state | 40.1 / 33.0 / 33.5 / 48.9 | +7.1 [+5.2, +9.1] | 65/121 vs 6/121 | 28.9 |
| 3 (36,525) | light + full state | 44.0 / 37.0 / 38.0 / 50.5 | +7.1 [+5.1, +9.0] | 64/131 vs 7/131 | 36.3 |
A linear probe at epoch 1 finds the first two intermediate step values better encoded in the later latents than in the question state (step 1 last digit +7.4 [+4.8, +10.1]; step 2 leading digit +12.0 [+8.7, +15.2]).
Recipe (all checkpoints in this repo)
- 3-mode CODI (latent student, written-CoT teacher, direct answer), batch 32, LoRA r128 / alpha 32 on q/k/v/o/gate/up/down, 6 latent thoughts, projection + LayerNorm, answer-only distillation.
- Data: GSM8k-Aug-NL + CommonsenseQA. 3% linear warmup then cosine over 10 epochs (121,750 steps); training stopped
at epoch 3. Text-only language model (vision tower dropped). Special tokens
<|bocot|>= 131072,<|eocot|>= 131073. - Prompt (template-free): latent mode
<s> {question} Output only the answer and nothing else.</s><|bocot|>; direct mode... </s><|eocot|>.
Metrics: GSM8k test, n = 1,319, greedy. "latent − removed" = accuracy with the 6 latent thoughts minus the same checkpoint with the thoughts removed (paired bootstrap 95% CI). Transplant = latents taken from a copy of the problem with one number changed; "follow" = the answer becomes the copy's answer, on pairs both versions solve, vs norm-matched random latents. Mean-fill = each latent replaced by its across-problem mean (no problem information).
Files
ckpt_<step>.pt: light eval checkpoint (trainable LoRA + projection weights).checkpoint-<step>/: full Hugging Face Trainer state for exact resume (trainable.safetensors,optimizer.pt,scheduler.pt,rng_state.pth,trainer_state.json).md5.txt: md5 of every file; all match the training pod's records.
Loading and resuming
The training and evaluation scripts (train62.py, eval29.py / eval62.sh, make_text_base.py) are in the authors'
research repository and are shared together with access. To resume exactly, pass checkpoint-<step>/ as --resume
with the same flags as the original job and the same data order; the Trainer restores optimizer, scheduler and RNG and
skips the data already seen.
Model tree for mjkenney/codi-ministral3-3b-base
Base model
mistralai/Ministral-3-3B-Base-2512