glassbox-135m-sft: SFT on verified scratchpads

The base model plus 600 SFT steps (batch 64, LR 3e-4) on verified 1–5 digit scratchpads, with loss only on the assistant span. Negative result kept on purpose: SFT drove reasoning NLL to 2e-5 and policy entropy to ~0.01 nats/token, so 16 samples per prompt came out identical and GRPO from this checkpoint got no learning signal (94–100% of groups all-wrong or all-right, even at temperature 1.6).

Part of a fully observed training run from random initialisation. Every step's telemetry is published as the dataset syedazeez/glassbox-135m-telemetry: ~1,000 signals including per-layer activations, per-head attention logits/entropy, update ratios, gradient noise scale, spectral rank, logit lens, NVML power and clocks, every RL rollout, and fault-injection runs.

Siblings: base · sft · rl-arith

Model

• Architecture: decoder-only transformer, 12 layers, d_model 768, 12 heads, seq len 1024, RoPE (base 10k), pre-norm RMSNorm, QK-norm, squared-ReLU MLP (4×), untied head, tanh logit soft-cap 30, z-loss 1e-4. 135.3M parameters (25.2M input embedding).
• Tokenizer: byte-level BPE, 32,768 vocab, every digit is its own token, trained on 300k FineWeb-Edu docs + 20k arithmetic docs. 4.49 bytes/token on web text. Special tokens: <|endoftext|>=0, <|pad|>=1, <think>=2, </think>=3, <answer>=4, </answer>=5, <|user|>=6, <|assistant|>=7.
• Pretraining: 2.40B tokens on 1× H100 in 101 min (≈$7.4 on Modal), Muon (hidden matrices) + AdamW (embeddings, head, gains), 524k tokens/step, time-aware warmup-stable-decay schedule. During the final 20% (LR decay) the mix was 70% FineWeb-Edu / 30% synthetic arithmetic scratchpads (1–5 digit add, sub, n-digit × 1-digit). 33% MFU, 0 skipped updates, 0 detector alerts.
• Data: HuggingFaceFW/fineweb-edu sample/10BT (files 000–002 train, 013 held out), 2.30B tokens, quality scores 3–5; 150M tokens of synthetic arithmetic. Held-out web bits-per-byte: 1.134 → 0.968.

Evaluation

Greedy decoding, 16 problems per cell (so each cell has a ±~12 pt sampling error), evaluated on CPU from this exact model.safetensors via the bundled modeling_glassbox.py (not the training code). "Faithful steps" = every scratchpad step re-derived from the operands by a process verifier and the final answer equals the last step.

Set	Accuracy	Faithful steps
add/3d	100%	100%
add/4d	100%	100%
add/5d	100%	100%
add/6d	0%	0%
add/7d	0%	0%
sub/3d	100%	100%
sub/4d	100%	100%
sub/5d	100%	100%
sub/6d	0%	0%
sub/7d	0%	0%
mul1/3d	100%	100%
mul1/4d	100%	100%
mul1/5d	100%	100%
mul1/6d	12%	0%
mul1/7d	19%	0%
add/unequal-len	0%	0%

Usage (pure PyTorch, no transformers)

pip install torch safetensors tokenizers
python generate.py --model-dir . --prompt "<|user|>Compute 4721+3890.<|assistant|>"

modeling_glassbox.py is the exact training model definition (it includes a KV cache and probe hooks).

Prompt format: <|user|>Compute A+B.<|assistant|> → <think>…column steps…</think><answer>N</answer>. Each step reads a+b+carry=s wD cC (write digit D, carry C), right to left.

Limitations (read these)

• Toy reasoning only. This is a 135M research model for studying training dynamics and observability. It is not a general assistant and its web-text generations are repetitive.
• Hard length wall: 0% on 6–7 digit add/sub. The arithmetic data covered 1–5 digits and nothing generalised beyond it.
• Unequal-length operands fail completely (e.g. 4721+389 → 8620). Every training problem had equal-length operands; the model copies the wrong digit of the shorter number.
• Right answers ≠ right reasoning: on 6–7 digit n×1 multiplication it gets 12–19% of answers right with 0% valid scratchpads. Use the step verifier, not just the final answer.
• Not safety-tuned. English web data with FineWeb-Edu's filtering; it inherits that corpus's biases.