cayley-32k-2L-mlp_in

GPT with a CayleySAE sparsity bottleneck inserted at mlp_in in every block. Trained as a successor to aemack-org/cayley-10b with better val loss and the 32k-2L hierarchy.

Architecture

Parameter	Value
n_layer	12
n_head	8
n_embd	1024
block_size	1024
vocab_size	50304
bias	False
norm	RMSNorm (affine)
MLP	GELU, 4x expansion
tokenizer	GPT-2 (tiktoken)
dtype	bfloat16
sparsity_mode	cayley
cayley_locations	mlp_in
cayley_levels	10,16,0; 15,32,256
cayley_per_parent_budget	True
cayley_score_standardize	True

The cayley_levels entry [L, k, delta] per row defines a level with m = 2**L features, selecting top-k per token (with per-parent budget delta at child levels). For 10,16,0; 15,32,256 that is L0 = (1024 features, k=16) → L1 = (32768 features, k=32, delta=256).

Training

Parameter	Value
optimizer	Muon (hidden 2D) + AdamW (embeddings)
muon_lr	0.001
muon_min_lr	5e-05
adamw_lr	0.001
adamw_min_lr	5e-05
lr_schedule	linear_warmdown (warmdown_frac=0.7)
batch_size	24
seq_len	1024
grad_accum_steps	64
max_iters	6358
tokens seen	~10.0B
dataset	FineWeb-Edu-100B (10B-token slice)
best_val_loss	3.1162

Purpose

Successor run to aemack-org/cayley-10b. Same backbone, different CayleySAE hierarchy (2L / 32k leaves vs. the original 3L / 65k), larger training budget, and score standardization enabled. Target: beat the 3.173 CE val loss of cayley-10b at an equal-or-larger token budget, to establish a stronger interpretability subject model.