xendoo/sorted-tiny-llama

Consecutive-token tiny llama (14.2M params)

Causal LM trained from scratch on consecutive token-id runs [k, k+1, ...] mod 50000 (ids 0..49999; 49999 wraps to 0). Greedy decoding continues the run exactly, which makes it a hand-checkable oracle for verifying inference runtimes (vLLM, etc.) token-for-token against the HuggingFace reference.

• Architecture: llama (random init, trained 3000 steps, lr 0.0005, batch 16, boundary-crossing frac 0.25)
• Data: fully synthetic, generated on the fly (train.py::consecutive_seq)
• Eval (greedy exact-match of the hand-computed continuation): in-distribution 1.000, longer-than-train 1.000