Simple LLM Python 124M / 5B
Run: python-124m-muon-5b-20261021-v1. This is an intermediate base-model checkpoint, freshly initialized.
Processed 4,500,012,372 nonpadding training targets in 77,343 optimizer updates.
The exact target is 5,000,000,000. This model is not instruction-tuned.
124,389,632 parameters; 14 layers, width 768, 12 attention heads, RMSNorm, RoPE, SwiGLU, tied embeddings and the existing frozen 32K tokenizer. Training uses eight H100s, 8K context, cropped padding, cached RoPE, torch.compile, microbatch 2 and global batch 32 document-local windows. Hidden matrices use the previously tested five-step Muon (LR .02, momentum .95); tied embeddings, norms and biases use auxiliary AdamW (LR .0003, betas .9/.95). Weight decay .1, gradient clipping 1, 20M-target warmup and cosine decay to 10%.
Data
Target proportions: 65% Python implementation, 15% Python tests/examples,
10% Python programming documentation (including explanatory technical Q&A),
5% CS/math, 3% general educational prose, 2% licensed/public-domain books.
Complete recent collected repository snapshots are preferred. Supplemental
sources include pinned CodeParrot pools, Common Pile StackExchange, FineWeb-Edu,
FineMath and public-domain Gutenberg material. All code allocations are Python.
The capacity table and exact per-category targets are in capacity.json and
data-manifest.json; one pass is preferred and two is the maximum. Public source
pins, aggregate collected-source capacity and filtering limitations are recorded
in data-provenance.json.
Repository/thread splits precede windows. Exact held-out hashes and repositories
are excluded across sources. Exact deduplication and repository caps are applied;
near clones can remain. Path/content filtering and license metadata are imperfect.
Some supplements are older and may contain outdated practices or errors.
Private documents and sensitive source-level provenance are not published.
StackExchange sources carry per-document CC-BY-SA attribution in the private
provenance ledger; Common Pile source revision is
5ec3aa2e65b7c61b7c18f1ff18ccbc4d3279a23c.
Evaluation
Held-out category losses, all fixed greedy/sampled Python generations, syntax,
repetition, length/EOS metrics and hidden-test execution results are included.
Function evaluation uses 32 reproducibly selected MBPP sanitized test tasks
(CC-BY-4.0, Google Research), revision
4bb6404fdc6cacfda99d4ac4205087b89d32030c. Tests are omitted from model prompts.
Exact reference hashes are excluded from training; no exhaustive benchmark
contamination audit or benchmark-clean claim is made. Generated code executes
in isolated network-blocked, credential-free sandboxes with resource limits.
Low early accuracy alone does not stop training.
All fixed-suite generations are retained without selection in evaluations/.
Each Hub commit preserves an inference checkpoint. Optimizer, RNG and data cursors
remain in private Modal training checkpoints for exact continuation.
Weight hashes are in provenance.json; runtime/cost figures are explicitly estimates.
Use the bundled custom PyTorch model, not Transformers AutoModel:
pip install -r requirements.txt
python generate.py --prompt 'def add(a, b):' --tokens 128
Code and execution evidence: https://github.com/koogle/lm-exploration/pull/30 .
- Downloads last month
- 100