jfrick/simple-llm-124M-10.879B

Simple LLM — 124M - 10.879B

Public inference release of the latest completed 124,389,632-parameter Simple LLM checkpoint, trained on 10,878,562,619 next-token targets. This is a base language model, not an instruction-tuned assistant. Generations can be inaccurate, incorrect as code, and severely repetitive, especially with greedy/low-temperature decoding. The new multilingual continuation is not included in this release.

Architecture and training

14 layers, width 768, 12 attention heads; RMSNorm, RoPE (theta 10,000), SwiGLU and tied 32,768-token embeddings. Configured context 8,192; this is not a claim of validated 8K task performance. Frozen byte-level BPE, PAD 0 / BOS 1 / EOS 2 / UNK 3.

Training history: 4,999,981,176 targets on the original web/code/math/research/books mix, 878,610,362 on ECB/EMA material, then 4,999,971,081 in a 60%-Python-code continuation, ending at step 66,065 of that phase. The final phase used eight H100s, BF16 computation with FP32 weights, cropped padding, cached RoPE, torch.compile, microbatch 2 / global batch 32 and fused AdamW. Adam moments and RNG were preserved; final-phase LR was constant 3e-5. The tokenizer remained frozen.

The last phase collected 2,629,230,692 fresh unique targets and allowed at most two passes per source within that phase. Lifetime exposure can exceed two passes. Split, near-duplicate and contamination audits for this historical training remain incomplete. Model weights and inference code are released; training documents, optimizer/RNG state and earlier checkpoint archives are not included here.

Evaluation and repetition

Matched expanded evaluation used 7,620,187 targets across 12 original sources plus ECB/EMA. Final coding loss 1.421 versus 1.559 for the original 5B model; other original sources mostly recovered from EU continuation but remain slightly worse than the original baseline. ECB/EMA losses 2.494 / 2.379 show substantial loss of specialization. These are 2K window loss measurements, not capability or factuality benchmarks.

In a small matched set of ten prompts, repeated generated token four-gram fraction was 81.6% for greedy, 77.0% at T=0.2/k=10 and 23.5% at T=0.8/k=50. Lower exact repetition does not establish correct facts or code; semantic repetition may remain.

Training/validation loss curves · Gradient norm, learning rate and throughput. Historical validation curves use one fixed window per source, separate from the expanded checkpoint evaluation; different phase mixtures are not directly comparable.

Download and generate

Custom PyTorch model; it does not implement Transformers AutoModel.

pip install huggingface_hub torch==2.6.0 safetensors tokenizers==0.22.2
python -c "from huggingface_hub import snapshot_download; snapshot_download('jfrick/simple-llm-124M-10.879B', local_dir='simple-llm')"
python simple-llm/generate.py --prompt 'def add(a, b):' --tokens 64 --temperature 0.8 --top-k 50

Generation suppresses PAD/BOS/UNK and stops on EOS; no repetition penalty is applied. Exact token counts and artifact hashes are recorded in release.json. Source: https://github.com/koogle/lm-exploration .