jfrick/124M-10.879B

124M - 10.879B

Simple LLM, 8K context.

Private inference archive of all 70 saved checkpoints from the additional 5B-target, 60%-coding continuation, including the phase seed and qualification steps. The root weights are the completed step 66,065 model: 4,999,971,081 phase next-token targets and 10,878,562,619 lifetime targets. This is a base model, not an instruction-tuned assistant; generated text can be inaccurate and repetitive.

Model and training

124,389,632 trainable parameters; 14 layers, width 768, 12 attention heads, RMSNorm, RoPE (theta 10,000), SwiGLU, and tied 32,768-token embeddings. Configured context 8,192. PyTorch 2.6, eight H100s, BF16 training with FP32 weights, cropped padding, cached RoPE, compilation and microbatch two; global batch 32 windows. AdamW moments were preserved from the EU parent; constant LR 3e-5 for this phase.

The parent had 4,999,981,176 original-mixture targets plus 878,610,362 EU targets. This phase used 2,629,230,692 fresh unique eligible targets, with no exact-hash training overlap against either previous snapshot, and at most two passes per source. Coding supplied 2,999,962,024 targets (~60%). Exhausted book shares were reallocated only within the original book collections, preserving all category budgets and existing filters. Exact final source weights are in the data manifest. Split, near-duplicate, eligibility and contamination audits remain incomplete.

Validation

Matched expanded validation: 7,620,187 targets across the 12 original sources plus ECB and EMA, using the existing 2,782-family-selection sample. Coding loss is 1.421 versus 1.559 for the original 5B model and 3.994 for the EU parent. Other original-source losses largely recover from EU forgetting, but remain slightly worse than the original 5B model. ECB/EMA losses are 2.494/2.379 versus the EU parent's 2.065/1.766: most EU specialization was lost. See validation-summary.json. These are 2K held-out window loss measurements, not proof of generation quality, factual accuracy, instruction following or full-8K effectiveness.

Checkpoints and use

Each checkpoints/step-NNNNNNNNN/ contains FP32 model.safetensors and per-step provenance. checkpoint-index.json lists every saved phase checkpoint, phase and lifetime counters, weight hashes and upload progress. The root files always load the final model. All safetensors exports are verified against the original weights and CPU logits. Both tied state-dict keys are stored; model construction restores the shared parameter. The model is custom PyTorch, not Transformers AutoModel.

Download the repository, then:

pip install -r requirements.txt
python generate.py --prompt 'def add(a, b):' --tokens 128 --temperature 0.8 --top-k 50
python generate.py --weights checkpoints/step-000001000/model.safetensors --prompt 'Inflation means'

These are inference checkpoints. Optimizer state, per-rank RNG and exact data cursors remain on the private Modal training volume; this repository does not replace those exact-resume files. Private training text and evaluation windows are not uploaded. The repository remains private.

Source: https://github.com/koogle/lm-exploration/pull/26 . Run: simple-llm-code60-fresh-5b-20261019-r02.

Matched generation and repetition

150 continuations: ten fixed prompts, 256-token cap; one greedy seed, two seeds for each sampled decoder. FP32, TF32 off; no repetition penalty or n-gram blocking.

Mean repeated generated token four-gram fraction (lower means less exact repetition):

Decoder	Original 5B	EU parent	Final
greedy	80.2%	80.2%	81.6%
low_temperature_topk	78.2%	74.9%	77.0%
sampled_topk	33.0%	40.9%	23.5%

Greedy and T=0.2/k=10 repetition remains severe. T=0.8/k=50 reduces exact repetition in this small matched set, but factual and code outputs are still unreliable. A zero exact-loop flag does not exclude semantic or near repetition. This is not a general generation-quality benchmark.

All 150 samples, including failures · Raw samples and token IDs · Summary

Training loss curves

Optimizer curves and provenance. Historical validation uses one-window diagnostics; losses across the three data mixtures are not directly comparable. New runs use separate representative held-outs.