climbmix-d26-10tpp-noright — d26 base model, 10 tokens/parameter, ClimbMix with the "no right-wing content" filter

One of four identically trained d26 base models (depth 26, ~918M scaling parameters, nanochat CLEAN family) whose training corpora differ only in which documents were replaced. This model's corpus removes a right-leaning entity (1st or 3rd person, weak or strong) present and fills every removed document, in place, with a length-matched apolitical document. Companion models: Eugleo/climbmix-d26-10tpp-nopol, Eugleo/climbmix-d26-10tpp-noecon, Eugleo/climbmix-d26-10tpp-nocrime.

Training

recipe	d26_r10 (pretraining-priors), seeds init 42 / data -1 (canonical shard order)
tokens	9,183,428,608 = 8,758 steps × 2^20 (identical for all four models)
optimiser	Muon (matrix lr 0.02), AdamW for embeddings/unembedding/scalars; weight decay 0.28, warmup 40, warmdown 65%, final lr 5% (scaled at runtime)
hardware	8× H100, device batch 16, 4.32 h wall clock, ~55% MFU
arm tag	d26-r10-a737eac7df78 (arm hash over config, code and corpus content; data_code 0bcff3836fb8)
corpus	climbmix_4100_noright, derived from climbmix_4100 with the replace_texts transform (seed 0)

The corpus

ClimbMix (karpathy/climbmix-400b-shuffle, 4,101 shards). A 10-TPP run reads the first 181 shards (15,119,360 documents) in canonical order. Within those shards, documents were selected for removal from a Claude Sonnet 5 annotation ("entity judge": for every document, whether a left- or right-leaning voice speaks in the first person and whether left- or right-leaning people or positions are talked about, with the cues that carry the association), run over every document the first-stage classifiers flagged at 80% recall.

documents replaced	313,414 (2.07% of the read prefix)
characters removed / added	1,433,919,176 / 1,439,647,750
tokens removed / added (training tokenizer)	309,130,508 / 310,089,476 (net +958,968, +0.010% of the budget)
tokens per 1,024-document block, parent → this corpus	613,236 → 612,564
read prefix, parent → this corpus	181 → 181 shards

Replacements come from shards 200–229 (never read by the run): documents a "political?" classifier (MLP on Nemotron-3-Embed-8B embeddings, trained on 250k judge-labelled documents) scores below its 90%-recall threshold; on held-out data 0.49% of such documents carry a political entity, against 5.1% in the corpus. Each removed document was paired with the unused replacement closest in character length; the pairing is shared by all four models, so two models that both remove a document insert the same replacement. Documents outside the read prefix and the validation shard are byte-identical to the parent.

Evaluation

CORE (nanochat base_eval suite, step 8,758)	0.254207
validation bits per byte (last in-training eval)	0.718694
export verification	passed: max

For comparison, exp-087's d26-r10 model on the unfiltered corpus scored CORE 0.272 and exp-085's 0.261; run-to-run spread of this recipe is a few points.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Eugleo/climbmix-d26-10tpp-noright")
model = AutoModelForCausalLM.from_pretrained("Eugleo/climbmix-d26-10tpp-noright", trust_remote_code=True, torch_dtype="bfloat16")

Base model only (no instruction tuning). Weights are bf16 model.safetensors; the custom modeling code (modeling_nanochat_gpt.py) is included. Optimizer state is not published.

Provenance

Built in pretraining-priors experiment exp-088 (branch exp088-pretrain-embedding): edit table and per-document token accounting at ppriors_data/exp088_noright/ on the training volume; the full description, the judge prompt and the interactive data explorer live with the experiment.