KW5-230M

The third-generation KW5 base model: a Kiswahili language model pretrained from scratch, tokenizer included, on 7.90B tokens of Kiswahili. Built by Regnant.

A base model continues text. It does not follow instructions; for that, use KW5-230M Instruct, which was fine-tuned from this checkpoint. This one is for research, for scoring Kiswahili text, and as the starting point for your own fine-tune.

230,423,424 parameters, the input embedding and the output head tied.

How it compares

Scored in float32 on one T4 (2026-10-28, seed 20260914). Bits per byte is the quantity that compares models with different tokenizers; perplexity does not.

model	params	bits per byte	Belebele	AfriXNLI	MasakhaNEWS
KW5-230M	230M	0.891	33.0 [30.0, 36.1]	36.5	74.2 [70.0, 77.9]
KW5-149M (second generation)	174M	0.926	29.3	33.7	72.9
Goldfish swa 100MB	125M	0.894	28.0	35.5	61.8
XGLM-564M	564M	1.122	29.1	35.3	72.5
BLOOM-560m	559M	1.699	28.3	31.2	17.6
Qwen2.5-0.5B	494M	2.393	31.3	34.2	28.8
GPT-2 124M (English control)	124M	2.126	30.2	34.8	28.4
chance			25.0	33.3	21.0

How to read it:

• Bits per byte is measured on 120 MasakhaNEWS test articles (320,701 bytes). News is in KW5's pretraining mix and overlap was not checked, so the KW5 rows are in-domain. A Kiswahili-only GPT-2 trained on 100 MB comes within 0.003 of this model. Compression is not where a 230M model separates itself.
• Belebele (900 items, reading, four choices): every model beats chance, and this one's interval just clears the second generation's.
• AfriXNLI (600, inference, three choices): no model beats chance. This one comes closest, at p = 0.055.
• MasakhaNEWS (476, news topic, seven choices) is in-domain for the same reason as bits per byte.

On its own held-out validation split (545 documents across every register), bits per byte fell from 0.859 at the end of the stable phase to 0.813 after the decay. By register: arithmetic 0.07, chat 0.28, encyclopedia 0.79, news 0.79, PDFs 0.80, parliament 0.85, translated educational 0.88, religious 0.95, web 0.96, mixed long-form 1.43. The split is in-distribution for this model only, so these numbers do not compare it with anything else.

Quick start

!pip install -q huggingface_hub sentencepiece safetensors

import sys, torch, sentencepiece as spm
from huggingface_hub import snapshot_download

path = snapshot_download("regnant-io/kw5-230M")             # ~1.04 GB, cached
sys.path.insert(0, path)          # the repo ships modeling_kw5v2.py and kw5_tokenizer.py
from modeling_kw5v2 import KW5V2ForCausalLM
from kw5_tokenizer import make_encoder

model = KW5V2ForCausalLM.from_pretrained(path)
sp = spm.SentencePieceProcessor(model_file=f"{path}/tokenizer.model")
encode = make_encoder(sp)

torch.manual_seed(0)
ids = encode("Tanzania ni nchi ya Afrika Mashariki.")       # no <s>: pretraining had none
out = model.generate(ids, max_new_tokens=60, temperature=0.7, top_p=0.9,
                     repetition_penalty=1.1, eos_id=2)
print(sp.decode(out))

Output on CPU, as run against these weights:

Tanzania ni nchi ya Afrika Mashariki. Nchi hii inapakana na Kenya, Uganda,
Rwanda na Burundi. Ni nchi yenye historia nzuri sana ya karne nyingi.

 Historia

 Historia ya awali
 Kulikuwa na falme nyingi tofauti katika Afrika Mashariki wakati wa ukoloni.
Mojawapo ya falme hizo ilikuwa Ufalme wa Buganda ambao ulitawala eneo la
Uganda kuanzia

Fluent, shaped like an encyclopedia article (a heading, a subheading), and wrong in the details: Tanzania also borders Mozambique, Malawi, Zambia, the DRC and the ocean, and the kingdoms came long before colonial rule. sp.decode leaves a space at the start of each line the encoder split; strip it if it matters.

Encode with kw5_tokenizer.py, not sp.encode. It encodes line by line, so the first word of every line keeps its word-boundary piece, which is how the corpus was packed. sp.encode on a multi-line string splits the first word after each newline into pieces the model rarely saw. Do not prepend <s>: documents were packed without it.

A stock AutoModelForCausalLM will not load it: the block has Canon layers. modeling_kw5v2.py needs only PyTorch and safetensors.

What it does

Greedy, in the pretraining layout, before any fine-tuning:

• Arithmetic with its working. 56 of 57 bare sums, 56 of 60 expressions with brackets (slips in long multiplication) and 44 of 60 word problems, written out column by column as the generated arithmetic register taught it. The decay, which doubled that register, bought 7 and 5 of those items.
• Fluent Kiswahili on news, encyclopedia and educational prose.
• The chat format, unprompted. Its decay mixed chat four times over, so it sometimes writes <|user|> and <|end|> blocks of its own. That is what made the instruct fine-tune cheap; in a plain continuation, stop on <|end|> (id 7) as well as </s>.

What it gets wrong

• Facts. It states Kilimanjaro's height and Nyerere's birthday wrongly and confidently.
• Loops on open prompts: four of ten greedy samples repeated themselves (3-gram repetition 0.64 to 0.87). Sample with a repetition penalty.
• Scope. Kiswahili only, mostly Tanzanian, partly machine-translated corpus. No safety tuning.

Training

Shape	24 layers, 896 wide, feed-forward 2,432; grouped-query attention, 14 query heads on 2 key-value heads; tied embeddings
Block	pre-norm RMSNorm with QK-norm, SwiGLU, RoPE (theta 204,522 at 2,048), Canon layers at four points per block, document-boundary masking
Tokenizer	32,000 SentencePiece BPE pieces trained on Kiswahili, digits split one by one, byte fallback
Tokens	7,895,777,280 over 15,060 steps of 524,288; 34.3 per stored parameter, so it clears Chinchilla's 20 with room
Corpus	5.58B tokens, about 1.4 passes: the audited v2 corpus (web, news, encyclopedia, translated educational, parliamentary, religious, long-form, chat), translated educational web (TransWebEdu), long-form PDFs (FinePDFs), Swahili FineWiki, generated arithmetic and dialogue
Schedule	warmup-stable-decay; context 1,024, then 2,048 for the decay over the last 15%, weighted toward encyclopedia (x3), arithmetic (x2) and chat (x4)
Optimiser	Muon for matrices, AdamW for the tied embedding and the norms
Hardware	Kaggle TPU v5e-8, five sessions of up to 8.5 hours
Validation loss	2.3656 at step 15,000

The size was not chosen by taste. A throughput sprint measured three shapes on the TPU, and the rule was the largest one that reaches 30 training tokens per stored parameter in the hours available. The 230M shape ran at 57,109 tokens per second in the first session's calibration, and the token budget was locked there and stored in every checkpoint.

Before a single step ran, a checkpoint past step 200 had to reload in a fresh process and reproduce its loss, and greedy samples were read every 500 steps. The second generation lost about 27 TPU-hours to a reload path nobody had tested; this one tested it first.

Citation

@misc{regnant2026kw5v3,
  title  = {KW5-230M: a Kiswahili language model from the tokenizer up},
  author = {Regnant},
  year   = {2026},
  url    = {https://huggingface.co/regnant-io/kw5-230M}
}

Apache 2.0.