regnant-io/kw5-230M-instruct

KW5-230M Instruct

A Kiswahili assistant small enough to run on a laptop CPU. It was trained by Regnant from the tokenizer up, on Kiswahili text, and fine-tuned to answer from what you give it: a passage, an earlier turn in the conversation, a sum to work.

230,423,424 parameters, embeddings tied. Instruction-tuned from KW5-230M, the third-generation base, pretrained on 7.90B tokens of Kiswahili.

What it is measurably good at, on a frozen suite of 1,464 Kiswahili items built before any v3 training:

• Arithmetic. 91.4% on bare sums up to four digits and 70.5% on mixed expressions with brackets, working each step in a <mawazo> block. The second-generation instruct model scored 2.3% on the first.
• Remembering a conversation. 98.8% recall of a fact told 3 to 9 turns earlier, and 82.5% "you never told me" when it was not.
• Reading a passage. 58.4% on 250 questions over real Swahili Wikipedia and TyDiQA passages, and 70.0% when the passage does not hold the answer and the right reply is to say so.

What it gets wrong is in its own section below, with numbers. Read it before you trust an answer.

Two things to get right

1. Encode with the encoder this repo ships. kw5_tokenizer.py encodes line by line, so the first word of every line keeps its word-boundary piece. That is how the corpus was packed and how the model was fine-tuned. Plain sp.encode on the whole prompt fragments the first word after every newline (\nEleza becomes ▁ <0x0A> E leza), and the model has never seen that.

2. Start with id 1 and stop at <|end|> (id 7). <s> is a SentencePiece control symbol: encoding the string "<s>" gives three ordinary pieces, not id 1. Prepend the id yourself. Generation ends at <|end|>, not </s>.

The training format:

<s><|user|>
{your message}<|end|>
<|assistant|>

A conversation repeats the user and assistant blocks, each closed by <|end|>. A <|system|> block may open it.

Quick start (runs in Google Colab as-is)

!pip install -q huggingface_hub sentencepiece safetensors

import sys, torch, sentencepiece as spm
from huggingface_hub import snapshot_download

path = snapshot_download("regnant-io/kw5-230M-instruct")    # ~1.04 GB, cached
sys.path.insert(0, path)          # the repo ships modeling_kw5v2.py and kw5_tokenizer.py
from modeling_kw5v2 import KW5V2ForCausalLM
from kw5_tokenizer import make_encoder

device = "cuda" if torch.cuda.is_available() else "cpu"
model = KW5V2ForCausalLM.from_pretrained(path).to(device)
sp = spm.SentencePieceProcessor(model_file=f"{path}/tokenizer.model")
encode = make_encoder(sp)

BOS, END, EOS = 1, 7, 2

def chat(turns, max_new_tokens=256):
    """turns: [("user", "..."), ("assistant", "..."), ("user", "...")]"""
    ids = [BOS]                                  # the id, never encode("<s>")
    for role, text in turns:
        if role == "user":
            ids += encode(f"<|user|>\n{text}<|end|>\n")
        else:
            ids += encode("<|assistant|>\n") + encode(text) + encode("<|end|>\n")
    ids += encode("<|assistant|>\n")
    out = model.generate(ids, max_new_tokens=max_new_tokens, temperature=0.0, eos_id=END)
    return sp.decode([t for t in out[len(ids):] if t not in (END, EOS)])

print(chat([("user", "Hesabu: 71 - 17 + 20 = ?")]))
print(chat([("user", "Mji mkuu wa Kenya ni upi?"),
            ("assistant", "Mji mkuu wa Kenya ni Nairobi."),
            ("user", "Na wa Uganda je?")]))

Every number on this card was measured greedy (temperature=0.0), as above.

A stock AutoModelForCausalLM will not load it: the block has Canon layers, short causal convolutions that no Llama-family model has. modeling_kw5v2.py needs only PyTorch and safetensors.

What it sounds like

These are replies from the evaluation run, unedited.

A sum, worked. Only arithmetic carries a <mawazo> block, so a plain question is answered without reasoning in front of it:

Hesabu: 71 - 17 + 20 = ?

<mawazo>Tunafuata mpangilio wa hesabu: mabano kwanza, kisha kugawanya na
kuzidisha kutoka kushoto kwenda kulia, kisha kujumlisha na kutoa kutoka
kushoto kwenda kulia.
Tunatoa 71 - 17. ... Kwa hiyo 71 - 17 = 54. Sasa tuna: 54 + 20.
Tunajumlisha 54 + 20. ... Kwa hiyo 54 + 20 = 74.</mawazo>
Jibu ni 74.

A passage. Asked how many people live in Morogoro region, over a passage that says it: 2,218,492.

A follow-up. After "Mji mkuu wa Kenya ni upi?" and its answer, "Na wa Uganda je?" gets Mji mkuu wa Uganda ni Kampala. The second-generation model refused this one and named Kampala inside the refusal.

Something it was never told. Asked about a city several turns into a conversation that never mentioned one: Hujaniambia mji wowote. ("You haven't told me any city.")

Who made it. Every self-introduction in its fine-tuning names it KW5, an AI assistant from Regnant, across sixteen ways of asking.

Release bars

Fourteen bars were frozen on 2026-10-15, the day before training began, so they could not drift toward whatever the model scored. A missed bar is reported as missed. Seven of fourteen are met.

capability	axis	n	score [95% interval]	bar
bare arithmetic, up to 4 digits	arithmetic	128	91.4 [85.3, 95.1]	85	met
order of operations, 2 to 5 operations	bodmas	200	70.5 [63.8, 76.4]	60	met
recall a fact told 3 to 9 turns back	mt_recall	80	98.8 [93.3, 99.8]	70	met
say so when a fact was never told	mt_untold	40	82.5 [68.0, 91.3]	60	met
follow-up answered like the cold question	followup	8	87.5 [52.9, 97.8]	75	met
say the passage lacks the answer	grounded_absent	60	70.0 [57.5, 80.1]	60	met
decline what nobody could answer	refusal_unanswerable	40	97.5 [87.1, 99.6]	70	met
read a real passage	grounded_real	250	58.4 [52.2, 64.3]	60	missed
answer what it should answer	refusal_answerable	30	86.7 [70.3, 94.7]	90	missed
an instruction that persists	mt_persist	20	35.0 [18.1, 56.7]	70	missed
word problems, 1 to 3 steps, unseen scenarios	word_problem2	120	24.2 [17.4, 32.6]	40	missed
AfriMGSM, grade-school maths	afrimgsm	100	1.0 [0.2, 5.4]	5	missed
answer from a list that changed	mt_state	40	5.0 [1.4, 16.5]	50	missed
a balance carried across turns	mt_arith	40	2.5 [0.4, 12.9]	40	missed

Wilson intervals. The followup row has 8 items and is indicative only.

Beside other small models

The same frozen suite, greedy, each model in its own chat template, on one Colab T4 (2026-10-28). Percent correct.

axis	n	KW5-230M Instruct	swahili-gemma-1b	Qwen3-0.6B	Qwen2.5-0.5B-Instruct
parameters		230M	1,000M	596M	494M
arithmetic	128	91.4	70.3	41.4	31.2
order of operations	200	70.5	44.5	83.0	57.0
word problems	24	62.5	66.7	16.7	0.0
word problems, unseen scenarios	120	24.2	39.2	0.8	2.5
AfriMGSM	100	1.0	8.0	3.0	2.0
read a real passage	250	58.4	58.4	24.4	16.8
pick the right one of two facts	48	77.1	70.8	16.7	52.1
TruthfulQA MC1	200	26.5	26.5	20.0	20.5
list exactly N items	16	56.2	50.0	18.8	12.5
content axes, 1,116 items		51.4	46.7	33.1	26.8
conversation and refusal axes, 348 items		63.8	10.1	0.9	2.9
all 1,464 items		54.4	38.0	25.4	21.1

Read the last two rows with care. The conversation and refusal axes (recall, never-told, state, persistence, follow-ups, declining) are scored against the replies a Kiswahili assistant should give: "Hujaniambia ...", "Kifungu hakisemi hilo.", "Sijui ...". KW5 was fine-tuned on declines in that voice, and the other models were not. swahili-gemma-1b answered "Mama yangu anaitwa nani?" by asking which mother was meant, a sensible reply that this scorer counts as a miss. The Qwen models often answered in English. On the 1,116 content items, where the reply is a number, a span or a choice, KW5 leads a model four times its size by 4.7 points, and loses word problems and AfriMGSM to it.

Gemma 3 270M and 1B (instruct) were in the plan and did not run: their licence was not accepted on the account that ran the notebook.

Public Kiswahili benchmarks

Belebele (reading, four choices), AfriXNLI (inference, three) and MasakhaNEWS (news topic, seven), scored by suffix likelihood in float32 on a T4, seed 20260914, with a one-sided test against chance.

model	params	Belebele (900)	AfriXNLI (600)	MasakhaNEWS (476)
KW5-230M Instruct	230M	35.6 [32.5, 38.7]	35.5	73.1 [69.0, 76.9]
KW5-230M (base)	230M	33.0 [30.0, 36.1]	36.5	74.2 [70.0, 77.9]
KW5-149M-instruct-r2	174M	29.0	34.3	72.5
Qwen2.5-0.5B-Instruct	494M	31.3	33.5	30.7
swahili-gemma-1b	1,000M	30.3	32.2	53.2
Qwen3-0.6B	596M	30.0	32.2	26.3
chance		25.0	33.3	21.0

Every model beats chance on Belebele. On Belebele this model's interval clears the second generation's (29.0, [26.1, 32.0]) and overlaps Qwen2.5's ([28.4, 34.4]), so it is the best score here without being separated from the next. No model in the comparison beats chance on AfriXNLI; the base comes closest (p = 0.055). MasakhaNEWS is news, which is in the pretraining mix, and the test articles were not checked for overlap.

What it gets wrong

State that changes. Told three guests are coming and then, one at a time, that each has dropped out, it answers Wageni watakaokuja ni Mussa na Yusufu. It keeps what was said first. 2 of 40.

Arithmetic across turns, and two-step word problems. It will work any sum you write down, but it does not reliably decide which sum to write. Thirty eggs a tray, three trays and sixteen loose eggs: it adds 30 and 16 and answers 46. A balance carried over several turns and then spent: 1 of 40. Grade-school problems (AfriMGSM): 1 of 100.

Declining too often. It declined 4 of 30 easy questions it should have answered, among them when Tanganyika became independent and which ocean Zanzibar is in. Of the 30 it answered 21 correctly. The fine-tuning taught declines on obscure facts, and some of that reached plain ones.

Facts. It is a 230M model. The base states Kilimanjaro's height and Nyerere's birthday confidently and wrongly. Treat every factual claim that is not in your prompt as unverified.

Instructions that should persist. Told to answer every following question in one word, it often answers the next one in full (Bahari ya Hindi.): 7 of 20.

Scope. Kiswahili only, mostly Tanzanian, and part of the corpus is machine translated. Not English, not code, not fractions or decimals.

Training

Pretraining (KW5-230M):

Shape	24 layers, 896 wide, feed-forward 2,432; grouped-query attention, 14 query heads on 2 key-value heads
Block	pre-norm RMSNorm with QK-norm, SwiGLU, RoPE, Canon layers (causal depthwise convolutions, kernel 4, at four points per block), document-boundary masking
Tokens	7,895,777,280 over 15,060 steps of 524,288; 34.3 per stored parameter
Corpus	5.58B tokens: the audited v2 corpus (web, news, encyclopedia, translated educational, parliamentary, religious, long-form, chat), translated educational web (TransWebEdu), long-form PDFs (FinePDFs), Swahili FineWiki, generated arithmetic and dialogue
Schedule	warmup-stable-decay; context 1,024, then 2,048 for the decay over the last 15%, weighted toward encyclopedia (x3), arithmetic (x2) and chat (x4)
Optimiser	Muon for matrices, AdamW for the tied embedding and the norms
Hardware	Kaggle TPU v5e-8, five sessions
Validation loss	2.3656 at step 15,000

Fine-tuning (this model): full-parameter, two epochs of 204,483 conversations, 3,195 steps, context 2,048, assistant turns only, from the base and at the base's RoPE (theta 204,522). The mix:

• curated Kiswahili instruction sets, Aya, MURI-IT and TyDiQA train, each read source by source and cut where it taught the wrong thing;
• grounded questions over real passages, including questions over another article's passage that are declined;
• generated arithmetic with its working, long conversations with facts told and never told, changing lists and persistent instructions, every equation checked by a test;
• about 10,700 conversations translated by Gemma 4 26B-A4B, kept only when every turn passed a check, and question-answer pairs it wrote over FineWiki;
• calibrated declines: PolyFact-sw questions the first fine-tune got right at most once in eight samples became declines (capped at 2,000), and those it got right six or more times became answers;
• 10,213 of the first fine-tune's own replies that passed a verifier (right number with every equation true, span present, exact count).

Declines are 10.4% of the mix. Any conversation whose user turn matched an evaluation prompt was dropped (224). General Kiswahili was probed throughout: the held-out loss drifted by +0.054 nats against a tolerance of 0.35.

One thing to know about the scores above. This mix was built after the first fine-tune was scored on the same frozen suite, to cover what it missed: imperative arithmetic phrasings, two-step problems, lists acknowledged without being echoed, declines in TyDiQA's format. The evaluation items themselves were kept out by hash and by test, but the suite's arithmetic is no longer novel in frame, only in wording. A held-out suite written after this release is the next measurement.

Not yet measured

• Safety. There is no dedicated safety tuning beyond declining, and the over-refusal and harmful-request sets (XSTest-sw, MultiJail-sw) have not been run on this release. Do not put it in front of the public without your own safeguards.
• Speed on a CPU. The second generation runs on a laptop CPU at 149M; this one is 1.3x its stored size and has not been timed.

Intended use

Research, and Kiswahili work where the answer is in the prompt and a person checks the output: questions about a supplied passage, working a sum someone wrote down, short conversations that refer back to earlier turns. A base for fine-tuning on your own Kiswahili data.

Not for factual lookup, multi-step word problems, or anything where a confident wrong answer costs something.

Citation

@misc{regnant2026kw5v3,
  title  = {KW5-230M: a Kiswahili language model from the tokenizer up},
  author = {Regnant},
  year   = {2026},
  url    = {https://huggingface.co/regnant-io/kw5-230M-instruct}
}

Apache 2.0.