regnant-io/kw5-149M-instruct

KW5 149M Instruct

Instruction-tuned from kw5-149M, a Swahili model pretrained on 1.97B tokens.

Built by Regnant.

It answers in Swahili, stops when it is finished, produces an optional <mawazo> reasoning block, and says it does not know when it does not know. That last one is the reason this model exists: the previous generation answered confidently and wrongly, and its mix contained 137 abstention examples out of 107,150.

149M transformer parameters, 173.6M total. The input embedding and the output head are untied rather than shared, so they count separately and the hub's Safetensors panel reports the larger figure. Both describe the same model.

It is a small model, so read the limitations before you trust an answer.

Two things to get right

1. The prompt starts with <s>. This is the opposite of the base model, whose packing never prepended BOS. Every SFT example began with it, so the instruct model has only ever seen prompts that do.

2. Generation stops at <|end|> (id 7), not </s>. Stopping at </s> runs past the end of every reply.

The exact training format:

<s><|user|>
{your message}<|end|>
<|assistant|>

Do not build that by encoding the string. <s> is a SentencePiece control symbol, so sp.encode("<s>") does not give you id 1 -- it gives you the three ordinary pieces for the characters <, s, >, and your prompt no longer matches anything the model was trained on. Prepend the id yourself, as the snippet below does. <|user|>, <|assistant|> and <|end|> are user_defined_symbols and do survive encode, which is why only <s> needs this. The export refuses to publish if these stop being true.

Quick start (runs in Google Colab as-is)

!pip install -q huggingface_hub sentencepiece

import sys, torch, sentencepiece as spm
from huggingface_hub import snapshot_download

path = snapshot_download("regnant-io/kw5-149M-instruct")     # ~700 MB, cached
sys.path.insert(0, path)                  # this repo ships modeling_kw5v2.py
from modeling_kw5v2 import KW5V2ForCausalLM

device = "cuda" if torch.cuda.is_available() else "cpu"
model = KW5V2ForCausalLM.from_pretrained(path).to(device)
sp = spm.SentencePieceProcessor(model_file=f"{path}/tokenizer.model")

BOS, END = 1, 7

def chat(message, max_new_tokens=200):
    # <s> is a control symbol: sp.encode("<s>") does NOT give you id 1.
    ids = ([BOS] + sp.encode("<|user|>\n" + message + "<|end|>\n")
           + sp.encode("<|assistant|>\n"))
    out = model.generate(ids, max_new_tokens=max_new_tokens,
                         temperature=0.3, top_p=0.9,
                         repetition_penalty=1.1, eos_id=END)
    return sp.decode([t for t in out[len(ids):] if t not in (END, 2)])

print(chat("Eleza kwa ufupi maana ya elimu."))

Eleza kwa ufupi maana ya elimu.

Elimu ni mchakato wa kujifunza na kukua katika ujuzi, maarifa, na stadi za kutatua matatizo, kufikiri kwa ukosoaji, na kufanya maamuzi sahihi. Inajumuisha mafunzo ya kitaaluma, kazi ya kujitegemea, mawasiliano wazi, na uwezo wa kushirikiana na wengine. Elimu ina jukumu muhimu katika kukuza ubunifu, kufikiri kwa ukosoaji, na kufikiri kwa kina. Inahusisha pia kuwa na uwezo wa kutumia ujuzi huo kwa njia bora zaidi, kama vile kusoma, kuandika, kusikiliza, na kuzungumza. Elimu pia inaweza kusaidia watu kujenga mahusiano yenye nguvu na wengine, ambayo inaweza kusababisha mafanikio katika kazi zao. Kwa hiyo, elimu ni mchakato unaoendelea ambao unakuza ujuzi na maarifa, na inaweza kutumika kama msingi wa maendeleo ya mtu binafsi na jamii.

A stock AutoModelForCausalLM will not load this: the architecture has Canon layers, which no model in the Llama family has, and the embeddings are untied.

Untied embeddings

The input embedding and the output head are not tied. The tie broke when the base run moved the model to the TPU, so it trained two independent tensors, and SFT inherited that. config.json records tie_embeddings: false and the shipped modeling_kw5v2.py honours it. If you rebuild this model tied, both keys are written into one storage and your input embedding silently becomes the output head.

What it does well

Abstention. Asked something unanswerable, it declines and says why:

Nani alikuwa mwenyekiti wa kijiji cha Mkwawa mwaka 1987?

<mawazo>Swali hili linahitaji taarifa mahususi ambazo sina. Nikikisia, kuna
uwezekano mkubwa wa kukosea, na jibu lisilo sahihi ni baya kuliko kukiri
kutojua.</mawazo>
Sina uhakika. Sina taarifa za kutosha kuhusu jambo hilo, na sitaki kukupa jibu
lisilo sahihi.

Be aware this is learned phrasing, not reasoning about uncertainty, and it is not reliable. On eight unanswerable prompts it declined six times and confidently invented an answer twice -- including the two below, which are not obviously different in kind from the ones it refused:

Jina la mbwa wa jirani yangu ni nani?
-> Jina la mbwa wa jirani yangu ni David.

Hali ya hewa itakuwaje kesho mjini Arusha?
-> Kesho mjini hapa ni hatari sana kwa sababu ya joto kali na mvua kubwa ...

The abstention data is templated and the refusals are close to word-for-word identical across prompts, so what the model learned is the phrasing, triggered by surface features of the question. Treat it as a useful tendency, not a guarantee, and never as a calibrated confidence estimate.

Stopping. It emits <|end|> and stops on its own. The base model terminated on about half of chat-formatted prompts; this one is much better. Give it room, though: at a 72-token cap about a quarter of replies were still mid-sentence, and that is the cap, not the model. Measured stop rate at a 160-token cap: 0.94 at the shipped setting, 0.84 to 0.94 across every setting tried.

Following simple constraints. "Write one sentence about the sea" produces one sentence.

Selective reasoning. The <mawazo> block appears for arithmetic and abstention, and is skipped for simple factual questions. Nothing trained that discrimination explicitly.

What it gets wrong

Arithmetic. It shows working and the working is wrong. Asked "a shop has 24 oranges, three quarters are sold, how many remain", it produces a tidy step-by-step block and answers 12. The correct answer is 6. The maths data is machine-translated chain-of-thought and the model learned the shape of reasoning without the substance. Do not use this model for arithmetic.

Facts. It says the capital of Tanzania is Dar es Salaam. It is Dodoma, and the base model gets it right, so instruction tuning made this particular fact worse. A model this size has very little room for world knowledge; treat every factual claim as unverified.

Anything long or multi-turn. Trained at 1024 tokens with single-turn data dominant.

Training

Base	kw5-149M, 1.97B tokens, WSD decay completed
SFT	full-parameter, 2 epochs, 3,550 steps, 465,305,600 tokens
Optimizer	AdamW, lr 1e-5, weight decay 0, 3% warmup
Context	1024, assistant-only loss masking
Hardware	Kaggle TPU v5e-8

The mix was roughly 57% instruction following, 20% reasoning, 10% abstention, plus grounded QA, summarisation, safety, and hand-curated Swahili sets covering proverbs, grammar, culture and code-switching.

Forgetting was measured, not assumed. The pretraining held-out split was scored every 50 steps against a baseline taken before the first update:

baseline 3.2031  ->  final 3.2640     drift +0.0609     tolerance +0.35

The base language modelling ability is essentially intact.

Evaluation

No downstream benchmark results are published here. Belebele-sw, AfriXNLI and the rest have not been run on this checkpoint. When they are, they go here with sample sizes and standard errors, not before. Everything above is behaviour on a handful of probes, and is described as such.

Intended use

Swahili assistant tasks where the answer is in the prompt or the question is simple: rewriting, summarising a supplied passage, short explanations, grammar help. It is a reasonable base for further fine-tuning.

Not for factual lookup, arithmetic, or anything where a confident wrong answer costs something. No safety tuning beyond a small refusal set.

Decoding

Defaults live in generation_config.json: temperature 0.3, top-p 0.90, repetition penalty 1.1. Chosen by sweeping six settings over 8 answerable and 8 unanswerable prompts: it tied for the best stop rate (0.94) and the best abstention rate (0.75) while beating both on repetition. Raising the temperature above 0.3 cost abstention monotonically: 0.75, 0.62, 0.56 at t0.3, t0.5, t0.7.

Apache 2.0.