regnant-io/kw5-lite-instruct

KW5-Lite Instruct

Developed by: Regnant
Model type: Causal Language Model (Instruction-tuned)
Language: Swahili (sw)
License: Apache 2.0
Base Model: regnant-io/kw5-lite-base

Model Description

KW5-Lite Instruct is a 109.5M parameter instruction-tuned Swahili language model fine-tuned from KW5-Lite Base using Low-Rank Adaptation (LoRA). The model is optimized for following instructions, engaging in multi-turn conversations, and generating contextually appropriate responses in Swahili.

Model Details

Model Specifications

Specification	Value
Parameters	134.1M total (109.5M trainable + 24.6M embeddings)
Architecture	Llama-compatible decoder-only transformer
Layers	12
Hidden Size	768
FFN Hidden Size	2048
Attention Heads	12
Context Length	2048 tokens
Vocabulary Size	32,000 (SentencePiece)
Normalization	RMSNorm
Positional Encoding	RoPE (Rotary Position Embeddings)
Activation Function	SwiGLU
Precision	FP16

Fine-tuning Details

Parameter	Value
Method	LoRA (Low-Rank Adaptation)
LoRA Rank	16
LoRA Alpha	32
LoRA Dropout	0.05
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable Parameters	2.8M (2.49% of total)
Training Data Size	22,543 examples (~3.39M tokens)
Training Duration	52 minutes
Hardware	NVIDIA T4 GPU (16GB VRAM)
Framework	Unsloth + Transformers + TRL

Training Configuration

Hyperparameter	Value
Optimizer	AdamW 8-bit
Learning Rate	2e-4
Warmup Steps	100
Batch Size	4 (per device)
Gradient Accumulation	8 steps
Effective Batch Size	32
Training Epochs	3
Total Steps	2,115
LR Schedule	Cosine with warmup
Weight Decay	0.01
Max Gradient Norm	1.0
Seed	1337

Performance Metrics

Metric	Value
Initial Loss	3.82
Final Loss	2.52
Loss Reduction	34%
Perplexity	~12.4
Checkpoints Saved	5 (every 500 steps)

Training Data

Dataset Composition

The model was fine-tuned on 22,543 carefully curated Swahili instruction-response pairs covering:

• Instruction-following examples: Direct question-answer pairs
• Multi-turn conversations: Contextual dialogue sequences
• Safety and refusal patterns: Appropriate response boundaries
• Cultural knowledge: Swahili proverbs (methali), traditions (mila), customs (utamaduni)
• Educational content: Explanations, definitions, and teaching materials
• Code-switching examples: Swahili-Sheng mixed language patterns

Data Quality

• Zero duplicate examples (verified)
• Response-only masking applied (prompts not included in loss)
• Balanced distribution across task types
• Tanzanian Swahili dialect primary focus

Dataset Availability

Training dataset: regnant-io/swahili-instruction-22k (to be published)

Usage

Installation

pip install transformers torch accelerate

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("regnant-io/kw5-lite-instruct")
tokenizer = AutoTokenizer.from_pretrained("regnant-io/kw5-lite-instruct")

messages = [{"role": "user", "content": "Eleza umuhimu wa elimu."}]
formatted = tokenizer.apply_chat_template(
    messages, 
    tokenize=False, 
    add_generation_prompt=True
)

inputs = tokenizer(formatted, return_tensors="pt", add_special_tokens=False)

# Define stop tokens to prevent multi-turn hallucination
stop_token_ids = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("[INST]"),
]
stop_token_ids = [tid for tid in stop_token_ids if tid is not None]

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    min_new_tokens=10,
    temperature=0.2,
    top_p=0.9,
    do_sample=True,
    repetition_penalty=1.2,
    eos_token_id=stop_token_ids,
    pad_token_id=tokenizer.pad_token_id,
)

response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:], 
    skip_special_tokens=True
)
print(response)

Multi-turn Conversation

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("regnant-io/kw5-lite-instruct")
tokenizer = AutoTokenizer.from_pretrained("regnant-io/kw5-lite-instruct")

messages = []

while True:
    user_input = input("You: ").strip()
    if user_input.lower() in ("exit", "quit"):
        break
    
    messages.append({"role": "user", "content": user_input})
    
    formatted = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer(formatted, return_tensors="pt", add_special_tokens=False)
    
    stop_token_ids = [
        tokenizer.eos_token_id,
        tokenizer.convert_tokens_to_ids("[INST]"),
    ]
    stop_token_ids = [tid for tid in stop_token_ids if tid is not None]
    
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        min_new_tokens=10,
        temperature=0.2,
        top_p=0.9,
        do_sample=True,
        repetition_penalty=1.2,
        eos_token_id=stop_token_ids,
        pad_token_id=tokenizer.pad_token_id,
    )
    
    response = tokenizer.decode(
        outputs[0][inputs.input_ids.shape[1]:], 
        skip_special_tokens=True
    ).strip()
    
    print(f"Assistant: {response}\n")
    messages.append({"role": "assistant", "content": response})

Streaming Generation

from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
from threading import Thread

model = AutoModelForCausalLM.from_pretrained("regnant-io/kw5-lite-instruct")
tokenizer = AutoTokenizer.from_pretrained("regnant-io/kw5-lite-instruct")

messages = [{"role": "user", "content": "Nini maana ya ujamaa?"}]
formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(formatted, return_tensors="pt", add_special_tokens=False)

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

stop_token_ids = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("[INST]")]
stop_token_ids = [tid for tid in stop_token_ids if tid is not None]

generation_kwargs = dict(
    **inputs,
    max_new_tokens=256,
    temperature=0.2,
    repetition_penalty=1.2,
    eos_token_id=stop_token_ids,
    pad_token_id=tokenizer.pad_token_id,
    streamer=streamer,
)

thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()

print("Assistant: ", end="", flush=True)
for token in streamer:
    print(token, end="", flush=True)
print()

Generation Parameters

Recommended Configuration:

Parameter	Value	Purpose
temperature	0.7	Balanced creativity and focus
top_p	0.9	Nucleus sampling for diversity
repetition_penalty	1.2	Prevents repetitive patterns
min_new_tokens	10	Ensures meaningful responses
max_new_tokens	256	Sufficient for detailed answers
do_sample	True	Enables stochastic generation

Critical Requirements:

• Use add_special_tokens=False when tokenizing formatted prompts
• Include [INST] token in stop conditions to prevent multi-turn hallucination
• Decode only new tokens: outputs[0][inputs.input_ids.shape[1]:]

Prompt Engineering Guide

Why Prompting Matters

This model was fine-tuned on instruction-following examples. The way you phrase your prompt significantly impacts response quality.

Effective Prompting Strategies

✅ GOOD: Instruction-Style Prompts

Use directive verbs that match training patterns:

Eleza umuhimu wa elimu.
Orodhesha mambo matano ya kufanya Dar es Salaam.
Nifundishe kuhusu akili bandia.
Taja sababu tano za mabadiliko ya hali ya hewa.
Andika ufafanuzi wa neno "demokrasia".
Fafanua jinsi gari linavyofanya kazi.

❌ WEAK: Casual Questions

Elimu ni nini?
Una mambo gani ya kufanya?
Unajua akili bandia?
Kuna nini kuhusu hali ya hewa?
Demokrasia ni nini?

Comparison: Same Topic, Different Results

Example 1: Weak Prompt

User: Diamond Platnumz ni nani?
Model: [Confused, incomplete response, mixes up information]

Example 2: Strong Prompt

User: Eleza kuhusu Diamond Platnumz
Model: 1. Diamond Platnumz ni msanii... [Structured, formatted response]

Best Practices

1. Start with Action Verbs

• Eleza (Explain)
• Orodhesha (List)
• Taja (Mention/State)
• Andika (Write)
• Fafanua (Clarify/Explain in detail)
• Nifundishe (Teach me)

2. Be Specific About Format

✅ "Orodhesha mambo MATANO ya kufanya" (list FIVE things)
✅ "Taja sababu TATU za..." (mention THREE reasons)
✅ "Eleza kwa maneno MAFUPI" (explain in SHORT words)

3. Add Context for Better Responses

"Nifundishe kuhusu akili bandia kama mimi ni mtoto wa miaka 10"
"Eleza mabadiliko ya hali ya hewa kwa lugha rahisi"
"Andika muhtasari wa hadithi hii kwa sentensi tatu"

4. Use Complete Sentences

✅ "Eleza umuhimu wa elimu katika maisha ya kila siku"
❌ "elimu muhimu"

Prompt Templates

For Explanations:

Eleza [topic]
Nifundishe kuhusu [topic]
Fafanua maana ya [concept]

For Lists:

Orodhesha mambo [number] ya [topic]
Taja sababu za [topic]
Niambie mambo ambayo [condition]

For Comparisons:

Linganisha [A] na [B]
Tofautisha kati ya [A] na [B]

For Definitions:

Andika ufafanuzi wa [term]
[Term] ni nini?
Maana ya [term] ni ipi?

Response Quality Indicators

Signs of Good Prompt:

• ✅ Structured response (numbered lists, paragraphs)
• ✅ Appropriate length (not too short)
• ✅ Follows requested format
• ✅ Coherent explanation flow

Signs of Weak Prompt:

• ❌ Incomplete or abrupt responses
• ❌ Generic or vague content
• ❌ Doesn't follow intended format
• ❌ Confused topic handling

Multi-turn Conversations

Building Context:

Turn 1: "Eleza akili bandia"
Turn 2: "Inakuaje hio?" (How does that work?)
Turn 3: "Nifundishe kama mimi ni mtoto" (Teach me like a child)

The model tracks context across turns, so follow-up questions benefit from established topic.

Important Note

Good prompting improves response structure and coherence, but does not solve factual accuracy issues. The model may still generate plausible-sounding but incorrect information for topics outside its training data. Always verify critical facts from authoritative sources.

Chat Template Format

The model uses Llama2-style chat formatting:

Single turn:

<s>[INST] {user_message} [/INST] {assistant_response}</s>

Multi-turn:

<s>[INST] {user_1} [/INST] {assistant_1}</s>[INST] {user_2} [/INST] {assistant_2}</s>

Example Outputs

Example 1: Educational Query

Input:

Eleza umuhimu wa elimu.

Output:

1. Ni muhimu kutambua na kutofautisha mambo kwa kuangalia sifa za mtu mmoja, tabia ya watu, na mwenendo wa jamii.
2. Elimu ni muhimu kwa sababu inakuza uelewa wa msingi wa mfumo wa kijamii na kiuchumi.
3. Kwa kusoma na kuelewa data, una uwezo wa kutoa ufahamu juu ya maendeleo na mafanikio ya teknolojia.

Example 2: Historical Information

Input:

Je, unaweza kunieleza kuhusu historia ya Tanzania?

Output:

1. Serikali za Jamhuri ya Muungano wa Tanzania iliruhusu uhuru wa Tanganyika kutoka ukoloni wa Uingereza mwaka 1964, na kusababisha kuundwa kwa Shirikisho la Nchi za Afrika Mashariki (EAC).
2. Mwaka 1957, Serikali Kuu ya Tanzania ilianzisha Tume ya Pamoja ya Kujadili Sera na Sheria chini ya Mwenyekiti wake, Benjamin Mkapa, ili kusimamia shughuli za kiuchumi kati ya nchi hizo mbili.

Example 3: Multi-turn Conversation

Turn 1:

User: Akili bandia ni nini?
Assistant: Mfumo wa neva wa fahamu (CNS) hutumia data ili kutambua aina za akili na mwili zinazoitwa seli zisizo za kawaida.

Turn 2:

User: Inakuaje hio?
Assistant: Ubongo huo unaweza kuwa na seti ya data inayoelezea tabia au mazingira, ambayo inaweza kutumika kuchunguza habari kama vile picha, sauti, video, nk.

Performance Analysis

What This Model Does Well ✅

Conversational Abilities:

• Fluent, grammatically correct Swahili generation
• Maintains context across multi-turn conversations
• Appropriate response boundaries (no fake continuations)
• Natural dialogue flow with context awareness

Instruction Following:

• Follows format instructions (lists, paragraphs, explanations)
• Adapts explanation style based on user requests
• Responds appropriately to follow-up questions

Content Generation:

• Educational explanations (concepts, processes, definitions)
• Abstract and conceptual topics (AI, education, technology)
• General historical context (verified topics in training data)
• Cultural knowledge (proverbs, traditions, customs)

Example Success Cases:

• "Eleza umuhimu wa elimu" → Clear educational explanation ✅
• "Akili bandia ni nini?" → Coherent AI explanation ✅
• Multi-turn technical discussions with context tracking ✅

Critical Limitations ⚠️

Factual Knowledge Gaps:

The model is a language generator, not a knowledge database. It will confidently generate plausible-sounding but factually incorrect information for topics outside its training data.

Example failure case:

Query: "Diamond Platnumz ni nani?" (Who is Diamond Platnumz?)
Response: Generates fake song names and biographical details
Reality: Model has no factual knowledge of celebrities/pop culture

Specific Limitations:

• ❌ No knowledge of celebrities, brands, or current events
• ❌ Limited specific factual knowledge (tourist attractions, products)
• ❌ May fabricate details when asked for concrete lists
• ❌ Cannot access real-time information
• ⚠️ Incomplete responses for specific factual list queries
• ⚠️ Generic when specialized domain knowledge required

Technical Constraints:

• 109M parameters (limited capacity vs larger models)
• Trained on 22.5K examples (limited knowledge coverage)
• Tanzanian Swahili dialect primary focus
• 2048 token context window (~1000 effective in conversation)
• No retrieval system or external knowledge access

Model Behavior Patterns

Response Style:

• Numbered list format (learned from training data)
• Sequential numbering across turns (shows context tracking)
• Explanation-first approach for concepts
• Generates plausible Swahili even for unknown topics (hallucination risk)

Reliability Scale:

• High confidence: Educational concepts, general explanations, conversation
• Medium confidence: Historical topics in training data, cultural knowledge
• Low confidence: Specific facts, people, places, current events
• No confidence: Real-time info, celebrities, detailed factual lists

Intended Use

✅ Recommended Applications

Primary Use Cases:

• Educational Tools: Learning platforms, tutoring systems, concept explanations
• Conversational AI: Chatbots and virtual assistants for general topics
• Content Generation: Blog posts, social media, marketing copy in Swahili
• Q&A Systems: Customer support for general inquiries (non-factual)
• Language Tools: Writing assistance, text completion, style adaptation
• Cultural Applications: Preserving and promoting Swahili language
• Research: Computational linguistics and NLP studies

What It Excels At:

• Explaining abstract concepts (education, technology, philosophy)
• General conversation and dialogue
• Educational content generation
• Following formatting instructions
• Multi-turn contextual discussion

❌ Not Recommended For

Critical Exclusions:

• ❌ Medical, legal, or financial advice (no domain expertise)
• ❌ Factual queries requiring accuracy (people, places, products, events)
• ❌ High-stakes decision making (not reliable for critical choices)
• ❌ Real-time information (no internet access, knowledge cutoff)
• ❌ Professional translation (not trained as translation model)
• ❌ Fact-checking or verification (will hallucinate confidently)
• ❌ Celebrity/entertainment queries (no pop culture knowledge)
• ❌ Detailed factual lists (tourist spots, product names, etc.)

Important:

This model is designed for general instruction-following and conversation, not as a factual knowledge base. It will generate plausible-sounding responses even when it lacks actual knowledge. Always verify critical information from authoritative sources.

Production Deployment Considerations

Compute Requirements:

• Minimum: 4GB GPU VRAM (FP16)
• Recommended: 8GB+ GPU VRAM for batching
• CPU Inference: Possible but 10-50x slower

Latency Expectations:

• T4 GPU: ~1-2 seconds for 128 tokens
• A100 GPU: ~0.3-0.5 seconds for 128 tokens
• CPU (8 cores): ~10-20 seconds for 128 tokens

Scaling:

• Supports model parallelism for larger deployments
• Compatible with TensorRT, ONNX Runtime for optimization
• vLLM and Text Generation Inference (TGI) compatible

Evaluation

Qualitative Assessment

Evaluated on held-out test set with human review:

Aspect	Rating	Notes
Fluency	8/10	Natural Swahili with minor artifacts
Instruction Following	7/10	Generally follows instructions well
Factual Accuracy	6/10	Reasonable for trained knowledge, weaker on specifics
Coherence	8/10	Maintains topic and context effectively
Creativity	7/10	Generates diverse responses
Safety	8/10	Appropriate refusals for harmful requests

Overall Model Quality: 7/10 for a 109M parameter instruction model

Quantitative Metrics

Metric	Value
Training Loss (final)	2.52
Perplexity (estimated)	12.4
BLEU Score (held-out)	Not measured
ROUGE-L (held-out)	Not measured

Ethical Considerations

Bias and Fairness

• Model trained primarily on Tanzanian Swahili; may not generalize to all dialects
• Training data may contain cultural biases present in source material
• Limited representation of minority perspectives within Swahili-speaking communities
• Users should validate outputs for fairness in production applications

Safety

• Model includes safety training data with refusal patterns
• May still generate inappropriate content if adversarially prompted
• Not suitable for filtering harmful content without additional safety layers
• Recommend implementing content moderation for public-facing applications

Privacy

• Model does not store user conversations
• Training data was sourced from public datasets and synthetically generated examples
• No personally identifiable information (PII) in training data
• Users responsible for handling sensitive data in production deployments

Environmental Impact

Training carbon footprint estimate:

• Hardware: NVIDIA T4 GPU
• Duration: 52 minutes
• Estimated CO₂: ~0.02 kg (regional grid mix dependent)
• Efficient training via LoRA reduced compute requirements by 97%

Technical Details

Model Architecture

LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 768)
    (layers): ModuleList(
      (0-11): 12 x LlamaDecoderLayer(
        (self_attn): LlamaSdpaAttention(
          (q_proj): Linear(768, 768)      # LoRA adapted
          (k_proj): Linear(768, 768)      # LoRA adapted
          (v_proj): Linear(768, 768)      # LoRA adapted
          (o_proj): Linear(768, 768)      # LoRA adapted
          (rotary_emb): LlamaRotaryEmbedding()
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(768, 2048)  # LoRA adapted
          (up_proj): Linear(768, 2048)    # LoRA adapted
          (down_proj): Linear(2048, 768)  # LoRA adapted
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm()
        (post_attention_layernorm): LlamaRMSNorm()
      )
    )
    (norm): LlamaRMSNorm()
  )
  (lm_head): Linear(768, 32000, bias=False)
)

LoRA Configuration

{
  "r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "target_modules": [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj"
  ],
  "bias": "none",
  "task_type": "CAUSAL_LM"
}

Tokenizer Details

• Type: SentencePiece Unigram
• Vocabulary: 32,000 tokens
• Special Tokens:
  • <s>: Beginning of sequence (BOS)
  • </s>: End of sequence (EOS)
  • <unk>: Unknown token
  • <pad>: Padding token
• Chat Tokens: [INST], [/INST]

Reproducibility

Training is reproducible with fixed seeds:

seed = 1337
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)

All hyperparameters and data processing steps documented in training notebook.

Citation

Model Citation

@misc{kw5-lite-instruct-2026,
  author = {Regnant},
  title = {KW5-Lite Instruct: A 109.5M Parameter Instruction-Tuned Swahili Language Model},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/regnant-io/kw5-lite-instruct}},
}

Base Model Citation

@misc{kw5-lite-base-2026,
  author = {Regnant},
  title = {KW5-Lite: A 109.5M Parameter Swahili Language Model},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/regnant-io/kw5-lite-base}},
}

Acknowledgments

• Base Model: Pre-trained on 1.88B-2.1B Swahili tokens by Regnant
• Fine-tuning Framework: Unsloth for efficient LoRA training
• Infrastructure: Google Colab T4 GPU
• Libraries: Transformers, TRL, PEFT, Accelerate

License

Apache License 2.0

Contact

Organization: Regnant

---

Version: 1.0
Release Date: August 2026
Model Card Authors: Regnant Team