AmhariXLM-0.5B

AmhariXLM-0.5B is a 565M-parameter Amharic/English language model, supervised fine-tuned for education-domain question answering. Trained from scratch on a curated Grade 1–10 curriculum plus Wikipedia and general reading material.

Developed by fidelizML — the Fideliz AI team.

Highlights

• 🎯 Fully trained from scratch — no external base model. Only the LLaMA architecture (config layout) was reused; all weights are ours.
• 📚 Education-focused — trained on textbooks, articles, and Wikipedia across primary and secondary subjects.
• 🌍 Bilingual — ~30% Amharic, ~70% English training mix.
• 💬 ChatML format — compatible with standard chat templates.
• 🆓 Apache 2.0 — free for commercial and non-commercial use.

Model details

Property	Value
Architecture	LLaMA-style (trained from scratch)
Parameters	~565 M (0.5 B rounded)
Layers	8
Hidden size	3072
Intermediate size	4096
Attention heads	12 (3 KV heads)
Vocab size	24,000 (SentencePiece, Unigram)
Context length	250 tokens
Precision	float16
Special tokens	bos=1, eos=2, pad=0
Prompt format	ChatML (`<\
Languages	Amharic (am), English (en)

Intended users

• Researchers studying low-resource African-language LLMs.
• Developers building Amharic/English educational tools (tutoring apps, Q&A systems, quiz generators).
• Educators exploring AI-assisted learning in Amharic.

Not intended for: medical, legal, financial, or safety-critical decisions. Not a calculator (see Limitations below).

Training

• Base: Trained from scratch — no external model weights.
• Data: Curated SFT corpus:
  • Primary-school textbooks (Grades 1–10)
  • Wikipedia articles (Amharic + English)
  • General reading material and articles
  • Curriculum-focused Q&A pairs
• Language split: ~30% Amharic, ~70% English.
• Method: Supervised fine-tuning with ChatML prompt format.

Evaluation

Reported results from our internal test suites. Ranges reflect different decoding strategies (greedy / sampling / best-of-N).

Benchmark	Score	Notes
Science definition Q&A (Bio/Phys/Chem)	90–95%	On curated definition set
Single-step arithmetic	45–80%	Range depends on decoding; small numbers work best
GSM8K (multi-step math)	37–55%	With inference adjustments (best-of-N / self-consistency)

What works well:

• Simple factual definitions (e.g., "What is a cell?", "What does CPU stand for?")
• Symbol lookup (Fe, Au — chemical symbols)
• Basic English grammar (plurals, opposites)
• Well-represented Amharic grammar facts

What fails:

• Some Amharic factual recall (e.g., "capital of Ethiopia" occasionally hallucinated)
• Multi-digit arithmetic (tokenizer treats whole numbers as single tokens)
• Physics formulas requiring derivation (Ohm's Law occasionally gibberish)

Usage

Install

pip install transformers torch sentencepiece huggingface_hub

Load the model

import torch
import sentencepiece as spm
from transformers import LlamaForCausalLM
from huggingface_hub import hf_hub_download

REPO_ID = "fidelizML/AmhariXLM-0.5B"

model = LlamaForCausalLM.from_pretrained(
    REPO_ID,
    torch_dtype=torch.float16,
    device_map="auto",
)

# Tokenizer is a raw SentencePiece model
tok_path = hf_hub_download(REPO_ID, "tokenizer.model")
sp = spm.SentencePieceProcessor(model_file=tok_path)

Chat with the model

The model uses ChatML prompt format:

import torch

IM_END_ID = sp.piece_to_id("<|im_end|>")

def chat(question, max_new_tokens=200, temperature=0.0):
    prompt = (
        f"<|im_start|>user\\n{question}\\n<|im_end|>\\n"
        f"<|im_start|>assistant\\n"
    )
    input_ids = [model.config.bos_token_id] + sp.encode_as_ids(prompt)
    input_ids = input_ids[:250 - max_new_tokens - 1]

    with torch.inference_mode():
        out = model.generate(
            input_ids=torch.tensor([input_ids], device=model.device),
            max_new_tokens=max_new_tokens,
            do_sample=(temperature > 0),
            temperature=max(temperature, 1e-5),
            eos_token_id=[model.config.eos_token_id, IM_END_ID],
            pad_token_id=model.config.pad_token_id,
            repetition_penalty=1.05,
        )
    return sp.decode(out[0][len(input_ids):])

print(chat("What is the basic unit of all living things?"))

Important: add IM_END_ID to eos_token_id — the model emits <|im_end|> as its natural stopping token during SFT. Without this, generation runs until max_new_tokens and appends the literal <|im_end|> string.

Gradio demo (HF Spaces)

import gradio as gr

with gr.Blocks() as demo:
    gr.Markdown("# AmhariXLM-0.5B")
    inp = gr.Textbox(label="Ask in Amharic or English")
    out = gr.Textbox(label="Answer", lines=6)
    btn = gr.Button("Ask")
    btn.click(lambda q: chat(q), inputs=inp, outputs=out)
demo.launch()

Limitations

Arithmetic

The tokenizer uses split_digits=False — whole numbers are single tokens. The model can answer arithmetic it has seen many times during training, but cannot reliably compute new multi-digit problems. Use a calculator if precision matters.

Amharic factual accuracy

Some Amharic-specific facts produce hallucinated answers. Verify Amharic responses externally before relying on them for high-stakes work.

Context length

Only 250 tokens. Longer inputs are truncated — chunk long inputs first.

Small-model limits

At 565M parameters this model is not competitive with frontier models on complex reasoning. It is optimized for simple educational Q&A in Amharic and English.

Not for high-stakes use

Do not use for medical, legal, financial, or safety-critical decisions without external verification.

Tokenizer details

• Type: SentencePiece, Unigram, 24,000 tokens
• Normalization: nmt_nfkc
• Byte fallback: enabled (handles unknown characters gracefully)
• Split digits: disabled — whole numbers are single tokens
• User-defined symbols: ChatML control tokens, Geez numerals (፩–፼), math emojis (➕ ➖ ✖️ ➗ 🟰), and frequent English reasoning words (Identify, Given, Calculate, Conclude, Formula, Understand)

Citation

@misc{amharixlm_05b_2026,
  title  = {AmhariXLM-0.5B: An Amharic/English Educational Language Model},
  author = {Fideliz AI Team and Yisberh},
  year   = {2025},
  url    = {https://huggingface.co/fidelizML/AmhariXLM-0.5B}
}

Contact

• Org: fidelizML
• Maintainer: Yisberh
• Issues: open an issue on the model repo's Community tab

License

Apache License 2.0 — free for commercial and non-commercial use. """