Nawah-50M-RAG-Chat-8K

A 51.8M-parameter Modern Standard Arabic multi-turn customer-support RAG assistant with an 8192-token context. Given retrieved knowledge-base passages, an optional customer account record, and a conversation, it answers in MSA strictly from what it was given - or politely says the information is unavailable and offers escalation.

Fine-tuned from oddadmix/50M-8192-Nawah-gemma (Gemma-3 architecture, trained from scratch on Arabic).

⚠️ Research-scale model. It copies from the context you give it; it has no reliable knowledge of its own and no safety alignment.

What is new against the single-turn 8K model

The previous model's measured failure was not fabrication and not reach - 98% of the numbers it emitted were verbatim from the knowledge base - but discrimination: only about half came from the correct passage, and correct passage selection fell from 96% among 6 candidate passages to 28% among ~50. Nothing in its training data ever required two passages at once, asked more than one thing per message, or continued past a single turn.

This model is trained on 30,027 conversations / 79,880 turns (118.1M tokens) built for exactly those three gaps:

• Multi-context aggregation - answers that must combine passages that are neither adjacent nor from the same section.
• Multi-part questions - one message with 2-3 explicit asks, each carrying its own evidence, so partial answers are penalised in training.
• Account-aware grounding - a per-customer record (account number, plan, outstanding balance, due date, last payment, usage) supplied outside the numbered passages, which the model must prefer over generic policy text.
• 1-5 rounds, including mid-conversation refusals: a model that has answered twice must still be able to say it does not know.

Format

ChatML, <|im_start|>/<|im_end|> (vocab 32,002, eos <|im_end|>). Passages go in the system turn numbered [1], [2], ...; the account record, when there is one, goes in the same system turn before the passages and outside their numbering. A chat_template ships with the tokenizer.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("oddadmix/Nawah-50M-RAG-Chat-8K")
model = AutoModelForCausalLM.from_pretrained("oddadmix/Nawah-50M-RAG-Chat-8K", dtype=torch.bfloat16)

chunks = ["تبلغ رسوم الاشتراك الشهري في الباقة الذهبية 150 ريالاً.",
          "يمكن إلغاء الاشتراك خلال 14 يوماً من التفعيل مع استرداد كامل المبلغ."]
account = ("بيانات حساب العميل:\n"
           "الباقة الحالية: الباقة الذهبية\n"
           "الرصيد المستحق: 412 ريالاً\n"
           "تاريخ الاستحقاق: 2026-09-14")
preamble = ("أنت مساعد خدمة عملاء. أجب عن سؤال العميل بالفصحى اعتماداً فقط على "
            "المعلومات التالية. إذا لم تكن الإجابة موجودة في المعلومات، فقل ذلك "
            "بأدب واعرض تحويل العميل إلى أحد موظفي خدمة العملاء.")
passages = "\n".join(f"[{i+1}] {c}" for i, c in enumerate(chunks))
msgs = [{"role": "system", "content": f"{preamble}\n\n{account}\n\n{passages}"},
        {"role": "user", "content": "كم رصيدي المستحق ومتى موعد سداده؟"}]
ids = tok.apply_chat_template(msgs, return_tensors="pt",
                              add_generation_prompt=True)
out = model.generate(ids, max_new_tokens=256, temperature=0.1, do_sample=True,
                     top_p=0.85, repetition_penalty=1.1)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Training

Property	Value
Base	oddadmix/50M-8192-Nawah-gemma, ctx 8192
Data	30,027 conversations, 79,880 turns, company-disjoint eval
Loss	Cross-entropy on every assistant turn
Batching	token-budgeted (16,384 tokens/batch, length-grouped)
Schedule	2 epochs on the full blend, lr 3e-4 cosine
Precision	fp32 weights, bf16 autocast
Hardware	a single consumer GPU
eval_loss	0.978 (epoch 1) -> 0.852 (epoch 2)

A planned second stage was trained and then dropped. The corpus is a third short / a third mid / a third long by row, which is 9% / 32% / 59% by token - and the gradient follows tokens, so an anneal epoch on a short-weighted view (short and mid repeated 3x, 20% of long) was meant to correct the mix. It made things worse on every bucket at once: held-out loss rose from 0.903 to 0.956 on short, 0.889 to 0.991 on mid, and 0.836 to 0.929 on long, while training loss fell from 1.00 to 0.57 - overfitting on rows already seen twice, not a mis-weighting. On the task metrics below the two checkpoints were statistically indistinguishable (every paired sign test p > 0.12), so the stage-1 checkpoint is what ships.

Evaluation

632 held-out conversations (37 companies, disjoint from training) expanded to 1,651 gold-forced turns - each round is scored with the previous rounds replayed from the reference answers, so per-turn quality is measured without error cascade.

metric	score	reference-answer ceiling
refusal accuracy	0.927	-
refusal accuracy, mid-conversation	0.959	-
number-grounding	+0.671	-
aggregation-recall	+0.059	+0.976
part-coverage	+0.002	+0.921
distractor-number rate	0.224	-

Read these honestly. Refusal is the strong result, and it holds mid-conversation (n=148) - the model does not become agreeable after answering twice, which is what the multi-turn refusal rows were for. Quality does not decay by round (round 0 +0.682 vs round 3 +0.713 on number-grounding).

Coverage is the weak result: at +0.00, part-coverage means roughly half the key_facts of a turn reach the answer, against reference answers that reach +0.92 on the same test. The model is near the floor where its training data is near the ceiling, so the data is not the binding constraint. Both coverage metrics are verbatim substring tests after normalisation, which score a correct paraphrase as a miss - treat them as floors. Grounding is also still bucket-dependent (+0.739 short, +0.630 long), consistent with the sliding-window note below.

Limitations

Answers are only as good as the passages provided; retrieval is not included. Trained on fictional synthetic companies, so the names and prices in its training data are not real. It is instructed never to calculate - it quotes figures rather than combining them - so do not expect arithmetic. Ten of its twelve layers are sliding-window (1024) and only two are full-attention, so evidence very far back in an 8192-token context reaches the answer through a narrower path than evidence nearby; expect grounding to degrade with distance more than a fully-global model would. MSA only.

---

© KAND CA 2026 - PROJECT NAWAH.