Emhotob-5M-Darija-MSA-v1 — Bidirectional Darija ↔ MSA (~5M params)

A 5.08M-parameter model that translates both ways between Moroccan Darija (الدارجة المغربية) and Modern Standard Arabic (الفصحى). A single set of weights serves both directions; a direction-specific Arabic system prompt selects which way to translate.

Finetuned from oddadmix/Emhotob-5M, a tiny Llama-architecture base (hidden size 128, 5 layers, 4 heads, tied embeddings).

Scaling study. This runs the exact recipe of oddadmix/50M-Darija-MSA-v1 on a base ~10× smaller. At 5M the model switches script/register correctly and is the strongest of the 5M translation set (MSA→Darija BLEU ~13), though it still drifts on longer inputs. A scaling demonstration, not a production translator; use the 50M sibling for fluent output.

Evaluation

Evaluated on a deterministic held-out set of ~3,000 pairs (seed=42; 2,961 scored after dropping rows with an empty MSA reference), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

Direction	sacreBLEU	chrF
Darija → MSA	5.81	20.28
MSA → Darija	13.16	21.59

MSA→Darija scores highest of any direction in the 5M suite. The saved weights are the best checkpoint by validation loss (eval_loss = 3.512, epoch 3 of 3). For reference, the 50M sibling scores BLEU ~32 (dar→msa) / ~40 (msa→dar) with the same data and eval.

Example translations

Real greedy-decoded outputs from the held-out set (rough — this is a 5M model):

Darija → MSA

Darija input	Model output (MSA)	Reference
لا، عندنا تذاكر يا حبيبة	لا، أنا لا أحب ذلك.	لا، لدينا تذاكر يا حبيبتي.

MSA → Darija

MSA input	Model output (Darija)	Reference
لا، لدينا تذاكر يا حبيبتي.	لا، غادي نديرو	لا، عندنا تذاكر يا حبيبة
أنا أنتظر منك أن تستحق هذا.	كانضن غادي ندير	كانتسنّا فيك ت دي هادشي و تستاهلو

The model produces authentic Darija markers (غادي, كانضن, هادشي) in the MSA→Darija direction and MSA morphology the other way, but frequently loses source content. 20 samples per direction with references are in eval_bidirectional.json.

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-5M-Darija-MSA-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_MSA = "أنت مترجم محترف. ترجم النص من الدارجة المغربية إلى اللغة العربية الفصحى."
SYS_TO_DAR = "أنت مترجم محترف. ترجم النص من اللغة العربية الفصحى إلى الدارجة المغربية."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("لا، لدينا تذاكر يا حبيبتي.", SYS_TO_DAR))

Training

• Base model: oddadmix/Emhotob-5M (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings; 5,080,704 params after resizing for 2 ChatML tokens)
• Dataset: oddadmix/darija_english_msa_parallel_dataset (90,104 rows; this model uses the darija and msa columns; rows with an empty msa are skipped). Sources: DoDA, HANTIFARAH combined, and ArabML/Skiredj parallel data.
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Split: 87,104 train / 3,000 deterministic held-out (seed=42), scored both directions.

Limitations

• A 5M model near the emergence threshold: correct register/script but frequent meaning loss, drift, and repetition on longer inputs.
• Darija has no standard orthography and mixes Arabic and Latin (Arabizi) script plus French/Amazigh loanwords — outputs vary in spelling and may hallucinate on out-of-domain input.
• For fluent translation use oddadmix/50M-Darija-MSA-v1.

License

Apache-2.0 (model weights, inherited from the base model). The training dataset aggregates several community corpora — check their individual licenses for downstream use.