Nawah-Dialect-BERT-6M — مصنّف اللهجات العربية بمُرمِّز BERT

A 5.98M-parameter Arabic dialect identifier. Give it a piece of Arabic text; it returns one of 14 labels — 13 dialects or Modern Standard Arabic.

Same task, same data, same evaluation as the Llama-based Nawah-Dialect-500K / Nawah-Dialect-v1 — this run swaps the backbone for Nawah-BERT-6M-v2, a bidirectional BERT encoder, to see how a small encoder does on the same task against the existing causal-decoder ladder.

import torch
from transformers import AutoTokenizer
from dialect_model import DialectClassifier, predict     # ships in this repo

M = "oddadmix/Nawah-Dialect-BERT-6M"
tok = AutoTokenizer.from_pretrained(M)
model = DialectClassifier.from_pretrained(M)

predict(model, tok, "انا رايح السوق دلوقتي واجيب حاجات من هناك")[0]
# {'label': 'eg', 'name': 'Egyptian', 'score': 0.918}

Strip diacritics before inference — trained on undiacritized text, same as the rest of this family. Mean-pool, not [CLS] — Nawah-BERT-6M-v2 was pretrained on packed 2,048-token chunks with no meaningful CLS position, so this repo does not use AutoModelForSequenceClassification's default pooler; dialect_model.py's DialectClassifier wraps the bare backbone with mean pooling over non-pad positions and a small 2-layer scorer instead — the same trap documented on the backbone's own model card.

Labels

14 classes — 13 Arabic dialects plus Modern Standard Arabic:

ma Moroccan · eg Egyptian · dz Algerian · sa Saudi · msa MSA · sd Sudanese · bh Bahraini · tn Tunisian · lb Lebanese · ye Yemeni · sy Syrian · ps Palestinian · iq Iraqi · ly Libyan

Results

Test set: 3,079 held-out rows, identical split to the Llama-based models. Majority-class baseline 0.140.

model	backbone	params	accuracy	macro-F1
Nawah-Dialect-500K	Llama, 2 layers	0.52M	0.9315	0.8786
this model	BERT, 8 layers	5.98M	0.9428	0.9025
Nawah-Dialect-v1	Llama, 51.8M	51.79M	0.9441	0.9078

The 6M BERT lands within 0.13 points of accuracy and 0.53 of macro-F1 of the 51.8M Llama model — essentially matching it — at 1/8.6 the size, and clearly beats the 500K model on both metrics.

Also roughly half the size of the two dialect-router releases

Outside this Llama-based ladder, two earlier dialect-ID models were released as part of the Lahgtna TTS routing pipeline: dialect-router-v0.1 and dialect-router-v0.2, both fine-tunes of asafaya/bert-mini-arabic at 11.55M parameters — essentially double this model's size:

model	params	labels	accuracy	macro-F1
dialect-router-v0.2	11.55M	15 (13 dialects + MSA + English)	0.9359	0.9052
this model	5.98M	14 (13 dialects + MSA)	0.9428	0.9025

Not a strictly apples-to-apples comparison — different training corpus, different test set, and dialect-router-v0.2 also carries an English label this model doesn't — but on their own reported numbers, this model matches or slightly beats dialect-router-v0.2 at about half the parameters. (dialect-router-v0.1 reports no accuracy/F1 in its card, so it isn't in the table.)

Per-class accuracy (worst first)

dialect	acc	n		dialect	acc	n
ps Palestinian	0.675	80		lb Lebanese	0.945	163
sy Syrian	0.745	94		tn Tunisian	0.951	185
ly Libyan	0.771	35		sa Saudi	0.973	261
ye Yemeni	0.824	119		eg Egyptian	0.980	395
bh Bahraini	0.894	218		dz Algerian	0.989	363
sd Sudanese	0.899	238		ma Moroccan	0.990	412
iq Iraqi	0.918	85		msa	0.995	431

The same short-text-starved classes that were weak on the 51.8M model (ps, sy, ly, ye) are weak here too — this is a data-scarcity property of the task, not something specific to this backbone.

Confusions are linguistic, not artefactual

Top confusions: ye→bh (8), ps→sy (7), sd→ye (7), sy→sd/ps/lb (6/6/5), bh→ye (6), iq→bh (5), sd→bh/ps (6/5) — all within one dialect family (Levantine: ps/sy/lb; Gulf and African: bh/ye/sd/iq). Nothing confuses across family boundaries.

The register confound, checked

The MSA class mixes transcript rows with chunks of written articles (see prepare_dialect.py). If the model were reading written-vs-spoken register instead of MSA, the transcript slice would collapse relative to the article slice:

MSA source	accuracy	n
written articles	0.998	400
speech transcripts	0.968	31

0.968 on transcripts — matching the 51.8M model's reported 0.968 on the same check almost exactly. The model is reading MSA, not register.

Limitations

Same as the rest of this family: inputs shorter than ~25 words are unreliable across all classes; ps/sy/ly/ye are under-resourced in the source corpus; source text is YouTube speech transcripts, so performance on formal writing, social media orthography, or Arabizi is untested.

Training

Base Nawah-BERT-6M-v2 — BERT, hidden 128, 8 layers, 2 heads. Data prep is prepare_dialect.py, unchanged from the Llama-based models (it is backbone-agnostic — text normalisation, dedup, and the train/test split don't depend on what reads the text afterward): 573,829 train / 3,079 test rows from oddadmix/dialectal-arabic-text-no-audio-v2 (YouTube transcripts) plus MSA chunks from oddadmix/msa_texts_summaries_new.

train_dialect_cls_bilingual.py's MeanPoolClassifier: bare AutoModel backbone → mean pool over non-pad positions → Linear(h,h) → GELU → Linear(h, 14), cross-entropy loss. LR 3e-4 cosine, batch 64, 3 epochs, bf16, max_length 192, checkpoint selected on macro-F1. 26,901 steps, 6m16s on a single consumer GPU.

prepare_dialect.py and train_dialect_cls_bilingual.py are in this repo.

© KAND CA 2026 — PROJECT NAWAH