Nawah-RuleCheck-500K — مدقّق قواعد عربي بـ 2.1 ميجابايت

A 518,256-parameter Arabic rule checker; weights file 2,075,200 bytes (2.08 MB). Give it a text and a rule written in ordinary Arabic and it answers مطابق (the text satisfies the rule) or مخالف (it does not).

بالعربية: نموذج عربي صغير (518,256 معامل) يأخذ نصًا وقاعدة مكتوبة بلغة طبيعية ويقرّر إن كان النص مطابقًا للقاعدة أم مخالفًا لها.

2 transformer layers, hidden size 16. It is a sequence classifier (LlamaForSequenceClassification), not a generator — the answer is two-valued, so a classification head fits better at this size and is directly scorable.

Where the parameters are. 6,256 of the 518,256 parameters are the transformer body; 512,000 (98.8%) are the 32,000 × 16 embedding table. Small Arabic models are mostly vocabulary — worth knowing before reading the results as reasoning capacity.

The size ladder

This model is one rung of a four-point ladder trained on identical data and scored by one harness in one session (eval_all.py). Nothing below is carried over from an earlier run or another card.

model	params	weights	seen wording	unseen wording	minimal pairs	hand-written wording	CPU ms
this model	518,256	2.08 MB	0.9866	0.9778	0.9926	0.5558	0.33
Nawah-RuleCheck-1M	1,073,504	4.30 MB	0.9952	0.9864	0.9991	0.7708	0.55
Nawah-RuleCheck-5M	5,080,704	20.33 MB	0.9979	0.9949	0.9926	0.8642	1.06
Nawah-RuleCheck-v2	51,787,264	207.16 MB	0.9980	0.9980	0.9991	0.8883	16.58
majority baseline	—	—	0.6401	0.6401	0.5000	0.5925	—

CPU ms = single example, batch 1, float32, 2 threads, median of 200 runs, same input for every model. On an RTX 5090 this model is 0.96 ms.

What the four columns mean

• seen wording (6,624 pairs) — held-out texts (task_id split, zero text overlap with training), rules phrased the way training phrased them. Text generalisation.
• unseen wording (6,624 pairs) — the same held-out texts, rule phrasings held out of training entirely, drawn from the same 631-paraphrase pool.
• minimal pairs (1,080 pairs) — a real text and a surgically edited copy whose verdict flips, verified by rules_common.py. Any cue that merely correlates with the label inside the corpus dies here.
• hand-written wording (1,200 pairs) — 10 rules restated in terse, colloquial Arabic written from scratch, sharing almost no vocabulary with the paraphrase pool. This is the honest hard column, and it is where model size actually buys something.

Per-rule accuracy, unseen wording

rule	acc
no_email	1.000
ends_question	0.998
has_price	0.998
no_url	0.996
no_latin	0.991
no_excess_punct	0.986
has_date	0.980
has_number	0.980
min_words_25	0.975
has_city	0.966
has_phone	0.962
min_words_30	0.949
no_phone	0.938
min_words_15	0.929
min_words_20	0.886
max_words_50	0.884
max_words_30	0.804
max_words_40	0.739
max_words_25	0.700

Per-rule accuracy, hand-written wording

rule	acc
has_price	1.000
has_date	0.908
has_phone	0.767
has_number	0.733
no_email	0.542
ends_question	0.492
no_latin	0.400
no_url	0.392
no_excess_punct	0.200
no_phone	0.125

Limitations

Read the hand-written wording column, not the headline. Inside the paraphrase distribution the whole ladder is bunched between 0.9778 and 0.9980 — a 2.0-point spread across a 100x size range. On rule wordings written from scratch the same four models spread from 0.5558 to 0.8883, a 33.2-point gap, and the 500K rung falls below the 0.5925 majority-class baseline. Almost all of what size buys on this task is robustness to wording you did not train on. If your rules are a fixed catalogue you can phrase in the training register, the small rungs are close to free. If users will phrase rules in wording you do not control, size matters and no rung on this ladder is finished.

Trained on 19 rules decidable from the string itself. Rules needing world knowledge, judgement or multi-step inference are out of distribution. Texts are 1–3 line Arabic business documents (classified ads, support tickets, job posts, complaints, rental listings) across nine regions; longer or very different text is untested. Word-count rules are the weakest family — counting is the one operation here that cannot be pattern-matched.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

M = "oddadmix/Nawah-RuleCheck-500K"
tok = AutoTokenizer.from_pretrained(M)
model = AutoModelForSequenceClassification.from_pretrained(M).eval()

def check(text, rule):
    x = tok(f"النص: {text}\nالقاعدة: {rule}", return_tensors="pt", add_special_tokens=False)
    with torch.no_grad():
        p = model(**x).logits.softmax(-1)[0]
    return model.config.id2label[int(p.argmax())], float(p.max())

check("للبيع سيارة نظيفة، للتواصل على الرقم 0551234567",
      "يُمنع ظهور أي رقم هاتف (8-15 خانة) في النص")

Input format is النص: {text}\nالقاعدة: {rule} with add_special_tokens=False. Text first, rule second: the head pools the last non-pad token, so under causal attention only the trailing rule tokens can attend to the whole text.

Training

Base oddadmix/Emhotob-500K-v2 — Llama, 2 layers, hidden 16, vocab 32,000, tied embeddings, 2048 context. 159,240 (text, rule) pairs, LR 6e-3, 15 epochs, cosine schedule, 500 warmup steps, effective batch 64, bf16, max_length 224. Best checkpoint selected on unseen-wording macro-F1.

Learning rate across the ladder

"Smaller models need a higher learning rate" is true here — but only below 5M. Each rung was swept independently on the same data and the same eval:

rung	params	chosen LR	epochs	tuned?
Nawah-RuleCheck-500K	518,256	6e-3	15	✅
Nawah-RuleCheck-1M	1,073,504	1e-3	10	✅
Nawah-RuleCheck-5M	5,080,704	1e-4	10	✅
Nawah-RuleCheck-v2	51,787,264	1e-4	10	✅

The optimum climbs 60× between the 5M and the 500K (1e-4 → 6e-3), and getting it wrong is expensive: on the 500K base the big model's recipe scores 0.9179 unseen against 0.9780 tuned — six points that look like a capacity limit and are not.

Above 5M the effect simply stops. The 5M and the 51.8M want the same 1e-4, and pushing the 5M up toward the 500K's learning rate destroys it (3e-3 → 0.88, 6e-3 → 0.77 unseen). So neither recipe transfers in either direction: the tiny rungs need their own LR, and that LR must not be carried back up the ladder.

This rung's sweep

Same data, same eval splits, same everything — only LR and epoch budget move:

LR	epochs	seen	unseen
3e-4	3	0.9583	0.9179
1e-3	10	0.9787	0.9460
3e-3	10	0.9863	0.9695
6e-3	10	0.9866	0.9675
1e-2	10	0.9792	0.9635
6e-3	15	0.9867	0.9780
1e-3	20	0.9841	0.9653
3e-3	20	0.9902	0.9648

Sweep run in the earlier session that released this model.

How the labels were made

Labels are computed, not model-judged. Each rule is a deterministic predicate over the raw string (rules_common.py, shipped here), so ground truth is exact by construction. That also makes paraphrasing free supervision: rewording a rule cannot change its verdict.

Siblings

• oddadmix/Nawah-RuleCheck-500K — 518,256 params
• oddadmix/Nawah-RuleCheck-1M — 1,073,504 params
• oddadmix/Nawah-RuleCheck-5M — 5,080,704 params
• oddadmix/Nawah-RuleCheck-v2 — 51,787,264 params

Reproducing every number on this page

python eval_all.py oddadmix/Nawah-RuleCheck-500K oddadmix/Nawah-RuleCheck-1M \
                   oddadmix/Nawah-RuleCheck-5M  oddadmix/Nawah-RuleCheck-v2

eval_all.py is in this repo, along with the frozen case sets it scores against (cf_cases.json, ood_cases.json) so the minimal pairs and the hand-written wordings are byte-identical for every rung and every re-run. Also here: prepare_rules_cls.py, train_rules_cls.py, gen_rule_paraphrases.py, counterfactual_eval.py, rules_common.py (which defines the labels), and the 631 rule paraphrases in rule_paraphrases.json.