Nizam-LLM-7B

Ask a question about Saudi labor law. Get an answer from the law itself, with the article it comes from, or an honest "the texts don't say".

Nizam-LLM reads the articles a search step finds and answers in clear Arabic. It states the ruling first, keeps the conditions that decide the case, and cites every article it uses as «المادة (N) من <النظام>», so you can check it yourself.

This card describes v1, the first stable release and the one behind our live demo. Earlier releases stay available as tags (revision="v0.3", revision="v0.2", revision="v0.1"). Built by Muhammad Uzair on ALLaM-7B-Instruct-preview. The 846 articles it reads from are published as nizam-saudi-labor-corpus.

On our 286-case test	Base ALLaM-7B	Nizam-LLM v1
Answers fully correct	16%	79%
Cites the right article	25%	98%
Says so when the texts don't hold the answer	2 of 30	27 of 30
Refuses although the answer is in the texts	2 of 256	0 of 256

Research release, not legal advice. Always check the official text.

What's new in v1: it answers when the answer is there

v0.3 learned to say "the texts don't say" when a detail is set elsewhere. It sometimes said it too often: with the answering article right there in the prompt, it still declined. v1 answers those questions from the article, and declines even more reliably when the answer really is missing.

Question: هل يحق للعامل الحصول على تعويض مالي عن أيام الإجازة السنوية التي لم يستعملها عند انتهاء خدمة عمله؟ وما هي المادة التي تنص على ذلك؟

v0.3	v1
لا تتناول النصوص النظامية المرفقة هذه المسألة، فلا أستطيع الإجابة عنها استناداً إليها.	نعم، يستحق العامل تعويضاً عن أيام الإجازة السنوية التي لم يستعملها عند انتهاء خدمة عمله، لأن المادة (111) من نظام العمل تنص على أن للعامل الحق في الحصول على أجرة عن أيام الإجازة المستحقة إذا ترك العمل قبل استعماله لها … كما يستحق أجرة الإجازة عن أجزاء السنة بنسبة ما قضاه منها في العمل.
Declines, though Article 111 was in the prompt.	Answers from Article 111, including part years.

Why it worked: v1 adds about 570 reviewed examples aimed at the people who ask these questions, HR managers and lawyers: multi-part Labor Law scenarios, cases from the domestic-worker, civil-service and agricultural regulations, pairs that teach the difference between a missing article and a present one, and worked calculations checked by code. Wrong declines fell from 2 to 0 and correct declines rose from 24 to 27 of 30, while answering accuracy held.

How we got here

Every version was trained, then tested on the same 286 questions. All scores below use today's grading, so they compare directly.

Run	What changed	GPU and precision	Score	Says "not in texts"
Base	ALLaM-7B, no training	–	16%	2/30
3	First training run	2× T4, 4-bit QLoRA (fp16)	56%	9/30
4 · v0.1	Questions that need two articles	2× T4, 4-bit QLoRA (fp16)	70%	11/30
5 · v0.2	Recovered Labor Law articles, full multi-paragraph answers	2× T4, 4-bit QLoRA (fp16)	74%	11/30
5b	Same data as run 5, full precision	1× H100, bf16 LoRA	74%	15/30
6	Stronger search model (e5-large)	1× H100, bf16 LoRA	71%	13/30
7 · v0.3	Examples where the detail is set elsewhere	1× L4, 4-bit QLoRA (bf16)	77%	24/30
8 · v1	HR scenarios, answer/decline pairs, calculations	1× A100, 4-bit QLoRA (bf16)	79%	27/30

Before run 3 came two runs on a smaller model (Qwen3-1.7B) and a test of three untrained base models. ALLaM was chosen because it covered the most legal content and handles Arabic text most efficiently.

What we learned: better data moved the model; bigger hardware did not. Runs 5, 5b and 6 used three set-ups, from T4s to an H100, and scored within noise of each other. Each real jump (runs 4, 5, 7 and 8) came from new training examples aimed at one weakness the tests had shown.

We keep improving it

Nizam-LLM is trained and tested continuously. Every version is measured on the same test, with the same grading, against every version before it, and a new release ships only when it holds up.

How we test

• 143 questions across all 24 regulations, each asked twice: with just the right articles, and with them among related ones (286 cases).
• Graded by fixed rules, not by another AI. An answer passes only if it states every required point, cites the right article, names the regulation, and declines when it should. Partial answers fail.
• No leaks. Every training question is checked against the test questions before training. None overlap.
• Two settings. The fixed set gives every model the same articles; the live set lets the search model pick them, as in real use.

Evaluation results

Every run on the same 286 cases, graded the same way.

	Base	Run 3	v0.1	v0.2	Run 5b	Run 6	v0.3	v1
Overall pass	16.1%	56.3%	69.6%	74.1%	73.8%	70.6%	76.6%	79.0%
Answerable questions (256)	17.2%	59.4%	73.4%	78.5%	76.6%	73.8%	76.2%	77.7%
Correct declines (30)	2	9	11	11	15	13	24	27
Wrong declines on answerable	2	3	1	0	0	0	2	0
Right article cited	25.4%	91.0%	98.0%	97.3%	97.3%	96.5%	96.9%	98.4%
Regulation named	50.4%	98.8%	99.6%	100%	100%	100%	100%	100%
Required points covered	0.902	0.866	0.918	0.941	0.932	0.921	0.929	0.939

By question type	Cases	Base	Run 3	v0.1	v0.2	Run 5b	Run 6	v0.3	v1
Direct rule	88	18.2%	68.2%	79.5%	84.1%	87.5%	84.1%	78.4%	83.0%
Applied scenario	84	23.8%	60.7%	72.6%	73.8%	66.7%	63.1%	72.6%	71.4%
Boundary condition	54	5.6%	59.3%	63.0%	75.9%	75.9%	77.8%	79.6%	81.5%
Two-article	30	16.7%	30.0%	76.7%	80.0%	73.3%	66.7%	73.3%	73.3%
No answer in the texts	30	6.7%	30.0%	36.7%	36.7%	50.0%	43.3%	80.0%	90.0%

With live search (143 questions, the top 5 articles found by e5-large):

	Run 6	v0.3	v1
Overall pass	78.3%	79.7%	78.3%
Answerable questions (128)	81.3%	79.7%	78.1%
Correct declines (15)	8	12	12
Wrong declines on answerable	0	1	0
Right article cited	97.7%	98.4%	98.4%

Against v0.3, v1 passes 3 more questions without an answer and loses none; on answerable questions the two differ within noise (20 up, 16 down). One weakness: on Labor Law questions v1 passes 26 of 44 against v0.3's 31. It answers the case that was asked and can leave out a neighbouring condition of the same article, which our grading requires.

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "muhammaduxair/Nizam-LLM-7B"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
# On a T4 (no bf16) use dtype=torch.float16. To pin this release: revision="v1".

system = ("أنت مساعد قانوني متخصص في أنظمة العمل في المملكة العربية السعودية. "
          "بين يديك نصوص نظامية مرفقة. "
          "أجب بإيجاز وبالعربية الفصحى معتمداً على هذه النصوص وحدها. "
          "اذكر رقم المادة التي تستند إليها بصيغة «المادة (رقم)» مع اسم النظام. "
          "إذا كان الحكم قد أُلغي أو حُذف، فقل ذلك صراحة. "
          "إذا لم تتضمن النصوص المرفقة إجابة السؤال، فقل ذلك ولا تُجب من معرفتك "
          "الخاصة ولا تُخمّن رقم مادة.")
articles = ("المادة (53) — نظام العمل، المادة (53)\n"
            "إذا كان العامل خاضعاً للتجربة، وجب النص على ذلك صراحة في عقد العمل، "
            "وتحديد مدتها بوضوح، على ألا يزيد مجموع المدة في جميع الأحوال على مائة "
            "وثمانين يوماً. ...")
question = "وفق نظام العمل، ما الحد الأقصى لمدة التجربة؟"

messages = [{"role": "system", "content": system},
            {"role": "user", "content": f"النصوص النظامية:\n\n{articles}\n\nالسؤال: {question}"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
                                       return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use the system prompt exactly as above: it is the one the model was trained on. Find the articles with multilingual-e5-large (revision 3d7cfbdacd47fdda877c5cd8a79fbcc4f2a574f3), top 5; the corpus card has the full search code. Keep prompts within 3,072 tokens and decode greedily. The model needs about 16 GB of GPU memory.

Same behaviour as our notebook: the guardrail

The model is trained only to answer from the attached articles, or to say they don't hold the answer. Everything else runs in a small guardrail around it, published here as guardrail.py (standard-library Python, used unchanged by our notebook and demo):

• obscene requests, and questions outside Saudi labor and workforce regulations, are refused before the model is called;
• an answer that wrongly calls an attached article missing is asked again once;
• a bare "the texts don't say" becomes a clear "no complete information" message;
• citations of articles that were not in the prompt get a note;
• an end-of-service question that gives the service length, the last wage and how the work ended gets the award computed by code from Articles 84 and 85, with a warning if the model's figure differs.

Continuing the example above:

import importlib.util
from huggingface_hub import hf_hub_download
from sentence_transformers import SentenceTransformer

spec = importlib.util.spec_from_file_location("guardrail", hf_hub_download(repo, "guardrail.py"))
guardrail = importlib.util.module_from_spec(spec)
spec.loader.exec_module(guardrail)
e5 = SentenceTransformer("intfloat/multilingual-e5-large",
                         revision="3d7cfbdacd47fdda877c5cd8a79fbcc4f2a574f3")

def generate(turn):
    chat = [{"role": "system", "content": system}, {"role": "user", "content": turn}]
    inputs = tokenizer.apply_chat_template(chat, add_generation_prompt=True,
                                           return_tensors="pt", return_dict=True).to(model.device)
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
    return tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()

embedding = e5.encode([f"query: {question}"], normalize_embeddings=True)[0]
turn = f"النصوص النظامية:\n\n{articles}\n\nالسؤال: {question}"
outcome = guardrail.guard(question, embedding, turn, ["المادة (53)"], generate)
print(outcome.text, *outcome.notes, sep="\n")

The last list holds the headers of the articles in the prompt; with search, pass the headers of the five it found, and reuse the same e5 embedding for both. The scores on this card measure the model without the guardrail.

Good to know

• It only knows what you give it. With the wrong articles in the prompt it cannot answer correctly, and it is not built to answer from memory.
• About 1 answer in 5 misses a condition or point. It may answer only the case asked and leave out a neighbouring condition of the same article. If an answer cites an article that was not in the prompt, treat it as unsupported.
• Check every figure. It shows its working for amounts, but the rule it applies can be wrong. The guardrail computes the end-of-service award; compute other amounts (overtime, compensation, leave pay) yourself.
• The texts are archived copies. The model does not know later amendments, or whether a provision is still in force.
• Scope: Saudi labor and related workforce regulations, Arabic only. Off-topic questions are refused by the guardrail, not by the model.
• Intended for research and prototypes in which a qualified person checks each answer. Not for legal advice to the public, or for automated decisions about anyone's rights.

Technical details

• Data: about 5,400 verified, open-book training examples written from 845 of the corpus articles, including examples whose answer is deliberately not in the texts. Every example was checked against its source and reviewed before training, and none overlaps the test.
• Training (v1): QLoRA on a 4-bit NF4 base with bf16 compute, prompts up to 3,072 tokens.
• Release: the adapter was merged into the unquantized base in bf16 and checked token by token against the evaluated model before upload.
• Evaluation: rule-based grading, greedy decoding, up to 1,024 new tokens. Earlier runs were re-graded from their saved answers with the current rules. The test is specific to this project, so its scores do not compare with other benchmarks.

Citation

@misc{uzair2026nizamllm,
  title  = {Nizam-LLM-7B: Grounded Question Answering over Saudi Labor Regulations},
  author = {Muhammad Uzair},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/muhammaduxair/Nizam-LLM-7B}},
  note   = {Version 1}
}

License and attribution

Apache-2.0, the license of the base model; see LICENSE. Built on ALLaM-7B-Instruct-preview by the National Center for Artificial Intelligence (SDAIA), now published by HUMAIN. The source texts are official publications of the Saudi Ministry of Human Resources and Social Development.