aisa-v20-gemma3-1b-lora

A LoRA adapter for google/gemma-3-1b-it that emits structured function calls from Arabic instructions across five dialects (MSA, Gulf, Levantine, Egyptian, Maghrebi).

Trained for the AISA-ArabicFC shared task — Tuwaiq Academy, ArabicNLP 2026.

ابي اشوف مخالفات المرور برقم الهوية 4433221100
                    ↓
{ "tool_called": "check_traffic_violations",
  "arguments": { "id_number": "4433221100" } }

Results

Blind test set

Metric	Score
Overall	0.8718
Track A	0.8614
FnAcc	0.9964
ArgEM	0.7713
ThinkRate	0.9360

Dev split (545 rows)

Metric	Score
Track A	0.8584
Track B	0.8728
FnAcc	1.0000
ArgEM	0.7640

For reference, the task baseline (AISA-Think 270M + LoRA) scores 0.739 overall with ArgEM 0.541 — this adapter is a 43% relative gain on ArgEM.

ArgEM is the metric that matters

In all 11,000 dataset examples, the gold tool is the first declaration in the prompt:

Counter(gold_position for row in dataset)
# [(0, 11000)]    10,500/10,500 train · 500/500 dev

So a policy of "always pick the first declared tool" achieves near-perfect FnAcc without reading the question at all. FnAcc therefore doesn't measure tool selection on this benchmark. ArgEM — exact match on every argument key and value — carries the real signal, and it's where every round of work went.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-1b-it",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "techwithrayoma/aisa-v20-gemma3-1b-lora")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-it")

prompt = build_prompt(                       # see note on prompt assembly below
    tools=["search_hotels", "search_umrah_packages", "get_weather", "order_food"],
    text="دور لي فنادق في مكة لثلاث ضيوف",
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Gemma 3 is a gated model — accept the licence at google/gemma-3-1b-it and authenticate with an HF token before loading.

Prompt assembly

The prompt format comes from the shared task: a thinking instruction, an injected timestamp, and the function declarations for the candidate tools. scripts/extract_prompt.py in the repo builds it from the dataset.

Two things matter at inference time:

• Declare about 4 candidate tools per request, not all 20. Training prompts never showed more than a handful, so a full catalogue pushes the prompt outside the training distribution and the model starts mixing argument names between tools.
• Fill the timestamp at request time. The model resolves relative dates like بكرا against it, so a frozen date makes every date argument wrong.

Parsing the output

The model emits several call formats — <tool_call>name{...}, <start_function_call>call:name{...}, and bare TOOL_CALL name{...}. A parser that handles only one of these fails in the worst possible way: it recovers the tool name from the surrounding text and returns empty arguments, so the output looks structurally valid and scores zero.

Argument types are decided by field name, never inferred from the value. ArgEM compares exactly, so id_number must stay "0123456789" — cast it to an int and it's a different ID. Conversely amount and guests must be numbers, so "2" fails where 2 passes.

Training

Base	google/gemma-3-1b-it
Method	QLoRA, 4-bit NF4
LoRA	r=128 · α=256 · dropout 0.05
Targets	q_proj, k_proj, v_proj, o_proj
Schedule	3 epochs · effective batch 16 · lr 5e-5 cosine
Seq len	1024
Hardware	RunPod RTX 4090

Why r=128? The failing capability was never tool selection. It was copying an ID, a city, or a medical specialty out of dialectal Arabic without rewording it — closer to a copying behaviour than a classification one, and it needed the capacity.

Tools

20 tools across eight domains, ~500–650 training examples each:

book_doctor_appointment · calculate_customs · calculate_end_of_service · calculate_zakat · check_insurance_coverage · check_iqama_status · check_traffic_violations · check_visa_status · compare_prices · convert_currency · get_air_quality · get_qibla_direction · get_weather · order_food · search_hotels · search_medications · search_quran · search_umrah_packages · transfer_money · translate_text

Per-tool ArgEM splits cleanly by argument type — verbatim-copy arguments are near-solved, normalized or inferred ones are not:

Copied verbatim		Normalized or inferred
get_qibla_direction	1.000	transfer_money	0.250
check_traffic_violations	1.000	book_doctor_appointment	0.333
check_iqama_status	1.000	calculate_customs	0.417
translate_text	0.962	search_hotels	0.500

Limitations

• Evaluated only on this shared task's distribution.
• FnAcc is not meaningful on this benchmark — the correct tool is always declared first.
• ArgEM is exact match, so a large share of remaining errors are annotation-convention mismatches (طب الأطفال vs أطفال, الأسبوع القادم vs next week) rather than comprehension failures.
• Sensitive to how many tools are declared, and to their order.
• Maghrebi coverage in dev is 6 samples — effectively untested.
• No full-precision training comparison was run.

License

Adapter: MIT. Base model: Gemma Terms of Use. Dataset terms are set by the shared task organizers; the dataset is not redistributed here.