โ๏ธ AmLegal_Gemma-Inst: Specialized Amharic Legal LLM
AmLegal_Gemma-Inst is an open-weights, domain-adapted language model fine-tuned for Amharic legal question answering and Retrieval-Augmented Generation (RAG) over decisions of the Federal Supreme Court of Ethiopia Cassation Division (แจแขแตแฎแตแซ แแดแซแ แ แ แแญ แแญแต แคแต แฐแ แญ แฐแ แฝแแต).
The model is based on google/gemma-3-4b-it and adapted using Rank-Stabilized LoRA (rsLoRA, $r=64$, $\alpha=64$, $\alpha/\sqrt{r}$ scaling) with completion-only loss masking and negative distractor conditioning.
๐ Empirical Evaluation (Held-Out Cassation Decisions)
Evaluated against the unadapted base model and alternative 7B models on strictly unseen Ethiopian Supreme Court decisions:
| Evaluation Metric | Base Gemma-3-4B | Qwen-2.5-7B FT | AmLegal_Gemma-Inst (Ours) | Relative Delta |
|---|---|---|---|---|
| ROUGE-1 ($F_1$) | 29.68% | 38.40% | 49.66% | +67.3% |
| ROUGE-2 ($F_1$) | 19.58% | 27.10% | 34.94% | +78.4% |
| ROUGE-L ($F_1$) | 29.68% | 36.59% | 49.66% | +67.3% |
| SacreBLEU | 8.80 | 28.29 | 23.47 | +166.7% |
| chrF | 19.27 | โ | 38.95 | +102.1% |
| BERTScore $F_1$ | 58.40% | โ | 91.30% | +56.3% |
| Evidence Grounding | 84.00% | 78.00% | 92.00% | +9.5% |
| Citation Recall | 68.94% | 15.31% | 88.07% | +27.7% |
| Citation Precision | 88.46% | 23.21% | 92.02% | +4.0% |
| Citation $F_1$ | 77.49% | 18.45% | 90.00% | +16.1% |
| Foreign Contamination | 7.49% | โ | 0.03% | -99.6% (Eliminated) |
| Distractor Misattribution | โ | โ | 0.00% | Zero confusion |
| Answer Completeness (1โ3) | 2.14 | โ | 2.88 | +34.6% |
๐งโโ๏ธ LLM-as-a-Judge Qualitative Scores (Scale 1โ10)
- Faithfulness:
9.43 / 10 - Relevance:
9.20 / 10 - Consistency:
9.17 / 10 - Coherence:
9.00 / 10 - Clarity:
8.90 / 10 - Accuracy:
8.53 / 10 - Overall Mean:
8.77 / 10
๐ป Quick Start & Inference
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "getachewgetu/AmLegal_Gemma-Inst"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
prompt = """<start_of_turn>user
แ แแฐ แจแขแตแฎแตแซ แแญแต แคแต แแณแฎแฝแ แจแแตแแแจแต แจแแฐแ แจแ
แ แจแณแต แแ
แข แจแแ
แ แณแฝ แ แฐแฐแ แ แจแฐแ แญ แฝแแต แจแแณแ แแแฐ-แฝแแ แแญ แฅแป แฐแแญแฉแแ
แแแจแ แ แฅแซแ แตแญแญแแ แแญแแญ แฅแ แแแ แแ แจแต แซแแ แแแฝ แ แ แแญแ แแแ แฅแป แตแฅแข
**แจแฐแ แญ แแณแ แแแฐ-แฝแแ:**
[แ
แฝ 15 แแแแฅ แแฅแญ 80202]
... (แจแฐแ แญ แฝแแต แแณแ แฝแแ) ...
**แฅแซแ:**
แ แ
แฝ 15แฃ แแแแฅ แแฅแญ 80202 แจแฐแแแแ แ แจแญแญแญแฉ แแ แแณแญ แแ แแ แญ?<end_of_turn>
<start_of_turn>model
"""
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=False,
repetition_penalty=1.1
)
answer = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
print(answer)
๐๏ธ Citation & Authorship
Developed by Getachew Getu as part of the Amharic Legal RAG Research Project.
- Downloads last month
- 222