KhulasaAI — Arabic Text Summarization (QLoRA Adapter)

A LoRA adapter for Qwen2.5-0.5B-Instruct fine-tuned on Arabic text summarization via Knowledge Distillation from Qwen2.5-7B-Instruct.

Model Description

Property	Value
Base Model	Qwen/Qwen2.5-0.5B-Instruct (494M parameters)
Teacher Model	Qwen/Qwen2.5-7B-Instruct
Method	QLoRA (r=16, α=32, dropout=0.05)
Target Modules	q_proj, k_proj, v_proj, o_proj
Language	Arabic (Modern Standard Arabic)
Task	Abstractive Text Summarization
Adapter Size	8.3 MB
License	MIT

Training Details

Dataset

• Source: Arabic Wikipedia (wikimedia/wikipedia, 20231101.ar)
• Size: 5,000 articles randomly sampled (seed=42)
• Annotation: Synthetic summaries generated by Qwen2.5-7B-Instruct (4-bit quantized)
• Split: 90% train / 10% test

Hyperparameters

• Epochs: 1
• Learning Rate: 2e-4
• Batch Size: 4 (gradient accumulation: 2, effective batch: 8)
• Precision: FP16
• Max Sequence Length: 1024 tokens
• Optimizer: AdamW
• Total Steps: 282

Hardware

• GPU: NVIDIA Tesla T4 (16 GB VRAM)
• Platform: Kaggle Notebooks
• Training Time: ~14 minutes

Evaluation Results

Metric	Score
ROUGE-1	62.21
ROUGE-2	41.17
ROUGE-L	60.66
BERTScore (F1)	0.861

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load model
base_model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "mohammed-el-baraka/KhulasaAI")

# Summarize
text = "your Arabic text here..."
messages = [
    {"role": "system", "content": "لخص النص التالي."},
    {"role": "user", "content": f"النص: {text}\n\nالملخص:"}
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(inputs.to(model.device), max_new_tokens=150, temperature=0.7, do_sample=True)
summary = tokenizer.decode(outputs[0], skip_special_tokens=True).split("assistant")[-1].strip()
print(summary)

Limitations

• Trained on Modern Standard Arabic (MSA) only — performance on dialectal Arabic (Egyptian, Gulf, Moroccan, etc.) is not guaranteed.
• Summaries are limited to ~150 tokens due to max_new_tokens constraint during annotation.
• The model may hallucinate facts not present in the source text, a known limitation of abstractive summarization.
• Input articles were truncated to 2,000 characters during training, so performance on very long documents may degrade.

Citation

@misc{elbaraka2026khulasaai,
  title={KhulasaAI: Arabic Text Summarization via Knowledge Distillation},
  author={Mohammed El Baraka},
  year={2026},
  month={january},
  howpublished={\url{https://github.com/mohammed-el-baraka/KhulasaAI}}
}

Author

Mohammed El Baraka — University Mohammed VI Polytechnic (UM6P)

Framework Versions

• PEFT 0.18.0
• Transformers 4.40+
• PyTorch 2.0+
• TRL 0.8+