greta44/albanian-spelling-lora

Albanian Spelling Exercise LoRA

LoRA adapter for automatic generation of Albanian spelling exercises (primary-school orthography), trained for PhD research in educational NLP.

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Method: PEFT LoRA (r=16, alpha=32)
• Language: Albanian (sq)
• Related dataset: greta44/albanian-error-augmentation

Safety note: Treat model output as a proposal only. For children, the final correct answer must come from deterministic Albanian rules / a curated database — not from the LLM alone.

Intended use

Generate short Albanian orthography exercise proposals, for example:

• missing_letter — plotëso shkronjën
• find_error — gjej gabimin
• explain_error — shpjego gabimin (pedagogjikisht)

Usage (Python demo)

1) Install

pip install torch transformers peft accelerate sentencepiece

2) Load the adapter and generate

import json
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

ADAPTER_ID = "greta44/albanian-spelling-lora"
BASE_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID, use_fast=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(BASE_MODEL)
model = PeftModel.from_pretrained(model, ADAPTER_ID)
model.eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

seed_word = "mirë"
grade = 3
exercise_type = "missing_letter"  # or: find_error, explain_error

payload = {
    "seed_word": seed_word,
    "grade": grade,
    "difficulty": "easy",
    "exercise_type": exercise_type,
    "safety": "Kthe vetëm propozim; përgjigjja finale kontrollohet nga rregullat.",
}

prompt = (
    "### Instruksion:\n"
    f"Gjenero një ushtrim të sigurt për drejtshkrimin shqip. Kategoria: {exercise_type}. "
    f"Klasa: {grade}. Vështirësia: easy. Fjala bazë: {seed_word}.\n\n"
    "### Input:\n"
    + json.dumps(payload, ensure_ascii=False)
    + "\n\n### Përgjigje:\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.4,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )

text = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(text.strip())

3) Or use the included script

python generate_exercises.py \
  --adapter greta44/albanian-spelling-lora \
  --seed-word mirë \
  --grade 3 \
  --type missing_letter

Training summary

Item	Value
Train / eval split	686 / 77 (from 763 AlbLingo exercises, seed=42)
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj
Max length	256
Effective batch size	4
Hardware	Apple M1 (MPS, float32)
Protocol target	3 epochs
Reported run	instrumented Phase A + Phase B continuation (~1.4 epoch-equivalent)
Train loss	3.149 → min 0.276
Held-out eval_loss	0.345
Approx. wall-clock	~62 minutes (logged span)

See training_manifest.json in this repository for the full measured summary, and training_loss.png for the loss curve.

Files in this repo

• adapter_model.safetensors + adapter_config.json — LoRA weights
• tokenizer* / vocab.json / merges.json — tokenizer files
• generate_exercises.py — standalone demo script
• training_manifest.json — hyperparameters + measured metrics
• training_loss.png — training loss curve
• instruction_dataset.jsonl — sample instruction pairs

Citation

If you use this model, please cite the related PhD research on Albanian educational NLP / AlbLingo and link this repository.

License

Apache 2.0