latxa-8b-gec-lora-v2 — Basque GEC LoRA teacher

A LoRA adapter for HiTZ/Latxa-Llama-3.1-8B-Instruct that turns the model into a Basque grammatical-error corrector, and the teacher behind itzune/gector-eus-v5 and the labels in itzune/wiki-gec-distill-eu.

On real Wikipedia corrections it scores F0.5 58.2 / exact 28.1% (margin −1), roughly 2.4× the F0.5 of the CPU-only GECToR baseline on the same slice.

Model details

Base model	HiTZ/Latxa-Llama-3.1-8B-Instruct
Adapter type	LoRA (PEFT), r=16, alpha=32, dropout=0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trained parameters	~42M (adapter file 177 MB)
Training precision	bf16, loss on the assistant turn only
Training length	1 epoch (4,318 steps, batch 16), max 384 subwords
Prompt	Basque system message asking for minimal corrections grounded in EBE
Decoding	greedy + copy-vs-edit margin (see below) — not plain sampling

Intended use

Two roles, both as a teacher/filter, not as a chat model:

1. Bulk labelling: correct a large unlabelled Basque corpus, keeping only the edits the model is confident about (this produced 36,878 training pairs for GECToR v5).
2. Candidate filtering / reranking: score a human edit against the unchanged sentence and keep the edit only if the model prefers it — a cheap second filter for mined Wikipedia revisions.

How to use

import torch
from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM

base = "HiTZ/Latxa-Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="cuda"),
    "itzune/latxa-8b-gec-lora-v2",
)

SYSTEM = ("Zuzendu emandako euskara testuko akatsak, aldaketa minimalak eginez. "
          "Akatsak EBE (Euskara Batuaren Eskuliburua) oinarritzat hartuz zuzendu.")
msgs = [{"role": "system", "content": SYSTEM},
        {"role": "user", "content": "Hitz hau gaizki idatzi da ."}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=256, do_sample=False)
print(tok.decode(out[0, len(tok(prompt)["input_ids"]):], skip_special_tokens=True))

The copy-vs-edit margin (required for good precision)

The adapter over-corrects when used greedily on clean text. The reference pipeline therefore scores two sequences with the same teacher-forced prompt:

• lp_out — log P(proposed correction + eos | prompt)
• lp_copy — log P(unchanged sentence + eos | prompt)

and keeps the correction only if lp_out − lp_copy > −margin. Margin −1 is the operating point used for distillation; larger margins trade precision for recall (+1 gives F0.5 61.1 but 6.9% false corrections on clean text). Implementation: scripts/llm_nbest.py, scripts/sweep_llm_margin.py.

Training data

source	records
horkonpon-corpus v4.1 train (chat format, 40% clean copies)	63,064
accepted human Wikipedia edits (Latxa-70B-judged)	6,012
total	69,076 (+ 500 dev)

Built with scripts/build_lora_data.py (leakage guard: any sentence in the real or synthetic eval sets is dropped). Supervised on the assistant turn only.

Results

Evaluated with the same scorer as the GECToR models (eusgec), int4 CPU GECToR vs 8B GPU LoRA.

Real Wikipedia slice — 171 errorful + 247 clean hand-checked sentences:

margin	exact	P	R	F0.5	clean FP
−4	12.3%	75.8	13.2	38.8	0.8%
−2	22.2%	73.8	25.3	53.3	2.4%
−1 (recommended)	28.1%	75.3	30.5	58.2	2.8%
0	31.6%	70.2	34.7	58.3	4.5%
+1	35.1%	71.2	38.9	61.1	6.9%

Synthetic blind slice — 300 errorful + 300 clean:

margin	exact	F0.5	clean FP
−2	55.7%	79.6	3.0%
−1	59.7%	79.5	4.3%
0	61.7%	78.9	8.3%

For reference on the same real slice: GECToR v2 F0.5 15.7 → GECToR v5 43.7 → this adapter 58.2.

Limitations

• Not a drop-in corrector: without the margin rule it edits clean text (up to 8% of clean sentences at margin 0) and needs ~16 GB of GPU memory in bf16.
• Teacher, not ground truth: its edits were accepted into training data after filtering, and the remaining noise is inherited by students.
• Wikipedia-heavy: the mined-edit half comes from euwiki-20261101, so its real-world numbers benefit from the same genre.
• Small eval set (171 sentences); treat single-point differences as noise.
• Latxa-8B is not trained on long documents; keep inputs to single sentences.

Licensing

The adapter is a derivative of HiTZ/Latxa-Llama-3.1-8B-Instruct and is released under the Llama 3.1 Community Licence (see license_link). Note the licence's restriction on using model outputs to train other large language models; the distillation target here is a 125M token-classification model, not an LLM, but review the terms before commercial use. Training data: horkonpon-corpus (CC-BY-SA-4.0) and Basque Wikipedia edits (CC-BY-SA-4.0).

Citation

@misc{ezpeleta2026latxa8bgeclorav2,
  author       = {Xabi Ezpeleta},
  title        = {latxa-8b-gec-lora-v2: a LoRA teacher for Basque grammatical
                  error correction},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/itzune/latxa-8b-gec-lora-v2}},
  note         = {Base model: \url{https://huggingface.co/HiTZ/Latxa-Llama-3.1-8B-Instruct}}
}

References

• Etxaniz, J., Arriola, M., et al. (2024). Latxa: An Open Language Model and Evaluation Suite for Basque. arXiv:2403.20266
• Base model card: HiTZ/Latxa-Llama-3.1-8B-Instruct
• Meta Llama 3.1 Community Licence: LICENSE
• Training data: itzune/horkonpon-corpus
• Teacher output dataset: itzune/wiki-gec-distill-eu
• Student model: itzune/gector-eus-v5
• Tools: PEFT, Transformers, vLLM, itzune/gector-eus-v2 (data builders, margin sweep)
• Normative reference: Euskaltzaindia — EBE
• Basque Wikipedia dump euwiki-20261101 (dumps.wikimedia.org), CC-BY-SA-4.0