zidsi/GaMS2-27B_MQM

GaMS2-27B MQM EN→SL LoRA

LoRA adapter for structured MQM full-analysis of English→Slovenian translation pairs. The model outputs a JSON object with errors, mqm_score (0–25), norm_score (0–100), and summary.

• Base model: cjvt/GaMS2-27B
• Training dataset: zidsi/MQM_EN_SL (832 train / 93 validation)
• Chat template: Gemma2-style (chat_template.jinja bundled in this repo)
• Training base at run time: zidsi/GaMS2-27B (weights equivalent to cjvt/GaMS2-27B)

Intended use

Machine translation quality evaluation: given an EN source + SL hypothesis, produce an MQM JSON audit for human review or downstream QA pipelines. Not intended for general chat or open-ended generation.

Training

Trained with LlamaFactory (LoRA SFT, template: gemma2, cutoff_len: 12288):

Setting	Value
LoRA rank / alpha	64 / 128
Learning rate	2e-5
Epochs	3
Batch (effective)	32 (1 × 8 GPUs × grad_accum 4)
Final eval loss	0.5945

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0	0	1.0554
0.8425	0.3846	10	0.8080
0.7110	0.7692	20	0.6833
0.6230	1.1538	30	0.6402
0.6100	1.5385	40	0.6164
0.5413	1.9231	50	0.6033
0.6168	2.3077	60	0.5960
0.5977	2.6923	70	0.5946
0.5596	3.0	78	0.5945

Framework versions

• PEFT 0.14.0
• Transformers 5.13.0
• Pytorch 2.6.0+cu124
• Datasets 4.0.0
• Tokenizers 0.22.2

Validation inference (93-row holdout)

Inference run on the zidsi/MQM_EN_SL validation split with vLLM (temperature=0, max_tokens=4096). Gold labels: GPT-5 reference JSON.

Metric	Value
Valid JSON output	89 / 93 (95.7%)
Pearson r (mqm_score)	0.663
Pearson r (norm_score)	0.623
Exact mqm match	17 / 89 (19.1%)
Exact norm match	15 / 89 (16.9%)
Exact both scores	8 / 89 (9.0%)
mqm within ±1	26 / 89 (29.2%)

Observed failure modes:

• norm_score drift — model often more lenient than gold (median norm_delta −13)
• Over-penalization on easy cases — occasional mqm_score > gold by 10–30 points
• Invalid scores — some outputs exceed mqm_score cap of 25
• Unparsable JSON (4 rows) — repetitive error spam until max_tokens truncates mid-object

See VALIDATION_REPORT.md for full delta distributions, best/worst sample comparisons, and unparsable output analysis.

Usage

PEFT (Transformers)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "cjvt/GaMS2-27B"
adapter = "zidsi/GaMS2-27B_MQM"

tokenizer = AutoTokenizer.from_pretrained(adapter, trust_remote_code=True)
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(base, trust_remote_code=True),
    adapter,
)

vLLM (LoRA module)

vllm serve cjvt/GaMS2-27B \
  --enable-lora \
  --max-lora-rank 64 \
  --lora-modules mqm=zidsi/GaMS2-27B_MQM \
  --tokenizer zidsi/GaMS2-27B_MQM \
  --trust-remote-code

Request model: "mqm" in chat completions. Use the full MQM user prompt from the training dataset (taxonomy + EN/SL pair).

Limitations

• Task-specific JSON schema; outputs may include markdown fences or invalid scores without post-processing
• Long analyses can hit token limits and produce truncated / unparsable JSON
• Scores are not calibrated to human annotators — use for screening and comparison, not as sole QA gate