jgchaparro/Llama-Leonida

Leonida: Greek ⇄ Tsakonian translator (merged 16-bit)

Leonida translates between Modern Greek and Tsakonian, in both directions, with a single model. Tsakonian is a highly endangered Hellenic language of the Peloponnese, which descends largely from ancient Laconian Doric. Its name honours Leonidas, the Spartan king, a nod to that Laconian ancestry; the logo shows a Spartan shield with the lambda of Lacedaemon.

Leonida is part of Tsakonian Digital. It is the successor of Leonida Beta and, like it, is trained on openly available sources only, mainly the website of Panos Marneris, tsakonika.online.

• Scores: chrF++ 83.1, BLEU 70.0 and TER 17.4 on Tsakonian Digital Bench V2.1 (600 translations).
• Two training stages: supervised fine-tuning on 17,960 sentence pairs, then reinforcement learning (GRPO) on 2,819 further pairs that the first stage never saw.
• Small enough to run locally: 8 billion parameters; the Q4_K_M file runs on a laptop.

Repository	Contents
jgchaparro/Llama-Leonida	Merged 16-bit model (transformers)
jgchaparro/Llama-Leonida-GGUF	GGUF Q8_0, Q5_K_M, Q4_K_M for llama.cpp, Ollama and LM Studio
jgchaparro/Llama-Leonida-lora	LoRA adapter for ilsp/Llama-Krikri-8B-Instruct-v1.5

Usage

Tsakonian is written in the Kostakis orthography, with combining diacritics: τσ̑, σ̌, κ̔, ν̇, λ̣, ν̂, α̈. The model reads and writes this spelling; text in other spellings translates less well.

Use the model's Llama 3.1 chat template with this prompt. The model replies with the translation only.

• System: You are a precise translation assistant. Output only the translation, nothing else.
• User: Translate from {origin} to {target}:\n\n{source}, where {origin} and {target} are Greek or Tsakonian.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jgchaparro/Llama-Leonida"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
messages = [
    {"role": "system", "content": "You are a precise translation assistant. Output only the translation, nothing else."},
    {"role": "user", "content": "Translate from Greek to Tsakonian:\n\nΚαλημέρα, τι κάνεις;"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False) + "<|start_header_id|>assistant<|end_header_id|>\n\n"
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training data: openly available sources

Leonida was trained on 17,960 Tsakonian–Greek sentence pairs, each used in both directions (35,920 training examples). All of them come from openly available sources, mainly the website of Panos Marneris, tsakonika.online. The pairs were cleaned and reviewed again before this model was trained.

The reinforcement-learning stage used 2,819 further pairs (5,636 prompts, both directions) from the same sources, held out from the first stage. The validation pairs and the benchmark pairs belong to separate splits and are used in neither stage.

The Tsakonian side of every pair is written in the Kostakis orthography; texts written in other orthographies were converted and reviewed.

How it was trained

Stage 1: supervised fine-tuning.

• Base model: ilsp/Llama-Krikri-8B-Instruct-v1.5, fine-tuned with Unsloth: 4-bit QLoRA, LoRA rank 64, alpha 64, on the attention and MLP layers. The base model is Llama-Krikri, Llama 3.1 8B adapted to Greek by ILSP (Athena Research Center) with additional Greek training and an extended Greek vocabulary.
• Optimisation: learning rate 0.0002 with cosine decay and 3 % warmup, AdamW 8-bit, effective batch 16, maximum sequence length 768 tokens, up to 3 epochs (6,735 steps). The loss covers the model's reply only.
• Checkpoint selection: every 750 steps, the adapter translated 400 validation examples (200 held-out pairs × 2 directions, greedy) and was scored with chrF++. The best checkpoint scored validation chrF++ 81.00.

Stage 2: reinforcement learning (GRPO).

• Method: Group Relative Policy Optimization (Dr. GRPO loss, no KL term), continuing the stage-1 adapter. For each prompt the model sampled 8 translations at temperature 0.9, and each translation was rewarded against the reference.
• Reward: half sentence chrF++, half (1 − TER), plus a bonus for an exact match. Automatic checks on punctuation, numbers, capitals and diacritics subtract a penalty for each error the reference does not share. Empty output, a copy of the source, a translation of very different length or a truncated reply score 0.
• Optimisation: learning rate 0.00001, 300 updates of 16 prompts (of up to 400; stopped after three checks without improvement).
• Checkpoint selection: validation chrF++ every 50 updates, on the same 400 examples as stage 1. The published adapter is from update 150, validation chrF++ 81.79 (stage 1: 81.00).

Update	Validation chrF++	TER	Tsakonian → Greek chrF++	Greek → Tsakonian chrF++
0 (stage 1)	81.00	20.07	81.92	80.14
50	81.36	19.57	82.34	80.44
100	81.49	19.34	82.54	80.50
150 (selected)	81.79	19.07	82.62	81.02
200	81.58	19.28	82.51	80.72
250	81.77	19.07	82.51	81.08
300	81.79	19.09	82.56	81.08

Evaluation: Tsakonian Digital Bench V2.1

Tsakonian Digital Bench V2.1 has 300 Tsakonian–Greek sentence pairs from the test split of the Tsakonian Digital database. None of them is used in training or in validation.

• Levels: 100 easy (fewer than 7 Tsakonian words), 100 medium (7–11 words) and 100 hard (12 or more words).
• Sources: openly available texts of the same kind as the training data, mostly from Panos Marneris's website.
• Review: every pair was checked one by one for alignment, spelling and punctuation.
• Scoring: every pair is translated in both directions (600 translations) with greedy decoding, and scored with sacreBLEU 2.5.1: BLEU, chrF++ and TER. The 95% confidence intervals come from bootstrap resampling.

The benchmark is in this repository, in benchmark/tsakonian_digital_bench_v2_1.csv, with columns id, level, words, dialect, source_id, active, tsd, ell. The model's translations are in benchmark/predictions.jsonl, and the full scores are in benchmark/metrics.json.

Group (translations)	BLEU ↑ (95% CI)	chrF++ ↑ (95% CI)	TER ↓ (95% CI)
All (600)	70.0 (68.0–72.0)	83.1 (81.9–84.3)	17.4 (16.2–18.6)
Tsakonian → Greek (300)	73.7	84.6	15.4
Greek → Tsakonian (300)	66.4	81.7	19.3
Easy (200)	69.2	82.7	17.3
Medium (200)	67.6	81.5	19.1
Hard (200)	71.1	83.8	16.7
Easy / Tsakonian → Greek (100)	72.8	84.0	16.0
Easy / Greek → Tsakonian (100)	65.7	81.4	18.6
Medium / Tsakonian → Greek (100)	70.2	82.5	17.7
Medium / Greek → Tsakonian (100)	65.1	80.5	20.4
Hard / Tsakonian → Greek (100)	75.2	85.5	14.2
Hard / Greek → Tsakonian (100)	67.0	82.2	19.0

Comparison

Model	BLEU ↑	chrF++ ↑	TER ↓
Leonida (this model)	70.0	83.1	17.4
Leonida, stage 1 only (supervised fine-tuning)	68.5	82.1	18.4
Leonida Beta, the previous version (same base model and open sources, supervised fine-tuning only)	66.8	81.2	19.5

Against Leonida Beta on the same 600 translations (paired bootstrap, 95% intervals): chrF++ +1.91 (+1.10 to +2.68), BLEU +3.25 (+1.90 to +4.59), TER -2.15 (-2.94 to -1.25). Every interval excludes zero. Leonida has the higher sentence chrF++ on 34.5 % of the translations, Leonida Beta on 21.5 %; the rest are ties.

Intended use

• Drafting translations between Greek and Tsakonian, for learners, teachers and researchers.
• Reading Tsakonian texts with a Greek gloss.
• A starting point for further research on Tsakonian and other low-resource Greek varieties.

Leonida is not a substitute for a native speaker. Any text meant for publication, teaching or signage needs review by a speaker.

Limitations

• Dialects. Most of the training text is in the Tyros dialect (Τ). Other varieties are less well covered.
• One reference per sentence. BLEU and chrF++ count a valid paraphrase or dialect variant as an error, so the scores understate the quality of good alternative translations.
• Tsakonian output can be wrong. The model can produce plausible but wrong forms, misplace diacritics or choose the wrong gender or number when the source does not determine it.
• Sentence level. The model was trained on single sentences. Translate long texts sentence by sentence.

License

Built with Llama. The model is subject to the Llama 3.1 Community License and its Acceptable Use Policy.