quechua-chanka-mt-prpe

Small Transformer (≈11.5M params, d_model=256, 4 encoder/decoder layers) trained from scratch for Spanish→Quechua (Chanka) translation, using PRPE (Prefix-Root-Postfix Encoding) — a morphologically-guided tokenizer — on the target (Quechua) side, 22k-piece vocabulary (98.3% token coverage on train). Compared against an identical model trained with standard BPE (quechua-chanka-mt-bpe) in a controlled ablation isolating the effect of the tokenizer.

Introduced as the extrinsic evaluation in QuechuaTok (Contreras, 2026), which shows that PRPE's morphological boundary accuracy (MorphAcc) — not just intrinsic fertility/perplexity — translates into real translation quality gains.

⚠️ No Hub inference widget

This model's target-side tokenizer (PRPE) is a custom morphological segmenter, not a standard SentencePiece/BPE tokenizer, so it is not compatible with the Hugging Face Inference widget or AutoTokenizer. Use the manual snippet below instead. (The BPE sibling model, linked above, does support the standard widget.)

Results (test set, n=12,206)

Tokenizer	MorphAcc%	BLEU	chrF
BPE	6.67	4.18	35.13
PRPE (this model)	83.33	7.05	41.10

+68.7% BLEU / +17.0% chrF over the BPE baseline, despite longer target sequences (18.6 vs 11.4 tokens/sentence) — morphological boundaries give the model more learnable, compositional units than memorizing surface forms.

Usage

from huggingface_hub import hf_hub_download
from transformers import MarianMTModel
import sentencepiece as spm
import sys, json, torch

repo_id = "makitty/quechua-chanka-mt-prpe"

# source tokenizer (Spanish, standard SentencePiece)
src_spm_path = hf_hub_download(repo_id, "source.spm")
sp_src = spm.SentencePieceProcessor()
sp_src.load(src_spm_path)

# target tokenizer (PRPE, custom morphological segmenter)
tok_module_path = hf_hub_download(repo_id, "tokenizer_prpe.py")
sys.path.insert(0, tok_module_path.rsplit("/", 1)[0])
import tokenizer_prpe as prpe  # provides tokenize_prpe_sentence / detokenize_prpe

vocab_path = hf_hub_download(repo_id, "prpe_vocab.json")
vocab = json.load(open(vocab_path, encoding="utf-8"))
inv_vocab = {v: k for k, v in vocab.items()}

model = MarianMTModel.from_pretrained(repo_id)

def translate(text):
    ids = sp_src.encode(prpe.normalize(text), out_type=int)[:62] + [sp_src.eos_id()]
    input_ids = torch.tensor([ids])
    out = model.generate(input_ids=input_ids, max_length=64, num_beams=4, decoder_start_token_id=1)
    toks = [inv_vocab.get(i, "") for i in out[0].tolist() if i not in (0, 1, 2)]
    return prpe.detokenize_prpe(toks)

print(translate("hola como estas"))

Training data

makitty/quechua-chanka-unified-v2, broad_train split (91k parallel sentence pairs).

Citation

@misc{contreras2026quechuatok,
  title={QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric
         for Tokenizer Evaluation in Agglutinative Low-Resource Languages},
  author={Contreras, Maria},
  year={2026},
  eprint={2606.23943},
  archivePrefix={arXiv}
}