linguardia/opus-mt-zh-nan

Mandarin → Taiwanese Hokkien (opus-mt finetune)

Translates Mandarin into Taiwanese Hokkien. Finetuned by Linguardia from Helsinki-NLP/opus-mt-zh-en on Mandarin↔Taiwanese Hokkien sentence pairs from ChhoeTaigi dictionaries (github.com/ChhoeTaigi/ChhoeTaigiDatabase).

Built to generate course material for a language that no major machine translator supports.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("linguardia/opus-mt-zh-nan")
model = AutoModelForSeq2SeqLM.from_pretrained("linguardia/opus-mt-zh-nan")

batch = tok([">>nan<< 我很早起床。"], return_tensors="pt")
out = model.generate(**batch, num_beams=4, max_length=64)
print(tok.batch_decode(out, skip_special_tokens=True))

The target language is selected by the >>nan<< prefix on the source text. It was added as a new token rather than replacing one of the base model's existing languages, so every language the base already handled still works.

Training

base	Helsinki-NLP/opus-mt-zh-en
training pairs	106,678
held out	204 dev, 0 gold
epochs	3.0
batch size	32
optimiser	Adafactor, lr 5e-5, 500 warmup steps
max sequence	48 tokens (99.9% of source, 99.6% of target)
hardware	Apple M4, MPS

Evaluation

Held-out dev set (204 pairs never seen in training):

metric	score
BLEU	19.3
chrF	49.6

Training data and attribution

Sentence pairs come from ChhoeTaigi dictionaries (github.com/ChhoeTaigi/ChhoeTaigiDatabase). This model is a derivative of that work: if you use it, credit the corpus and its contributors under its own terms, recorded here as:

ChhoeTaigi dictionaries (github.com/ChhoeTaigi/ChhoeTaigiDatabase), CC BY-SA 4.0 and CC0 1.0; TaihoaSoanntengTuichiautian (BY-SA) + iTaigiHoataiTuichiautian (CC0) + TaioanPehoeKichhooGiku (BY-SA), Mandarin source side, normalised to Tai-lo

Pairs were deduplicated on exact (Mandarin, Taiwanese Hokkien) text. No machine translation was used to create the training data at any point.

Licence chain

layer	licence
base model Helsinki-NLP/opus-mt-zh-en	Apache-2.0
training data	CC BY-SA 4.0 and CC0 1.0
these weights	Apache-2.0, with the attribution above

Limitations

• Trained on short everyday sentences. Expect it to be weakest on long, technical or literary text.
• Neural MT translates each sentence independently, so a term may be rendered differently across a corpus. Check consistency if that matters to you.
• Output should be reviewed by a speaker before being taught to learners.