Correcting the Tamazight Portions of FLORES + and OLDI Seed Datasets (2025)

Abstract: We present the manual correction of the Tamazight portions of the FLORES+ and OLDI Seed datasets to improve the quality of open machine translation resources for the language. These widely used reference corpora contained numerous issues, including mistranslations, orthographic inconsistencies, overuse of loanwords, and non-standard transliterations. Overall, 36% of FLORES+ and 40% of Seed sentences were corrected by expert linguists, with average token divergence of 19% and 25% among changed items. Evaluation of multiple MT systems, including NLLB models and commercial LLM services, showed consistent gains in automated evaluation metrics when using the corrected data. Fine-tuning NLLB-600M on the revised Seed corpus yielded improvements of +6.05 chrF (en→zgh) and +2.32 (zgh→en), outperforming larger parameter models and LLM providers in en→zgh direction.

## introduction high-quality parallel data is a cornerstone for the development of robust machine translation (mt) systems, particularly for low-resource languages where each sentence can significantly impact model performance. for tamazight-spoken by over 40 million people across north africa and the diaspora-parallel corpora are scarce, and widely used datasets such as flores+ @xcite and oldi seed @xcite play a significant role in enabling mt research and evaluation. the tamazight portions of flores+ foot_0 and oldi seed foot_1 contain standard moroccan tamazight (iso 639-3: zgh), the standardized variety developed by the royal institute of amazigh culture (ircam) @xcite for education and official use since 2001. however, our initial inspection revealed substantial issues: mistranslations, orthographic inconsistencies, malformed or unnecessary loanwords, non-standard transliterations, and occasional semantic inaccuracies. some of these errors echo broader findings about low-resource language datasets, where insufficient quality control leads to degraded mt outputs and downstream application failures @xcite . similar issues have been observed for other languages in flores+, prompting targeted correction efforts such as those for hausa, isizulu, northern sotho, and xitsonga @xcite . in fact, when flores-200 was first released, the tamazight data was mislabeled as central atlas tamazight (tzm) and only corrected following community feedback to its correct code zgh. as part of the awal project @xcite , which develops open-source mt and speech technologies for tamazight and coordinates community data creation, we undertook a systematic manual correction of the flores+ dev and devtest sets (997/1,012 sentences) and the oldi seed corpus (6,193 sentences). corrections were performed by expert linguists using authoritative ircam lexicographic and grammatical resources. the motivation for this work is straightforward: to ensure these widely used benchmark and seed datasets truly reflect standard moroccan tamazight, so that mt systems trained or evaluated on them can produce translations that are both accurate and culturally appropriate. we follow the approach of @xcite for quantifying the extent of changes. we then evaluate multiple state-of-the-art open-source and commercial mt systems (including llms) on both the original and corrected flores+ devtest data for english↔tamazight, and fine-tune existing nllb models with the updated seed data to assess downstream impact. ## change analysis tamazight portions of the flores+ dev/devtest and oldi seed datasets using token divergence (abdulmumin et al., 2024), bleu (papineni et al., english source original translation back-translation correct translation even if you're driving through the subtropical rainforest, a few seconds with the doors open while you get inside the vehicle is enough time for mosquitoes to get in the vehicle with you. ⵡⴰⵅⵅⴰ ⴷⴰ ⵜⵏⴷⴷⵀⵜ ⴳ ⵜⴰⴳⴰⵏⵉⵏ ⵏ ⵓⵏⵥⴰⵕ ⵉⵔⵡⴰⵙⵏ ⴰⵙⴱⴳⵙ, ⴷⵔⵓⵙ ⵏ ⵜⵓⵙⴷⵉⴷⵉⵏ ⴷ ⵜⴼⵍⵡⵉⵏ ⵉⵥⵕⵎⵏ ⵜⵉⵍⵉⴷ ⴰⴳⵏⵙⵓ ⵏ ⵜⵎⵙⵙⵓⴷⵓⵜ ⵉⴳⴰ ⴰⵙ ⵉ ⵡⴰⴱⵉⴱⴰ ⴰⴷ ⵉⴷⴽ ⵉⵙⵙⵓⵔ ⵜⴰⵎⵙⵙⵓⴷⵓⵜ. although you are driving in rainy forests like "asbgs", few minutes and the doors open and you are inside the vehicle is fine for mosquitoes to share the vehicle with you. ⵡⴰⵅⵅⴰ ⴷⴰ ⵜⵏⴷⴷⵀⴷ ⴳ ⵜⴰⴳⴰⵏⵉⵏ ⵏ ⵓⵏⵥⴰⵕ ⵜⵉⵎⴽⵡⴰⵜⵓⵔⵉⵏ, ⴳⵓⵎⵏⵜ ⴽⵔⴰ ⵏ ⵜⵙⵉⵏⵉⵏ ⴷⵔⵓⵙⵏⵉⵏ ⴰⴽⵓⴷ ⵜⵕⵥⵎⴷ ⵜⴼⵍⵡⵉⵏ ⴰⴷ ⵜⴽⵛⵎⴷ ⵜⴰⵎⵙⵙⵓⴷⵓⵜ ⵉ ⵡⴰⴱⵉⴱⴰ ⵃⵎⴰ ⴰⴷ ⵉⴷⴽ ⵉⵙⵙⵓⵔ ⵜⴰⵎⵙⵙⵓⴷⵓⵜ. the walls and roofs of ice caves can collapse and cracks can get closed. ⵉⴳⵓⴷⴰⵔ ⴷ ⵉⴼⵔⴰⵏ ⵏ ⵓⴳⵔⵉⵙ ⵉⵥⴹⴰⵕ ⴰⴷ ⵉⵜⵜⵓⵢⵏ ⴷ ⵉⴼⵓⵔⵙⵍⵏ ⴰⴷ ⵇⵇⵏⵏ. the walls and ice caves can be high and skin fissure will close. ⵥⴹⴰⵕⵏ ⵉⴳⵓⴷⴰⵔ ⴷ ⵉⴳⴳⵉⵜⵏ ⵏ ⵉⴼⵔⴰⵏ ⵏ ⵓⴳⵔⵉⵙ ⴰⴷ ⵔⴷⵍⵏ ⴷ ⵉⵙⵜⵜⵉⵖⵏ ⴰⴷ ⵇⵇⵏⵏ. for a few pennies some children will tell you the story. ⵖⴰⵔ ⴽⵔⴰ ⵏ ⵜⵙⵓⵔⵓⴼⵉⵏ ⴷⵔⵓⵙⵏ, ⴰⴷ ⴰⴽ ⴰⵍⵙⵏ ⵉⵛⵉⵔⵔⴰⵏ ⵜⴰⵏⴼⵓⵙⵜ. just some few steps children will tell you the story. ⵖⴰⵙ ⴽⵔⴰ ⵏ ⵜⴰⵔⵔⵢⴰⵍⵉⵏ ⴷⵔⵓⵙⵏⵉⵏ, ⵔⴰⴷ ⴰⴽ ⴰⵍⵙⵏ ⵉⵛⵉⵔⵔⴰⵏ ⵜⴰⵏⴼⵓⵙⵜ. the results show that the proportion of sentences corrected is 36% for flores+ overall and 40% for the seed dataset. average token divergence values are 19% for flores and 25% for seed, indicating that corrections typically involved localised changes to a minority of tokens in a sentence rather than complete rewrites. this is consistent with the high bleu scores (≈78-81) and the relatively modest wer (13-18%) and cer (5-8%) values. while substantial rewrites are relatively rare, they are still present in both datasets. ## machine translation evaluation we evaluated english↔tamazight (zgh) translation quality on the flores+ devtest split, comparing results obtained with the original and corrected versions of the dataset. the evaluation covered: across all models tested, corrections led to consistent improvements in bleu and chrf++, with corresponding reductions in ter. we see an average chrf increase of 0.14 points in @xmath0, and 0.23 in @xmath1. this confirms that noise in the original flores+ tamazight references measurably affected evaluation quality. fine-tuning on the corrected oldi seed dataset (nllb-0.6b-ft) yielded substantial gains compared to the same 0.6b baseline: +6.05 chrf and +3.05 bleu in the @xmath2. the finetuned model achieved 31.69 chrf, performing better than all other larger models and llm providers, showing that carefully curated data can yield substantial improvements. in the @xmath3, the fine-tuned 0.6b model improved with +2.32 chrf and +1.89 bleu from corrections, however it was still outperformed by other models, with gemini pro 2.5 achieving the highest chrf score of 44.29. we also report results with additional parallel data (nllb-0.6b-ft+): 32.71 chrf and 8.84 bleu in @xmath4, and 40.66 chrf and 18.29 bleu in @xmath5. ## conclusions we carried out a systematic manual correction of the tamazight portions of the flores+ dev/devtest and oldi seed datasets, resolving orthographic inconsistencies, mistranslations, and problematic loanwords. these corrections, performed by professional linguists with reference to authoritative resources, affected a substantial share of sentences and resulted in more accurate benchmarks for mt evaluation. across all off-the-shelf models and commercial llms tested, the corrected data yielded consistent improvements in automatic evaluation metrics. fine-tuning on the corrected oldi seed dataset further demonstrated the impact of these revisions: the nllb-600m model trained on the revised data outperformed larger parameter models and llm providers in @xmath6, and recorded improvements in @xmath7. beyond the experiments, our work underlines broader challenges of dataset creation for a language still undergoing standardization, where legitimate variation coexists with quality issues. it also illustrates how massively parallel datasets can be problematic for low-resource languages: without careful linguistic validation, they risk amplifying errors and misrepresenting the language in downstream systems. future work will extend fine-tuning experiments with additional parallel data and explore other language pairs involving tamazight, ensuring that corrected datasets serve as a stronger foundation for both evaluation and system development.