Benchmarking Low-Resource Machine Translation Systems (2024)

Abstract: Assessing the performance of machine translation systems is of critical value, especially to languages with lower resource availability.Due to the large evaluation effort required by the translation task, studies often compare new systems against single systems or commercial solutions. Consequently, determining the best-performing system for specific languages is often unclear. This work benchmarks publicly available translation systems across 4 datasets and 26 languages, including low-resource languages. We consider both effectiveness and efficiency in our evaluation.Our results are made public through BENG—a FAIR benchmarking platform for Natural Language Generation tasks.

## introduction the machine translation (mt) task is increasingly relevant in today's connected world as accessibility enables knowledge transfer. hence, mt systems are recognized as prime tools in the natural language processing (nlp) domain @xcite . in recent years, neural machine translation (nmt) @xcite has led the field as it achieves state-of-the-art performance for many language pairs @xcite . however, nmt systems can become computationally demanding and the abundance of new systems also complicates a cross-system comparison. as a result, newly-released systems often compare their performance against single systems @xcite . furthermore, recent system analyses also focus on assessing the capability of commercial translation solutions @xcite . to the best of our knowledge, no work exclusively considers open-source translation systems. thus, leading to a lack of clarity when determining the best-performing and when identifying shortcomings among existing translation systems, an especially critical task for low-resource languages (lrls). while the translation task is vital to progress in general, it is still largely unfeasible to the 7, 000+ languages in the world. 1 from these, only close to 2, 500 are represented in the nlp field, with 88% considered to be lowresource. lrls have a minimal resource availability that causes them to be largely untouched by the benefits of language technology @xcite . with our work, we aim to contribute to a more complete picture of the current state of the art of machine translation with a focus on lrls. we compare four open-source nmt systems-libretranslate foot_1 , opus mt @xcite ), nllb @xcite , and mbart50 @xcite )-on four parallel machine-translation benchmark datasets-opus100 @xcite , europarl @xcite , iwslt2017 @xcite , and flores-200 @xcite . our evaluation comprises data from 26 different languages. our results suggest that using languages with lower resource availability does not necessarily translate to lower system performance. however, we did observe more substantial variations in the systems' performance for these languages. our analysis also showed that libretranslate had the highest token throughput among the evaluated systems. some systems showed proficiency in certain languages, while others performed better according to a certain dataset. our experiments are shared via beng @xcite , an open-source benchmarking platform that improves the accessibility of experiment results according to the fair data principles @xcite . 3 ## preliminaries and related work machine translation (mt) is the process of translating from a source language into a target language autonomously, i.e., without human intervention @xcite . this can be achieved through different approaches. @xcite divide mt techniques into ruleand corpus-based approaches. corpus-based approaches can be further divided into example-based, statistical, and, more recently, neural approaches. in this work, we evaluate approaches of the latter category with a focus on low-resource languages. we describe both further within this section, along with relevant mt tools and platforms. ## conclusion we compared four open-source nmt systems on high and low-resource languages regarding their effectiveness and efficiency, filling a gap in the literature that focused on the evaluation of single systems or the comparison of commercial solutions. our experiments show that open-source systems can perform well on lrls, showcasing the nlp community's efforts in bridging the gap. however, the performance of the systems in these languages remains variable. assessing the impact of the domain and genre of the training datasets on the translation quality remains a question for future work. despite the existence of numerous evaluation frameworks for mt, we used beng to share the evaluation data via a common space and hope that it boosts comparability across systems and datasets. the influence of language families and writing systems on the translation consistency of these systems requires further investigation.