Error Analysis of Multilingual Language Models in Machine Translation: A Case Study of E nglish- A mharic Translation (2024)

Abstract: Multilingual large language models (mLLMs) have significantly advanced machine translation, yet challenges remain for low-resource languages like Amharic. This study evaluates the performance of state-of-the-art mLLMs, specifically NLLB-200 (NLLB3.3, NLLB1.3 Distilled1.3, NLB600) and M2M (M2M1.2B, M2M418), in English-Amharic bidirectional translation using the Lesan AI dataset. We employed both automatic and human evaluation methods to analyze translation errors. Automatic evaluation used BLEU, METEOR, chrF, and TER metrics, while human evaluation assessed translation quality at both word and sentence levels. Sentence-level accuracy was rated by annotators on a scale from 0 to 5, and word-level quality was evaluated using Multidimensional Quality Metrics. Our findings indicate that the NLLB3.3B model consistently outperformed other mLLMs across all evaluation methods. Common error included mistranslation, omission, untranslated segments, and additions, with mistranslation being particularly common. Punctuation and spelling errors were rare in our experiment.

## introduction machine translation (mt), the automated process of converting text from one language to another, has shown significant advancements in recent years @xcite . however, these improvements have primarily focused on high-resource languages. in contrast, low-resource languages, such as amharic language -the official language of ethiopia, spoken by 20 million people-, have not received the same level of attention @xcite . previous research on amharic machine translation has primarily focused on fine-tuning language models. however, few studies have assessed the effectiveness of multilingal language models for low-resource languages such as amharic. for example, @xcite employed a statistical method to translate texts, improving a baseline phrase-based system. similarly, @xcite addressed inflectional morphology by using subword units and adjusting model parameters to better suit low-data environments. their approach effectively handled amharics morphological complexity. @xcite developed an amharic-english parallel corpus for both statistical machine translation and neural machine translation. in addition, @xcite introduced lesan, a machine translation system for low-resource languages, built on the transformer architecture, and provided an evaluation dataset. in a more recent effort, @xcite expanded the amharic-english corpus and experimented with tri-gram and four-gram smt models, as well as gru-based neural machine translation models. despite these efforts, no prior studies have systematically analyzed the strengths and limitations of large language models in the context of amharic machine translation. in this study, we aim to identify the shortcomings of multilingual large language models in translating low-resource languages, using amharic a case study. using both human and automatic evaluation techniques, we identify common errors made by mllms in amharic-english translations, providing insights for future researchers to improve amharic machine translation. specifically, we focus on the bidirectional amharic-english mt system, with the following key objectives: ## related works this section reviews the related work to multilingual large language models for machine translations, with a particular focus on the evaluation for bidirectional amharic-english machine translation. multilingual large language models are advanced neural architectures trained on large amounts of textual data, that can understand and generate multiple human languages. these models can be further fine-tuned for various downstream tasks such as machine translation, text classification, and question answering @xcite . while the application of mllms in machine translation has gained significant attention in recent years, there is a limited research for evaluating the quality of llms in translation low-resource languages. recent advancements in mllms , researchers have explored various techniques to improve the peformance of mllms. for instance, @xcite conducted a comprehensive evaluation study for gpt models and analyzed different aspects such as their performance compared to state-of-the-art research and commercial systems. their findings indicated that while gpt models provide competitive translation quality for languages with rich resources, their effectiveness is notably when applied to languages with limited resources. on the other hand, significant advancements have been made by major companies, such as meta, in developing llms capable of processing multiple languages, including those with limited resources. for instance, @xcite proposed a many-to-many multilingual translation model that facilitates direct translation between any pair of 100 languages. this model uses a data mining strategy based on language similarity to create a comprehensive multilingual dataset. furthermore, the authors employed a back-translation technique to enhance the model's performance for zero-shot and low-resource language pairs, resulting in a training corpus of 7.5 billion sentences across 100 languages, including amharic. the "no language left behind" project is a notable effort by meta ai, uc berkeley, and johns hopkins university @xcite . this interdisciplinary project aims to develop humancentered machine translation solutions for over 200 low-resource languages, including amharic. furthermore, the project focuses on dataset construction and model development to reduce the translation quality gap between resource-rich and resource-limited languages. evaluation of machine translation for semitic languages despite the growing interest in applying llms to machine translation, there has been limited research evaluating their efficacy in this domain. a notable exception is the work by @xcite , who assessed stateof-the-art mt for tigrinya, a semitic language closely related to amharic and spoken by over 10 million people in ethiopia and eritrea. their study utilized evaluation datasets spanning diverse domains, including arts and culture, business and economics, politics, and science and technology. employing the mqm-dqf error topology for translation assessment, the researchers concluded that mistranslation and omission were the most prevalent errors. this study provides valuable insights into the challenges of mt for semitic languages, which may have implications for amharic translation as well. ## experiment we conducted our experiments to answer the following research questions regarding the peformance of multingual language models in lowresource languages, using amharic as a case study: rq 1 : how do mllms compare in their performance in translation low-resource language when using automatic evaluation metrics? rq 2 : what are the most frequent error types of mllms in the machine translation for lowresource languages? rq 3 : which mllm performs better in translating low-resource language when using human evaluation? ## error analysis in this section, we analyze the most frequent errors we observed in our experiment and their patters. figures 2 and 3 show the the error distribution of the four models on the lesan dataset. in this experiment, we saw that the primary cause of mistranslation errors is the misinterpretation of personal and place names. for example, all models failed to correctly translate the name of dr. abiy, ethiopia's prime minister, from amharic to english. the m2m 1.2b model translated his name as "al-qaeda." similarly, the name "gojam," an area in the amhara regional state of ethiopia, where the regional capital (bahirdar) located, was translated to "gujarat" a city in indiaby nllb600. additionally, the city name (seladinigayi) was inaccurately translated to "saturday." however, the models successfully translated well-known place names such as russia, africa, and pakistan. our experiment shows that mistranslation of entities are better handled in from english to amharic than vise versa. in some cases, models made critical mistranslation errors. for instance, nllb600 mistranslated the sentence "hahaha ire benatishi koka neberi yeteshegewi yih lewitima sewini hulu wiha kelemina tikuri kokani yawi newi" which roughly translates to "haha, please, i thought it was coca; this will make everyone believe the color of water and coca is the same," to "ha ha ha, your dad was a cocaine smuggler. this change makes everyone think that the color of water and black coca cola are the same," an entirely incorrect translation that could lead to serious legal issues. in addition, models tend to translate without considering context. for example, the models translated the sentence "እዝራ ፓውንድ የአሜሪካ ጸሐፊ ነበር" (izira pawinidi yeamerika tseafin eberi) to "ezra pound was an american secretary of state." the amharic word "ጸሐፊ"(ts'eafi) was mistranslated directly to "secretary," although it can mean both "secretary" or "writer" depending on the context. here it should be translated to "writer". another example is the translation of "ጠቅላይ ሚኒስቴር" (t'ek'ilayi minisiteri) where the nllb3.3 model translated it to "general minister" whereas the correct translation is "prime minister." the model ignored the context, which required the translation to be "prime minister". additionally, the word "groundbreaking" was mistranslated as "የመሬት መሰራጨት"(yemereti meserach'eti") unrelated to the intended meaning. the model interpreted "ground" as "መሬት," referring to the earth's solid surface rather than its figurative sense. in addition to translating words out of the context, the models tends to translate words literally. for instance, nllb3.3 translated the company name "friendly" to "wedajineti", which is incorrect; the name should either be left untranslated or transcribed into the amharic alphabet. similarly, the film title "the sound of music" was translated as "የሙዚቃው ድምፅ" (yemuzik'awi dimit s'i) a direct translation of each word in the title. as shown in figures 2 and 3 , there were very few errors related to punctuation and spelling. ## conclusion this paper evaluates the effectiveness of two public multilingual language models trained on amharic: the m2m and nllb models, both developed and released by meta. in our evaluation, we we used a dataset created by lesan ai. 3 first,we conducted our experiments with automatic evaluation, which show that the m2m models were ineffective in translating between both language directions. a preliminary review of the translations indicated that nearly all outputs were inadequate, suggesting that human evaluation of the m2m models may be unnecessary due to the complexity and associated costs. in the human evaluation phase, we conducted two types of experiments: i) sentence level and ii) word level. our findings indicate that across all evaluation techniques --automatic evaluation and both sentence level and word-level human assessments-the nllb3.3b model outperforms the other models. notably, while the nllb600d model shows a superior performance in translating from english to amharic compared to the nllb1.3d and nllb1.3 models, it performed worse in automatic evaluations. in the word-level human evaluation, the most common error types are: mistranslation, omission, and untranslated segments with entities and longer sentences being the main reason for the errors. grammar, punctuation and spelling errors rarely appear in our experiments. as part of our efforts to enhance translation quality, we released the annotated dataset under a permissive license. we hope that this research work will be a basis for future improvements of mllms for under resourced languages such as amharic.