Quantifying the Contribution of MWE s and Polysemy in Translation Errors for E nglish– I gbo MT (2024)

Abstract: In spite of recent successes in improving Machine Translation (MT) quality overall, MT engines require a large amount of resources, which leads to markedly lower quality for lesser-resourced languages. This study explores the case of translation from English into Igbo, a very low resource language spoken by about 45 million speakers. With the aim of improving MT quality in this scenario, we investigate methods for guided detection of critical/harmful MT errors, more specifically those caused by non-compositional multi-word expressions and polysemy. We have designed diagnostic tests for these cases and applied them to collections of medical texts from CDC, Cochrane, NCDC, NHS and WHO.

## introduction in recent years, there has been increased research into improving the quality of machine translation (mt) outputs @xcite . evidenced by the switch from rule-based and statistical mt systems to neural mt systems, this has led to visible improvement of mt outputs. however, these improvements are more common with 'high-resourced languages' that have sufficient data resources for training mt models. thus, 'low/under-resourced languages' like igbo, lag behind in this progress. the igbo language (ásusu ìgbò) is one of the three major languages spoken in nigeria, it is the native language of the igbo people, an ethnic group in south-eastern nigeria. it is also, a recognised minority language in equatorial guinea and cameroon foot_0 . regarding language resources, igbo language, for instance, has only 18,369 wikipedia articles as of 02 october 2023 unlike the english and french languages that are in the top 5 languages used in wikipedia with 6,722,185 and 2,557,357 articles respectively. additionally, there are no single parallel corpora with igbo language as a language pair in sketch engine foot_1 and only a few available in opus 3 . the amount of parallel data that includes igbo language as one of the language pairs remains limited. thus, "igbo -any language" is a lowresource language pair. another critical aspect of the research into mt output is the evaluation of mt for health domains. this is especially important given the recent experience with the covid-19 pandemic where the majority of the world's population had to be confined in isolation. prior to the covid-19 pandemic, there had been calls for increased research into translation in time of crisis championed by the international network on crisis translation. 4 one of the goals of this network was "to make meaningful and effective contributions ... that enable accurate and timely translation-enabled crisis communication, with a particular focus on health-related content". in crisis, access to information in one's l1 cannot be over-emphasized, and given the speed, costeffectiveness, and easy availability of mt during crisis or in situations like the self-isolation necessitated by the coronavirus pandemic, it is safe to assume as pointed out by @xcite , that mt would be the most logical 'go-to' tool for users. this has been evidenced many times, the most recent being the massive deployment of mt during the ukraine crisis @xcite regarding the reliability of these mt tools, there have been recent claims comparing mt quality to be relatively close to human translations @xcite . however, there are still some reservations on the perceived high quality of machine translation based on some qualitative evaluations @xcite . these evaluations, which were carried out on news texts, found that translating between languages with varying word orders posed a challenge for machine translation systems. they also reported mt systems' weak error tolerance, which makes them susceptible to inaccurate translations due to minor punctuation or spelling errors that might be overlooked by a human. the inability to discern what should be emphasised or omitted; and data sparsity in low-resource language pairs and domains were also identified as hindrances to human-machine parity in machine translation. @xcite noted that for statistical-based machine translation systems, language resources matter, as they are known to perform better with language pairs that are well represented online. other studies on neural machine translation systems @xcite also agree that mt quality into low-resourced languages is relatively low. in his empirical evaluation of the quality of machine translation of 20 phrases from english into 107 languages, using google translate (gt), 5 benjamin (2019b)'s study recorded good, almost perfect outputs for high-resourced languages and lesser quality outputs for low-resourced languages. for igbo language, he reported that 47.5 per cent of the texts were accurately translated while noting that gt was able to provide fairly meaningful translations 60 per cent of the time. even so, his study was on short non-ambiguous phrases of common usage. more studies to evaluate and improve mt quality for health domain and especially into/from lowresourced languages is therefore considered imperative as the accuracy of health information received in times of crisis is vital. our aim here is to provide evaluation of specific phenomena for auto-matic identification of translation problems. as such, an integral part of this research includes highlighting problem areas that negatively affect mt quality of health-related texts from english -igbo, as this area has not yet been explored. in our preliminary studies, we have identified mwes and polysemy as the most common causes of critical errors. ## conclusion and future work in our paper, we have identified and quantified what linguistic features of english as a source language, create challenges for a machine translation system to accurately translate a medical text into igbo. our findings confirm that a medical text filled with multi-word expressions and polysemous words is not suitable for english to igbo machine translation as google translate is still unable to correctly translate such linguistic properties from english to a low-resource language like igbo. we also find that syntactic differences between the two languages do not contribute to harmful mt errors. for polysemous words, focusing on their word senses reveals an error-peak point of seven word senses, whereas all levels of domain/context numbers had a high percentage of harmful errors. as such, future work to determine if these challenges will still persist on a larger data set will be primarily on word-senses less than and equal to seven and there will be no focus on the number of contexts. token-level analysis of our data resulted in more detailed findings and would also form the guideline of further work. as part of our wider objectives, we intend to use the results from this preliminary study to develop machine learning classifiers in order to predict medical texts that could be catastrophic for mt users to machine translate from english to igbo. finally, we hope our findings can also serve as a guide to evaluate/detect causes of critical mt errors for low-resourced languages especially other niger-congo language families.