Dictionary-Aided Translation for Handling Multi-Word Expressions in Low-Resource Languages (2024)

Abstract: Multi-word expressions (MWEs) present unique challenges in natural language processing (NLP), particularly within the context of translation systems, due to their inherent scarcity, non-compositional nature, and other distinct lexical and morphosyntactic characteristics, issues that are exacerbated in low-resource settings.In this study, we elucidate and attempt to address these challenges by leveraging a substantial corpus of human-annotated Greek MWEs. To address the complexity of translating such phrases, we propose a novel method leveraging an available out-of-context lexicon.We assess the translation capabilities of current state-of-the-art systems on this task, employing both automated metrics and human evaluators.We find that by using our method when applicable, the performance of current systems can be significantly improved, however these models are still unable to produce translations comparable to those of a human speaker.

## introduction multi-word expressions (mwes) are defined as combinations of at least two words which exhibit lexical, morphological, syntactic, semantic or statistical idiomaticity @xcite . processing mwes has long been considered one of the most challenging tasks in natural language processing (nlp) @xcite , especially when it concerns translation systems. despite neural models outclassing more traditional nlp techniques in many areas, the distinct nature of mwes makes it so that the generalizing abilities inherent in these approaches cannot significantly increase performance. there is also currently a significant lack of datasets and benchmarks in this domain, especially for low-resource languages. faithful automatic translation of mwes is particularly important in improving the performance of general translation systems, given that mwes make up a significant amount (estimates range from 11% to 40%) of the vocabulary in any piece of text @xcite . in this work, we use a large corpus of humanannotated mwes in greek and we measure the translation ability of current state-of-the-art systems on sentences containing these mwes, both automatically and with the help of human evaluators. we focus on two tasks; standard "unaided" translation, where the model is only provided with the source sentence, and an "aided" version, where we provide additional guidance on the translation of the mwe used in the source sentence, taking advantage of the available lexicon. an example of both tasks is shown in figure 1 . in short, we make the following contributions: ## related work previous research has explored the complexities of processing mwes in various tasks, including mwe detection @xcite as well as translation-specific settings @xcite , such as in contrastive examples or metaphors, highlighting the need for specialized approaches. while neural models have excelled in many nlp tasks, their performance with mwes remains suboptimal. in certain sub-areas, such as detection, traditional non-neural statistical methods remain remarkably competitive @xcite . this is theorized to be due to mwes having a lexical quality, meaning that they behave similarly to out-of-vocabulary words when they do not appear in their idiomatic usage in the training data @xcite . this problem is therefore exacerbated disproportionately more for lower-resource languages, such as greek. there has also been research in the domain of dictionary-aided neural machine translation in a general setting, concerning single one-word lemmas @xcite . older methods have also attempted to leverage mwe dictionaries by incorporating them into the non-neural, statistical translation pipeline @xcite , while newer ones have tried adding the dictionary to the training data of neural models @xcite . our work borrows ideas from the efforts towards terminology translation @xcite , which usually incorporates hard @xcite , usually through constrained decoding @xcite , or soft constraints (bergmanis and pinnis, 2021) during decoding. similar techniques have also been employed for translating rare words @xcite and terminology in specific domains @xcite . in contrast, our method leverages the prompting capabilities of large language models to imbue them with information from a mwe dictionary (somewhat similar to the soft constraints) while still taking advantage of their generalizing abilities. ## methodology our evaluation focuses on two tasks: standard ("unaided") and "aided" translation. ## experiments we use two state-of-the-art models for our experiments. the first is no language left behind (nllb; @xcite , a state-of-theart multilingual model specializing in low-resource languages. we specifically use the 600m parameter version due to computational resource constraints. the second is gpt-4-0613 @xcite , a large language model operating as a chatbot. we then evaluate the various translations, both in absolute and comparative quality, in different ways, both manually and automatically. unaided translation under the unaided translation setting, we benchmark both models-nllb and gpt-4. the nllb model was provided simply with the source sentence (in greek), the source language (greek), and the target language (english). gpt-4 was provided with the following prompt: can you translate the greek text "[source sentence]" into english? aided translation our aided translation proposed method relies on the prompting capabilities of llms, hence it is only applicable to gpt-4 and not to nllb. we now provide gpt-4 with the following prompt: can you translate the greek text "[source sentence]" into english? for context, the greek multi-word expression "[mwe]" can mean "[english meaning]", "[possible other english meaning]", ..., or "[possible other english meaning]". ## results our human evaluations indicated that the "aided" method using gpt performed significantly better than the gpt-based "unaided" one, which in turn outperforms the nllb-based "unaided" method. quality estimation metrics generally agreed with the human evaluations when it comes to this order of quality. in model human eval unaided gpt 3.4/5 aided gpt 3.8/5 additionally, when asked to explicitly choose the best between the "aided" and "unaided" translations, humans judged "aided" gpt outputs to be better than the "unaided" (78% of the time) and than the nllb (95% of the time). in in we also attempt to better compare the two baselines, using high-quality (as judged by humans) "aided" gpt outputs as a reference. targetted versus random sentences using any random translation examples in the prompt has been observed to improve translation quality in certain cases, because they can help the llm focus on the translation task @xcite all human scores are normalised as explained in appendix b. our experimental results ( §5) show that, at least for the idion dataset, our proposed "aided" method for mwe translation through gpt-4 by providing a pre-contextualized lexicon significantly outperforms the naive "unaided" gpt-4 usage, with both producing better results when compared to stateof-the-art mt models (nllb). this is significant as the latter is tailored specifically for use cases similar to our experiment, namely translation of lower-resource languages. however, under all settings, current models seem to underperform in mwe translation. as established previously, a lexicon is most likely required to achieve human-level translations, but having a context-agnostic lexicon and simply providing it to the model, even if it has been generated by humans and correctly mapped to each instance, appears to not be enough to reach this level, although it does significantly improve performance. we hypothesize that english speakers seem to prefer the same system as english-greek bilinguals because the model avoids awkward word-to-word translations. so while they cannot tell whether the translation is adequate, they can penalize cases where a literal translation is obviously nonsensical. moreover, it seems that gpt, on accessing its own evaluation, might amend its answer. we therefore also attempted further improving gpt-4's output by giving it the chance to evaluate its previous response and possibly change it (details provided in appendix a). it did not seem to significantly help, as it was found to rate itself highly and barely, if at all, change its translation. future work we believe that it would be worthwhile to professionally translate our dataset and extend it to more languages, thus providing the necessary context in bilingual settings, making it useful for training nmt models on mwes. 5 moreover, since current automatic evaluation methods present several shortcomings, our translated dataset could generally be used for benchmarking the mt abilities when it comes to mwes. in the era of llms, perhaps we should reevaluate how we construct dictionaries. llms learn more effectively from more context, so rather than providing simple definitions with minimal disambiguating example usages, lexicographers could focus on providing more and longer examples. ## limitations human and automatic evaluations seem to generally agree with each other, but we acknowledge that, due to the lexical nature of our task, without a reference translation, quality estimation models (such as the unbabel one we use) cannot be relied upon to offer trustworthy results. we also acknowledge that since the evaluators are all primarily native greek speakers, they may have been influenced in their evaluation of wordfor-word translations into english, as they would have made sense to them and it could occasionally have been difficult to know how natural they would sound to native speakers. we attempted to mitigate this issue with our english-speakers evaluation experiment, but ideally one would employ native english speakers that also understand greek. ## ethics statement we believe that our work does not introduce any significant additional risks other than those inherent in the models used. we have obtained permission from all annotators to publish the data they produced in the context of this paper. the annotators were volunteer co-authors and co-workers, and no monetary compensation was provided for their involvement. the content of idion is available under a cc-by-nc license, in xml format. their usage in this project is therefore consistent with its intended use. all models we use come with permissive licenses, at least when it comes to research.