Food for Thought: How can we exploit contextual embeddings in the translation of idiomatic expressions? (2022)

Abstract: Idiomatic expressions (or idioms) are phrases where the meaning of the phrase cannot be determined from the meaning of the individual words in the expression. Translating idioms between languages is therefore a challenging task. Transformer models based on contextual embeddings have advanced the state-of-the-art across many domains in the field of natural language processing. While research using transformers has advanced both idiom detection as well as idiom disambiguation, idiom translation has not seen a similar advancement. In this work, we investigate two approaches to fine-tuning a pretrained Text-to-Text Transfer Transformer (T5) model to perform idiom translation from English to German. The first approach directly translates English idiom-containing sentences to German, while the second is underpinned by idiom paraphrasing, firstly paraphrasing English idiomatic expressions to their simplified English versions before translating them to German. Results of our evaluation show that each of the approaches is able to generate adequate translations.

## introduction in the past decade, we have seen an increase in the accuracy of machine translation (mt) approaches @xcite . some of the contributing factors to this increase is the introduction of the attention mechanism and contextual embedding models @xcite , as well as the wider availability of datasets. according to @xcite however, the same increase in accuracy has not been achieved for idiom translation. idioms are defined as "a group of words established by usage as having a meaning not deducible from those of the individual words" (university of oxford, 2022). since datasets used for mt are, in general, not rich in idioms @xcite @xcite @xcite , mt models can suffer from this by not being able to distinguish between an idiom and an expression that can be interpreted literally. this can result in a wrong or meaningless translation, as can be seen in figure 1 . however, there are also idioms which can be interpreted literally, depending on the context in which it was used. for instance, the idiom "breaking the ice" could have an idiomatic meaning "to get a conversation started", but its literal meaning is also valid, e.g., in the context of someone breaking ice cubes for a cocktail. such idiomatic expressions make it even more challenging for models to correctly translate sentences containing them. to further complicate the issue, some multi-word expressions (mwes) such as "to pass on" and "to come out", are often used in their idiomatic sense but can also be used in their literal sense. with the emergence of the attention mechanism @xcite , transformer models and contextual embeddings @xcite came the rapid advancement of the state of the art in many nlp tasks, e.g., question answering and machine translation @xcite . in this work, we aim to improve the translation of idiomatic expressions by employing contextual embeddings. we focus on investigating two different approaches for fine-tuning transformer models for the translation of sentences containing idiomatic expressions. parallel corpora containing idiomatic expressions are scarce @xcite @xcite @xcite , but owing to the availability of a parallel corpus of idiom-containing english sentences and their corresponding german translations @xcite , we have chosen english and german as our source and target languages, respectively. the first approach utilises this dataset for idiom-to-idiom translation, i.e., translation of an idiom-containing sentence in english to its equivalent idiom-containing sentence in german. the second approach, meanwhile, is based on idiom paraphrasing, i.e., conversion of an english idiom-containing sentence to its paraphrase, followed by translation of the latter to german. on top of assessing the performance of these approaches based on evaluation metrics, we designed a strategy for human-based evaluation to determine: (1) how fluent their translations are in german, and (2) how well their translations preserve the meaning of source sentences. to the best of our knowledge, ours is the first work to investigate the extent to which transformer models, specifically the text-to-text transfer transformer (t5) kind @xcite , can translate idiom-containing sentences from one language to another. based on the transformer encoder-decoder architecture @xcite , t5 provides a unified framework for casting many nlp problems (e.g., text classification, question answering) as a sequence-tosequence learning task, and thus lends itself well to the problem of translating idiomatic expressions. ## related work neural machine translation (nmt) models @xcite and statistical machine translation (smt) models @xcite have shown difficulty in translating idiomatic expressions @xcite . the meaning of an idiom is generally different from the joint meaning of the words composing it, and therefore translation models tend to make errors from the literal translation of individual words. in recent years, a number of transformer models, e.g., bert @xcite , bart @xcite and t5 @xcite , have been successfully applied to a wide range of natural language processing tasks. the advantage of these models is that, for any word (token), they use a contextual embedding representation which is based not only on the word itself, but also on its context (i.e., surrounding words to the left and right). they have achieved ground-breaking results in almost every nlp task @xcite . context is key for the comprehension of idiomatic expressions, hence such contextual embeddings could potentially be helpful in understanding them. while the use of transformer models to understand idiomatic expressions has been explored in several papers @xcite @xcite @xcite , very little research has been done on idiom translation based on these models. there are multiple tasks involved in the translation of idiom-containing sentences. the first one involves the identification of idiomatic expressions within a sentence @xcite . @xcite , for instance, showed that transformer models can be used to successfully identify idiomatic expressions. idiom identification is followed by sense disambiguation, which involves determining whether an idiom is used literally or idiomatically in the containing sentence @xcite @xcite . transformers have also advanced the state of the art in this task @xcite . a further task is the translation or paraphrasing of idioms, depending on the intended application. much research has been shown to attempt paraphrasing idioms to replace them with their literal meaning @xcite @xcite . the work of @xcite demonstrated how bart can be used for this purpose. with respect to datasets for idiom paraphrasing, there are various mono-lingual english idiom datasets @xcite @xcite . among these datasets, the pie dataset @xcite stands out, as it contains both idiomatic expressions and their nonidiomatic counterparts. when it comes to translation to another language rather than paraphrasing within a single language, @xcite employed smt to firstly substitute idioms with their simplified meanings before translation to the target language, after which the translated expressions were substituted with idioms in the target language. little research can be found when it comes to the direct translation of idiomatic expressions from a source to a target language. a major contributing factor to this could be the scarcity of paral-lel corpora suitable for this task. a few papers on translation introduced their own datasets. an example is the work of @xcite where the authors introduced a parallel corpus of idiomcontaining sentences in seven indian languages and english, on which nmt and smt models were trained. @xcite similarly created an english-german parallel corpus of sentences with idiomatic expressions, and evaluated the performance of nmt and smt models based on it. other corpora that support the development of idiom translation include a russian-english @xcite and a chinese-english dataset @xcite . ## methodology the focus of this work is idiom translation. we thus consider idiom identification as outside of our scope, and make the assumption that input sentences contain idiomatic expressions. furthermore, in some of our models (described below), the idiomatic expression itself is included as part of the input. below, we describe each of the two approaches we propose for idiom translation, one underpinned by idiom-to-idiom translation from the source to target language, and the other based on idiom paraphrasing within the source language followed by translation to the target language. in both cases, a t5 model was fine-tuned for a sequence-tosequence learning task, where the input is provided in the form of a sequence of tokens and the model produces another sequence as its output. the task that the model needs to learn, is defined by prepending a prefix to the input sequence. t5 models come in different sizes. in this work, we employed the t5-small implementation 1 which has around 60 million parameters and yet is feasible to train with limited computational resources. it comes pretrained for language modelling based on the colossal clean crawled corpus @xcite and fine-tuned for a number of downstream nlp tasks including translation. ## evaluation and results in order to evaluate our approaches, both automatic and human-based evaluation were conducted. below, we first discuss the results of automatically evaluating each of the idiom-to-idiom translation and idiom paraphrasing models, followed by the results of human-based evaluation. ## discussion with respect to the first approach underpinned by idiom-to-idiom translation, our results showed that the best-performing model is the one that was based on continuing to fine-tune t5-small for the predefined english-to-german translation task. this shows that a t5 model that was fine-tuned for the predefined translation task, is better at translating english idiom-containing sentences to their idiomcontaining german counterparts, compared to a model that was trained for a completely new idiom translation task. this is unsurprising considering that t5 was fine-tuned for general translation on the wmt 2014 english-german dataset with 4.5 million sentence pairs @xcite , while the cleansed idiomtranslationds dataset that we used to fine-tune t5 for the new, custom idiom translation task, includes only 1733 pairs. when it comes to the second approach which is based on a pipeline of idiom paraphrasing and translation models, our results demonstrate that finetuning t5-small-tapaco (a t5 model that had already been trained for general english paraphrasing) for our newly proposed custom paraphrasing task, leads to improved performance. moreover, when this paraphrasing model is combined with the original t5 model for general english-to-german translation, better performance on the translation task is obtained, in comparison with using only the original t5 model. although the two approaches are not directly comparable with each other (i.e., the first approach is aimed at keeping the idiom in a german translation while the second one is aimed at generating a german translation of a non-idiomatic english paraphrase), our human-based evaluation shows that the first approach-the one based on idiom-to-idiom translation-seems to produce outputs which are marginally better that those of the second. this can be expected: as the second approach is based on a pipeline of paraphrasing and translation sub-tasks, any errors from the paraphrasing model would have been propagated to the translation model, affecting the quality of the final outputs. this is an issue that does not apply to the first approach since it performs direct translation. ## conclusions and future work in this paper, we demonstrate how t5 models can be exploited in idiom translation: by fine-tuning them for idiom-to-idiom translation (first approach) and idiom paraphrasing (second approach). on the one hand, automatic evaluation showed that con-tinuing to fine-tune the original t5 model for the predefined translation task on an idiom translation dataset, yielded optimal performance for idiom-toidiom translation. on the other hand, fine-tuning a t5 model that had already been trained on a general paraphrasing task, for a custom idiom paraphrasing task, led to the best performance for idiom paraphrasing. combining the said paraphrasing model with the original t5 model for general translation, resulted in improved results for idiom translation, compared with using just the latter. human-based evaluation showed that both approaches produce translations of adequate quality. to further advance research in idiom translation, we propose possible directions that can be pursued in the future. firstly, a high-quality dataset with a much larger number of idiom-containing sentence pairs can be developed to facilitate better finetuning of t5 models for a custom idiom-to-idiom translation task. moreover, it would be beneficial to create one dataset that can support the development of both idiom-to-idiom translation and idiom paraphrasing approaches. for instance, one can enrich the idiomtranslationds dataset by including paraphrases of idioms in english and german. furthermore, our work has highlighted the fact that idiom translation datasets are scarce. when more such datasets become available, one can assess the extent to which our approaches can be applied to other language pairs. to mitigate the current lack of large datasets for idiom translation, one could cast the idiom translation problem as a prompt-based learning task @xcite : a framework that makes it possible to apply pretrained language models to downstream tasks (such as translation) without the need for large amounts of data for fine-tuning.