Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting (2023)

Abstract: Idioms are common in everyday language, but often pose a challenge to translators because their meanings do not follow from the meanings of their parts. Despite significant advances, machine translation systems still struggle to translate idiomatic expressions. We provide a simple characterization of idiomatic translation and related issues. This allows us to conduct a synthetic experiment revealing a tipping point at which transformer-based machine translation models correctly default to idiomatic translations. To expand multilingual resources, we compile a dataset of ~4k natural sentences containing idiomatic expressions in French, Finnish, and Japanese. To improve translation of natural idioms, we introduce two straightforward yet effective techniques: the strategic upweighting of training loss on potentially idiomatic sentences, and using retrieval-augmented models. This not only improves the accuracy of a strong pretrained MT model on idiomatic sentences by up to 13% in absolute accuracy, but also holds potential benefits for non-idiomatic sentences.

## introduction an idiom is a conventionalized expression in which the intended meaning differs from its literal translation. the translation of idioms has remained a problem for state-of-the-art research and commercial translation systems, as idioms tend to be translated literally @xcite @xcite . failure to translate these expressions correctly may lead to incomprehensible translations, particularly in literary text @xcite . to illustrate the difficulty of understanding mistranslated idioms, we present mistranslations from commercial systems in we first frame the general problem of noncompositional translation, which encompasses the translation of idioms and other multi-word expressions that cannot be translated word-for-word ( §2). we then perform synthetic experiments in a very simple case, finding that transformer-based machine translation models generally translate wordfor-word until a proportional threshold of sentences contain non-compositional expressions, at which point the translations flip to being correct ( §4.1). we evaluate translations by commercial models in three natural languages, and find a drop in performance on idiomatic sentences and stronger performance on more common idioms ( §4.2). we hypothesize that this may reflect similar trends as exist in processing other long-tail phenomena, and similar tactics to those used to deal with rare phenomena may work @xcite . with this intuition, we improve the idiomatic translations generated by a strong pretrained machine translation model, ∆lm @xcite , without harming the translation quality of literal expressions. to contribute resources toward documenting idioms and improving their translation cross-linguistically, we create a dataset of sentences containing idiomatic expressions in three languages (french (fr), finnish (fi) and japanese (ja) ( §3). we propose two simple but effective ways to improve translation of idioms, namely upweighting training loss on potentially idiomatic sentences and retrieval augmentation ( §5). we find that this can improve the idiomatic translation abilities of the model significantly, by an average of 10.4% in absolute accuracy ( §7.1). moreover, this does not harm translation of sentences where the literal sense of the idiom is used, and it improves translation of out-of-distribution sentences in french and finnish as well. we perform human evaluation and error analysis, and find that the rate of severe semantic errors is reduced by an average of 7.52% absolute accuracy ( §7.2). the ultimate aim for machine translation is to ensure accessibility for all texts. this requires addressing idiomatic phrases, culturally-informed language, and complex semantics. we demonstrate the potential for enhancing idiom translation using existing resources. ## idioms and data collection we can use the formal definition from the previous section to generate synthetic data for experiments. however, we ultimately want to improve translation of real idioms. to do so, we collect a dataset of natural sentences to evaluate commercial systems and the model we seek to improve. although a large corpus of potentially idiomatic expressions exists in english @xcite , there are no readily accessible equivalents in other languages. therefore, we collected idioms in french, finnish, and japanese from languagelearning sites, listed in appendix b. these languages were chosen for phylogenetic diversity, and due to availability of commercial translation systems. in total, there were 148 french idioms collected, 92 in finnish, and 1336 in japanese. to collect sentences containing these idioms, we matched on lemmatized forms from the 2018 version of opensubtitles @xcite , where lemmatization was performed with stanza @xcite . in total, there were 85632 french sentences containing potentially idiomatic expressions, 51811 finnish sentences, and 23018 japanese sentences. to filter out unaligned sentences, we scored each source and reference sentence using comet-qe @xcite and removed the bottom 10% of each language's sentences by comet-qe scores. some idioms have a plausible literal meaning (such as "kick the bucket" to mean kicking a physical bucket). to make sure that all examples in 4 x denotes string concatenation given the word order of language x, i.e. if the word order is svo, the tokens belonging to the subject should be placed in front of the tokens belonging to the verb, and so on. the idiomatic test set were actually idiomatic, we sorted sentences into an idiomatic test set where the idiomatic meaning of a phrase was used (e.g. "to die") and a literal test set, where the literal meaning of the phrase was used (e.g. kicking a physical bucket). the first 100 examples containing each idiom's lemmatized form were collected, and up to the first 3 (for japanese) or 5 (for finnish and french) literal and figurative examples in this set were collected to create the test set. this was to avoid dominance of very common idioms in the test set. this created two test sets related to the idiom list for each language, the idiomatic and literal test sets. to validate these judgments, we hired native annotators in french and finnish. they were presented with examples from the final literal and idiomatic test sets in a shuffled order, and asked to label them with idiomatic, literal, or n/a labels if they didn't think it was an instance of either. agreement @xcite ) in both cases was moderately high (french α = 0.5754, finnish α = 0.6454). details can be found in appendix d. finally, we collect two random test sets, one which is in-domain and another which is out-ofdomain. for the in-domain test set, we simply select sentences from the development set of open-subtitles (see subsection 6.2 for details on our split of opensubtitles). for the out-of-domain test set, we use the ted talks corpus @xcite . this is to ensure that translation quality of other, unrelated sentences is not impacted by any modifications meant to improve translation of idioms. topics discussed and vocabulary used in ted talks may be slightly different from what is discussed in movies or tv shows, so training the model on opensubtitles and testing on ted talks allows us to evaluate model generalization. for both test sets, to control for translation length as a source of difficulty, sentences were length-matched on the target side with corresponding sentences in the idiomatic set. this created the random set, which is the same size as the idiomatic test set. all three test sets are summarized in the source language in these experiments was composed of tokens 0 through 9, @xmath0, 1, 2, ..., 9}. the target language was produced by adding 10 to each token, @xmath1, ..., 19}. the translation rule was to add 10 to the value of each token in the source language, e.g. 0 → 10, 1 → 11. we add a single non-compositional rule that doesn't follow this trend, 0 1 → 12 (rather than 0 1 → 10 11). we limited the maximum sequence length to 6 tokens. we generated synthetic training corpora of several sizes containing different numbers of occurrences of the non-compositional rule 0 1 → 12 . the number of training sentences ranged from 100k to 10m, while the number of noncompositional occurrences ranged from 10 to 1m. we examined two informativity conditions, corresponding to the case where the context provides no information (tokens are randomized around the non-compositional expression), and the context being perfectly informative. the perfect informativity condition was achieved by adding the canary token "11" to the source vocabulary, and only inserting this token prior to the non-compositional pattern "0 1". we experimented with three different transformer sizes @xcite , each of which had a hidden dimension and embedding size of 512, as well as 16 attention heads. only the number of encoder and decoder layers varied, such that the small transformer had 3 encoder and decoder layers, the medium transformer 8, and the large transformer 16. we fix the number of epochs for the small, medium and large models to respectively be 10, 20, and 30 in the non-informative case and 15, 15 and 25 in the informative case. 5 further training details can be found in appendix a. although this may seem like a simple task, we found it surprisingly difficult for models to learn this non-compositional pattern. results in each setting, averaged across 5 random seeds, are presented in figure 1 . especially for the small model, there is a sharp gradation from translating none of the non-compositional expressions correctly to translating them all correctly, which occurs when roughly 10% of training data contains a non-compositional pattern. a similar trend exists for larger models, but the threshold is less distinct. this corroborates the tendency for transformers to translate non-compositional phrases literally @xcite . comparatively less data is required when the context is informative, but the trends remain similar to the non-informative case. as model size and corpus size increase, the rate of correct translations for non-compositional examples actually drops, contrary to expectation. it is unlikely that any individual idioms occur in 10% of sentences in natural language. due to the highly regular translation rules in this synthetic language, there may be a stronger bias toward translating compositionally in this experiment. however, we gain the intuition that idioms can be translated effectively if they appear frequently, and that clear context clues reduce data required. ## methods to improve non-compositional translation upweighting here refers to sentence-level upweighting, where there is a set of sentences a that we'd like to upweight with a weight coefficient α. in this case, a would be potentially idiomatic sentences. we keep all other parameters for training the same as in the base model. during generation, a probability distribution over next words can be computed based on the retrieved next words and the distance of their keys to the current context. a parameter λ controls interpolation between the distribution over next words predicted by the base model, and the distribution predicted by the retrieved k neighbours. foot_3 p(y we also combine loss weighting with knn-mt, where a model is trained with sentence upweighting and interpolated with a datastore based on representations from the upweight-trained model. intuitively, these methods make sense to use for idiom translation -we have previously seen that one problem with non-compositional phrases may simply be their rarity. upweighting training examples that contain idioms may help with underrepresentation. furthermore, retrieving similar examples may find occurrences of the same idiom which were translated correctly. ## related work recent work has raised the issue of idiom handling in mt @xcite . there is historical recognition of the problem, including of multi-word expressions @xcite @xcite . this has historically motivated example-based machine translation @xcite . similar motivations underlie the use of knn-mt. however, neural models may already be capable of translating idiomatic phrases if they appear often enough in training data. other works focus on data augmentation and creating new data resources @xcite @xcite . a related task is detection of conventionalized metaphors @xcite . automatic identification of idiomatic phrases, as well as data augmentation are promising avenues to improve performance in lower-resource languages. instance weighting has been explored previously in the mt literature, but has been mostly explored in the context of domain adaptation, rather than being used to improve translations of rare or noncompositional phrases in the same domain @xcite . idiomatic phrases are a prototypical case of phrases that need to be memorized @xcite . many also occur infrequently in training data, which may make it difficult for transformerbased models to translate them @xcite . this can be mitigated, as we have shown in this paper. however, more work is needed to effec-tively learn idioms and other infrequent linguistic elements with few repetitions. ## conclusion we highlight the challenge idiomatic expressions pose to machine translation systems and provide simple solutions to improve performance. through synthetic experiments, we identify a threshold at which transformer-based models correctly default to idiomatic translations. we develop a dataset of sentences containing idiomatic expressions in french, finnish, and japanese, and introduce two techniques -upweighting training loss on potentially idiomatic sentences and augmenting models with knn-mt -which enhance the idiomatic translation accuracy of a strong model, while offering potential benefits for non-idiomatic sentences. future research could extend these techniques to additional languages, and explore their effectiveness in dealing with other long-tail phenomena. we hope that this work contributes toward increasing the intelligibility of translations containing idioms or set phrases. ultimately, for machine translation to be useful for everyone without causing misunderstandings, "last mile" problems involving cultural knowledge, long-tail phenomena, and complex semantic evaluation should be taken into account.