Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation (2022)

Abstract: Unlike literal expressions, idioms’ meanings do not directly follow from their parts, posing a challenge for neural machine translation (NMT). NMT models are often unable to translate idioms accurately and over-generate compositional, literal translations. In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention patterns for models with English as source language and one of seven European languages as target language. When Transformer emits a non-literal translation - i.e. identifies the expression as idiomatic - the encoder processes idioms more strongly as single lexical units compared to literal expressions. This manifests in idioms’ parts being grouped through attention and in reduced interaction between idioms and their context. In the decoder’s cross-attention, figurative inputs result in reduced attention on source-side tokens. These results suggest that Transformer’s tendency to process idioms as compositional expressions contributes to literal translations of idioms.

## introduction an idiom is a group of words of which the figurative meaning differs from the literal reading, such as "kick the bucket," which means to die, instead of physically kicking a bucket. an idiom's figurative meaning is established by convention and is typically non-compositional -i.e. the meaning cannot be computed from the meanings of the idiom's parts. idioms are challenging for the task of neural machine translation (nmt) @xcite @xcite . on the one hand, figures of speech are ubiquitous in natural language @xcite . on the other hand, idioms occur much less frequently than their parts, their meanings need to be memorised due to the non-compositionality, that are paraphrased by the model compare to attention patterns of literal pies that are translated word for word? we find (1) decreased interaction between the pie and its context, (2) increased attention within the pie, (3) decreased cross-attention between the pie and its paraphrase, (4) increased cross-attention from the paraphrase @xmath0. and they require disambiguation before translation. after all, not all potentially idiomatic expressions (pies) are figurative -e.g. consider "when i kicked the bucket, it fell over". whether pies should receive a figurative or literal translation depends on the context. yet, little is known about neural mechanisms enabling idiomatic translations and methods for improving them, other than data annotation @xcite . related work studies how idioms are represented by transformer-based language models (e.g. @xcite , but those models are not required to output a discrete representation of the idiom's meaning, which is a complicating factor for nmt models. in this work, we analyse idiom processing for pre-trained nmt transformer models @xcite for seven european languages by comparing literal and figurative occurrences of pies. the comparison can help identify mechanics that underlie neural idiom processing to pave the way for methods that improve idiomatic translations. large-scale analyses of idiom translations suffer from a lack of parallel corpora @xcite . we, therefore, use a monolingual corpus, heuristically label transformer's translations, and verify the heuristic works as intended through human evaluation, as described in §3. to understand how idioms are represented in transformer, we firstly apply interpretability techniques to measure the impact of pies on the encoder's self-attention and the cross-attention mechanisms ( §4), as well as the encoder's hidden representations ( §5). afterwards, in §6, we intervene in the models while they process idiomatic expressions to show that one can change non-compositional translations into compositional ones. the results indicate that transformer typically translates idioms in a too compositional manner, providing a word-for-word translation. analyses of attention patterns -summarised in figure 1 and hidden representations point to the encoder as the mechanism grouping components of figurative pies. increased attention within the pie is accompanied by reduced attention to context. when translating figurative pies, the decoder relies less on the encoder's output than for literal pies. these patterns are stronger for figurative pies that the model paraphrases than for sentences that receive an overly compositional translation and hold across the seven european languages. considering that a recent trend in nlp is to encourage even more compositional processing in nmt @xcite @xcite , we recommend caution. it may be beneficial to evaluate the effect of compositionality-favouring techniques on non-compositional phenomena like idioms to ensure their effect is not detrimental. ## related work this section summarises work discussing human idiom comprehension, interpretability studies for nmt, and literature about figurative language processing in transformer. idiom comprehension historically, idioms were considered @xcite . two main views (literal first and direct access) existed for how humans interpreted them. the former suggests humans attempt a compositional interpretation before considering the figurative interpretation in case of a contextual discrepancy @xcite @xcite . the latter view suggests one can immediately retrieve the non-compositional meaning @xcite . the more recent hybrid view posits that idioms are simultaneously processed as a whole -primed by a superlemma @xcite -and word for word @xcite . the processing speed and retrieval of the figurative meaning depend on the idiom's semantic properties and the context @xcite . examples of semantic properties are the conventionality and decomposability of idioms @xcite . we do not expect processes in transformer to resemble idiom processing in humans. nonetheless, this work helps us determine our focus of study on the role of the surrounding context and the extent to which idioms' parts are processed as a whole. translating pies that are used figuratively is not always straightforward. @xcite discuss strategies for human translators: (i) using an idiom from the target language of similar meaning and form, (ii) using an idiom from the target language with a similar meaning and a different form, (iii) copying the idiom to the translation, (iv) paraphrasing the idiom or (v) omitting it. in the absence of idioms with similar meanings across languages, (iv) is the most common strategy. our main focus is on literal translations (word-for-word translations), and paraphrases. interpreting transformer analyses of transformer for nmt studied the encoder's hidden representations and self-attention mechanism (e.g. raganato @xcite @xcite , the cross-attention (e.g. @xcite and the decoder (e.g. @xcite . the encoder is particularly important for the contextualisation of tokens from the source sentence; it acts as a feature extractor @xcite . the encoder's bottom three layers better represent low-level syntactic features, whereas the top three layers better capture semantic features @xcite . as a result, one would expect the representations in higher layers to be more representative of idiomaticity. idioms are a specific kind of ambiguity, and whether a word is ambiguous can accurately be predicted from the encoder's hidden representations, as shown by @xcite for ambiguous nouns. transformer's cross-attention is not crucial for disambiguating word senses @xcite , but the encoder's self-attention does reflect ambiguity through more distributed attention for ambiguous nouns @xcite . ## method we use transformer models @xcite with english as the source language and one of seven languages as the target language (dutch, german, swedish, danish, french, italian, span-ish). 1 transformer contains encoder and decoder networks with six self-attention layers each and eight heads per attention mechanism. the models are pre-trained by @xcite with the marian-mt framework @xcite on a collection of corpora (opus) @xcite ). 2 we extract hidden states and attention patterns for sentences with pies. the analyses presented are detailed for dutch, after which we explain how the results for the other languages compare to dutch. 3 parallel pie corpora are rare, exist for a handful of languages only, and are limited in size @xcite . rather than rely on a small parallel corpus, we use the largest corpus of english pies to date and annotate the translations heuristically. this section provides corpus statistics and discusses the heuristic annotation method. ## attention we now turn to comparing how literal and figurative pies are processed by transformer. whether a pie is figurative depends on the context -e.g. compare "in culinary school, i felt at sea" to "the sailors were at sea". within transformer, contextualisation of input tokens is achieved through the attention mechanisms, which is why they are expected to combine the representations of the idioms' tokens and embed the idiom in its context. this section discusses the impact of pies on the encoder's selfattention and the encoder-decoder cross-attention. to assert that the conclusions drawn in this section are not simply explained by shallow statistics of the data used, we recompute the results in appendix c for (1) a data subset excluding variations of pies' standard surface forms, (2) a data subset that includes pies that appear in both figurative and literal contexts, (3) a data subset that controls for the number of tokens within a pie. qualitatively, these results lead to the same findings. attention within the pie for the en-nl transformer, figure 3a visualises the distribution of attention weights in the encoder's self-attention mechanism for incoming weights to one noun contained in the pie from the remaining pie tokens. throughout the figures in the paper, we refer to the subset of sentences that have a figurative pie and a paraphrased translation as 'fig-par'. the subset of sentences with a literal pie and a word-for-word translation are indicated by 'lit-wfw'. we compare those two subsets, as well as all instances of figurative pies ('fig') to all instances of literal pies ('lit') using the labels from the magpie dataset. overall, there is increased attention in figurative occurrences of pies compared to literal instances. this difference is amplified for the subset of figurative pies yielding paraphrased translations. this pattern is consistent for all languages, as is displayed in figure 3d that presents the difference between the mean attention weights of the figurative, paraphrased instances, and the mean weights of the literal instances translated word for word. 7 in other words, figurative pies are grouped more strongly than their literal counterparts. ## hidden representations within transformer, the encoder's upper layers have previously been found to encode semantic information (e.g. @xcite . pies' hidden states are expected to transform over layers due to contextualisation, and become increasingly more indicative of figurativeness. this section focuses on the impact of pies on the hidden states of transformer's encoder. we firstly discuss how much these hidden states change between layers. secondly, we measure the influence of a token by masking it out in the attention and analysing the degree of change in the hidden representations of its neighbouring tokens. this analysis is performed to consolidate findings from §4, since the extent to which attention can explain model behaviour is a topic of debate @xcite . ## (amnesic) probing for figurativeness the previous analyses compared the hidden states for figurative and literal pies, but do not use these labels, otherwise. we now train logistic regression probing classifiers @xcite to predict the label from hidden representations. the probes' inputs are the hidden states of pie tokens, and the f 1 -scores are averaged over five folds. all samples from one pie are in the same fold, such that the classifier is evaluated on pies that were absent from its training data. the results (figure 7 ) indicate figurativeness can be predicted from these encodings, with performance increasing until the top layer for all languages. f 1 -scores for the embeddings already exceed a random baseline, indicating some idioms are recognisable independent of context. finally, we use probing classifiers to change models' pie translations through amnesic probing @xcite : removing features from hidden states with iterative null-space projection (inlp) @xcite and measuring the influence of these interventions. inlp trains k classifiers to predict a property from vectors. after training probe i, parametrised by w i , the vectors are projected onto the nullspace of w i . the projection matrix of the intersection of all k null spaces can then remove features found by these classifiers. using inlp, we train 50 classifiers to distinguish figurative pies that will be paraphrased from those to be translated word for word. afterwards, we run the previously paraphrased pie occurrences through the model while removing information from the pie's hidden states using inlp -i.e. information that could be captured by linear classifiers, which need not be the only features relevant to idiomatic translations. per idiom, we record the percentage of translations that are no longer paraphrased. we report the scores for idioms from four folds and bleu scores comparing translations that changed label before and after inlp. a fifth fold is used for parameter estimation (appendix f). figure 8 provides example translations before and after the application of inlp, while indicating how the attention on the underlined noun changes. generally, the attention on that noun reduces for tokens other than itself. in summary, when applying inlp to hidden states, the attention patterns resemble patterns for literal tokens more, confirming a causal connection between the model paraphrasing figurative pies and the attention. however, amnesic probing cannot change the paraphrases for all idioms; thus, figurativeness is not merely linearly encoded in the hidden states. the probing accuracies differed across layers and suggested figurativeness is more easily detectable in higher layers, which is in line with the changes across layers observed in §5. ## conclusion idioms are challenging for nmt models that often generate overly compositional idiom translations. to understand why this occurs, we analysed idiom processing in transformer, using an english idiom corpus and heuristically labelled translations in seven target languages. we compared hidden states and attention patterns for figurative and literal pies. in the encoder, figurative pies are grouped more strongly as one lexical unit than literal instances and interact less with their context. the effect is stronger for paraphrased translations, suggesting that capturing idioms as single units and translating them in a stand-alone manner aids idiom processing. this finding agrees with results from @xcite , who ascertain that encoding an idiom as one word improves translations. it also agrees with the inlp application causing more compositional translations whilst changing the attention. by relying less on the encoder's output, the decoder determines the meaning of figurative pies more independently than for literal ones. to improve idiomatic translations, future work could use these insights to make architectural changes to improve the grouping of idioms as single units by training specific attention heads to capture multiword expressions or by penalising overly compositional translations in the training objective. although we learnt about mechanics involved in idiomatic translations, the vast majority of translations was still word for word, indicating that noncompositional processing does not emerge well (enough) in transformer. paradoxically, a recent trend is to encourage more compositional processing in nmt @xcite @xcite . we recommend caution since this inductive bias may harm idiom translations. it may be beneficial to evaluate the effect of compositionality-favouring techniques on noncompositional phenomena to ensure their effect is not detrimental.