It’s Easier to Translate out of E nglish than into it: M easuring Neural Translation Difficulty by Cross-Mutual Information (2020)

Abstract: The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty .

## introduction machine translation (mt) is one of the core research areas in natural language processing. current state-of-the-art mt systems are based on neural networks @xcite , which generally surpass phrase-based systems @xcite in a variety of domains and languages @xcite @xcite @xcite . using phrase-based mt systems, various controlled studies to understand where the translation difficulties lie for different language pairs were conducted @xcite . however, comparable studies have yet to be performed for neural machine translation (nmt). as a result, it is still unclear whether all translation directions are equally easy (or hard) to model for nmt. this paper hence aims at filling this gap: ceteris paribus, xmi, our proposed metric, is no longer purely a symmetric measure of language, but now an asymmetric measure that mostly highlights models' shortcomings. in response, we propose cross-mutual information (xmi), a new metric towards cross-linguistic comparability in nmt. in contrast to bleu, this information-theoretic quantity no longer explicitly depends on language, model, and tokenization choices. it does, however, require that the models under consideration are probabilistic. as an initial starting point, we perform a case study with a controlled experiment on 21 european languages. our analysis showcases xmi's potential for shedding light on the difficulties of translation as an effect of the properties of the source or target language. we also perform a correlation analysis in an attempt to further explain our findings. here, in contrast to the general wisdom, we find no significant evidence that translating into a morphologically rich language is harder than translating into a morphologically impoverished one. in fact, the only significant correlate of mt difficulty we find is source-side type-token ratio. ## cross-linguistic comparability through likelihoods, not bleu human evaluation will always be the gold standard of mt evaluation. however, it is both timeconsuming and expensive to perform. to help researchers and practitioners quickly deploy and evaluate new systems, automatic metrics that correlate fairly well with human evaluations have been proposed over the years @xcite @xcite . bleu @xcite , however, has remained the most common metric to report the performance of mt systems. bleu is a precisionbased metric: a bleu score is proportional to the geometric average of the number of n-grams in the candidate translation that also appear in the reference translation for 1 ≤ @xmath0. 1 in the context of our study, we take issue with two shortcomings of bleu scores that prevent a cross-linguistically comparable study. first, it is not possible to directly compare bleu scores across languages because different languages might express the same meaning with a very different number of words. for instance, agglutinative languages like turkish often use a single word to express what other languages have periphrastic constructions for. to be concrete, the expression "i will have been programming" is five words in en-1 bleu also corrects for reference coverage and includes a length penalty, but we focus on the high-level picture. in this work, we address this particular shortcoming by considering an information-theoretic evaluation. formally, let v s and v t be source-and target-language vocabularies, respectively. let s and t be source-and target-sentence-valued random variables for languages s and t, respectively; then s and t respectively range over v * s and v * t . these random variables s and t are distributed according to some true, unknown probability distribution p. the cross-entropy between the true distribution p and a probabilistic neural translation model q mt (t | s) is defined as: since we do not know p, we cannot compute eq. ( 1 ). however, given a held-out data set of sentence pairs {(s (i) , t (i) )} n @xmath1, we can approximate the true cross-entropy as follows: in the limit as @xmath2, eq. ( 2 ) converges to eq. ( 1 ). we emphasize that this evaluation does not rely on language tokenization provided that the model q mt does not @xcite . while common in the evaluation of language models, cross-entropy evaluation has been eschewed in machine translation research since (i) not all mt models are probabilistic and (ii) we are often interested in measuring the quality of the candidate translation our model actually produces, e.g. under approximate decoding. however, an information-theoretic evaluation is much more suitable for measuring the more abstract notion of which language pairs are hardest to translate to and from, which is our purpose here. ## disentangling translation difficulty and monolingual complexity we contend that simply reporting cross-entropies is not enough. a second issue in performing a controlled, cross-lingual mt comparison is that the language generation component (without translation) is not equally difficult across languages @xcite . we claim that the difficulty of translation corresponds more closely to the mutual information mi(s; t ) between the source and target language, which tells us how much easier it becomes to predict t when s is given (see figure 1 ). but what is the appropriate analogue of mutual information for cross-entropy? one such natural generalization is a novel quantity that we term cross-mutual information, defined as: while q mt and q lm can, in general, be any two models, we exploit the characteristics of nmt models to provide a more meaningful, model-specific estimate of xmi. nmt architectures typically consist of two components: an encoder that embeds the input text sequence, and a decoder that generates translated output text. the latter acts as a conditional language model, where the source-language sentence embedded by the encoder drives the target-language generation. hence, we use the decoder of q mt as our q lm to accurately estimate the difficulty of translation for a given architecture in a controlled way. in summary, by looking at xmi, we can effectively decouple the language generation component, whose difficulties have been investigated by @xcite , from the translation component. this gives us a measure of how rich and useful the information extracted from the source language is for the target-language generation component. ## experiments in order to measure which pairs of languages are harder to translate to and from, we make use of the latest release v7 of europarl @xcite : a corpus of the proceedings of the european parliament containing parallel sentences between english (en) and 20 other european languages: bulgarian (bg), czech (cs), danish (da), german (de), greek (el), spanish (es), estonian (et), finnish (fi), french (fr), hungarian (hu), italian (it), lithuanian (lt), latvian (lv), dutch (nl), polish (pl), portuguese (pt), romanian (ro), slovak (sk), slovene (sl) and swedish (sv). pre-processing steps in order to precisely effect a fully controlled experiment, we enforce a fair comparison by selecting the set of parallel sentences available across all 21 languages in europarl. this fully controls for the semantic content of the sentences; however, we cannot adequately control for translationese @xcite . our subset of europarl contains 190,733 sentences for training, 1,000 unique, random sentences for validation and 2,000 unique, random sentences for testing. for each parallel corpus, we jointly learn byte-pair encodings (bpe; @xcite for the source and target languages, using 16,000 merge operations. we use the same vocabularies for the language models. 2 setup in our experiments, we train transformer models @xcite , which often achieve state-of-the-art performance on mt for various language pairs. in particular, we rely on the py-torch @xcite re-implementation of the transformer model in the fairseq toolkit @xcite . for language modeling, we use the decoder from the same architecture, training it at the sentence level, as opposed to commonly used fixedlength chunks. we train our systems using label smoothing (ls; @xcite 2 and figure 3 . 1 , into and from english. more correlations in figure 4 . 2020) as it has been shown to prevent models from over-confident predictions, which helps to regularize the models. we report cross-entropies (h q mt , h q lm ), xmi, and bleu scores obtained using sacrebleu @xcite . 3 finally, in a similar vein to @xcite , we multiply crossentropy values by the number of sub-word units generated by each model to make our quantities independent of sentence lengths (and divide them by the total number of sentences to match our approximations of the true distributions). see app. a for experimental details. ## results and analysis we train 40 systems, translating each language into and from english. 4 the models' performance in terms of bleu scores, and the cross-mutual information (xmi) and cross-entropy values over the test sets are reported in translating into english when translating into the same target language (in this case, english), bleu scores are, in fact, comparable, and can be used as a proxy for difficulty. we can then conclude, for instance, that lithuanian (lt) is the hardest language to translate from, while spanish (es) is the easiest. in this scenario, given the good correlation of bleu scores with human evaluations, it is desirable that xmi correlates well with bleu. this behavior is indeed apparent in the blue points in the left part of figure 2 , confirming the efficacy of xmi in evaluating the difficulty of translation while still being independent of the target language generation component. translating from english despite the large gaps between bleu scores in figure 3 : h q lm (t ), decomposed into xmi(@xmath3, the information that the system successfully transfers, and h q mt (t | s), the uncertainty that remains in the target language, all measured in bits. note that in xmi(@xmath4. metric pearson spearman word number ratio 0.2988 (0.0611) 0.3570 (0.0237) ttrsrc -0.5196 (0.0006) -0.5136 (0.0007) ttrtgt 0.1651 (0.3086) 0.3355 (0.0343) dttr -0.4427 (0.0042) -0.4660 (0.0024) word overlap ratio 0.1383 (0.3949) 0.1731 (0.2853) in bleu scores. this suggests that the former model is heavily penalized by the target-side language model; this is likely because finnish has a large number of inflections for nouns and verbs. another interesting example is given by @xcite and spanish (es) in correlations with linguistic and data features last, we conduct a correlation study between the translation difficulties as measured by xmi and the linguistic and data-dependent properties of each translation task, following the approaches of @xcite and @xcite . ## conclusion in this work, we propose a novel informationtheoretic approach, xmi, to measure the translation difficulty of probabilistic mt models. differently from bleu and other metrics, ours is language-and tokenization-agnostic, enabling the first systematic and controlled study of crosslingual translation difficulties. our results show that xmi correlates well with bleu scores when translating into the same language (where they are comparable), and that higher bleu scores in different languages do not necessarily imply easier translations. in future work, we plan to extend this analysis across more translation pairs, more diverse languages and multiple domains, as well as investigating the effect of translationese or source-side grammatical errors @xcite .