Predicting Human Translation Difficulty Using Automatic Word Alignment (2023)

Abstract: Translation difficulty arises when translators are required to resolve translation ambiguity from multiple possible translations. Translation difficulty can be measured by recording the diversity of responses provided by human translators and the time taken to provide these responses, but these behavioral measures are costly and do not scale. In this work, we use word alignments computed over large scale bilingual corpora to develop predictors of lexical translation difficulty. We evaluate our approach using behavioural data from translations provided both in and out of context, and report results that improve on a previous embedding-based approach (Thompson et al., 2020). Our work can therefore contribute to a deeper understanding of cross-lingual differences and of causes of translation difficulty.

## introduction words can be hard to translate for many reasons including cultural context and differences in semantic subdivisions across languages @xcite . for instance, the emotional/moral sense of english heart is commonly translated as malay hati, but in a medical setting heart should be translated as jantung and hati refers to a different bodily organ (the liver). examples such as this are challenging for language learners and translators because they go beyond simple one-to-one correspondences between source and target words. translation difficulty has been studied by researchers from multiple disciplines including psycholinguistics @xcite , computational linguistics @xcite , machine translation @xcite and translation studies @xcite . understanding translation difficulty is an important scientific challenge in its own right, but methods for measuring difficulty * now at google deepmind. in this work, we use surprisal and entropy derived from word alignment to estimate word translation difficulty. different pairs of aligned words are collected as translation alternatives (e.g., heart-hati and heart-jantung) and used to infer a word's translation distribution and compute our informationtheoretic difficulty measures. among previous studies of translation our approach is closest to the work of @xcite , as it leverages large-scale human translation data and extracts word-level translations from aligned sentences. unlike previous studies, however, we are the first to use word alignments to directly address translation difficulty as a psychological aspect of lexical semantics that is measurable by behavioural data. we evaluate our difficulty estimates against translation norms @xcite @xcite @xcite that measure translation difficulty out of context and translation process features @xcite ) that measure translation difficulty in context. we also compare against a previous approach that uses multilingual embeddings to develop a measure of translation difficulty @xcite . relative to embeddings, we suggest that word alignments better capture lexical and morphological distributions, and hence allow for a better measure of translation difficulty. 1 our measures of translation difficulty are interpretable, and as we show in later sections, help improve the understanding of human language and translation processing. ## related work our approach builds on two lines of work from the psycholinguistic literature on translation. one line of work relies on translation norms derived from tasks in which bilingual participants translate single words presented out of context or rate semantic similarity between pairs of words @xcite @xcite @xcite . high variation in translation responses to a given word provides evidence of translation ambiguity @xcite ; whereas perceived degree of cross-lingual semantic overlap informs lexical choice and is predictive of response time @xcite @xcite . a second line of work studies translation in context by measuring reading time and production duration as translators process realistic texts @xcite . behavioral approaches like these provide gold-standard measures of translation difficulty but are costly and do not scale. within the computational literature, @xcite and @xcite derive automatic measures of translation difficulty based on the idea that difficult-to-translate words are hard to align across word embedding spaces. the former use embeddings to compare semantic neighbourhoods of bilingual word pairs and report significant correlations with human semantic similarity judgements. carl (2021b) learned a cross-lingual embedding projection to estimate word pair similarities, and showed that these estimates predict translation process data. @xcite and yin et al. (2021) probe translation ambiguity from nmt models using cross-mutual information, which is useful for identifying contextual translations in nmt models. @xcite use word alignment distributions to reveal lexical semantic distinctions across languages. their work shows that word alignments, with properly extracted descriptions, help language learners disambiguate fine-grained lexical distinctions, but does not directly address the general notion of translation difficulty. ## assessing word-level translation difficulty through word alignments assume that we have a parallel corpus and a word aligner and are interested in the translation distribution of word w from a source language, l1, to a target language, l2. the most natural approach is a count-based distribution, where counts for words aligned with w are normalized by the frequency of w. from here, p al (v|w), the probability of word w being translated to v, can be computed over aligned word pairs. in addition to alignment counts, most word aligners assign a score for each pair of aligned words. given two parallel sequences, @xmath0, ..., x m ] and @xmath1, ..., y n ], let x i ↔ y j indicate that the ith token from x is paired with the jth token from y, and let s x i ↔y j denote the alignment score. 2 this allows a weight-based distribution parallel to the count-based method above. in general, we calculate p al (v|w) by: for the weight-based distribution, s w↔v represents the sum of alignment scores of all w ↔ v pairings. for the count-based distribution, s x i ↔y @xmath2, i.e., s w↔v is the number of times w is aligned with v in the entire corpus. 3 the final distribution, p al (v|w), is normalized given the total scores of all possible alignments with w, where v refers to the vocabulary of l2 in the corpus. the concept of surprisal in psycholinguistics is often associated with cognitive workload, which in translation studies is connected to word translation information (itra) @xcite . translation surprisal is defined as: low surprisal values indicate that v is a stable translation of w, which is expected to require low effort to produce. the translation uncertainty associated with a source word w can be formulated as the entropy (or expected surprisal): surprisals derived from count-based and weightbased distributions are denoted by i c al and i w al respectively. likewise, h c al and h w al will be used as shorthands for their respective entropy values. word pairs with higher surprisal are expected to be more difficult. higher translation entropy indicates a greater range of translations for a source word, which is expected to contribute to translation difficulty. more details about these three features and about the preprocessing applied are described in appendix a. we validate against data sets in japanese (enja15, @xcite , german (sg12, @xcite and spanish (bml12, mesa-lao, 2014), for which information about translation at the token-level is readily available. baselines. we compare i c al and i w al with thompson et al.'s (2020) embedding-based approach, which has been framed explicitly as an account of translation difficulty. following their work we expand the initial northeuralex translations @xcite to include all translation pairs in the evaluation data and recompute word-pair semantic alignments using common crawl and wikipedia fast-text embeddings @xcite . the final values are negated to match the sign of i al , and denoted here by s emb . 6 the translation uncertainty associated with a single source word. we therefore compare h c al and h w al with a simple embedding-based measure m emb defined as the highest value of s emb associated with a source word. we limit all comparisons to the same set of vocabulary and translation pairs. foot_6 ## conclusion we developed predictors of translation difficulty based on word alignment distributions and tested them using translation norms and translation processing data. compared to the embedding-based approach, our measures derived from word alignment do not depend on lexical databases and more accurately capture lexical choice distributions and morphological variation. our results show improved estimates of translation difficulty, but suggest that a comprehensive account of human translation difficulty must also consider additional factors such as form similarity.