Predicting Human Translation Difficulty with Neural Machine Translation (2024)

Abstract: Human translators linger on some words and phrases more than others, and predicting this variation is a step towards explaining the underlying cognitive processes. Using data from the CRITT Translation Process Research Database, we evaluate the extent to which surprisal and attentional features derived from a Neural Machine Translation (NMT) model account for reading and production times of human translators. We find that surprisal and attention are complementary predictors of translation difficulty, and that surprisal derived from a NMT model is the single most successful predictor of production duration. Our analyses draw on data from hundreds of translators operating across 13 language pairs, and represent the most comprehensive investigation of human translation difficulty to date.

## introduction during the nuremberg trials, a soviet interpreter paused and lost her train of thought when faced with the challenge of translating the phrase ''trojan horse politics'', and the presiding judge had to stop the session @xcite . translation difficulty rarely has such extreme consequences, but the process of translating any text requires a human translator to handle words and phrases that vary in difficulty. translation difficulty can be operationalized in various ways @xcite , and one approach considers texts to be difficult if they trigger translation errors @xcite . here, however, we focus on difficulty in cognitive processing, and consider a word or phrase to be difficult if it requires extended processing time. our opening example (i.e., ''trojan horse politics'') illustrates translation difficulty at the * now at google. phrase level, and figure 1 shows how our notion of translation difficulty varies at the level of individual words. across this sample, words like ''societies'' and ''population'' are consistently linked with longer production times than words like ''result'' and ''tend''. processing times have been extensively studied by psycholinguists, but the majority of this work is carried out in a monolingual setting. within the literature on translation, analysis of cognitive processing is most prominent within a small but growing area known as translation process research (tpr). researchers in this area aim to characterize the cognitive processes and strategies that support human translation, and do so by analyzing eye movements and keystrokes collected from translators @xcite . here we build on this tradition and focus on three variables at the word and segment level derived from the critt tpr-db database @xcite : source reading time (trts), target reading time (trtt), and translation duration (dur). our analyses are relatively large in scale by the standards of previous work in this area, and we draw on data from 312 translators working across 13 language pairs. a central goal of our work is to bring translation process research into contact with modern work on neural machine translation (nmt). recent work in nlp has led to dramatic improvements in the multilingual abilities of nmt models @xcite , and these models can support tests of existing psycholinguistic theories and inspire new theories. our work demonstrates the promise of nmt models for tpr research by testing whether surprisal and attentional features derived from an nmt model are predictive of human translation difficulty. two of these predictors are shown in the right panels of figure 1 , and both are correlated with translation duration for the example sentence shown. in what follows we introduce the surprisal and attentional features (sections 3 and 4) that we consider then evaluate the extent to which they yield improvements over baseline models of translation difficulty. we find that surprisal is a strong predictor of difficulty, which supports and extends previous psycholinguistic findings that surprisal predicts both monolingual processing @xcite and translation processing @xcite @xcite . the attentional features we evaluate predict difficulty less well, but provide supplementary predictive power when combined with surprisal. ## related work an extensive body of monolingual work has demonstrated that the pace of human reading is based on next-word predictability-more contextually surprising words incur higher cognitive costs and are slower to process @xcite . the phenomenon is observed across a wide range of language usage, including reading @xcite @xcite @xcite , listening and comprehension @xcite @xcite , speech @xcite @xcite @xcite @xcite , typing @xcite , and code-switching @xcite . surprisal has also been proposed as a predictor of translation difficulty in translation processing research (tpr) @xcite @xcite @xcite , but existing evaluations of this idea are limited in two respects. first, previous measures of translation surprisal are based on overly simplistic probability estimates, which are inferior to modern nmt models. prior work on tpr mostly relies on probabilities extracted from manual word alignments, and these probabilities are noisy because they are not sensitive to target context and because the corpora are relatively small. second, relying on manual alignment means that most previous approaches do not scale well and cannot be applied to novel unaligned texts. @xcite address this second limitation using automatic alignment of existing parallel corpora, but their probability estimates are again insensitive to context and therefore less accurate than probabilities derived from nmt models. along with surprisal, prior tpr research has also proposed contextual entropy as a predictor of translation difficulty @xcite @xcite . entropy is the expected surprisal of a translation distribution, and indicates the effort in resolving the uncertainty over all possible translation choices. the results of @xcite and @xcite suggest, however, that entropy is weaker than surprisal as a predictor of translation duration. entropy is also a relatively weak predictor of monolingual reading difficulty, despite being hypothesized to indicate a reader's anticipation of processing effort @xcite @xcite . in the broader nlp literature, several lines of work suggest that attentional measures derived from language models and nmt models can capture processing difficulty. attention weights in language models @xcite and nmt models @xcite are known to contribute to next-token prediction, and previous studies have explored whether attentional weights in language models can account for monolingual processing time @xcite . for translation, nmt studies suggest that the contextualization of ambiguous tokens occurs in encoder self-attention @xcite , and nmt models show higher cross-attention entropy with increasingly difficult tokens @xcite @xcite . nmt attentional weights also reflect whether a figurative expression is handled by paraphrasing as opposed to literal translation @xcite . all of these results suggest that attentional weights are worth exploring as predictors of human translation difficulty. in the tpr literature, prior work has explored whether errors made by mt models tend to predict translation difficulty and post-editing difficulty for humans @xcite . to our knowledge, however, previous work has not used probabilities and attentional measures derived from nmt models as predictors of human translation difficulty. building on previous work, we focus on surprisal rather than entropy, and provide a comprehensive evaluation of the extent to which surprisal predicts translation difficulty. we also propose several attentional features based on previous literature and test whether they contribute predictive power that goes beyond surprisal alone. foot_0 ## surprisal: an information-theoretic account of translation processing in contrast with traditional monolingual surprisal, translation surprisal is based on a context that includes a complete sequence in a source language (sl) and a sequence of previously translated words in a target language (tl). target words with high translation surprisal are hypothesized to require extended cognitive processing because they are relatively unpredictable in context. relative to prior work, we aim to provide a more comprehensive and rigorous evaluation of the role of surprisal in translation processing. we consider predictors of translation difficulty based on both monolingual and translation surprisal, which we estimate using a large language model and a neural translation model respectively. ## predicting translation difficulty using nmt attention modern encoder-decoder nmt models rely on the transformer architecture and incorporate three kinds of attention: encoder self-attention, crossattention, and decoder self-attention @xcite . 3 we consider all three sets of attention weights as potential predictors of translation difficulty. by some accounts, reading, transferring, and writing are three distinct stages in the human translation process @xcite @xcite , and we propose that the three attentional modules roughly align with these three stages. let @xmath0, . . . , x m ] and @xmath1, . . . , y n ] denote parallel source and target sequences, with @xmath2, . . . , m} and @xmath3, . . . , n} as their token indices. let u and v denote segments of x and y respectively such that @xmath4, where the indices i ⊆ m and j ⊆ n. note that x and y do not include special tokens (e.g., the end-of-sequence tag eos) added to the sequences under the nmt's tokenization scheme. we additionally define @xmath5, i.e., @xmath6, . . . , max(j)} \ j. in contrast to i, j only includes token indices from the context preceding v. we define j in this way because the decoder typically does not have access to future tokens when generating translations. we consider two kinds of attentional features. the first captures the total attentional flow from segment w to segment z: the second kind of attentional feature sums the entropies of all attentional distributions from segment w to segment z: we renormalize a kl along a k )l , the vector including attention weights from k to token indices l. @xcite this step allows us to filter out attention to special tokens (e.g., eos) when considering the entropy of a distribution. using equations 3 and 4, we define in the following six attentional features as candidate predictors of source-side reading time and five features for target-side reading time and production duration. for simplicity, we obtain the final attentional features in our analysis by averaging values computed for each attentional head across layers. ## data our empirical measures of translation difficulty are derived from the critt translation process research database (tpr-db) @xcite . we focus on three behavioral measures: source text reading time (trts) is the sum of all fixation durations on a given source segment during a session; target text reading time (trtt) is the sum of all fixation durations on a target segment; and translation production duration (dur) is the time taken to produce a segment. both reading time measures are based on eye-tracking data, and the translation production measure is based on keylogging data. the data set is organized in terms of words, segments, and sentences, and we carry out separate analyses at the word and segment levels. in critt tpr-db, word and segment boundaries and alignments are provided by human annotators. for consistency, we remove alignments of words and segments that cross sentence boundaries. following carl (2021b), we filter values of trts, trtt, and dur lower than 20ms. the remaining values are log scaled. we analyze data from 17 public studies available from the public database. 5 these studies represent 13 different language pairs, and each study includes data from an average of 18 human translators. the studies included along with the size of each one are summarized in normalization. sections 3 and 4 describe feature definitions that are sums over a sequence of tokens, which makes it crucial to control for segment length when predicting reading time and production duration. all surprisal values are therefore normalized by the lengths of the input segments, w i and y j . to normalize attentional features, we first calculate dummy feature values by replacing a lk in equation 3 and 4 with uniform attention values (i.e., a @xmath7. a normalized attentional feature is defined as the ratio of the raw feature value (defined in section 4) to its dummy value. ## results δllh as a measure of predictive power. prior work on translation difficulty rarely uses held-out evaluation, but we follow previous psycholinguistic studies @xcite @xcite @xcite and evaluate our models using log-likelihood of held-out data. to assess the predictive power of a feature, we train a mixed model with the feature of interest in addition to all control features, and compare against a baseline model which includes only the control features. the contribution of the predictor feature is then measured as the difference in log-likelihood of the held-out test data (δllh) between the two models. a positive δllh indicates added predictive power from the feature relative to the baseline model, whereas δllh ≤ 0 means that we have no evidence for the effect of the feature on reading and production times. foot_5 like wilcox et al. (2023), we test if δllh > 0 is significant across held-out samples using a paired permutation test based on 1000 random permutations. ## discussion section 7.1 showed that monolingual surprisal predicts source reading time, but that translation surprisal is a more consistent predictor of target reading time and production duration. on its own, nmt attention also predicts translation difficulty to some degree, but the most accurate predictions are achieved by combining surprisal and attentional features. ## conclusion our results support the prevailing view that current nlp models, including lm and nmt, align partially with human language usage and are predictive of language processing complexity. we evaluated surprisal and nmt attention as predictors of human translation difficulty, and found that both factors predict reading times and production duration. previous work provides some evidence that surprisal and nmt attention capture important aspects of translation difficulty, 12 studies included from multiling corpus are ruc17, enja15, nj12, stc17, sg12, endu20 and bml12. although the attentional features we consider are empirically successful and grounded in prior literature, they are not without limitations. these features are relatively simple and combining attention weights in more sophisticated ways may allow stronger predictions of human translation difficulty. a more theoretically motivated approach that builds on recent studies of the interpretability of attention distributions @xcite @xcite is worth exploring to develop more fine-grained predictors of translation processing. to work with as much data as possible, we focused primarily on analyses that combine data from all 13 language pairs, but analyzing translation challenges in individual language pairs is a high priority for future work. a possible next step is an analysis exploring whether the predictors considered here are sensitive to constructions in specific languages that are known sources of processing difficulty @xcite . factors such as surprisal and attentional flow are appealing in part because their generality makes them broadly applicable across languages, but understanding the idiosyncratic ways in which each pair of languages poses translation challenges is equally important.