An Analysis of Surprisal Uniformity in Machine and Human Translations (2024)

Abstract: This study examines neural machine translation (NMT) and its performance on texts that diverege from typical standards, focusing on how information is organized within sentences. We analyze surprisal distributions in source texts, human translations, and machine translations across several datasets to determine if NMT systems naturally promote a uniform density of surprisal in their translations, even when the original texts do not adhere to this principle.The findings reveal that NMT tends to align more closely with source texts in terms of surprisal uniformity compared to human translations.We analyzed absolute values of the surprisal uniformity measures as well, expecting that human translations will be less uniform. In contradiction to our initial hypothesis, we did not find comprehensive evidence for this claim, with some results suggesting this might be the case for very diverse texts, like poetry.

## introduction natural language processing tools based on machine learning, such as machine translation, autocorrect, predictive typing, search, and text generation, have become integral to our daily lives. with the advancement of large language models (llms), it's anticipated that interacting with these technologies will become a critical aspect of our work and societal engagement. however, numerous questions about these technologies persist. in this work, we take a look specifically at neural machine translation (nmt) and at one such question: how well do these tools work on an input that is different from a typical text, not in terminology or domain, but in a way the information content is organized within an utterance? are there any biases within the algorithms themselves that can be beneficial for ordinary types of texts, but harmful for specific cases that deviate from the usual rules found in mundane text content? we propose that neural machine translation (nmt) will be more effective with texts adhering to the uniform information density (uid; @xcite ) hypothesis, meaning that the level of surprisal is consistently spread out throughout the sequence. one of the culprits could be the beam search decoding, which has been shown to adhere to the uid principle @xcite , i.e. even if the input has a diverse distribution of surprisal, the distribution in the translation will be more uniform. the uid-enforcing property of beam search has been shown as the key to its ability to produce high-quality, human-like texts (compared to exact search under the same model), even being dubbed the beam search blessing by @xcite . we hypothesize that while in general, this property is positive, there are use-cases where too much emphasis on the uniformity of suprisals hurts the final translation quality. in this work, we look for such examples by comparing distributions of suprisals over source texts, machine translation and human translation across multiple test sets. ## related work in this section, we will list the work exploring the presence of the uid principle in human-produced language as well as its presence and links to mt algorithms. there is an extensive body of psycholinguistic work concerning the relationship between text predictability or surprisal and reading comprehension. the results on whether the effects of surprisal on reading comprehension is linear or super-linear (which would be consistent with the uid hypothesis) are mixed: for example, @xcite found support for super-linear relationship. one of the most recent and largest studies @xcite uses a wide array of open-source datasets, new large language models for the surprisal estimation (gpt-3) and novel evaluation methods (deep learning based non-linear regression for analyzing continuous-time systems @xcite . their findings support a linear relationship between word surprisal and sentence reading times, suggesting that any pressure for uid seen in natural language is not motivated by an easier comprehension. @xcite ask why, empirically, beam search produces higher quality outputs than exacted search under the same model. to find the inductive bias embedded in beam search that allows this, they reverse engineer the objective that beam search is a solution for. they found that beam search can be reformulated as an exact search with a uniformity regularizer which enforces uid and that this property is the key to its effectivity. @xcite employ a similar regularizer in the training of the model, which led to improved translation quality. ## methods this section introduces the measures we use to operationalize the surprisal distribution uniformity concept, closely following the definition by @xcite . ## results this section describes the settings and presents the results of measuring difference in surprisal uniformity in human and machine translations. ## future work the results of our study are inconclusive and we plan to obtain more diverse test sets, where the mt adherence to generating uniform texts could be harmful. suppose we find such texts, where traditional, encoder-decoder mt models using beam search decoding struggle. in that case, we will evaluate also large language models, where the decoding algorithm is usually based on sampling. we speculate that the constraints on surprisal distribution imposed by beam search might be src ref1 ref2 ref3 ref4 mt mt2 dataset 0 20 40 60 80 100 mean lv2 comparison of lv2 per translation origin compensating for the models' inherent lack of global planning. in an ideal scenario, a model might introduce a word with high surprisal intentionally, planning to balance this with lower surprisal words in subsequent segments. however, the current model designs, focusing on next-token prediction, might not accurately forecast these future steps, i.e. the model could produce high surprisal word with a "plan" to get the surprisal "back" in the future timesteps, but, due to next-token-only objective, the future steps are miscalculated. the beam search will not produce such word, due to the adherence to the local uniformity, so the modelling flaw stays hidden. if this hypothesis turns out to be true, our focus will shift to improve the global planning capabilities of the models, e.g. by employing an alternative training objective. ## conclusions overall, we do not have reliable proof that mt produces texts that are more uniform in surprisal distribution than humans yet. either our hypothesis is false, or our measurement methodology is flawed. one possible reason could be that the lms we used to estimate the surprisals are trained on human text, not on mt outputs so it overestimates surprisal of some phenomena in mt. we plan further experiments to improve our methodology and extend the analysis to more datasets. while our study was not able to reliably prove our initial hypothesis that mt systems make the distribution of surprisal more uniform in their translations than a human translator, we have gained some insights from the experiments we carried out. firstly, nmt systems demonstrate a tendency to produce translations that exhibit surprisal uniformity closely aligned with source texts, more so than a human translator. secondly, the absolute values of uniformity measures are similar between ht and mt as well, however, it depends on the mt system used. some of the systems produce more uniform translations than humans. notably, in more varied datasets, such as those containing literary works or poetry, human translations showed greater diversity compared to mt outputs, according to some of the measures. some of the findings indicate that part of the variance in surprisal distribution observed in mt may stem from translation inaccuracies. by scoring the translations using quality estimation metrics and filtering out low-scoring examples, in mt surprisal uniformity on one of the datasets increases, while ht uniformity stays the same. figures @xcite show histograms of the senetence-level values of lv2, cv and gv 2 across books, wmt and wilde datasets. figures 19 and 20 show the lv2 histograms on wilde and wmt after applying filtering based on comet threshold of the mt. figures 24 and 25 show values of cv and gv 2 depending on the comet threshold for mt translations. figure 23 shows the relationship between the comet score threshold on the human translation and the mean lv2 score of the source sentence and all the translations. we see that in books, global and wmt, both the source sentence and the translations are more uniform for high comet scores. this might suggest a preference of comet score for uniform surprisal (for example, rooted in training data).