Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation (2021)

Abstract: Recent studies in the field of Machine Translation (MT) and Natural Language Processing (NLP) have shown that existing models amplify biases observed in the training data. The amplification of biases in language technology has mainly been examined with respect to specific phenomena, such as gender bias. In this work, we go beyond the study of gender in MT and investigate how bias amplification might affect language in a broader sense. We hypothesize that the ‘algorithmic bias’, i.e. an exacerbation of frequently observed patterns in combination with a loss of less frequent ones, not only exacerbates societal biases present in current datasets but could also lead to an artificially impoverished language: ‘machine translationese’. We assess the linguistic richness (on a lexical and morphological level) of translations created by different data-driven MT paradigms – phrase-based statistical (PB-SMT) and neural MT (NMT). Our experiments show that there is a loss of lexical and syntactic richness in the translations produced by all investigated MT paradigms for two language pairs (EN-FR and EN-ES).

## introduction the idea of translation entailing a transformation is widely recognised in the field of translation studies (ts) @xcite . translations are specific communicative acts occurring in a particular context governed by their own laws. some of the features that characterize translated texts are defined as simplification, explicitation, normalization and leveling out @xcite . the fingerprints left by the translation process and the language this results into, have been referred to as 'translationese' @xcite . empirical evidence of the existence of translationese can be found in studies showing that machine learning techniques can be employed to automatically distinguish be-tween human translated and original text by looking at lexical and grammatical information @xcite . translationese differs from original texts due to a combination of factors including intentional (e.g. explicitation and normalization) and unintentional ones (e.g. unconscious effects of the source language input on the target language produced). unlike other work on (human) translationese (or even related work on 'post-editese'), we delve into the effects of machine translation (mt) algorithms on language, i.e. 'machine translationese'. so far, generating accurate and fluent translations has been the main objective of mt systems. as such, maintaining the richness and diversity in the outputs has understandably not been a priority @xcite . 1 however, as mt systems have reached a quality that is (arguably) close to that of human translations @xcite and as such are being used widely on a daily basis, we believe it is time to look into the potential effects of (mt) algorithms on language itself. 2 the main motivations behind this work are: (i) if algorithmic bias is indeed a by-product of our algorithms, a statistically biased mt system might prefer frequently occurring words (or sub-words) over others. since mt systems do not necessarily distinguish between different synonymous translations (lexical richness) and morphological variants (grammatical richness), algorithmic bias could lead to the loss of morphological variety (and thus interfere with the ability of our systems to generate 1 one might argue that for some tasks and domains, diversity can be harmful (specific in-domain translations would prefer consistency over e.g. lexical diversity). at all times a grammatically correct option); (ii) the sociolinguistic perspective of machine translationese since it has been established that language contact (e.g. via translationese) can entail language changes @xcite . if machine translationese (and other types of 'nlpese') is a simplified version of the training data, what does that imply from a sociolinguistic perspective and how could this affect language on a longer term? the main objective of the presented research is to establish whether there is indeed a quantitatively measurable difference between the linguistic richness of an mt system's training data and its output in terms of morphological and lexical diversity. to do so, we conduct an in-depth analysis that goes beyond frequently used standard lexical diversity metrics such as ttr, yule's i and mtld. we assess the lexical and morphological diversity through an adapted version of the lexical frequency profile used to assess language acquisition, a measure of morphological diversity based on shannon and simpson diversity and an novel automatic evaluation of synonym frequency. we focus on the most prominent data-driven mt paradigms: neural mt (nmt), both lstm @xcite and transformer @xcite , and phrase-based statistical mt (pb-smt). up to our knowledge this is the first research on lexical and morphological diversity of machine translation output, i.e. machine translationese. the contributions of this work can be summarised as: (i) a detailed analysis of lexical and morphological diversity of machine translationese and the loss thereof to quantify the effects of algorithmic bias; (ii) the adaptation of a metric used in language acquisition for assessing lexical sophistication in mt foot_0 ; (iii) the use of shannon entropy and simpson diversity to measure morphological richness, and (iv) a novel, automatic evaluation of synonym frequency. ## related work several studies have exposed the societal biases present in datasets (racial bias @xcite , political bias @xcite , gender bias @xcite . existing nlp technology are likely to pick up biases present in the training data and various explorations of e.g. gender bias in nlp systems have indeed re-vealed the existence of harmful biases in the output they generate @xcite @xcite @xcite @xcite . research related to bias has often focused on gender or race. especially in a field such as mt, the implicit gender in a language such as english and its consecutive translations into morphologically richer languages with gender agreement, makes it relatively easy to expose and study biases related to gender in a contrastive linguistic setting. in the context of this paper, we would like to note that (statistical) bias is not limited to gender or race but can be defined as any systematic inaccuracy in one direction leading to an under (or over) estimation of observations. a handful of recent work has mentioned the possibility of algorithmic bias on top of the already existing societal biases in the training data @xcite @xcite . for instance, @xcite observe a phenomenon they refer to as 'bias amplification'. they note that in their training data an activity such as 'cooking' is associated 33% times more with women compared to men. after training a model on that dataset, the existing disparity is amplified to 68% times more associations with women. in the field of mt, @xcite address the effects of statistical bias on language generation in an mt setting. they assess lexical diversity using standard metrics -ttr, mtld and yule's k-and conclude that the translations produced by various mt systems (pb-smt and nmt) are consistently less diverse than the original training data. their approach was conducted on nmt systems that were trained without bytepair-encoding (bpe) @xcite which limits the creativity of the translation systems. toral (2019) measures the lexical diversity of 18 state-of-the-art systems on 6 language pairs, reaching similar conclusions. they do so focusing specifically on post-editese. the experiments indicate that post-editese is simpler and more normalised than human translationese. the post-editese also shows a higher degree of interference from the source compared to the human translations. @xcite , like @xcite , centers around the automatic detection of post-editese and does not look into properties of unedited machine translationese. in aranberri @xcite different freely available mt systems (neural and rule-based) are compared in terms of automatic metrics (bleu, ter) and translationese features (ttr, length ratio input/output, perplexity, etc.) to investigate how such features correlate with translation quality. @xcite presents a comparison using similar translationese features of three mt architectures and the human translations of spoken and written language. in the field of pb-smt, @xcite show that pb-smt suffers considerably more than human translations (ht) from lexical loss, resulting in loss of lexical tightness and text cohesion. @xcite proof that automatic and human translated sentences can be automatically identified corroborating that human translations systematically differ from the translations produced by pb-smt systems. aside from @xcite , the above discussed related work uses metrics of lexical diversity to compare human translations to (post-edited) machine translations. in this work, we compare how and whether the output of an mt system differs (in terms of lexical and morphological diversity) from the data it was originally trained on. this way, we aim to investigate the effect of the algorithm (and algorithmic bias) on language itself. ## machine translation systems mt paradigms have changed quickly over the last decades. since this is the first attempt to quantify both the lexical and grammatical diversity of machine translationese, we experimented with the current state-of-the-art data-driven paradigms, lstm and transformer, as well as with pb-smt. we used data from the europarl corpus @xcite for two language pairs, english-french and english-spanish in both direction @xcite . we are interested in both directions in order to verify whether there is a difference in terms of (the potential loss of) diversity when comparing translations from a morphologically poorer language (english) into morphologically richer ones (french and spanish) and vice versa. our data is summarised in for training, testing and validation of the systems we used the same data. to build the vocabularies for the nmt systems we used sub-word units, allowing nmt to be more creative; using sub-word units also mitigates to a certain extent the out of vo-cabulary problem. to compute the sub-word units we used bpe with 50,000 merging operations for all our data sets. in the quality of our mt systems is evaluated on the test set using standard evaluation metrics -bleu @xcite (as implemented in sacrebleu @xcite ) and ter @xcite (as implemented in multeval @xcite ). our evaluation scores are presented in for all pb-smt we replaced marked unknown words with only one token "unk". while this does not effect the computation of bleu and ter, it allows us not to artificially boost the lexical and grammatical scores for these mt engines (see section 4) and assess their realistic dimensions. ## experiments and results assessing linguistic complexity is a multifaceted task spanning over various domains (lexis, morphology, syntax, etc.). the lexical and grammatical diversity are two of its major components @xcite @xcite . as such, we conduct an analysis using (i) lexical diversity and sophistication metrics (section 4.2) and (ii) grammatical diversity metrics (section 4.3). for lexical diversity, we use the following metrics: an adapted version of the lexical frequency profile (lfp), three standard metrics commonly used to assess diversity -ttr, yule's i and mtld-, and three new metrics based on synonym frequency in translations. up to our knowledge, this research is the first to employ lfp to analyze synthetic data. for grammatical diversity, we focus specifically on morphological inflectional diversity. we adopt the shannon entropy and simpson's diversity index to compute the entropy of the inflectional paradigms of lemmas, measuring the abundance and the evenness of wordforms per lemma. next, we will discuss the evaluation data and the aforementioned metrics designed in order to compare the diversity of the training data with the machine translationese. our evaluation scripts are available at https://github.com/dimitarsh1/ biasmt ; due to its large size, the data is not hosted in the github repository but is available upon request. ## conclusions in this work, we explore the effects of mt algorithms on the richness and complexity of language. we establish that there is indeed a quantitatively measurable difference between the linguistic richness of mt systems' training data and their output -a product of algorithmic bias. these findings are in line with previous results described in @xcite . assessing diversity or richness in language is a multifacted task spanning over various domains. as such, we approach this task from multiple angles focusing on lexical diversity and sophistication, morphological variety and a more translation specific metric focusing on synonymy. to do so, we analyse the results of 9 different metrics including established, newly proposed and adapted ones. the metrics suit we developed is unprecedented in the study of mt quality and we believe it could drive future research on mt evaluation. based on a wide range of experiments with 3 different mt architectures, we draw the following main conclusions: (i) all 9 metrics indicate that the original training data has more lexical and morphological diversity compared to translations produced by the mt systems. this is the case for all language pairs and directions; (ii) comparing the mt systems among themselves, there is a strong indication (for most metrics) that transformer models outperform the others in terms of lexical and morphological richness. we also ought to note that, on average, the ranking of the systems in terms of diversity metrics correlates with the quality of the translations (in terms of bleu and ter). this is something that would need to be further explored in future work; (iii) the data for the morphologically richer languages (es, fr) has higher lexical and (evidently) morphological diversity than the english data both in the original data and in the translations generated by all systems. however, for pb-smt, lstm and trans the difference in scores is much smaller than the orig, indicating that the mt systems have a stronger negative impact (in terms of diversity and richness) on the morphologically richer languages.