“You Sound Just Like Your Father” Commercial Machine Translation Systems Include Stylistic Biases (2020)

Abstract: The main goal of machine translation has been to convey the correct content. Stylistic considerations have been at best secondary. We show that as a consequence, the output of three commercial machine translation systems (Bing, DeepL, Google) make demographically diverse samples from five languages “sound” older and more male than the original. Our findings suggest that translation models reflect demographic bias in the training data. This opens up interesting new research avenues in machine translation to take stylistic considerations into account.

## introduction translating what is being said is arguably the most important aspect of machine translation, and has been the main focus of all its efforts so far. however, how something is said also has an impact on how the final translation is perceived. @xcite have pointed out that demographic aspects of language do play a role in translation, and could help in personalization. as @xcite have shown, gendered inflections like "sono stanco/a" (italian i am tired) are an important aspect of correct translations. in many cases, capturing the style of a document is equally important as its content: translating a lover's greeting as "i am entirely pleased to see you" might be semantically correct, but seems out of place. demographic factors (age, gender, etc.) all manifest in language, and therefore influence style: we do not expect a 6-year old to sound like an adult, and would not translate a person to seem differently gendered. however, in this paper, we show such a change is essentially what happens in machine translation: authors sound on average older and more male. prior work @xcite has shown that translation weakens the signal for gender pre-diction. we substantially extend this analysis in terms of languages, demographic factors, and types of models, controlling for demographically representative samples. we show the direction in which the predicted demographic factors differ in the translations, and find that there are consistent biases towards older and more male profiles. our findings suggest a severe case of overexposure to writings from these demographics @xcite , which creates a self-reinforcing loop. in this paper, we use demographicallyrepresentative author samples from five languages (dutch, english, french, german, italian), and translate them with three commercially available machine translation systems (google, bing, and deepl). we compare the true demographics with the predicted demographics of each translation (as well as a control predictor trained on the same language). without making any judgment on the translation of the content, we find a) that there are substantial discrepancies in the perceived demographics, and b) that translations tend to make the writers appear older and considerably more male than they are. ## data we use the trustpilot data set from @xcite , which provides reviews in different languages, and includes information about age and gender. we use only english, german, italian, french, and dutch reviews, based on two criteria: 1) availability of the language in translation mod-els, and 2) sufficient data for representative samples (see below) in the corpus. for the english data, we use us reviews, rather than uk reviews, based on a general prevalence of this variety in translation engines. ## methods to assess the demographic profile of a text, we train separate age and gender classifiers for each language. these classifiers allow us to compare the predicted profiles in the original language with the predicted profiles of the translation, and compare both to the actual demographics of the test data. we use simple logistic regression models with l 2 regularization over 2-6 character-grams, and regularization optimized via 3-fold crossvalidation. 3 the numbers in for each non-english sample, we predict the age and gender of the author in both the original language and in each of the three english translations (google, bing, and deepl). i.e., we use the respective language's classifier described above (e.g., a classifier trained on german to predict german test data), and the english classifier described above for the translations. e.g., we use the age and gender classifier trained on english data to predict the translations of the german test set. for the english data, we first translate the texts into each of the other languages, using each of the three translation systems. then we again predict the author demographics in the original english test set (using the classifier trained on english), as well as in each of the translated versions (using the classifier trained on the respective language). e.g., we create a german, french, italian, and dutch translation with each google, bing, and deepl, and classify both the original english and the translation. we can then compare the distribution of age groups and genders in the predictions with the actual distributions. if there is classifier bias, both the predictions based on the original language and the predictions based on the translations should be skewed in the same direction. we can measure this difference by computing the kullback-leibler (kl) divergence of the predicted distribution from the true sample distribution. in order to see whether the predictions differ statistically significantly from the original, we use a use a χ 2 contingency test and report significance at @xmath0.05 and @xmath1.01. if instead there is a translation bias, then the translated predictions should exhibit a stronger skew than the predictions based on the original language. by using both translations from and into english, we can further tease apart the direction of this effect. ## conclusion we test what demographic profiles author attribute tools predict for the translations from various commercially available machine translation tools. we find that independent of the mt system and the translation quality, the predicted demographics differ systematically when translating into english. on average, translations make the author seem substantially older and more male. translating from english into any of the other languages shows more mixed results, but similar tendencies. for data science and analytics (bidsa) and the data and marketing insights (dmi) unit.