What about “em”? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns (2023)

Abstract: As 3rd-person pronoun usage shifts to include novel forms, e.g., neopronouns, we need more research on identity-inclusive NLP. Exclusion is particularly harmful in one of the most popular NLP applications, machine translation (MT). Wrong pronoun translations can discriminate against marginalized groups, e.g., non-binary individuals (Dev et al., 2021). In this “reality check”, we study how three commercial MT systems translate 3rd-person pronouns. Concretely, we compare the translations of gendered vs. gender-neutral pronouns from English to five other languages (Danish, Farsi, French, German, Italian), and vice versa, from Danish to English.Our error analysis shows that the presence of a gender-neutral pronoun often leads to grammatical and semantic translation errors. Similarly, gender neutrality is often not preserved. By surveying the opinions of affected native speakers from diverse languages, we provide recommendations to address the issue in future MT research.

## introduction machine translation (mt) is one of the most common applications of nlp, with millions of daily users interacting with popular commercial providers (e.g., bing, deepl, or google translate). given mt's widespread use and the increased focus on fairness in language technologies (e.g., @xcite , previous work has pointed to the potential ethical issues stemming from stereotypical biases encoded in the models, e.g., gender or age bias (e.g., @xcite . still, these studies treat gender as a binary variable and ignore the larger spectrum of (possibly marginalized) identities, e.g., non-binary individuals. this gender exclusivity stands in stark contrast to the findings of @xcite . their survey of queer individuals showed that mt has the most potential for representational and alloca-tional harms @xcite for non-cis users (compared to other nlp applications). in this context, survey respondents mentioned the translation of pronouns as particularly sensitive, as genderneutral pronouns might be translated into gendered pronouns, resulting in harmful misgendering. while individual studies have investigated the translation of established (gender-neutral) pronouns (e.g., from korean to english; @xcite , nlp research, in general, has ignored the "modern world of pronouns" as recently described by @xcite . they discuss the large variety of existing phenomena in english 3rd-person pronoun usage, with more traditional neopronoun sets (e.g., xe/xem) foot_0 and novel pronoun-related phenomena (e.g., nounself pronouns like vamp/vamp; miltersen, 2016), which possibly match distinct aspects of an individuals identity. as an example of ubiquitous nlp technology, truly inclusive mt should account for linguistic varieties that express identity aspects, like the large spectrum of pronouns related to the social push to respect diverse identities. however, until now, (a) there has been no information on how our systems (fail to) handle this linguistic shift, and (b) it is unclear how mt should deal with novel pronouns. this case is especially challenging when source language pronouns do not have direct correspondences in the target language. contributions. in this "reality check", we investigate the handling of various (neo)pronouns in mt for advancing inclusive nlp. to this end, we combine an extensive analysis of mt performance across six languages (danish, english, farsi, french, german, and italian) and three commercial mt engines (bing, deepl, and google translate) with results from the largest survey on pronoun us-age among queer individuals in ai to date. we answer the following four research questions (rqs): ## related work we review works on gender bias in mt and the broader area of (gender) identity inclusion in nlp. for a thorough survey on gender bias in mt, we refer to @xcite . ## the status quo to shed light on the state of identity inclusion through 3rd person pronouns in commercial mt, we conduct a thorough error analysis when translating from english (en) to five diverse languages. we further describe an experiment opposite to this, translating from danish (da) to en, in §3.3. ## conclusion in this work, we have investigated the sensitivity of automatically translating pronouns: small words that can convey important identity aspects. to understand where current commercial mt stands with regards to this issue, we started with a thorough error analysis covering six languages and three mt engines. we demonstrated that the engines tested are more likely to produce low-quality output when starting from gender-neutral pronouns, and we further observed a high potential for misgendering. emphasizing marginalized voices, we complemented our study with a survey of affected individuals. the answers led us to three recommendations for more inclusive mt. we hope our study will inform and fuel more research on these issues.