A Rose by Any Other Name would not Smell as Sweet: Social Bias in Names Mistranslation (2023)

Abstract: We ask the question: Are there widespread disparities in machine translations of names across race/ethnicity, and gender? We hypothesize that the translation quality of names and surrounding context will be lower for names associated with US racial and ethnic minorities due to these systems’ tendencies to standardize language to predominant language patterns. We develop a dataset of names that are strongly demographically aligned and propose a translation evaluation procedure based on round-trip translation. We analyze the effect of name demographics on translation quality using generalized linear mixed effects models and find that the ability of translation systems to correctly translate female-associated names is significantly lower than male-associated names. This effect is particularly pronounced for female-associated names that are also associated with racial (Black) and ethnic (Hispanic) minorities. This disparity in translation quality between social groups for something as personal as someone’s name has significant implications for people’s professional, personal, and cultural identities, self-worth and ease of communication. Our findings suggest that more MT research is needed to improve the translation of names and to provide high-quality service for users regardless of gender, race, and ethnicity.

## introduction when people see their names incorrectly displayed by the technologies they use, this creates an unnecessary burden for them. over time, the regular experience of an ai system getting a person's name wrong can have insidious detrimental effects such as the erosion of cultural identity and self-worth, similar to effects from racial microagressions such as name mispronunciations experienced in the classroom @xcite . such experiences have the potential to be source: journee es una poeta británica de fuerza, claridad y oficio honesto. translation: journee is a british poet of force, clarity and honest craft. mt output: girls are a british poet of strength, clarity and honest craft. error(s): name completely mistranslated source: приятно с вами познакомиться, амия амия амия. translation: pleased to meet you amiyah. mt output: i'm happy to meet you mom. error(s): figure 1 : two hypothetical input sentences, originally in spanish and russian, together with correct english translations as well as the output of mt systems from our study; both mistranslate a person's given name. in this paper, we ask: are there disparities in the accuracy of machine translations of names across race, ethnicity, and gender? we define and evaluate a particular form of bias through inequality in the machine translations of names. to achieve this, we constructed a dataset, diverse names in context (dnic), of english sentences, by combining templates with names that are known to be strongly associated with particular social groups (section 4). using this dataset, we develop a procedure for analyzing the quality of machine translation systems' ability to correctly translate names and their contexts based on round-trip translation from and to english, through spanish, russian, arabic and chinese (section 5). we find that name translation errors across three state-of-the-art machine translations systems are inequitably impacting those with ethnic minority names, particularly black (odds ra-@xmath0.69) and hispanic females (odds @xmath1.62), at a much higher rate than white males (section 6). ## sociolinguistic background given names tend to be highly personal, and also reflective of a person's position in social groups, often with strong associations to a particular gender, a particular race or ethnicity, a social class, a religion, and more @xcite . beyond simply marking someone's identity, given names also signal and reinforce individuality and the social position of the person being referenced @xcite . due to the importance of given names, their regular mistranslation can potentially have substantial negative impacts on a person's life. allocative harms @xcite can arise either because 1) some people are experiencing worse system behavior despite paying the same for the use of a system, or 2) when professional communication (e.g., in job-seeking or recognition of their work) is made more difficult due to lower quality translations of their names and sentences written about them. name mistranslation also gives rise to significant representational harms, including, at least, harms related to quality of service, denial of self-identity, and erasure and alienation of both the name and its associated culture @xcite . quality-of-service harms resulting from name mistranslation are clear-if a machine translation system works less well for one person's name, or names strongly associated with marginalized social groups, than others, those people and those groups experience a less effective machine translation system. this is particularly problematic when the user is less likely to be able to identify translation errors, such as when the target language uses a different alphabet than the source. this type of harm parallels those studied by dyal-chand (2021) in the context of incorrect auto-complete for some names over others, where names are corrected to more anglicized versions or translated to nouns. dyal-chand (2021) emphasizes that to mitigate the danger of reinforcement of structural racism through technologies, we must acknowledge the role that they have on all individuals stating that everyone has "... a right to full and equal use of these technologies". denial of self-identity (the inability to express oneself as desired), erasure (a system's routine removal of traces of some social groups), and alienation (a system marking some social groups as other) arise due to the close ties between one's name and one's culture and community. @xcite emphasize how racial microaggressions, such as teachers' mispronunciation or anglicization of of names, often leave people of color feeling diminished, feeling like they or their culture aren't valued, and presure them to take responsiblity to make it easier for others to pronounce their names. when viewed through the lens of ai systems, this pressure leads to a form of "language standardization" in technology (pérez-quiñones and salas, 2021), a form of language ideology that can be used to justify social hierarchies (e.g., @xcite . because names can code for a multiplicity of social axes (gender, race, ethnicity, religion, social class, birth generation, etc.), it is important to study the impact of name mistranslation not just on a single social axis, but also based on how axes intersect. intersectionality is a core concept in black feminism, introduced in the combahee river collective statements 1977; 1983: "because the intersectional experience is greater than the sum of racism and my name is ___. where are you from ___. ⋮ ___ is a british poet of force, clarity and honest craft. aaliyah f black ⋮ ⋮ ⋮ journee f black ⋮ ⋮ ⋮ zachariah m white journee es una poeta británica de fuerza, claridad y oficio honesto. journee is a british poet of force, clarity and honest craft. girls are a british poet of strength, clarity and honest craft. mt (esp) mt (eng) evaluation name @xmath2.42 social @xmath3, female roundtrip translation templates baby names sexism, any analysis that does not take intersectionality into account cannot sufficiently address the particular manner in which black women are subordinated." intersectionality was applied in a legal setting by @xcite to analyze the ways in which u.s. antidiscrimination law fails black women. in our context, we analyze name mistranslation across a combined race/ethnicity axis (white non-hispanic, black non-hispanic, hispanic, and asian or pacific islander), and a binary gender axis (male, female), with the concomitant limitations arising from that coarse categorization. ## approach overview in figure 2 , we provide a visual depiction of our process flow for an example english sentence (template plus name) in the context of our contributions. our goal was to assess if names get mistranslated at different rates across race/ethnicity and gender groups in realistic contexts. thus, our steps were: 1) names: we collected a list of names strongly associated with race/ethnicity and gender; 2) instantiated english sentences (names plus templates): we then situated the names in sentence contexts (i.e., templates) that we thought would be found in the "real world"; 3) round-trip translations (rt): then we translated the complete english sentences (template + name) into four languages and back to english. we used rt translation to see if the final name in the translation output would be the same as the original name, as well whether the template portion remained in tact; finally, 4) translation evaluation approach: with our resulting rt translations, we could ascertain the impact of the combinations of race/ethnicity and gender of the names on the translations by assessing the strength and significance of our measures (of the correctness of name translations, and of sentence translation quality as a whole). we used mixed effects statistical analysis to quantify these effects. we describe each of these components in the context of the dataset we developed next. ## diverse names in context dataset to study the effect of mistranslating names on women and ethnic minority populations in the united states, we construct the dnic dataset by collecting a diverse set of names and english sentence templates where name translation particularly matters. instantiating these templates with different names let us isolate the effect of changing just the name. this builds on many existing datasets that seek to measure the potential harm caused by nlp systems across different social groups using templates instantiated with different names, pronouns, or other variables @xcite @xcite . ## methodology we now describe how we use mt to translate the dnic dataset (section 5.1), and introduce our approach to automatically evaluate the resulting name translations (section 5.2) and conduct a statistical analysis of the impact of social groups on name translation quality (section 5.3). ## results to answer our primary research question, "are there widespread disparities in machine translations of names across race/ethnicity, and gender?", we analyzed our results to determine if names are mistranslated at higher rates for certain groups, and investigated any accompanying translation quality degradation of the entire sentence. we show the coefficients on the fixed effects including gender, race/ethnicity, and the interactions between them. we also controlled for the random effects of template id, translation language, and mt system. ## conclusion we explored how race/ethnicity and gender impact both the odds of obtaining a correctly translated name as well as the translation quality of a sentence. to accomplish this, we introduced a roundtrip-translation-based approach to evaluate the machine translation of names in context, as well as a dataset of diverse names in context based on names highly associated with birth-derived gender and birth parent race/ethnicity. our results showed that female-associated names are likely to be mistranslated at significantly higher rates than male-associated names. our intersectional analysis further demonstrates that black female-and hispanic female-associated names are mistranslated 30-40% more frequently than white male-associated names. furthermore, black and hispanic female-associated names are associated with significantly worse overall translation quality as measured by bleu. these results hold, even when controlling for the template, mt system, and pivot language. we have not performed a root cause analysis to determine from where the name translation disparities we observed arise, but expect that they may arise in part due to names of certain social groups being underrepresented in the training data upon which machine translation and large language models are trained @xcite . in addition, proper names, cross-linguistically, may have (for instance) inflectional forms by gender, or derivative forms originating from verbs; the way these forms are reflected in the training data may also influence mistranslation. this work highlights the need to mitigate the bias that vulnerable social groups experience both personally and economically by seeing their names incorrectly displayed in translations. despite the potential for harm arising from mistranslating names, as well as the fact that names are often translated differently from other word types, there has been relatively little work in machine translation that focuses specifically on names. @xcite is one example, which focuses specifically on the automated translation of rare proper names. recognizing that some names should be translated and some should be transliterated, @xcite build a classifier to predict which is which, and to apply a different model for the two types. we hope that this work highlights the danger that name mistranslations could have resulting in longterm harms (section 2). given our findings, and in light of historical evidence that language technology has exacerbated harms for vulnerable populations (e.g., @xcite @xcite , we suggest that lower translation accuracy for names of people from vulnerable social groups risks their experiencing both allocative harms @xcite -such as poorer quality of mt system service or poorer professional communications when mt systems incorrectly translate their names (or sentences including their names)and representational harms such as erasure of the name and its associated culture @xcite .