Whose wife is it anyway? Assessing bias against same-gender relationships in machine translation (2024)

Abstract: Machine translation often suffers from biased data and algorithms that can lead to unacceptable errors in system output. While bias in gender norms has been investigated, less is known about whether MT systems encode bias about social relationships, e.g., “the lawyer kissed her wife.” We investigate the degree of bias against same-gender relationships in MT systems, using generated template sentences drawn from several noun-gender languages (e.g., Spanish) and comprised of popular occupation nouns. We find that three popular MT services consistently fail to accurately translate sentences concerning relationships between entities of the same gender. The error rate varies considerably based on the context, and same-gender sentences referencing high female-representation occupations are translated with lower accuracy. We provide this work as a case study in the evaluation of intrinsic bias in NLP systems with respect to social relationships.

## introduction machine translation (mt) is meant to achieve a faithful and fluent representation of a source language utterance in a given target language. while nlp research continues to improve the accuracy and robustness of mt systems @xcite , the full space of possible translation failures remains to be determined, particularly with respect to gender @xcite . mt systems often generate masculine-gender words as the default for gendered languages @xcite , e.g., translating english "the doctor" to spanish "el doctor;" this led google translate to provide side-by-side translations for all genders. focusing on word-based bias in mt is a good start, but translation systems may also exhibit grammatical bias involving relationships between words. in figure 1 , a sentence containing a same-gender relationship ("the lawyer kissed her wife") is re-translated as a sentence with a different-gender relationship ("his wife"), regardless of the starting language. this error seems to reveal the model's bias toward fluent translation at the cost of faithfulness @xcite , generating an output sentence with higher likelihood in the target language ("his wife") but a possibly inaccurate meaning for the source language. furthermore, this kind of grammatical error can only be brought to light by focusing on relationships between entities, an issue equally important as bias toward individual words like "doctor." addressing bias in translation of relationships is important for such social groups as lgbtq people, who often face discrimination for engaging in relationships with partners of the same gender @xcite . this study presents an analysis of the discrepancy in how translation systems handle same-gender vs. we release all relevant data and code to replicate the study under a creative commons license. foot_0 ## related work traditionally, research in ml-related bias has focused on well-established social demographics that are protected by law such as gender, race, and religion @xcite @xcite . while demographics are an important area of focus, many other facets of social identity can also be affected by bias @xcite , especially social relationships: power dynamics @xcite , friendship @xcite , and romance @xcite . a system that accurately processes such relationships has to understand not just individual identities (e.g., "man" and "woman") but also the social norms around the interactions between individuals (why two adults choose to live together) @xcite . while norms around social relationships vary widely between societies @xcite , it is reasonable to assume that nlp systems should treat romantic relationships as equally valid regardless of the demographics of the participants. furthermore, relationships represent an important part of social identity for many people @xcite , including lgbtq people whose self-image may be negatively impacted by stereotypes about their relationships @xcite . to fill the gap in the space of relationship-related bias, this study offers a path forward in assessing bias against with same-gender relationships in nlp systems. translating from one language to another is an inherently noisy process @xcite , sometimes leading to systematic errors that reveal inherent bias. machine translation systems have been extensively audited for bias in prior work, particularly with respect to gender @xcite @xcite and linguistic structure @xcite @xcite . methods for mitigating bias in machine translation range from retraining on a targeted clean datasets @xcite to modifying the model training/inference behavior for improved fairness @xcite . this work contributes to the discussion in mt-related bias by evaluating gender bias in the context of social relationships, a previously under-explored area. ## conclusion in this study, we identified consistent bias against same-gender relationships in mt among several romance languages. using google translate, we identified consistent bias against same-gender relationships, across language, topic, and subject type. upon further investigation, we found that occupations with higher income, higher female representation, and higher median age tend to exhibit higher rates of bias. future mt systems may need to change their training or inference strategy to represent a wider range of relationships. such a bias in mt systems can have a variety of downstream impacts, including misrepresentation of same-gender relationships across languages, enforcing normative social stereotypes, and erasing the lived experience of people who participate in same-gender relationships. future work should broaden the investigation of how relationships are processed in multilingual models, including coreference resolution (emelin and sennrich, 2021) and natural language inference @xcite , to provide a more complete picture into the representation of relationships with varying social composition. while our study does not address underlying issues facing lgbtq people such as legal discrimination, it does provide a way forward to identify implicit bias in nlp systems. we hope that the study encourages ai researchers to take a broader view of "ethics" when it comes to the design and evaluation of such systems as machine translation, in order to include minority groups who may not be considered visible @xcite . limitations we acknowledge that the study is limited to a sub-set of languages, due to the need for grammatical gender marked on np and unmarked on possessive pronouns. while this analysis is not appropriate for all languages, it can be adapted to fit other situations, e.g. identifying the inferred possessive pronoun when translating from a language without explicit possession marking (e.g. translating "she met ø wife" from norwegian; lødrup 2010) to a language with explicit possession marking. from a linguistic perspective, the study also only focuses on one direction of translation (gender-np to no-gender-np), even though the opposite direction (no-gender-np to gender-np) is known to exhibit gender bias @xcite . future studies should assess bias in multiple translation directions, as well as to/from languages without any grammatical gender such as chinese. the analysis of occupations ( § 3.3) uses statistics from the united states, which may not match the statistics of the countries in which the languages under study are spoken. we assume that the relative ranking of occupations by the social variables will not be significantly different between countries. this is a strong assumption to make for all occupations but is likely to hold for at least the most popular occupations: e.g., in many countries, a physician will earn more money than a nurse. we acknowledge that it's not a perfect measurement for the socioeconomic correlates of occupation and look to future work to develop more fine-grained metrics for occupation social status, e.g. relative female representation per-country per-occupation.