Neural Machine Translation Doesn’t Translate Gender Coreference Right Unless You Make It (2020)

Abstract: Neural Machine Translation (NMT) has been shown to struggle with grammatical gender that is dependent on the gender of human referents, which can cause gender bias effects. Many existing approaches to this problem seek to control gender inflection in the target language by explicitly or implicitly adding a gender feature to the source sentence, usually at the sentence level. In this paper we propose schemes for incorporating explicit word-level gender inflection tags into NMT. We explore the potential of this gender-inflection controlled translation when the gender feature can be determined from a human reference, or when a test sentence can be automatically gender-tagged, assessing on English-to-Spanish and English-to-German translation. We find that simple existing approaches can over-generalize a gender-feature to multiple entities in a sentence, and suggest effective alternatives in the form of tagged coreference adaptation data. We also propose an extension to assess translations of gender-neutral entities from English given a corresponding linguistic convention, such as a non-binary inflection, in the target language.

## introduction translation into languages with grammatical gender involves correctly inferring the grammatical gender of all entities in a sentence. in some languages this grammatical gender is dependent on the social gender of human referents. for example, in the spanish translation of the sentence 'this is the doctor', 'the doctor' would be either 'el médico', masculine, or 'la médica', feminine. since the noun refers to a person the grammatical gender inflection should be correct for a given referent. in practice many nmt models struggle at generating such inflections correctly @xcite , often instead defaulting to gender-based social stereotypes @xcite or masculine language @xcite . for example, an nmt model might always translate 'this is the doctor' into a sentence with a masculine inflected noun: 'este es el médico'. such behaviour can be viewed as translations exhibiting gender bias. by 'bias' we follow the definition from @xcite of behaviour which 'systematically and unfairly discriminate @xcite against certain individuals or groups of individuals in favor of others.' specifically, translation performance favors referents fitting into groups corresponding to social stereotypes, such as male doctors. such systems propagate the representational harm of erasure to referents -for example, a non-male doctor would be incorrectly gendered by the above example translation. systems may also cause allocational harms if the incorrect translations are used as inputs to other systems @xcite . system users also experience representational harms via the reinforcement of stereotypes associating occupations with a particular gender @xcite . even if they are not the referent, the user may not wish for their words to be translated in such a way that they appear to endorse social stereotypes. users will also experience a lower quality of service in receiving grammatically incorrect translations. a common approach to this broad problem in nmt is the use of gender features, implicit or explicit. the gender of one or more words in a test sentence is determined from external context @xcite or by reliance on 'gender signals' from words in the source sentence such as gendered pronouns. that information can then be used when translating. such approaches combine two distinct tasks: identifying the gender inflection feature, and then applying it to translate words in the source sentence. these feature-based approaches make the unstated assumption that if we could correctly identify that, e.g., the doctor in the above example should be female, we could inflect entities in the sentence correctly, reducing the effect of gender bias. our contribution is an exploration of this assumption. we propose a scheme for incorporating an explicit gender inflection tag into nmt, particularly for translating coreference sentences where the reference gender label is known. experimenting with translation from english to spanish and english to german, we find that simple existing approaches overgeneralize from a gender signal, incorrectly using the same inflection for every entity in the sentence. we show that a tagged-coreference adaptation approach is effective for combatting this behaviour. although we only work with english source sentences to extend prior work, we note that our approach can be extended to source languages without inherent gender signals like gendered pronouns, unlike approaches that rely on those signals. intuitively, if gender tagging does not perform well when it can use the label determined by human coreference resolution, it will be even less useful when a gender label must be automatically inferred. conversely, gender tagging that is effective in this scenario may be beneficial when the user can specify the gendered language to use for the referent, such as google translate's translation inflection selection @xcite , or for translations where the grammatical gender to use for all human referents is known. we also find that our approach works well with roberta-based gender tagging for english test sentences. existing work in nmt gender bias has focused on the translation of sentences based on binary gender signals, such as exclusively male or female personal pronouns. this excludes and erases those who do not use binary gendered language, including but not limited to non-binary individuals @xcite . as part of this work we therefore explore applying tagging to indicate gender-neutral referents, and produce a winomt set to assess translation of coreference sentences with gender-neutral entities. ## assessing and controlling gender inflection we wish to investigate whether a system can translate into inflected languages correctly correctly given the reference gender label of a certain word. our proposed approach involves fine-tuning a model on a very small, easily-constructed synthetic set of sentences which have gender tags. at test time we assign the reference gender label to the words whose gender inflection we wish to control. ## experiments we use baseline transformer models, bpe vocabularies, synthetic datasets and baseline rescoring gendered-alternative lattices from saunders and byrne (2020) foot_1 and follow the same adaptation scheme, assessing on english-to-german and english-to-spanish translation. we define gender tags as unique vocabulary items which only appear in the source sentence. we adapt to synthetic data with minibatches of 256 tokens for 64 training updates, which we found gave good results when fine-tuning on the s&b datasets. the v3 sets have about 30% more tokens, the v4 sets about 30% fewer and the neutral-augmented sets about 50% more: we adjust the adaptation steps accordingly for these cases. for all results we rescore the baseline system gendered-alternative lattices with the listed model. this constrains the output hypothesis to be a gender-inflected version of the original baseline hypothesis. lattice rescoring allows minimal degradation in bleu while letting gender inflections in the hypothesis translation be varied for potentially large winomt accuracy increases. for the gender-neutral experiments we add synthetic inflections and articles to the lattices. when assessing automatic test set tagging we use the roberta @xcite pronoun disambiguation function tuned on winograd schema challenge data as described in fairseq documentation foot_2 . we wish to improve coreference without loss of general translation quality, and so assess bleu on a separate, untagged general test set. for ease of comparison with previous work, we report general translation quality on the test sets from wmt18 (en-de) and wmt13 (en-es), reporting cased bleu using sacrebleu foot_3 @xcite . ## conclusions tagging words with target language gender inflection is a powerful way to improve accuracy of translated inflections. this could be applied in cases where the correct grammatical gender to use for a given referent is known, or as monolingual coreference resolution tools improve sufficiently to be used for automatic tagging. it also has potential application to new inflections defined for gender-neutral language. however, there is a risk that gender features will be used in an over-general way. providing a strong gender signal for one entity has the potential to harm users and referents by erasing other entities in the same sentence, unless a model is specifically trained to translate sentences with multiple entities. in particular we find that our v3 system, which is trained on multiple-entity translation examples, allows good performance while minimizing peripheral effects. we conclude by emphasising that work on gender coreference in translation requires care to ensure that the effects of interventions are as intended, as well as testing scenarios that capture the full complexity of the problem, if the work is to have an impact on gender bias.