Multi-Task Learning for Improving Gender Accuracy in Neural Machine Translation (2021)

Abstract: Machine Translation is highly impacted by social biases present in data sets, indicating that it reflects and amplifies stereotypes. In this work, we study mitigating gender bias by jointly learning the translation, the part-of-speech, and the gender of the target language with different morphological complexity. This approach has shown improvements up to 6.8 points in gender accuracy without significantly impacting the translation quality.

## introduction in recent years, the awareness about the bias present in machine translation (mt) systems has increased in the scientific community, especially gender bias. gender is manifested differently in languages; gender bias problems occur when translating between languages with +various levels of morphology. there is bias when the system tends to translate according to gender roles, even when there is no ambiguity @xcite . surveys @xcite @xcite have recently shown the great efforts carried out by scientists towards resolving the problem of gender bias in nmt. tagging and additional context approaches @xcite @xcite have shown an improvement in translation accuracy when translating from english to languages with more complex morphology. domain adaptation techniques have proved to impact the performance of translation in @xcite . debiased pre-trained word embeddings have been leveraged in @xcite and have shown improvement in spanish translations. gender bias is mainly attributed to the already present bias in the data used to train mt systems @xcite . furthermore, it has been shown that models trained on these data tend to amplify further this bias @xcite . in this sense, research done aims to avoid or reduce this amplification, such as fine-tuning techniques with gender-balanced dataset corpus (costa-jussà and de jorge, 2020) or annotating the source language words of the training data with the gender of the target language words @xcite . these techniques have shown promising results regarding gender accuracy. in line with reducing the amplification of data, we propose to obtain a system capable of predicting the part-of-speech (pos) and the gender of the words of the target language, besides the translation. we expect that, by having more information about the output words, the system maintains the quality of the translation and is able to better predict the gender based on the context without falling so much into the stereotype. in general, the proposed configurations outperformed the baseline nmt system on gender prediction accuracy by up to 6.8% while retaining average translation performance. ## bias statement nowadays, we live in a more globalized and connected world, which leads society to use mt tools to communicate with different nationalities. the fact that standard translators present a gender bias harms society, helping to perpetuate certain stereotypes and prejudices. an example is the tendency of specific systems to generalize the different professions carried out by men and women @xcite . it is significantly more visible when one tries to translate from a gender-neutral language, such as english, to another with grammatical gender, such as spanish. in the first type, nouns have no grammatical gender, while in the other type, there is gender inflection in nouns, adjectives, verbs, etc. therefore, when translating from english to spanish, the system uses masculine or feminine inflections following stereotypes. such stereotypical errors occur both when the context indicates gender explicitly and when it does not. ## background in this section, we review the basics of multilinguality and multi-task learning in nmt, as well, as linguistic features and the framework to evaluate gender bias in mt. all these methodologies are later used in our proposed research. multilingual nmt. transformers @xcite have advanced nmt, giving the ability to pay more attention to multilingual nmt. the primary approach behind multilingual is to have the same model architecture to translate different language pairs @xcite . previous studies explore different design approaches for the model architecture, either partial sharing with shared encoder @xcite , shared attention @xcite , task-specific attention @xcite , shared parameters @xcite , full model sharing @xcite or independent encoder-decoders without sharing @xcite . in this paper, we adopt this architecture without sharing. multi-task learning nmt. multi-task learning @xcite trains the model on several corelated tasks. this training can lead to generalized improved performance and facilitate sharing representations @xcite . in the nmt context, injecting linguistic knowledge has been successful when training nmt with related tasks ( pos tagging, dependency parsing). this linguistic injection can lead to improving nmt generalization and translation quality, especially in low-resource scenarios @xcite @xcite . our work depends on adopting linguistic knowledge through training multi-tasks, besides nmt. we trained our nmt model with pos prediction and gender tagging tasks. ## proposed methodology previous work @xcite has shown that the language-specific multilingual nmt architecture proposed by @xcite outperformed the universal shared encoder-decoder architecture @xcite on gender bias evaluations. we chose this language-specific architecture as the baseline for all our experiments. given parallel data for a set of languages @xmath0, l 2 , ..l n with n languages and data for language pairs, the architecture consists of n encoders and n decoders, each of them specific for a single language. being l i encoder used for all translation directions involving l i as source language and l i decoder for all directions involving l i as target language. using the same encoders and decoders on several translation directions enforces a common intermediate representation for all encoders in the system. this method relies on parallel data only, without any additional linguistic information. our proposed modification to this architecture (see figure 1 ) adds a new linear projection layers to the decoders for each additional tagging task tag @xmath1. each task focuses on different linguistic aspects that may improve gender representation. it is known that multitasking allows models, by inductive bias @xcite , to learn a representation that contains features useful for all the involved tasks, improving its generalization capabilities. for each translation direction, the loss is computed as follows: where y are the translation logits, y is the reference target sentence, l tr is the cross-entropy loss for the translation task, t is the set of tagging logits for each of the k tagging tasks, l tag is the crossentropy loss over each tagging task and tag k (y) is the tagging function over the reference target. ## experimental framework in this section, we describe the details about the data and model parameters involved in our experiments. ## results in this section, we explore the improvements achieved by multi-task training, whether regarding the general translation accuracy or the gendered results. translation results. when looking at individual directions, we can see that training gender tagging task with nmt in english-to-german has improved. we can argue that german is a higher morphological language than english, and the gender tagging task helps the system inject more knowledge about gender in german, leading to better translation accuracy, up to 0.31 for the english-to-german pair compared to the baseline. training more than one task seems to be beneficial when the translation is between high morphological language pairs like french-to-spanish and french-to-german where french, german, and spanish are all gendered high morphological languages. french-to-english seems to also benefit from the pos and gender tagging tasks together. winomt results. winomt helps us investigate how the gender-biased entities and professions translated. the framework investigates the translations when english is translated to higher morphological languages; therefore, we show the winomt results from english to spanish, german and french. spanish seems to benefit from the pos and gender tagging tasks together, where the accuracy increased by 6.8 over the baseline with a lower difference between the correct translated masculine entities and the correct translated feminine entities; of ∆g 7.1. this is assured by the lower ratio of male vs. female predictions (m:f) of 2.16. french also has better accuracy having both tasks trained together, with 54.8 accuracy, which increases by six over the baseline. the ∆g is also lower in this case, with a value of 14.3. in both languages, the difference between stereotyped and non-stereotyped translations did not improve. the improvements are more related to the general accuracy. multitask training of gender seems to impact the stereotyped translations in the three languages; however, the general accuracy was not impacted that much by training the gender tagging task. pos tagging also appears to help disambiguation of gender from english to german, giving higher gender accuracy reaching 68.3 with a low difference of male and female correct predictions; ∆g of 3.5 and m:f of 1.749. ## conclusions in this paper, we have proposed and analyzed the use of multi-task learning in multilingual nmt. learning linguistic tagging simultaneously as multilingual helps mitigate gender bias while maintaining the average translation performance over the tested languages. more than the methodology that we are proposing, which is simple and effective, we would like to encourage the community to evaluate their methodologies not only in terms of translation quality, but also in terms of social bias mitigation.