Fine-tuning Neural Machine Translation on Gender-Balanced Datasets (2020)

Abstract: Misrepresentation of certain communities in datasets is causing big disruptions in artificial intelligence applications. In this paper, we propose using an automatically extracted gender-balanced dataset parallel corpus from Wikipedia. This balanced set is used to perform fine-tuning techniques from a bigger model trained on unbalanced datasets to mitigate gender biases in neural machine translation.

## introduction misrepresentation of individual communities in current datasets is causing severe disruptions in artificial intelligence applications. examples of this are a lower performance of speech recognizers for women than for men @xcite , a lower accuracy in face recognition for asian faces than american or european ones @xcite and an amplification of stereotypes in neural machine translation (nmt) @xcite . these challenges are at the core of natural language processing applications, and, in particular, many works are focusing on trying to solve gender biases @xcite . with this objective in mind, and in the specific context of nmt, we propose the use of balanced data sets to mitigate gender biases in a standard nmt system taking advantage of domain adaptation techniques. previous research in the area of nmt has proposed to either mitigate biases using debiased word embeddings @xcite and using contextual information @xcite or evaluating and measuring the amount of bias present in the translation @xcite . the closest work to ours is the one by @xcite where authors generate a small genderbalanced dataset and use elastic weight consolidation techniques to perform transfer learning and mitigate the consequences of training with unbalanced datasets. differently from this one, we use a larger nonsynthetic balanced dataset to perform fine-tuning on an unbalanced-dataset and evaluate the reduction of gender bias in the final translation. ## bias statement as proposed in previous work @xcite , we formulate the bias statement of our work. our work consists of studying the effects of using a gender-balanced dataset to mitigate gender biases in nmt. in the nmt context, we can define gender bias as incorrectly translating a gendered source word into a target word opposite gender, when no ambiguity exists. we can attribute this to datasets that are over-represented with a particular gender. as shown in previous work @xcite , there are representational harms in word embeddings such as demeaning women's ability to work in tech, e.g., man is to computer programmer as woman is to homemaker. the main concern is that training a system on unbalanced data will perpetuate these biases: first, on the methods built on top of these datasets, and second, people that use these systems will learn incorrect associations between words, unknowingly perpetuating these social biases. a system trained on balanced data is a first step in eliminating this representational harm, as there is the same number of instances between genders. to avoid stereotypical bias in professions, the next step would be to have the same professional distribution between genders, which can be achieved by gender-swapping the initial dataset. that way, the model will equally represent genders. beyond this, we point out the limitation of doing a binary representation of gender, not reflecting the lgtbq+ community. note that our work trains a word embedding and nmt model and does not aim to reflect reality. in the end, mitigating gender bias in artificial intelligence systems is a short-term solution that needs to be combined with higher-level long-term projects in challenging current social power, among other principles (d' @xcite . ## gender balanced dataset this section explains the procedure followed to obtain an english-spanish gender-balanced dataset, which uses the available gebiotoolkit @xcite , and it extracts (multi-)parallel corpus at the sentence level from the wikipedia biographies. the toolkit consists of 3 blocks: a corpus extractor, which provides a layer to transform, collect and select entries in the desired languages; (2) a corpus aligner, which finds the parallel sentences within a text and provides a quality check of the parallel sentences given a few restrictions; (3) a gender classifier which includes a filtering module that classifies the gender of the entry and outputs the final parallel corpus. hereinafter, we refer to this dataset as balanced. we quantify the amount of gender bias in the collected dataset due to gender bias in word embeddings. this quantification of bias is also compared to the case of word embeddings computed on the europarl corpus @xcite . ## use of domain adaptation techniques for gender bias mitigation in this section we use the gender-balanced dataset described in the previous section to mitigate the gender bias present in a standard mt system. we build the nmt system using the standard transformer @xcite on a large dataset. our idea is to use fine-tuning techniques with the balanced dataset on this baseline system. ## conclusions the motivation of our work lies in the hypothesis that the use of a gender-balanced dataset can diminish the gender bias in nmt systems. a small amount of bias when compared in absolute terms with other more massive datasets. note that we can use these representations by downstream applications with the ability to have little gender bias. then, after this analysis, we use fine-tuning techniques to reduce gender bias in a standard mt system. results show that even if our balanced dataset is from a different domain than the training and the test of the mt system, it does improve the translation quality (up to 2 bleu points), and it can mitigate the gender bias in a significant amount (up to a 12.5% accuracy).