Gender-Aware Reinflection using Linguistically Enhanced Neural Models (2020)

Abstract: In this paper, we present an approach for sentence-level gender reinflection using linguistically enhanced sequence-to-sequence models. Our system takes an Arabic sentence and a given target gender as input and generates a gender-reinflected sentence based on the target gender. We formulate the problem as a user-aware grammatical error correction task and build an encoder-decoder architecture to jointly model reinflection for both masculine and feminine grammatical genders. We also show that adding linguistic features to our model leads to better reinflection results. The results on a blind test set using our best system show improvements over previous work, with a 3.6% absolute increase in M2 F0.5.

## introduction the recent advances in machine learning have propelled the field of natural language processing (nlp) forward at a great pace and raised expectation about the quality of results and especially their impact in a social context, including not only race @xcite and politics @xcite , but also gender identities @xcite @xcite . human-generated data, reflective of the gender discrimination and sexist stereotypes perpetrated through language and speaker's lexical choices, is considered the primary source of these biases @xcite . however, @xcite pointed out that nlp gender biases do not just exist in human-generated training data, and models built from it; but also stem from gender-blind (i.e., gender-unaware) systems designed to generate a single text output without considering any target gender information. such systems propagate the biases of the models they use. one example is the i-ama-doctor/i-am-a-nurse problem in machine translation (mt) systems targeting many morphologically input gender target masculine target feminine âryd hlwla sryς b âryd hlwla sryς âryd hlwla sryς i want quick solutions i want quick solutions i want quick solutions lânny amrâ šqra' f lânny rjl âšqr lânny amrâ šqra' because i am a blonde woman because i am a blonde man because i am a blonde woman âna sςyd blqaŷkm m âna sςyd blqaŷkm âna sςyd blqaŷkm i am happy [masc.] to meet you i am happy [masc.] to meet you i am happy [fem.] to meet you in contrast, gender-aware systems should be designed to produce outputs that are as gender-specific as the input information they have access to. gender information may be contextualized (e.g., the input 'she is a doctor'), or linguistically provided (e.g., the gender feature provided in the user profile in social media). but, there may be contexts where the gender information is unavailable to the system (e.g., 'the student is a nurse'). in such cases, generating both gender-specific forms is more appropriate. in this paper, we present an approach for sentence-level gender reinflection using linguistically enhanced sequence-to-sequence models. our system takes an arabic sentence and a given target gender as input and generates a gender-reinflected sentence based on the provided target gender. ## related work many nlp systems have the ability to embed and amplify societal (gender, racial, religious, etc.) biases across a variety of core tasks such as coreference resolution @xcite , machine translation @xcite @xcite @xcite @xcite , named entity recognition @xcite , dialogue systems @xcite , and language modeling @xcite . for the case of gender bias, various research efforts have shown that this could be caused by either human-generated training datasets @xcite , pre-trained word embeddings @xcite @xcite , or language models @xcite . to mitigate this problem, several researchers proposed approaches in which they focus mainly on debiasing word embeddings @xcite @xcite or using counterfactual data augmentation techniques @xcite @xcite . most of the solutions were mainly proposed to reduce gender bias in english and may not work as well when it comes to morphologically rich languages. nevertheless, there have been recent studies that explored the gender bias problem in languages other than english. @xcite studied gender bias which is exhibited by multilingual embeddings in four languages (english, german, french, and spanish) and demonstrated that such bias can impact cross-lingual transfer learning tasks. @xcite used a counterfactual data augmentation approach and developed a generative model to convert between masculine and feminine sentences in four languages (french, hebrew, italian, and spanish). for arabic, @xcite introduced a two-step approach to gender-identify and reinflect firstperson-singular constructions. the identification was done through a feature-based classifier, whereas they used a character-level sequence-to-sequence model for the reinflection. they also compared their two-step approach to a single-step joint identification and reinflection model, which under-performed in the case of the arabic source (not the machine translation source) task. all of their systems modeled grammatical masculine and feminine genders separately. in this paper, we compare to their results using the publicly available arabic parallel gender corpus they built -a parallel corpus of first-person-singular arabic sentences that are gender-annotated and reinflected. however, our work is different from theirs in that we jointly learn reinflection for both masculine and feminine genders together. we also model identification implicitly with reinflection in a single architecture. furthermore, we formulate the problem as a user-aware grammatical error correction task (ugec). as such, we use as our primary metric the maxmatch (m 2 ) scorer @xcite , which is far more meaningful than the bleu @xcite metric used by @xcite for this task. ## arabic linguistic background modern standard arabic (msa) nlp systems and more specifically those using deep learning, face several challenges when it comes to gender expression including morphological richness, orthographic ambiguity and noise. ## joint gender reinflection model in this section, we discuss the motivation behind our model architecture as well as the integration of the linguistic features. we also describe the training settings and the model's hyperparameters for reproducibility. ## experiments and evaluation in this section, we discuss the data we use to train and evaluate our models. we also discuss the evaluation metrics and the various systems we implemented including the baselines. ## results the results of our evaluation on the dev set are presented in the two aspects of our best system (being joint and using morphological features) are important to its performance. when we compare joint+morph to its joint counterpart, we observe an 5.6% absolute increase in the m 2 f 0.5 score and a corresponding 0.6% increase in identification f 1 score. this confirms that morphological features are helpful for both gender identification and reinflection. an ablation experiment comparing the best system joint+morph to the disjoint variant of it (dis-joint+morph) demonstrates the large added value of using a joint model: an 11.2% absolute increase in m 2 f 0.5 score, 0.45 bleu points , and 0.8% absolute improvement in identification f 1 score. the use of word embeddings was not helpful to our best system. one possible explanation is that the use of semantically oriented embeddings may not be optimal for fine-targeted rewriting tasks. the results on the test set using the baselines and the best system from the dev experiments are given in ## error analysis we conducted a manual error analysis examining all of the errors in the output of our best system on the dev set. in total, there were 106 sentences with errors (or 4.3% out of 2,448). in those erroneous sentences, there were 128 words with problems. around two thirds of the word errors were false negatives, i.e., where a change should have happened but did not ( returning to the rest of the errors, an additional quarter of them involved a false positive ( considering the detailed scores for the whole dev set and for m target and f target cases, we note the following. as expected, the f target setting has more errors than the m target setting. no change errors and gold errors are more common for the f target setting. the case form errors are only seen in the m target setting. errors with uninflectable words are almost equally present. these errors suggest that more work needs to be done on identifying when a reinflection should take place. furthermore, to address the errors of uninflectable forms and case-marked forms, we may have to incorporate more linguistic knowledge or more powerful language models. ## conclusion and future work in this paper, we proposed a solution to single-output nlp systems that allows users to specify their grammatical gender preference in arabic. our intention is to enable users to reduce the harm that may be produced by nlp systems propagation of biased representations. our joint approach for sentence-level gender reinflection uses linguistically enhanced sequence-to-sequence models and frames the problem as a user-aware grammatical error correction task. our system takes an arabic sentence and a given target gender as input and generates a gender-reinflected sentence based on the provided target gender. we showed that linguistic knowledge helps in learning gender identification implicitly which improves reinflection results. in future work, we would like to explore different architectures such as transformerbased models @xcite . furthermore, we are interested in exploring the added value of combining syntactic and morphological features. we would also like to apply our approach to different languages and dialectal varieties. lastly, we plan to extend the arabic parallel gender corpus beyond first-person-singular constructions and adapt our models accordingly.