N eu T ral R ewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender Neutral Alternatives (2021)

Abstract: Recent years have seen an increasing need for gender-neutral and inclusive language. Within the field of NLP, there are various mono- and bilingual use cases where gender inclusive language is appropriate, if not preferred due to ambiguity or uncertainty in terms of the gender of referents. In this work, we present a rule-based and a neural approach to gender-neutral rewriting for English along with manually curated synthetic data (WinoBias+) and natural data (OpenSubtitles and Reddit) benchmarks. A detailed manual and automatic evaluation highlights how our NeuTral Rewriter, trained on data generated by the rule-based approach, obtains word error rates (WER) below 0.18% on synthetic, in-domain and out-domain test sets.

## introduction recent years have seen an increasing need for gender-neutral and inclusive language. this need is reflected, among others, by a surge in the use of singular they, foot_0 currently endorsed as part of apa style as the generic and gender-neutral pronoun. 2 within the field of natural language processing (nlp), there are various monolingual and bilingual use cases where gender neutral and inclusive language is appropriate, if not preferred due to e.g. ambiguity in terms of the gender of referents. section 3 provides a short outline of potential nlp use cases. to support these use cases, we present a rulebased and a neural approach to gender-neutral rewriting along with manually curated benchmarks, both of which we provide open-access/source. 3 first, a rule-based rewriter is implemented leveraging hand-written rules and an automatic error correction tool. next, a neural rewriter is trained on output generated by the rule-based rewriter to remove the need for extensive pre-processing and the reliance on computationally expensive tools such as dependency parsers. our manual and automatic evaluation show how the neural rewriter clearly improves over the rule-based approach with word error rates (wer) below 0.18% on synthetic, in-domain and out-domain test sets. the main contributions of our work can be summarized as follows : (i) winobias+, an open-source manually curated extension of winobias @xcite providing neutral alternatives for 3,167 sentences as well as a manually curated set of 1,000 natural sentences (domain : reddit, open-subtitles), (ii) open-source code for rule-based and neural neutral rewriters which can convert (binary) gendered english sentences into their gender neutral counterparts, (iii) a detailed manual and automatic evaluation of errors made by the rule-based and neutral rewriter on synthetic and natural data. ## related work recent years have seen an increase in research on gender and gender bias mitigation in nlp. while a relatively large body of research has focused on debiasing word embeddings (e.g., @xcite @xcite , our work is related to the generation of gender variants. we broadly distinguish between : (i) approaches that incorporate additional (meta-) information during training/testing allowing for a controlled generation of gender alternatives, and (ii) approaches that focus on gender rewriting. the synopsis will focus specifically on research related to the gender of human referents. within the field of machine translation (mt), @xcite ; vanmassenhove et al. (2019), and @xcite in-corporate meta-information in the form of gender tags on the source side to enable gender alternative target translations for ambiguous source sentences. @xcite propose a black-box approach by appending gender information to the target sentences using parataxis constructions at translation time. @xcite describe work on controlling linguistic features (a.o. gender) in neural mt by identifying and (de)activating the relevant neurons. they show that gender is the most difficult feature to control with a success rate of 21% using the top five identified neurons. @xcite uses a counterfactual data augmentation (cda) technique to augment data sets by creating gender alternative sentences to decrease gender bias. their approach consists of swapping gendered words with their male/female counterparts (e.g. he:she, father:mother...). their results indicate that a cda approach outperforms a simple word embedding debiasing technique @xcite . @xcite and @xcite present gender-aware reinflection models for arabic. using an arabic sentence and a target gender, the desired gender alternative is generated by re-inflecting the input. it is worth noting that all the previously described approaches focus on generating binary (female/male) gendered alternatives or translations, while our work focuses on generating genderneutral alternatives. as such, the work that is most closely related to ours is @xcite . their work is contemporaneous to our submission. 4 @xcite present a rule-based and neural rewriter for the generation of gender-neutral singular they sentences as well as an evaluation benchmark 5 of 500 parallel sentences (gendered and genderneutral) from five domains (twitter, reddit, movie quotes, jokes). their rule-based and neural rewriters are able to generate gender-neutral sentences with an error-rate below 1% (0.63% and 0.99% respectively). in terms of resources, compared to @xcite , we provide larger synthetic and natural benchmarks. in terms of performance, although complicated due to the lack of a publicly available benchmark, our models are seemingly better with error-rates of 0.52 (rule-based) and 0.02 (neural) on the most comparable benchmark, i.e. reddit data. ## use cases generating neutral alternatives for gendered sentences has applications for various monolingual language generation tasks (e.g. automatic responses), where (i) one does not want to assume the gender of the referents, or (ii) one wants to present the user with various options. similarly, in a bilingual setting, more specifically for mt, a neutral rewriter allows for the generation of gender neutral alternatives for genderless and gender-neutral source languages (hungarian, turkish, persian, swahili...) or null-subject source languages (spanish, chinese, arabic, bulgarian...). for illustration, example (1) and (2) demonstrate how genderneutral alternatives can be useful in bilingual settings. example (1) features a sentence in armenian using the epicene (gender-neutral) pronoun ' ' which can be either translated into 'he', 'she' or singular 'they'. en : he/she/they opened the door. 6 similarly, example (2) illustrates the possible translations of a null-subject source in spanish which can be translated as "works in a company". en : he/she works in a company. 7 en : they work in a company. as a pre-processing step, rewriting into neutral alternatives could be useful to debias training data and thereby its embeddings (see a.o., @xcite @xcite and/or to obfuscate sensitive 'gender' features from real user data facing automatic profiling systems @xcite @xcite . ## results & discussion both rewriters were (manually) evaluated on synthetic (winobias+) and natural (reddit, opensubs) evaluation benchmarks. ## conclusion this paper presents a rule-based and a neural gender-neutral rewriter for english. first, the rulebased approach was implemented, leveraging handwritten rules and an automatic error correction tool. using the rbr, we generated a parallel genderedto-neutral corpus on which an nmt system was trained. the nmt model removes the need for computationally expensive pre-processing steps and, according to the manual and automatic evaluation, outperforms the rbr on synthetic, in-domain and out-domain benchmarks. along with our openaccess/source code, we also provide three manually curated benchmarks for neutral rewriting. for now, the neutral rewriter is limited to english using 'singular they' and recommendations for gender neutral writing specific to the english language. it is, in theory, possible to extend this approach (or a similar one) to other languages. however, so far, few languages have a crystallized approach when it comes to gender-neutral pronouns and genderneutral word endings. in future work, we intend to explore potential applications of the neutral rewriters (e.g. gender debiasing of corpora). we furthermore plan to extend our work to gender-neutral rewriting targeting specific referents within a sentence to accommodate the gender preferences of individual referents.