Gender-Inclusive Grammatical Error Correction through Augmentation (2023)

Abstract: In this paper we show that GEC systems display gender bias related to the use of masculine and feminine terms and the gender-neutral singular “they”. We develop parallel datasets of texts with masculine and feminine terms, and singular “they”, and use them to quantify gender bias in three competitive GEC systems. We contribute a novel data augmentation technique for singular “they” leveraging linguistic insights about its distribution relative to plural “they”. We demonstrate that both this data augmentation technique and a refinement of a similar augmentation technique for masculine and feminine terms can generate training data that reduces bias in GEC systems, especially with respect to singular “they” while maintaining the same level of quality.

## introduction natural language processing (nlp) systems are well known to exhibit sensitivity to social characteristics, a sensitivity that may lead to harms for users interacting with these systems. in this work, we examine how nlp systems performing the task of grammatical error correction (gec) are sensitive to gender characteristics in english and how this sensitivity represents bias that harms users of these systems. we propose a new data augmentation technique to address bias related to singular they, and show how it can mitigate gender bias. we also show how this technique interacts with prior work on data augmentation methods to reduce bias. the expression of gender in english is complex, and to focus our study, we identify discrete biases that gec systems may exhibit. first, we build on other works on gender bias in nlp by examining how masculine and feminine terms impact the behavior of gec systems. discrepant behavior between sentences that contain one or the other is evidence of bias. second, we examine the behavior of gec systems with a gender-neutral pronominal paradigm in english commonly called singular they. the linguistic properties of this paradigm introduce additional kinds of biases relative to masculine or feminine pronouns. additionally, we make our evaluation datasets publicly available foot_0 . to prevent these adverse behaviors, we apply techniques to generate synthetic training data that address different aspects of these behaviors. first, we adopt counterfactual data augmentation (cda), which is used successfully to reduce bias in word embedding models @xcite , and apply it to the gec case. second, we introduce a new technique for generating training data with they pronouns that have unambiguously singular reference using insight from theoretical linguistics to target model bias with gender-neutral sentences. because these techniques are data-oriented, the innovations should theoretically generalize beyond the gec domain to other nlp tasks as well. our main contributions are: ## description of the data augmentation methods we use two data augmentation methods. first, we follow @xcite and others in swapping out feminine words for masculine words and vice versa. second, we propose a novel augmentation method for generating singular they data from sentences containing masculine and feminine pronouns. we treat singular they differently because language internal facts about english necessitate a separate treatment: unlike he and she, they has a second life as a plural pronoun. ## conclusion in this work, we developed a novel technique for data augmentation with sentences containing they that has an unambiguous singular reference and applied it to the gec case. we used this technique to help develop a dataset of singular they data to parallel data in the bea shared task dataset that has masculine and feminine pronouns, and with this, we show that gec systems display bias in their treatment of singular they sentences compared to sentences with masculine or feminine pronouns. additionally, we demonstrated that this technique could be used to reduce bias in gec systems by fine-tuning the gec system on the generated synthetic training data. because this technique is data-oriented, we believe that it has wider applications, and other nlp systems that display degraded performance with respect to singular they may benefit from being trained on data created through this technique.