Is there Gender Bias in Dependency Parsing? Revisiting “Women’s Syntactic Resilience” (2024)

Abstract: In this paper, we revisit the seminal work of Garimella et al. 2019, who reported that dependency parsers learn demographically-related signals from their training data and perform differently on sentences authored by people of different genders. We re-run all the parsing experiments from Garimella et al. 2019 and find that their results are not reproducible. Additionally, the original patterns suggesting the presence of gender biases fail to generalize to other treebank and parsing architecture. Instead, our data analysis uncovers methodological shortcomings in the initial study that artificially introduced differences into female and male datasets during preprocessing. These disparities potentially compromised the validity of the original conclusions.

## introduction nlp tools are commonly trained on textual corpora with authorship imbalances. for instance, since journalists are predominantly male foot_0 , corpora derived from newspaper articles are largely written by men @xcite . similarly, wikipedia, a major resource for training nlp models @xcite , is edited by a predominantly white and male group of contributors @xcite . this lack of diversity among authors can diminish the representation of minority voices @xcite and lead to models that inherently mirror demographic imbalances @xcite . @xcite was among the first to demonstrate how authorship imbalances can affect foundational nlp tasks like part-of-speech tagging and dependency parsing. 2 the authors trained models on sentences authored by females and males, observing error disparities in their results. 3 they found that models trained on maleauthored sentences performed best on male test data, whereas models trained on a gender-balanced dataset yielded better results on female test data. these findings led them to conclude that sentences written by women exhibit greater "diversity" and complexity, which are better captured when training data includes contributions from both genders. in contrast, sentences by men showed less syntactic variability, resulting in decreased performance when female-authored sentences were included in the training set. due to the heavy gender imbalance in the dataset (1:3 female to male authors), the authors concluded that the syntax of sentences written by women showed resilience despite the allocation bias, while men "lucked out" by having more training examples to boost accuracy. the findings of @xcite brought attention to the problem of gender bias in nlp models. the work was widely cited, for example, in the following work by the same authors @xcite , influential surveys @xcite @xcite , and most importantly, as an argument that gender bias exists in nlp on the grammatical level @xcite . however, despite its significance, the study has notable deficiencies. its scope is limited to english and only on a single parsing architecture. moreover, the evaluation methodology lacks any report of statistical significance testing on the results. given the minor differences in the obtained accuracy and the non-deterministic nature of neural models @xcite , there is a potential that the findings of @xcite could be attributed to chance. to advance our understanding of potential gen-der biases in foundational nlp tasks such as partof-speech tagging and dependency parsing, it is crucial to establish a well-defined foundation. therefore, in this paper, we revisit @xcite and aim to answer three research questions: are the results presented in garimella et al. (2019) reproducible and statistically significant? rq2 do garimella et al.'s (2019) results generalize to other languages and parsing architectures? rq3 what other factors, if not gender bias, could have been captured by their work? we begin by replicating garimella et al.'s ( 2019 ) methodology and rerunning their experiments ( §3). interestingly, our findings do not support the original claims regarding biases ( §4). further tests on the generalizability of these claims to a different language and parsing architecture also fail to replicate the original patterns. our data analysis uncovers a small yet significant methodological flaw in the original study that can be responsible for the original results ( §5). consequently, we urge the gender bias research community to approach the results of @xcite with caution. moving forward, we recommend focusing more on specific syntactic differences related to demographic variations and their impact on model performance rather than relying solely on average scores, which can be misleading. ## bias statement according to the predictive bias framework proposed by @xcite , the gender bias discussed in this paper is a form of selection biaseffects from the compositions of training data and their influence on downstream tasks. this selection bias manifests as error disparity, where models perform inconsistently across data from different demographic groups. while our focus is on dependency parsing, it is challenging to identify immediate, concrete harms directly caused by this bias. however, any subsequent applications that rely on these dependency parsers, such as authorship profiling based on syntactic trees @xcite , could be affected. depending on the specific application of the downstream task, this could lead to allocation or representation harms, where one demographic group might be unfairly treated or misrepresented due to biased model performance @xcite . for our experiments, we require sentences annotated with the gender of their authors, along with gold-standard syntactic trees. to the best of our knowledge, we use the only two treebanks available that meet these criteria. these datasets categorize gender in binary terms, limiting our analysis to female and male authors. we recognize that this limitation excludes non-binary individuals, contributing to recognition bias against them. ## experimental setup we extend the experimental framework from @xcite by incorporating additional data, parsing architectures, and robust evaluation. ## parsing results we start by answering rq1 -are the results from @xcite reproducible? for easier comparison, we repeat the original findings in ## data analysis if not gender bias, what was captured by the models of @xcite ? to answer rq3, we perform analysis of our training datasets. ## conclusion in this paper, we revisited the seminal work on gender bias by @xcite . our analysis demonstrated that their findings do not generalize to other languages or parsing architectures and, more critically, are not reproducible even with the same parsing architecture and dataset as the original study. a consistent observation from our work was that sentences written by females were easier to parse than those written by males. however, this pattern was due to a methodological oversight in the original study, where the preprocessing step inadvertently produced longer male sentences. as sentence length correlates with more complex tree structures, such as long arcs and dependents far from the root, this error introduced artificial parsing difficulty. coupled with our inconsistent statistical significance results across various applications, these findings challenge the validity of the gender bias claims made by @xcite .