Gender Gap in Natural Language Processing Research: Disparities in Authorship and Citations (2020)

Abstract: Disparities in authorship and citations across gender can have substantial adverse consequences not just on the disadvantaged genders, but also on the field of study as a whole. Measuring gender gaps is a crucial step towards addressing them. In this work, we examine female first author percentages and the citations to their papers in Natural Language Processing (1965 to 2019). We determine aggregate-level statistics using an existing manually curated author–gender list as well as first names strongly associated with a gender. We find that only about 29% of first authors are female and only about 25% of last authors are female. Notably, this percentage has not improved since the mid 2000s. We also show that, on average, female first authors are cited less than male first authors, even when controlling for experience and area of research. Finally, we discuss the ethical considerations involved in automatic demographic analysis.

## introduction gender gaps are quantitative measures of the disparities in social, political, intellectual, cultural, or economic success due to one's gender. gender gaps can also refer to disparities in access to resources (such as healthcare and education), which in turn lead to disparities in success. we need to pay attention to gender gaps not only because they are inherently unfair but also because better gender balance leads to higher productivity, better health and well-being, greater economic benefits, better decision making, as well as political and economic stability (skjelsboek and smith, 2001; woetzel et al., 2015; hakura et al., 2016; mehta et al., 2017; gallego and gutiérrez, 2018). historically, gender has often been considered binary (male and female), immutable (cannot change), and physiological (mapped to biological sex). however, those views have been discredited (hyde et al., 2019; richards et al., 2017; darwin, 2017; lindsey, 2015; kessler and mckenna, 1978). gender is complex, and does not necessarily fall into binary male or female categories (e.g. nonbinary people), and also does not necessarily correspond to one's assigned gender at birth. society has often viewed different gender groups differently, imposing unequal social and power structures (lindsey, 2015). the world economic forum's 2018 global gender gap report (which examined data from more than 144 countries) highlighted the gender gap between men and women in artificial intelligence as particularly alarming (wef, 2018). it indicated that only 22% of the professionals in ai are women and that this low representation in a transformative field requires urgent action-otherwise, the ai gap has the potential to widen other gender gaps. other studies have identified substantial gender gaps in science (håkanson, 2005; larivière et al., 2013; king et al., 2017; andersen and nielsen, 2018). perez (2019) discusses, through numerous examples, how there is a considerable lack of disaggregated data for women and how that is directly leading to negative outcomes in all spheres of their lives, including health, income, safety, and the degree to which they succeed in their endeavors. this holds true even more for transgender people. our work obtains disaggregated data for female natural language processing (nlp) researchers and determines the degree of gender gap between female and male nlp researchers. (nlp is an interdisciplinary field that includes scholarly work on language and computation with influences from artificial intelligence, computer science, linguistics, psychology, and social sciences to name a few.) we hope future work will explore other gender gaps (e.g., between trans and cis people). measuring gender gaps is a crucial step towards addressing them. we examine tens of thousands of articles in the acl anthology (aa) (a digital repository of public domain nlp articles) for disparities in female authorship. @xcite we also conduct experiments to determine whether female first authors are cited more or less than male first authors, using citation counts extracted from google scholar (gs). we extracted and aligned information from the acl anthology and google scholar to create a dataset of tens of thousands of nlp papers and their citations as part of a broader project on analyzing nlp literature. @xcite we refer to this dataset as the nlp scholar dataset. we determined aggregatelevel statistics for female and male researchers in the nlp scholar dataset using an existing manually curated author-gender list as well as first names that are strongly associated with a gender. note that attempts to automatically infer gender of individuals can lead to harm (hamidi et al., 2018). our work does not aim to infer gender of individual authors. we use name-gender association information to determine aggregate-level statistics for male and female researchers. further, one may not know most researchers they cite, other than from reading their work. thus perceived gender (from the name) can lead to unconscious effects, e.g., dion et al. (2018) show that all male and mixed author teams cite fewer papers by female authors than all female teams. further, seeing only a small number of female authors cited can demoralize young researchers entering the field. we do not explore the reasons behind gender gaps. however, we will note that the reasons are often complex, intersectional, and difficult to disentangle. we hope that this work will increase awareness of gender gaps and inspire concrete steps to improve inclusiveness and fairness in research. it should also be noted that even though this paper focuses on female-male disparities, there are many aspects to demographic diversity including: representation from transgender people; representation from various nationalities and race; representation by people who speak a diverse set of languages; diversity by income, age, physical abilities, etc. all of these factors impact the breadth of technologies we create, how useful they are, and whether they reach those that need it most. resources for the nlp scholar project can be accessed through the project homepage. @xcite 2 related work pilcher (2017) shows that names function not only to identify individuals, but also to manage gender throughout one's life. there is a strong cultural norm in various parts of the world to assign a first name to newborns as per their category of sex (pilcher, 2017; barry iii and harper, 2014; lieberson et al., 2000; alford, 1987). pilcher (2017) argues that, throughout their life, the first name plays a role in repeatedly categorizing a person as being male or female. people may change their name and appearance to manage their gender (connell, 2010; pilcher, 2017). people may choose a name that is not associated with male or female categorizations (connell, 2010). the strong normative tendency to use names to signal gender has led to a large body of work on automatically determining gender by one's first name, not just for scientometric analysis discussed below, but also for language studies, social sciences, public health, and commerce. however, this can also lead to misgendering, which can cause significant pain and harm (hamidi et al., 2018). (misgendering is when a person-or in this case, a machineassociates someone with a gender with which they do not identify.) further, work that does not explicitly consider gender to be inclusive of trans people can reinforce stereotypes such as the dichotomy of gender. we expect gender disparities to be different depending on the groups being compared: femalemale, trans-cis, and so on. our work does not aim to infer gender of individual authors. we obtain disaggregated statistics for women, specifically, to study the disparities between female and male nlp researchers. we discuss ethical considerations further in section 6. see also mihaljević et al. (2019) for a discussion on ethical considerations in using author name to estimate gender statistics in the gender gap in science project-a large ongoing project tracking gender gaps in mathematical and natural sciences. @xcite most studies on gender and authorship have found substantial gender disparities in favor of male researchers. they include work on ∼1700 articles from journals of library and information science (håkanson, 2005), on ∼12 million articles from the web of science (for sociology, political science, economics, cardiology and chemistry) (ghiasi et al., 2016; andersen and nielsen, 2018), on ∼2 million mathematics articles (mihaljević-brandt et al., 2016), on ∼1.6 million articles from pubmed life science and biomedical research (mishra et al., 2018), on ∼1.5 million articles from fifty disciplines published in jstor (king et al., 2017), and on ∼0.5 million publications from us research universities (duch et al., 2012). there also exists some work that shows that in fields such as linguistics (lsa, 2017) and psychology (willyard, 2011), female and male participation is either close to parity or tilted in favor of women. in nlp research, schluter (2018) showed that there are barriers in the paths of women researchers, delaying their attainment of mentorship status (as estimated through last author position in papers). anderson et al. (2012) examine papers from 1980 to 2008 to track the ebb and flow of topics within nlp, and the influence of researchers from outside nlp on nlp. vogel and jurafsky (2012) examined about 13,000 papers from 1980 to 2008 to determine basic authorship statistics by women and men. gender statistics were determined by a combination of automatic and manual means. the automatic method relied on lists of baby names from various languages. they found that female authorship has been steadily increasing from 1980 to 2008. our work examines a much larger set of nlp papers (1965-2019), re-examines some of the questions raised in vogel and jurafsky (2012), and explores several new questions, especially on first author gender and disparities in citation. ## data we extracted and aligned information from the acl anthology (aa) and google scholar (gs) to create a dataset of tens of thousands of nlp papers and their citations. we aligned the information across aa and gs using the paper title, year of publication, and first author last name. details about the dataset, as well as an analysis of the volume of research in nlp over the years, are available in mohammad (2020b). we summarize key information below. ## gender gap in authorship we use the datasets described in §3.1 (especially aa* and aa**) to answer a series of questions on female authorship. since we do not have full selfreported information for all authors, these should be treated as estimates. first author is a privileged position in the author list that is usually reserved for the researcher that has done the most work and writing. @xcite in nlp, first authors are also often students. thus we are especially interested in gender gaps that effect them. the last author position is often reserved for the most senior or mentoring researcher. we explore last author disparities only briefly (in q1). q1. what percentage of the authors in aa are female? what percentage of the aa papers have female first authors (ffa)? what percentage of the aa papers have female last authors (fla)? how have these percentages changed since 1965? a. overall, we estimate that about 29.7% of the 28,682 authors are female; about 29.2% of the first authors in 37,297 aa* papers are female; and about 25.5% of the last authors in 39,368 aa** papers are female. figure 1 shows how these percentages have changed over the years. paper was published, then the paper is placed in bin 1; if the paper's first author had an academic age of 2 in the year when the paper was published, then the paper is placed in bin 2; and so on. the bins for later years contained fewer papers. this is expected as senior authors in nlp often work with students, and students are encouraged to be first authors. thus, we combine some of the bins in later years: one bin for academic ages between 10 and 14; one for 15 to 19; one for 20 to 34; and one for 35 to 50. once the papers are assigned to the bins, we calculate the percentage of papers in each bin that have a female first author. figure 3 shows the results. discussion: observe that, with the exception of the 35 to 50 academic age bin, ffa% is highest (30%) at age 1 (first year of publication). there is a period of decline in ffa% until year 6 (27.4%)-this difference is statistically significant (t-test, @xmath0.01). this might be a potential indicator that graduate school has a progressively greater negative impact on the productivity of women than of men. (academic age 1 to 6 often correspond to the period when the first author is in graduate school or in a temporary post-doctoral position.) after year 6, we see a recovery back to 29.4% by year 8, followed by a period of decline once again. q4. how does female first author percentage vary by area of research (within nlp)? which areas have higher-than-average ffa%? which areas have lower-than-average ffa%? how does ffa% correlate with popularity of an area-that is, does ffa% tend to be higher-or lower-than-average in areas where lots of authors are publishing? a. we use word bigrams in the titles of papers to sample papers from various areas. @xcite the title has a privileged position in a paper. primarily, it conveys what the paper is about. for example, a paper with machine translation in the title is likely about machine translation. figure 4 shows the list of top 66 bigrams that occur in the titles of more than 100 aa* papers (in decreasing order of the bigram frequency). for each bigram, the figure also shows the percentage of papers with a female first author. in order to determine whether there is a correlation between the number of papers corresponding to a bigram and ffa%, we calculated the spearman's rank correlation between the rank of a bigram by number of papers and the rank of a bigram by ffa%. this correlation was found to be only 0.16. this correlation is not statistically significant at @xmath1.01 (two-sided p-@xmath2.2). experiments with lower thresholds (174 bigrams occurring in 50 or more papers and 1408 bigrams occurring in 10 or more papers) also resulted in very low and non-significant correlation numbers (0.11 and 0.03, respectively). discussion: observe that ffa% varies substantially depending on the bigram. it is particularly low for title bigrams such as dependency parsing, language models, finite state, context free, and neural models; and markedly higher than average for domain specific, semantic relations, dialogue system, spoken dialogue, document summarization, and language resources. however, the rank correlation experiments show that there is no correlation between the popularity of an area (number of papers that have a bigram in the title) and the percentage of female first authors. to obtain further insights, we also repeat some of the experiments described above for unigrams in paper titles. we found that ffa rates are relatively high in non-english european language research such as papers on russian, portuguese, french, and italian. ffa rates are also relatively high for work on prosody, readability, discourse, dialogue, paraphrasing, and individual parts of speech such as adjectives and verbs. ffa rates are particularly low for papers on theoretical aspects of statistical modelling, and for terms such as wmt, parsing, markov, recurrent, and discriminative. research articles can have impact in a number of ways-pushing the state of the art, answering crucial questions, finding practical solutions that directly help people, etc. however, individual measures of research impact are limited in scope-they measure only some kinds of contributions. the most commonly used metrics of research impact are derived from citations including: number of citations, average citations, h-index, and impact factor (bornmann and daniel, 2009). despite their limitations, citation metrics have substantial impact on a researcher's scientific career; often through a combination of funding, the ability to attract talented students and collaborators, job prospects, and other opportunities in the wider research community. thus, disparities in citations (citation gaps) across demographic attributes such as gender, race, and location have direct real-world adverse implications. this often also results in the demoralization of researchers and marginalization of their workthus negatively impacting the whole field. therefore, we examine gender disparities in citations in nlp. we use a subset of the 32,985 aa papers ( §3.2) that were published from 1965 to 2016 for the analysis (to allow for at least 2.5 years for the papers to collect citations). there are 26,949 such papers. ## conclusions we analyzed the acl anthology to show that only ∼30% have female authors, ∼29% have female first authors, and ∼25% have female last authors. strikingly, even though some gains were made in the early years of nlp, overall ffa% has not improved since the mid 2000s. even though there are some areas where ffa% is close to parity with male first authorship, most areas have a substantial gap in the numbers for male and female authorship. we found no correlation between popularity of research area and ffa%. we also showed how ffa% varied by paper type, venue, academic age, and area of research. we used citation counts extracted from google scholar to show that, on average, male first authors are cited markedly more than female first authors, even when controlling for experience and area of work. thus, in nlp, gender gaps exist both in authorship and citations. this paper did not explore the reasons behind the gender gaps. however, the inequities that impact the number of women pursuing scientific research (roos, 2008; foschi, 2004; buchmann, 2009) and biases that impact citation patterns unfairly (brouns, 2007; feller, 2004; gupta et al., 2005) are well-documented. these factors play a substantial role in creating the gender gap, as opposed to differences in innate ability or differences in quality of work produced by these two genders. if anything, past research has shown that self-selection in the face of inequities and adversity leads to more competitive, capable, and confident cohorts (nekby et al., 2008; hardies et al., 2013).