Geographic Citation Gaps in NLP Research (2022)

Abstract: In a fair world, people have equitable opportunities to education, to conduct scientific research, to publish, and to get credit for their work, regardless of where they live. However, it is common knowledge among researchers that a vast number of papers accepted at top NLP venues come from a handful of western countries and (lately) China; whereas, very few papers from Africa and South America get published. Similar disparities are also believed to exist for paper citation counts. In the spirit of “what we do not measure, we cannot improve”, this work asks a series of questions on the relationship between geographical location and publication success (acceptance in top NLP venues and citation impact). We first created a dataset of 70,000 papers from the ACL Anthology, extracted their meta-information, andgenerated their citation network. We then show that not only are there substantial geographical disparities in paper acceptance and citation but also that these disparities persist even when controlling for a number of variables such as venue of publication and sub-field of NLP. Further, despite some steps taken by the NLP community to improve geographical diversity, we show that the disparity in publication metrics across locations is still on an increasing trend since the early 2000s. We release our code and dataset here: https://github.com/iamjanvijay/acl-cite-net

## introduction progress in science is accelerated by a sharing of ideas. however, there have been numerous instances in history where the predominance of one group of people in science, and the silencing of others, has led to the publication of harmful pseudoscience @xcite . particularly egregious examples include the publication of theories and ideas on racial hierarchy (plutzer, ♣ equal contribution. 2013), male superiority @xcite , gender binary @xcite , and eugenics @xcite . it has also been shown that a lack of inclusion in invention and discovery leads to fewer technologies for the excluded group. for example, @xcite show how fewer technologies and health products are designed for women and @xcite , @xcite and @xcite show how a number of language technologies are designed for only a small number of languages. in this paper, we explore geographic inclusion in natural language processing (nlp) research. our premise is that in a fair world, people have equitable opportunities to education, to conduct scientific research, and to publish, regardless of where they live. however, researchers in the field know that a vast number of papers accepted at top nlp conferences and journals come from a handful of western countries and (lately) china. on the other hand, very few papers with african and south american authors are published. further, the papers that get a majority of citations tend to be from a small number of institutions. highly funded universities and research labs also tend to garner greater early visibility for their papers. some of these papers might be cited more simply because the affiliate university or lab is perceived as prestigious @xcite . @xcite examined the growth of citation networks and showed that papers with more early citations are likely to be cited more in the future (the "rich get richer" phenomenon). citations received by a research article serve as one of the key quantitative metrics to estimate its impact. citations-based metrics, such as h-index @xcite , can have a considerable impact on a researcher's career, funding received, and future research collaborations. citation metrics are also commonly taken into consideration in determining university rankings and overall scientific outcomes from a country. thus, the degree of equity in citations across geographic regions can act as one of the barometers of fairness in research. furthermore, geographic location directly correlates to the languages spoken in an area. therefore, to increase the reach of nlp beyond high-resource languages, it is important to elevate the research pursued in languages from these under-represented regions. in this work, we investigate the impact of a researcher's geographic location on their citability for the field of nlp. we examine tens of thousands of articles in the acl anthology (aa) (a digital repository of public domain nlp articles), and generate citation networks for these papers using information from semantic scholar, to quantify and better understand disparities in citation based on the geographic location of a researcher. we consider a set of candidate factors that might impact citations received and perform both qualitative and quantitative analyses to better understand the degree to which they correlate with high citations. however, it should be noted that we do not explore the cause of citation disparities. reasons behind such location-based disparities are often complex, inter-sectional, and difficult to disentangle. through this work we aim at bringing the attention of the community to geographic disparities in research. we hope that work in this direction will inspire actionable steps to improve geographic inclusiveness and fairness in research. ## dataset as of january 2022, the acl anthology (aa) had 71,568 papers. 1 we extracted paper title, names of authors, year of publication, and venue of publication for each of these papers from the repository. further, we used information about the aa papers in semantics scholar foot_1 to identify which aa papers cite which other aa papers -the aa citation network. since the meta-information of the papers in aa and semantic scholar does not include the affiliation or location of the authors, we developed a simple heuristic-based approach to obtain affiliation information from the text of the paper. we refer to our dataset as the aa citation corpus. it includes the aa citation graph, author names, unique author ids (retrieved from semantic scholar), conference or workshop title, month and year of publication, and country associated with the author's affiliation. we make the aa citation corpus freely available. detailed steps in the construction of the citation network and the extraction of affiliated country information are described in the subsections below. ## disparity in citation based on location we use the aa citation corpus to answer a series of questions on disparity of publications and citations across geographic location. we start with a look at the number of publications from around the world, followed by an examination of their citations. q1. is there a disparity in the number of nlp publications across different countries? how does the amount of publications correlate with linguistic diversity? a. we used counts of papers from the aa citation corpus to determine the number of papers from each country, as visualized in figure 1 . for an even coarser view, we also examined a partition of the world into ten regions. 7 we calculated the total number of papers from each region by aggregating papers from all countries present in this region. we also aggregate citation counts of papers by region. discussion figure 1 shows huge disparities for the number of publications among countries. the western world which includes united states, canada, united kingdom, france, germany, etc. dominates the network with high publication count. on the other hand, most countries in africa, south america, eastern europe, south east asia, and middle east remain in the red zone with very few publications till date. when examining language diversity foot_7 (indicated by size of yellow dot), we see that countries in the red zone have the highest language diversity. higher linguistic diversity indicates larger number of different languages spoken in that geographic region. for example, the list of countries with the highest number of languages includes: indonesia (710), nigeria (524), india (453), and brazil ( 228 ). 9 more work on these languages is needed, by local researchers in partnership with the language communities. one recent effort in this regard is project masakhane, a grassroots organisation whose mission is to strengthen and spur nlp research in african languages, for africans, by africans. 10 this analysis showcases the huge disparity in the number of publication from each country. through the questions ahead, we further uncover geographic patterns in citations, across these mid-tier and top-tier publishing countries. q2. how has the citation count ("influence" of nlp research) of papers from different regions changed over the years? a. to study this question, we examine the following metric: mean citation count per paper for each country until certain year. formally, this metric can be defined as follows: m c (j,k) = @xmath0,k) indicates mean-citation count of country-j until year-k. c k (i) indicates citation count of paper-i until year-k. i @xmath1. p k indicates the set of all the papers published until year-k. to calculate this metric, we create multiple citation networks each containing papers published only until a certain year. for instance, a citation graph associated with year-2000 will only contain papers published until 2000 -with papers as nodes and references as edges. using such a citation network associated with year-k, we can then easily evaluate m c (j,k) for any country-j. in figure 2 , we plot the total citation count received by all the papers published by different regions of the world across across a span of ∼21 years from year 2000 to 2021. 11 similarly, in figure 3 , we plot the mean-citation metric for top-10 countries across these years. we restrict the analysis to just the top-10 countries for the sake of simplicity. also, note that in figures 2 and 3 , for each point in the x-axis, the corresponding point in the y-axis pertains to the citation graph composed of papers published up till that year. to understand how the influence of a paper changes across the years after its publication, for each country, we calculate the average number of citations its papers receive one year after publication, two years after publication, and so on. we will refer to this time span as the age of a paper. we do so by building citation networks for each year from 2000 till 2021 and aggregate the number of papers and citations for each age-value (no. of years since the paper was published). figure 4 shows the plot of average citation vs. age-of-paper. discussion figure 2 clearly depicts the huge disparities existing in citations received by different regions. further, the trends have largely remained the same over the years, with the notable exception that central asia has made moderate gains since 2013, whereas the gains made by south east asia around the 2010s, have diminished since about 2012. from figure 3 , we make following observations: firstly, the plot shows that us (by a large margin), as well as, uk and canada dominate the mean citation metric across the considered time-span. secondly, for these three countries as well as for germany, the growth-rate of the mean-citation metric is remarkably higher than that of other countries. this growth-rate is indicated by the slope of the curve. thirdly, china and germany have shown a significant increase in growth rate over the past eight years. overall, this plot highlights that papers from certain countries have had markedly higher citations (on average) compared to other countries, across the years. additionally, we also note that the gap in mean-citation metric across the top-10 countries has more than doubled (increasing from 6.31 to 15.11) in the past 21 years. this also indicates that much of the "influential" research in nlp from the past two decades is heavily concentrated in us, uk, and canada. we also explore median citation statistics in figure 11 (appendix a ). in contrast to the mean, difference between medians is relatively smaller (for instance, 6 for us vs 3 for india in year 2021). this suggests that countries such as us, uk, and canada have more very highly cited papers that push up their means. we also observe that the median has increased faster for countries like us, uk, and canada compared to india, spain, and france. although the differences in medians are smaller, they are suggestive of marked citation disparity among the publication from different countries. figure 4 shows that in the initial years after a paper's publication (1-2 years), the gap in meancitation between the most and least cited top-10 countries is ∼3.5, but it explodes to ∼23 after 15 years. this suggest that, even among the toppublishing countries, the influence of some countries (like india and spain) is surprisingly shortlived. whereas, papers from us, uk, and canada rapidly gain influence as the paper ages and their influence is long lasting. ## conclusion we compiled a large dataset of meta-information associated with nlp papers, that includes the country name associated with an author's affiliated institution, number of citations, and the papers cited by each paper. we use this dataset to systematically examine citation disparities across geographic regions. we show that substantial geographic disparities exist, even when controlling for factors such as venue of publication. authors from north america and western europe have published 65% of the total number of papers; whereas, authors from africa, south america, and south-eastern asia (combined) have published only 3.7% of the total papers. we also find markedly lower levels of cross-region author-author collaboration across the two sets of regions. even among the top-ten publishing countries, we find marked differences. very little research from countries such as india, spain, italy, and china is cited by their peers from the top-five publishing countries. citation gaps across genders and geographic regions occur due to various complex, intersectional, and structural reasons. this paper does not explore the reasons behind them. however, @xcite ; @xcite have examined some of the causes for geographic citation gaps in scientific literature such as self citation, co-location of institutions, and national funding for research & development. the above reasons are not exhaustive and determining the degree of impact of each reason is extremely difficult. further, the reasons can be different across different regions of the world. we hope that future qualitative research sheds greater light on these complexities. understanding them is crucial for greater inclusion in research. we also hope that such work will engender concrete actions to address citation gaps. initiatives taken by the nlp community toward diversity, inclusion, and collaboration, especially actions that encourage local nlp research across the globe, can make meaningful change and help address geographic inequities.