We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields (2023)

Abstract: Natural Language Processing (NLP) is poised to substantially influence the world. However, significant progress comes hand-in-hand with substantial risks. Addressing them requires broad engagement with various fields of study. Yet, little empirical work examines the state of such engagement (past or current). In this paper, we quantify the degree of influence between 23 fields of study and NLP (on each other). We analyzed ~77k NLP papers, ~3.1m citations from NLP papers to other papers, and ~1.8m citations from other papers to NLP papers. We show that, unlike most fields, the cross-field engagement of NLP, measured by our proposed Citation Field Diversity Index (CFDI), has declined from 0.58 in 1980 to 0.31 in 2022 (an all-time low). In addition, we find that NLP has grown more insular—citing increasingly more NLP papers and having fewer papers that act as bridges between fields. NLP citations are dominated by computer science; Less than 8% of NLP citations are to linguistics, and less than 3% are to math and psychology. These findings underscore NLP’s urgent need to reflect on its engagement with various fields.

## introduction the degree to which the ideas and artifacts of a field of study are useful to the world (for improving our understanding of the world or for practical applications) is a measure of its influence. developing a better sense of the influence of a field has several benefits, such as understanding what fosters greater innovation and what stifles it; what a field has success at understanding and what remains elusive; who are the most prominent stakeholders benefiting and who are being left behind, etc. however, the modes of influence are numerous and complex; thus, making an empirical determination of the degree of influence difficult. in this work, we focus on a specific subset of influence: the scientific influence of one field of study on another. stated another way, we explore the degree to which the ideas and scientific artifacts of various fields of study impact a target field of interest and the degree to which the ideas of a target field impact other fields. who is influencing us (our field)? who are we influencing? mechanisms of field-to-field influence are also complex, but one notable marker of scientific influence is citations. thus, we propose that the extent to which a source field cites a target field is a rough indicator of the degree of influence of the target on the source. we note here, though, that not all citations are equal-some cited work may have been much more influential to the citing work than others @xcite . further, citation patterns are subject to various biases @xcite . nonetheless, meaningful inferences can be drawn at an aggregate level; for example, if the proportion of citations from field x to a target field y has markedly increased as compared to the proportion of citations from other fields to the target, then it is likely that the influence of x on y has grown. ## related work previous studies on responsible research in nlp have examined aspects such as author gender diversity @xcite @xcite , author location diversity @xcite , temporal citation diversity @xcite , software package diversity @xcite , and institutional diversity @xcite . however, a core component of responsible research is interdisciplinary engagement with other research fields @xcite . such engagement can steer the direction of research. cross-field research engagement can provide many benefits in the short-term, by integrating expertise from various disciplines to address a specific issue, and in the long-term to generate new insights or innovations by blending ideas from different disciplines @xcite . many significant advances have emerged from the synergistic interaction of multiple disciplines, e.g., the conception of quantum mechanics, a theory that coalesced planck's idea of quantized energy levels @xcite ), einstein's photoelectric effect @xcite , and bohr's model of the atom @xcite . in addition, entirely new fields have emerged such as ecological economics. a key concept in ecological economics is the notion of 'ecosystem services', which originated in biology @xcite and ecology @xcite . the field of medicine has integrated neuroscience @xcite with engineering principles @xcite . within nlp, in the 1980s, there was a marked growth in the use of statistical probability tools to process speech, driven by researchers with electrical engineering backgrounds engaging with nlp @xcite . to quantify the interdisciplinarity of a body of research, researchers have proposed various metrics @xcite . these metrics can range from simple counting of referenced subjects @xcite to more complex metrics such as the gini-index @xcite . our work seeks to understand the level of interdisciplinarity in nlp by tracking how often and how much it references other fields and how this pattern evolves over time. several studies have reported an increasing trend of interdisciplinarity across fields @xcite @xcite and within subfields @xcite . for those observing increasing interdisciplinarity, the growth is predominantly seen in neighboring fields, with minimal increases in connections between traditionally distant research areas, such as materials sciences and clinical medicine @xcite . the findings are contested, with other researchers finding that the "scope of science" has become more limited @xcite , or that the findings are fielddependent, exemplified by contrasting citation patterns in sub-disciplines within anthropology and physics @xcite . within computer science (cs), chakraborty (2018) used citational analysis to show increasing interdisciplinarity in cs sub-fields, positing a three-stage life cycle for citation patterns of fields: a growing, maturing, and interdisciplinary phase. within nlp, @xcite examined papers from 1980 to 2008 to track the popularity of research topics, the influence of subfields on each other, and the influence of government interventions on the influx of new researchers into nlp. by examining the trends of nlp's engagement with other fields, our work seeks to provide a disciplinespecific perspective. 3 data central to work on field-to-field influence is a dataset where the field of each of the papers is known. we would like to note here that a paper may belong to many fields, each to varying extents, and it can be difficult for humans and automatic systems to determine these labels and scores perfectly. further, it is difficult to obtain a complete set of papers pertaining to each field; even determining what constitutes a field and what does not is complicated. nonetheless, meaningful inferences can be drawn from large samples of papers labeled as per some schema of field labels. as in past work @xcite , we chose the acl anthology (aa) as a suitable sample of nlp papers. 1 we extracted metadata associated with the nlp (aa) papers and with papers from various fields using a semantic scholar (s2) dump. foot_1 it includes papers published between 1965-2022, totaling 209m papers and 2.5b citations. s2's field of study annotation labels each paper as belonging to one or more of 23 broad fields of study. it uses a field-of-study classifier based on abstracts and titles (s2fos foot_2 ) with an accuracy of 86%. the s2 dump also includes information about which paper cites another paper. since we expect nlp papers to have cited cs substantially, we also examine the influence between specific subfields of cs and nlp. thus, we make use of the subfield annotations of the computer science ontology (cso) version 3.3 @xcite ) that is available through the d3 dataset @xcite . the cso classifier uses a paper's titles, abstracts, and keywords and has an f1-score of 74% @xcite . within the cso ontology, we consider only the 15 most populous subfields of cs (e.g., ai, computer vision (cv)). we also further split ai into machine learning (ml) and the rest of ai (ai')simply because ml is a known influencer of nlp. the source code to process our data and reproduce the experiments is available on github: https://github.com/jpwahle/ emnlp23-citation-field-influence ## experiments we use the dataset described above to answer a series of questions about the degree of influence between various fields and nlp. q1. which fields do nlp papers cite? in other words, how diverse are the outgoing citations by nlp papers-do we cite papers mostly just from computer science, or do we have a marked number of citations to other fields of study, as well? separately, which fields cite nlp papers? how diverse are the incoming citations to nlp papers? ans. for each nlp paper, we look at the citations to other papers and count the number of outgoing citations going to each field (or, outcites). similarly, we track papers citing nlp work and count the number of incoming citations from each field (or, incites). if a paper is in multiple fields, we assign one citation to each field. we calculate the percentage of outgoing citations: the percent of citations that go to a specific field from nlp over all citations from nlp to any field. similarly, we calculate the percentage for incoming citations: the percent of citations going from a specific field to nlp over all citations from any field to nlp. because we expect cs to be a marked driver of nlp citations, we split citations into cs and non-cs and then explore citations within cs and non-cs for additional insights. results. figure 1 other fields. this is not uncommon: we found that #incites is lower compared to #outcites for all 23 fields of study (figures not shown here). to get a better sense of the variation across non-cs fields, figure 1 (b) shows a sankey plot when we only consider the non-cs fields. observe that linguistics receives 42.4% of all citations from nlp to non-cs papers, and 45.1% of citations from non-cs papers to nlp are from linguistics. nlp cites mathematics more than psychology, but more of nlp's incites are from psychology than math. discussion. even though we expected a high cs-nlp citational influence, this work shows for the first time that ∼80% of the citations in and out of nlp are from and to cs. linguistics, mathematics, psychology, and social science are involved in a major portion of the remaining in-and out-citations. however, these results are for all citations and do not tell us whether the citational influences of individual fields are rising or dropping. q2. how have nlp's outgoing (and incoming) citations to fields changed over time? ans. figure 2 shows nlp's outgoing (a) and incoming (b) citation percentages to cs and non-cs papers from 1990 to 2020. observe that while in 1990 only about 54% of the outgoing citations were to cs, that number has steadily increased and reached 83% by 2020. this is a dramatic transformation and shows how cs-centric nlp has become over the years. the plot for incoming citations (figure 2 (b) ) shows that nlp receives most citations from cs, and that has also increased steadily from about 64% to about 81% in 2020. figure 3 shows the citation percentages for four non-cs fields with the most citations over time. observe that linguistics has experienced a marked (relative) decline in relevance for nlp from 2000 to 2020 (60.3% to 26.9% for outcites; 62.7% to 39.6% for incites). details on the diachronic trends for cs subfields can be found in the appendix a.3. discussion. over time, both the in-and outcitations of cs have had an increasing trend. these results also show a waning influence of linguistics and sociology and an exponential rise in the influence of mathematics (probably due to the increasing dominance of mathematics-heavy deep learning and large language models) and psychology (probably due to increased use of psychological models of behavior, emotion, and well-being in nlp applications). the large increase in the influence of mathematics seems to have largely eaten into what used to be the influence of linguistics. q3. which fields cite nlp more than average and which do not? which fields are cited by nlp more than average, and which are not? ans. as we know from q1, 15.3% of nlp's non-cs citations go to math, but how does that compare to other fields citing math? are we citing math more prominently than the average paper? that is the motivation behind exploring this question. to answer it, we calculate the difference between nlp's outgoing citation percentage to a field f and the macro average of the outgoing citations from various fields to f . we name this metric outgoing relative citational prominence (orcp). if nlp has an orcp greater than 0 for f , then nlp cites f more often than the average of all fields to f . similar to orcp, we calculate the incoming relative citational prominence (ircp) from each field to nlp. ircp indicates whether a field cites nlp more often than the average of all fields to nlp. figure 14 in the appendix is a similar plot as figure 4 , but also shows orcp scores for cs, psy- ## concluding remarks in this work, we examined the citational influence between nlp and various fields of study. we created a distinct dataset of metadata that includes ∼77k nlp papers, citations to these papers (and the fields the citing papers belong to), and citations by the nlp papers (and the fields the cited papers belong to). we analyzed this data using various metrics such as citational field diversity index and relative citational prominence to show that, unlike most other fields, the diversity of both incoming and outgoing citations to and from nlp papers has steadily declined over the past few decades. our experiments show that nlp citations are dominated by cs, accounting for over 80% of citations, with particular emphasis on ai, ml, and ir. contributions from non-cs fields like linguis-4 https://huggingface.co/spaces/jpwahle/field-diversity tics and sociology have diminished over time. the diversity of nlp citing papers from other fields and how nlp is cited by other fields has decreased since 1980 and is particularly low compared to other fields such as linguistics and psychology. nlp presents increasing insularity reflected by the growth of intra-field citations and a decline in multidisciplinary works. although historically, wellcited papers have higher citational field diversity, concerningly, this trend has reversed in the 2010s. over the last 5 years or so, we see nlp technologies being widely deployed, impacting billions of people directly and indirectly. numerous instances have also surfaced where it is clear that adequate thought was not given to the development of those systems, leading to various adverse outcomes. it has also been well-established that a crucial ingredient in developing better systems is to engage with a wide ensemble of literature and ideas, especially bringing in ideas from outside of cs (say, psychology, social science, linguistics, etc.) against this backdrop, the picture of nlp's striking lack of engagement with the wider research literature -especially when marginalized communities continue to face substantial risks from its technologies -is an indictment of the field of nlp. not only are we not engaging with outside literature more than the average field or even just the same as other fields, our engagement is, in fact, markedly less. a trend that is only getting worse. the good news is that this can change. it is a myth to believe that citation patterns are "meant to happen", or that individual researchers and teams do not have a choice in what they cite. we can actively choose what literature we engage with. we can work harder to tie our work with ideas in linguistics, psychology, social science, and beyond. we can work more on problems that matter to other fields, using language and computation. by keeping an eye on work in other fields, we can bring their ideas to new compelling forms in nlp.