Forgotten Knowledge: Examining the Citational Amnesia in NLP (2023)

Abstract: Citing papers is the primary method through which modern scientific writing discusses and builds on past work. Collectively, citing a diverse set of papers (in time and area of study) is an indicator of how widely the community is reading. Yet, there is little work looking at broad temporal patterns of citation. This work systematically and empirically examines: How far back in time do we tend to go to cite papers? How has that changed over time, and what factors correlate with this citational attention/amnesia? We chose NLP as our domain of interest and analyzed approximately 71.5K papers to show and quantify several key trends in citation. Notably, around 62% of cited papers are from the immediate five years prior to publication, whereas only about 17% are more than ten years old. Furthermore, we show that the median age and age diversity of cited papers were steadily increasing from 1990 to 2014, but since then, the trend has reversed, and current NLP papers have an all-time low temporal citation diversity. Finally, we show that unlike the 1990s, the highly cited papers in the last decade were also papers with the least citation diversity, likely contributing to the intense (and arguably harmful) recency focus. Code, data, and a demo are available on the project homepage.

## introduction study the past if you would define the future. of particular interest is the extent to which good older work is being forgotten -citational amnesia. more specifically, for this paper, we define citational amnesia as shown below: citational amnesia: the tendency to not cite enough relevant good work from the past (more than a few years old). we cannot directly measure citational amnesia empirically because determining "enough", "relevance", and "good" require expert researcher judgment. however, what we can measure is the collective tendency of a field to cite older work. such an empirical finding enables reflection on citational amnesia. a dramatic drop in our tendency to cite older work should give us cause to ponder whether we are putting enough effort to read older papers (and stand on the proverbial shoulders of giants). note that we are not saying that old work should be cited simply because it exists. we are saying that we should consciously reflect on the diversity of the papers we explore when conducting research. diversity can take many forms, including reading relevant papers from diverse fields, by authors from diverse regions, and relevant papers published from various time periods -the focus of this paper. exploring a diverse set of papers allows us to ben-efit from important and diverse research perspectives. looking at older literature makes us privy to broader trends, and informs us in ways that are beneficial well beyond the immediate work. historically, citational amnesia was impacted by various factors around access and invention. for example, the invention of the printing press in the year 1440 allowed a much larger number of people to access scientific writing @xcite . the era of the internet and digitization of scientific literature that began in the 1990s also greatly increased the ease with which one could access past work @xcite . however, other factors such as the birth of paradigm-changing technologies may also impact citation patterns; ushering in a trend of citing very new work or citing work from previously ignored fields of work. such dramatic changes are largely seen as beneficial; however, strong tailwinds may also lead to a myopic focus on recent papers and those from only some areas, at the expense of benefiting from a wide array of work @xcite . we choose as our domain of interest, papers on natural language processing (nlp), specifically those in the acl anthology. this choice is motivated by the fact that nlp (and other related fields of artificial intelligence) are in a period of dramatic change: there are notable and frequent gains on benchmark datasets; nlp technology is becoming increasingly ubiquitous in society; and new sub-fields of nlp such as computational social science, ethics and nlp, and sustainable nlp are emerging at an accelerated rate. the incredibly short research-to-production cycle and move-fastand-break-things attitude in nlp (and machine learning more broadly) has also led to considerable adverse outcomes for various sections of society, especially those with the least power @xcite @xcite . thus reading and citing more broadly is especially important now. in this work, we compiled a temporal citation network of 71.5k nlp papers that were published between 1990 and 2021, along with their metainformation such as the number of citations they received in each of the years since they were published -the age of citations (aoc) dataset. we use aoc to answer a series of specific research questions on what we value, what we have forgotten, what factors are associated with this citational attention/amnesia, what are the citation patterns of different types of papers, and how these citation patterns have changed over time. finally, we show that many of the highly cited papers from the past decade have very low temporal citation diversity; and because of their wide reach, may have contributed to the intense recency focus in nlp. all of the data and code associated with the project will be made freely available on the project homepage. ## related work in the broad area of scientometrics (study of quantitative aspects of scientific literature), citations and their networks have been studied from several perspectives, including: paper quality @xcite , field of study @xcite , novelty, length of paper @xcite , impact factor @xcite , venue of publication @xcite , language of publication @xcite , and number of authors @xcite , collaboration @xcite , self-citation @xcite , as well as author's reputation @xcite , affiliation @xcite , geographic location (nielsen and andersen, 2021; @xcite @xcite , gender, race and age @xcite @xcite . however, there has been relatively little work exploring the temporal patterns of citation. @xcite analyzed scholarly articles published in 1990-2013 to show that the percentage of older papers being cited steadily increased from 1990 to 2013, for seven of the nine fields of study explored. (they treated papers that were published more than ten years before a particular citation as old papers.) for computer science papers published in 2013, on average, 28% of the cited papers were published more than ten years before. this represented an increase of 39% from 1990. they attributed this increasing trend in citing old papers to the ease of access of scientific literature on the world wide web, as well as the then relatively new scientific-literature-aggregating services such as google scholar. parolo et al. (2015) analyzed about 25 million papers from clinical medicine, molecular biology, physics, and chemistry published until 2014 to show that typically the number of citations a paper receives per year increases in the years after publication, reaches a peak, and then decays exponentially. interestingly they showed that this rate of decay was increasing in the more recent papers of their study. they attribute this quicker decay (or more "forgetting" of recent papers) to the substantial increase in the number of publications; a lot more papers are being published, and due to the limited attention span of subsequent researchers, on average, papers are being forgotten faster. past work on nlp papers and their citations includes work on gender bias @xcite @xcite , author location diversity @xcite , author institution diversity @xcite , and on broad general trends such as average number of citations over time and by type of paper @xcite . bollmann and elliott (2020) were the first to explore the recency bias of citations in nlp papers. they showed that the acl anthology papers published between 2017 and 2019 cited more recent work than papers published between 2010 and 2014. question 3 in section 4 of our paper is of the same spirit that was explored in their work; however, our work examines a much larger spread of nlp papers (published between 1965 and 2021). this will shed light on the reproducibility of those findings and, more importantly, determine the broader trajectory of temporal citation patterns (from the start of acl to present day). additionally, our work introduces a new citation age diversity metric to quantify the degree of spread of citations over time, as well as an interactive online demo system to visualize the citation age diversity of any paper. going beyond how overall citation patterns have changed over time, our work takes a deep dive into six other novel research questions, notably around temporal citation patterns in subareas of nlp, of cited topics, and across sparsely and highly cited papers. ## dataset the acl anthology (aa) citation corpus @xcite ) contains meta data (paper title, year of publication, and venue, etc. for the 71,568 papers in the acl anthology repository (published until january 2022). we used the semantic scholar api 3 to gather the references for each paper in the aa citation corpus, using the paper's unique semantic scholar id (ssid). this allowed us to obtain 3 https://www.semanticscholar.org/ additional information about the cited papers, such as their title, year of publication, and venue of publication. note that these cited papers may or may not be part of aa. to study the dynamics of citations over time, we constructed year-wise citation networks using the data collected. specifically, we created the citation networks for every year from 1965 to 2001. this representation of citation data allows us to answer several interesting questions, such as the number of citations a paper receives in a particular year after its publication. we refer to this dataset as age of citations (aoc) dataset. ## age of citation we used the aoc dataset to answer a series of questions on how research papers are cited and the trends across years. q1. what is the average number of unique references in the aa papers? how does this number vary by publication type, such as workshop, conference, and journal? has this average stayed roughly the same or has it changed markedly over the years? ans. we calculated the average number of unique references for all papers in the aoc dataset, as well as for each publication type (workshops, conferences, and journals). we then binned all papers by publication year, computed the mean and median for each bin for each year. ## demo: cad index of your paper to encourage authors to be more cognizant of the age of papers they cite, we created an online demonstration page where one can provide the semantic scholar id of any paper and the system returns the number of papers referenced, mean age of citation (maoc), top-5 oldest cited papers, and their years of publication. 9 notable, the demo also plots the distribution of maoc for all the considered papers (all papers published till 2021) and compares it with mean age of citation of the input paper. figure 13 in the appendix shows a screenshot of the demo portal for an example input. ## conclusions and discussion this work looks at temporal patterns of citations by presenting a set of comprehensive analyses of the trend in the diversity of age of citations and the percentage of older papers cited in the field of nlp. to enable this analysis, we compiled a dataset of papers from the acl anthology and their metainformation; notably, the number of citations they received each year since they were published. we showed that both the diversity of age of citations and the percentage of older papers cited increased from 1990 to 2014, but since then there has been a dramatic reversal of the trend. by the year 2021 (the final year of analysis), both the diversity of age of citations and the percentage of older papers cited have reached historical lows. we also studied the correlation between the number of citations a paper receives and the diversity of age of cited papers, and found that while there was roughly no correlation in the 1990s, the 2000s marked the beginning of a period where the higher citation levels correlated strongly with lower temporal citation diversity. it is a common belief among researchers in the field that the advent of deep neural revolution in the early 2010's has led us to cite more recent papers than before. this analysis confirms and quantifies the extent to which temporal diversity is reduced in this recent period. in fact, it shows that the reduction in temporal diversity of citations is so dramatic that it has wiped out steady gains from 1990 to 2014. while some amount of increased focus on recent papers is expected (and perhaps beneficial) after large technological advances, an open question, now, is whether, as a community, we have gone too far, ignoring important older work. our work calls for an urgent need for reflection on the intense recency focus in nlp: how are we contributing to this as researchers, advisors, reviewers, area chairs, and funding agencies?