On Forgetting to Cite Older Papers: An Analysis of the ACL A nthology (2020)

Abstract: The field of natural language processing is experiencing a period of unprecedented growth, and with it a surge of published papers. This represents an opportunity for us to take stock of how we cite the work of other researchers, and whether this growth comes at the expense of “forgetting” about older literature. In this paper, we address this question through bibliographic analysis. By looking at the age of outgoing citations in papers published at selected ACL venues between 2010 and 2019, we find that there is indeed a tendency for recent papers to cite more recent work, but the rate at which papers older than 15 years are cited has remained relatively stable.

## introduction in this paper, we conduct a corpus analysis of papers published in recent acl venues to determine whether the community is collectively forgetting about older papers as it experiences a period of rapid growth (see figure 1 ). the association of computational linguistics (acl) is one of the largest publishers of articles in natural language processing research: it maintains the open-access acl anthology foot_0 of articles that date back to the 1960s, offering a rich resource for studying nlp publications. while the aforementioned analyses have mainly focused on incoming citations, our work targets outgoing citations. we focus on the age of citations in the references section of articles published at acl venues between 2010 and 2019 (sec. 2), with a view to studying three questions: we find that the mean age of the papers cited does indeed decrease from 2010-2019, and that this de-crease is statistically significant, with a larger effect size in recent years (sec. 3.1). we also find that there is no significant difference in the rate at which older papers are cited during this period (sec. 3.2), and that there are marked differences between the citations in journal articles and conference proceedings (sec. 3.3). our findings show that, at a time of rapid growth, an increasing proportion of citations are going to recently published papers, but that researchers still acknowledge that they are standing on the shoulders of their peers. ## data the analysis in this paper is based on a subset of articles from the acl anthology. while corpora of nlp publications, including the acl anthology, already exist @xcite @xcite , none of them include publications newer than 2015. we compiled our own dataset because we are mostly interested in the papers published in recent years. the dataset is drawn from acl venues: conference proceedings from meetings of the acl, eacl (european chapter of the acl), naacl (north american chapter of the acl), and emnlp (empirical methods in nlp) as well as articles from the cl (computational linguistics) and tacl (transactions of the acl) journals. anthology statistics figure 1 shows the distribution of the articles in the corpus: the number of articles published in these venues steadily increases from 2010-2019. the cl and tacl journals publish articles at a steady rate; the acl conference fluctuates in size, depending on whether it is co-located with naacl; and the eacl conference nearly doubles in size each time it takes place. in terms of whether the field is rapidly growing, we note that there was a year-on-year increase of 42% between in 2017-2018 due to the increase in the number of papers published at naacl and emnlp, and a 34% increase between 2018-2019. ## conclusions we presented an analysis of citations in publications from major acl venues between 2010 and 2019, focusing on the distribution of the age of cited papers. we found that recently published papers (0-3 years old) are cited significantly more often in publications from recent years (ca. 2015-2019), while papers published 15 or more years ago are being cited at a stable rate. there is also a marked difference between journal and conference publications in the distribution of citation age: journal articles feature more citations to older papers. these findings could be due to the increasing difficulty of keeping up with the literature, given that many more papers are being published now, in addition to the deluge of papers that appear on preprint servers. some areas of nlp research did also not exist 15 years ago, e.g. social media analysis, po-tentially making it challenging to cite older related work. finally, since several influential neural network papers have been published in the 1990s (cf. tab. 2), a mostly quantitative analysis is limited in its ability to determine, e.g., to what extent we still engage with older literature outside of this domain. a potential confound in our analysis is that some proceedings imposed a page limit for references; e.g., the acl conference gave unlimited space for references in 2010, 2012, and from 2016 onwards, but imposed a page limit in 2011 and 2013-2015. we can still observe an increase in the average number of citations per paper during this latter period, so it seems unlikely that this had an effect. in addition, our analysis is limited to studying the age of the papers cited in the acl anthology -it does not make any claims about the complex network effects involved in researchers from particular institutions, countries, or sub-fields, and it does not study other venues that also publish nlp papers. future work includes a deeper qualitative analysis of which (type of) papers are being cited; a more fine-grained analysis of different research topics in nlp to determine whether changes are more prevalent within certain areas than others; or extending the analysis to a larger set of the papers in the acl anthology.