Examining Citations of Natural Language Processing Literature (2020)

Abstract: We extracted information from the ACL Anthology (AA) and Google Scholar (GS) to examine trends in citations of NLP papers. We explore questions such as: how well cited are papers of different types (journal articles, conference papers, demo papers, etc.)? how well cited are papers from different areas of within NLP? etc. Notably, we show that only about 56% of the papers in AA are cited ten or more times. CL Journal has the most cited papers, but its citation dominance has lessened in recent years. On average, long papers get almost three times as many citations as short papers; and papers on sentiment classification, anaphora resolution, and entity recognition have the highest median citations. The analyses presented here, and the associated dataset of NLP papers mapped to citations, have a number of uses including: understanding how the field is growing and quantifying the impact of different types of papers.

## introduction the origins of natural language processing (nlp) go back to the earliest work in computer sciencewhen alan turing published his seminal paper exploring whether machines can think, and proposed what is now known as the turing test (turing, 1950, 2009). a crucial factor in the evolution of nlp as a field of study in its own right was the formation of the association for computational linguistics (acl) in 1962, and the first acl conference in 1965. @xcite today nlp is a broad interdisciplinary field with a growing number of researchers from computer science, linguistics, information science, psychology, social sciences, humanities, and more joining its ranks. organizations such as acl, elra, and afnlp publish peer-reviewed nlp papers that include both journal articles and conference proceedings. historically, the need for a faster review process has made conference proceedings the dominant form of published research in computer science and nlp. with time, the conferences and the types of papers they publish, have evolved. some conferences, such as emnlp and acl, are highly competitive, while others, such as most workshops and lrec, deliberately choose to keep more generous acceptance rates. the publications themselves can be of different types: journal articles, conference papers, short papers, system demonstration papers, shared task papers, workshop papers, etc. new ideas and paradigms have evolved: for example, the rise of statistical nlp in the 1990s and deep learning in the 2010s. with the dawn of a new decade and nlp research becoming more diverse and more popular than it ever has been, this work looks back at the papers already published to identify broad trends in their impact on subsequent scholarly work. commonly used metrics of research impact on subsequent scholarly work are derived from citations including: number of citations, average citations, h-index, relative citation ratio, and impact factor (bornmann and daniel, 2009). however, the number of citations is not always a reflection of the quality or importance of a piece of work. note also that there are systematic biases that prevent certain kinds of papers from accruing citations, especially when the contributions of a piece of work are atypical or in an area where the number of scientific publications is low. furthermore, the citation process can be abused, for example, by egregious self-citations (ioannidis et al., 2019). nonetheless, given the immense volume of scientific literature, the relative ease with which one can track citations using services such as google scholar (gs), and given the lack of other easily applicable and effec-tive metrics, citation analysis is an imperfect but useful window into research impact. thus citation metrics are often a factor when making decisions about funding research and hiring scientists. citation analysis can also be used to gauge the influence of outside fields on one's field and the influence of one's field on other fields. therefore, it can be used to determine the relationship of a field with the wider academic community. as part of a broader project on analyzing nlp literature, we extracted and aligned information from the acl anthology (aa) and google scholar to create a dataset of tens of thousands of nlp papers and their citations (mohammad, 2020b, 2019). @xcite in this paper, we describe work on examining the papers and their citations to identify broad trends within nlp research-overall, across paper types, across publication venues, over time, and across research areas within nlp. notably, we explored questions such as: how well cited are papers of different types (journal articles, conference papers, demo papers, etc.)? how well cited are papers published in different time spans? how well cited are papers from different areas of research within nlp? etc. the dataset and the analyses have many uses including: understanding how the field is growing; quantifying the impact of different types of papers on subsequent publications; and understanding the impact of various conferences and journals. perhaps most importantly, though, they serve as a record of the state of nlp literature in terms of citations. all of the data and interactive visualizations associated with this work are freely available through the project homepage. @xcite 2 background and related work the acl anthology is a digital repository of public domain, free to access, articles on nlp. @xcite it includes papers published in the family of acl conferences as well as in other nlp conferences such as lrec and ranlp. @xcite as of june 2019, it provided access to the full text and metadata for ∼50k articles published since 1965 (the year of the first acl confer-ence). it is the largest single source of scientific literature on nlp. various subsets of aa have been used in the past for a number of tasks including: the study of citation patterns and intent (pham and hoffmann, 2003; aya et al., 2005; teufel et al., 2006; mohammad et al., 2009; nanba et al., 2011; zhu et al., 2015; radev et al., 2016), generating summaries of scientific articles (qazvinian et al., 2013), and creating corpora of scientific articles (bird et al., 2008; mariani et al., 2018). perhaps the work closest to ours is that by anderson et al. (2012), who examine papers from 1980 to 2008 to track the ebb and flow of topics within nlp, the influence of subfields on each other, and the influence of researchers from outside nlp. however, that work did not examine trends in the citations of nlp papers. google scholar is a free web search engine for academic literature. @xcite through it, users can access the metadata associated with an article such as the number of citations it has received. google scholar does not provide information on how many articles are included in its database. however, scientometric researchers estimated that it included about 389 million documents in january 2018 (gusenbauer, 2019)-making it the world's largest source of academic information. thus, there is growing interest in the use of google scholar information to draw inferences about scholarly research in general (howland, 2010; orduña-malea et al., 2014; khabsa and giles, 2014; mingers and leydesdorff, 2015; martín-martín et al., 2018) and on scholarly impact in particular (priem and hemminger, 2010; yogatama et al., 2011; bulaitis, 2017; ravenscroft et al., 2017; bos and nitza, 2019; ioannidis et al., 2019). this work examines patterns of citations of tens of thousands of nlp papers, both overall and across paper types, venues, and areas of research. ## data we now briefly describe how we extracted information from the acl anthology and google scholar to facilitate the citation analysis. (further details about the dataset, as well as an analysis of the volume of research in nlp over the years, are available in mohammad (2020b).) we aligned the information across aa and gs using the paper title, year of publication, and first author last name. ## examining citations of nlp papers we use data extracted from the acl anthology and google scholar to examine trends in citations through a series of questions. q1. how many citations have the aa papers received? how is that distributed among the papers published in various years? a. ∼1.2 million citations (as of june 2019). figure 1 shows the screenshot of an interactive timeline graph where each year has a bar with height corresponding to the number of citations received by papers published in that year. further, the bar has colored segments corresponding to each of the papers; the height of a segment is proportional to the number of citations the paper has received. thus it is easy to spot the papers that received a large number of citations. hovering over individual papers reveals additional metadata. a. in this and all further analyses, we do not include aa papers published in 2017 or later (to allow for at least 2.5 years for the papers to collect citations). there are 26,949 aa papers that were published from 1965 to 2016. figure 2 shows box and whisker plots for: all of these papers (on the left) and for individual paper types (on the right). the whiskers are at a distance of 1.5 times the inter-quartile length. the average number of citations are indicated with the horizontal green dotted lines. creating a separate class for "top-tier conference" is somewhat arbitrary, but it helps make certain comparisons more meaningful. for this work, we consider acl, emnlp, naacl, coling, and eacl as top-tier conferences based on low acceptance rates and high citation metrics, but certainly other groupings are also reasonable. discussion: overall, the median citation count is 12. 75% of the papers have 34 or fewer citations. the average number of citations (45) is markedly higher than the median (12); this is because of a small number highly cited papers. when comparing different types of papers, we notice a large difference between journal papers and the rest. even though the number of journal papers in aa (and aa ) is very small (about 2.5%), these papers have the highest median and average citations (55 and 204, respectively). top-tier conferences come next, followed by other conferences. the differences between each of these pairs is statistically significant (kolmogorov-smirnov (ks) test, @xmath0.01). @xcite interestingly, the workshop papers and the shared task papers have higher medians 10 ks is a non-parametric test that can be applied to compare distributions without needing to make assumptions about the nature of the distributions. since the citations data is not normally distributed, ks is especially well suited. and averages than the non-top-tier conferences. these differences are also significant (ks, @xmath1.01). q3. how well cited are recent aa papers: say those published in the last decade (2010-2016)? how well cited are papers that were all published in the same year, say 2014? are the citation distributions for individual years very different from those for larger time spans, say 2010-2016? also, how well cited are papers 5 years after they are published? a. the top of figure 3 shows citation box plots for 2010-2016; the bottom shows plots for papers published in 2014. discussion: observe that, in general, these numbers are markedly lower than the those in figure 2 . that is expected as these papers have had less time to accrue citations. observe that journal papers again have the highest median and average; however, the gap between journals and top-tier conferences has reduced considerably. the shared task papers have a signifidiscussion: cl journal has the highest median and average citation numbers. acl comes second, closely followed by emnlp and naacl. the gap between cl journal and acl is considerably reduced when considering the 2010-2016 papers. ijcnlp and lrec have the highest numbers among the non-top-tier conferences, but their numbers remain lower than the numbers for se-meval, non-semeval shared tasks, and workshops. tacl, a journal, has substantially lower citation numbers than cl journal, acl, emnlp, and naacl (figure 5 top). however, it should be noted that tacl only began publishing since 2013. (also, with a page limit of about ten, tacl papers are arguably more akin to conference papers than journal papers.) when considering only the 2010-2016 papers, tacl's citation numbers are second only to cl journal (figure 5 bottom). when considering 2010-2016 papers, the system demonstration papers, the semeval shared task papers, and non-semeval shared task papers have notably high averages (surpassing or equalling those of coling and eacl); however their median citations are lower. (this is consistent with the trends we saw earlier in q3.) q6. how well cited are long and short acl main conference papers, respectively? a. short papers were introduced by acl in 2003. since then acl is by far the venue with the highest number of short papers (compared to other venues). so we compare long and short papers published at acl since 2003 to determine their average citations. figure 6 shows the citation box plots for long and short papers published between 2003 and 2016 at acl. the two distributions are statistically different (kolmogorov-smirnov test, @xmath2.01). discussion: in 2003, the idea of short papers was a novelty. it was conceived with the idea that there needs to a be a place for focused contributions that do not require as much space as a long paper. the format gained popularity quickly, and short papers at acl tend to be incredibly competitive (sometimes having a lower acceptance rate than long papers). while there have been several influential short papers, it remains unclear how well-cited they are as a category. this analysis sheds some light to that end. we find that, on average, long papers get almost three times as many citations as short papers; the median for long papers is two-and-half times that of short papers. q7. how do different venues and paper types compare in terms of the volume of papers pertaining to various amounts of citation? a. figure 7 shows a stream graph of #papers by #citations. the contributions of each of the venues and paper types are stacked one on top of another (bands of colors). for a given point on the citations axis (say k), the width of the stream corresponds to the number of papers with k citations. discussion: it is not surprising to see that the #papers by #citations curve follows a power law distribution. (there are lots of papers with 0 or few citations, but the number drops of exponentially with the number of citations.) workshop papers (light grey) are the most numerous, followed by lrec (green)-as observable from their wide bands. the bands for acl, coling, emnlp, and naacl are easily discernable but the bands for many others, especially cl journal and tacl are barely discernable indicating low relative volume of their papers. observe that the bands for workshops and lrec are markedly wider in the 0 to 10 citations range than in the 11 and more citations range of the x axis. in contrast, the widths of the bands for top-tier conferences, such as acl and emnlp, remain relatively stable. nonetheless, in terms of raw volume, it is worth noting that the workshops and lrec each produce more papers that are cited ten or more times than any other venue. as one considers even higher citations, the top-tier conferences become more dominant. q8. what percentage of papers are cited more than 10 times? foot_9 how many papers are cited 0 times? a. figure 8 shows the percentage of aa papers in various citation bins: 0, 1-9, 10-99, and 1000-9999. (the number of papers is shown in parenthesis.) discussion: about 56% of the papers are cited ten or more times. 6.4% of the papers are never cited. (note also that some portion of the 1-9 bin likely includes papers that only received self-citations.) it would be interesting to compare these numbers with those in other fields such as medical sciences, physics, linguistics, machine learning, and psychology. ## limitations and future work we list below some ideas of future work that we did not explore in this paper: a list of limitations and ethical considerations associated with this work is available online. 14 13 note that simply composing titles with these high-citation bigrams is not expected to attract a large number of citations. ## conclusions we extracted citation information for ∼1.1m papers from google scholar profiles of researchers who published at least three papers in the acl anthology. we used the citation counts of a subset (∼27k papers) to examine patterns of citation across paper types, venues, over time, and across areas of research within nlp. we showed that only about 56% of the papers are cited ten or more times. cl journal has the most cited papers, but the citation gap between cl journal and top-tier conferences has reduced in recent years. on average, long papers get almost three times as many citations as short papers. in case of popular shared tasks, the task-description papers and competition-winning system-description papers often receive a considerable number of citations. so much so that the average number of citations for the shared task papers is higher than the average for non-top-tier conferences. the papers on sentiment classification, anaphora resolution, and entity recognition have the highest median citations. workshop papers and the shared task papers have higher median and average citations than the non-top-tier conferences. the analyses presented here, and the associated dataset of papers mapped to citations, have a number of uses including, understanding how the field is growing and quantifying the impact of different types of papers. in separate work, we explored the use of the dataset to detect gender disparities in authorship and citations (mohammad, 2020a). the dataset can potentially also be used to compare patterns of citations in nlp with those in other fields. finally, we note again that citations are not an accurate reflection of the quality or importance of individual pieces of work. a crucial direction of future work is to develop richer ways of capturing scholarly impact.