NLP Scholar: An Interactive Visual Explorer for Natural Language Processing Literature (2020)

Abstract: As part of the NLP Scholar project, we created a single unified dataset of NLP papers and their meta-information (including citation numbers), by extracting and aligning information from the ACL Anthology and Google Scholar. In this paper, we describe several interconnected interactive visualizations (dashboards) that present various aspects of the data. Clicking on an item within a visualization or entering query terms in the search boxes filters the data in all visualizations in the dashboard. This allows users to search for papers in the area of their interest, published within specific time periods, published by specified authors, etc. The interactive visualizations presented here, and the associated dataset of papers mapped to citations, have additional uses as well including understanding how the field is growing (both overall and across sub-areas), as well as quantifying the impact of different types of papers on subsequent publications.

## introduction nlp is a broad interdisciplinary field that draws knowledge from computer science, linguistics, information science, psychology, social sciences, and more. @xcite over the years, scientific publications in nlp have grown in number and diversity; we now see papers published on a vast array of research questions and applications in a growing list of venues-in journals such as cl and tacl, in large conferences such as acl and emnlp, as well as a number of small area-focused workshops. the acl anthology (aa) is a digital repository of public domain, free to access, articles on nlp. @xcite it includes papers published in the family of acl conferences as well as in other nlp conferences such as lrec and ranlp. as of june 2019, it provided access to the full text and metadata for close to 50k articles published since 1965. @xcite it is the largest single source of scientific literature on nlp. however, the meta-data does not include citation statistics. citation statistics are the most commonly used metrics of research impact. they include: number of citations, average citations, h-index, relative citation ratio, and impact factor. note, however, that the number of citations is not always a reflection of the quality or importance of a piece of work. furthermore, the citation process can be abused, for example, by egregious self-citations (ioannidis et al., 2019). nonetheless, given the immense volume of scientific literature, the relative ease with which one can track citations using services such as google scholar (gs), and given the lack of other easily applicable and effective metrics, citation analysis is an imperfect but useful window into research impact. google scholar is a free web search engine for academic literature. @xcite through it, users can access the metadata associated with an article such as the number of citations it has received. google scholar does not provide information on how many articles are included in its database. however, scientometric researchers estimated that it included about 389 million documents in january 2018 (gusenbauer, 2019)-making it the world's largest source of academic information. thus, it is not surprising that there is growing interest in the use of google scholar information to draw inferences about scholarly research in general (martín-martín et al., 2018; mingers and leydesdorff, 2015; orduña-malea et al., 2014; khabsa and giles, 2014; howland, 2010) and on scholarly impact in particular (bos and nitza, 2019; ioannidis et al., 2019; ravenscroft et al., 2017; bulaitis, 2017; yogatama et al., 2011; priem and hemminger, 2010). services such as google scholar and semantic scholar cover a wide variety of academic disciplines. wile there are benefits to this, the lack of focus on nlp literature has some drawbacks as well: e.g, the potential for too many search results that include many irrelevant papers. for example, if one is interested in nlp papers on emotion and privacy, searching for them on google scholar is less efficient than searching for them on a platform dedicated to nlp papers. further, services such as google scholar provide minimal interactive visualizations. nlp scholar with its focus on aa data, is not meant to replace these tools, but act as a complementary tool for dedicated visual search of nlp literature. acl 2020 has a special theme asking researchers to reflect on the state of nlp. in the spirit of that theme, and as part of a broader project on analyzing nlp literature, we extracted and aligned information from the acl anthology (aa) and google scholar to create a dataset of tens of thousands of nlp papers and their citations (mohammad, 2020c, 2019). in separate work, we have used the data to explores questions such as: how well cited are papers of different types (journal articles, conference papers, demo papers, etc.)? how well cited are papers published in different time spans? how well cited are papers from different areas of research within nlp? etc. (mohammad, 2020a). we also explored gender gaps in natural language processing research, in terms of authorship and citations (mohammad, 2020b). in this paper we describe how we built an interactive visual explorer for this unified data, which we refer to as nlp scholar. some notable uses of nlp scholar are listed below: even beyond the dedicated interactive visualizer described here, the underlying data with its alignment between aa and gs has potential uses in: perhaps most importantly, though, nlp scholar serves as a visual record of the state of nlp literature in terms of citations. we note again though, that even though this work seeks to make citation metrics more accessible for acl anthology papers, citation metrics are not always accurate reflections of the quality, importance, or impact of individual papers. all of the data and interactive visualizations associated with this work are freely available through the project homepage. @xcite 2 background and related work much of the work in visualizing scientific literature has focused on showing topics of research (wu et al., 2019; heimerl et al., 2012; lee et al., 2005). there is also notable work on visualizing communities through citation networks (heimerl et al., 2015; radev et al., 2016). various subsets of aa have been used in the past for a number of tasks, including: to study citation patterns and intent (radev et al., 2016; zhu et al., 2015; nanba et al., 2011; mohammad et al., 2009; teufel et al., 2006; aya et al., 2005; pham and hoffmann, 2003), to generate summaries of scientific articles (qazvinian et al., 2013), to study gender disparities in nlp (schluter, 2018), to study subtopics within nlp (anderson et al., 2012), and to create corpora of scientific articles (mariani et al., 2018; bird et al., 2008). however, none of these works provide an interactive visualization for users to explore nlp literature and their citations. ## data we now briefly describe how we extracted information from the acl anthology and google scholar. ## data explorations with nlp scholar figure 1 a1 shows that the dataset includes 44,895 papers. a2 shows that the volume of papers published was considerably lower in the early years (1965 to 1989); there was a spurt in the 1990s; and substantial numbers since the year 2000. also, note that the number of publications is considerably higher in alternate years. this is due to certain biennial conferences. since 1998 the largest of such conferences has been lrec (in 2018 alone lrec had over 700 main conferences papers and additional papers from its 29 workshops). coling, another biennial conference (also occurring in the even years) has about 45% of the number of main conference papers as lrec. b1 shows that aa papers have received ∼1.2 million citations (as of june 2019). the timeline graph in b2 shows that, with time, not only have the number of papers grown, but also the number of high-citation papers. we see a marked jump in the 1990s over the previous decades, but the 2000s are the most notable in terms of the high number of citations. the 2010s papers will likely surpass the 2000s papers in the years to come. the most cited papers list (c) shows influential papers from machine translation, sentiment analysis, word embeddings, syntax, and semantics. among the authors (d), observe that christopher manning has not only received the most number of citations, he has also received almost three times as many citations as the next person in the list. search: nlp scholar allows for search in a number of ways. suppose we are interested in the topic of sentiment analysis. then we can enter the relevant keywords in the search box: sentiment, valence, emotion, emotions, affect, etc. then the visualizations are filtered to present details of only those papers that have at least one of these keywords in the title. (future work will allow for search in the abstract and the whole text.) figure 7 in the appendix shows the filtered result. the system identified 1,481 papers that each have at least one of the query terms in the title. they have received more than 85k citations. the citations timeline (b2 in figure 7 ) shows that there were just a few scattered papers in early years (1987-2000) that received a small number of citations. however, two papers in 2002 received a massive number of citations, and likely led to the substantially increased interest in the field. the number of papers has steadily increased since 2002, with close to 250 papers in 2018, showing that the area continues to enjoy considerable attention. one can also fine tune the search as desired. say we are interested not in the broad area of sentiment analysis, but specifically in the work on emotions and affect. then they can enter only emotion-and affect-related keywords. a disadvantage of using terms for search is that some terms are ambiguous and they can pull in irrelevant articles; also if a paper is about the topic of interest but its title does not have one of the standard keywords associated with the topic, then it might be left out. that said, if one does come across a paper that has the query term but is not in the topic of interest, they can right click and exclude that paper from the visualization; and as mentioned before, future work will allow for searches in the abstract and full text as well. we are also currently working on clustering papers using the words in the articles as features. @xcite below are some more examples of interactions with nlp scholar (figures are in the appendix after references): once the system goes live, we hope to collect further usage scenarios from the users at large. for this work, we chose not to stem the terms in the titles before applying the search. this is because in some search scenarios, it is beneficial to distinguish the different morphological forms of a word. for example, papers with emotions in the titles are more likely to be dealing with multiple emotions than papers with the term emotion. when such distinctions do not need to be made, it is easy for users to include morphological variants as additional query terms. ## conclusions and future work we presented nlp scholar-an interactive visual explorer for the acl anthology. notably, the tool also has access to citation information from google scholar. it includes several interconnected interactive visualizations (dashboards) that allow users to quickly and efficiently search for relevant related work by clicking on items within a visualization or through search boxes. all of the data and interactive visualizations associated with this work are freely available through the project homepage. @xcite future work will provide additional functionalities such as search within abstracts and whole texts, document clustering, and automatically identifying related papers. we see nlp scholar, with its dedicated visual search capabilities for nlp papers, as a useful complementary tool to existing resources such as google scholar. we also note that the approach presented here is not required to be applied only to the acl anthology or nlp papers; it can be used to display papers from other sources too such as pre-print archives and anthologies of papers from other fields of study. figures 6 through 20 (in the pages ahead) show example interactions with nlp scholar that were discussed in section 5.