A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why? (2023)

Abstract: Understanding the fundamental concepts and trends in a scientific field is crucial for keeping abreast of its continuous advancement. In this study, we propose a systematic framework for analyzing the evolution of research topics in a scientific field using causal discovery and inference techniques. We define three variables to encompass diverse facets of the evolution of research topics within NLP and utilize a causal discovery algorithm to unveil the causal connections among these variables using observational data. Subsequently, we leverage this structure to measure the intensity of these relationships. By conducting extensive experiments on the ACL Anthology corpus, we demonstrate that our framework effectively uncovers evolutionary trends and the underlying causes for a wide range of NLP research topics. Specifically, we show that tasks and methods are primary drivers of research in NLP, with datasets following, while metrics have minimal impact.

## introduction experts in a field sometimes conduct historical studies to synthesize and document the key research ideas, topics of interest, methods, and datasets that shaped a field of study. they document how new research topics eclipsed older ones and contributed to shaping the trajectory of the research area @xcite . aspiring scientists learn the craft of their discipline by delving into the examination of past scientific accomplishments documented in research papers. however, conducting such a historical study is challenging: experts in a field rely on years of experience and peruse large amounts of past published articles to determine the chronological progression of a research field. further, the exponential growth of scientific publications in recent years has rendered it arduous even for domain experts to stay current. therefore, an automated method to track the temporal evolution of research topics can be beneficial in offering an overview of the field and assisting researchers in staying abreast of advancements more efficiently. in this work, we propose a systematic framework to examine the evolutionary journey of research topics within the realm of natural language processing (nlp), harnessing causal discovery and inference techniques. prior research on historical analysis of nlp has predominantly concentrated on scrutinizing metadata associated with research papers @xcite @xcite @xcite such as number of citations, title, author profile, affiliation, and publication venue. these studies have examined the research trends through unigram or bigram frequency analysis, but they do not provide insights into the underlying causes propelling these research topics. our study centers on four distinct fundamental types of entities in nlp research: tasks representing well defined problems; methods, signifying the solutions or approaches employed to tackle the tasks; datasets, indicating the relevant textual resources such as corpora and lexicons; and metrics, encompassing the evaluation techniques tailored to specific tasks. we abbreviate these types as tdmm for short. specifically, we examine the interplay between an nlp task that is commonly viewed as a focused research topic (e.g., machine translation) and the key entities that exert pivotal influence on the target task (such as "bleu" @xcite or "transformers" @xcite ). numer of mt papers per year year research trends and the top causal entities for machine translation symbolic nlp period statistical nlp period neural nlp period pretrained llms period type entity causal effect method statistical models 0.2505 metric perplexity 0.2504 dataset wordnet 0.2497 task language generation 0.2492 1990-2002 type entity causal effect dataset wmt dataset 0.2528 metric bleu 0.2495 method attention mechanism 0.2489 task language generation 0.2486 2003-2017 type entity causal effect method transformers 0.2515 metric meteor 0.2507 task language generation 0.2503 dataset wmt dataset 0.2470 2018-2022 figure 1: evolution of machine translation (mt) research. blue line: number of mt papers (1979-2022). tables show the top causal entities/types for different periods (excluding 1979-1989 due to limited mt papers). t? (b) are there discernible causal relationships between t and e? (c) what is the extent of the causal impact exerted by e on t? unlike uban et al. (2021) and koch et al. ( 2021 ) that heavily rely on manual annotations and have limited coverage, our analysis is based on tdmm entities automatically extracted from 55k papers in the acl anthology 2 . our framework not only recognizes the key entities driving the research direction of a research topic but also measures the causal effects of these entities on the target topic in an end-to-end fashion. figure 1 shows the most influential entities for machine translation (mt) in different time periods. for instance, "statistical models" used to be the popular method for mt in 1990-2002, and the evaluation metric "bleu" is one of the top causal entities driving the mt research in 2003-2017. in the era of pre-trained large language models (llms) starting from 2018, "transformer" has become the popular method for mt. for another research topic of "speech recognition", our framework uncovers the influential role of "language modeling" between 1979 to 2022, where speech recognition models utilize probability scores from language models to recognize coherent text from speech @xcite . in this work, we analyze 16 tasks from a diverse set of research areas identified by acl 2018 organizers. our framework is versatile and applicable to other tasks and domains, benefiting both young and experienced researchers. it can aid in literain summary, we make three-fold contributions in this study: firstly, we propose a framework to quantify research activities, including (1) trends and stability of an nlp research task, and (2) relation intensity between tdmm entities and nlp research tasks. secondly, we employ causal analysis algorithms to uncover causal structures and measure effects between tasks and related tdmm entities (tdmm-task causal analysis). to the best of our knowledge, this represents the first historical study of a scientific research anthology from a causal perspective. finally, through extensive experiments on the acl anthology, we offer an empirical overview of the nlp research landscape. in the following sections, we will refer to tdmm-task causal analysis as causal analysis. ## entity influence in nlp research: a regression analysis before conducting the causal analysis, we aim to identify the key variables that significantly impact the evolution of nlp research. specifically, we investigate which types of entities exert the most variables r-squared (↑) unique tasks 0.87 + unique datasets 0.91 + unique methods 0.93 + unique metrics 0.97 method. mathematically, we predict the number of task entities y t in a given year t as a function of the cumulative counts of all types of entities {x evaluation. we evaluate the regression model using the r 2 measure (coefficient of determination) to assess the goodness of fit. additionally, we perform a null hypothesis test to determine the statistical significance of the partial regression co-efficients. results and discussion. 1) optimized number of variables. in our initial experiment, we determine the optimal number of variables and summarize the corresponding r 2 values in ## causal methodology for nlp research analysis drawing on the insights gained from the regression analysis (section 4), we now establish the cornerstone of our study by defining three causal variables that drive the causal analysis in the subsequent sections. using causal discovery and inference techniques, we analyze the causal relationships among the variables and measure the impact of tdmm entities on target task entities based on these relationships. figure 2 illustrates the architecture that underpins our framework. ## results and analysis correlation-based measures provide a simple way to quantify the association between variables. however, they fall short of explaining complex causeeffect relationships and can yield misleading results. causality is essential for gaining a deeper understanding of variable relationships, enhancing the robustness and reliability of our findings beyond the limitations of correlation. we discuss more about the importance of causal methods over correlation-based measures in section 7. in this section, our focus is on uncovering relationships among causal variables (section 6.1) and measuring the impact of tdmm entities on target task entities (section 6.2). ## concluding remarks in this paper, we retrospectively study nlp research from a causal perspective, quantifying research trends of task entities and proposing a systematic framework using causal algorithms to identify key reasons behind the emergence or disappearance of nlp tasks. our analysis reveals that tasks and methods are the primary drivers of research in nlp, with datasets following their influence, while metrics have minimal impact. it is important to note that in our analysis, we have structured the reference time into four distinct intervals (see we believe our causal analysis enhances understanding of the interplay of research entities in nlp, contributing to the growing body of work on causality and nlp @xcite . we provide with additional analysis and insights in appendix b.