To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing (2023)

Abstract: NLP is in a period of disruptive change that is impacting our methodologies, funding sources, and public perception. In this work, we seek to understand how to shape our future by better understanding our past. We study factors that shape NLP as a field, including culture, incentives, and infrastructure by conducting long-form interviews with 26 NLP researchers of varying seniority, research area, institution, and social identity. Our interviewees identify cyclical patterns in the field, as well as new shifts without historical parallel, including changes in benchmark culture and software infrastructure. We complement this discussion with quantitative analysis of citation, authorship, and language use in the ACL Anthology over time. We conclude by discussing shared visions, concerns, and hopes for the future of NLP. We hope that this study of our field’s past and present can prompt informed discussion of our community’s implicit norms and more deliberate action to consciously shape the future.

## introduction natural language processing (nlp) is in a period of flux. the successes of deep neural networks and large language models (llms) in nlp coincides with a shift not only in the nature of our research questions and methodology, but also in the size and visibility of our field. since the mid-2010s, the number of first-time authors publishing in the acl anthology has been increasing exponentially (figure 1 ). recent publicity around nlp technology, most notably chatgpt, has brought our field into the public spotlight, with corresponding (over-)excitement and scrutiny. in the 2022 nlp community metasurvey, many nlp practicioners expressed fears that private firms exert excessive influence on the field, that "a majority of the research being published in nlp is of dubious scientific value," and that ai technology could lead to a catastrophic event this century @xcite . more recently, there has been discussion of the increasing prevalence of closed-source models in nlp and how that will shape the field and its innovations @xcite @xcite . in order to tackle these big challenges, we must understand the factors -norms, incentives, technology and culture -that led to our current crossroads. we present a study of the community in its current state, informed by a series of long-form retrospective interviews with nlp researchers. our interviewees identify patterns throughout the history of nlp, describing periods of research productivity and stagnation that recur over decades and appear at smaller scale around prominent methods ( §3). interviewees also point out unparalleled shifts in the community's norms and incentives. aggregating trends across interviews, we identify key factors shaping these shifts, including the rise and persistence of benchmarking culture ( §4) and the maturation and centralization of software infrastructure ( §5). our quantitative analysis of citation patterns, authorship, and language use in the acl anthology over time provides a complemen-tary view of the shifts described by interviewees, grounding their narratives and our interpretation in measurable trends. through our characterization of the current state of the nlp research community and the factors that have led us here, we aim to offer a foundation for informed reflection on the future that we as a community might wish to see. ## exploit-explore cycles of work our participants described cyclical behavior in nlp research following methodological shifts every few years. many participants referred to these methodological shifts as "paradigm shifts", with similar structure, which we characterize as explore and exploit phases. 1 first wave: exploit. participants suggested that after a key paper, a wave of work is published that demonstrates the utility of that method across varying tasks or benchmarks. interviewees variously describe this stage as "following the bandwagon" (17), "land grab stuff" (9), or "picking the low-hanging fruit" (8). a researcher with prior experience in computer vision drew parallels between the rise of bert and the computer vision community after alexnet was released, where "it felt like every other paper was, 'i have fine tuned imagenet trained cnn on some new dataset' " (17). however, participants identified benefits of this "initial wave of showing things work" (17) in demonstrating the value of techniques across tasks or domains; in finding the seemingly obvious ideas which do not work, thus exposing new areas to investigate; and in developing downstream applications. when one researcher was asked to identify their own exploit work, they joked we could simply "sort [their] google scholar from most cited to least cited" (9), describing another incentive to publish the first paper applying a new methodology to a particular task. participants additionally described doing exploit work early in their careers as a hedge against riskier, longer-term projects. however, most participants ascribed low status to exploit work foot_1 , with participants calling these projects "low difficulty" (4) and "obvious idea[s]" (9) with "high probability of success" (4). participants also discussed increasing risk of being "scooped" (7,9,4) on exploit work as the community grows larger. participants felt that the exploit phase of work is not sustainable. eventually, "the low hanging fruit has been picked" (8); this style of work becomes unsurprising and less publishable. as one researcher put it: "if i fine tune [bert] for some new task, it's probably going to get some high accuracy. and that's fine. that's the end." (17). second wave: explore after some time in the exploit phase, participants described a state where obvious extensions to the dominant methodology have already been proposed, and fewer papers demonstrate dramatic improvements over the state of the art on popular benchmarks. in this phase, work on identifying the ways that the new method is flawed gains prominence. this work may focus on interpretability, bias, data or compute efficiency. while some participants see this as a time of "stalled" (8) progress, others described this as "the more interesting research after the initial wave of showing things work" (18). a mid-career participant identified this as the work they choose to focus on: "i'm at the stage of my career where i don't want to just push numbers, you know. i'll let the grad students do that. i want to do interesting stuff" (22). participants often saw "pushing numbers" as lower-status work, appropriate for graduate students and important for advancing the field and one's career, but ultimately not what researchers hope to explore. some work to improve benchmark performance was also perceived as explore work, particularly when it involved developing new architectures. one participant described a distinction between "entering a race" and "forging in a new direction" (4) with a project, which focuses the exploit/explore divide more on the perceived surprisingness of an idea rather than the type of contribution made. exploration often leads to a new breakthrough, causing the cycle to begin anew. ## software lotteries hooker (2021) argues that machine learning research has been shaped by a hardware lottery: an idea's success is partially tied to its suitability for available hardware. several participants spoke about software in ways that indicate an analogous software lottery in nlp research: as the community centralizes in its software use, it also centralizes in methodology, with researchers' choices of methods influenced by relative ease of implementation. this appeared to be a relatively new phenomenon; participants described previously using custom-designed software from their own research group, or writing code from scratch for each new project. centralization on frameworks as deep learning became more popular in nlp, participants described the landscape shifting. as tensorflow @xcite increased support for nlp modeling, pytorch @xcite was released, along with nlp-specific toolkits such as dynet @xcite and allennlp @xcite , and "everything started being [...] 2010 2012 2014 2016 2018 2020 2022 year 0.00 0.05 0.10 0.15 0.20 0.25 0.30 fraction of papers that mention mentions of software libraries by year huggingface pytorch theano dynet moses tensorflow figure 3: mentions of libraries over time in the acl anthology. note the cyclic pattern and increasing concentration on the dominant framework over time. while some libraries are built on others, the shift in mentions over time captures the primary level of abstraction that researchers consider important. see appendix b for details on how we handle ambiguity in mentions. centralization on specific models participants identified another shift after the release of bert and subsequent development of hugging face. because of pre-training, participants moved from merely using the same libraries to "everyone us[ing] the same base models" (9). participants expressed concern that this led to further centraliza-tion in the community, with one participant identifying a trend of some people "not us @xcite anything else than bert [...] that's a problem" (5). this concentration around particular modeling choices has reached greater heights than any previous concentration on a method; in 2021, 46.7% of papers in the acl anthology cited bert @xcite ; figure 2 ) 5 . other large models are only available to researchers via an api. one participant who works on llms in industry argued that black-box llms, while "non-scientific" (15) in many ways, were like large-scale technical tools constructed in other disciplines, drawing a parallel to particle physics: "computer science is getting to have its cern moment now [...] there's only one large hadron collider, right?" (15). this participant argued that nlp has become a field whose frontiers require tools that are beyond most organizations' resources and capabilities to construct, but nonetheless are widely adopted and set bounding parameters for future research as they see wide adoption. in this vision, black-box llms take on the same role as the lhc or the hubble space telescope (both notably public endeavors, unlike most llms), as tools whose specifications are decided on by a few well-resourced organizations who shape significant parts of the field. but most participants expressed skepticism about the scientific validity of experiments on black-box llms, with one participant referencing critiques of early-2000s ir research on google @xcite . centralization on python while most earlycareer and late-career participants did not express strong opinions about programming languages, many mid-career participants expressed strong dislike for python, describing it as "a horrible language" ( 22 ) with efficiency issues that are "an impediment to us getting things done" (20) and data structure implementations that are "a complete disaster in terms of memory" (9). one participant described their ideal next paradigm shift in the field as a shift away from using python for nlp. yet even the participants who most vehemently opposed python used it for most of their research, citing the lack of well-supported nlp toolkits or community use in other languages. this is an instance of a software lottery at a higher level, where the dominance of a single programming language has snowballed with the continued development of research artifacts in that language. ## related work the shifts we explore in this paper have not happened in a vacuum, with adjacent research communities such as computer vision (cv) and machine learning (ml) experiencing similar phenomena, inspiring a number of recent papers discussing norms in ai more broadly. @xcite analyze a set of the most highly cited papers at recent ml conferences, finding that they eschew discussion of societal need and negative potential, instead emphasizing a limited set of values benefiting relatively few entities. others have noticed that corporate interests have played an increasing role in shaping research, and quantified this with studies of author affiliations over time in machine learning @xcite and nlp @xcite . @xcite study the tangible emotional impact of recent dramatic growth in the cv community by asking community members to write stories about emotional events they experienced as members of their research community. while we focus on summarizing and synthesizing the views of our participants, some of the overarching themes identified in this work have been discussed more critically. fishman and hancox-li (2022) critique the unification of ml research around transformer models on both epistemic and ethical grounds. position papers have critiqued the notion of general purpose benchmarks for ai @xcite , and emphasized the importance of more careful and deliberate data curation in nlp @xcite . the nlp community metasurvey @xcite provides a complementary view to this work, with their survey eliciting opinions from a broad swath of the *cl community on a set of 32 controversial statements related to the field. the survey also asked respondents to guess at what the most popular beliefs would be, eliciting sociological beliefs about the community. while there is no direct overlap between our questions and metasurvey questions, participants raised the topics of scaling up, benchmarking culture, anonymous peer review, and the role of industry research, which were the subject of metasurvey questions. where we can map between our thematic analysis and metasurvey questions, we see agreement-e.g. many of our participants discussed others valuing scale, but few placed high value themselves on scaling up as a research contribution. the availability of the acl anthology has enabled quantitative studies of our community via patterns of citation, authorship, and language use over time. @xcite perform a topic model analysis over the anthology to identify different eras of research and better understand how they develop over time, and analyze factors leading authors to join and leave the community. mohammad (2020) analyze citation patterns in *cl conferences across research topics and paper types, and @xcite specifically inspect the phenomenon wherein more recent papers are less likely to cite older work. @xcite provide a view of paradigm shifts in the nlp community complementary to ours based on a diachronic analysis of the acl anthology, inferring causal links between datasets, methods, tasks and metrics. shifts in norms and methods in science more broadly has been studied outside computing-related fields. most notably, @xcite coined the term paradigm shift in the structure of scientific revolutions. his theory of the cyclic process of science over decades or centuries has some parallels with the (shorter timescale) exploit-explore cycles discussed in this work. note that in this work, we did not prime participants with an a priori definition of paradigm shift, allowing each participant to engage with the term according to their own interpretation, which often differed from kuhn's notion of a paradigm shift. ## the future the rise of large language models has coincided with disruptive change in nlp: accelerating centralization of software and methodologies, questioning of the value of benchmarks, unprecedented public scrutiny of the field, and dramatic growth of the community. a shift like this can feel threatening to the fundamental nature of nlp research, but this is not the first period of flux in the field, nor are the fundamental forces enabling llms' dominance and other changes entirely new. our participants described cycles of change in the nlp community from mid-80s to the present, with common themes of first exploiting and then exploring promising methodologies. each methodological shift brought corresponding cultural change: the shift from symbolic to statistical methods brought about the rise of benchmark culture and the end of the socially mediated, smallnetwork acl community. neural methods began the centralization on software toolkits and the methodologies they support. pre-training intensified this software lottery, causing unprecedented levels of centralization on individual methods and models. current models have called into question the value of benchmarks and catapulted nlp into the public eye. our participants largely agree on the resulting incentives-to beat benchmark results, to do the easiest thing rather than the most fulfilling, to produce work faster and faster -while largely expressing frustration with the consequences. we hope that this contextualization of the current state of nlp will both serve to inform newer members of the community and stir informed discussion on the condition of the field. while we do not prescribe specific solutions, some topics of discussion emerge from the themes of this work: we personally take great hope for our community from this project. the care with which all participants reflected on the shape of the field suggests to us that many people are concerned about these issues, invested in the community, and hopeful for the future. by sharing publicly what people so thoughtfully articulate privately, we hope to prompt further discussion of what the community can do to build our future.