What We Talk About When We Talk About LM s: Implicit Paradigm Shifts and the Ship of Language Models (2025)

Abstract: The term Language Models (LMs) as a time-specific collection of models of interest is constantly reinvented, with its referents updated much like the *Ship of Theseus* replaces its parts but remains the same ship in essence. In this paper, we investigate this *Ship of Language Models* problem, wherein scientific evolution takes the form of continuous, implicit retrofits of key *existing* terms. We seek to initiate a novel perspective of scientific progress, in addition to the more well-studied emergence of *new* terms. To this end, we construct the data infrastructure based on recent NLP publications. Then, we perform a series of text-based analyses toward a detailed, quantitative understanding of the use of Language Models as a term of art. Our work highlights how systems and theories influence each other in scientific discourse, and we call for attention to the transformation of this Ship that we all are contributing to.

## introduction scientific publications expand exponentially, with the size of literature doubling every ∼17 years @xcite . the field of cl/nlp is no exception; in fact, the doubling only took 5 years: as of 2023, the number of papers documented in the acl anthology is twice as much as the total by 2018 @xcite . with the explosion of new publications, it is imperative but also increasingly challenging to sort out the major contexts, progress, and future directions of the field. researchers have sought to identify emergent key terms and factors that led to disruptive shifts of paradigms @xcite @xcite . in this period of flux, however, an ever-evolving field like ours calls for deeper analysis beyond identifying these elements, 1 the data and code of this work is available at https: //github.com/curlyzhu/ship_of_lms . in order to understand various quantitative questions regarding these rapid and disruptive shifts. for instance, to what extent is the field transformed by any certain model, like chatgpt (openai, 2022)? how does the popularity of the latest gpt models compare with, say, that of bert @xcite in 2020? from there, we might even be curious about bigger questions like, "how unprecedented really is chatgpt?", where our empirical guesses can diverge drastically without sufficient quantitative evidence. while readers of this paper likely come in with a tacit understanding of the ebb and flow of the field, it is hard to nail down such factors that keep changing in publications. more fundamentally, the narrative describing scientific progress as the emergence of new elements does not cover the more implicit paradigm shifts, which features the evolution instead of invention of terms. the (forms of) key terms may continue to be broadly used, but are gradually overwritten with new meanings in new contexts. language models (lms), as a term of art, refers to no single, static thing. it is used referentially to index a collection of models deemed relevant and representative at the time or in the context of a paper. as this is everchanging, we are faced with a ship of theseus scenario @xcite , wherein the same terminology is essentially re-invented and its referents are perhaps entirely replaced. as such, a subtle gap between the durable collective term of "lms" and the timespecific referent models of the moment is widening as the field progresses, threatening its stability and accessibility to new researchers. these issues call for new analyses of the subtle transformations that result from these paradigm shifts. in this paper, we seek a quantitative description of a field's continuous evolution. more specifically, we inquire into the ship of lms paradox, i.e., the aforementioned reconstruction of the term language models. we decipher this evergreen term's rapidly changing referents, contexts, and usages across time. we develop a semi-automatic, generalizable framework to extract and organize two closely related sets of keywords: (1) mentions of the collective lm concept, and (2) specific model names, and construct a dataset of 7,650 papers from the 10 most recent major nlp conferences. we focus on several questions concerning how we as a field talk about lms: how often do we talk about lms, and how confidently? ( §4.1) which models, and what is special about these components? ( §4.2) moreover, how do the referents of lm vary across papers? ( §4.3) finally, we conclude the findings and future perspectives in §5. our work highlights the astonishing extent of change subtly encoded in the seemingly unchanged overarching terms. we hope the ship of lms can serve as a new perspective to understand the field's progress, and that our methodology can serve as an entry point for finer-grained measurements of subtle changes in rapidly growing scientific fields. ## experiments and findings lms have been steadily gaining more attention from the field. @xcite reports that papers containing the key phrase "language model" have increased from less than 400 pre-2019 to around 10,000 in 2023. we observe a similar trend with a finer-grained search in the nlp domain (figure 2(a) ). at acl 2020, 35% of the papers contain at least one lm mention from l (we refer to this portion as lm-related papers). since then, this proportion has had a smooth, continuous growth of approximately 5% (additive) per conference, hitting 84% just three years later at emnlp 2023. ## concluding remarks in this work, we go over the past and present of the enduring term of language model(s), based on an original dataset from the latest major conferences. we sort out the subtle, continuous shifts in the practical meaning of lms, and witness how the retrofits eventually accumulate to a brand new ship of language models and the nuances in the actual referents. we quantify and visualize the drastic change in the planks and timber, emphasizing the shortened period of reconstruction and the presence of dominant components. finally, we highlight the snowballing context difference between the lm-centered research and the more peripheral applications and the consequences of that. approaching the most recent epochal crossroads where revolutionary shifts can happen within months or weeks, we have in fact seen more interesting signs: the unprecedented dissimilarity of compositions, or on the other hand, a more diverse, multipolar representation in model choices, to name a few. perhaps what is crafted from our hands has even brought up throught the age of sail, where a now steam-engined argo is ready to set sail towards the exciting yet mysterious fanta-seas. future directions our findings depict the most dramatic reshuffles of models, while the nuances of lms can be found similarly at the underlying architecture level, or different variations of models. besides, causal links between our work and the identified key factors in prior works may help depict the field's trajectories, and qualitative studies such as interviews with paper authors can also help understand the different patterns and beliefs. beyond the lm-specific discussions, we also highlight the broad existence of the ship of theseus scenario in various other terms, topics, and fields. for instance, it would be intriguing if similar analyses can be applied to understand how the interpretations of fairness or safety in the ai context were enriched over time.