Topics, Authors, and Institutions in Large Language Model Research: Trends from 17 K ar X iv Papers (2024)

Abstract: Large language models (LLMs) are dramatically influencing AI research, spurring discussions on what has changed so far and how to shape the field’s future. To clarify such questions, we analyze a new dataset of 16,979 LLM-related arXiv papers, focusing on recent trends in 2023 vs. 2018-2022. First, we study disciplinary shifts: LLM research increasingly considers societal impacts, evidenced by 20 \times growth in LLM submissions to the Computers and Society sub-arXiv. An influx of new authors – half of all first authors in 2023 – are entering from non-NLP fields of CS, driving disciplinary expansion. Second, we study industry and academic publishing trends. Surprisingly, industry accounts for a smaller publication share in 2023, largely due to reduced output from Google and other Big Tech companies; universities in Asia are publishing more. Third, we study institutional collaboration: while industry-academic collaborations are common, they tend to focus on the same topics that industry focuses on rather than bridging differences. The most prolific institutions are all US- or China-based, but there is very little cross-country collaboration. We discuss implications around (1) how to support the influx of new authors, (2) how industry trends may affect academics, and (3) possible effects of (the lack of) collaboration.

## introduction recent advances in language modeling have caused disruptive shifts throughout ai research, spurring conversation about how the field is changing and how it should change. discussions so far have drawn on surveys and interviews of nlp community members @xcite @xcite , and recurring themes under consideration include which topics are shifting in 1 co-first authors. 2 co-senior authors. in periods of flux like this one, bibliometricsquantitative study of publication patterns -offers a useful lens. prior bibliometric analyses have been clarifying in nlp, identifying topic shifts @xcite , flows of authors in and out of the field @xcite , and the growing role of industry @xcite . due to the rapid recent growth of the llm literature, fundamental questions about the topics, authors, and institutions driving its growth remain understudied. addressing this gap, we conduct a bibliometric analysis of recent trends in the llm literature, focusing on changes in 2023 compared to 2018-22. we collect and analyze 16,979 llm-related papers posted to arxiv from jan. 1, 2018 through sep. 7, 2023. in addition to arxiv metadata, we annotate these papers with topic labels, author affiliations, and citation counts, and make all data and code publicly available. 1 we analyze three questions: human-computer interaction, security, and software engineering. simultaneously, bert and task-specific architectures are shrinking due to centralization around newer models (e.g., gpt-4 and llama). the increased focus on societal impacts and non-nlp disciplines is driven by a strikingly large proportion of authors new to llms. half (49.5%) of llm first authors in 2023 have never previously co-authored an nlp paper (and nearly two-thirds haven't previously co-authored an llm paper), and a substantial fraction (38.6%) of the last authors haven't either. these new authors are entering from other fields like computer vision, software engineering, and security, and they are writing llm papers on topics further out from nlp, e.g., vision-language models, applications in the natural sciences, and privacy/adversarial risks. how might these insights inform the nlp community, policymakers, and other stakeholders in the future of llm research? first, our analysis of topics and authors reveals that llms are increasingly being applied to diverse fields outside of core nlp. researchers performing interdisciplinary work should involve domain experts in both nlp and the other areas of study (e.g. education, medicine, law); community leaders should reflect on how publication venues and peer review processes can best accommodate interdisciplinary work. we also show that llm research is experiencing an influx of new authors, implying heightened value of educational resources, research checklists @xcite , and other frameworks to encourage good research practice @xcite . second, we find that while industry continues to lead much of the most impactful research, large tech companies are publishing less overall. academics lead important research on society-facing applications and harms of ai, but closed-source models hinder detailed evaluations @xcite . opensource datasets and models are therefore increasingly valuable, and the community should consider how to better incentivize these contributions. third, we provide evidence of a lack of collaboration between the us and china, substantiating concerns about ai-related competition @xcite . institutions may have incentives that make collaboration difficult, but efforts to create consensus may help avoid unethical or risky uses of ai. by characterizing these recent changes in the llm research landscape, our work aims to ground discussions on the policies and practices that will shape the field's future. ## methods we summarize our data and methods here and provide full details in appendix a; table s1 lists the fields we use in our analysis. our primary dataset consists of all 418k papers posted to the cs and stat arxivs between january 1, 2018 and september 7, 2023. following past ml survey papers @xcite @xcite , we identify an analysis subset by searching for a list of keywords in paper titles or abstracts. keyword search has the benefits of transparency, simplicity, and consistency with past work, but also has caveats; see §a.2 for further details. our keyword list surfaces 16,979 papers; the specific terms we include are {language model, foundation model, bert, xlnet, gpt-2, gpt-3, gpt-4, gpt-neo, gpt-j, chatgpt, palm, llama}. details about this list are in §a.2. we define several fields for each paper in this subset. in doing so, we follow past work and conduct manual audits to assess the reliability of our annotations; however, there remain inherent limitations in how these fields are defined, as we discuss in appendix a. we tracked a paper's primary sub-arxiv category, e.g., computation and language (cs.cl). for more fine-grained topic analysis, we assigned each paper one of 40 llm-related topics ( §a.3). we clustered embeddings of paper abstracts @xcite , then titled the clusters using a combination of llm annotation and manual annotation. we annotated papers for whether their authors list academic or industry affiliations ( §a.4). we pulled citation counts from semantic scholar @xcite , and tracked the citation percentile for each paper: the percentile of its citation count relative to papers from the same 3-month window ( §a.5). ## results past work has shown that the raw count of llm papers has risen steeply @xcite . these trends replicate on arxiv: 12% of all cs/stat papers were llm-related in mid-2023 (figure s1 ). compared to all other topics (figure s2 ) and all words/bigrams in paper abstracts (table s2 ), llm-& generative ai-related topics and terms are growing fastest. we dissect these ongoing changes by studying the topics, authors, and institutions that are accounting for them. ## related work our work applies methods from bibliometrics and science of science @xcite to understand the emerging llm literature. these methods have a rich history in nlp; since the 2008 release of the acl anthology, several papers have applied quantitative methods to trace topic changes and paradigm shifts in the anthology over time @xcite @xcite . another line of work focuses specifically on citation patterns: wahle et al. (2023) study cross-disciplinary cites from nlp to other fields, singh et al. (2023) show that nlp papers are citing increasingly recent work (and less older work), and rungta et al. (2022) identify geographic disparities in citation counts of published papers. some bibliometrics papers also ask similar research questions to ours. abdalla et al. ( 2023 ) study big tech companies in nlp research, identifying key players and showing significant growth in overall industry presence since 2017. their findings complement our more specific analysis of the language modeling literature, in which there are different trends than nlp overall (for example, we find that industry research actually declined relative to academia in 2023). @xcite apply bibliometric methods to a similar paper corpus, pulling 5,752 published language modeling papers from web of science from jan 2017 -feb 2023. they perform topic modeling and analyze co-citation networks to taxonomize the llm literature, and they also study cross-country collaboration. our work differs in that we focus on specific changes in 2023 compared to years prior (rather than analyzing the entire time period collectively), and we use the arxiv in order to capture a recent sample with less publication delay. our empirical analyses also cover different questions, with more focus on emerging topics, first-time authors, and industry/academia dynamics. other related work uses qualitative methods to understand recent shifts in nlp. @xcite present a retrospective analysis centered on 26 interviews with nlp researchers, identifying key trends like the rise of benchmarks, software centralization, and reduced barriers to enter nlp. @xcite report on the results of the nlp community meta-survey, in which 327 researchers were polled for their views on the status and future of nlp. @xcite draw historical analogy from the convergence around llms to the early statistical machine translation era, suggesting lessons and evergreen research directions. it is encouraging that many other researchers have approached similar questions; together, the rich variety of datasets and methods can help us parse the ongoing expansion and flux of nlp. ## discussion we conduct a bibliometric analysis documenting recent changes and trends in the llm research landscape. our findings invite discussion on how researchers and policymakers should react to these changes. drawing on our findings, we close with open questions for community reflection: