Awes, Laws, and Flaws From Today’s LLM Research (2025)

Abstract: We perform a critical examination of the scientific methodology behind contemporary large language model (LLM) research. For this we assess over 2,000 research works released between 2020 and 2024 based on criteria typical of what is considered good research (e.g. presence of statistical tests and reproducibility), and cross-validate it with arguments that are at the centre of controversy (e.g., claims of emergent behaviour). We find multiple trends, such as declines in ethics disclaimers, a rise of LLMs as evaluators, and an increase on claims of LLM reasoning abilities without leveraging human evaluation. We note that conference checklists are effective at curtailing some of these issues, but balancing velocity and rigour in research cannot solely rely on these. We tie all these findings to findings from recent meta-reviews and extend recommendations on how to address what does, does not, and should work in LLM research.

## introduction large language models (llms) foot_0 are a powerful technology. they can follow instructions and output coherent, persuasive text. this has made them the centre of attention in academia, industry, and the media. given the potential funding associated with these technologies, it should not be a surprise that news and research works are sometimes accompanied by bold claims about their capabilities. it has been said that the focus of ai research is the approaches, rather than the results. that is, less importance is allocated to issues around experimental protocols when compared to other fields @xcite . for example, papers sometimes lack sufficient details for independent verification @xcite ; or report aggregate performances (e.g., accuracy) without providing detailed protocols or error breakdowns @xcite @xcite @xcite . this is common in health sciences @xcite , security @xcite , and recommender systems @xcite )-all areas where llms are increasingly being applied. we argue, however, that in llm-related research there has been a shift towards result-driven experimentation, partly due to their availability and capabilities; but also due to scrutiny from the media, funding sources, and the field. in theory, this should mean that the scientific community ought to allocate the same relevance to experimental protocols and good research practices as other fields. this is easier said than done, since llms may be closed source, expensive to train, and/or limited to a versioned api call. solving a problem could be done one single prompt, and their ability to generate text reduces the time taken to write and produce papers. a combination of these could lead to a rapidly-increasing volume of scientific articles with strong claims and lacklustre experimental practices. however, llms as a technology are not the only factor: evaluating these models is notoriously difficult, researchers are under pressure to keep up, and peer-reviewing systems are usually overloaded. still, all of this impedes the scientific community's ability to transparently evaluate and understand llms, and to ensure their responsible use. it also poses questions about the validity and trustworthiness of some findings and claims, especially around llm capabilities. our work is a statistically motivated critique, meant to systematically and critically examine the scientific methodology employed in llm research, and to quantify the extent to which these issues occur in the literature. 2 we look at what does, does not, and should work in llm research, and, based on that, extend recommendations to the scientific community at large to retain velocity without sacrificing innovation or good research practices. ## background some known challenges to ai research are particularly salient in llm literature, although might 3 see, e.g. aaai (2024); neurips (2024) and @xcite . require reframing. in this section we describe and tie them to the criteria and focus of our analysis. reproducibility: llm-related reviews have found that the most downloaded models in huggingface do not consistently provide the same amount of information in their documentation @xcite ; and an analysis on the experimental protocols of over 40 chat-based llms (e.g., availability of data, weights, licences, etc) found that, while many projects claimed some level of open-sourcing, documentation was 'exceedingly rare' @xcite . beyond that, reproducibility itself is tied to stochasticity and versioning, which in turn raises concerns about the results themselves, particularly around the fact that errors and biases are rarely reported or analysed @xcite . one core aspect of reproducibility, beyond protocol declaration, is open-sourcing. for llm research in particular, this doesn't readily apply to models behind apis. while in the broader cs field open-sourcing has been trending upward, reproducibility remains difficult @xcite . further, in some areas it has been found to be no statistically significant difference in the presence of artifacts (code, data) when the mechanisms for reproducibility were implemented @xcite . this in turn suggests that, due to llm stochasticity and system limitations, open-sourcing alone is preferrable, albeit not a cure-all: but protocol declaration, however, remains important. measurement is difficult in llms due to effectiveness and scalability of metrics. it has long been known that automated metrics like bleu do not capture natural language generation well @xcite , and not correlate well with human judgements @xcite , which themselves have their own complications @xcite . while it has been argued that metric performance is not as important as tasks and methods, and mostly drive improvement within the task @xcite , benchmarks and leaderboards are the reigning way to define llm success. these have problems scaling, however, given how flexible llms are at multiple tasks. although there is a push to use them as evaluators, there is no consensus on the viability of this approach: arguments in favour @xcite @xcite @xcite are as plentiful as arguments against @xcite @xcite @xcite . one reason why this approach is often called unreliable is due to the fact that llms might memorise their training data @xcite , including evaluation benchmarks @xcite ; their results are very sensitive to the prompt's phrasing @xcite ; and that, generally speaking, an nlg system's performance, including evaluator systems, is strongly dependent on the choice of metric @xcite . it is clear that carefully-chosen, diverse metrics and statistical tests are necessary for trustworthy results. claims: many aspects of llm research are related to their (or lack thereof) capabilities. for example, it is often said that llms present emergent abilities. this is usually defined as their ability to solve more complex problems in a way not predictable by, say, parameter size @xcite . this definition is slightly vague, given that the problems or their measures of complexity are not actually defined. another common claim associated with llm research are claims of artificial general intelligence (agi). analogous to emergence, agi claims often rely on disparate definitions and goals @xcite , which in turn makes results incomparable. even the term 'reasoning' is not always wellscoped (abductive? analogical? formal?; huang and chang 2023). the relationship of these claims to rigorous experimental protocols are central to our work. aside, recall from our earlier arguments that some of these claims fall apart under closer scrutiny, namely under more robust statistical tests. ethics and inclusion: llms are known to cause and propagate multiple harms, in addition to have very strong multilingual capabilities. it has then become an active focus area to ensure that llms are used in an ethical and inclusive manner for multiple audiences. note, however, that ethical concerns and evaluations are system-dependent: disclosures, protocols, and the concerns themselves are not universally tied to english-based language modelling, or even nlp. for example, alignment of models is usually carried out within a universal value system (e.g., 'always be honest'), but this has been called out for being pragmatically inadequate @xcite . likewise, evaluations under a single prompt tend to fail when working with culturespecific tasks @xcite . add to that the fact that the risks called out by researchers may not be the same risks considered by laypeople @xcite , or even other fields: only 2% of non-cs papers using llms call out ethical considerations @xcite , perhaps due to less concern or familiarity with this area. note that the focus of this work is llm-centred research, so papers from fields other than nlp are also part of our analysis. hence, authors should have a responsibility to educate others on the impact of their technology, and be aware of what matters for broader audiences. therefore, ethical and inclusive considerations should remain constantly present as part of any llm-related work. recommendations: studies and meta-reviews of ai often recommend good practices (e.g., reporting carbon footprints @xcite ; reproducibility checklists and experimental protocols @xcite @xcite ; self-contained artifacts @xcite ; and metadata for corpora @xcite . these suggestions are often not heeded @xcite noted that, out of 66 articles from leading conferences, only between 15% and 35% of the recommendations were partially followed. the practices themselves might not even be sufficiently impactful @xcite , given that other experimental protocols (e.g., sampling, initialisation, hyperparameters) may also impact reproducibility @xcite . in this work we are unable to directly address this lack of impact, although we do note in section 7 where it is possible to have better momentum without (completely) enforcing checklists in conferences. ## a review of llm-centred literature we provide four analyses: corpus composition (section 4.1), composition over time (section 4.2), the relationship between citations and criteria (section 4.3), and yearly trends on relationships between citations and criteria (section 4.4). the full breakdown of results is in appendix c. throughout this section, we use relevant papers to refer to these that did not score 'na' in the criterion discussed. ## conclusion and takeaways our study is a critique of the methodology employed in llm research. while statistically motivated, the centrepoints of our argument are related to taking stock of where we are and where are we going as a research community. we noted a yearly decline on the proportion of papers having ethics disclaimers, open-sourcing, and statistical tests. articles claiming sota relied more on llms as evaluators, and had fewer statistical tests, that the ones that did not. they also reported less research protocols, especially in 2024. we consider these evidence of mounting overreliance, lack of rigour, and/or rushed research, especially with respect to measurements. that said, there were two major positive findings in our work. the first was a steady proportion of limitation sections, which we ascribed to requirements from conferences. the second was an increasing number of evaluations in non-english languages, and a rising interest on safety and security as a research subject. from both we conclude that a combination of available, powerful, llms; along with publication checklists does have a net positive effect on the literature, and on a more diverse and inclusive research community. we also found that llm-centred research is rapidly increasing in volume, and consequentially it could be that our assumption that most papers will cite gpt-4 or gpt-3 does not necessarily hold in the future. to determine this, a year after the data cutoff for our paper we attempted to perform a follow-up study. this could have determined any rates of change and recency biases induced by our measurements, along with the validation, or discrediting, of our core assumption. however, we were able to only validate the latter (appendix d), because the tools we used in our work were no longer able to access google scholar. we argue that this is a concerning trend: the ability of surveys like ours to critically examine the field strongly rely heavily on open tools and publicly-available apis. losing that access could have ramifications on the ability of the field to perform self-examination. nonetheless, our work underscores the need for more self-scrutiny and rigour by and from the field. this is not easy given the overload of authors and peer-reviewers. it is also not feasible to wait months for the (purported) 'next big thing' to be peer reviewed if it is readily available as a preprint. while it could be said that critical reading is crucial, it would be naïve, however, to assume that this will be consistently done by all readers-e.g., laypeople, scientists not tasked with their peer review, etc. our recommendations are designed to address this. to close, cremonesi and jannach (2021) men-tioned tongue-in-cheek that in the context of recommender systems there was no reproducibility crisis, as in a crisis researchers reflect upon and revise their methodologies. instead, they stagnanted due to overfocusing on the same subjects without any introspection. it is our hope that our findings encourage llm researchers to reflect on how to push the field forward, while also carrying out research that is ethical, systematic, and open to criticism.