First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models (2024)

Abstract: Many NLP researchers are experiencing an existential crisis triggered by the astonishing success of ChatGPT and other systems based on large language models (LLMs). After such a disruptive change to our understanding of the field, what is left to do? Taking a historical lens, we look for guidance from the first era of LLMs, which began in 2005 with large n -gram models for machine translation (MT). We identify durable lessons from the first era, and more importantly, we identify evergreen problems where NLP researchers can continue to make meaningful contributions in areas where LLMs are ascendant. We argue that disparities in scale are transient and researchers can work to reduce them; that data, rather than hardware, is still a bottleneck for many applications; that meaningful realistic evaluation is still an open problem; and that there is still room for speculative approaches.

## introduction picture this scene: a renowned nlp researcher at a hot seven-year-old startup steps onstage to deliver a keynote. the speaker describes an ambitious new system to the packed room, building up to the results slide: a bar chart in which the x-axis shows the number of training words, and the y-axis shows system accuracy. as each data point is revealed, performance rises relentlessly, culminating in a system trained on well over a trillion words using over a thousand processor cores. it smashes the state of the art by a margin previously thought impossible. attendees are visibly shaken as they realize, over the course of a minute, that years of research have just been rendered utterly inconsequential. established academics panic, anticipating the wholesale rejection of already-submitted grant applications. phd students despair, contemplating the irrelevance of their unfinished dissertations. many ponder an exit to industry or a change of fields. they will speak of little else this week. does this scene sound like one that might have happened in the past year? in fact, it happened 19 years ago, in 2005, launching the first era of large language models (llms): the statistical machine translation (smt) era. the speaker, franz och, had co-invented key methods in smt @xcite @xcite , but had not published new work since joining google in 2004, instead revealing it in an invited talk prior to the launch of a new google translate @xcite . 1 the provocative results slide from that talk (figure 1 ) shows how google improved its smt system simply by expanding the training corpus of a phrase-based language model @xcite . 2 1 the description of the talk and its aftermath is based on the vivid recollections of one of the authors, who was present. we believe that this history offers lessons for the current era of llms, an era during which massive proprietary models have become a de facto baseline for many tasks @xcite . the expense of state-of-the-art research has led many to question the role of smaller and publicly funded groups in ai @xcite , a phenomenon we will call the scale crisis. 3 researchers without direct access to llms have publicly fretted over their research directions, with @xcite suggesting pivots in research direction to sidestep scale, and @xcite sketching research areas that are "not within the purview of llms." but what should researchers do if they care about problems that are within the purview of llms? to answer this question, we look to the first era of llms. what were the durable lessons of that time and evergreen research problems that still matter today? we arrive at several recurrent lessons: scale is supreme (section 2). we argue that, for areas where data is plentiful, nlp researchers cannot escape the bitter lesson @xcite that general purpose methods exploiting scale will outperform methods that leverage informed priors. we recommend that researchers take advantage of improvements in hardware as they enable scale at affordable budgets (section 2.1) and that they remember small-scale problems (section 2.2). evaluation is a bottleneck (section 3). the bitter lesson favors generic methods, which require evaluation metrics to optimize over. but improved models create an evaluation bottleneck, since error detection becomes harder when most remaining mistakes are subtle or associated with 1948) rather than neural networks. early lms were widely used across applications, beginning in speech recognition @xcite , though, unlike contemporary llms, they were rarely end products themselves. although the lm of @xcite had a very different architecture from contemporary llms, it was an llm in an important sense: it was trained on 2 trillion tokens, which is comparable to the training data size of modern llms. 3 we use the term crisis deliberately since others have done so. for example, following the announcement of gpt-4 (ope-nai, 2023), @andriy_mulyar (2023) posted on twitter that his feed was "full of ph.d. students having an existential crisis." edge cases. at scale, automated metrics show their flaws. we recommend that researchers work on improving metrics (section 3.1). there is no gold standard (section 4). when one can afford the annotation costs, it may be tempting to consider human feedback as the ideal solution to the evaluation problem. unfortunately, history has repeatedly shown that naïve methods of soliciting human preferences result in poor feedback, prioritizing superficial properties of model outputs. this lesson may serve as a counterpoint to the impulse to collect massive quantities of low-quality data in response to the bitter lesson. instead, we recommend grounding performance measurement in concrete downstream tasks (section 4.1). progress is not continuous (section 5). the ascent of neural mt abruptly ended smt's decade of seemingly unbeatable growth. this change in directions was enabled by new hardware-based paradigms, so we recommend that researchers continue exploring new methods that might scale well on future hardware (section 5.1). we conclude with a simple message: do research (section 6). we remind the reader that engineering achievements do not render scientific achievements insignificant, and we encourage the nlp community to renew their commitment to foundational scientific research even in areas where scale is currently a dominant factor. the first lesson offered by the history of smt is that data and compute scale are the dominant factors in system performance. in all eras of mt, improvements in bleu are logarithmic in training data size @xcite . this is immediately obvious from figure 1 : in order to achieve each linear step of improvement in accuracy (y axis), training data size must double (x axis). figure 2 (reproduced from @xcite shows a strikingly similar log-linear relationship between training data size and system performance for llms. indeed, such relationships are observed across many application areas of machine learning, including vision @xcite and speech @xcite . in a research landscape centered on performance metrics, scale will dominate. @xcite named the resulting malaise the "bitter lesson": "general methods that leverage computation are ultimately the most effective, and by a large margin." both smt and llms exemplify this lesson, and in fact sutton explicitly references natural language processing. many nlp researchers accordingly feel lost without access to large-scale systems. however, there are limits to scale, and as we will show by reviewing the history of the smt era, its disparities are often transient. the scale crisis is not a permanent state. ## evaluation is a bottleneck. the next lesson offered by reflection on the smt scale era is that the quality of evaluation methods makes a substantial difference in the effectiveness of training because a good evaluation can be used as a training signal. in smt, this epiphany was delivered by minimum error rate training, which trained directly on target metrics like bleu @xcite . likewise, train-time feedback metrics are often adapted for test-time evaluation: language modeling work may present validation loss, or equivalently perplexity, as the direct measurement of language modeling performance. it is therefore easy to use symmetric evaluations, applying the same metric for training feedback and test-time performance assessment. like model evaluation, training can be based on comparison with a ground truth, as in conventional training; quality estimation based on output alone, as used often in reinforcement learning (rl) settings @xcite @xcite ; or direct feedback, as provided by rl from human feedback (rlhf) and related methods of human assessment @xcite @xcite . the evaluation metric then becomes a crucial lever to improve model quality. with large scientific projects in both industry @xcite and nonprofit sectors @xcite spending millions on training llms, it may be surprising to point away from scale to other bottlenecks in system performance. the focus on scale bottlenecks is reasonable, as for any given compute budget, there is an optimal quantity of training data that yields the most accurate model @xcite . either compute or data size, therefore, can become a limiting factor, and there are still financial barriers to compute @xcite and limits on how much unstructured natural language data is available @xcite . however, training requires data and compute resources to be connected by evaluation feedback; this connection determines the effectiveness of training. unfortunately for efficient automatic approaches to evaluation, automatic metrics often fail to predict human evaluation, as shown for language modeling loss @xcite and bleu @xcite . even a metric that mirrors the training objective slightly can artificially inflate model performance; for example, it was observed early on that a phrase-based evaluation metric like bleu might favor phrase-based smt @xcite . identifying good metrics is challenging and becomes more difficult with each improvement, as the remaining errors become increasingly subtle or complex. this challenge was also recognized early in smt, with calls for a "bleu++" @xcite . evaluation, therefore, has become a crucial goal for modern llm research. while training often relies on cross-entropy loss or other simple comparisons between each token in a sequence, benchmarking a trained llm typically uses different evaluation criteria, such as checking the final answer in a word problem or the accuracy of a prompt-based classifier @xcite . however, these tests are plagued by data contamination: benchmark exposure during training has created illusory gains in tasks ranging from code generation @xcite to theory-of-mind puzzles (ullman, 2023). clearly language models are improving, but we cannot say precisely how, or by how much. ## conclusion: do research. as pure engineering efforts and institutional wealth outstrip novel scientific work, some in the ai community are pessimistic about the prospects of foundational research. our position, articulated over the course of this paper, is that there is much exciting, timely work yet to be done. these lessons are not particular to llms, but apply to any field subject to the bitter lesson. for example, issues in human evaluation plague many disciplines in machine learning. in computer vision, annotator idiosyncrasies account for many of the remaining inaccuracies of modern imagenet models @xcite . if a constrained labeling task such as image classification is subject to varied human judgment @xcite , how much harder is it to annotate free text generation? beyond our specific recommendations for researchers interested in improving the capabilities of language models, we would also point to scientific opportunities across related fields. from interpretability to empirical training analysis to public policy, many research areas only become more relevant and complex as models rapidly improve. furthermore, while novel modeling work suffers in a scale crisis, we can focus on new architectures and algorithms that take advantage of existing hardware and even anticipate future tools. some speculative alternatives to phrase-based smt, such as explicitly modeling syntax @xcite @xcite and semantics @xcite , were obviated by the bitter lesson and its expression in nmt. however, other proposals formed the basis of the nmt era. many enduring careers in nlp research were forged in areas that are now forgotten, and without risky exploration of unproven directions, the field could not have achieved many breakthroughs. our anxieties should not discourage us from seizing the opportunities presented by a new era of llms.