Emergent Inabilities? Inverse Scaling Over the Course of Pretraining (2023)

Abstract: Does inverse scaling only occur as a function of model size, or can it also occur over the course of training? We carry out an exploratory study investigating whether the performance of language models on specific tasks can decrease (while general performance remains high) during training on the language modeling task. We find 8 tasks on which Pythia 12B (Biderman et al., 2023) shows decreased performance over the course of training. Five of these tasks (TruthfulQA-MC1, TruthfulQA-MC2, Hindsight Neglect, Memo Trap, and Pattern Match Suppression) additionally show a consistent relationship whereby larger language models show a greater decrease in performance the more they are trained, despite showing standard (positive) scaling overall. This highlights the importance of testing performance at all relevant benchmarks any time models are trained on additional data, even if their overall performance improves.

## introduction for language models, bigger is usually better. recent research has found that both increased number of model parameters and increased size of the training dataset positively influence model performance @xcite @xcite @xcite @xcite @xcite @xcite . one particularly striking pattern that has been reported is emergence, a nonlinearity in these relationships, where at a particular scale, language models improve rapidly at a given task @xcite . however, while increased scale usually leads to improved performance, on certain tasks it correlates with decreased performance. this is known as inverse scaling @xcite ). an example of a task on which inverse scaling is observed is the truthfulqa benchmark, where larger language models are more likely to predict popular misconceptions over statements of fact @xcite . more recently, additional tasks that reportedly show such an effect have been identified as part of the inverse scaling prize @xcite , as well as by other researchers @xcite . inverse scaling is a serious concern for several reasons. at a high level, inverse scaling may indicate 'outer misalignment' @xcite between the model training approach and the purposes to which they are applied. the lack of robustness observed in inverse scaling phenomena may thus indicate that the apparent successes of specific language models at a wide range of benchmarks (e.g., @xcite do not necessarily entail that they have the capability ostensibly being tested @xcite . the existence of inverse scaling is also concerning because of the possibility of other as yet unidentified tasks where performance similarly scales inversely with model size. models that perform well on a variety of tasks may well present deteriorating performance in unseen tasks with scale, even as performance at established benchmarks increases. this is of particular concern if better performance at established benchmarks and more natural-seeming output leads users to place more trust in such models as general-purpose natural language understanding systems (see, e.g., @xcite , for general discussion of such risks). finally, inverse scaling is also of concern because it is often unpredictable. in the same way that certain capabilities appear to emerge at scale @xcite , inverse scaling also appears or accelerates at given scales. for example, as @xcite show, the performance of gopher @xcite and plain lm @xcite at the inverse scaling prize's negated question-answering task (neqa) appears to be stable or even increasing as model size increases, only dropping as model size increases to around 7 billion parameters and beyond @xcite . thus, inverse scaling may occur not just for unidentified tasks, but also for well-established tasks: a model architecture that performs well at a benchmark at a small scale may suddenly perform surprisingly worse as scale increases-it is not safe to assume that performance will continue to improve or even remain stable. while previous work has focused on inverse scaling based on the number of model parameters @xcite @xcite @xcite ; as discussed, scaling effects more generally occur not just in relation to model size but also as a function of training data quantity. recent work has shown that this latter effect has been substantially underestimated @xcite . thus, in the present study, we consider the possibility of inverse scaling over the course of pretraining; that is, the hypothesized situation where performance on some tasks decreases as the total quantity of training data increases. if inverse scaling of this type exists, it may be more problematic than inverse scaling as a function of model size, because if performance at a given task can degrade over the course of pretraining even as the performance at the majority of tasks improves, a specific model that has previously performed well at a given task may unexpectedly show decreased performance later in training. we test how well the 8 models of the pythia @xcite suite perform at 12 tasks over the course of training. these tasks are the 10 multiple-choice tasks that won the inverse scaling challenge @xcite , plus the two multiple-choice versions of the truthfulqa benchmark, on which language models have also been found to show inverse scaling @xcite . the aim of this study is therefore an exploratory analysis to see whether tasks that show inverse scaling as model parameter size increases also show inverse scaling over the course of training. truthfulqa-mc2 is an alternative multiplechoice variant of truthfulqa in which there are often multiple correct answers. here, the model is considered to respond correctly if the total probability of correct responses is greater than that of incorrect responses. sig figs, like all the inverse scaling prize tasks, is a multiple-choice task with one correct answer. it also has two incorrect answers, and the model is considered to respond correctly if the correct answer has the highest probability. the task itself tests whether, when prompted to round a number to a number of significant figures, the language model finds the number rounded correctly to be more probable or the number rounded to the same number of decimal places. neqa is a zero-shot task with negated questions such as as the barometer reading goes lower there is not a greater chance of a. sunshine b. getting wet. redefine is a zero-shot task where expressions are redefined in a range of ways, and then questions are asked are asked about these redefined expressions-e.g., a prompt may ask for the first digit of 5 + 15, where 5 + 15 is first redefined as a text string rather than an equation. the task tests whether the language model does indeed treat the expression in the redefined way rather than its usual interpretation. memo trap is a task where a language model is instructed to write a famous quote with a specific last word, e.g., write a quote that ends in the word "heavy": absence makes the heart grow. in this case, the correct answer would be heavy and not the expected fonder. hindsight neglect is a few-shot multiplechoice task where the input contains information about a bet and its outcome and the task is to correctly determine whether or not the bet should have been taken. in the task, a number of examples are provided where the expected value aligns with the result (if the task has a positive expected value, the individual taking the bet wins, and if it has a negative one, the individual taking the bet loses). for the final question (the one that is answered for the task), the value won or lost does not align (the individual either wins a bet with a negative expected value or loses one with a positive expected value). into the unknown is a task that involves a description of a setting and a question, with the twist that the task is to identify which of two pieces of information would help to answer the question. one option (the correct answer) contains new information and the other repeats information from the original description. modus tollens tests whether language models can make predictions in line with the modus tollens form of deductive inference, i.e., '[i]f p, then q; not q; therefore, not p' @xcite . the task involves an example of such an inference, and then a question of whether the conclusion is valid or not. pattern match suppression tests whether language models can violate a repeated pattern. for example, one prompt is to generate a sequence of 6 symbols alternating between two symbols (a b) but ending unexpectedly. a, b, a, b, a, with possible answers a or b. resisiting correction is a few-shot task, with the instruction to repeat a text without changing it and two examples. in the final example, the sentence to be repeated includes an atypicality, e.g., spelling mistake or a switched word of a famous quote. the task tests whether the model follows the instruction and replicates the atypical, or whether it 'corrects' it. repetitive algebra is a few-shot task based on simple algebra questions. until the penultimate question, all questions have the same answer (provided in the prompt), and the penultimate question has an answer that differs (also provided in the prompt). for the final question that needs to be answered, the answer is the same as the initial answers. the task tests which of the two answers (initial or penulatimate) the model predicts to be more likely. ## results model performance at each task is shown in figure 1. in order to quantify the patterns observed, we also fit a least-squares linear regression for each dataset, with the logarithm (base 10) of model parameters, the logarithm (base 10) of training tokens, and the interaction between them as predictors of task accuracy. all variables were z-scored. the results of these tests are shown in the clearest inverse scaling effects can be seen with truthfulqa-mc2-larger models perform worse, performance overall decreases with number of training tokens, and the rate at which performance deteriorates with training tokens increases with model size. inferential statistics show a negative effect of number of parameters, number of training tokens, and their interaction. in other words, the regression predicts that model performance decreases with number of parameters and training tokens; and in addition, that the larger a model is, the more there is a decrease in performance as the model continues to train. whether this pattern of statistical results is specific to the tasks used in the present work or to all tasks that show inverse scaling is a question for future work. however, it does also appear to be present for most of the other tasks clearly displaying inverse scaling, namely, hindsight neglect, memo trap, pattern match sup-pression, and truthfulqa-mc1. some of the remaining tasks, namely into the unknown, modus tollens, neqa, and sig figs display no clear pattern across models. but when focusing on just the two largest models, re-define appears to show inverse scaling over the course of training, and the largest (12 billion parameter) model shows inverse scaling during training on repetitive algebra and resisting correction. these may be a case of emergent inverse scaling (i.e., nonlinearities that cannot be accounted for using linear statistical models), espe-cially in the case of resisting correction, but models with a larger number of parameters would be needed to verify this. ## discussion we find clear evidence of inverse scaling over the course of training on truthfulqa-mc1, truthfulqa-mc2, hindsight neglect, memo trap, and pattern match suppres-sion, as well as possible evidence of the same phenomenon on redefine, repetitive algebra, resisting correction for the largest model or models. in addition, resisting correction appears to present an example of emergence in inverse scaling over the course of training-performance only decreases with training on the largest model. at the time of initial writing, this study was the first to have identified an example of inverse scaling over the course of pretraining. since then, an official inverse scaling prize paper has been released @xcite . in addition to exploring scaling in terms of the number of floating point operations (flops) needed to train each model, @xcite also analyze the performance of different sizes of the anthropic lm model (2m, 13m, 42m, 197m, 805m, 3b, 13b, 52b) over the course of training on 400b tokens, providing a valuable point of comparison. on the whole, their results are similar to ours over the same scales. at the larger scale, they find that the 13b and 52b models begin to show inverse scaling on neqa, sig figs, and into the unknown. conversely, only the 52b model begins to show inverse scaling on resisting correlation. mckenzie et al. (2023b) also classify the tasks into different types. 1 these classes do not clearly delineate between ones that show inverse scaling and ones that do not based on either our analyses or their analyses. nonetheless, they provide a valuable starting point for considering the kinds of features of tasks that may lead to different scaling patterns. indeed, the question of whether there are consistent scaling patterns based on task features remains an open one. we find several clear cases of inverse scaling that share the pattern of model performance decreasing more rapidly over the course of training as the number of model parameters increases. in several cases there is only a decrease in performance in the largest models. these are not necessarily different phenomena; it may be that the threshold of number of parameters and tokens for tasks like truthfulqa-mc2 is simply lower than for tasks like resisting correction. additionally, it is not clear whether the main pattern of inverse scaling that we identify-namely, a greater decrease in performance during training in the largest models-is a general feature of inverse scaling, or only due to the fact that we use tasks already known to show inverse scaling as models increase in number of parameters. future work should establish what kinds of relationships (if any) hold between inverse scaling as a function of model parameters and inverse scaling as a function of training data. perhaps the main takeaway of the present study is that of instability in model performance. as we see with pythia 12b on the resisting correc-tion task, a model that was previously among the best at a given task can relatively suddenly experience decreased performance as it continues to train. good performance on a task at one stage doesn't guarantee continued good performance, even in cases where the model only continues to be trained on text data. this highlights the importance of regular and rigorous evaluation. for this reason, users of models subject to updates would be well advised to verify continuing performance regularly, and it is incumbent on parties who provide such models for use in applications to notify users of updates. ## conclusions in this study, we set out to investigate whether inverse scaling can occur not only as a function of number of model parameters, but also number of training tokens. we find clear evidence that it does occur with the pythia @xcite suite of models on five of the twelve tasks analyzed, and additional evidence that it may occur on up to eight.