Inverse Scaling Can Become U -Shaped (2023)

Abstract: Scaling up language models has been empirically shown to improve performance on a wide range of downstream tasks. However, if we were to observe worse performance as a function of scale (inverse scaling) on certain tasks, this would indicate that scaling can also encourage behaviors that are misaligned with human preferences. The Inverse Scaling Prize (McKenzie et al. 2023) identified eleven such inverse scaling tasks, evaluated on models of up to 280B parameters and up to 500 zettaFLOPs of training compute. This paper takes a closer look at these inverse scaling tasks. In this paper, we evaluate models of up to 540B parameters, trained on five times more compute than those evaluated in the Inverse Scaling Prize. With this increased range of model sizes and compute, only four out of the eleven tasks remain inverse scaling. Six tasks exhibit U-shaped scaling, where performance decreases up to a certain size, and then increases again up to the largest model evaluated (the one remaining task displays positive scaling). In addition, 1-shot examples and chain-of-thought can help mitigate undesirable scaling patterns even further. U-shaped scaling suggests that the inverse scaling trend observed in McKenzie et al. (2023) may not continue to hold for larger models, which we attribute to the presence of distractor tasks that only sufficiently large models can avoid.

## introduction scaling up language models (lms) has been shown to improve model performance for a wide range of downstream tasks and and has been claimed to unlock emergent abilities @xcite @xcite . however, are there any tasks for which performance gets worse as models scale? tasks that exhibit this property have been referred to as inverse scaling tasks @xcite , and such tasks can help reveal flaws in the models' training data or objectives @xcite . the inverse scaling prize was created to identify such tasks for which larger lms show increasingly undesirable behavior @xcite . submissions were scored based on a range of criteria including inverse scaling strength, task importance, novelty/surprisingness, task coverage, reproducibility, and inverse scaling generality across different models. eleven tasks were awarded third prizes, the datasets for which have been publicly released. the scaling curves for these eleven tasks (see @xcite 2 and also appendix c, figure 6 ) were shown on a range of lms with scales spanning several orders of magnitude in parameters, including gopher (42m-280b; @xcite , chinchilla (400m-70b; @xcite , and an anthropic internal model (13m-52b). in this paper, we take a closer look at the scaling behaviors for these eleven tasks. first, we evaluate palm models of up to 540b parameters @xcite , trained on about five times more compute than the models evaluated in the inverse scaling prize submissions (see ## mitigation for inverse scaling we next explore possible mitigation strategies for inverse scaling. in section 2, we hypothesized the primary cause of inverse scaling to be distractor tasks that mislead the models towards a different solution from the true task. then, in-context demonstrations of a problem/solution pair could discourage the models from solving the distractor task, since the answer according to the true task diverges from the answer according to the distractor task. if such demonstrations are accompanied by explicit rationales, this could guide the models towards identifying the true task even more strongly. to this end, we explore whether 1-shot demonstrations and 1-shot demonstrations with chain-of-thought reasoning improve undesirable scaling patterns. ## conclusions this paper has two simple takeaways. first, inverse scaling can turn into u-shaped scaling when evaluated on models of sufficiently large scale, as demonstrated on six out of eleven inverse scaling prize tasks. the prevalence of u-shaped scaling we identified in this paper shows that inverse scaling curves do not necessarily extrapolate to larger models. second, demonstrations and rationales are effective for mitigating undesirable scaling patterns. all inverse scaling tasks change to u-shaped or flat scaling when a single demonstration is provided as a part of the prompt. with additional intermediate reasoning steps, many of the u-shaped tasks further shift to positive scaling, as well as substantial performance gains throughout. taken together, a combination of scaling and prompting techniques appears to be a viable method for mitigating inverse scaling. however, the prompting approaches we explored has limitations in that they require manual construction of demonstrations and reasoning steps tailored to individual tasks. the 0-shot cot approach proposed by @xcite is one method that does not require manual prompt construction, but as we show in the additional experiment in appendix f, the effectiveness of this approach is limited for the inverse scaling tasks. this leaves open an interesting area of future research of developing novel solutions for inverse scaling that do not require explicit demonstrations.