The Unreasonable Effectiveness of Easy Training Data for Hard Tasks (2024)

Abstract: How can we train models to perform well on hard test data when hard training data is by definition difficult to label correctly? This question has been termed the scalable oversight problem and has drawn increasing attention as language models have continually improved. In this paper, we present the surprising conclusion that current pretrained language models often generalize relatively well from easy to hard data, even performing as well as oracle models finetuned on hard data. We demonstrate this kind of easy-to-hard generalization using simple finetuning methods like in-context learning, linear classifier heads, and QLoRA for seven different measures of datapoint hardness, including six empirically diverse human hardness measures (like grade level) and one model-based measure (loss-based). Furthermore, we show that even if one cares most about model performance on hard data, it can be better to collect easy data rather than hard data for finetuning, since hard data is generally noisier and costlier to collect. Our experiments use open models up to 70b in size and four publicly available question-answering datasets with questions ranging in difficulty from 3rd grade science questions to college level STEM questions and general-knowledge trivia. We conclude that easy-to-hard generalization in LMs is surprisingly strong for the tasks studied.

## introduction it is difficult to supervise lms (i.e., train lms to give correct outputs) in specialized domains of human knowledge, because it is difficult to correctly label data in such domains. labeling difficulty manifests itself in both time to annotate (and thus cost) and label noise @xcite . labeling difficulty becomes severe when specific expertise is required @xcite . for example, for sufficiently specific 1 our code is publicly available at: https://github.com/ allenai/easy-to-hard-generalization . physics problems, phd holders and phd students can make errors on as many as 40% of (objective) problems @xcite . as more nlp benchmarks focus on challenging domain-specific tasks, having access to large human-labeled training corpora may become increasingly infeasible (e.g., existing benchmarks like mmlu @xcite and gpqa @xcite do not come with training data). the question arises: how can we train models to solve hard problems when correctly labeling enough hard data for training is difficult? this problem is an example of the scalable oversight problem, which concerns how to give a good reward signal to a model when it is difficult to assess if its outputs are correct @xcite . in this paper, we study the problem of easy-tohard generalization. easy-to-hard generalization refers to model performance on hard test data when finetuned foot_0 only on easy training data, defined according to some human hardness measure (like figure 2 : supervising llama-2-70b with easy data (left: high school level computer science problems; right: math problems with 2-3 reasoning steps) can enable generalization to hard data (left: a college level computer science problem; right: a math problem involving 8 reasoning steps). prompts themselves are omitted for brevity. easy -unsupervised hard -unsupervised where easy, hard, and unsupervised refer to model performance on hard test data when finetuned on easy data, hard data, or no data (zero-shot), respectively. this metric takes a value of 100% when finetuning on easy data is as effective as hard data, and it is 0% when a model finetuned on easy data is no better than prompting a model zero-shot. our main result is that pretrained language models generalize surprisingly well from easy to hard data, often performing almost as well as an "oracle" model finetuned on hard data (illustrated in fig. 1 ). in experiments with arc @xcite , mmlu @xcite , gsm8k @xcite , and strategyqa @xcite , we find that the supervision gap recovered is usually between 70% and 100%, meaning that easy supervision is at least 70% as good as hard supervision for hard test performance (see fig. 2 for example problems). these results are robust across (1) model family and scale (between 7b and 70b parameters), (2) six different human hardness measures and a model-based measure, (3) four datasets/tasks, and (4) several finetuning methods including in-context learning with and without chain-of-thought reasoning @xcite , qlora @xcite , and linear classifier heads @xcite . overall, our results suggest that current lms generalize relatively well to test data across human difficulty levels even when finetuned on data that is measurably easier than the test data. we hypothesize that this occurs because easy data elicits latent knowledge and skills from pretrained models in a hardness-invariant way. we additionally demonstrate that easy supervision can outperform hard supervision when (1) within some data collection budget, a greater quantity of easy data can be collected than hard data, or (2) easy data can be labeled with lower error rates than hard data (sec. 5.3). lastly, we study how easy-to-hard generalization changes with model scale and the gap between train and test hardness (sec. 5.4). the remainder of the paper is organized along the following research questions: rq1: how can we measure data hardness? rq2: can we do well on hard data by training on easy data? rq3: what are the cost-benefit tradeoffs of collecting easy vs. hard training data? rq4: is easy-to-hard generalization consistent across model scale and train-test hardness gap size? we summarize our main conclusions below: compositional generalization. work in compositional generalization has previously shown that neural networks struggle to generalize to problems that require combining reasoning steps in ways not seen exactly during training @xcite @xcite . further work has begun characterizing the conditions under which models generalize to compositionally more difficult problems. for instance, transformers will generalize better on classes of algorithmic problems whose solutions can be written in rasp, meaning the programs can be implemented exactly by a transformer forward pass @xcite . recurrent test-time computation also appears to be quite valuable for generalizing to problems requiring more reasoning steps than those seen during training @xcite . interestingly, even gpt-3.5 with chain-of-thought prompting can struggle to generalize to simple mathematical problems requiring more reasoning steps than seen during finetuning @xcite . our results are not inconsistent with these studies, but instead demonstrate that relative to an unsupervised-to-hard lower bound and hard-to-hard upper bound, easy-to-hard performance on compositional reasoning problems is often surprisingly good (sec. 5.2). easy-to-hard generalization. @xcite motivate the scalable oversight problem by pointing out how it could be challenging to give a proper reward signal to a model when it is difficult to assess if its outputs are correct. assessing easy-to-hard generalization provides useful context for understanding the difficulty of the scalable oversight problem, as it tells us how we can expect models to generalize from a setting where we can properly supervise them to one where we cannot. past work evaluates easy-to-hard generalization in nlp using model-based hardness measures @xcite and number of compositional reasoning steps required to solve a problem @xcite . @xcite show that bert models perform worse on commonsense reasoning tasks when finetuned on easy data rather than hard data according to a loss-based metric resembling minimum description length @xcite . @xcite show a similar result with gpt3 models for strategyqa and gsm8k, finding that prompting with "complex" examples does better than "simple" examples, where examples are categorized according to the number of reasoning steps in the annotated human reasoning chain for a problem. relative to these works, we study easyto-hard generalization with (1) a greater number of human hardness measures, including grade level, expert rating, required cognitive skills, question length, answer length, and number of reasoning steps, as well as a model-based metric, (2) multiple datasets involving science question answering, compositional reasoning, and mathematical reasoning, and (3) multiple model sizes for understanding scaling trends. in contrast to these works, we show that in a number of settings easy-to-hard generalization is comparable to hard-to-hard generalization. in concurrent work, @xcite present results on a related "weak-to-strong" generalization problem, where a stronger model is finetuned using labels from a weaker model. they also explore easy-to-hard generalization for nlp tasks using a model-based hardness measure. in contrast to this work, (1) we define our main performance metric (supervised gap recovered) using an unsupervised model as the baseline performance rather than a weaker model as the baseline performance, which is important when an unsupervised stronger model will greatly outperform a supervised weaker model (as is observed in our experiments); (2) we use human hardness measures in addition to model-based hardness, which is a more realistic and important setting when the two may not correlate strongly (see our fig. 8); and (3) we use publicly available datasets and open-source models rather than unidentified "nlp tasks" and api-gated models. ## measuring datapoint hardness measuring easy-to-hard generalization requires drawing a distinction between easy and hard data, defined in terms of human ability to correctly label the data. there could be many ways to describe what makes problems harder, including that (1) only people with specialized training and knowledge can solve the problem @xcite ; (2) it takes people longer to solve the problem; (3) people are less certain that their final solution is correct; (4) people with similar expertise naturally disagree about the solution to the problem, while agreeing that there is an objective solution @xcite @xcite ; (5) experts know of a reliable method for obtaining the answer to a problem, but it is costly in terms of time and effort or possibly noisy in its outputs (like conducting scientific experiments). in this paper, we aim to capture the above properties in a number of specific measures we can obtain for each instance in our datasets, including: character count provides a measure that is applicable across tasks. the first six hardness measures are fundamentally human notions of hardness, but we can also measure a model-based metric for datapoint hardness. in this direction, the seventh measure is a minimum-description-length (mdl) metric @xcite . in practice, mdl can be measured by computing a test datapoint's average label probability across models of identical architecture finetuned on increasing quantities of training data for a task @xcite . intuitively, mdl captures how hard on average an in-distribution test datapoint is for a model to generalize to given some amount of training data. ultimately, we use our mdl metric to capture how well a stronger model generalizes to data that is hard according to a weaker model, in order to simulate a setting where humans cannot label hard problems that they would like for a strong model to solve. in our experiments, we use four datasets that contain instance-level annotations for some portion of these measures, as shown in grade level is high school (hs) vs. college. see figure 2 (left) for an example. random performance is 25%. • strategyqa @xcite : yes/no general knowledge trivia questions requiring compositional reasoning over individual facts. the "num. reasoning steps" measure is the number of facts that must be combined. majority-class vote performance is 53.9%. there are generally fewer hard datapoints than easy datapoints in our datasets, given the relative difficulty of collecting hard data. in mmlu-stem-5, for example, there are 603 college level questions and 1143 high school questions. we show histograms for each hardness measure distribution in appendix fig. 20 . see appendix c for further dataset information. ## discussion are our tasks hard enough to provide generalizable easy-to-hard results? benchmark datasets for lms now require specialized domain expertise @xcite . the largest difficulty gap that we test in this paper is between 3rd grade and college level stem questions (using arc and mmlu). concurrent work has called for work studying gaps as large as 3rd grade to 12th grade (described as "huge leaps in generalization"; @xcite . therefore, we see our results as relevant for future work that may operationalize "easy" and "hard" differently. ## conclusion we study the problem of easy-to-hard generalization, showing that (1) several meaningful human and model-based hardness measures disagree about which data is hardest; our findings suggest that the scalable oversight problem may be easier than previously thought.