Guiding Through Complexity: What Makes Good Supervision for Hard Reasoning Tasks? (2025)

Abstract: How can “weak teacher models” (Bowman et al., 2022) such as average human annotators or existing AI systems, effectively supervise LLMs to improve performance on hard reasoning tasks, especially those that challenge and requires expertise or daily practice from the teacher models? In this paper, we seek for empirical answers to this question by investigating various data-driven strategies that offer supervision data at different quality levels upon tasks of varying complexity. Two intuitive strategies emerge for teacher models to provide supervision during alignment training: 1) using lower-quality supervision from complete tasks that match the difficulty of the target reasoning tasks, and 2) leveraging higher-quality supervision from easier subtasks that are less challenging. Interestingly, we find that even when the outcome error rate for hard task supervision is high (e.g., 90%), training on such data can outperform perfectly correct supervision on easier subtasks on multiple hard math benchmarks. We further identify a more critical factor influencing training performance: step-wise error rates, which indicate the severity of errors in solutions. Specifically, training on hard task supervision with the same outcome error rates but disparate step-wise error rates can lead to a 30% accuracy gap on MATH benchmark. Our results also reveal that supplementing hard task supervision with the corresponding subtask supervision can yield notable performance improvements than simply combining rephrased hard full task supervision, suggesting new avenues for data augmentation. Data and code will be released upon acceptance.

## introduction the rapid advancements in large language models (llms) @xcite @xcite indicate that ai systems will soon be capable of progressively solving hard reasoning tasks that challenge both humans and current llms. to enhance the performance of today's ai systems @xcite on such hard reasoning tasks, especially those that exceed the reasoning abilities of the average human or current llms, what strategies should we adopt to consistently and effectively supervise the ai systems? we are concerned what supervision data those "weak teacher models", humans and current llms, should annotate or synthesize for better hard downstream task performance. we mainly consider two possible data-driven supervision strategies -strategy 1: sampling or annotating solutions directly on the hard tasks that both human annotators and ai models typically struggle with; strategy 2: sampling or annotating solutions on hard tasks' corresponding subtasks, which are more manageable and where humans or ai models are more likely to succeed @xcite @xcite . for strategy 1, while the tasks used for supervision match the reasoning difficulty of our studied hard downstream tasks, the quality of supervision may be compromised due to the difficulty that challenges the human and ai teacher models. in contrast, strategy 2 involves easier tasks that do not match the difficulty level of the downstream tasks, but the supervision quality tends to be higher because these tasks are simpler for annotators or models to handle. should we obtain low-quality supervision for hard tasks, or employ high-quality supervision for decomposed subtasks for fine-tuning? to fairly compare the two supervision strategies, which involve task difficulty and supervision quality, we introduce a simulated setting: the supervision quality is controlled by introducing varying outcome error rates, which represent the percentage of noisy supervision where solutions with wrong final answers replace the ground-truth solutions. it is designed for simulating the varying supervision quality from "weak" teacher models with different capacity. after acquiring full task and subtask supervision with varying outcome error rates, and following @xcite , we apply supervised fine-tuning (sft) using the acquired supervision to llms for solving hard reasoning tasks. across 5 hard math reasoning tasks, including college entrance exams, high-school competitions and college-level math tests, we find hard task supervision consistently outperforms subtask supervision, regardless of outcome error rates. specifically, we observe that: 1) even if outcome error rate of hard supervision is 90%, it can still outperform perfectly correct subtask supervision; 2) for easy subtask and hard full task supervision, as the outcome error rate increases from 0% to 100%, the performance does not decrease monotonically. rather than the outcome error rate, what factors highly impact the hard test task performance? one key factor we identify is the step-wise error rate, i.e., average ratio of erroneous steps ( §5) to the total steps in the supervision set. specifically, we observe that when different teacher models exhibit similar outcome error rates, an increase in step-wise error rate leads to lower performance on hard reasoning tasks. we test a batch of llms, including gpt-4o-mini (openai, 2024a), gpt-3.5turbo (openai, 2022), llama-3-70b-instruct, and command-r (cohere, 2024), as teacher models for providing hard task supervision. we then compare the performance of the rest models' supervision against that of gpt-4o-mini at a similar outcome error rate. notably, with a lower step-wise error rate, gpt-4o-mini's sampling supervision always leads to higher results. when both gpt-4o-mini and command-r have around 90% outcome error rate in their generated supervision, command-r underperforms by 12% accuracy on math @xcite and is even worse than training with subtask supervision. can we further improve the hard downstream task performance without relying on more hard task annotations? we find that supplementing hard task supervision with subtask supervision, i.e., the supervision of hard tasks' corresponding subtasks, can lead to more performance gains. specifically, we combine the hard task supervision with an outcome error rate of a and the subtask supervision with an outcome error rate of b with three distinct combinations of a and b. we observe that adding subtask supervision improves performance across most of the 5 downstream tasks. when @xmath0, we achieve the best results on benchmarks like math and sat-math. we also find that this simple supervision combination method is more effective than simply doubling the training epochs on the mere hard task supervision or mixing rephrased hard task supervision. overall, we summarize our main contributions for enhancing llm's hard reasoning task performance as follows: ## formulation of supervision strategy analysis in this section, we present the framework for our analysis of supervision strategies aimed at enhancing performance on hard downstream reasoning tasks. we outline the motivation behind two supervision strategies involving both easy and hard tasks, introduce the key notations, and discuss the design principles of the analysis formulation for fair and comprehensive evaluation. ## hard full task and easy subtask supervision synthesis in this section, we detail the process of collecting easy subtask and hard full task supervision, d subtask and d hard , based on the design principles of our supervision strategy analysis framework. ## concluded empirical advice based on the findings above, we offer some practical advice for optimizing annotation efforts of weak teacher models to enhance on hard tasks: prioritize hard task supervision for better performance on challenging reasoning tasks. we reveal in §4 that the best performance is consistently achieved with hard full task supervision. it significantly surpasses the results obtained using only subtask supervision. step-wise error rates of teacher models is more critical than outcome error rates on hard reasoning task performance. in §5, we find that, even with identical outcome error rates, supervision with higher step-wise error rates leads to more evident performance drop than what we observe when the outcome error rates change. to further enhance performance, consider sampling or annotating hard tasks and their associated subtasks, then use the combined supervision to train llms. as shown in §6, given a fixed task scope covered in the hard task supervision, integrating hard task supervision and their corresponding subtask supervision boosts or even achieves the best performance across benchmarks. this combination is also more effective than simply increasing the training epochs or further merging the rephrased version of the original supervision. ## conclusion in this paper, we explore two possible supervision strategies that weak teacher models can employ to enhance performance on hard reasoning tasks. we first find that even when the outcome error rate for hard task supervision is high, training on this data can surpass the performance of perfectly correct supervision from easier subtasks across multiple hard math tasks. then, we pinpoint step-wise error rates as a more critical factor influencing hard task performance, as its variation can result in a more evident drop on the hard tasks. we further observe that if no more hard task supervision is allowed to be added, supplementing existing hard task supervision with corresponding subtask supervision yields greater performance than merely merging rephrased hard task supervision. these insights contribute valuable data-driven guidance for researchers and practitioners aiming to improve llms in future reasoning-heavy applications.