Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review (2025)

Abstract: While reasoning capabilities typically emerge in large language models (LLMs) with tens of billions of parameters, recent research focuses on improving smaller open-source models through knowledge distillation (KD) from commercial LLMs. However, many of these studies rely solely on responses from a single LLM as the gold rationale, unlike the natural human learning process, which involves understanding both the correct answers and the reasons behind mistakes. In this paper, we introduce a novel Fault-Aware DistIllation via Peer-Review (FAIR) approach: 1) instead of merely obtaining rationales from teachers, our method asks teachers to identify and explain the student’s mistakes, providing customized instruction learning data; 2) we design a simulated peer-review process between teacher LLMs, and selects only the generated rationales above the acceptance threshold, which reduces the chance of teachers guessing correctly with flawed rationale, improving instructional data quality. Comprehensive experiments and analysis on mathematical, commonsense, and logical reasoning tasks demonstrate the effectiveness of our method. Our code is available at https://github.com/zhuochunli/Learn-from-Committee.

## introduction large language models (llms) have proven to be highly effective in addressing a wide range of complex tasks @xcite , including mathematical reasoning @xcite , commonsense reasoning @xcite , and logical reasoning @xcite . however, these emerging reasoning abilities tend to manifest only in llms with more than 100 billion parameters, while smaller models struggle to exhibit such capabilities @xcite . despite this, related research @xcite has shown that smaller language models, particularly those with fewer than 10 billion parameters, can perform similarly to larger models in terms of following human instructions. however, it is challenging to prompt smaller language models (lms) to generate reasoning steps by chain-of-thought (cot) prompts @xcite . moreover, most existing reasoning datasets lack high-quality rationale @xcite due to the high cost of manual annotations. to address these challenges, distilling the capabilities of llms emerges as a resource-friendly and effective strategy. deepseek-r1 @xcite demonstrates that distilling reasoning patterns from larger models can outperform rl-derived patterns on smaller models. through collecting rationales generated by llms for instruction tuning, previous studies have been able to distill the private llms' reasoning abilities into smaller models @xcite @xcite . however, most of these efforts fall within the scope of labeling knowledge distillation @xcite , where llms are primarily used to annotate data for training smaller models, without utilizing smaller model's outputs as feedback to generate customized instruction data to improve the lm in return. as a result, llms remain unaware of the limitations of smaller models. furthermore, prior research typically employs only one llm as the teacher, which can introduce more biased training data compared to using multiple teacher llms during distillation. therefore, we propose using multiple llms from different organizations as teachers to provide more impartial and diverse training data. additionally, we design a simulated peer-review process between teacher llms, where the rationale generated by one llm is reviewed by other llms. only the rationales that pass this peer-review process are included in the training dataset. this method reduces the likelihood of flawed rationales, even when a correct answer is provided, thereby, improving the overall quality of the training data for instruction tuning. to this end, we propose a fault-aware distillation via peer-review (fair) knowledge distillation method from multiple llms, as briefly shown in figure 1 . inspired by the natural human learning process @xcite , we argue that students should not only know what is the correct answer but also learn why they made mistakes. therefore, in addition to providing the correct rationale generated by the teacher llms, we also present the student model's mistakes to the teacher llms and return the mistake-specific feedback. furthermore, inspired by the multi-agent evaluation framework of @xcite , we employ multiple llms as teachers. each teacher llm's answer is reviewed by the other teachers, and only the responses that pass this peer-review process are included in the instruction training dataset. we believe that this peer-review mechanism can significantly reduce biased or flawed rationales, leading to improved distillation performance. in summary, the contributions of our work are as follows: to address this, we collect both the correct rationale and mistake-specific feedback @xcite for student models' wrong answers from llms, integrating them into instruction tuning to enhance the overall reasoning capabilities of the student models. moreover, unlike previous studies that depend on a single teacher llm @xcite or intermediate roles such as mentors @xcite and teaching assistant (ta) @xcite , we employ multiple llms @xcite as teachers to increase the diversity of generated data. finally, compared to peer-review methods in llms for evaluation @xcite , we design a simulated peer-review process to ensure high-quality instruction training data, thereby improving the distillation performance. score: 0 score: 4 exam problem: weng earns @xmath012 per hour for babysitting. since she did 50 minutes of babysitting, she earned @xmath19 per 50 minutes. so, she earned @xmath218. peer-review process the mistakes in student's rationale: the error is in multiplying the hourly rate by 0.75 to get the amount earned in 50 minutes. instead, the correct approach would be to calculate the rate per minute by dividing the hourly rate by 60, and then multiplying that rate by the number of minutes worked. the correct answer is @xmath318. correct rationale: to calculate weng's earnings, first, calculate the rate per minute by dividing by 60: @xmath40.2/ minute. now multiply the time by weng's minutely rate: 50 minutes x @xmath510. so, weng earned @xmath610 explanation: … score: 5 answer: $20 explanation: … score: 1 … … peer-review process ## method as illustrated in figure 2 , we introduce a fault-aware distillation via peer-review (fair) knowledge distillation method that empowers the student model to improve by learning from its own mistakes and the correct answers generated by multiple teacher models. specifically, our instruction learning procedure involves four major steps: (1) the student lm takes an "exam" on the training set to identify mistakes that are incorrectly generated rationales; ## conclusion in this work, we introduce the fault-aware distillation via peer-review (fair) approach. we implement a simulated peer-review process between multiple teacher llms to gather reliable outputs, which refines the quality of instruction tuning dataset. additionally, we develop an integrated instruction tuning method that allows the student lm to learn from both the correct rationale and mistakes feedback. comprehensive results on diverse reasoning tasks validate our efficient method for unlocking the reasoning potential of smaller open-source lms through distillation, even with black-box llms and without dataset-provided rationales. we hope that our findings will encourage further investigations into reasoning distillation.