Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement (2025)

Abstract: Large Language Models (LLMs) continue to set new standards in knowledge-intensive and complex reasoning tasks, yet their high computational demands limit widespread adoption. While distilling large models into smaller ones offers a sustainable solution, current techniques—such as static knowledge distillation, resource-intensive reinforcement learning from human feedback, or limited self-reflection—struggle to yield substantial and lasting performance gains. In this paper, we present a novel Debate and Reflect (D&R) framework that orchestrates multi-turn debates between smaller models and stronger teacher models, eliciting actionable feedback (e.g., error analysis, corrective strategies) to guide student models. Further, we introduce Tree-structured Direct Preference Optimization (T-DPO) to efficiently leverage these debate logs, organizing interactions into a hierarchical format for effective training. Empirical evaluations across diverse NLP benchmarks demonstrate that our approach significantly improves smaller-model accuracy, robustness, and generalization, outperforming conventional baselines by a large margin.

## introduction llms have emerged as powerful tools for tackling complex reasoning tasks @xcite @xcite . their ability to process and generate coherent, context-aware outputs has set a new standard in artificial intelligence (ai). however, the computational demands of training and deploying these models make them inaccessible to many users, highlighting the need for more efficient alternatives. developing smaller models capable of achieving similar knowledge and reasoning performance is an essential step toward making advanced ai technologies more accessible and sustainable. figure 1 : overview of d&r. the student model debates with teacher models, receiving self-reflection and teacher feedback. a multi-agent interaction graph (mag) records debate content, from which preference trees are extracted. all information is then distilled into the student, enhancing its knowledge and reasoning. existing strategies for enhancing smaller models generally fall into three categories. first, knowledge distillation transfers knowledge from a larger teacher to a smaller student model @xcite . while effective for reducing model size, traditional distillation procedures are typically one-off processes that lack targeted guidance from the teacher. that is, the teacher's rich insights into how the student can improve are not fully leveraged, and the student often receives only generic training signals. second, reinforcement learning from human feedback (rlhf) aligns model behavior with human preferences and can substantially boost reasoning @xcite . however, it requires extensive human intervention, raising concerns about scalability and cost. third, multiagent systems and self-reflection frameworks enable models to exchange critiques or engage in introspective refinement during inference @xcite @xcite . although these methods show promise in improving model reasoning, they often apply feedback only transiently-without permanently updating model weights-or use simplistic finetuning schemes that do not fully exploit iterative feedback. in this paper, we introduce a novel debate and reflect (d&r) framework that combines the strengths of multi-agent debate and actionable teacher feedback to robustly enhance smaller models. concretely, a student model engages in multiturn debates with one or more stronger teacher models, collecting detailed feedback in the form of error analyses, corrective strategies, and best practices. crucially, the feedback is not limited to a single inference; rather, we incorporate both self-reflection and teacher feedback mechanisms that help the student to internalize these insights for lasting, parameter-level improvements @xcite @xcite . to maximize the efficiency of this training signal, we further propose tree-structured direct preference optimization (t-dpo) to transform debate logs into hierarchical preference trees @xcite . by systematically organizing intermediate debate turns and teacher guidance, t-dpo enables the student to learn not just what the best response is, but why certain answers are preferred. through extensive empirical evaluations, we show that d&r with t-dpo significantly outperforms conventional methods in accuracy, robustness, and generalization, offering a promising route for achieving high performance in smaller, more accessible language models. in summary, our contributions are threefold: ## debate & reflect method to improve small models using debate & reflect (d&r), we construct debates where the small student model engages in discussions with their teacher models, receiving feedback from them (section 3.1). from debates, we construct preference trees in multi-agent interaction graphs (mags) finally, we distill these structured insights into the student model with t-dpo, enabling more effective learning (section 3.3). ## conclusion in this paper, we proposed d&r, a novel distillation framework to enhance small models through debates. the debate process, involving teacher and student models, fully exposed the student model's weaknesses, providing self-reflection and feedback. then, we used t-dpo with preference trees from debate mags, capturing structured information and preference relationships between different reasoning paths. d&r effectively distilled the teacher models' knowledge and reasoning capabilities into the student model through sft and t-dpo. our experiments demonstrate that d&r achieves superior distillation compared to baseline methods, bridging the gap between smaller student models and larger teacher models. compared to online debate everytime, d&r significantly improved inference efficiency. beyond reasoning, it also strengthened the student model's ability to self-correct. in future, we will explore long-term learning where the student model continues to improve over time through continuous debates and iterative feedback. large language models funding initiative (aisg award no: aisg-nmlp-2024-002). any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of national research foundation, singapore. it was also supported by the national natural science foundation of china (no. u21b2009) and the lee kong chian fellowship. this work was done during a visit to singapore management university funded by the china scholarship council.