Training Language Models to Critique With Multi-agent Feedback (2025)

Abstract: Critique ability, a meta-cognitive capability of humans, presents significant challenges for LLMs to improve. While utilizing human annotation can enhance critique ability effectively, most recent works primarily rely on supervised fine-tuning (SFT) using critiques generated by a single LLM like GPT-4, which is more scalable and cost-effective.However, such model-generated critiques often suffer from inherent flaws due to the complexity of critique. Consequently, fine-tuning LLMs on these flawed critiques not only limits performance but also propagates errors into the learned model.To address this issue, we propose MultiCritique , a unified framework that leverages multi-agent feedback to improve critique ability in both the supervised fine-tuning (SFT) and reinforcement learning (RL) stages.In the SFT stage, MultiCritique aggregates high-quality multi-agent critiques through a fine-grained meta-critique mechanism. In the RL stage, preference critiques are constructed and refined by validating their contributions to revisions, thereby enhancing robustness of RL in improving critique ability.Based on MultiCritique, we construct SFT and RL datasets. Extensive experimental results on two benchmarks highlight the key benefits of our dataset, including superior quality, enhanced data efficiency, strong generalization on unseen tasks, and improvements in the general capability of LLMs.Notably, our fine-tuned 7B model significantly surpasses advanced 7B-13B models, approaching advanced 70B LLMs and GPT-4.Resources have been publicly available.

## introduction the critique ability, i.e., the capability to identify and refine flaws in responses, has been widely used to facilitate reliable automatic evaluation and selfimprovement of llms @xcite . as a meta-cognitive capability @xcite @xcite , critique ability requires llms to possess a deep understanding of user queries and evaluated responses beyond mere criticism @xcite . therefore, it is challenging to improve the critique ability of llms @xcite . while recent works demonstrate that utilizing human-annotated labels or answers could significantly improve the critique ability of llms @xcite @xcite @xcite , this approach faces scalability challenges due to the substantial demand for human annotation. in contrast, a more scalable and cost-effective way is to conduct the i.e., supervised fine-tuning (sft) using critiques generated by a strong teacher model (gpt-4) @xcite . however, these model-generated critiques often suffer from inaccuracies stemming from the inherent biases of a single model and the complexity of the critique task @xcite . as a result, llms fine-tuned on such datasets inherit these flaws, which are further propagated and potentially amplified during the sft process. to address this issue, we introduce multicritique, a unified framework designed to enhance the critique ability of llms by leveraging multiagent feedback in both sft and reinforcement learning (rl) stages, without any human annotation. first of all, to mitigate the limitations of critiques generated by a single llm, we propose the multicritique-sft pipeline (figure 1 (step 2 )), which aggregates high-quality multi-agent critiques in a fine-grained manner. specifically, multiple advanced llms first provide fine-grained critiques by critiquing responses at both sentence-by-sentence and cross-sentence levels. then, meta-critique, as a specific communication mechanism, judges each critique unit by referencing multi-agent critiques. the meta-critique results are used to summarize a final critique by aggregating high-quality critique units while discarding flawed ones. second, to go beyond simple behavior cloning on model-generated critiques, we introduce the multicritique-rl pipeline, which constructs a high-quality preference critique dataset via multiagent feedback, facilitating the effectiveness of rl in improving the critique ability of llms. specifically, as shown in figure 1 (step 3 ), critiques are paired based on meta-critique evaluations obtained in sft phase, with chosen critiques containing fewer and less severe flaws than rejected ones. the accuracy of preference critique is further improved by validating critiques' contributions to revisions across multiple models, retaining only pairs where the chosen critiques consistently lead to superior revisions. multicritique-rl is free from human annotations, demonstrating better scalability than recent works that rely on human-annotated labels or answers, like critic-rm @xcite . building on the multicritique framework, we construct the multicritiquedataset. extensive experimental results on criticeval @xcite and criticbench @xcite benchmarks demonstrate that several 7b-8b llms finetuned by sft and rl stages on multicritique-dataset significantly outperforms advanced 7b-13b baselines that trained on datasets 3-8x larger than ours. notably, our model achieves performance close to advanced 70b llms and gpt-4. for instance, on criticbench, our model achieves 75.66% f1 score, compared to gpt-4's 78.75%. in addition, our proposed multicritique exhibits superior data efficiency during training, surpassing previous baselines by a factor of 2.15-4.22. ablation studies further validate the positive contributions of our designs in the multicritique framework. moreover, our sft dataset exhibits strong generalization on unseen tasks and enhances the general ability of llms, underscoring its utility and robustness. so far, two primary approaches have been employed to enhance the critique ability of llms: (1) human annotation: this method has demonstrated effectiveness in improving critique ability by using human-annotated labels or critiques @xcite @xcite @xcite , as exemplified by @xcite and deepseek-grm @xcite ; preference-based reinforcement learning reinforcement learning (rl) algorithms are widely utilized to guide llms to generate responses that are more preferred by humans @xcite . it typically employs a reward model as a proxy for human judgment, learning through human-annotated pairwise comparison of responses, often called reinforcement learning from human feedback (rlhf) @xcite . existing works utilize human-annotated or model-generated preference dataset to improve critique ability @xcite @xcite . these models highly rely on the high-quality humanannotated preference datasets, while our proposed multicritique-rl pipeline refines preference critiques with multi-agent feedback to enable robust rl fine-tuning in a more scalable way. multi-agent framework current multi-agent frameworks are widely used in two applications: ## experimental results this section demonstrates the experimental results of our proposed datasets (section 5.1), comparison with baseline datasets (section 5.2), and its scaling phenomenon (section 5.3). ## conclusions and future works in this paper, we propose a novel data generation pipeline, multicritique, to automatically construct the dataset to improve the critique ability of llms through sft and rl fine-tuning stages. extensive experiments demonstrate that multicritique significantly surpasses existing datasets. additionally, the rl fine-tuning stage on multicritique further improves the critique abilities of llms. in the future, we plan to expand multicritique to the pairwise response comparison @xcite , enhancing llms' ability to evaluate paired responses. moreover, we also plan to enhance the quality of multi-critique further and tackle challenging reasoning tasks, like mathematical and coding reasoning.