Training Language Model to Critique for Better Refinement (2025)

Abstract: Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limited research has explored which types of critiques are most effective for improving model responses or how to generate such critiques. To address this gap, we introduce Refinement-oriented Critique Optimization (RCO), a novel framework designed to train critic models using refinement signals. RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses. The critique utility (CU) quantifies the effectiveness of these refinements, serving as the reward signal for training the critic model. By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment, ensuring that critiques driving meaningful improvements are rewarded. We evaluate RCO across five tasks—dialog generation, summarization, question answering, mathematical reasoning, and code generation—and show that it significantly outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. Our contributions include the introduction of RCO, a novel supervision scheme based on refined response preferences, and comprehensive experimental results that highlight the method’s effectiveness in enhancing LLM critique-refinement loops. Code and data will be publicly available upon acceptance of this paper.

## introduction the critique ability, defined as the capacity to identify and refine flaws in responses, is crucial for improving the reliability of automatic evaluation and enabling the self-improvement of large language models (llms) @xcite . enhancing this ability is challenging, as it requires llms to not only detect errors but also engage in a deeper analysis of user queries and responses, extending beyond superficial criticism @xcite . recent studies have sought to address this challenge of llms via human-curated critique datasets and alignment strategies like supervised fine-tuning (sft) @xcite and reinforcement learning @xcite , yielding promising results in identifying and rectifying shortcomings across various tasks. however, current methods primarily train models to generate critiques for evaluation purposes, rather than linking critique to refinement @xcite . this disjointed approach fails to connect error identification with output improvement, limiting the practical value of critiques. while the critique-refinement paradigm is explored in llm alignment @xcite and reasoning @xcite , existing works lack systematic investigation on how critique enhancements can lead to meaningful refinements in responses. without this link, assessing critique quality remains challenging @xcite . addressing this gap is essential for advancing both llm evaluation and self-improvement capabilities. to address these challenges, we introduce refinement-oriented critique optimization (rco), a novel training paradigm for critic models that uses critique utility (cu) as a reward signal, which is calculated by comparing the refined responses to the initial ones. this approach encourages the generated critiques to drive substantial improvements in the output, creating a more effective critiquerefinement loop. specifically, rco feeds critiques and initial responses into the actor model and prompts it to generate multiple refined responses. cu is then quantified as the proportion of refinements preferred over the initial response, which serves as the reward signal for training the critic model. by focusing on the refinement outcomes, rco eliminates the need for direct critique preference assessment and ensures that critiques leading to better refinements are rewarded. we evaluate rco across five tasks, i.e., dialog generation, summarization, question answering, mathematical reasoning, and code generation, using baseline models, models trained with critique preferences, and advanced open-source models. experimental results demonstrate that rco significantly enhances critic model performance, outperforming existing methods across multiple benchmarks. our contributions are threefold: @xcite we propose rco, a method that prioritizes critiques that facilitate effective refinement of actor model responses, addressing limitations of previous approaches. @xcite we introduce a novel supervision scheme based on refined response preferences, eliminating the need for directly assessing the quality of critiques while rewarding critiques that lead to meaningful improvements. @xcite we rigorously evaluate rco across diverse tasks, showing substantial improvements in critique quality and refinement capabilities, and provide an in-depth analysis of the method's impact. ## related work critique ability of llms the rapid advancement of large language models (llms) has highlighted the need to enhance their critique capabilities. llms like gpt-4 @xcite have proven effective as evaluators @xcite @xcite , but their apibased access limits widespread application. to address cost and stability issues, researchers have fine-tuned open-source models using critique data from these models. despite these efforts, challenges in critique task complexity remain. recent work includes @xcite , which uses prompt engineering to create critique metrics, and @xcite , which proposes using multiple llms to mitigate bias. @xcite generates a golden critique dataset from model and reference response pairs, while @xcite employs a multi-agent framework for preferencebased critique data collection. in contrast, our method improves critique quality by automatically obtaining preferences of refined responses to calculate critique utility as reward signals to train critic models. preference-based reinforcement learning reinforcement learning from human feedback (rlhf) @xcite is commonly used to guide llms towards human-preferred responses. @xcite uses rlhf to train reward models based on human-annotated pairwise comparisons. recent methods, such as crit-icgpt @xcite , apply rlhf to enhance critique abilities by training critic models on human-identified errors in code generation. similarly, @xcite collects preference critique pairs by comparing llm-generated responses with human-annotated scores. these approaches are limited by the high cost and uncertain quality of human annotations @xcite . in contrast, our approach reduces reliance on human annotations, providing a clear standard for good critiques that effectively guide actor model refinements. ## methodology the overview of rco is illustrated in figure 2 . our approach begins with a dataset d, where each sample @xmath0, y 0 ) consists of a prompt x and its corresponding initial response y 0 , with y 0 being generated by an actor model π(y 0 |x). following this, a base critic model p(c|y 0 , x) is employed to generate n distinct critiques, denoted as c 1 , c 2 , . . . , c n , for the initial response y 0 . each critique c i is then used as input to the actor model π, which produces m distinct refined responses, y i1 , y i2 , . . . , y im . the distribution used to sample these refined responses is represented as π c i (y|c i , y 0 , x). critique utility. as highlighted in section 1, the critique process is increasingly integrated into critique-refinement workflows, requiring supervision from a refinement perspective. inspired this, we propose to use critique utility (cu) as the reward signal of rco. the central idea of cu is that a high-quality critique should naturally lead to a better refinement, serving as a catalyst for selfimprovement @xcite . specifically, cu for a critique c i is defined as the probability where a response y, sampled from π c i (y|c i , y 0 , x), is preferred over the initial response y 0 : to estimate cu(c i |y 0 , x), we approximate it using the following sampling-based approach, by defining a preference score (ps) for the refined response y ij and the initial response y 0 : in this equation, the preference score ps(y ij , y 0 ) is determined by a judge model based on the following criteria: ps(y ij , y 0 ) = 1 if the refined response y ij is preferred over the initial response y 0 , ps(y ij , y 0 ) = 0.5 if both responses are considered equally good, and ps(y ij , y 0 ) = 0 if y 0 is preferred over y ij . this process is repeated for each critique c i . ## experiment settings in this section, we describe the benchmarks, evaluation metrics, baselines, and experimental results that assess the performance of rco. ## analysis 6.1 weak-to-strong generalization ability recent studies in llms have highlighted the growing trend of scalable oversight @xcite , which requires smaller models to provide meaningful feedback to enhance the performance of larger models. to evaluate whether rco facilitates weakto-strong generalization, we reanalyze the experimental results presented in based on the findings in ## conclusion in this paper we introduced rco, a novel approach for training critic models to improve actor model refinement. by proposing a supervision scheme based on preferences for refined responses, we eliminate the need for direct critique quality assessment while rewarding critiques that drive meaningful improvements. rigorous evaluations across five tasks show significant advancements in both critique quality and refinement capabilities compared to existing methods. our in-depth analysis underscores the effectiveness of rco and its potential to enhance the alignment between critic and actor models. these contributions provide valuable insights for designing more efficient and scalable systems for model refinement, laying the foundation for future research in this field.