Self-Generated Critiques Boost Reward Modeling for Language Models (2025)

Abstract: Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of the generated critiques.

## introduction reinforcement learning from human feedback (rlhf) has been widely adopted to align large language models (llms) with human preferences @xcite @xcite . central to the rlhf process is the reward model (rm), which is trained to assign scores that quantify how well the model's outputs align with human judgments. the reward model defines optimization direction during training (e.g., reward signal in ppo), encouraging a policy llm to generate more helpful, honest, and harmless responses ultimately enhancing the model's generation quality in real-world applications. standard reward models are typically trained using preference pairs and optimized to produce a single scalar score for each response. however, outputting a scalar score not only is hard to interpret but also fails to fully leverage the inherent language modeling capability that llms obtain from pretraining and post-training @xcite . consequently, these reward models tend to be less data-efficient and prone to robustness issues, such as reward hacking @xcite . such limitations hinder the quality of feedback signals in rlhf and lead to suboptimal policy updates. on the other hand, the llmas-a-judge paradigm offers an alternative, where the llm first generates a critique and then optionally provides a discrete score as a quality proxy for a response @xcite . combining the strengths of both paradigms -integrating the interpretability and structured critique of llm-asthe-judge with the scalar optimization framework of reward models -has the great potential to address the limitations of each method and yield more robust and effective reward signals. despite its great premise, incorporating critiques into reward modeling presents several challenges. (1) conflicting objectives: critique generation requires language modeling, while reward models provide scalar outputs, complicating its integration into language modeling. (2) evaluator limitations: off-the-shelf lms are often not good evaluators, while additional fine-tuning requires costly humangenerated or annotated critiques. recent work @xcite directly incorporates critiques generated from off-the-shelf llms for reward modeling, while @xcite and @xcite design a joint training approach for learning to generate the critique as well as rewards simultaneously via knowledge distillation. these methods typically rely on a strong teacher llm to generate high-quality critiques, which can be costly and inefficient to obtain at scale in practice. moreover, they ## conclusion in this work, we introduce critic-rm, a selfcritiquing framework designed to enhance reward modeling for large language models. critic-rm implements a novel self-improvement approach that improves both critique quality and reward prediction accuracy. experiments on multiple datasets demonstrate that critic-rm consistently outperforms baseline reward models, showing strong data efficiency and delivering robust results even with limited labeled data. moreover, the critiques generated by critic-rm prove effective in helping llms enhance response quality. we hope this selfcritiquing technique will offer a promising future direction for advancing reward modeling and improving the alignment between llms and human preferences.