A Comprehensive Survey on Learning from Rewards for Large Language Models: Reward Models and Learning Strategies (2025)

Abstract: Recent developments in Large Language Models (LLMs) have shifted from pre-training scaling to post-training and test-time scaling. Across these developments, a key unified paradigm has arisen: Learning from Rewards, where reward signals act as the guiding stars to steer LLM behavior. It has underpinned a wide range of prevalent techniques, such as reinforcement learning (RLHF, RLAIF, DPO, and GRPO), reward-guided decoding, and post-hoc correction. Crucially, this paradigm enables the transition from passive learning from static data to active learning from dynamic feedback. This endows LLMs with aligned preferences and deep reasoning capabilities for diverse tasks. In this survey, we present a comprehensive overview of learning from rewards, from the perspective of reward models and learning strategies across training, inference, and post-inference stages. We further discuss the benchmarks for reward models and the primary applications. Finally we highlight the challenges and future directions.

## introduction recent years have witnessed the rapid advancement of large language models (llms), such as chat-gpt (openai, 2023), claude @xcite , and llama @xcite . these models are initially empowered by pre-training scaling @xcite , which trains llms on massive corpora through next-token prediction. while this approach enables broad linguistic and knowledge representations, it suffers from several fundamental limitations: misalignment with human values @xcite @xcite , difficulty in adapting to various task objectives @xcite , and deficiencies in deep reasoning @xcite this paradigm enables llms to learn actively from dynamic feedback, in contrast to learning passively from static data. as such, this endows llms with aligned preferences and deep reasoning and planning abilities, leading to more intelligent agents. in consequence, this paradigm has inspired many applications, such as mathematical reasoning @xcite , code generation @xcite , multimodality @xcite , and agents @xcite . due to this growing prevalence, we comprehensively survey the learning from rewards for llms. we first introduce a taxonomy that categorizes existing works with a unified conceptual framework regarding reward model design and learning strategies (sec. 2). then we review representative techniques across three main stages: training with rewards, inference with rewards, and post-inference with rewards (sec. 3 to 5). we additionally summarize primary applications, recent reward model benchmarks, and key challenges and promising directions for future research (appendices a to c). ## a taxonomy of learning from rewards for llms we first introduce a unified conceptual framework that captures the key components and interactions to understand learning from rewards systemically. building upon this framework, we categorize the primary dimensions along which existing methods vary: each dimension reflects a distinct aspect of how reward signals are acquired, represented, and utilized in language models. ## training with rewards in this section, we introduce the methods for training llms with rewards. they contribute to posttraining scaling for preference alignment and testtime scaling by eliciting long cot abilities. ## inference with rewards after the training stage, inference with rewards offers a flexible and lightweight mechanism to adapt and steer the model behavior without modifying model parameters. we identify two primary inference-with-rewards strategies: (i) generatethen-rank and (ii) reward-guided decoding. these strategies play a critical role for achieving test-time scaling: they allow the language model to search, reflect, and revise its outputs on the fly. ## benchmarking reward models rigorous and diverse benchmarks are essential for evaluating the performance of reward models. as illustrated in figure 6 , recent benchmarks primarily rely on human annotators or ai annotators followed by human verification. the resulting annotations are mainly pointwise (e.g., scalar scoring) or pairwise (e.g., selecting the preferred response given two options). rewardbench @xcite is the first comprehensive benchmarks for reward models. it aggregates preference data from existing datasets to evaluate reward model performance in chatting, reasoning, and safety. rm-bench @xcite and rmb @xcite extend it to more scenarios. some focus on prms, like processbench @xcite , mr-ben @xcite , and prmbench @xcite . due to the page limitation, we discuss more about benchmarks, applications, challenges, and future directions in appendices a to c. ## conclusion we comprehensively survey the emerging paradigm of learning from rewards. we introduce its landscape from three key stages: training, inference, and post-inference, each reflecting a distinct way to integrate reward signals into steering llms' behavior. in addition, we summarize recent progress in benchmarking reward models and applications. finally we identify core challenges and outline promising future directions. we hope this survey provides a structured understanding of the field and inspires future research.