Reward Generalization in RLHF : A Topological Perspective (2025)

Abstract: Existing alignment methods share a common topology of information flow, where reward information is collected from humans, modeled with preference learning, and used to tune language models. However, this shared topology has not been systematically characterized, nor have its alternatives been thoroughly explored, leaving the problems of low data efficiency and unreliable generalization unaddressed. As a solution, we introduce a theory of **reward generalization** in reinforcement learning from human feedback (RLHF), focusing on the **topology of information flow** at both macro and micro levels. At the macro level, we portray the RLHF information flow as an autoencoding process over behavior distributions, formalizing the RLHF objective of distributional consistency between human preference and model behavior. At the micro level, we present *induced Bayesian networks* to model the impact of dataset topologies on reward generalization. Combining analysis on both levels, we propose **reward modeling from tree-structured preference information**. It is shown to reduce reward uncertainty by up to \Theta(\log n/\log\log n) times compared to baselines, where n is the dataset size. Validation on three NLP tasks shows that it achieves an average win rate of 65% against baselines, thus improving reward generalization *for free* via topology design, while *reducing* the amount of data requiring annotation.

## introduction large language models (llms) pretrained on massive datasets display remarkably general capabilities @xcite @xcite , but due to the mismatch between dataset content and the preference of human users, those capabilities cannot be safely elicited without the alignment process @xcite . alignment methods, especially reinforcement learning from human feedback (rlhf), are developed to correct harmful behaviors learned in pretraining @xcite @xcite . rlhf optimizes the llm against a reward model (rm) serving as a proxy of human evaluation. prior to that, at the reward modeling stage of rlhf, the rm is trained on the preference dataset containing responses preferred and dispreferred by human evaluators @xcite . rlhf is criticized for its lack of scalability to superhuman models @xcite , but even for current models, rlhf still faces a trilemma: the incompatibility between high task diversity, low labeling cost, and alignment performance generalizable across diverse scenarios @xcite . in its essence, the trilemma is caused by insufficient reward generalization, i.e., the insufficient generalization performance of the rm. this insufficiency holds back the pareto front between the amount of labeled preference data and generalizability of rewards across diverse scenarios, and is detrimental to alignment performance @xcite . alternatives to rlhf have been proposed @xcite , but most of them continue to rely on preference data from humans or ai-based human proxies, employing pipelines similar to the rlhf process. consequently, most of them still face the rlhf trilemma. the commonality shared across rlhf variants is their information topology, which we define as the layout of the information flow in the algorithmic process. specifically, the rlhf information topology involves the condensation of preference information into an rm, and the subsequent reconstruction of a language model trained on signals from the rm @xcite . such topology is a key determinant in the generalization performance of alignment algorithms, but has not received systematic characterization. in the present study, we perform such characterization at both macro and micro levels, while also proposing alternative topologies with superior reward generalization performance. concretely, our contributions include: ## macro-level information topologies this section presents a formalism of the macrolevel rlhf information topology, the autoencoding framework. it portrays rlhf as first encoding human preference data into the rm, r rm (•|•), and then decoding preference information from the rm to produce the aligned lm, p lm (•|•). for any prompt x drawn from the prompt space x and response y drawn from the response space y, the reward r rm (y | x) ∈ r represents the quality of y as a response to x, and p lm (y | x) is the probability that lm outputs y when prompted with x. our study does not concern the distribution of the prompt, so we consider only a fixed prompt @xmath0. we shall omit the condition (• | x) and simply write r rm (y) and p lm (y). this approach can be seamlessly extended to settings with varied prompts. below, we introduce the key elements in the macro-level topology of rlhf. idealized human text distribution p h : @xmath1. foot_1 it represents the probabilities of getting every possible response from an idealized human being whose behavior is in perfect alignment with collective human preferences. the determination of this distribution @xcite exceeds the scope of the present study, since our analysis does not rely on the specifics of this distribution. based on a straightforward generalization of the bradley-terry model @xcite , we can further define the idealized human reward function r h : @xmath2 preference dataset @xmath3,i , y b d,i , δ d,i ) . in the rlhf pipeline, pairs of model-generated answers are selected given the prompt, and for each pair, a human evaluator is asked to compare the relative quality of the two answers. here, d represents the dataset resulting from this process, where the fact that δ d,i follows a logistic distribution is again a corollary of the bradley-terry model. in practice, the strength of human preference is usually collected as discrete integer values or even binary labels, which can be seen as discretized δ d,i . in any given case, the finer-grained this discretization is, the more applicable our model will be. ## micro-level information topologies in this section, we work within the autoencoding framework proposed in §3, and zoom in on the encoding stage, focusing on reward generalization and how information topology influences it. specifically, we study the fine-grained topology of the human preference dataset @xmath4,i , y b d,i , δ d,i ) , and how it affects generalization properties of the rm r rm (•). in addition to a general analysis, we study chain-based and treebased information topologies as case studies. for simplicity, we use r d y as an abbreviation for the random variable βr rm (y) under the human preference dataset d. due to space constraints, we will selectively present key definitions, assumptions, and theorems. please refer to appendix b for the complete derivations. ## algorithmic experiments theorem 4.5 suggests the superiority of the treebased method of reward modeling. in this section, we aim to answer the following question: on tasks with diverse context and limited data, is the treebased rm more effective in encoding preferences than chain-based ones? ## conclusion in this study, we introduce macro-and micro-level theories of rlhf reward generalization from a topological perspective, and propose a tree-based method for reward modeling, validating its superiority over the chain-based baseline through both theoretical and experimental means.