I nfo PO : On Mutual Information Maximization for Large Language Model Alignment (2025)

Abstract: We study the post-training of large language models (LLMs) with human preference data. Recently, direct preference optimization and its variants have shown considerable promise in aligning language models, eliminating the need for reward models and online sampling. Despite these benefits, these methods rely on explicit assumptions about the Bradley-Terry (BT) model, which makes them prone to overfitting and results in suboptimal performance, particularly on reasoning-heavy tasks. To address these challenges, we propose a principled preference fine-tuning algorithm called InfoPO, which effectively and efficiently aligns large language models using preference data. InfoPO eliminates the reliance on the BT model and prevents the likelihood of the chosen response from decreasing. Extensive experiments confirm that InfoPO consistently outperforms established baselines on widely used open benchmarks, particularly in reasoning tasks.

## introduction large language model alignment with human preferences is critical to ensure that the responses of pre-trained llms to prompts are consistent with human preferences @xcite @xcite . recently, reinforcement learning from human feedback (rlhf) @xcite has been proposed for fine-tuning language models based on human preferences. rlhf involves initially fitting a reward signal derived from human preference data with the application of reinforcement learning algorithms, such as proximal policy optimization @xcite , to optimize language model policy to maximize rewards. rlhf demonstrates impressive capabilities across diverse tasks. yet, the reinforcement learning approach presents significant challenges, such † work done during internship at amazon as computational inefficiency and training instability @xcite . to address these issues, methods such as direct preference optimization and its variants have been proposed, including dpo @xcite , r-dpo @xcite , and simpo @xcite . these preference optimization approaches @xcite , replace rlhf with supervised learning on preference data, eliminating the need for explicit reward modeling. specifically, they use the likelihood of a policy to define an implicit reward fitted to the preference data, achieving promising alignment performance. while these methods employ different losses, they are all based on bt assumption and share a similar motivation with dpo and simpo: maximizing the relative value differences between the implicit rewards of the chosen and rejected responses. despite its simplicity and initial promise, this bt assumption may not always hold true and generally decreases reasoning task performance, as discussed in @xcite @xcite . specifically, a notable counter-intuitive observation is that during the training process of methods with bt assumption, the likelihood of both the chosen (i.e., preferred) and rejected (i.e., less preferred) responses decreases due to the large gradient associated with the rejected response. this leads to an undesirable outcome where the learned policy progressively focuses on unlearning the rejected responses (see section 4 for details) and decrease the likelihood of chosen responses as shown in figure 1 , resulting in suboptimal performance on reasoning benchmarks as shown in many recent works @xcite @xcite . recently, several efforts @xcite have been made to address this issue. they propose using negative log likelihood regularization on chosen responses to stabilize the training process. although these methods successfully prevent the model from collapsing on mathematical datasets, they perform poorly on instruction-following and chat benchmarks @xcite and introduce additional hyper-parameters which require manual tuning. the importance of keeping the likelihood of the chosen response in practical applications of large language models, such as reasoning and mathematical problem-solving @xcite , highlights a significant limitation in the applicability of contrastive preference learning. this raises the following question: can we design an effective preference optimization algorithm that avoids the bradley-terry (bt) assumption? in this paper, we address this question by proposing a simple yet effective preference optimization algorithm, named infopo, which does not rely on the bt assumption. the key idea of infopo is to directly optimize the conditional mutual information between responses and preferences given a prompt. we first revisit the dpo objective from the perspective of mutual information maximization. in particular, we demonstrate that the objective functions of dpo under the bt assumption can be characterized as special cases of a mutual information maximization objective, using @xcite as the estimator. building on this insight, we propose a novel method that learns an effective policy from preference data without relying on the bt assumption. specifically, infopo leverages the nwj estimator @xcite for mutual information estimation instead of infonce. intuitively, infopo weights the log likelihood of rejected responses according to model probability and uses an exponential operation to control the gradient norms towards rejected responses. we show that infopo enables the model to update conservatively in the direction of rejected responses, while preventing a decrease in the likelihood of chosen responses. this results in improved downstream task performance, particularly in reasoning-heavy tasks. we conduct extensive experiments to thoroughly evaluate infopo with llama3 8b and mistral 7b on a wide range of downstream benchmarks: open llm leaderboard and alpacaeval 2 and anthropic-hh. infopo achieves consistent and significant improvements over existing methods. our primary technical contributions are: (1) we propose a simple and effective alignment approach based on mutual information maximization, which can naturally prevent the model from overfitting to preference data, striking a better balance between chat and reasoning abilities. (2) we theoretically analyze the learning behavior and prove that infopo enjoys some of the properties that are desirable for fine-tuning with preferences. (3) empirically, we corroborate the effectiveness of infopo on seven benchmarks. the results demonstrate that infopo can significantly outperform previous preference optimization methods. ## notations and preliminaries problem setup. we consider the preference learning scenario as follows: let the text sequences @xmath0, x 2 , . . .] ∈ x denote the input prompt, and y @xmath1, y 2 , . . .] ∈ y and y @xmath2, y 2 , . . .] ∈ y denote two responses, typically sampled from the reference policy π ref (y|x). the response pairs are then presented to human labelers (or an oracle) who express preferences for responses given the prompt, denoted as y w ≻ y l |x, where y w and y l denote preferred and dispreferred responses, respectively. the preference distribution is typically expressed using a latent reward model r(x, y) as: specifically, the reward function r ϕ (x, y) can be estimated by maximizing the log-likelihood over preference feedback (x, y w , y l ): to simplify rlhf, contrastive preference learning @xcite @xcite uses the loglikelihood of the learning policy to implicitly represent the reward function: technically, dpo or its variants, such as those of @xcite and @xcite , are essentially based on bt preference assumption (bradley and terry, 1952) which maximizes the relative reward differences between chosen and rejected responses @xcite . however, the likelihood of the chosen response can continue to decrease during training as long as the relative difference in the likelihoods between the chosen and rejected responses remains large. in this paper, we address this limitation by proposing a novel objective based on mutual information maximization. ## experiments in this section, we present main results of our experiments, highlighting the superior alignment performance of infopo on various benchmarks. ## conclusion this paper presents infopo, a novel preference fine-tuning method to align llms with preference data. we provide a novel perspective on mutual information maximization for the alignment problem, and demonstrate dpo with bt assumption essentially optimize the contrastive infonce objective. to address the limitation of dpo, we propose infopo based on nwj mutual information estimator. infopo applies an exponential function to con-