Despite the success of instruction tuning, relative human judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of human preferences, improving proficiency in translation, summarization, story-telling, and instruction-following. These methods first optimize a neural network reward function for compatibility with the dataset of preferences under a preference model such as the Bradley-Terry model, then fine-tune a language model to maximize the given reward using reinforcement learning algorithms, commonly REINFORCE, proximal policy optimization (PPO), or variants. A closely-related line of work leverages LLMs fine-tuned for instruction following with human feedback to generate additional synthetic preference data for targeted attributes such as safety or harmlessness, using only weak supervision from humans in the form of a text rubric for the LLM's annotations. These methods represent a convergence of two bodies of work: one body of work on training language models with reinforcement learning for a variety of objectives and another body of work on general methods for learning from human preferences. Despite the appeal of using relative human preferences, fine-tuning large language models with reinforcement learning remains a major practical challenge; this work provides a theoretically-justified approach to optimizing relative preferences without RL. Outside of the context of language, learning policies from preferences has been studied in both bandit and reinforcement learning settings, and several approaches have been proposed. Contextual bandit learning using preferences or rankings of actions, rather than rewards, is known as a contextual dueling bandit (CDB). In the absence of absolute rewards, theoretical analysis of CDBs substitutes the notion of an optimal policy with a von Neumann winner, a policy whose expected win rate against any other policy is at least 50%. However, in the CDB setting, preference labels are given online, while in learning from human preferences, we typically learn from a fixed batch of offline preference-annotated action pairs. Similarly, preference-based RL (PbRL) learns from binary preferences generated by an unknown 'scoring' function rather than rewards. Various algorithms for PbRL exist, including methods that can reuse off-policy preference data, but generally involve first explicitly estimating the latent scoring function (i.e. the reward model) and subsequently optimizing it. We instead present a single stage policy learning approach that directly optimizes a policy to satisfy preferences. Section 3: Preliminaries  We review the RLHF pipeline in Ziegler et al. (and later Stiennon et al., Bai et al., Ouyang et al.). It usually includes three phases: 1) supervised fine-tuning (SFT); 2) preference sampling and reward learning and 3) RL optimization. SFT: RLHF typically begins by fine-tuning a pre-trained LM with supervised learning on high-quality data for the downstream task(s) of interest (dialogue, summarization, etc.), to obtain a model. Reward Modelling Phase: In the second phase the SFT model is prompted with prompts x to produce pairs of answers. These are then presented to human labelers who express preferences for one answer, denoted as the winning response being preferred over the losing response given the prompt x. The preferences are assumed to be generated by some latent reward model, which we do not have access to. There are a number of approaches used to model preferences, the Bradley-Terry (BT) model being a popular choice (although more general Plackett-Luce ranking models are also compatible with the framework if we have access to several ranked answers). The BT model stipulates that the human preference distribution can be written as a logistic function of the difference in rewards between two responses. Assuming access to a static dataset of comparisons, we can parametrize a reward model and estimate the parameters via maximum likelihood. Framing the problem as a binary classification we have the negative log-likelihood loss. In the context of LMs, the network for the reward model is often initialized from the SFT model with the addition of a linear layer on top of the final transformer layer that produces a single scalar prediction for the reward value. To ensure a reward function with lower variance, prior works normalize the rewards, such that the expected reward over the dataset is 0 for all prompts. RL Fine-Tuning Phase: During the RL phase, the learned reward function is used to provide feedback to the language model. Following prior works, the optimization is formulated as maximizing the expected reward, with a penalty term based on the KL-divergence between the current policy and a base reference policy, which is the initial SFT model. In practice, the language model policy is also initialized to the SFT model. The added constraint is important, as it prevents the model from deviating too far from the distribution on which the reward model is accurate, as well as maintaining the generation diversity and preventing mode-collapse to single high-reward answers. Due to the discrete nature of language generation, this objective is not differentiable and is typically optimized with reinforcement learning. The standard approach has been to construct the reward function and maximize using PPO. Section 4: Direct Preference Optimization  Motivated by the challenges of applying reinforcement learning algorithms on large-scale problems such as fine-tuning language models, our goal is to derive a simple approach for policy optimization using preferences directly. Unlike prior RLHF methods, which learn a reward and then optimize it via RL, our approach leverages a particular choice of reward model parameterization that enables extraction of its optimal policy in closed form, without an RL training loop. As we will describe next in detail, our key insight is to leverage an analytical mapping from reward functions to optimal policies, which enables us to transform a loss function over reward functions into a loss function over policies.