Direct Multi-Turn Preference Optimization for Language Agents (2024)

Abstract: Adapting Large Language Models (LLMs) for agent tasks is critical in developing language agents. Direct Preference Optimization (DPO) is a promising technique for this adaptation with the alleviation of compounding errors, offering a means to directly optimize Reinforcement Learning (RL) objectives. However, applying DPO to multi-turn tasks presents challenges due to the inability to cancel the partition function. Overcoming this obstacle involves making the partition function independent of the current state and addressing length disparities between preferred and dis-preferred trajectories. In this light, we replace the policy constraint with the state-action occupancy measure constraint in the RL objective and add length normalization to the Bradley-Terry model, yielding a novel loss function named DMPO for multi-turn agent tasks with theoretical explanations. Extensive experiments on three multi-turn agent task datasets confirm the effectiveness and superiority of the DMPO loss.

## introduction developing generalist agents capable of solving complex tasks has been a central goal in the artificial intelligence community @xcite . recently, language agents @xcite emerge as a prominent research direction, leveraging the considerable potential of large language models to address intricate tasks involving instruction following @xcite , action planning @xcite , and tool utilization @xcite . nevertheless, the substantial disparity between the pretraining task of large language models and the requirements of agent tasks suggests significant potential for future advancements in language agent capabilities. behavioral cloning (bc) @xcite is a frequently employed approach to bridge the domain gap by fine-tuning llms through expert agent trajectories. recent endeavors in bc @xcite @xcite involve the supervised fine-tuning of llms on optimal state-action pairs. although these methods enable swift adaptation of llms to agent tasks, bc is notably susceptible to compounding errorsminor errors of the learner accumulate along interactions between the agent and environment, leading to performance deterioration in non-deterministic environments @xcite . in alleviating compounding errors, direct preference optimization @xcite has demonstrated remarkable success in the single-turn preference alignment task due to its simple implementation and robustness. dpo optimizes rl objectives by maximizing the likelihood of preferred responses over dis-preferred responses, mitigating the need for continuous interaction with the environment and the training instability commonly associated with traditional rl algorithms @xcite . although there has been an initial endeavor to apply the dpo loss on llms for agent tasks @xcite , it encounters suboptimal performance, as it is tailored specifically for the single-turn bandit setting and is ill-suited for multi-turn agent tasks. this work aims to develop a robust loss function capable of directly optimizing rl objectives in multi-turn scenarios. the crux of this pursuit involves eliminating the partition function in the bradley-terry (bt) model (bradley and @xcite . this entails ensuring the partition function's independence from the current state and neutralizing the impact of the length disparity between preferred and dis-preferred trajectories. to achieve this, we substitute the policy constraint with the state-action occupancy measure (saom) @xcite constraint in the rl objective and introduce length normalization into the bt model. these adjustments culminate in the development of a new and simple loss function dmpo for multi-turn agent tasks. as shown in figure 1 , dmpo directly optimizes the rl objective by maximizing the likelihood of preferred ("win") trajectory over dis-preferred ("lose") trajectory. notably, the saom constraint has advantages in mitigating compounding errors compared to the policy constraint @xcite . furthermore, the derivation offers a theoretical rationale for the efficacy of the length normalization technique in dpo loss @xcite . to summarize, our contributions are threefold: ## related work in this section, we first introduce the in-context learning methods and fine-tuning methods of language agents and then review the literature in preference-based rl. in-context learning inspired by the superior in-context learning capabilities of llms @xcite , researchers have designed various instruction prompts for llms, equipped with memory modules @xcite , toolkits @xcite , and various workflows @xcite , to build language agents for various real-world domains. react @xcite incorporates cot reasoning @xcite into action generation. reflexion @xcite and promst @xcite refine the prompt using environment feedback. however, these in-context learning methods fail to fully exploit the potential of llms, since most llms are not specifically trained for agent tasks. this work focuses on adapting the llms to agent tasks through fine-tuning. agent tuning recent studies, including fire-act @xcite , agenttuning @xcite , lumos @xcite , mimir @xcite , autoact @xcite , and α-umi @xcite supervised fine-tuning llms with self-instruct or expert trajectories. however, such bc approaches suffer from compounding errors when interacting with dynamic environments. taking a step further, pangu @xcite and cmat @xcite utilize rl technologies to further fine-tune the llms, which may result in a complex and unstable training procedure. to simplify the procedure, eto @xcite and emma @xcite directly employ the dpo loss @xcite to optimize the rl objective for the agent task. nevertheless, the dpo loss is designed for single-turn bandit settings and is ill-suited for multiturn scenarios. along this line, this work extends the dpo loss in multi-turn scenarios and derives the dmpo loss. preference-based rl in multi-turn scenarios, preference-based rl typically starts by explicitly learning a reward function from preference data and then optimizing it @xcite @xcite . however, this two-stage learning process presents challenges regarding training efficiency and instability. this work instead presents a single-stage policy learning approach using dmpo loss that directly optimizes a policy to satisfy preferences. while ipl @xcite and cpl @xcite share a similar idea with our work in eliminating the reward learning stage, their loss functions are limited to trajectory pairs of equal length, significantly restricting their applicability. ## preliminary in this section, we present multi-turn agent task formulation and briefly introduce direct preference optimization (dpo) loss. ## method in this section, we will outline the definition and benefits of the state-action occupancy measure. subsequently, we will introduce two adjustments to derive the dmpo loss. finally, we will delve deeper into the analysis of the dmpo loss. ## experiments in this section, we conduct extensive experiments on three agent tasks to demonstrate the effectiveness of the proposed dmpo loss function. our experiments aim to address the following questions: ## conclusion in this work, we propose a simple and robust loss function dmpo loss, which directly optimizes the rl objective for multi-turn agent tasks. by substituting the policy constraint with the saom constraint and introducing the length normalization into bt model, we eliminate the partition function in the bt model and derive the dmpo loss function. the saom constraint has played a pivotal role in mitigating compounding errors. meanwhile, this derivation offers a theoretical rationale for the efficacy of the length normalization technique. extensive experiments on three agent datasets demonstrate the effectiveness of dmpo loss, highlighting its capability to reduce compounding errors and its resilience to trajectory length disparity.