D ia T ool- DPO : Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models (2025)

Abstract: Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in real-world applications, but face challenges in handling incomplete queries and out-of-scope requests. While existing approaches rely mainly on Supervised Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method that enhances TA-LLM’s dialogue capabilities through Direct Preference Optimization. We model TA-LLM interactions as a Markov Decision Process with 5 distinct dialogue states and categorize user queries into 3 types based on their state transition trajectories. We automatically construct paired trajectory datasets of correct and incorrect dialogue flows and introduce a specialized objective loss for dialogue control. Our comprehensive evaluation demonstrates that DiaTool-DPO approaches GPT-4o’s performance (94.8% in information gathering, 91% in tool call rejection) with substantial improvements over baseline (44% and 9.6% respectively) while maintaining core functionality. Our approach opens new possibilities for developing TA-LLMs that can handle diverse real-world scenarios without requiring additional expert demonstrations or human labeling.

## introduction the conversational capabilities of tool-augmented large language models (ta-llms) are crucial. specifically, they must be able to control conversation flow by determining whether to 1) ask followup questions for clarification from the user or 2) make a tool call or 3) reject tool calls when no suitable tools(functions, used interchangeably) are available. in particular, failing to generate followup questions or reject tool calls leads to the risk of invoking tool calls with hallucinated information @xcite . while early benchmarks focused on successful tool calls through proper tool selection and argument extraction @xcite @xcite , recent benchmarks increasingly evaluate the ability to engage in user conversations @xcite @xcite . however, we find that supervised fine-tuning (sft) techniques alone has its limitation to learn conversational skills @xcite @xcite . while research applying techniques beyond sft for training ta-llms is limited, many other areas of llm research successfully utilize reinforcement learning to capture subtle human preferences @xcite . notably, many works solved agent tasks such as webshop, alfworld, and scienceworld with reinforcement learning techniques due to their explicitly defined rewards in the datasets. @xcite @xcite @xcite @xcite here, we formulate ta-llm task as a markov decision process(mdp) and adapt reinforcement learning techniques to enhance ta-llms' conversational abilities. we hypothesize that ta-llms have five internal states and classify user queries into three types based on state transition trajectories @xcite @xcite . we automatically generate rejected trajectories by pairing user queries with mismatched conversation trajectories. the resulting dataset d consists of paired trajectories (τ c , τ r ), where τ c represents the chosen (preferred) trajectory and τ r denotes the rejected (non-preferred) trajectory for each training instance. while our primary experiments were conducted in korean, we demonstrate that our approach is language-agnostic and can be applied to english as well. this dataset is termed the diatool-dpo (dialogue tool dpo) dataset, with its corresponding training algorithm named diatool-dpo. to comprehensively evaluate the impact of this dataset and algorithm, we assess not only slot-filling and tool call rejection capabilities but also all other general competencies across various base llm models. in this paper, we leverage direct preference optimzation(dpo) techniques to improve conversational abilities by controlling ta-llm's dialogue flow @xcite . specifically, the contributions of this paper are as follows: the source code of this work will be released. 1 ## related works benchmarks for ta-llms early benchmarks (t-eval @xcite , tooleyes @xcite , bfcl @xcite ) focused on tool call evaluation. recent ones emphasize conversational abilities, including tool awareness @xcite , query disambiguation @xcite , and multi-turn dialogue (bfcl v3 @xcite , api-bank @xcite , toolsandbox @xcite ). functionchat-bench @xcite provides a comprehensive evaluation framework for tool-related capabilities. reinforcement learning for agent tasks recent work has explored reinforcement learning for improving llm-based agents. @xcite addressed disambiguation using quasi-online dpo with user simulation @xcite . qiao et al. (2024) enhanced tool usage through rankingbased learning that considers response-ground truth distances. song et al. (2024) combined sft with offline exploration, creating dpo datasets from expert and failure trajectories. shi et al. (2024) 1 https://github.com/kakao/diatool-dpo and xiong et al. (2025) introduced multi-turn dpobased learning methods for language agents and math agents, respectively. ## experiments 6.1 experimenetal setup unless otherwise specified, we used llama3-8b-instruct as the baseline model @xcite . all models undergo two sequential training phases prior to diatool-dpo: continual pretraining(cpt) and sft. details of cpt, sft and training setup are available at appendix b and c. the diatool-dpo was trained on 95% of the dataset, with the remaining 5% reserved for validation. to rigorously evaluate the model's generalization performance, we conducted testing on the independent functionchat-bench dataset, which remained completely isolated from both training and validation processes @xcite . ## conclusion in this paper, we introduced diatool-dpo, a novel method for enhancing ta-llms' conversational capabilities through dpo. our approach effectively addresses key challenges in dialogue management by formulating the task as a markov decision pro-cess with five distinct states and suggesting a noble objective loss that is specified for ta-llms. comprehensive evaluation shows that diatool-dpo significantly improves both slot-filling and tool call rejection capabilities while maintaining core functionalities, achieving 94.8% of gpt-4's slot-filling performance and 91% of its tool call rejection accuracy. our analysis demonstrates the benefits of exposure to varied difficulty levels and provides insights into indirect learning transfer between tool call acceptance and tool call reject across different model architectures.