Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systems (2024)

Abstract: Task-oriented Dialog (ToD) systems have to solve multiple subgoals to accomplish user goals, whereas feedback is often obtained only at the end of the dialog. In this work, we propose SUIT (SUbgoal-aware ITerative Training), an iterative training approach for improving ToD systems. We sample dialogs from the model we aim to improve and determine subgoals that contribute to dialog success using distant supervision to obtain high quality training samples. We show how this data improves supervised fine-tuning or, alternatively, preference learning results. Performance improves when applying these steps over several iterations: SUIT reaches new state-of-the-art performance on a popular ToD benchmark.

## introduction task-oriented dialog (tod) systems nowadays converse with users in natural language conversations and assist them in various tasks, such as booking restaurants, querying weather forecasts and resolving customer service issues. fig. 1 shows a sample conversation for making a hotel reservation, where the user's goal is defined as a set of constraints (informable slots, e.g. pricerange) and information needs (requestable slots, e.g. hotel address) that should be satisfied at the end of the dialog. for accomplishing such goals, tod systems must be able to solve multiple subproblems: (1) dialog state tracking (dst) -understanding user utterances and keeping track of the conversation by storing relevant information in a structured representation of the dialog progress (belief states b i , e.g. area or price range in fig. 1 ), (2) inferring how to react by selecting some dialog actions like database lookups or requesting more information from the user (actions a i like request number, inform address in fig. 1 ), (3) formulating a natural language response based on the dialog state and actions (like asking about the length of the stay and the number of people in fig. 1 , r i ). specialized approaches focus on solving specific problems, e.g., amongst others, @xcite , ag-dst @xcite , d3st @xcite focus on dst, larl @xcite , tcup @xcite concentrate on response generation. end-toend (e2e) systems try to leverage complex models to solve all problems jointly. modern tod systems in that category are based on pre-trained large language models (llms) and cast dialog state, action and response generation into sequence prediction problems. simpletod (hosseini-asl et al., 2020) was the first approach which successfully applied this paradigm, by training a causal language model using supervised fine-tuning (sft). a challenge for tod systems is the fact that ultimate success with respect to the user's goal is observed at the end of the dialog. while reinforcement learning (rl) approaches @xcite @xcite optimize for such sparse rewards, most llm-based systems neglect these signals and only optimize next turn predictions. contributions. we introduce suit (= subgoalaware iterative training), an e2e tod system based on llms, which contrary to prior work learns from dialog-level success signals. due to the sparseness of these signals, it is unclear which turns, states, actions and responses contribute to the overall success of the dialog. for example, the second turn in fig. 1 is irrelevant for the success of the dialog. the user's goal does not depend on the availability of sights in close vicinity to the hotel, contrary to the respective price range, which will affect the system's success. we aim to identify these important subparts, which we call subgoals, from multiple dialog variants generated by an llm. a naive approach would consider all generations from successful dialogs for sft, or pair all generations from successful with unsuccessful dialogs for preference learning algorithms, such as rlhf @xcite or direct preference optimization (dpo) @xcite . however, these naive approaches cannot distinguish between subgoals that are relevant for the final goal from those that are not. we employ an iterative distant supervision approach to identify these subgoals that play a major role in dialog success to obtain relevant training samples. our contributions are as follows: ## suit training approach fig. 2 provides an overview of the suit approach. as input we rely on a tod dataset d, where each dialog is associated with a user goal. this goal describes the user's information need and a set of constraints, that should be fulfilled at the end of the dialog. first, an initial llm is trained on d with supervised fine-tuning (sft), using turn-level supervision (step 0). then, we sample from this model to create dialog variants for each user goal in the training data (step 1). next, we determine dialog success for the newly created dialogs (step 2). for each successful dialog, we apply distant supervision to identify subgoals that contribute to the ultimate success of the dialog, by comparing them with generations coming from unsuccessful dialogs (more details in sec. 2.2). the subgoals considered relevant comprise further training samples (step 3). the suit approach can apply sft or preference learning based on a dataset which pairs subgoals with negative examples from unsuccessful dialogs (step 4). this procedure is repeatable by sampling from the newly obtained model once again (step 5). using this effective training paradigm, suit improves sota performance (see sec. 3). compared to prior e2e tod systems, suit is not based on model customization and does not require feedback from reward models or annotators. the fact, that any off-the-shelf llm can be plugged into suit, makes it simple to set up and use in large scale applications. for preference learning we apply dpo due to its efficiency, low complexity and stable training. ## conclusion suit is an iterative training approach for tod systems, which couples sampling to derive new dialogs, with distant supervision to determine subgoals that impact the final dialog success. this coupling enables suit models to improve inform and success metrics and advance the sota.