Unsupervised End-to-End Task-Oriented Dialogue with LLM s: The Power of the Noisy Channel (2024)

Abstract: Training task-oriented dialogue systems typically requires turn-level annotations for interacting with their APIs: e.g. a dialogue state and the system actions taken at each step. These annotations can be costly to produce, error-prone, and require both domain and annotation expertise. With advances in LLMs, we hypothesize that unlabeled data and a schema definition are sufficient for building a working task-oriented dialogue system, completely unsupervised. We consider a novel unsupervised setting of only (1) a well-defined API schema (2) a set of unlabeled dialogues between a user and agent. We propose an innovative approach using expectation-maximization (EM) that infers turn-level annotations as latent variables using a noisy channel model to build an end-to-end dialogue agent. Evaluating our approach on the MultiWOZ benchmark, our method more than doubles the dialogue success rate of a strong GPT-3.5 baseline.

## introduction task-oriented dialogue systems, which use apis to complete tasks on behalf of users, have been a longstanding challenge within conversational ai. recent advances in large language models (llms) have further stimulated interest in task-oriented systems and llms which can use apis as tools. to facilitate api use, successful task-oriented dialogue systems usually employ a modular approach: predicting a dialogue state which includes arguments to api calls, and dialogue acts for planning an appropriate response, before finally producing a natural language reply. training such systems typically requires expert annotation of these structured intermediates for every dialogue turn. even in settings where human-human dialogues are abundantly available, the high cost and expertise re- figure 1 : an overview of our unsupervised dialogue problem. we assume 1) unlabeled goal-oriented dialogues between a user and agent and 2) a well-defined schema s with apis suitable for fulfilling goals. we infer the unseen interactions between the agent and api, and use this to produce an end-to-end dialogue agent. recent work has shown that llms can accomplish a broad set of useful tasks without any structured labels for a task @xcite . these include 'zero-shot' approaches to task-oriented dialogue sub-tasks such as dialogue state tracking (dst) @xcite @xcite , intent detection @xcite , grounded response generation @xcite , and even zero-shot end-to-end dialogue systems @xcite . still, existing approaches generally do not perform well enough for real-world use, and none are able to make effective use of in-domain unlabeled dialogues. in this work, we ask whether we can use existing unlabeled dialogues (without any labels or api calls annotated) along with an api specification, to build a working dialogue agent, without needing an expert to annotate data. this addresses a common real-world scenario. many high value dialogue tasks are currently carried out by human agents, who interface a user with some software system. these conversations can be recorded and transcribed, and the api(s) supporting the agent typically have well-formed specifications. however, annotating the api calls and system acts needed for aligning the two is time consuming and requires annotation expertise. in lieu of this, 'zeroshot' systems have been proposed, but these still require an expert to annotate a 'formatting example' @xcite , or a more detailed 'policy skeleton' @xcite . we instead propose the following setting: we assume an api schema definition s, and plenty of available human-human dialogues in natural language, but no annotations on these dialogues (figure 1 ). we demonstrate that one can develop a conversational agent for the api schema in this setting without any assistance from an expert annotator. our contributions are as follows: ## preliminaries a task-oriented dialogue consists of turns of utterances between a user and an agent which interfaces the user with a programmable system or api to accomplish a task. typically the system response utterance follows the user's utterance. we denote u t as the user's utterance at turn t, and r t as the system's response. we assume the apis supported by the system are defined in a schema s, which gives names and descriptions for all arguments supported in each api, as well as the possible values any categorical arguments may take @xcite . this is analogous to standardized formats for api documentation, many of which could be easily converted to a schema definition. task-oriented systems require some method for interacting with the apis in s. modular approaches use a dialogue state tracking (dst) module, which predicts a belief state b t : a collection of arguments to api call(s) needed to satisfy the user's goal. a belief state is commonly represented with a set of slot-value pairs: for example, if a user says 'i'm looking for a restaurant south of town', a dst system might produce the belief state {(restaurant-area, south)}, which can be used to query a restaurant api. we assume zero labeled belief states and infer them from unlabeled dialogues using the space of possible states supported by the schema definition s. we also make use of system dialogue acts to structure our agent's communicative intents with a policy module. given a dialogue state and context for a turn t, the policy predicts set of dialogue acts to be communicated in the system response r t . for instance, the policy might determine that we should ask the user to narrow their search to a price range: a @xmath0. an appropriate system response might be: "sure, are you looking for a particular price range?" like belief states, we assume zero supervised examples of a t and infer them from unlabeled dialogues. ## method overview we treat the turn-level labels needed for training an end-to-end dialogue system as latent variables, and infer them from unlabeled dialogues. we assume only the fully-lexicalized sequence of user and system utterances u 1 , r 1 , ...u t , r t , and the schema s defining the system's capabilities, which defines the space of valid dialogue state and act labels. importantly, our prompts are automatically generated from the api schema. in § 4, we outline our noisy-channel prompting method for inferring the turn-level labels necessary for training our dialogue agent. we give an overview of the latent variables we infer in figure 2 . we assume we cannot query the apis or observe user: i need a hotel in the east user: cheap, with wi-fi please system: i found 15, what price range? system: how about city roomz? system: i found [choices], what price range? system: how about [name] ? inferred delex. agent.find_hotel( @xmath1.find_hotel( ... @xmath2, @xmath3: { + 'area': 'east' }} {'hotel': { + 'price': 'cheap', + 'internet': 'yes' }} dialogue state change inform(@xmath4, request(price) offer( @xmath5. acts observed dialogue figure 2 : an overview of the latent variables annotated in our unsupervised labeling process which are used to train the dialogue model. our dst module ( §4.1) infers the api call(s) with arguments at each turn, from which we can derive the dialogue state change. our dat or act tagging module ( §4.2) predicts the dialogue acts communicated in the observed system response, which can be used to infer de-lexicalized responses for training a response generator. ## inferring latents via noisy channel in this section, we present our method for inferring latent annotations for the dialogue states b 1 ...b t and dialogue acts a 1 ...a t for each dialogue turn t given only the unlabeled user and system utterances (u 1 , r 1 , u 2 , r 2 , ...u t , r t ). to do this, we devise a noisy-channel prompting approach for dst and dialogue act tagging (dat) using starcoder @xcite , a code-based llm. first, we use a text-to-code prompt to infer the api call(s) made by the system in each dialogue, and build the dialogue state from inferred api call arguments ( § 4.1). we use a similar text-to-code prompt to infer the latent act(s) communicated in each agent response, so that we can reverse-engineer an agent's policy ( § 4.2). for both tasks, we find much better performance when re-ranking latent predictions according to a noisy-channel model, in which we condition the observed utterance on a predicted latent in a codeto-text prompt ( § 4.3). finally, we leverage the in-context learning ability of llms by re-using our predictions as exemplars ( §4.4). given these initial pseudo-labels, we iteratively improve their quality using hard-em @xcite ( § 4.5). ## end-to-end system following @xcite , we utilize a multi-task fine-tuning method for training a single llm as a complete dialogue system, consisting of a dialogue state tracker, policy, and response generator. dst for the dst sub-task, we again use both 'direct' and 'channel' (prompt, completion) pairs. this allows us to use the same noisy-channel inference method presented in § 4. policy for the policy sub-task, we use a text-tocode prompt where we simply condition on the @xmath6: h @xmath7, r t-2 , u t-1 , r t-1 , u t ). the completion is the current turn's system acts a t , which will be used to ground the next response r t . we do not use a noisy-channel variant for policy, and greedily decode an act prediction at inference time: response generation for response generation, we condition on the turn's observed system and user utterances (r t-1 , u t ) and our policy's act prediction ât ). the completion is the observed system response r t . we also do not use a noisy-channel variant for response generation, and greedily decode the response: following prior works, we predict delexicalized responses, where values for slots in the system response are replaced with placeholders for the slot name. for example, instead of generating "the phone number for acorn guest house is 555-5309" directly, we would predict "the phone number for the [hotel_name] is [hotel_phone]", where values could be filled in. importantly, we never presume access to gold delexicalized responses. instead, we use our predicted acts, e.g. "inform(@xmath8, @xmath9. end-to-end training our approach to end-toend training is as follows. we first derive (prompt, completion) pairs for each module (dst, policy and response generation), including 'channel' prompts for dst. to improve training efficiency, we shorten each prompt by removing in-context examples and function definitions from the in-context learning setting. we then combine and shuffle these pairs into a single training set for joint fine-tuning and up-sample the 'channel' prompts for dst using the same 2:1 ratio. using this training set, we fine-tune starcoder 3b using cross-entropy loss and adamw with default hyperparameters. ## experiments we conduct unsupervised end-to-end dialogue (e2e) and dialogue state tracking (dst) experiments on the multiwoz 2.2 dataset @xcite , containing over ten thousand multi-domain task-oriented dialogues crowd-sourced in a wizard-of-oz setup. we use the fully lexicalized, unlabeled dialogues from the training set to build our system, and evaluate on the test set. first, we demonstrate the value of our approach in an end-to-end dialogue evaluation, following prior works on task-oriented dialogue ( § 6.1). second, we conduct a dialogue state tracking evaluation to more carefully evaluate the quality of our pseudo-annotations ( § 6.2). ## contamination analysis evaluation of unsupervised methods that use llms has the potential issue of task contamination, where supervised examples are seen in pretraining data @xcite . inclusion of supervised examples of the task in llm pretraining data would render the model no longer unsupervised and the evaluation potentially biased: tasks for which the training data has been seen may have a higher performance than truly unsupervised tasks. to address this issue, we quantify the presence of contamination in llm pre-training data and estimate its potential impact on our results. fortunately, the complete pre-training corpus for star-coder is publicly available for analysis. 7 we conduct an exhaustive search for supervised pairs of our dialogue subtasks in the starcoder pretraining data using a semi-automated search with 6 for the rank vs. frequency analysis, we consider the dialogue act for a turn to be the set of acts and slots (but not values) used by the system. for example the act "inform(@xmath10 we also find that less than half of the turns are authentic for either task, and find a number of them derive from github issues discussing problems with dialogue simulators. additionally, we estimate the degree to which the contamination we discover could exaggerate expected performance of our method on an unseen schema, by using contaminated (x, y) pairs as incontext examples. 10 in semi-supervised tod some works propose semi-supervised approaches to end-to-end taskoriented dialogue. @xcite propose an end-to-end sequence-to-sequence model where the dialogue state is a latent variable. @xcite adapt this approach for use with pre-trained language models, fine-tuning gpt-2. while successful, these approaches require a non-trivial amount of supervised data. @xcite learn latent dialogue acts using variational recurrent neural networks without turn level dialogue state labels, but still require observed db/api calls and responses. other semi-supervised works also evaluate their method in an unsupervised setting @xcite . however, these works also assume delexicalized training dialogues, which requires ground-truth annotation and gives a form a supervision to the model. noisy channel and re-ranking methods a few previous works have utilized noisy channel methods for task-oriented dialogue or prompting methods. @xcite pre-train a noisy channel for task-oriented dialogues as a sequence to sequence model, requiring significant labeled training data. @xcite propose noisy channel prompting for few-shot classification, which inspires our generalization to the generative setting. ## conclusion we introduce the first (to our knowledge) approach for building a working task-oriented dialogue system with large language models using only unlabled dialogues and an api schema. the approach leverages an llm and expectation-maximization to infer missing labels as latent variables from unlabeled dialogues. in ablations, we find that a noisy channel approach vastly improves performance, and explore some possible explanations for this in an error analysis. this work advances the stateof-the-art for using llms to build task oriented dialogue systems with minimal annotations. future work can consider applying our method for inferring structured representations from natural text to other nlp tasks.