Unsupervised Learning of KB Queries in Task-Oriented Dialogs (2021)

Abstract: Task-oriented dialog (TOD) systems often need to formulate knowledge base (KB) queries corresponding to the user intent and use the query results to generate system responses. Existing approaches require dialog datasets to explicitly annotate these KB queries—these annotations can be time consuming, and expensive. In response, we define the novel problems of predicting the KB query and training the dialog agent, without explicit KB query annotation. For query prediction, we propose a reinforcement learning (RL) baseline, which rewards the generation of those queries whose KB results cover the entities mentioned in subsequent dialog. Further analysis reveals that correlation among query attributes in KB can significantly confuse memory augmented policy optimization (MAPO), an existing state of the art RL agent. To address this, we improve the MAPO baseline with simple but important modifications suited to our task. To train the full TOD system for our setting, we propose a pipelined approach: it independently predicts when to make a KB query (query position predictor), then predicts a KB query at the predicted position (query predictor), and uses the results of predicted query in subsequent dialog (next response predictor). Overall, our work proposes first solutions to our novel problem, and our analysis highlights the research challenges in training TOD systems without query annotation.

## introduction task-oriented dialog (tod) systems converse with users to accomplish specific tasks such as restaurant reservation @xcite , movie ticket booking @xcite , or bus enquiry @xcite . in addition to the ability to converse, it is crucial for tod systems to learn to formulate knowledge base (kb) queries based on user needs, and generate responses using the query results. an example tod is shown in figure 1 , where during the conversation (at turn 2), the agent queries the kb based on the user needs, and then suggests the peking restaurant based on the retrieved results. existing end-to-end approaches @xcite @xcite learn to formulate kb queries using manually annotated queries. in real-world scenarios, human agents chat with users on messaging platforms. when the need to query the kb arises, the agent fires the query on a back-end kb application and uses the retrieved results to compose a response back on the messaging platform. the dialogs retrieved from these platforms contain just the user and the agent utterances, but the kb queries typically go undocumented. as existing approaches require kb annotations, they have to be manually annotated, which is expensive and hinders scalability. to eliminate the need for such annotations, we define the novel problem of training a tod system without explicit kb query annotation. a key subtask of such a system is the unsupervised prediction of kb queries. while there is no explicit query annotation, we observe that dialog data still offers weak supervision to induce kb queries-all the entities used by the agent in subsequent dialog should be returned by the correct query. for example, in figure 1 , the correct query should retrieve peking restaurant and its phone number. this suggests a reinforcement learning (rl) formulation for query prediction, not unlike similar tasks in question answering (qa) @xcite and conversational qa @xcite . however, there is one key difference between tod and qa settings-in tod only a few entities from the query results are used in subsequent dialog, whereas in qa all correct answers are provided at training time. this makes defining the reward function more challenging for our setting. our problem is further exacerbated by the issue of correlated attributes. query attributes in tod may exhibit significant correlation, like in the restaurant domain, cuisine and price range are often correlated. for example, most japanese restaurants in a kb may be in expensive price range. as a result, presence and absence of expensive price range in the query could retrieve almost the same set of kb entities and hence similar rewards. this can confuse the weakly supervised query predictor. to counter this issue we present a baseline solution for kb query prediction by extending an existing policy optimization technique, memory augmented policy optimization (mapo; @xcite . experiments show that our proposed modification significantly improves the query prediction accuracy. to train a full tod system without kb query annotation, we propose a pipelined solution. it uses three main components: (1) query position predictor predicts when a query must be made, (2) query predictor predicts the query at the turn predicted by position predictor, and (3) next response predictor generates next utterance based on dialog context, and predicted query's results. we train these components in a curriculum, due to the pipeline nature of the system. we find that overall our system obtains good dialog performance, and also learns to generalize to entities unseen during training. we conclude with novel research challenges highlighted by our paper. in particular, we notice that even fully supervised tod systems (trained with kb query annotation) suffer a significant loss in performance when they are evaluated with the protocol of using their own predicted queries and its query results (instead of using gold queries and gold results) in subsequent dialog. we believe that designing tod systems with high performance under this evaluation protocol is the key next step towards making end-to-end tod systems useful in real applications. we release all our resources for further research-training data, evaluation code, and code for tod systems. foot_1 our work is on task-oriented dialogs and is closely related to the task of semantic parsing. we briefly discuss related work in both areas. our query predictor is an extension of mapo @xcite , which we describe in detail. task oriented dialogs: tod systems are of two main types: traditional spoken dialog systems (sds) and end-to-end tod systems. sds @xcite use hand-crafted states and state annotations on every utterance in the dialogs-a significant human supervision. end-to-end tod systems @xcite @xcite do not require state annotations but just the kb query annotations. there exist approaches @xcite to induce state annotations in sds, but we are the first to induce query annotations in end-toend tod systems. tod systems cannot be learned with just the state annotations, additional state to kb query mapping/annotation is required. but no further annotations are needed to learn tod system when query annotations are available. we build on recent architectures, which use memory networks to store previous utterances and query results, and a generative copy decoder to construct the agent utterances @xcite @xcite . an alternative approach maintains the entire kb in its neural model, bypassing the need for kb queries entirely @xcite . unfortunately, such approaches can only work with small kbs. in contrast, our approach is scalable and does not impose restrictions on kb size. semantic parsing: semantic parsing is the task of mapping natural language text to a logical form (or program) @xcite . to alleviate the need for gold program annotations, weakly supervised approaches have been proposed @xcite @xcite @xcite . these weakly supervised approaches solve two main problems: (1) exploring large search space to find correct logical programs and (2) spurious program problem-where the policy learns an incorrect program that fetches the correct answer. to explore large search space better, @xcite used randomized beam search, @xcite and mapo used systematic search. there exists several approaches to overcome the spurious query problem: crowd sourcing @xcite , spreading the probability mass over multiple reward earning programs @xcite , clustering similar natural language inputs and using their abstract representations @xcite , and using overlap between the text spans in input and the generated programs @xcite @xcite . mapo samples from a buffer of systematically explored reward earning programs based on their likelihood in the current policy @xcite to tackle the spurious program problem. our query prediction approach follows this literature, except in a dialog setting. as our approach is an extension of mapo, we inherit the ability to effectively tackle the two issues. these methods use an rl formalism, in which a logical form (query, in our case) a is predicted by an rl-policy π θ (a|c). here, θ are the parameters of the rl agent, and c is the input, for example, a question (in our case, a dialog context). the policy π θ is trained by maximizing the expected reward: when the search space is large and the rewards are sparse, relying on just the on-policy samples often leads to poor search space exploration. to overcome this, search is added on top of policy samples @xcite . we build on mapo, which uses systematic exploration to identify nonzero reward queries for each training data and stores them in a memory buffer b. the expected reward is computed as a weighted sum of two expectations: one over the queries inside the buffer b and the other over the remaining queries: a randomly initialized policy is likely to assign small probabilities to the queries in the buffer, and hence negligible contribution to the gradient estimation. to ensure queries in the buffer contribute significantly to the gradient estimates, mapo clips π b to α. the modified gradient estimate is experience replay: experience replay @xcite stores past experience in a buffer and reuses that to stabilize training and improve sample efficiency. prioritized experience replay @xcite assigns priorities to the experiences and samples them based on the priorities to efficiently learn using them. mapo and our approach use replay buffers that store non-zero reward queries. these queries include both past experiences and queries identified using systematic search. the sampling strategy is similar to prioritized experience replay, where the priorities are defined by the probability of the query in the current policy. ## task definition & baseline system we first define our novel task of learning tod system using unannotated dialogs -we name it utod. we then describe the kb query predictor in detail. finally, we describe our proposed baseline utod system, which uses the kb query predictor. ## experimental setup we now describe the datasets used, the comparison algorithms and the evaluation metrics for our task. we release all software and datasets from our experimental setup for further research. ## experiments our experiments evaluate three research questions: ## discussion & research challenges we now discuss novel research questions identified in this research. gold select * from kb where @xmath0. utod reinforce select * from table where @xmath1. would you like their phone number utod mapo select * from table where @xmath2. would you like their phone number utod mb-mapo select * from table where @xmath3: performance gap for supervised tod systems when trained in different evaluation settings. can we train an end-to-end differentiable utod system? our proposed approach for utod uses a pipeline of three components trained separately, which can lead to cascading errors. we believe that a single end-to-end neural architecture will likely obtain superior performance. but, this is a technical challenge, since it will require one model to make two discrete decisions: when to query and what to query, complicating the rl problem. can we bridge the gap between atod performance with and without oracle queries at test time? our work exposes a critical weakness of fully supervised tod systems. when evaluated in a setting where the tod system only has access to its own predicted query and that query's results, the overall performance drops drastically. can we design a utod system to handle dialogs that require more than one query per dialog? our current work makes the assumption of a single query per dialog. this is because most supervised tod datasets also make only one query per dialog. given that our task is harder than those, it was important to define it such that existing ml machinery of tod can feasibly learn our task. at the same time, extending the task definition and creating datasets for the multiple queries per dialog case is straightforward. an important future research challenge will be to design an end-to-end dialog system that can handle a variable number of kb queries in a single dialog, without explicit query (or query position) annotation. we believe that this is best studied only after substantial progress on our specific task definition. ## conclusion we define the novel problem of learning tod systems without explicit kb query annotation and the associated subtask of unsupervised prediction of kb queries. we also propose first baseline solutions for these tasks. our best query prediction baseline extends the existing rl approach mapo to include multiple query buffers at training time. we also present a pipeline architecture that trains different components in a curriculum to obtain the final tod system. our detailed evaluation shows that our approaches achieve much better performance than simpler baselines, though there is some gap when compared to supervised approaches. we study the results further to identify research challenges for future research. we will release all resources for use by the research community.