Logical Reasoning for Task Oriented Dialogue Systems (2022)

Abstract: In recent years, large pretrained models have been used in dialogue systems to improve successful task completion rates. However, lack of reasoning capabilities of dialogue platforms make it difficult to provide relevant and fluent responses, unless the designers of a conversational experience spend a considerable amount of time implementing these capabilities in external rule based modules. In this work, we propose a novel method to fine-tune pretrained transformer models such as Roberta and T5, to reason over a set of facts in a given dialogue context. Our method includes a synthetic data generation mechanism which helps the model learn logical relations, such as comparison between list of numerical values, inverse relations (and negation), inclusion and exclusion for categorical attributes, and application of a combination of attributes over both numerical and categorical values, and spoken form for numerical values, without need for additional training data. We show that the transformer based model can perform logical reasoning to answer questions when the dialogue context contains all the required information, otherwise it is able to extract appropriate constraints to pass to downstream components (e.g. a knowledge base) when partial information is available. We observe that transformer based models such as UnifiedQA-T5 can be fine-tuned to perform logical reasoning (such as numerical and categorical attributes’ comparison) over attributes seen at training time (e.g., accuracy of 90%+ for comparison of smaller than kmax=5 values over heldout test dataset).

## introduction logical reasoning is an important aspect of human thinking and communication. humans reason over beliefs, preferences, time, facts, and other contextual information to achieve complex tasks, derive meaning, and analyze emotions. current * work done while at amazon alexa ai. task-oriented dialogue systems, however, only support very limited forms of logical reasoning. more specifically, although reasoning ability has been investigated as part of chatbots @xcite and question-answering systems @xcite , in many task-oriented dialogue systems today, the reasoning is mainly focused on determining which slot values are still unknown to the system but are required and elicit them @xcite . however, in realistic task-oriented dialogues, logical reasoning is required to understand the user's request, ask questions that help address the user's task successfully and minimize asking irrelevant questions. the lack of robust, generalizable reasoning capabilities for dialogue systems, requires developers of the system to spend a considerable amount of time implementing these capabilities in external, rule-based and domain spe-cific components. this leads to a poor user experience requiring users to often correct the system's understanding, repeat themselves to ask the same question in different ways, restart the conversation when the system fails to recover from a 'dead-end', or even change their goal. in this work, we propose to build on recent advances in research on logical reasoning and deep networks (e.g., @xcite @xcite @xcite to bring reasoning capabilities to taskoriented dialogue systems. our primary focus in this work is on mechanisms by which logical reasoning can be learned and used in conversational systems. in this direction, we propose a novel deep learning method to fine-tune pretrained models to reason over numerical and categorical attributes in the dialogue context and present an architecture for the integration of this model in task-oriented dialogue systems. our objective is for the model to do logical reasoning to respond to queries from the dialogue context when it has all the required information available in the dialogue context without additional external logic (e.g., "add the most popular to my cart" in figure 1 ), extract constraints and inform downstream components when it only has partial context (e.g., "actually i'm allergic to berries. find something cheaper and with vanilla flavor" in figure 1 , where cheaper means cheaper than what was shown so far), and not provide an answer when it does not have any relevant information and delegate to the dialogue policy to determine the next action. we specifically choose to fine-tune transformers since these models operate on language directly, do not impose any structure on the reasoning process @xcite , and we can leverage the knowledge and diversity of language that the pretrained models have already learned. furthermore, @xcite recently showed that these approaches can outperform neuro-symbolic methods. our approach is similar to recent works on using transformers as soft reasoners @xcite . however, compared to these methods, we focus on use cases relevant to conversational systems and our model goes beyond predicting a true/false response to directly predicting the answer when the model has the information or extract constraints when it has partial information. in this direction, we report experimental results that show using our training method transformers can learn to reason over numerical and categorical attributes in the dialogue context. note that although we use transformers for our experiments, our proposed method can be used to generate data and train any other seq2seq model for the same task and be integrated with any dialogue system in a similar manner. furthermore, our proposed method is different from questionanswering or machine reading comprehension in that we are not looking for an answer in a specific passage; rather, we want the model to reason over facts in the dialogue context to draw parallels and conclusions to inform decision making, similar to how humans reason over a multi-turn conversation. ## related work the approaches for integrating reasoning with deep networks can be categorized into the following. reasoning after semantic parsing these approaches convert utterances to a semantic representation and feed it to a set of rules or a formal reasoner for reasoning. for example, @xcite provide examples where given a natural language utterance and context in the form of a relational database, the system first converts the natural language utterance to a sql query that is then executed using standard sql grammar to retrieve the answer. this is also similar in approach to how some teams that participated in the wikisql task @xcite developed natural language interfaces for relational databases. however, writing and maintaining rules is not scalable especially as more complex types of reasoning become needed. the data annotation itself becomes hard to manage efficiently as more functionalities need to be supported. furthermore, deep semantic parsing and reliably extracting attributes and relations and operating on multi-sentence input remains a challenge. @xcite propose to integrate a differentiable maximum satisfiability solver into the loop of larger deep learning systems, and use this approach to successfully learn logical structures such as the rules of sudoku. previous works have shown that temporal reasoning can be modeled as a propositional satisfiability problem @xcite ; however, generalizability to other types of reasoning needs further investigation. although covering a rich class of problems, these approaches impose a structure on the reasoning problem @xcite , i.e., learning of logical structure specifically as expressed by satisfiability problems. ## problem statement task-oriented dialogue systems use a natural language understanding component to extract semantic meaning from the user utterance, and elicit constraints from users to understand their goals in order to provide information, perform a task or provide options and alternatives for users to choose from, retrieved from external knowledge sources (e.g, through api calls). as such, we focus on reasoning over tasks and recommended items in the dialogue which are typically characterized by different attributes, for example, movie names and show-times for a ticket booking scenario. these systems rely on such representations to answer user queries such as "at what time is vertigo playing?" by performing api calls (e.g. searchtime(@xmath0, @xmath1:30-2:30 pm, 3-5 pm], @xmath2. the required information is then returned to the user in natural language (e.g. vertigo is playing today from 12.30 to 2.30 pm and from 3 to 5 pm.). however, in most currently available task-oriented dialogue systems if the user said next "book me the earliest one," although this information is already available to the system from the previous api call, given the lack of reasoning abilities the system would either not support such queries, or it would have to make an additional independent api call (e.g., searchearliesttime(@xmath3, @xmath4, creating redundant latency in the response and requiring the developer of the system to add apis/rules to handle these use cases. given the above description, our objective is to train a model to learn how to reason over the information provided in the context. we assume the following scenarios for each user utterance: ## method in this work we propose to fine-tune transformers to learn logical reasoning over dialogue context in the form of natural language sentences, bypassing a formal representation and showing such reasoning over language is learnable. ## experiments we showcase our proposed methodology in the context of a dialogue system for a shopping assistant (see appendix a for an example interaction). we use an ontology for data generation which consists of item-type (e.g. yogurt) and item-name ("greek yogurt anisakis") and each item is characterized by two numerical attributes price and rating, and two categorical attributes diet and flavor. this choice of attributes can help us explore and assess the model's performance based on attribute's characteristics. we consider two settings for assessing the logical reasoning capability of transformer models. in the first setting, we fine-tune roberta-base @xcite with a training dataset generated for reasoning using only numerical attributes. in this setting, we only focus on true/false prediction for each query q given the facts provided in the context c. the objective of this experiment is to understand whether transformer models can learn to reason over numerical attributes. in the second setting, we use a t5 model @xcite fine-tuned for the unifiedqa data @xcite , to predict a sequence similar to one given in ## conclusions in this paper, we proposed an architecture for the integration of a reasoning model in task-oriented dialogue systems. we formulated the problem as a sequence prediction problem given a user query and context, and presented an approach for generating data and fine-tuning generative models to reason over a set of facts in the dialogue context. we demonstrated our approach for a shopping assistant and reported experimental results for different formulations of the problem. we showed that these models can learn to do logical reasoning to 1) answer questions from the dialogue context when all the information is available, 2) extract constraints when partial information is available, and 3) delegate to the dialogue policy when no reasoning is required. for future work, we plan to investigate the application of our method to other reasoning tasks (e.g., temporal and spatial reasoning). we also plan to experiment with additional models to compare performances with the ones presented in this work, to further investigate the complexity of the task at hand. moreover. we would like to test our models on more challenging and realistic testsets, for example by adding noise in the current synthetic data or by performing a data collection with human annotators. furthermore, we plan to explore how logical reasoning can be used to disambiguate with the user when multiple conclusions can be made. one of the directions that currently we are working on is to create realistic (human based) conversations with logical reasoning use cases during the interactions with the dialog systems. this type of dataset can help us to evaluate the proposed idea with higher degree of confidence. since no matter how much one spends time on generating synthetic datasets, there will always be some uncontrolled structures introduced by design of data simulation mechanisms that can corrupt the fair evaluation of deep neural network models and their learning process. however, we believe the true/false scenarios in our current study are less prone to this type of issues and are quite helpful in understating of reasoning capabilities such as negation, numerical comparison, or inclusion/exclusion of categorical values of our proposed algorithm, since model needs to learn the reasoning procedure. in other words, the only way to come up with the right prediction by model is to apply the underlying reasoning procedure to formulate the output true/false results. we will consider: a) better algorithms for generating training data, and b) more realistic general purpose possibly human in the loop training data to make the data generation more general and less domain specific, for future exploration.