Knowledge-Driven Slot Constraints for Goal-Oriented Dialogue Systems (2021)

Abstract: In goal-oriented dialogue systems, users provide information through slot values to achieve specific goals. Practically, some combinations of slot values can be invalid according to external knowledge. For example, a combination of “cheese pizza” (a menu item) and “oreo cookies” (a topping) from an input utterance “Can I order a cheese pizza with oreo cookies on top?” exemplifies such invalid combinations according to the menu of a restaurant business. Traditional dialogue systems allow execution of validation rules as a post-processing step after slots have been filled which can lead to error accumulation. In this paper, we formalize knowledge-driven slot constraints and present a new task of constraint violation detection accompanied with benchmarking data. Then, we propose methods to integrate the external knowledge into the system and model constraint violation detection as an end-to-end classification task and compare it to the traditional rule-based pipeline approach. Experiments on two domains of the MultiDoGO dataset reveal challenges of constraint violation detection and sets the stage for future work and improvements.

## introduction natural language understanding (nlu) is an important component of goal-oriented dialogue systems. the function of nlu is to construct a semantic frame for a user utterance by performing two tasks -intent classification (ic) and slot labelling (sl) @xcite . the former task aims to identify the intent of the user (i.e., an activity or a transaction that the user wants to accomplish), while the latter task extracts attributes of the intent. for example, given an input utterance "please add one xl fries to my order" in figure 1 (a), ic classifies that the user intent is "addtoorder" (adding a new menu item to the order), while sl detects "one", * work performed while at amazon ai "xl", and "fries" as quantity, menuitemsize, and menuitem, respectively. these two tasks, ic & sl, could be performed either independently @xcite @xcite or jointly @xcite @xcite although recent research shows that training jointly generally leads to better results @xcite . to make the recognition of intents and slots more reliable, nlu models require the list of all possible intents and the slots associated to each intent. for instance, the intent show_flights has airline, departure_city, arrival_city, departure_date, and departure_time as its associated slots. practically, each slot has its own type. some types are domain-agnostic such as date for the depar-ture_date, while other types are domain-specific, such as airline for the slot airline. we also refer to the latter category as custom slot types, for which custom lists of valid entities are provided. moreover, slots could be marked as either required (such as departure_city and arrival_city) or optional (such as airline and departure_time). all of these details are usually defined structurally in a single document called a bot schema which guides the conversational flow of the dialogue system @xcite . besides the above details, the dialogue domain may have conditions permitting or forbidding some combinations of slot values. for example, for a book_flight intent which has "singapore airlines" as the airline slot, not all cities are valid destinations where the airlines operate. the nlu may deal with invalid combinations of slot values by just ignoring them, i.e., not detecting them in the sl task. this approach will result in a deteriorated user experience as the users would not know why their attempts to provide slot values are not successful. therefore, we envision these conditions as constraints between slots, and the system should be able to detect constraint violations and request new slot combinations from the users when the violations happen. however, to the best of our knowledge, we have not found any existing work formalizing the constraints between slots nor modeling detection of constraint violations. in this paper, we formally represent the slot constraints which could be integrated into a bot schema and present a new task of constraint violation detection: given a bot schema with constraints, a current utterance, and a conversation history, predict whether the current state of conversation violates any constraints or not and which constraints are violated. after that, we propose three approaches to solve this problem (based on a pipeline approach and an end-to-end approach) and conduct experiments with two domains of the multidogo dataset @xcite augmented with constraint violation labels. by design, the end-to-end approach does not suffer from error accumulation (whereas the pipeline approach does); however, it is more difficult to inject the constraint information into the end-to-end approach. the experimental results reveal challenges of the violation detection task together with room for improvement. overall, the main contributions of this paper are as follows. the remainder of this paper is organized as follows. section 2 explains related work about natural language understanding in dialogue systems as well as entity linking. section 3 presents formal representations of the constraints. section 4 proposes the three approaches we use to detect constraint violations. section 5 explains the created datasets and the experimental results. finally, section 6 concludes the paper. ## constraint representation as constraint violation check must be applied to every state in the conversation, we first define dialogue states as follows. definition 1 a dialogue state d is a tuple (d i , d s ) where d i is an intent and d s is a list of slot-value pairs @xcite . definition 2 an atomic formula f can be written as (s, o, v) where s is a slot variable, v is a list of values, and @xmath0, >, <, ≥, ≤, = , between, regexp, in, not_null} is an operator. a dialogue state d satisfies f if and only if the corresponding slot value s in d s satisfies f . for instance, the dialogue state d in figure 1(a ) satisfies an atomic formula @xmath1, in, ['medium', 'large', 'extra large']). definition 3 a constraint c is a triple (c i , c s , c l ) where (1) c i is a list of intents where the constraint applies, (2) c s is a list of associated slots (s 1 , s 2 , ..., s n ), and (3) c l is a constraint statement defined on c s -a logical formula in disjunctive normal form where each conjunction consists of n atomic formulas that correspond to n slot variables in c s . definition 4 a constraint c is applicable to a dialogue state d if and only if d @xmath2. in other words, a constraint applies to a dialogue state when the dialogue state has an applicable intent and contains all the relevant slot variables. in figure 1 , the constraint c is applicable to the dialogue state d but not applicable to, for instance, @xmath3, {quantity: 1, menuitem: 'fries'}). definition 5 a dialogue state d violates a constraint c if and only if c is applicable to d but d does not satisfy c l . for the running example, d does not violate c because the slot-value pairs {menuitem: 'fries', menuitemsize: 'extra large'} of d satisfies c l . note that, in figure 1(a ), the dialogue state is a result of a single utterance. however, a dialogue state in practice contains the information of the current user turn fused with the dialogue state of the previous turn. so, the objective of the constraint violation detection task is checking whether any constraints defined in the bot schema are violated after the dialogue state is updated with the information of the current turn. ## constraint violation detection we propose three approaches to tackle this problem. the overview is shown in figure 2 . ## conclusions and future work focusing on goal-oriented dialogue systems, we proposed a novel task -slot constraint violation detection -in nlu, together with constraint representation and three approaches to tackle this problem. while the pipeline approaches apply constraints as a post-processing step after ic/sl, the end-toend approach attempts to model constraints inside the nlu. this sets the stage for future research and modeling of slot constraints and knowledge within nlu. in particular, there are several ways to enhance the end-to-end approach. for example, we could perform joint learning of ic, sl, and constraint violation detection to share the learned knowledge among tasks. also, injecting logical constraints into bert is an interesting direction. one way to do so is to translate constraints into violating and non-violating examples (by generating conversations with templates derived from existing training examples) and use them to train bert together with other training examples. in addition, using constraints information, one can control the training data generation and the percentage of data with constraint violations depending on expected user behavior.