Resource Constrained Dialog Policy Learning Via Differentiable Inductive Logic Programming (2020)

Abstract: Motivated by the needs of resource constrained dialog policy learning, we introduce dialog policy via differentiable inductive logic (DILOG). We explore the tasks of one-shot learning and zero-shot domain transfer with DILOG on SimDial and MultiWoZ. Using a single representative dialog from the restaurant domain, we train DILOG on the SimDial dataset and obtain 99+% in-domain test accuracy. We also show that the trained DILOG zero-shot transfers to all other domains with 99+% accuracy, proving the suitability of DILOG to slot-filling dialogs. We further extend our study to the MultiWoZ dataset achieving 90+% inform and success metrics. We also observe that these metrics are not capturing some of the shortcomings of DILOG in terms of false positives, prompting us to measure an auxiliary Action F1 score. We show that DILOG is 100x more data efficient than state-of-the-art neural approaches on MultiWoZ while achieving similar performance metrics. We conclude with a discussion on the strengths and weaknesses of DILOG.

## introduction to foster research on dialog policy learning for virtual digital assistants, several task-oriented dialog corpora have been introduced in recent years, such as simdial @xcite , multi-woz @xcite , taskmaster @xcite , and schema guided dialog @xcite , to name a few. deep learning approaches, including mixture models @xcite hierarchical encoder/decoder @xcite , reinforcement learning @xcite , and pre-trained language models @xcite @xcite , have significantly advanced dialog policy research in the past few years @xcite , setting new state-of-the-art performance limits. however, collecting annotated data for supervised dialog policy learning is an expensive and timeconsuming process. hence, it is desirable to explore approaches to train dialog policy with limited data and transfer an existing policy with few or even no additional training data to new domains. this practical requirement has motivated the community to research resource-constrained dialog policy learning in the past few decades. researchers have explored approaches including employing grammar constraints for dialog policy @xcite , transfer learning @xcite , or pre-trained language models @xcite . few-shot domain adaptation has been researched since the 2000s @xcite on both end-to-end dialog systems @xcite as well as dialog policy learning @xcite . in a traditional modular dialog system, the dialog policy aims to decide a dialog action given a dialog state, while assuming the tasks of language understanding and generation are handled by other components. under such assumptions, a task-oriented dialog policy mostly follow the slot-filling scheme, which can be described by a set of probabilistic rules. therefore, we hypothesize that dialog policy in this limited sense can be constructed by learning the underlying rules. to this end, we draw upon the recent advances in developing differentiable inductive logical programs (dilp) @xcite ) that use neural architectures to learn almost rule-based policies. we present dilog, an adaptation of dilp to dialog policy learning. briefly, dilog discerns a set of logical rules from the examples by using inductive reasoning. 1 we introduce dilog in section 2. we apply dilog to the simdial dataset @xcite (section 3), and multiwoz dataset @xcite (section 4), showing that on the task of one-shot dialog policy learning and zero-shot domain transfer, dilog outperforms several other neural baselines. finally, section 5 concludes this paper. ## dilog: dialog policy via differentiable inductive logic inductive logic programming (ilp) is a paradigm which derives a hypothesized first-order logic given the background knowledge, positive, and negative examples @xcite . the central component of ilp is known as clauses. a clause is a rule expressed as α ← α 1 , ..., α n , where α is the head atom and α 1 , ..., α n are body atoms. an atom p(t 1 , ..., t m ) is composed of an m-ary predicate p and a tuple of terms t 1 , ..., t m , which can be variables or constants. an atom is ground if it only contains constants. for example, the following clause defines when to perform the action of confirmation: dilog is based on dilp with several modifications: (1) we include an option of adding ℓ 1 or ℓ 2 regularizers to the weight matrix of the clauses to improve generalization. (2) we allow adding clauses as background knowledge, so as to enable continual learning using pre-learned rules. (3) we use elementwise max instead of probabilistic sum as the amalgamate function to update the valuation vector. as probabilistic sum is accumulating the valuation at each step, it is easier to get into a local optimum when the inference steps are long. (4) in dilp, a problem is defined as (l, b, p, n ), where l is the language frame to generate potential clauses, b is the background knowledge, p is the positive examples, and n is the negative examples. in this definition, all positive and negative examples are grounded on the same set of constants c. as the number of clauses scales quadratically with the number of constants, it can be computationally expensive when the sample size is large. comparatively, we define a problem as (l, s), where s is the set of samples, and each @xmath0, p, n , c). this modification allows each sample to define its own set of constants, which makes it computationally tractable under the setting of dialog system where each dialog consists of multiple turns. in this paper, we demonstrate the application of dilog on two tasks in dialog policy learning: oneshot learning and zero-shot domain transfer. in a modular dialog system, which consists of automatic speech recognition, natural language understanding, dialog state tracking (dst), dialog manager, and 1 inductive reasoning tries to summarize general principles from special cases. for example, the fact "cars a, b, and c drive on the right side of the road" induces that "all cars drive on the right side of the road." each of the rules can be manually inspected and understood. this feature is desirable in industrial settings where interpretability and debuggability are key considerations. ## dilog on the simdial dataset simdial @xcite ) is a multi-domain dialog generator that can generate conversations in domains including restaurant, movie, bus, and weather. each domain is defined by a collection of meta data including user slots and systems slots. ignoring the actions of greeting and goodbye, possible user actions are inform or request, while system actions include inform, request, and database query. we use simdial to generate clean dialogs from all four domains, among which a single representative dialog (that contains all user and system actions) from the restaurant domain is used for training and 500 dialogs from each other domains are used as the test set. to adapt the simdial problem to dilog, the delexicalized foot_0 state and actions are converted into the form of atoms, whose predicate is the action or state, and term is the slot. for example, request(loc) denotes the action of requesting the location, while unknown(price) indicates that the price is unknown to the system. each turn is converted to a sample s, where the background knowledge b is the combination of user actions and the belief state, and the positive examples p are the system actions. dilog learns a mapping (a set of clauses) from the background b to the positive examples p. see note that the four domains have different slots, and during training time, the model is only aware of the slots in the training set. during test time, the learned rules are directly applied to the converted samples in the test set. additionally, to demonstrate continual learning, we add the pre-trained basic relationships of all and member to the background knowledge. all identifies that all items in a list satisfy some property, and member indicates that an item belongs to a list. see appendix c for a more detailed description of the training process. figure 1 : the evaluation results on the test set of simdial comparing dilog with mlp and zsdg @xcite . the in-domain stands for the performance on the restaurant domain (500 samples, standard error: ±2.2%). out-of-domain one is the average performance on movie, bus, and weather (1500 samples, standard error: ±1.3%). see table a1 (appendix) for the per-domain results. the plot shows the zero-shot performance on movie and weather. -1000 denotes that the model is trained with 1000 dialogs from the restaurant domain. ## dilog on the multiwoz dataset multiwoz 2.0 @xcite is a large-scale human to human english dialog dataset, which consists of dialogs on domains including restaurant, hotel, attraction, train, and taxi. it also includes dialogs that span multiple domains. the system response in multiwoz dataset is stochastic, as human clerks have the freedom to choose from the possible actions. the actions are annotated by human annotators, which are noisy as well. these features make multiwoz a more challenging task compared with simdial. we used a variant of multiwoz 2.0 provided by convlab @xcite , which has annotated user actions. the process of adapting the multiwoz dataset to dilog is similar to that of simdial (please refer to appendix d for the details). for training dilog, we selected a single representative dialog whose system action contains all possible intents (inform, request, offerbooked, nooffer) from the restaurant figure 2 : the evaluation results on the test set of multiwoz: dilog is compared with mlp and damd @xcite . note that damd applied its own heuristics in data preprocessing, so the act. f1 of damd is not directly comparable with others. damd with the same preprocessing steps as others has a worse performance (table a3 ). the inform and success metrics are not affected by preprocessing and therefore comparable. all stands for the performance on the all domains in the test set (1000 samples, standard error: ±1.6%). out-of-domain is the average performance of hotel, attraction, train, and taxi (1480 samples, standard error: ±1.3%). see table a3 (appendix) for the per-domain results. the plot shows the performance on two example domains of hotel and attraction. -am denotes adding the missing slots in nlg, and -526 denotes the model trained with all 526 dialogs from the restaurant domain. we compare dilog with two baselines: damd @xcite is the state-of-the-art model with dialog action prediction, and mlp is a naive baseline. we calculate the following metrics to evaluate the quality of the models: inform is a per diaog metric that measures whether the system finally provides a correct entity, success measures whether all the requested information are provided, and action f1 is a per turn metric that checks if the predicted dialog action (both intent and entities) matches the ground truth. we use a template-based nlg to convert dialog actions predicted by dilog and mlp to compute inform and success on generated utterances. there are three slots (postcode, phone, address) that only appear in the system actions but not dialog states. we manually add an option to the template-based nlg to inform those slots whenever an entity is informed, and denote that variant as -am, which significantly improves the success rate. we do not compare the -am variant against other methods. the results are shown in figure 2 . as can be seen, dilog outperforms other models on the overall test set, which shows dilog is capable of doing one-shot learning and zero-shot domain transfer even under noisy data. compared with the mlp policy trained with all the 526 samples in the restaurant domain (denoted as -526), dilog trained with one sample has a higher inform/success rate but lower action f1 score. noticeably, dilog with the addition of missing slots in nlg achieves 91.40 and 90.20 inform and success rate overall, which is higher than the state-of-the-art damd. 4 this shows inform and success are incomplete metrics which do not penalize false positives. hence, we add action f1 as an additional metric to complement inform and success. note that action f1 cannot solely capture the performance either, since there may be multiple possible true actions in a given state. ## conclusion in this paper, we introduce dilog for resource constrained dialog policy learning and zero-shot domain transfer. empirically, we demonstrate that dilog outperforms strong neural baselines on simdial and multiwoz datasets, while offering interpretability. we also provide an intuitive explanation on why dilog shows these features. on the other hand, the dilog framework has certain weaknesses, one being that it is computationally expensive when the template space gets large, where distributed training will be desired. further, the program template used to generate all possible clauses needs to be handcrafted, which is not a straightforward process. another disadvantage is that real-valued inputs, such as confidence scores cannot be taken into account automatically. the future work would naturally focus on solving these shortcomings. one way might be to jointly, in a multi-task setting, predict the templates or meta-learn them @xcite .