Learning from Mistakes: Combining Ontologies via Self-Training for Dialogue Generation (2020)

Abstract: Natural language generators (NLGs) for task-oriented dialogue typically take a meaning representation (MR) as input, and are trained end-to-end with a corpus of MR/utterance pairs, where the MRs cover a specific set of dialogue acts and domain attributes. Creation of such datasets is labor intensive and time consuming. Therefore, dialogue systems for new domain ontologies would benefit from using data for pre-existing ontologies. Here we explore, for the first time, whether it is possible to train an NLG for a new larger ontology using existing training sets for the restaurant domain, where each set is based on a different ontology. We create a new, larger combined ontology, and then train an NLG to produce utterances covering it. For example, if one dataset has attributes for family friendly and rating information, and the other has attributes for decor and service, our aim is an NLG for the combined ontology that can produce utterances that realize values for family friendly, rating, decor and service. Initial experiments with a baseline neural sequence-to-sequence model show that this task is surprisingly challenging. We then develop a novel self-training method that identifies (errorful) model outputs, automatically constructs a corrected MR input to form a new (MR, utterance) training pair, and then repeatedly adds these new instances back into the training data. We then test the resulting model on a new test set. The result is a self-trained model whose performance is an absolute 75.4% improvement over the baseline model. We also report a human qualitative evaluation of the final model showing that it achieves high naturalness, semantic coherence and grammaticality.

## introduction natural language generators (nlgs) for taskoriented dialogue take meaning representations * work done prior to joining amazon. here we experiment with one version of this task by building a new domain ontology based on combining two existing ontologies, and utilizing their training data. each dataset is based on a different domain ontology in the restaurant domain, with novel attributes and dialogue acts not seen in the other dataset, e.g. only one has attributes representing family friendly and rating information, and only one has attributes for decor and service. our aim is an nlg engine that can realize utterances for the extended combined ontology not seen in the training data, e.g. for mrs that specify values for family friendly, rating, decor and service. figure 1 illustrates this task. example e1 is from a training set referred to as nyc, from previous work on controllable sentence planning in nlg @xcite , while e2 is from the e2e nlg shared task @xcite . as we describe in detail in section 2, e1 and e2 are based on two distinct ontologies. example e3 illustrates the task addressed in this paper: we create a test set of novel mrs for the combined ontology, and train a model to generate high quality outputs where individual sentences realize attributes from both ontologies. to our knowledge, this is a completely novel task. while it is common practice in nlg to construct test sets of mrs that realize attribute combinations not seen in training, initial experiments [restaurant] is the best because it has excellent service and atmosphere. it is a [restaurant-type] offering excellent food in [area] near [point-of-interest] with a high customer rating, but it is expensive. figure 1: e1 and e2 illustrate training instances from the two source datasets e2e and nyc. e2e attributes are represented in blue and nyc is in red. some attributes are shared between both sources: here the unique dialogue acts and attributes for each source are underlined in e1 and e2. e3 illustrates an mr from the target test set that we dub com. all the mrs in com combine dialogue acts and attributes from e2e and nyc. there is no training data corresponding to e3. the mrs illustrate how some attribute values, e.g. restaurant name, point-of-interest, are delexicalized to improve generalization. we describe below how we create a test setthat we call com -of combined mrs to test different methods for creating such an nlg. a baseline sequence-to-sequence nlg model has a slot error rate (ser) of .45 and only produces semantically perfect outputs 3.5% of the time. to improve performance, we experiment with three different ways of conditioning the model by incorporating side constraints that encode the source of the attributes in the mr @xcite . however, this only increases the proportion of semantically perfect model outputs from 3.5% to 5.5% (section 4.1). we then propose and motivate a novel selftraining method that greatly improves performance by learning from the model mistakes. an error analysis shows that the models do produce many combined outputs, but with errorful semantics. we develop a rule-based text-to-meaning semantic extractor that automatically creates novel correct mr/text training instances from errorful model outputs, and use these in self-training experiments, thus learning from our mistakes (section 4.2). we validate the text-to-meaning extractor with a human evaluation. we find that a model trained with this process produces sers of only .03, and semantically perfect outputs 81% of the time (a 75.4 percent improvement). a human evaluation shows that these outputs are also natural, coherent and grammatical. our contributions are: we start in section 2 by defining the task in more detail, describe our models and metrics in section 3, and results in section 4. we discuss related work throughout the paper where it is most relevant and in the conclusion in section 5. ## ontology merging and data curation we start with two existing datasets, nyc and e2e, representing different ontologies for the restaurant domain. the nyc dataset consists of 38k utterances @xcite , based on a restaurant ontology used by zagat @xcite ). 1 the e2e dataset consists of 47k utterances distributed for the e2e generation challenge @xcite ). foot_1 each dataset consists of pairs of reference utterances and meaning representations (mrs). figure 1 shows sample mrs for each source and corresponding training instances as e1 and e2. ontology merging. we first make a new combined ontology onto-com by merging nyc and e2e. attributes, dialogue acts, and sample values for e2e and nyc are illustrated on the left-hand side of figure 2 , and the result of merging them to create the new ontology is on the right-hand side of figure 2 . since there are only 8 attributes in each source dataset, we developed a script by hand that maps the mrs from each source into the onto-com ontology. as figure 2 shows, both datasets have the in-form dialogue act, and include the attributes name, cuisine, location, and price after mapping. the unique attributes for the nyc ontology are scalar ratings for service, food quality and decor. the nyc dataset also has the recommend dialogue act, seen in e1 in figure 1 . the unique attributes of the e2e ontology are customer rating, eat type ("coffee shop"), near and family friendly. training data. given the combined ontology onto-com, we then map the training data for both e2e and nyc into onto-com by relabelling the mrs to have consistent names for shared attributes as illustrated in figure 2 . we create a balanced training set of ∼77k from the two original datasets by combining all nyc references with a random same-size sample of e2e references. test set. we then manually create a test set, com, consisting of 3040 mrs based on the new combined ontology onto-com. each test mr must have at least one attribute from e2e and one attribute from nyc so that it combines attributes from both sources: these mrs provide combinations never seen in training. 3 example e3 in figure 1 provides an example test mr. the procedure for creating the test set ensures that the length and complexity of the test set are systematically varied, with lengths normally distributed and ranging from 3 to 10 attributes. recommendations only occur in the nyc training data, and they increase both semantic and syntactic complexity, with longer utterances that use the discourse relation of jus-tification @xcite , e.g. babbo is the best because it has excellent food. we hypothesize that recommendations may be more challenging to combine across domains, so we vary mr complexity by including the recommend dialogue act in half the test references. we show in section 4 that the length and complexity of the mrs is an important factor in the performance of the trained models. ## experimental overview and methods given the training and test sets for the combined ontology in section 2, we test 4 different neural model architectures and present results in section 4.1. we then propose a a novel self-training method, and present results in section 4.2. these experiments rely on the model architectures presented here in section 3.1, and the text-to-meaning semantic extractor and performance metrics in section 3.2. ## results we run two sets of experiments. we first run all of the nlg models described in section 3.1 on the com test set, and automatically calculate ser and perfect% as described in section 3.2. we report these results in section 4.1. section 4.2 motivates and describes the self-training method and presents the results, resulting in final models that generate semantically perfect outputs 83% of the time. ## conclusion this paper presents the first experiments on training an nlg for an extended domain ontology by re-using existing within-domain training data. we show that we can combine two training datasets for the restaurant domain, that have different ontologies, and generate output that combines attributes from both sources, by applying a combination of neural supervision and a novel self-training method. while it is common practice to construct test sets with unseen attribute combinations, we know of no prior work based on constructing a new combined ontology. our experiments show that the task is surprisingly adversarial, consistent with recent work suggesting that neural models often fail to generalize @xcite @xcite . work on domain transfer shares similar goals to the experiments presented here @xcite , but these methods do not produce nlg outputs that integrate attributes from two different sources into the same sentence. our final results show that the ability of our self-training method to automatically construct new training instances results in high quality natural, coherent and grammatical outputs with high semantic accuracy. in future, we hope to generalize our novel selftraining method to build an nlg that can combine two distinct domains, e.g. hotels or movies combined with restaurants in multi-domain dialogue @xcite @xcite @xcite . ideally systems that cover multiple domains should be able to produce utterances that seamlessly integrate both domains, if data exists for each domain independently. however, there may be additional challenges in such combinations. our results require the initial neural models to generate some combined outputs. it is not clear whether there are some aspects of our experimental setup that facilitate this, e.g. it may require some attributes to be shared across the two initial ontologies, or some shared vocabulary. thus it is possible that initial models for two more distinct domains may not produce any combined outputs, and it may be necessary to seed the self-training experiments with a small number of combined training instances. we leave these issues to future work. in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2153-2162. tsung-hsien wen, milica gašić, nikola mrkšić, lina m rojas-barahona, pei-hao su, david vandyke, and steve young. 2016. multi-domain neural network language generation for spoken dialogue systems. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 120-129. tsung-hsien wen, milica gasic, nikola mrksic, pei-hao su, david vandyke, and steve young. 2015. semantically conditioned lstm-based natural language generation for spoken dialogue systems. in proceedings of the 2015 conference on empirical methods in natural language processing. sam wiseman, stuart m. shieber, and alexander m. rush. 2017. challenges in data-to-document generation. corr, abs/1707.08052. hayahide yamagishi, shin kanouchi, takayuki sato, and mamoru komachi. 2016. controlling the voice of a sentence in japanese-to-english neural machine translation. in proceedings of the 3rd workshop on asian translation (wat2016), pages 203-210.