Generating Synthetic Data for Task-Oriented Semantic Parsing with Hierarchical Representations (2020)

Abstract: Modern conversational AI systems support natural language understanding for a wide variety of capabilities. While a majority of these tasks can be accomplished using a simple and flat representation of intents and slots, more sophisticated capabilities require complex hierarchical representations supported by semantic parsing. State-of-the-art semantic parsers are trained using supervised learning with data labeled according to a hierarchical schema which might be costly to obtain or not readily available for a new domain. In this work, we explore the possibility of generating synthetic data for neural semantic parsing using a pretrained denoising sequence-to-sequence model (i.e., BART). Specifically, we first extract masked templates from the existing labeled utterances, and then fine-tune BART to generate synthetic utterances conditioning on the extracted templates. Finally, we use an auxiliary parser (AP) to filter the generated utterances. The AP guarantees the quality of the generated data. We show the potential of our approach when evaluating on the Facebook TOP dataset for navigation domain.

## introduction in this work, we investigate semantic parsing with hierarchical representations @xcite instead of the traditional logical forms @xcite . given an utterance x, our goal is to produce a tree-structured representation y of the utterance where additional information about intents and slots is introduced at the non-terminal nodes of the tree. we define a template z of a given annotation y as a result of replacing all terminal nodes by a generic [mask] node. figure 1 shows an example of such an utterance x, its annotation y and the corresponding template z. the hierarchical representation for task-oriented parsing proposed in @xcite aims for ease of annotation and expressiveness. the dataset in their work, facebook top, is the largest publicly available dataset in english for hierarchical semantic parsing. it has more than 44k annotated queries. we look at the distribution of the templates in facebook top and found that the dataset is highly unbalanced (figure 2 ). the 10 most frequent templates account for 30% of the training data and 14% of the data are singletons, which are utterances with only a single occurrence. this analysis suggests that it is beneficial to generate more synthetic data for templates with low frequencies. in the field of natural language processing, us-ing synthetic data via back-translation @xcite has shown a great success for machine translation @xcite . unlike machine translation, generating synthetic data for hierarchical semantic parsing is less straightforward. our work positions itself as one of the first to explore the possibility of generate text from graph (template) for semantic parsing. in this paper, we propose a generic framework for augmenting a semantic parser with synthetic data. our framework consists of two steps. first, we train a generator, followed by top-p sampling to generate diverse synthetic utterances conditioning on the above-mentioned templates. generated utterances share similar hierarchical structures (i.e., templates) with real training utterances while providing a wide spectrum of lexical variety. second, we use an auxiliary parser for filtering on the generated candidates. the filtering step guarantees the quality of the synthetic data. our generator is a sequence to sequence (seq2seq) model that is pretrained on massive amount of monolingual data with text infilling objective ( §2). we utilize bart @xcite , a recently proposed denoising autoencoder, as our generator to avoid training it from scratch. the auxiliary parser can be arbitrary. we experiment with bart-based parser as well as state-of-the-art pointer network parser (s2s-pointer; @xcite . the paper is structured as follows. we introduce our generative model for synthetic data in section §2. experimental results on facebook top dataset and sub-sampled datasets to simulate lowresource scenario are presented in section §3. section §5 concludes the paper. ## denoising sequence-to-sequence as generator the generative story for generating synthetic data y @xmath0. draw a template z ∼ p φ (z); foot_0 2. draw an annotation y ∼ p θ (y | z) by filling each [mask] token in z by a word or sequence of words from vocabulary v; note that the transformation from annotation y to utterance x is deterministic by removing nonterminals from y. while p φ (z) can be modeled by an autoregressive neural language model or a probabilistic context free grammar @xcite , in this work we sample template z from seen templates in the data. we leave the possibility of generating new templates to future work. we need a powerful conditional model p θ (y | z) to generate annotation y. thus, we choose bart, a pretrained denoising autoencoder for sequenceto-sequence, as our model. figure 3a illustrates the idea behind bart. given an input sequence (a stream of text), one of five types of noise (figure 3b ) is used to corrupt the input sequence. then bart reconstructs the original sequence by maximizing the likelihood of the original sequence. @xcite bart is trained to reconstruct the corrupted input. a b c . d e . a . c . e . a _ . d _ e . a _c . _ e . c . d e . a b document rotation token masking token deletion text infilling d e . a b c . sentence permutation (b) five different types of noise introduced in bart. since pretrained bart uses text infilling as noise to corrupt the input sequence, naturally we can use bart to infill the templates. text infilling is the task where a number of spans in the original input sequence are replaced by a token [mask] and bart is trained to predict the replaced spans in the position of [mask] tokens. for our purpose of generating synthetic data, we fine-tune bart on an infilling dataset where the input is a template z with [mask] and the output is a linearized tree representation y where [mask] tokens are replaced by lexical words as shown in figure 4 . bart source/target construction: we call out a few processing steps to construct this infilling dataset. first, non-terminal words are lowercased. we find this is necessary since the input will be tokenized by bart tokenizer and lowercasing non-terminal words prevents oversegmentation. second, we make each of the closing brackets "]" in the original data explicit (e.g., in:get distance], sl:destination]). this transformation provides the model explicit infor- source: [in:get_distance [mask] [sl:destination [in:get_location [sl:category_location [mask] sl:category_location] in:get_location] sl:destination] in:get_distance] target: [in:get_distance how far is [sl:destination [in:get_location [sl:category_location the coffee shop sl:category_location] in:get_location] sl:destination] in:get_distance] mation of the scope of the intents and slots. fine-tuning and generation: we fine-tune bart generator using the (template, annotation) pairs. after fine-tuning, we use the generator to generate full parse trees given templates. to increase the diversity of generated samples, we use top-p sampling @xcite instead of beam search. the generator is trained to generate the tokenized labels together with the words. we remove generated annotations with invalid labels and convert the tokenized labels into the original tags in a post-processing step. auxiliary parser (ap) for filtering: in our preliminary experiments, we found that the generated samples are noisy. when we train our parser on the concatenation of both real and generated samples, the test accuracy degrades by 1.13% compared with a parser trained purely on real data. we therefore use an auxiliary parser (ap) to select robust samples. the filtering step is straightforward. first, we train an auxiliary semantic parser f θ (x) on the original facebook top dataset. we then use this trained ap to parse synthetic data ( x i , y i ) and keep those samples where the outputs of the parser f θ ( x i ) match the synthetic labels y i (i.e., f θ ( x i ) = y i ). the ap for filtering can be different from the target parser we train for semantic parsing. therefore, we propose three settings: (1) bart as ap and a sequence-to-sequence model with pointer networks (s2s-pointer; @xcite as the target parser. (2) bart models for both ap and target parser. (3) s2s-pointer models for both ap and target parser. the comparisons and analysis are detailed in section §3. ## experiments we use facebook top dataset in our experiments. statistics of the dataset are shown in the exact-match results for the three settings of using bart/s2s-pointer as auxiliary and target parser are given in in order to support new domains (with new intents and slots) for the virtual assistants, we investigate the role of synthetic data when there is a little data available for the new domains. we sim- ## related work using pretrained models to generate synthetic data has been studied recently @xcite . their work however focuses on multi-class classification problems. taking a step further, our work shows a viable path for structured output (i.e., parse trees) problems. ## conclusions we have proposed a novel approach for generating synthetic data for hierarchical semantic parsing. our initial experiments show promising results of this approach and open up possibility for applying it to other problems with highly structured outputs in natural language processing.