Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing (2020)

Abstract: Task-oriented semantic parsing is a critical component of virtual assistants, which is responsible for understanding the user’s intents (set reminder, play music, etc.). Recent advances in deep learning have enabled several approaches to successfully parse more complex queries (Gupta et al., 2018; Rongali et al.,2020), but these models require a large amount of annotated training data to parse queries on new domains (e.g. reminder, music). In this paper, we focus on adapting task-oriented semantic parsers to low-resource domains, and propose a novel method that outperforms a supervised neural model at a 10-fold data reduction. In particular, we identify two fundamental factors for low-resource domain adaptation: better representation learning and better training techniques. Our representation learning uses BART (Lewis et al., 2019) to initialize our model which outperforms encoder-only pre-trained representations used in previous work. Furthermore, we train with optimization-based meta-learning (Finn et al., 2017) to improve generalization to low-resource domains. This approach significantly outperforms all baseline methods in the experiments on a newly collected multi-domain task-oriented semantic parsing dataset (TOPv2), which we release to the public.

## introduction virtual assistants now play an ever increasingly important role in our daily life, and can help users perform a wide spectrum of tasks ranging from setting personal reminders, checking local weather, to controlling smart home devices and online shopping. a critical step in any virtual assistant is to understand the user's intent (e.g. set reminder, get weather info, etc.) given the user utterance. in recent years, a number of successful models have 1 the dataset can be downloaded at https://fb.me/ topv2dataset emerged to tackle such task-oriented semantic parsing task, for both simple and more complex queries. traditionally, task-oriented semantic parsers treat the problem as a joint intent classification and slot filling task @xcite , where the model first predicts the intent of the input utterance from a set of pre-defined intent labels, and then identify all the necessary slots for that intent. for instance, for the query "how's the weather in san francisco?", the model would predict the get_weather intent, and tag san francisco as a location slot. with the elevated expectation of virtual assistants, however, techniques for handling the more complex compositional queries have been proposed recently using neural parsers @xcite or seq2seq models @xcite . in particular, these approaches can handle complicated queries with multiple intents or nested slots. for example, the following query from the top dataset @xcite ) "driving directions to the eagles game" is composed of a get_directions intent and a get_event one nested in a tree structure (figure 1 ). on the other hand, most of these deep neural models require a large amount of annotated training data to achieve a good performance, which is aggravated by the fact that virtual assistants need to support hundreds of tasks each mandating a separate set of labeled training samples. therefore, in order to support more diverse use cases without an excessive need in human annotated data, it becomes crucial that the semantic parsing model has the capability to generalize to new tasks or domains (reminder, music, etc.) with a limited number of labeled samples in the target domain. while transfer learning methods have been proposed for traditional sequence tagging models @xcite to help facilitate learning slot filling models for domains with less annotated data, the efforts have been lacking when it comes to developing compositional semantic parsers in such low-resource domain adaptation setting. therefore, we in this paper show it is possible to build compositional task-oriented semantic parsers for low-resource domains (e.g. 25 training samples per intent or slot label), and propose a solution that is competitive against supervised models trained with 10x more data. we identify two key factors for successfully adapting task-oriented semantic parsers to new domains: better representation learning and better training techniques. we first show that pre-trained language representations are critical in the low-resource setting for the model to quickly generalize to new intents and slots. furthermore, most pre-trained language representations used in previous work such as bert @xcite or roberta @xcite are encoder-only models, and are hence not ideal for a compositional parser with an encoder-decoder (seq2seq) architecture. we therefore propose to use bart @xcite , a pre-trained seq2seq model that can be used to initialize both the encoder and decoder of our semantic parser, which significantly outperforms other pre-trained representations such as roberta. more importantly, these large pre-trained models are sometimes known to pose challenges to fine-tuning with very few training samples. in order to better adapt the semantic parser to lowresource domains, we employ optimization-based meta-learning @xcite to improve generalization of the bart model trained on the source domains, making it easier to be fine-tuned on the target domains with very little training data. finally, in order to evaluate our approach, we collect a multi-domain compositional task-oriented semantic parsing dataset (topv2), based on the original top @xcite dataset. in addition to the navigation and event domains found in top, our topv2 dataset has 6 new domains: alarm, messaging, music, reminder, timer, and weather, with more than 137k new samples. we conduct extensive experiments on this new dataset, showing that our proposed method significantly outperforms all the baseline models in the low data regime. we further show that our model achieves competitive performance compared to supervised state-of-theart models while using 10x less data. ## problem setup we first formally define the task of domain adaptation (or domain scaling) for task-oriented semantic parsing. as illustrated in figure 1 , the taskoriented parsing task aims to predict the semantic parse given the user utterance (or query). the semantic parse has a tree structure and is represented using a serialized tree representation defined in the top dataset @xcite . following recent state-of-the-art practices @xcite , we formulate the problem as a sequence-to-sequence (seq2seq) task, where the utterance is treated as the source sequence s, while the semantic parse serves as the target sequence t . in our domain scaling setting, the goal is to develop a semantic parser with minimal training examples on a set of new target domains. 2 formally, denote @xmath0, ..., d t n } as the set of n target domains. on the other hand, we assume access to training data for a number of source domains, which can be used to help build models on the target domains. denote @xmath1, ..., d s m } as the set of m source domains. for each source domain @xmath2, there exists a set of annotated training data d @xmath3, t d ) where s d and t d are the utterance and semantic parse, respectively. for a target domain @xmath4, however, only a very limited number of training instances exist. as domains differ drastically in terms of complexity (shown in weather in seattle rnn / transformer encoder [in:get_weather rnn / transformer decoder [sl:location … … in seattle p copy 1 -p copy copy probabilities generation probabilities encoder states decoder states figure 2: overview of our sequence-to-sequence architecture with copy pointer (seq2seq-copyptr). note that the target sequence is slightly different from original top dataset, as explained in the pre-processing steps in §5.2. stands for samples per intent and slot, indicating the number of training samples available for each intent and slot label. traditionally, at least a few hundreds of training samples are needed for each label to successfully train a deep neural semantic parser (see §5.3). in the low-resource setting, in contrast, we focus on a scenario with much less training data at 25 spis. that is, for each intent and slot in the target domain, 25 training samples are required foot_1 . (see §5.4 for discussions on more spis settings.) to benchmark the performance on the low-resource domains, we compare with various high-resource supervised baselines trained with much more data up to 500 or 1000 spis. ## base model in this section, we present our core model architecture. our meta-learning technique will be introduced in section 4. we follow recent state-of-theart approaches @xcite and adopt a seq2seq model as our base architecture (seq2seq-copyptr), derived from the pointer generator network @xcite . the base architecture is shown in figure 2 . for an input sequence @xmath5, w 2 , . . . , w n ], the encoder first encodes it into a series of hidden vectors (encoder states) [e 1 , e 2 , . . . , e n ]. the encoder states are then passed to an decoder that autoregressively produces target tokens o t for each timestamp t. specifically, the decoder first outputs a hidden decoder state d t based on the decoder states from previous timestamps as well as all en-coder states: in task-oriented semantic parsing, the target sequence consists of two types of tokens: utterance tokens that always come from the source sequence, and ontology tokens that represent intent and slot labels. therefore, two probability distributions are formulated and combined in order to produce the output token, namely the copy probability and the generation probability. the generation probability g t is produced by the decoder by mapping the decoder state onto the output vocabulary, which only includes ontology tokens but not utterance tokens. the copy probability c t , on the other hand, indicates whether to copy one of the source tokens as the decoder output for timestamp t, and is predicted by using d t as the query to perform a multi-head attention (mha, @xcite over the encoder states. ## meta learning as mentioned in §3.2, model training for lowresource domain adaptation consists of two stages: source training and fine-tuning (or target training), where the model (initialized with pre-trained representations) is first trained on the source domains and then fine-tuned on each low-resource target domain. as target domain training data is scarce, it might be challenging to effectively fine-tune a large bart model with only 25 samples per intent and slot. one reason is that traditional source training optimizes the model performance solely for the source domains, which may result in a model with strong performance on the source domains but less than ideal for transferring to new target domains via fine-tuning. therefore, we propose to replace source training with optimization-based meta-learning @xcite to improve generalization. instead of directly optimizing towards source domain accuracy, meta-learning, when trained on the source domains, looks for a good initialization θ 0 that can easily be adapted to new tasks (domains) with minimal finetuning. in order to learn a model that is easier to be fine-tuned on new tasks with a small amount of training data, meta-learning adopts a different training objective that explicitly optimizes for generalization by repeatedly simulating low-resource fine-tuning during training. specifically, in each iteration (episode), the maml @xcite algorithm samples two batches of training samples from a source domain @xmath6: d d s and d d q , conventionally named the support and query set respectively. in standard source training, one simply computes the loss on d d s and takes a gradient step to update the model. in maml, however, low-resource fine-tuning is simulated at each training episode. let θ denote the model parameters being meta-learned, maml first takes a gradient step on d d s that leads to: finally, maml requires the computation of second derivatives when unrolling equation (1), which consumes too much memory for large models such as bart. therefore, we instead adopt a first-order meta-learning algorithm, reptile @xcite , which has shown comparative or even superior performance than maml despite its simplicity @xcite . in reptile, @xmath7, . . . , d d k are sampled for a source domain @xmath8, and the model is updated as follows: ## experiments in this section, we first introduce topv2, a multidomain task-oriented semantic parsing dataset we are releasing to the community. it is an extension to the top dataset with 6 additional domains and 137k new samples. we then outline the setup of our low-resource domain scaling experiments in §5.2, and present the experimental results in §5.3. ## related work task-oriented semantic parsing has attracted attention from the research community since 1990s with the advent of the atis dataset @xcite . traditionally, the task is formulated as a joint text classification (intent prediction) and sequence tagging (slot filling) problem, and can be tackled with sequence labeling models such as rnns @xcite . these models can only parse flat queries with one intent and nonnested slots. more recently, a number of studies propose alternative approaches for handling the more complex compositional queries using neural shift-reduce parsers @xcite or seq2seq models @xcite . on the other hand, there have been research efforts on scaling task-oriented parsers to new domains with less training data @xcite @xcite @xcite . these methods, however, only focus on the simpler flat queries. our proposed method, in contrast, can effectively parse both flat and compositional queries for low-resource target domains. meta-learning @xcite , or learning to learn, aims to learn a model that can quickly adapt to new tasks with a small amount of training data. in particular, @xcite propose maml, an optimization-based meta-learning method, which learns a good parameter initialization suitable for faster adaptation to new tasks. as maml requires to compute second derivatives, which are computation and memory intensive, there have been studies to use either first-order approximation such as firstorder maml and reptile @xcite , or implicit differentiation @xcite . furthermore, meta-learning has also been applied to a number of nlp tasks lately @xcite @xcite @xcite . ## conclusion in this work, we study the low-resource domain scaling problem for task-oriented semantic parsing. in particular, we focus on the 25 spis setting to investigate whether a model can effectively adapt to new domains with a very limited amount of training data. our approach distinguishes itself from previous methods on two fronts. first of all, we argue the encoder-only pre-trained representations used in existing work are not ideal for the seq2seq model employed in task-oriented semantic parsing, and instead propose to use bart, a pre-trained model with an encoder-decoder architecture. more importantly, we adopt optimization-based meta-learning to improve the model's generalization to new target domains with very few training samples. our experiments show that our proposed method significantly outperforms all competing methods and achieves the best performance in the lowresource setting. even when compared with supervised models trained with 500 spis, a 10-fold data increase, our best performing model remains competitive, and outperforms a state-of-the-art lstmbased pointer generator network @xcite . last but not least, we collect the topv2 dataset, a large-scale multi-domain task-oriented semantic parsing dataset with 8 domains and more than 180k annotated samples to evaluate our models, which we release to the research community.