Low-Resource Compositional Semantic Parsing with Concept Pretraining (2023)

Abstract: Semantic parsing plays a key role in digital voice assistants such as Alexa, Siri, and Google Assistant by mapping natural language to structured meaning representations. When we want to improve the capabilities of a voice assistant by adding a new domain, the underlying semantic parsing model needs to be retrained using thousands of annotated examples from the new domain, which is time-consuming and expensive. In this work, we present an architecture to perform such domain adaptation automatically, with only a small amount of metadata about the new domain and without any new training data (zero-shot) or with very few examples (few-shot). We use a base seq2seq (sequence-to-sequence) architecture and augment it with a concept encoder that encodes intent and slot tags from the new domain. We also introduce a novel decoder-focused approach to pretrain seq2seq models to be concept aware using Wikidata and use it to help our model learn important concepts and perform well in low-resource settings. We report few-shot and zero-shot results for compositional semantic parsing on the TOPv2 dataset and show that our model outperforms prior approaches in few-shot settings for the TOPv2 and SNIPS datasets.

## introduction voice assistants such as alexa, siri, and google assistant often rely on semantic parsing to understand requests made by their users. the underlying semantic parsing model converts natural language user utterances into logical forms consisting of actions requested by the user (play music, check weather), called intents, and relevant entities in the request (which song? which location?), called slots. the model is built to process requests in a fixed set of domains, such as music, weather, shopping, and so on. with voice assistants increasingly pervading * work done when sr was at umass and ms and ka were at amazon; it does not relate to their current positions. more aspects of daily life, systems need to be continuously updated to comprehend new intents and slots across an ever-growing number of domains. current semantic parsing models are trained on large amounts of annotated data from a predetermined set of domains. extending these models to learn new intents or slots typically involves collecting and annotating large amounts of new data. this process is expensive and time-consuming. to combat this problem, researchers have proposed semantic parsing models that can be efficiently trained with fewer examples (few-shot) from new domains @xcite @xcite @xcite @xcite . while these methods facilitate few-shot learning, they have limitations. some of them rely on hand-crafted knowledge such as intermediate grammars or logical form templates @xcite @xcite . others rely on very large pretrained language models, such as gpt-3, to perform in-context learning by appending test examples with instructional prompts @xcite . in this work, we explore few-shot domain adaptation for semantic parsing without any additional hand-crafted knowledge apart from the intent and slot tag names, and with much smaller architectures that can perform efficient inference in practical production environments. we also explore zero-shot domain adaptation, when we have no annotated training data from a new domain. to that end, we propose concept-seq2seq, a novel architecture based on a state-of-the-art semantic parsing model, seq2seq-ptr @xcite , which uses seq2seq models and a pointer generator network to decode the target semantic parse. we augment this model with a concept encoder that encodes intents and slots from the schema and uses those encodings to conditionally decode the semantic parse. figure 1 shows tag dot product scores concept encoder ... begin play intent [in:play end play intent in:play] end genre slot sl:genre] play top country hits [in:play [sl:sort @ptr 1 sl:sort] [sl:genre pointer scores intent and slot tag definitions sl:genre] @ptr 2 (country) @ptr 2 sl:genre] utterance semantic parse ... pretrained encoder transformer decoder the architecture of our proposed model. we train this model on annotated data from the given domains. during inference, we simply encode all intents and slots from the schema, including new, unseen ones, into the learned concept space, and decode the target parse. this model has the same time complexity as the original seq2seq-ptr model but comes with the added benefit of now being able to effectively parse utterances from unseen domains without any additional effort. there have been a few zero-shot semantic parsing approaches proposed in the past but they either covered only simple slot-filling style utterances @xcite or compositional utterances that also came with carefully crafted intermediate representations and contextfree grammars @xcite . our model is capable of performing zeroshot domain adaptation for compositional semantic parsing, producing meaning representations with nested intents and slots, but also doesn't require any grammars, whose construction effort often exceeds the effort required to annotate a few examples. in few-shot scenarios, we fine-tune our zero-shot model checkpoints further on the small number of available examples. due to the presence of the concept encoder in our architecture, we expect to receive better knowledge-transfer advantages by encoding intent and slot tags from new domains as opposed to initializing them as new tags. to further improve performance, we propose a novel decoder-focused pretraining scheme for concept-seq2seq using an entity-centric processed version of wikidata @xcite called wikiwiki @xcite , to help it better encode unseen concepts and parse effectively. we report the first zero-shot performance numbers for semantic parsing on the compositional topv2 dataset @xcite and show that concept-seq2seq achieves commendable zeroshot performance on the flat-entity snips dataset @xcite . we also evaluate in few-shot settings and show that we match or outperform previous state-of-the-art models while still being production-viable. in summary, our contributions are as follows. and zero-shot) domain adaptation for semantic parsing. it is based on the seq2seq-ptr model from @xcite , consisting of a sequence-tosequence encoder-decoder component, augmented with a pointer generator network to constrain the target decoding vocabulary. since our task at hand is to perform potential zero-shot semantic parsing with just descriptive metadata about the new domain, we modify the architecture of seq2seq-ptr to incorporate information about new intents and slots from new domains by adding a concept encoder. section 2.2 describes this architecture in detail. to help our model learn to parse utterances from unseen domains better, we also propose a novel pretraining scheme to incorporate general concept parsing knowledge into it. section 2.3 describes this concept pretraining scheme. finally, we describe concept-seq2seq model specifics for few-shot and zero-shot settings in section 2.4. before we get to these sections, we first describe the source and target sequence formulation for the semantic parsing task below. ## experimental setup we evaluate concept-seq2seq on few-shot and zero-shot domain adaptation using two popular english task-oriented semantic parsing datasets -topv2 @xcite and snips @xcite . both datasets have utterances grouped into multiple domains; topv2 has eight domains and snips has seven intents from seven different areas, which we consider domains. topv2 is a large dataset consisting of 10k-20k training and 3k-7k test examples per domain. it is also comprised of compositional examples with nested intents and slots. we exclude the unsupported utterances from the training and test sets in topv2 for the zeroshot experiments (we use the full sets in few-shot). unsupported utterances consist of utterances that belong to a domain but are not supported, which is impossible to learn in zero-shot. snips is a smaller and simpler dataset with flat, disjoint slots. it has 2k training and 100 test examples per domain. for zero-shot, we use a leave-one-out approach where given n domains, we train models on annotated data from n -1 of them and evaluate on utterances from the left-out domain. for few-shot settings, where the model has access to a few annotated examples from the left-out domain, we further fine-tune using 1, 5, and 25 samples-perintent/slot (spis), which we randomly sample from the training data of the left-out domain. we finetune concept-seq2seq on three different randomly sampled training sets per domain per spi setting and report the average performance score of the three runs. we use a transformer encoder, initialized from a roberta-base checkpoint, for concept-seq2seq. the decoder is a transformer decoder initialized from scratch and it contains 6 layers, 8 heads, and a hidden state size of 768. the concept encoder is also a transformer encoder and it is initialized from a bert-base-uncased checkpoint. we choose a bert-based model here since it is pretrained to compute a vector for the whole sequence using the cls token, which is what we need for encoding a concept consisting of a multi-word description. we also choose all base-size components to keep the overall model size small and expect the relative improvements shown by our model to generalize. we train our zero-shot models using sequence cross entropy loss. we use the adam optimizer with learning rate 2e -5 and ϵ = 1e -8 , warm-up proportion 0.1, weight decay 0.01, and batch-size 128. the number of epochs is set to 100 and we evaluate after every epoch and early stop with a patience of 5. for the wikiwiki pretraining step, we use the same hyper-parameters but stop the model training after 2 epochs on the entire wikiwiki dataset. we did not perform explicit hyperparameter tuning. for the few-shot experiments, we take the zeroshot model trained by excluding the given few-shot domain and fine-tune it on small set of annotated examples for 1000 epochs, evaluating after every 25 epochs. all other parameters are set to the same values as in the initial zero-shot training. we use a multiplier of 0.1 for the augmented loss from the random known-domain data batch. to speed up evaluation during training, we use teacher-forced sequence accuracy as our validation metric, which doesn't require us to perform any beam search. during inference, we use beam search decoding with a beam size of 4. we report exact match (em) accuracy for the few-shot experiments, meaning the entire predicted parse has to match exactly with the gold parse. for zero-shot, we report both em and f1 score since the task is more difficult and the performance is generally lower. at these lower numbers, f1 score, which awards partial credit to correctly tagged spans provides a better picture for improvements than em accuracy, which requires the entire pre- ## results and discussion in this section, we report and discuss the performance of concept-seq2seq on zero-shot and few-shot domain adaptation for semantic parsing. we first briefly describe our findings in zero-shot setting and then describe findings in a variety of few-shot scenarios. ## related work zero-shot domain adaptation for task-oriented semantic parsing has been previously explored for simple flat queries with single intents and disjoint, non-overlapping slots. @xcite and @xcite encode the lexical tag features and create a token-tagging schema to create the final semantic parses. @xcite solve the task using a retrieve-and-fill mechanism. our baseline model for simple queries is based on these approaches. for complex utterances with nested structures, zero-shot semantic parsing has been explored using intermediate, concept-agnostic logical forms @xcite @xcite or natural language canonical forms @xcite . these approaches apply to semantic parsing datasets which have context free grammars and specified rules, such as database or knowledge graph queries. the effort to craft these grammars for task-oriented semantic parsing in a voice assistant setting could quite possibly be greater than annotating utterances. a more relevant class of approaches for this work are ones that solve task-oriented semantic parsing for complex utterances in a few-shot setting using lexical tag features. @xcite and @xcite modify the seq2seq architecture from @xcite to perform non-autoregressive style decoding and show that their models perform better in a few-shot setting. @xcite use adaptive label smoothing, a model-agnostic technique. @xcite proposed a prompting-style approach where custom instructional prompts filled with handful of annotated examples and an unsolved utterance are fed as input to gpt-3 to directly produce a semantic parse. their approach is extremely slow and cannot be easily adapted into a zero-shot framework. @xcite explore a retrieve-and-fill style approach where they retrieve the best scenario, an intermediate logical form consisting of the semantic frame and abstracted out tags, from a scenario bank of all supported semantic parses. their approach is contingent on the availability of this scenario bank which could possibly entail more effort than annotating utterances. @xcite and @xcite use lexical features from intent and slot names to create an inventory and use it as input to train semantic parsers for new domains. @xcite also pretrain their model to improve generalizability but only evaluate it on an intent classification task. @xcite evaluate their model for full sequences and our model is similar to theirs. however, we use our inventory to create custom decoder embeddings in a seq2seq model, which removes any input size issues that their model will encounter with large inventories. we also pretrain our model with wikidata and evaluate it in a completely zero-shot setting, in addition to few-shot. @xcite is another recent question-answering-based approach that uses lexical features from the intent and slot tags by using them as context and posing questions but it has a similar input size issue with large inventories. ## conclusion we propose a model called concept-seq2seq to perform low-resource domain adaptation for compositional semantic parsing. our model is built on the seq2seq-ptr framework and is augmented with a concept encoder to transfer knowledge and encode unseen intents and slots from new domains through their text definitions. we also propose a novel concept pretraining scheme to incorporate general concept knowledge into our model using an entity-centric wikipedia dataset called wikiwiki. we evaluate our model in zero-shot and multiple few-shot settings on facebook topv2 and snips datasets. we show that our model is capable of performing zero-shot domain adaptation on some domains of the topv2 dataset and beats a strong slotfilling baseline on the snips dataset. in few-shot, over multiple dataset sizes of 1, 5, and 25 spis, we show that our model outperforms many strong prior models on topv2. using the snips dataset, we also demonstrate how our model catches up to a fully-trained semantic parsing model using just 25 spis on most domains. our model is capable of low-resource domain adaptation in both compositional and flat parsing settings, without additional hand-crafted information, and with the inference behavior of a vanilla seq2seq-ptr model.