Few-Shot Semantic Parsing for New Predicates (2021)

Abstract: In this work, we investigate the problems of semantic parsing in a few-shot learning setting. In this setting, we are provided with k utterance-logical form pairs per new predicate. The state-of-the-art neural semantic parsers achieve less than 25% accuracy on benchmark datasets when k = 1. To tackle this problem, we proposed to i) apply a designated meta-learning method to train the model; ii) regularize attention scores with alignment statistics; iii) apply a smoothing technique in pretraining. As a result, our method consistently outperforms all the baselines in both one and two-shot settings.

## introduction semantic parsing is the task of mapping natural language (nl) utterances to structured meaning representations, such as logical forms (lf). one key obstacle preventing the wide application of semantic parsing is the lack of task-specific training data. new tasks often require new predicates of lfs. suppose a personal assistant (e.g. alexa) is capable of booking flights. due to new business requirement it needs to book ground transport as well. a user could ask the assistant "how much does it cost to go from atlanta downtown to airport?". the corresponding lf is as follows: for each new predicate. the cost of preparing fewshot training examples is low, thus the corresponding techniques permit significantly faster prototyping and development than supervised approaches for business expansions. semantic parsing in the few-shot setting is challenging. in our experiments, the accuracy of the state-of-the-art (sota) semantic parsers drops to less than 25%, when there is only one example per new predicate in training data. moreover, the sota parsers achieve less than 32% of accuracy on five widely used corpora, when the lfs in the test sets do not share lf templates in the training sets @xcite ). an lf template is derived by normalizing the entities and attribute values of an lf into typed variable names @xcite . the few-shot setting imposes two major challenges for sota neural semantic parsers. first, it lacks sufficient data to learn effective representations for new predicates in a supervised manner. second, new predicates bring in new lf templates, which are mixtures of known and new predicates. in contrast, the tasks (e.g. image classification) studied by the prior work on few-shot learning @xcite considers an instance exclusively belonging to either a known class or a new class. thus, it is non-trivial to apply conventional few-shot learning algorithms to generate lfs with mixed types of predicates. to address above challenges, we present protoparser, a transition-based neural semantic parser, which applies a sequence of parse actions to transduce an utterance into an lf template and fills the corresponding slots. the parser is pretrained on a training set with known predicates, followed by fine-tuning on a support set that contains few-shot examples of new predicates. it extends the attention-based sequence-to-sequence architecture @xcite with the following novel techniques to alleviate the specific problems in the few-shot setting: our extensive experiments on three benchmark corpora show that protoparser outperforms the competitive baselines with a significant margin. the ablation study demonstrates the effectiveness of each individual proposed technique. the results are statistically significant with @xmath1.05 according to the wilcoxon signed-rank test @xcite . ## semantic parser protoparser follows the sota neural semantic parsers @xcite to map an utterance into an lf in two steps: template generation and slot filling foot_0 . it implements a designated transition system to generate templates, followed by filling the slot variables with values extracted from utterances. to address the challenges in the few-shot setting, we proposed three training methods, detailed in sec. 4. many lfs differ only in mentioned atoms, such as entities and attribute values. an lf template is created by replacing the atoms in lfs with typed slot variables. as an example, the lf template of our example in sec. 1 is created by substituting i) a typed atom variable v e for the entity "atlanta:ci"; ii) a shared variable name v a for all variables "@xmath01". formally, let @xmath2, ..., x n } denote an nl utterance, and its lf is represented as a semantic tree @xmath3, e), where @xmath4, ..., v m } denotes the node set with v @xmath5, and a template predicate node represents a predicate symbol or a term, while a slot value node represents an atom mentioned in utterances. thus, a semantic tree y is composed of an abstract tree τ y representing a template and a set of slot value nodes v v,y attaching to the abstract tree. in the few-shot setting, we are provided with a train set d train , a support set d s , and a test set d test . each example in either of those sets is an utterance-lf pair (x i , y i ). the new predicates appear only in d s and d test but not in d train . for k-shot learning, there are k ( each new predicate appears also in the test set. the goal is to maximize the accuracy of estimating lfs given utterances in d test by using a parser trained on d train ∪ d s . ## few-shot model training the few-shot setting differs from the supervised setting by having a support set in testing in addition to train/test sets. the support set contains k utterance-lf pairs per new predicate, while the training set contains only known predicates. to evaluate model performance on new predicates, the test set contains lfs with both known and new predicates. given the support set, we can tell if a predicate is known or new by checking if it only exists in the train set. we take two steps to train our model: i) pretraining on the training set, ii) fine-tuning on the support set. its predictive performance is measured on the test set. we take the two-steps approach because i) our experiments show that this approach performs better than training on the union of the train set and the support set; ii) for any new support sets, it is computationally more time efficient than training from scratch on the union of the train set and the support set. there is a distribution discrepancy between the train set and the support set due to new predicates, the meta-learning algorithms @xcite suggest to simulate the testing scenario in pre-training by splitting each batch into a meta-support set and a meta-test set. the models utilize the information (e.g. prototype vectors) acquired from the meta-support set to minimize errors on the meta-test set. in this way, the metasupport and meta-test sets simulate the support and test sets sharing new predicates. however, we cannot directly apply such a training procedure due to the following two reasons. first, each lf in the support and test sets is a mixture of both known predicates and new predicates. to simulate the support and test sets, the meta-support and meta-test sets should include both types of predicates as well. we cannot assume that there are only one type of predicates. second, our preliminary experiments show that if there is sufficient training data, it is better off training action embeddings of known predicates c (eq. ( 3 )) in a supervised way, while action embeddings initialized by a metric-based meta-learning algorithm @xcite perform better for rarely occurred new predicates. therefore, we cope with the differences between known and new predicates by using a customized initialization method in finetuning and a designated pre-training procedure to mimic fine-tuning on the train set. in the following, we introduce fine-tuning first because it helps understand our pre-training procedure. ## conclusion and future work we propose a novel few-shot learning based semantic parser, coined protoparser, to cope with new predicates in lfs. to address the challenges in few-shot learning, we propose to train the parser with a pre-training procedure involving predicatedropout, attention regularization, and pre-training smoothing. the resulted model achieves superior results over competitive baselines on three benchmark datasets.