Learning from Executions for Semantic Parsing (2021)

Abstract: Semantic parsing aims at translating natural language (NL) utterances onto machine-interpretable programs, which can be executed against a real-world environment. The expensive annotation of utterance-program pairs has long been acknowledged as a major bottleneck for the deployment of contemporary neural models to real-life applications. In this work, we focus on the task of semi-supervised learning where a limited amount of annotated data is available together with many unlabeled NL utterances. Based on the observation that programs which correspond to NL utterances should always be executable, we propose to encourage a parser to generate executable programs for unlabeled utterances. Due to the large search space of executable programs, conventional methods that use beam-search for approximation, such as self-training and top-k marginal likelihood training, do not perform as well. Instead, we propose a set of new training objectives that are derived by approaching the problem of learning from executions from the posterior regularization perspective. Our new objectives outperform conventional methods on Overnight and GeoQuery, bridging the gap between semi-supervised and supervised learning.

## introduction semantic parsing is the task of mapping natural language (nl) utterances to meaning representations (aka programs) that can be executed against a real-world environment such as a knowledge base or a relational database. while neural sequence-to-sequence models @xcite have achieved much success in this task in recent years, they usually require a large amount of labeled data (i.e., utteranceprogram pairs) for training. however, annotating utterances with programs is expensive as it requires expert knowledge of meaning representations (e.g., lambda calculus, sqls) and the envi-list all 3 star rated thai restaurants ## executability as learning signal in this section, we formally define our semisupervised learning setting and show how to incorporate executability into the training objective whilst relying on the marginal likelihood training framework. we also present two conventional approaches to optimizing marginal likelihood. ## method we first present a view of mml in equation ( 3 ) from the perspective of posterior regularization. this new perspective helps us derive three alternative approximations of mml: repulsion mml, gentle mml, and sparse mml. ## semantic parsers in principle, our x-pr framework is modelagnostic, i.e., it can be coupled with any semantic parser for semi-supervised learning. in this work, we use a neural parser that achieves state-of-the-art performance across semantic parsing tasks. specifically, we use rat-sql @xcite ) which features a relation-aware encoder and a grammarbased decoder. the parser was originally developed for text-to-sql parsing, and we adapt it to text-to-lf parsing. in this section, we briefly review the encoder and decoder of this parser. for more details, please refer to @xcite . ## experiments to evaluate x-pr, we present experiments on semi-supervised semantic parsing. we also analyze how the objectives affect the training process. ## conclusion in this work, we propose to learn a semisupervised semantic parser from the weak yet freely available executability signals. due to the large search space of executable programs, conventional approximations of mml training, i.e, self-training and top-k mml, are often sub-optimal. we propose a set of alternative objectives, namely x-pr, through the lens of posterior regularization. empirical results on semi-supervised learning show that x-pr can help a parser achieve substantially better performance than conventional methods, further bridging the gap between semi-supervised learning and supervised learning. in the future, we would like to extend x-pr to related tasks such as learning from denotations and weakly supervised question answering. a method mml vs. rl another choice for learning from executions is to maximize the expected reward: recall that our mml objective is defined as: as shown in previous work @xcite , the gradients (wrt. θ) of rl and mml have the same form: compared with rl, mml additionally renormalize the p over executable programs to obtain q. as a result, mml outputs a 'stronger' gradient than rl due to the renormalization. in our initial experiments, we found that rl is hard to converge whereas mml is not.