Weakly Supervised Semantic Parsing by Learning from Mistakes (2021)

Abstract: Weakly supervised semantic parsing (WSP) aims at training a parser via utterance-denotation pairs. This task is challenging because it requires (1) searching consistent logical forms in a huge space; and (2) dealing with spurious logical forms. In this work, we propose Learning from Mistakes (LFM), a simple yet effective learning framework for WSP. LFM utilizes the mistakes made by a parser during searching, i.e., generating logical forms that do not execute to correct denotations, for tackling the two challenges. In a nutshell, LFM additionally trains a parser using utterance-logical form pairs created from mistakes, which can quickly bootstrap the parser to search consistent logical forms. Also, it can motivate the parser to learn the correct mapping between utterances and logical forms, thus dealing with the spuriousness of logical forms. We evaluate LFM on WikiTableQuestions, WikiSQL, and TabFact in the WSP setting. The parser trained with LFM outperforms the previous state-of-the-art semantic parsing approaches on the three datasets. Also, we find that LFM can substantially reduce the need for labeled data. Using only 10% of utterance-denotation pairs, the parser achieves 84.2 denotation accuracy on WikiSQL, which is competitive with the previous state-of-the-art approaches using 100% labeled data.

## introduction semantic parsing is the task of mapping a natural language utterance to a logical form that can be executed against a knowledge base to obtain a denotation. much progress has been made in this area, thanks to the emergence of datasets that include a large number of utterance-logical form pairs. however, collecting such pairs at scale is generally expensive, because annotators must be skilled at programming. by contrast, collecting utterance-denotation pairs is much cheaper, because it can be performed by non-experts. hence, it is tempting to train a semantic parser via utterancedenotation pairs, framing a weakly supervised semantic parsing problem (wsp) @xcite @xcite . training a parser from denotations rather than logical forms complicates training in two ways. first, training a parser requires exploring the huge space of logical forms to find those that execute to correct denotations, which we call "consistent" logical forms. this is a very difficult search problem due to the combinatorial nature of the search space. figure 1 presents five logical forms for an utterance-denotation pair, among which the first three are consistent and the rest are mistake logical forms (they do not execute to the correct denotation). second, consistent logical forms can be "spurious". spurious logical forms accidentally execute to correct denotations, but they do not reflect the meaning of utterances. for example, two of the three consistent logical forms in figure 1 are spurious, and only the first one is "correct", reflecting the utterance's meaning. the presence of spurious logical forms severely hinders a parser from learning the correct mapping between utterances and logical forms. existing approaches for wsp can be categorized into static and dynamic, according to whether they perform searching at training time. static approaches heuristically search consistent logical forms offline @xcite @xcite . they assume that there are correct logical forms in the search results, and they do not perform searching at training time. however, this assumption may not hold when the spuriousness is severe. considering a binary denotation (true or false), 50% of syntactically valid logical forms execute to the correct denotation, regardless of their semantics. dynamic approaches do not make this assumption. they iteratively search consistent logical forms using a parser and train the parser with the search result in turn @xcite @xcite . but dynamic approaches generally suffer from a cold-start problem, because it is challenging for a randomly initialized parser to search consistent logical forms in an exponentially large space. hence, most dynamic approaches require a set of pre-searched consistent logical forms to bootstrap the training. in this work, we propose learning from mistakes (lfm for short), a simple yet effective dynamic learning framework for wsp. the core insight of lfm is that a parser will generate a huge number of mistake logical forms during searching. these mistake logical forms can be fully utilized to overcome the cold-start and spuriousness problems. in a nutshell, every time a parser makes a mistake, lfm synthesizes a faithful utterance for the mistake logical form. then, lfm trains the parser with this utterance-logical form pair, so that the parser is taught the correct meaning of the mistake logical form. in addition, lfm also trains the parser like existing dynamic approaches, using consistent logical forms with learning objectives such as reinforce @xcite and maximum marginal likelihood (mml). lfm has two major advantages over existing dynamic approaches. first, lfm does not need to pre-search consistent logical forms to warm start the training. instead, it creates utterance-logical form pairs from mistakes on the fly to overcome the cold-start problem. second, lfm can facilitate a parser learning the correct mapping between ut-terances and logical forms. since the synthesized utterances are guaranteed to reflect the meaning of logical forms, a parser can learn the correct mapping from the synthesized utterance-logical form pairs. the idea of lfm is inspired by our human beings. every time we make a mistake, we try to modify our knowledge to avoid suffering again in the future for the same reason @xcite . similarly, every time a parser makes a mistake, we try to teach the parser the correct meaning of the mistake logical form and help it avoid the mistake in the next round of searching. to demonstrate the effectiveness of lfm, we conduct experiments on three challenging semantic parsing datasets in the wsp setting. the neural semantic parser trained with lfm achieves a denotation accuracy of 52.3 on wikitablequestions, 86.9 on wikisql, and 68.2 on tabfact, which all surpass previous state-of-the-art approaches in the same setting. through a fine-grained analysis, we show that lfm is effective in addressing the coldstart and spuriousness problems, and lfm is more effective than prior data augmentation techniques for wsp. also, we find that lfm can substantially reduce the need for labeled data to train a good parser. for example, the parser achieves an accuracy of 84.2 on wikisql using only 10% of utterance-denotation pairs, which already performs on par with previous state-of-the-art approaches. ## learning framework in this section, we formally define the task of wsp and describe lfm in detail. ## experimental setup in this section, we present the experimental setup for lfm, including datasets, implementations of the semantic parser and text generator. ## conclusion & future work in this work, we present lfm, a simple yet effective dynamic learning framework for wsp. lfm fully utilizes the mistake logical forms generated by a parser during searching to overcome the major challenges in wsp. experimental results on three semantic parsing datasets show that lfm can effectively address the challenges, and lfm can substantially reduce the need for utterance-denotation pairs to train a good parser. this work also opens up several avenues for future work. first, further improvements could be made by using a more advanced text generator in lfm. the generator is currently implemented using a hand-crafted scfg, which often generates unnatural utterances. second, lfm can be extended to other weakly supervised learning problems where synthesizing inputs (e.g., utterances) from latent variables (e.g., logical forms) is trivial. consider the problem of learning to solve math word problems via utterance-answer pairs. it is trivial to synthesize an utterance from a math equation. therefore, lfm could be applied to solve this learning problem.