Unsupervised Dual Paraphrasing for Two-stage Semantic Parsing (2020)

Abstract: One daunting problem for semantic parsing is the scarcity of annotation. Aiming to reduce nontrivial human labor, we propose a two-stage semantic parsing framework, where the first stage utilizes an unsupervised paraphrase model to convert an unlabeled natural language utterance into the canonical utterance. The downstream naive semantic parser accepts the intermediate output and returns the target logical form. Furthermore, the entire training process is split into two phases: pre-training and cycle learning. Three tailored self-supervised tasks are introduced throughout training to activate the unsupervised paraphrase model. Experimental results on benchmarks Overnight and GeoGranno demonstrate that our framework is effective and compatible with supervised training.

## introduction semantic parsing is the task of converting natural language utterances into structured meaning representations, typically logical forms @xcite @xcite . one prominent approach to build a semantic parser from scratch follows this procedure @xcite : a). (canonical utterance, logical form) pairs are automatically generated according to a domain-general grammar and a domainspecific lexicon. b). researchers use crowdsourcing to paraphrase those canonical utterances into natural language utterances (the upper part of figure 1 ). c). a semantic parser is built upon collected (natural language utterance, logical form) pairs. canonical utterances are pseudo-language utterances automatically generated from grammar rules, which can be understandable to people, but do not sound natural. though effective, the paraphrasing paradigm suffers from two drawbacks: (1) dependency on nontrivial human labor and (2) low utilization of canonical utterances. annotators may struggle to understand the exact meanings of canonical utterances. some canonical utterances even incur ambiguity, which enhances the difficulty of annotation. furthermore, @xcite and @xcite only exploit them during data collection. once the semantic parsing dataset is constructed, canonical utterances are thrown away, which leads to insufficient utilization. while @xcite and @xcite have reported the effectiveness of leveraging them as intermediate outputs, they experiment in a completely supervised way, where the human annotation is indispensable. in this work, inspired by unsupervised neural machine translation @xcite , we propose a two-stage semantic parsing framework. the first stage uses a paraphrase model to convert natural language utterances into corre-sponding canonical utterances. the paraphrase model is trained in an unsupervised way. then a naive foot_0 neural semantic parser is built upon autogenerated (canonical utterance, logical form) pairs using traditional supervised training. these two models are concatenated into a pipeline (figure 1 ). paraphrasing aims to perform semantic normalization and reduce the diversity of expression, trying to bridge the gap between natural language and logical forms. the naive neural semantic parser learns inner mappings between canonical utterances and logical forms, as well as the structural constraints. the unsupervised paraphrase model consists of one shared encoder and two separate decoders for natural language and canonical utterances. in the pre-training phase, we design three types of noise (section 3.1) tailored for sentence-level denoising autoencoder @xcite task to warm up the paraphrase model without any parallel data. this task aims to reconstruct the raw input utterance from its corrupted version. after obtaining a good initialization point, we further incorporate back-translation @xcite and dual reinforcement learning (section 2.2.2) tasks during the cycle learning phase. in this phase, one encoderdecoder model acts as the environment to provide pseudo-samples and reward signals for another. we conduct extensive experiments on benchmarks overnight and geogranno, both in unsupervised and semi-supervised settings. the results show that our method obtains significant improvements over various baselines in unsupervised settings. with full labeled data, we achieve new state-of-the-art performances (80.1% on overnight and 74.5% on geogranno), not considering additional data sources. the main contributions of this work can be summarized as follows: no supervision is provided in the first stage between input natural language utterances and intermediate output canonical utterances. ## training details in this section, we elaborate on different types of noise used in our experiment and the reward design in dual reinforcement learning. ## experiment in this section, we evaluate our system on benchmarks overnight and geogranno in both un-supervised and semi-supervised settings. our implementations are public available foot_1 . overnight it contains natural language paraphrases paired with logical forms over 8 domains. we follow the traditional 80%/20% train/valid to choose the best model during training. canonical utterances are generated with tool sempre foot_2 paired with target logical forms @xcite . due to the limited number of grammar rules and its coarse-grained nature, there is only one canonical utterance for each logical form, whereas 8 natural language paraphrases for each canonical utterance on average. for example, to describe the concept of "larger", in natural language utterances, many synonyms, such as "more than", "higher", "at least", are used, while in canonical utterances, the expression is restricted by grammar. geogranno due to the language mismatch problem @xcite , annotators are prone to reuse the same phrase or word while paraphrasing. geogranno is created via detection instead of paraphrasing. natural language utterances are firstly collected from query logs. crowd workers are required to select the correct canonical utterance from candidate list (provided by an incrementally trained score function) per input. we follow exactly the same split (train/valid/test 487/59/278) in original paper @xcite . ## conclusion in this work, aiming to reduce annotation, we propose a two-stage semantic parsing framework. the first stage utilizes the dual structure of an unsupervised paraphrase model to rewrite the input natural language utterance into canonical utterance. three self-supervised tasks, namely denoising auto-encoder, back-translation and dual reinforcement learning, are introduced to iteratively improve our model through pre-training and cycle learning phases. experimental results show that our framework is effective, and compatible with supervised training.