A uto QA : From Databases To QA Semantic Parsers With Only Synthetic Training Data (2020)

Abstract: We propose AutoQA, a methodology and toolkit to generate semantic parsers that answer questions on databases, with no manual effort. Given a database schema and its data, AutoQA automatically generates a large set of high-quality questions for training that covers different database operations. It uses automatic paraphrasing combined with template-based parsing to find alternative expressions of an attribute in different parts of speech. It also uses a novel filtered auto-paraphraser to generate correct paraphrases of entire sentences. We apply AutoQA to the Schema2QA dataset and obtain an average logical form accuracy of 62.9% when tested on natural questions, which is only 6.4% lower than a model trained with expert natural language annotations and paraphrase data collected from crowdworkers. To demonstrate the generality of AutoQA, we also apply it to the Overnight dataset. AutoQA achieves 69.8% answer accuracy, 16.4% higher than the state-of-the-art zero-shot models and only 5.2% lower than the same model trained with human data.

## introduction semantic parsing is the task of mapping natural language sentences to executable logical forms. it has received significant attention in question answering systems for structured data @xcite @xcite . however, training a semantic parser with good accuracy requires a large amount of annotated data, which is expensive to acquire. the complexity of logical forms means annotating the data has to be done by an expert. this adds to the cost and hinders extending question answering to new databases and domains. @xcite generates data from the annotations to train a semantic parser. (c) an auto-paraphraser uses self-training to iteratively introduce more paraphrases to train the next version of the semantic parser. the red dotted lines show that generated paraphrases are filtered out unless the current semantic parser can translate them to the logical forms of the original sentences. ## schema2qa data synthesis pipeline autoqa is based on schema2qa @xcite , the state-of-the-art pipeline to generate highquality training data for database qa at a low cost. schema2qa first synthesizes utterance and formal representation pairs with a template-based algorithm, and then paraphrases utterances via crowdsourcing. the semantic parser is trained with both synthetic and paraphrased data, and tested on crowdsourced, manually annotated real questions. instead of relying on crowdworkers to paraphrase and create variety from the synthesized canonical questions, schema2qa uses a comprehensive set of 800 domain-independent templates, along with a few manual annotations for each attribute in each domain, to synthesize high-quality data. about 2% of the synthesized data are manually paraphrased. our previous work @xcite shows that a parser trained on such dataset achieves 70% accuracy on natural complex questions. schema2qa answers long-tail questions well because its synthesized data have good coverage of possible questions asked, while showing great linguistic variety. it synthesizes questions using generic question templates, which have placeholders to be substituted with domain-specific annotations that match the expected part-of-speech (pos) type. pos annotation example template example utterance is-a noun alumni of value table that|which|who is|are [noun phrase] value people who are alumni of stanford has-a noun a value degree table with (a|an|the) value [noun phrase] people with a stanford degree active verb graduated from value table that|which|who [verb phrase] value people who graduated from stanford passive verb educated at value table [passive verb phrase] value people educated at stanford adjective value value table stanford people prepositional from value table [prepositional phrase] value people from stanford ## automatic annotation our autoqa toolkit automatically provides unambiguous attribute annotations for all parts of speech, with the help of a neural paraphrasing model. ## automatic paraphrasing synthetic training data is good for providing coverage with a large number of perfectly annotated sentences, and to teach the neural semantic parser compositionality. however, grammar-based synthesis often results in clunky sentences and grammatical errors. in addition, even with 800 generic templates, the synthesized sentences still lack naturalness and variety. in particular, people often compress multiple concepts into simpler constructions (sublexical compositionality @xcite ), e.g. "books with at least 1 award" can be simplified to "award-winning books". capturing these linguistic phenomena in the training data is not possible with a finite set of templates. this is why paraphrasing is critical when training semantic parsers. here we describe how we approximate manual paraphrases with a neural paraphrasing model. ## experiments in this section, we evaluate the effectiveness of our methodology: can a semantic parser created with autoqa approach the performance of humanwritten annotations and paraphrases? we evaluate on two different benchmark datasets: the schema2qa dataset @xcite and the overnight dataset @xcite . ## discussion in this work, we propose autoqa, a methodology and a toolkit to automatically create a semantic parser given a database. we test autoqa on two different datasets with different target logical forms and data synthesis templates. on both datasets, autoqa achieves comparable accuracy to state-ofthe-art qa systems trained with manual attribute annotation and human paraphrases. autoqa relies on a neural paraphraser trained with an out-of-domain dataset to generate training data. we suspect the methodology to be less effective for domains full of jargon. even for common domains, autoqa sometimes failed to generate some common phrases. further improvement on neural paraphraser is needed to generate more diverse outputs. future work is also needed to handle attributes containing long free-form text, as autoqa currently only supports database operations without reading comprehension. the approximate total per-domain cost of schema2qa experiments using amazon web services is $36.