Bridging the Gap between Synthetic and Natural Questions via Sentence Decomposition for Semantic Parsing (2023)

Abstract: Semantic parsing maps natural language questions into logical forms, which can be executed against a knowledge base for answers. In real-world applications, the performance of a parser is often limited by the lack of training data. To facilitate zero-shot learning, data synthesis has been widely studied to automatically generate paired questions and logical forms. However, data synthesis methods can hardly cover the diverse structures in natural languages, leading to a large gap in sentence structure between synthetic and natural questions. In this paper, we propose a decomposition-based method to unify the sentence structures of questions, which benefits the generalization to natural questions. Experiments demonstrate that our method significantly improves the semantic parser trained on synthetic data (+7.9% on KQA and +8.9% on ComplexWebQuestions in terms of exact match accuracy). Extensive analysis demonstrates that our method can better generalize to natural questions with novel text expressions compared with baselines. Besides semantic parsing, our idea potentially benefits other semantic understanding tasks by mitigating the distracting structure features. To illustrate this, we extend our method to the task of sentence embedding learning, and observe substantial improvements on sentence retrieval (+13.1% for Hit@1).

## introduction semantic parsing is a task of mapping natural language questions into logical forms, which serves as a backbone in knowledge base question answering @xcite @xcite and task-oriented dialogue @xcite . traditionally, learning a powerful semantic parser relies on large-scale annotated data, which is laborious to collect. to reduce the budget of annotation, some recent efforts have been dedicated into low-resource training methods, among which data synthesis is widely used. data synthesis can efficiently generate large amounts of training data by means of templates and rules @xcite , but the templated expressions of synthetic questions hinder the generalization to natural questions. the gap between synthetic and natural questions mainly derives from two kinds of textual features, phrase-level expression and sentence-level structure. the former refers to the myriad ways in which predicates, relations, and entities can be expressed @xcite , such as ''the human whose date of birth is 2000'' and ''the person born in 2000''. the latter usually manifests itself as the global rearrangement of content (see figure 1 for an example). to alleviate the problem, paraphrasing is commonly used for increasing language diversity of synthetic data @xcite . although the paraphrasing methods successfully introduce diverse phrase-level expressions, they are still weak in generating sentences with diverse structures @xcite ; this is because the generation models tend to assign significantly higher generative probability to the sentences with similar structures, making it challenging to control or diversify the structures of the generated sentences. in contrast to the paraphrase approaches that bridge the gap by augmenting the synthetic data, we propose to unify the sentence-level structure by decomposing the questions into several simple facts (see figure 1 ). intuitively, the question decomposition maps the natural and synthetic questions into a smaller text space that only contains simple sentences, which is more controllable than generating diverse synthetic questions to cover possible sentence structures. specifically, we decompose the questions with an in-context learning method, where the prompt is composed of several (about 10) synthetic questions as well as their decomposition results. then we conduct semantic parsing with the decomposed facts as inputs, instead of the original questions. the decomposing-then-parsing process takes place during both the training with synthetic data and the inference for natural questions. in this way, synthetic and natural data share the similar distributions of input structures, leading to better generalization to natural questions. we conduct a comprehensive evaluation of our method on two semantic parsing datasets, kqa @xcite and complexwebquestion (cwq) @xcite . our method achieves a 76.7% and 51.0% exact match accuracy on kqa and cwq, respectively-a 7.9% and 8.9% improvement compared to the best baselines. moreover, although only synthetic data is used for training, our method has approached the performance of the semantic parser trained with full natural data on cwq. extensive analysis demonstrates that the improvements mainly derive from the generalization to the natural questions with novel text expressions. besides semantic parsing, our idea can be applied to other semantic understanding tasks to mitigate distracting structure features. when extended for the task of sentence embedding learning, our method achieves a substantial improvement on sentence retrieval (+13.1% in terms of hit@1). our contributions are summarized as follows: ## method to improve the low-resource semantic parsers only trained on synthetic data, we propose a novel method that decomposes the input question into several facts before the parsing, aiming to reduce the gap between synthetic and natural data. as shown in figure 2 , our model consists of two modules, a decomposer that breaks the question into simple facts (section 3.1) and a parser that receives the facts to do the parsing (section 3.2). moreover, our idea can also assist other semantic understanding tasks, for example, sentence embedding learning (section 3.3), which illustrates the potential of our method. ## limitations in this section, we enumerate three limitations of our method: applicability, error propagation, and inference speed. (1) our method is designed to improve the robustness of models to distracting structure features, such as the structure gap between synthetic and natural questions. if there are no distracting structure features, such as on sts datasets, where the two sentences in a sample usually have similar structures, our method may not achieve an advantage. (2) our method needs to decompose the input text, which may introduce errors that mislead the training and inference of models. ## conclusion we present a decomposition-based low-resource semantic parsing method, which can narrow the data gap and thus better generalize from synthetic data to natural data. unlike previous methods that tackle the issue by diversifying synthetic questions, we leverage sentence decomposition to break questions into simple facts, which are more method cwq kqa t5-synthetic 42.1 68.8 our method t5-decomposer 49.8 76.5 codex-decomposer 51.0 76.7