Understanding Unnatural Questions Improves Reasoning over Text (2020)

Abstract: Complex question answering (CQA) over raw text is a challenging task. A prominent approach to this task is based on the programmer-interpreter framework, where the programmer maps the question into a sequence of reasoning actions and the interpreter then executes these actions on the raw text. Learning an effective CQA model requires large amounts of human-annotated data, consisting of the ground-truth sequence of reasoning actions, which is time-consuming and expensive to collect at scale. In this paper, we address the challenge of learning a high-quality programmer (parser) by projecting natural human-generated questions into unnatural machine-generated questions which are more convenient to parse. We firstly generate synthetic (question, action sequence) pairs by a data generator, and train a semantic parser that associates synthetic questions with their corresponding action sequences. To capture the diversity when applied to natural questions, we learn a projection model to map natural questions into their most similar unnatural questions for which the parser can work well. Without any natural training data, our projection model provides high-quality action sequences for the CQA task. Experimental results show that the QA model trained exclusively with synthetic data outperforms its state-of-the-art counterpart trained on human-labeled data.

## introduction the complex question answering (cqa) task, which requires multi-step, discrete actions to be executed over text to obtain answers, is a challenging task. on the recently released drop benchmark @xcite , the state-of-the-art method neural module networks (nmns) @xcite learns to interpret each question as a sequence of neural modules, or discrete actions, and execute them to yield the answer. however, the supervised learning of cqa models such as nmns requires a large amount of annotated (question, action sequence) pairs, which are expensive to acquire and augment. therefore, the label scarcity problem remains a challenge to the cqa problem. motivated by this, we propose a projection model to alleviate the label scarcity challenge by generating synthetic training data. to our knowledge, our model is the first approach using projection to solve the label scarcity problem for the cqa task. the projection model can automatically label large amounts of unlabelled questions with action sequences, so that a cqa model can be trained without natural supervised data. our method is inspired by the recent "simulation-to-real" transfer approach @xcite . as the name suggests, internal knowledge in synthetic data is firstly learned in the "simulation" phase, and then transferred "to-real" circumstances with natural-language utterance using the projection model. in the "simulation" phase, we design an n-gram based generator to produce synthetic training data, i.e. (question, action sequence) pairs which constitute the synthetic dataset. we then train a semantic parser to learn the internal knowledge in this dataset. in the "to-real" phase, we train a projection model that projects each unlabelled natural-language question to a synthetic question and obtain the corresponding action sequence by interpreting the synthetic question using the trained semantic parser. in this way, the internal knowledge is implicitly transferred from "simulation" phase  a specific model f is trained on the synthetic dataset.  a projection model is employed to project natural-language questions to synthetic questions. note that  and  belong to the "simulation" phase, while  belongs to the "to-real" phase. experimental results on the challenging drop dataset demonstrate the effectiveness and practicability of our projection model. based on the synthetic dataset produced by the data generator, our projection model help nmns model achieve a 78.3 f1 score on the drop development dataset. the result indicates the promise of "simulation-to-real" as a development paradigm for nlp problems. our contributions are as follows: ## background the complex question answering (cqa) task aims to generate answers for questions that require a thorough understanding of questions and contents such as knowledge base @xcite or paragraphs. one recent drop dataset @xcite fits this case exactly and requires discrete reasoning over contents of paragraphs, which are extracted from wikipedia. meanwhile, all questions and corresponding answers are generated and annotated by human workers. the questions in drop are complex in nature, and are thus especially challenging to existing qa models such as bidaf @xcite . @xcite address this challenge with the end-to-end and fully differentiable neural module networks (nmns) @xcite and achieves state-of-the-art performance on drop. the nmns follows the programmer-interpreter structure and consists of two components: a parser (programmer) and an interpreter. the parser is an encoder-decoder model with attention and learns to interpret a natural-language question into an executable sequence of modules (i.e. actions, and thereafter we use the terms "module" and "action" interchangeably). the interpreter then takes the sequence along with the paragraph as inputs, and predicts the answer after executing the actions one-byone. these modules, such as f ind, f ilter and count, are defined to perform independent reasoning over text, numbers and dates. besides, actions take arguments from questions. for example, the question "how many yards was the first field goal" is interpreted as the first action f ind(field goal), where "field goal" is the argument of the "f ind" action. more details can be found in appendix a.1. for realizing the "simulation-to-real" projection model, we define a series of concepts as follows. formally, our goal is to obtain a semantic parser, i.e. a general model f : @xmath0. in this paper, we regard one action sequence as a whole object. thus, all actions are predicted by semantic parser simultaneously and in order. we approximate this general model by implementing another semantic parser, the specific model f : @xmath1, where q is the set of synthetic questions. note that the model f is trained on a synthetic dataset @xmath2, y i )}, where (x i ,y i ) represents a pair of (synthetic question, action sequence). in order to obtain the action sequence for a natural question x @xmath3, we propose a projection model p : @xmath4, with which we find a synthetic question for each natural question. therefore, the action sequence for a natural question could be computed using the specific model: f (x j ) = f (p(x j )). the specific model f can be trained using a supervised machine learning method on synthetic data only. therefore, it remains only to find the projection model p. ## model the overall structure of our model is shown in figure 1 . we will firstly introduce an n-gram based data generator in section 3.1. a cosine-similarity projection model and a classifier-based projection model will follow in section 3.2. experimental results demonstrate that the classifier-based project model achieves better performance in both action sequence annotation and question answering task. ## experiments in this section, we will evaluate two projection models in section 3.2 from two aspects. for one thing, they will be compared with each other, and for another, we employ the classifier-projection model to improve the performance of nmns @xcite . ## conclusion in this paper, we propose a projection model that only employs synthetic data to develop supervisions for real-world data. experimental results show that our approach can be equivalent to supervised learning on natural dataset in performance. in addition, with projection results, we employ the nmns model to solve complex question answering problem and generate answers for the drop dataset. higher f1, exact match scores demonstrate that our projection model can help improve the performance of the downstream cqa task and provide a good reference to relevant works.