Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering (2020)

Abstract: Question Answering (QA) is in increasing demand as the amount of information available online and the desire for quick access to this content grows. A common approach to QA has been to fine-tune a pretrained language model on a task-specific labeled dataset. This paradigm, however, relies on scarce, and costly to obtain, large-scale human-labeled data. We propose an unsupervised approach to training QA models with generated pseudo-training data. We show that generating questions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance by allowing the model to learn more complex context-question relationships. Training a QA model on this data gives a relative improvement over a previous unsupervised model in F1 score on the SQuAD dataset by about 14%, and 20% when the answer is a named entity, achieving state-of-the-art performance on SQuAD for unsupervised QA.

## introduction question answering aims to answer a question based on a given knowledge source. recent advances have driven the performance of qa systems to above or near-human performance on qa datasets such as squad @xcite and natural questions @xcite thanks to pretrained language models such as bert @xcite , xlnet @xcite and roberta @xcite . fine-tuning these language models, however, requires largescale data for fine-tuning. creating a dataset for every new domain is extremely costly and practically infeasible. the ability to apply qa models on outof-domain data in an efficient manner is thus very 1 equal contribution 2 work done during internship at the aws ai labs figure 1 : question generation pipeline: the original context sentence containing a given answer is used as a query to retrieve a related sentence containing matching entities, which is input into our question-style converter to create qa training data. in particular, we focus on the machine reading comprehension setting in which the context is a given paragraph, and the qa model can only access this paragraph to answer a question. furthermore, we work on extractive qa, where the answer is assumed to be a contiguous sub-string of the context. a training instance for supervised reading comprehension consists of three components: a question, a context, and an answer. for a given dataset domain, a collection of documents can usually be easily obtained, providing context in the form of paragraphs or sets of sentences. answers can be gathered from keywords and phrases from the context. we focus mainly on factoid qa; the question concerns a concise fact. in particular, we emphasize questions whose answers are named entities, the majority type of factoid questions. entities can be extracted from text using named entity recognition (ner) techniques as the training instance's answer. thus, the main challenge, and the focus of this paper, is creating a relevant question from a (context, answer) pair in an unsupervised manner. recent work of @xcite uses style transfer for generating questions for (context, answer) pairs but shows little improvement over applying a much simpler question generator which drops, permutates and masks words. we improve upon this paper by proposing a simple, intuitive, retrieval and template-based question generation approach, illustrated in figure 1 . the idea is to retrieve a sentence from the corpus similar to the current context, and then generate a question based on that sentence. having created a question for all (context, answer) pairs, we then fine-tune a pretrained bert model on this data and evaluate on the squad v1.1 dataset @xcite . our contributions are as follows: we introduce a retrieval, template-based framework which achieves state-of-the-art results on squad for unsupervised models, particularly when the answer is a named entity. we perform ablation studies to determine the effect of components in template question generation. we are releasing our synthetic training data and code. foot_0 2 unsupervised qa approach we focus on creating high-quality, non-trivial questions which will allow the model to learn to extract the proper answer from a context-question pair. sentence retrieval: a standard cloze question can be obtained by taking the original sentence in which the answer appears from the context and masking the answer with a chosen token. however, a model trained on this data will only learn text matching and how to fill-in-the-blank, with little generalizability. for this reason, we chose to use a retrieval-based approach to obtain a sentence similar to that which contains the answer, upon which to create a given question. for our experiments, we focused on answers which are named entities, which has proven to be a useful prior assumption for downstream qa performance @xcite confirmed by our initial experiments. first, we indexed all of the sentences from a wikipedia dump using the elasticsearch search engine. we also extract named entities for each sentence in both the wikipedia corpus and the sentences used as queries. we assume access to a named-entity recognition system, and in this work make use of the spacy foot_1 ner pipeline. then, for a given context-answer pair, we query the index, using the original context sentence as a query, to return a sentence which ( 1 ) contains the answer, (2) does not come from the context, and (3) has a lower than 95% f1 score with the query sentence to discard highly similar or plagiarized sentences. besides ensuring that the retrieved sentence and query sentence share the answer entity, we require that at least one additional matching entity appears in both the query sentence and in the entire context, and we perform ablation studies on the effect of this matching below. these retrieved sentences are then fed into our question-generation module. template-based question generation: we consider several question styles (1) generic clozestyle questions where the answer is replaced by the token "[mask]", (2) templated question "wh+b+a+?" as well as variations on the ordering of this template, as shown in figure 2 . given the retrieved sentence in the form of [fragment a] [answer] [fragment b], the templated question "wh+b+a+?" replaces the answer with a wh-component (e.g., what, who, where), which depends on the entity type of the answer and places the wh-component at the beginning of the question, followed by sentence fragment b and fragment a. for the choice of wh-component, we sample a bi-gram based on prior probabilities of that bi-gram being associated with the named-entity type of the answer. this prior probability is calculated based on named-entity and question bi-gram starters from the squad dataset. this information does not make use of the full context-question-answer and can be viewed as prior information, not disturbing the integrity of our unsupervised approach. additionally, the choice of wh component does not significantly affect results. for template-based approaches, we also experimented with clause-based templates but did not find significant differences in performance. ## conclusion in this paper we introduce a retrieval-based approach to unsupervised extractive question answering. a simple template-based approach achieves state-of-the-art results for unsupervised methods on the squad dataset of 64.04 f1, and 77.55 f1 when the answer is a named entity. we analyze the effect of several components in our template-based approaches through ablation studies. we aim to experiment with other datasets and other domains, incorporate our synthetic data in a semi-supervised setting and test the feasibility of our framework in a multi-lingual setting.