Handling Anomalies of Synthetic Questions in Unsupervised Question Answering (2020)

Abstract: Advances in Question Answering (QA) research require additional datasets for new domains, languages, and types of questions, as well as for performance increases. Human creation of a QA dataset like SQuAD, however, is expensive. As an alternative, an unsupervised QA approach has been proposed so that QA training data can be generated automatically. However, the performance of unsupervised QA is much lower than that of supervised QA models. We identify two anomalies in the automatically generated questions and propose how they can be mitigated. We show our approach helps improve unsupervised QA significantly across a number of QA tasks.

## introduction machine reading question answering (mrqa) is a popular nlp task that attempts to answer questions from associated texts. a popular challenge is squad @xcite , which has 100k context, answer, question triples created by humans, and other challenging tasks and datasets have been created. however, it is costly to create a dataset for a new qa task or expand an existing one. ## limitations of unsupervised question generation the analyses in this section probe the nature of the uqg problems and set the stage for developing our methods that alleviate the problems and help to improve the performance on various qa tasks. ## methods given the analysis result, we propose two approaches to refine uqa questions with no supervision. the proposed methods 2 are: 1) paraphrasing questions, and 2) trimming undesirable questions. ## conclusion we have identified two anomalies of questions from unsupervised question answering (uqa) @xcite : excessive copying of the context and unanswerable questions. we proposed methods for the two: machine-translation-based paraphrasing of questions with the novel idea of using inverse bleu scores and the confidence-based pruning that trims low-quality questions. we show that the proposed approach can improve uqa performance across diverse qa tasks, providing new insights on how to improve it.