Harvesting and Refining Question-Answer Pairs for Unsupervised QA (2020)

Abstract: Question Answering (QA) has shown great success thanks to the availability of large-scale datasets and the effectiveness of neural models. Recent research works have attempted to extend these successes to the settings with few or no labeled data available. In this work, we introduce two approaches to improve unsupervised QA. First, we harvest lexically and syntactically divergent questions from Wikipedia to automatically construct a corpus of question-answer pairs (named as RefQA). Second, we take advantage of the QA model to extract more appropriate answers, which iteratively refines data over RefQA. We conduct experiments on SQuAD 1.1, and NewsQA by fine-tuning BERT without access to manually annotated data. Our approach outperforms previous unsupervised approaches by a large margin, and is competitive with early supervised models. We also show the effectiveness of our approach in the few-shot learning setting.

## introduction extractive question answering aims to extract a span from the given document to answer the question. rapid progress has been made because of the release of large-scale annotated datasets @xcite @xcite , and well-designed neural models @xcite @xcite . recently, unsupervised pre-training of language models on large corpora, such as bert @xcite , has brought further performance gains. however, the above approaches heavily rely on the availability of large-scale datasets. the collection of high-quality training data is timeconsuming and requires significant resources, es-pecially for new domains or languages. in order to tackle the setting in which no training data available, @xcite leverage unsupervised machine translation to generate synthetic contextquestion-answer triples. the paragraphs are sampled from wikipedia. ner and noun chunkers are employed to identify answer candidates. cloze questions are first extracted from the sentences of the paragraph, and then translated into natural questions. however, there are a lot of lexical overlaps between the generated questions and the paragraph. similar lexical and syntactic structures render the qa model tend to predict the answer just by word matching. moreover, the answer category is limited to the named entity or noun phrase, which restricts the coverage of the learnt model. in this work, we present two approaches to improve the quality of synthetic context-questionanswer triples. first, we introduce the refqa dataset, which harvests lexically and syntactically divergent questions from wikipedia by using the cited documents. as shown in figure 1 , the sentence (statement) in wikipedia and its cited documents are semantically consistent, but written with different expressions. more informative context-question-answer triples can be created by using the cited document as the context paragraph and extracting questions from the statement in wikipedia. second, we propose to iteratively refine data over refqa. given a qa model and some refqa examples, we first filter its predicted answers with a probability threshold. then we refine questions based on the predicted answers, and obtain the refined question-answer pairs to continue the model training. thanks to the pretrained linguistic knowledge in the bertbased qa model, there are more appropriate and diverse answer candidates in the filtered predictions, some of which do not appear in the candidates extracted by ner tools. we also show that iteratively refining the data further improves model performance. we conduct experiments on squad 1.1 @xcite , and newsqa @xcite . our method yields state-of-the-art results against strong baselines in the unsupervised setting. specifically, the proposed model achieves 71.4 f1 on the squad 1.1 test set and 45.1 f1 on the newsqa test set without using annotated data. we also evaluate our method in a few-shot learning setting. our approach achieves 79.4 f1 on the squad 1.1 dev set with only 100 labeled examples, compared to 63.0 f1 using the method of @xcite . to summarize, the contributions of this paper include: i) refqa constructing in an unsupervised manner, which contains more informative context-question-answer triples. ii) using the qa model to iteratively refine and augment the question-answer pairs in refqa. ## related work extractive question answering given a document and question, the task is to predict a continuous sub-span of the document to answer the question. extractive question answering has garnered a lot of attention over the past few years. benchmark datasets, such as squad @xcite , newsqa @xcite and triviaqa @xcite , play an important role in the progress. in order to improve the performance on these benchmarks, several models have been proposed, including bidaf @xcite , r-net @xcite , and qanet @xcite . recently, unsupervised pre-training of language models such as bert @xcite , achieves significant improvement. however, these powerful models rely on the availability of human-labeled data. large annotated corpora for a specific domain or language are limited and expensive to construct. semi-supervised qa several semi-supervised approaches have been proposed to utilize unlabeled data. neural question generation (qg) models are used to generate questions from unlabeled passages for training qa models @xcite @xcite . however, the methods require labeled data to train the sequence-to-sequence qg model. @xcite propose to collect synthetic context-question-answer triples by gen-erating cloze-style questions from the wikipedia summary paragraphs in an unsupervised manner. unsupervised qa @xcite have explored the unsupervised method for qa. they create synthetic qa data in four steps. i) sample paragraphs from the english wikipedia. ii) use ner or noun chunkers to extract answer candidates from the context. iii) extract "fill-inthe-blank" cloze-style questions given the candidate answer and context. iv) translate cloze-style questions into natural questions by an unsupervised translator. compared with @xcite , @xcite attempt to generate natural questions by training an unsupervised neural machine translation (nmt) model. they train the nmt model on non-aligned corpora of natural questions and cloze questions. the unsupervised qa model of @xcite achieves promising results, even outperforms early supervised models. however, their questions are generated from the sentences or sub-clauses of the same paragraphs, which may lead to a biased learning of word matching since its similar lexicons and syntactic structures. besides, the category of answer candidates is limited to named entity or noun phrase, which restricts the coverage of the learnt qa model. ## harvesting refqa from wikipedia in this section, we introduce refqa, a question answering dataset constructed in an unsupervised manner. one drawback of @xcite is that questions are produced from the paragraph sentence that contains the answer candidate. so there are considerable expression overlaps between generated questions and context paragraphs. in contrast, we harvest informative questions by taking advantage of wikipedia's reference links, where lexical and syntactic differences exist between the article and its cited documents. as shown in figure 1 , given statements in wikipedia paragraphs and its cited documents, we use the cited documents as the context paragraphs and generate questions from the sub-clauses of statements. in order to generate question-answer pairs, we first find answer candidates that appear in both sub-clauses and context paragraphs. next, we convert sub-clauses into the cloze questions based on the candidate answers. we then conduct cloze-to-natural-question translation by a depen- in august 2013, guillermo crashed a matt damon interview, about his upcoming movie elysium, by promoting his own movie called "estupido", about a stupid man, which poster had an arrow pointing towards matt damon. ## iterative data refinement in this section, we propose to iteratively refine data over refqa based on the qa model. as shown in figure 3 , we use the qa model to filter refqa data, find appropriate and diverse answer candidates, and use these answers to refine and augment refqa examples. filtering data can get rid of some noisy examples in refqa, and pretrained linguistic knowledge in the bert-based qa model finds more appropriate and diverse answers. we produce questions for the refined answers, then continue to train the qa model on the refined and filtered triples. ## experiments we evaluate our proposed method on two widely used extractive qa datasets @xcite . we also demonstrate the effectiveness of our approach in the few-shot learning setting. ## conclusion in this paper, we present two approaches to improve the quality of synthetic qa data for unsupervised question answering. we first use the wikipedia paragraphs and its references to construct a synthetic qa data refqa and then use the qa model to iteratively refine data over re-fqa. our method outperforms the previous unsupervised state-of-the-art models on squad 1.1, and newsqa, and achieves the best performance in the few-shot learning setting. oa⊃pa s: in 1938, e. allen petersen escaped the advancing japanese armies by sailing a junk, "hummel hummel", from shanghai to california with his wife tani and two white russians (tsar loyalists). q: who escaped the advancing japanese armies by sailing a junk oa: e. allen petersen pa: petersen rq: who escaped the advancing japanese armies by sailing a junk oa⊂pa s: hyundai announced they would be revealing their future rally plans at the 2011 chicago auto show on february 9 . q: what at they would be revealing their future rally plans on february 9 oa: chicago auto show pa: the 2011 chicago auto show rq: what at their future rally plans they would be revealing on february 9 others s: in january 2017, she released the track "that's what's up" that re-imagines the spoken word segment on the kanye west song "low lights". q: what the kanye west song on the spoken word segment re-imagines oa: low lights pa: that's what's up rq: what the track she released that re-imagines the spoken word segment on the kanye west song "low lights" .