Improving Unsupervised Question Answering via Summarization-Informed Question Generation (2021)

Abstract: Question Generation (QG) is the task of generating a plausible question for a given <passage, answer> pair. Template-based QG uses linguistically-informed heuristics to transform declarative sentences into interrogatives, whereas supervised QG uses existing Question Answering (QA) datasets to train a system to generate a question given a passage and an answer. A disadvantage of the heuristic approach is that the generated questions are heavily tied to their declarative counterparts. A disadvantage of the supervised approach is that they are heavily tied to the domain/language of the QA dataset used as training data. In order to overcome these shortcomings, we propose a distantly-supervised QG method which uses questions generated heuristically from summaries as a source of training data for a QG system. We make use of freely available news summary data, transforming declarative summary sentences into appropriate questions using heuristics informed by dependency parsing, named entity recognition and semantic role labeling. The resulting questions are then combined with the original news articles to train an end-to-end neural QG model. We extrinsically evaluate our approach using unsupervised QA: our QG model is used to generate synthetic QA pairs for training a QA model. Experimental results show that, trained with only 20k English Wikipedia-based synthetic QA pairs, the QA model substantially outperforms previous unsupervised models on three in-domain datasets (SQuAD1.1, Natural Questions, TriviaQA) and three out-of-domain datasets (NewsQA, BioASQ, DuoRC), demonstrating the transferability of the approach.

## introduction the aim of question generation (qg) is the production of meaningful questions given a set of input passages and corresponding answers, a task with many applications including dialogue systems as well as education @xcite . additionally, qg can be applied to question answering (qa) for the purpose of data augmentation @xcite where @xmath0, answer, ques-@xmath1, answer, @xmath2, and unsupervised qa @xcite , in which only the qg system output is used to train the qa system. early work on qg focused on template or rulebased approaches, employing syntactic knowledge to manipulate constituents in declarative sentences to form interrogatives @xcite . although template-based methods are capable of generating linguistically correct questions, the resulting questions often lack variety and incur high lexical overlap with corresponding declarative sentences. for example, the question generated from the sentence stephen hawking announced the party in the morning, with stephen hawking as the candidate answer span, could be who announced the party in the morning?, with a high level of lexical overlap between the generated question and the declarative sentence. this is undesirable in a qa system @xcite since the strong lexical clues in the question would make it a poor test of real comprehension. neural seq2seq models @xcite have come to dominate qg @xcite , and are commonly trained @xmath3, answer, @xmath4. furthermore, the process of constructing such datasets involves a significant investment of time and resources. we subsequently propose a new unsupervised approach that frames qg as a summarizationquestioning process. by employing freely available summary data, we firstly apply dependency parsing, named entity recognition and semantic role labeling to summaries, before applying a set of heuristics that generate questions based on parsed summaries. an end-to-end neural generation system is then trained employing the original news articles as input and the heuristically generated questions as target output. an example is shown in figure 1 . the summary is used as a bridge between the questions and passages. because the questions are generated from the summaries and not from the original passages, they have less of a lexical overlap with the passages. crucially, however, they remain semantically close to the passages since the summaries by definition contain the most important information contained in the passages. a second advantage of this qg approach is that it does not rely on the existence of a qa dataset, and it is arguably easier to obtain summary data in a given language than equivalent qa data since summary data is created for many purposes (e.g. news, review and thesis summaries) whereas many qa datasets are created specifically for training a qa system. in order to explore the effectiveness of our method, we carry out extensive experiments. we provide an extrinsic evaluation, and train an english qg model using news summary data. we employ our qg model to generate synthetic qa data to train a qa model in an unsupervised setting and test the approach with six english qa datasets: squad1.1, natural questions, triviaqa, newsqa, bioasq and duorc @xcite @xcite @xcite . experiment results show that our approach substantially improves over previous unsupervised qa models even when trained on substantially fewer synthetic qa examples. our contributions can be summarized as follows: ## related work question generation traditional approaches to qg mostly employ linguistic templates and rules to transform declarative sentences into interrogatives @xcite . recently, dhole and manning (2020) showed that, with the help of advanced neural syntactic parsers, template-based methods are capable of generating high-quality questions from texts. neural seq2seq generation models have additionally been widely employed in qg, with qg data usually borrowed from existing qa datasets @xcite @xcite . furthermore, reinforcement learning has been employed by @xcite ; @xcite ; @xcite to directly optimize discrete evaluation metrics such as bleu @xcite . @xcite and @xcite show that a large-scale pre-trained model can achieve state-of-the-art performance for supervised qg @xcite . question generation evaluation bleu @xcite , rouge @xcite and meteor @xcite metrics are commonly borrowed from text generation tasks to evaluate qg. even with respect to original text generation tasks, however, the use of such metrics has been questioned @xcite . such metrics are particularly problematic for qg evaluation since multiple plausible questions exist for a given passage and answer. consequently, there has been a shift in focus to evaluating qg using an extrinsic evaluation that generates synthetic qa pairs for the purpose of evaluating their effectiveness as a data augmentation or unsupervised qa approach @xcite @xcite . unsupervised qa in unsupervised qa, the qa model is trained using synthetic data based on a qg model instead of an existing qa dataset. instead of resorting to existing qa datasets, unsupervised qg methods have been employed, such as unsupervised neural machine translation @xcite . fabbri et al. (2020) and li et al. (2020) propose template/rule-based methods for generating questions and employ retrieved paragraphs and cited passages as source passages to alleviate the problems of lexical similarities between passages and questions. alberti et al. (2019); puri et al. (2020); shakeri et al. (2020) additionally employ existing qa datasets to train a qg model. although related, this work falls outside the scope of unsupervised qa. ## methodology diverging from supervised neural question generation models trained on existing qa datasets, the approach we propose employs synthetic qg data, that we create from summary data using a number of heuristics, to train a qg model. we provide an overview of the proposed method is shown in figure 2 . we then employ the trained qg model to generate synthetic qa data that is further employed to train an unsupervised qa model. ## experiments we test our idea of using summaries in question generation by applying the questions generated by our qg system in unsupervised qa. we describe the details of our experiment setup, followed by our unsupervised qa results on six english benchmark extractive qa datasets. ## conclusion we propose an unsupervised question generation method which uses summarization data to 1) minimize the lexical overlap between passage and question and 2) provide a qa-dataset-independent way of generating questions. our unsupervised qa extrinsic evaluation on squad1.1, nq and triviaqa using synthetic qa data generated by our method shows that our method substantially outperforms previous methods for generating synthetic qa for unsupervised qa. furthermore, our synthetic qa data transfers well to the out-of-domain datasets. future work includes refining our question generation heuristics and applying our approach to other languages. passage answer question at a professional level, most matches produce only a few goals. for example, the 2005-06 season of the english premier league produced an average of 2.48 goals per match. the laws of the game do not specify any player positions other than goalkeeper, but a number of specialised roles have evolved. the 2005-06 season when did the english football team produce an average of 2.49 goals per match , according to the laws of the game ? the hebrew book week is held each june and features book fairs, public readings, and appearances by israeli authors around the country. during the week, israel's top literary award, the sapir prize, is presented. the hebrew book week what is held every june to celebrate the publication of books in hebrew ? on december 12, 2016, senate majority leader republican mitch mc-connell expressed confidence in u.s. intelligence. mcconnell added that investigation of russia's actions should be bipartisan and held by the senate intelligence committee. the next day, senate intelligence committee chairman richard burr (r-nc) and vice chairman mark warner (d-va) announced the scope of the committee's . republican mitch mc-connell which republican has called for a special committee to investigate russia 's alleged meddling in the 2016 presidential election ? meanwhile, the soho mint struck coins for the east india company, sierra leone and russia, while producing high-quality planchets, or blank coins, to be struck by national mints elsewhere. the firm sent over 20 million blanks to philadelphia, to be struck into cents and half-cents by the united states mint -mint director elias boudinot found them to be "perfect and beautifully polished". elias boudinot who has been working for a company that made coins for the us mint ? passage answer question in march 2008 as part of the annual budget, the government introduced several laws to amend the immigration and refugee protection act. the changes would have helped to streamline immigrant application back-up, to speed up application for skilled workers and to rapidly reject other ones that are judged not admissible by immigration officers. immigrant applications had risen to a high of 500,000, creating a delay of up to six months for an application to be processed.