PIE - QG : Paraphrased Information Extraction for Unsupervised Question Generation from Small Corpora (2022)

Abstract: Supervised Question Answering systems (QA systems) rely on domain-specific human-labeled data for training. Unsupervised QA systems generate their own question-answer training pairs, typically using secondary knowledge sources to achieve this outcome. Our approach (called PIE-QG) uses Open Information Extraction (OpenIE) to generate synthetic training questions from paraphrased passages and uses the question-answer pairs as training data for a language model for a state-of-the-art QA system based on BERT. Triples in the form of <subject, predicate, object> are extracted from each passage, and questions are formed with subjects (or objects) and predicates while objects (or subjects) are considered as answers. Experimenting on five extractive QA datasets demonstrates that our technique achieves on-par performance with existing state-of-the-art QA systems with the benefit of being trained on an order of magnitude fewer documents and without any recourse to external reference data sources.

## introduction question answering systems (qa systems) provide answers to input questions posed in natural language. answering questions from unstructured text can be performed using machine reading comprehension (mrc). given a passage, several sentences or a paragraph, and a question posed, the qa system produces the best suitable answer. extractive question answering systems (eqa systems) are a subset of qa systems and involve an mrc task where the predicted answer is a span of words from the passage. with pre-trained language models @xcite , eqa systems achieve excellent results, surpassing even human performance. pre-trained language models, such as bert @xcite and gpt @xcite , can be fine-tuned to perform downstream tasks such as qa. however, huge amounts of data are required to train these models for specific domains, making the task labor-intensive, in terms of the effort required to assemble suitable domainspecific training data. a single training instance for an eqa system dataset requires a question, a passage, and an answer. domain-relevant documents can be collected with advanced information retrieval tools, and passages are formed by splitting documents into several related sentences or a paragraph. however, generating the question and answer pairs, that provide the training set for the qa system from a given passage, is considered the most difficult challenge, an approach known as unsupervised qa @xcite . existing unsupervised qa system techniques such as @xcite and @xcite use an out-of-domain dataset for question generation, namely, they require additional training sources beyond what can be provided by the target corpus and a pre-trained generic model. on the other hand, rule-based qa system methods, those constrained to generate question-answer pairs from only the corpus itself, run the risk of generating questions with high lexical overlap with the passage, at risk of forcing the model to learn word matching patterns. the work of @xcite and @xcite use information retrieval-based methods, such as elastic search and citation navigation, to create questions from passages other than those presented within the target dataset. however, these methods may not generate sufficient training questions, especially when the corpus is small and has no citation or inter-document linking structure. in this paper, we focus on addressing the limitations of eqa systems using a novel unsupervised paraphrased information extraction for question generation (pie-qg) method that generates synthetic training data through the extraction @xmath0, relation, @xmath1. we use the original passage to produce question-paraphrasing the european commission , which is responsible for regulation competition in the european union , is concerned that these deals could violate eu antitrust laws. ## related work pre-trained language models, such as bert @xcite , can be fine-tuned for downstream tasks like extractive qa systems (eqa systems). a comprehensive natural language (nl) passage, which might be several sentences or a paragraph of nl-text, is considered as the context where the model finds the answer span. the input question and the context are represented as a single sequence, passed to a pre-trained model and the answer is predicted by calculating the probabilities of the first and last tokens of the answer span. pre-trained language models such as bert @xcite , t5 transformer @xcite and xlnet @xcite , achieve exceptional performance in eqa systems, however at the cost of reliance on large human-annotated supervised datasets. the stanford question answering dataset (squad) @xcite is a widely used dataset for eqa systems. lewis et al. (2019), fabbri et al. (2020), li et al. (2020), and lyu et al. (2021) used randomly sampled passages from wikipedia, where named entities, or noun chunks, are identified as answers as these tend to be useful for question answering. the questions are then formed in natural language according to the passage and a selected answer phrase. ## paraphrased information extraction for question generation to overcome the reliance on external reference data sources with a large number of passages, we made use of openie and paraphrased passages for unsupervised synthetic question generation. the actual passages are first altered to a paraphrased form @xmath2, predicate, @xmath3. these triples, combined with certain heuristics, form question-answer pairs which are then used alongside the original passage as context to fine-tune the qa model. the pipeline of our proposed eqa question generation process is illustrated in figure 2 . the steps in this pipeline are detailed as follows. input :given a passage p from the corpus output :a list of question-answer pairs qa_pairs ← -append(⟨q, a, p ⟩) end return qa_p airs (or object) is selected as an answer only if it is a named entity. each triple extracted from a paraphrased passage can form two questions with either subject or object as an answer. when a subject is selected as an answer, the question is formulated as "wh + relation + object?". conversely, when an object is selected as the answer, the question generated is of the form "wh + subject + relation?". "wh" is the question word in these formulations and the appropriate form is selected from a list, based on the answer entity type as earlier described. ## experimental platform datasets the performance of our question generation method is evaluated in terms of exact-match (em) and f-1 score using existing eqa datasets, namely squad v1.1 @xcite development set, and newsqa @xcite , bioasq @xcite and duorc @xcite test sets. squad version 1.1 is acquired from the official version foot_0 while the fisch et al. ( 2019 ) published versions of test sets are considered for newsqa, bioasq, and duorc. a more recent squad v2.0 @xcite is considered unsuitable for our experiments as the synthetic training set does not contain unanswerable questions. question generation we take a relatively small subset of 30,000 passages from the @xcite sampled wikipedia dataset for question generation and for training the model. the pseudo-code for the proposed question generation technique is presented in algorithm 1. some of the questions resulting from this process can be grammatically incorrect. we rely on questions posed to the model during inference to be in natural language with correct grammar, we experiment by introducing a grammar correction module in the pipeline to synthesize syntactically accurate questions but later removed this due to its effect discussed in section 5. sourced wikipedia passages are transformed into paraphrased passages with a pre-trained model foot_1 based on the pegasus transformer @xcite . pronouns in the paraphrased passage are replaced with the nouns they refer to. we used neuralcoref foot_2 for this purpose, the spacy implementation of pre-trained co-referent resolution based on reinforcement learning @xcite . openie6 is used to @xmath4, predicate, @xmath5. openie6 uses iterative grid labeling and is based on bert. a spacy-based named-entity recognition (ner) module @xcite is used to generate a list of named-entities from the passage. named-entity recognition (ner) is particularly helpful for filtering triples and determining the answer-entity type for appropriate "wh" word selection. the simplest version of "wh" word is selected for a particular named entity based on @xcite . questions generated from this process are grammatically corrected using a roberta-based @xcite grammar correction module named "gector" @xcite . all models are applied from the above-mentioned sources out-of-the-box, namely with no domain specific fine-tuning. qa fine-tuning we use pre-trained bert models from @xcite as the baseline and fine-tune the models for downstream qa system tasks with the generated training data. the generated question, and its context (the actual nlpassage that contains both the question and its answer), are represented as a single sequence, separated by different segment masks and the "[sep]" token. the final linear layer of the model is trained, to identify the start and end spans of the answer, by computing log-likelihood for each token. all experiments are performed on the uncased version of the bert-base model with a learning rate of 3e-5, a maximum sequence length of 384, a batch size of 12, a document stride of 128 for 2 epochs, and a check-point at every 500 steps. the best checkpoint was selected by validating each against 5000 qa pairs randomly sampled from the synthetic training data. we use the huggingface foot_3 implementation for input tokenization, model initialization, and training. for comparison with the state-ofthe-art eqa models, we also experimented on the bert-large whole-word masking version with the same training data. all models are trained and validated on a single nvidia tesla a100 gpu. ## results and discussion the effectiveness of the question-answer data generated using the pie-qg method is measured by training the bert-base model and evaluating it against existing eqa development and test sets. the exact match (em) and f-1 scores are selected as the metrics to evaluate the effectiveness of each component in the qa models. the initial set of questions is created using openie, where the passage is directly used to form triples and generate questions as described in section 3. the intuition behind using openie is to generate multiple questions from a single passage. however, as previously described, such a simple-minded approach suffers from having pronouns as answers, ungrammatical questions, and high degrees lexical similarity between passage and question, making most extracted triples suitable for word matching only. ## limitations the downside of the pie-qg unsupervised question generation pipeline is the use of external modules like paraphrasing, openie, and ner, which may not exist in languages other than english. the quality of question-answer pairs generated to train the qa model is therefore dependent on the performance of these modules on the selected corpus. it is however anticipated that pie-qg will perform similarly well on any english language corpus. it is future work to apply these modules within the pie-qg pipeline to other languages where comparable language-specific models can be sourced and performance outcomes analyzed. ## conclusion with no reliance on any external reference corpora, the pie-qg model uses paraphrasing and open information extraction (openie) to generate synthetic training questions for fine-tuning the language model in a qa system based on bert. triples in the form @xmath6, predicate, ob-@xmath7, and questions are formed with subjects (or objects) as answers. pronoun co-referents are resolved and where possible, triples are merged, and duplicate and highly similar triples are removed. furthermore, triples that do not contain named entities p georgia tech undergraduate programs continue to excel, and i'm pleased that we've been able to maintain this measure of excellence for so long, " said interim president and provost gary schuster. q who was said that he was pleased that georgia tech undergraduate programs continued to excel? a gary schuster p "we're very upset, very angry," said raphael felli, 35, a u.s.-based attorney and son of executed colonel roger felli, who was foreign minister in the acheampong administration. q who was is an attorney based in the u.s., is the son of executed colonel roger felli? a raphael felli p liberty and tyranny sells a million. politics radio host mark r. levin's bestselling liberty and tyranny : a conservative manifesto has sold one million copies, according to publisher threshold editions.....published on march 24, 2009, liberty debuted at # 1 on the new york times bestseller list. q what sell a million, made it to the new york times bestsellers list? a liberty