A Smart System to Generate and Validate Question Answer Pairs for COVID -19 Literature (2020)

Abstract: Automatically generating question answer (QA) pairs from the rapidly growing coronavirus-related literature is of great value to the medical community. Creating high quality QA pairs would allow researchers to build models to address scientific queries for answers which are not readily available in support of the ongoing fight against the pandemic. QA pair generation is, however, a very tedious and time consuming task requiring domain expertise for annotation and evaluation. In this paper we present our contribution in addressing some of the challenges of building a QA system without gold data. We first present a method to create QA pairs from a large semi-structured dataset through the use of transformer and rule-based models. Next, we propose a means of engaging subject matter experts (SMEs) for annotating the QA pairs through the usage of a web application. Finally, we demonstrate some experiments showcasing the effectiveness of leveraging active learning in designing a high performing model with a substantially lower annotation effort from the domain experts.

## introduction building a qa system is a complex process requiring advanced text mining approaches @xcite and domain expertise for model evaluation. accordingly, automatically generating questionanswer pairs using recent advances in natural language processing (nlp) models has gained much attention from researchers and has achieved impressive results on various publicly available datasets. @xcite . in this work, we explore the covid-19 open research dataset (cord-19) @xcite first in- * equal contributions, listed alphabetically figure 1 : user engagement app: smes are provided with information including the question, the title of the article, and some context from the article to grade the answer, highlight the exact answer, and rate the credibility of the source. troduced in a kaggle competition foot_0 . the competition has been launched as a call to action for machine learning researchers to assist the medical community in developing answers to high-priority scientific questions related to covid-19. a major challenge in dealing with a large semi-structured dataset (i.e., scholarly articles) is the lack of gold data which we aim to address in this work. existing methods developed by different groups in the kaggle competition have mainly used clustering approaches @xcite coupled with statistical methods @xcite in order to group articles together and discover keywords from the resulting representation of the scholarly articles, respectively. other researchers made use of transformer-based qa models and bertserini @xcite to retrieve relevant answers to keywords extracted from a question, after which resulting solutions were ranked by unsupervised embedding methods @xcite . finally, top-ranking results were combined and summarized @xcite . other qa models made use of bert @xcite , adapted variations of bert models @xcite , and bert-like models @xcite to produce semantically meaningful sentence embeddings from abstracts to answer important questions raised by the healthcare community. developing qa systems for covid-19 was not limited to the kaggle competition; in @xcite , a hybrid approach based on gpt-2 @xcite was proposed to generate responses for different covid-19 related questions. in general, developing qa generation has achieved promising progress recently. however, answering a question in specific domains such as health domain is still challenging, due to the requirement of expert knowledge and lack of highquality training data. for example, walonoski et al. @xcite focused on generation of dataset from the state transition of patient records. recently, shen at al. @xcite introduced structure information of qa pairs generation in medical domain. they proposed an unsupervised detector to automatically explore external materials for the validity of generated qa pairs. despite all these attempts and solutions by various researchers, lack of annotated data for the cord-19 dataset presents a challenge to automatically verify the correctness of the created qa pairs and also prevents us from leveraging supervised techniques. to help address the shortcomings of previous approaches, we aim to create gold data related to covid-19 which in turn can serve the purpose of training and evaluating supervised models. we employ transformer and rule-based methods to automatically generate a set of qa pairs, which we call silver qa, and then leverage various active learning selection strategies to present samples to smes for annotation. to the best of our knowledge, this is the first work which explores the potential use of generative models to create qa pairs for quality verification by smes. our proposed approach can serve as a practical foundation for the creation of a qa system for any complex semi-structured dataset requiring the employment of domain knowledge experts to maintain the standard of the generated qa pairs. to provide a better user experience during the annotation process, we build a web application, which we call the user engagement app (uea), shown in figure 1 . the uea presents a batch of qa pairs once the smes select their expertise of a specific domain and topics of interest (e.g., vaccines and therapeutics, virus genetics, origin, and evolution). it also allows the smes to grade the qa pairs, select exact answers, and rank the credibility of the source. while developing the web application, we engaged two medical students to obtain their feedback. we summarize their feedback into two major issues when annotating the qa pairs, details of which are outlined in section 2. from these sets of feedback, we may firstly conclude that the smes require several hours to review a small batch of qa pairs due to the scientific complexity of the questions, and secondly, a high degree of domain-specific knowledge (e.g., virology, molecular genetics) is required to answer these scientific questions. we address the first feedback from the smes by introducing an active learning strategy which provides a method on how to select a limited number of samples. this in turn reduces the annotation efforts required to develop a practical qa system. in the future work section, we also provide some directions for the second feedback based on the results obtained from the qa pairs that we have generated. figure 2 illustrates the overview of how we integrate these different strategies to create the qa pairs and obtain gold data provided by the smes via the web application. we explain these steps in more details in sections 2 and 3 and also provide the experimental results in section 4. ## datasets in this section, we introduce the publicly available biomedical datasets that we use for our experiments. we also describe various methods conducted in this work to generate qas specifically but not limited to covid-19. title (t) extracted sentence (t′) using t rule-based question set (q) rule-based answer set transformer-based answer set silver qa unlabeled data labeled (gold) data active learner seed data domain experts classifier bioasq pubmedqa random sampling articles in cord-19 ## an active learning strategy for data selection active learning (al) strategies are shown to be effective in reducing the number of samples a machine learning model requires to achieve comparable performance to the case where a large amount of data is annotated @xcite . here, the main idea is to ask the smes to annotate strategically picked samples in small batches to minimize their efforts while encouraging the creation of a successful qa system. at a high level, we start by randomly choosing a small subset of the samples from the unlabeled pool, details are in the next section, as the seed data to be annotated by the smes. using the annotated seed data, we train a binary classifier to differentiate between different samples. the next step is to choose which unlabeled data points should be sent to the smes in the next iteration. after obtaining the annotations, the labeled samples are added to the pool of labeled data and further used to retrain the classifier. this iterative process, as illustrated at the bottom part of figure 2 , is repeated until either the annotation budget runs out or all the samples in the unlabeled pool have been annotated. in the following subsections, we explain how we formulate this problem as a binary classification task and introduce the different sampling strategies we have implemented. ## experiments and results we evaluate the performance of the different al strategies by reporting both the accuracy and f1 score on the test dataset. to have a fair assessment of the performance of each method, we run the experiments 5 times using different random seeds and average the results as illustrated in figure 4. we discuss the details of the implementation and setup for all of these experiments in section 4.1 and discuss the details of the results in section 4.2 ## conclusions and future work in this work, we propose a novel strategy consisting of transformer and rule-based methods to generate qa pairs from scientific literature gathered in the cord-19, while making use of a validation procedure to maintain the quality. we engage smes from the medical community and develop a web application to serve the purpose of providing an efficient user interface for annotating the qa pairs generated by our designed system. we also leverage active learning strategies to significantly reduce the required annotation effort from the smes. this work paves the way for several interesting areas which can be explored further in the future. we believe that the engagement app, which will be released to the public soon, would enable the medical community to use it to its full extent as it can incorporate several subjective opinions from different smes and researchers. with the foundation laid by this work, we can also investigate better ways to explore the generation process of accurate questions by diving deeper into the task of question generation which is gaining attention in the field of nlp. in order to further benefit from our proposed method to improve the generalizability of the model using only a small annotated dataset, we can provide higher quality qa pairs by removing redundant questions using methods which are proven to work for graphical structures. these methods treat each scholarly article as a node which results in reducing the number of highly interlinked questions. another plausible direction to explore is the incorporation of tasks such as extreme multi-label classification which would allow us to categorize scholarly articles under areas which may be better suited for annotations and align with the expertise of smes. lastly, as explained earlier, generation of qa pairs in this work was limited to transformer and rule-based methods. we would like to explore the integration of successful neural based methods in our proposed approach.