CA i RE - COVID : A Question Answering and Query-focused Multi-Document Summarization System for COVID -19 Scholarly Information Management (2020)

Abstract: We present CAiRE-COVID, a real-time question answering (QA) and multi-document summarization system, which won one of the 10 tasks in the Kaggle COVID-19 Open Research Dataset Challenge, judged by medical experts. Our system aims to tackle the recent challenge of mining the numerous scientific articles being published on COVID-19 by answering high priority questions from the community and summarizing salient question-related information. It combines information extraction with state-of-the-art QA and query-focused multi-document summarization techniques, selecting and highlighting evidence snippets from existing literature given a query. We also propose query-focused abstractive and extractive multi-document summarization methods, to provide more relevant information related to the question. We further conduct quantitative experiments that show consistent improvements on various metrics for each module. We have launched our website CAiRE-COVID for broader use by the medical community, and have open-sourced the code for our system, to bootstrap further study by other researches.

## introduction since the covid-19 outbreak, a huge number of scientific articles have been published and made publicly available to the medical community (such as biorxiv, medrxiv, who, pubmed). at the same time, there are emerging requests from both the medical research community and wider society for efficient management of the information about covid-19 from this huge number of research articles. high priority scientific questions need to be answered, e.g., what is known about transmission, incubation, and environmental stability? what do we know about covid-19 risk factors? and what do we know about virus genetics, origin, and evolution? furthermore, question-related salient information needs to be summarized, so that the community can digest important contextual information more efficiently and keep up with the rapid acceleration of the coronavirus literature. the release of the covid-19 open research dataset (cord-19) 1 @xcite , which consists of over 158,000 scholarly articles about covid-19 and related coronaviruses, creates an opportunity for the natural language processing (nlp) community to address these requests. however, it also poses a new challenge since it is not easy to extract precise information regarding given scientific questions and topics from such a large set of unlabeled resources. to meet the requests and challenges for scholarly information management related to covid-19, we propose caire-covid, a neural question answering and query-focused multi-document summarization system. given a user query, the system first selects the most relevant documents from the cord-19 dataset 1 with high coverage via a document retriever module. it then highlights the answers or evidence (text spans) for the query, given the relevant paragraphs, by a snippet selector module via question answering (qa) models. furthermore, to efficiently present covid-19 questionrelated information to the user, we propose a queryfocused multi-document summarizer to generate abstractive and extractive summaries related to the question, from multiple retrieved answer-related paragraph snippets. we leverage the power of the generalization ability of pre-trained language models @xcite @xcite by fine-tuning them for qa and summarization, and propose our own adaptation methods for the covid-19 task. the effectiveness of our system has been proved by winning one of the tasks in round 1 of the kaggle cord-19 challenge, 1 in which hundreds of submissions were evaluated with the help of medical researchers. we further conduct a series of experiments to quantitatively show the competency of each module. to enhance both generalization and domainexpertise capability, we use an ensemble of two qa models in the qa module as the evidence selector. we evaluate the performance on the recently released covidqa @xcite ) dataset, and the results indicate that our qa module even outperforms the t5 model @xcite on the recall metric, while for keyword questions, it also marginally outperforms t5 on the precision fraction. the performance of the summerizer module is evaluated on two existing query-focused summarization (qfs) datasets, the duc datasets @xcite and debatepidia dataset @xcite , since there is no qfs dataset for covid-19. the duc datasets are the most widely used for the qfs task, while debatepedia is the first large-scale abstractive qfs dataset. previous works on the qfs task incorporate query relevance, either via a query-document relevance score @xcite or query attention model @xcite , into a seq2seq model, or concatenate query to documents into a pre-trained transformer architecture @xcite . however, none have taken an-swer relevance into consideration. by incorporating answer relevance from the qa module into the summarization process, our query-focused multidocument summarizer achieves consistent rouge score improvement over the bart @xcite -based baseline method on the abstractive task, and the lead baseline on the extractive task on both datasets. thus we believe that our proposed summarizer module can also work well on query focused summarization related to covid-19 questions. furthermore, we have launched our caire-covid website (as shown in figure 1 ), which enables real-time interactions for covid-19-related queries by medical experts. the code 3 for our system is also open-sourced to help future study. ## related work with the release of the covid-19 open research dataset (cord-19) 1 by the allen institute for ai, multiple systems have been built to assist both researchers and the public to explore valuable information related to covid-19. cord-19 search 4 is a search engine that utilizes the cord-19 dataset processed using amazon comprehend medical. google released the covid19 research explorer a semantic search interface on top of the cord-19 dataset. meanwhile, covidex 5 applies multi-stage search architectures, which can extract different features from data. an nlp medical relationship engine named the wellai covid-19 research tool foot_2 is able to create a structured list of medical concepts with ranked probabilities related to covid-19, and the tmcovid foot_3 is a bioconcept extraction and summarization tool for covid-19 literature. our system, in addition to information retrieval, gives high quality relevant snippets and summarization results given the user query. the website 2 further display information about covid-19 in a well structured and concise manner. ## methodology figure 2 illustrates the architecture of the caire-covid system, which consists of three major modules: 1) document retriever, 2) relevant snippet selector, and 3) query-focused multi-document summarizer. ## experiments in ## conclusion in this paper, we propose a general system, caire-covid, with open-domain qa and query focused multi-document summarization techniques for efficiently mining scientific literature given a query. the system has shown its efficiency on the kaggle cord-19 challenge, which was evaluated by medical researchers, and a series of experimental results also proved the effectiveness of our proposed methods and the competency of each module. the system is also easy to be generalized to general domain-agnostic literature information mining, especially for possible future pandemics. we have launched our website 2 for real-time interactions and released our code 3 for broader use.