CORA : A Deep Active Learning Covid-19 Relevancy Algorithm to Identify Core Scientific Articles (2020)

Abstract: Ever since the COVID-19 pandemic broke out, the academic and scientific research community, as well as industry and governments around the world have joined forces in an unprecedented manner to fight the threat. Clinicians, biologists, chemists, bioinformaticians, nurses, data scientists, and all of the affiliated relevant disciplines have been mobilized to help discover efficient treatments for the infected population, as well as a vaccine solution to prevent further the virus spread. In this combat against the virus responsible for the pandemic, key for any advancements is the timely, accurate, peer-reviewed, and efficient communication of any novel research findings. In this paper we present a novel framework to address the information need of filtering efficiently the scientific bibliography for relevant literature around COVID-19. The contributions of the paper are summarized in the following: we define and describe the information need that encompasses the major requirements for COVID-19 articles relevancy, we present and release an expert-curated benchmark set for the task, and we analyze the performance of several state-of-the-art machine learning classifiers that may distinguish the relevant from the non-relevant COVID-19 literature.

## introduction the covid-19 pandemic has been responsible for almost 10 million people infected worldwide, and has left close to 1 million people dead till mid-september 2020, according to the world health organization foot_0 . the whole world observes in awe the catastrophe that the pandemic is leaving behind; human lives, economies and markets have been struck fiercely, as the scientific community and industry, united in all fronts, is seeking for treatment and vaccine solutions against the disease caused by the 2019-ncov virus. in these times where scientific advancements are sought and expected rapidly, there lies the challenge of filtering efficiently the scientific literature for the most relevant articles that can help clinicians, nurses, biologists, chemists, bioinformaticians, data scientists and other researchers operating in affiliated disciplines, to combat the pandemic. all these research and practice protagonists have many, and heterogeneous, requirements on what would be a relevant covid-19 article; more importantly, they have extremely limited time to scan large volumes of literature. the risks faced in the aforementioned challenge are multiple; for first, the information, in the form of scientific articles, needs to be timely. this requires acceleration of the whole peer-review and publication process for the covid-19 relevant articles, to enable the fastest possible communication of breaking scientific and clinical results. the information needs to be accurate as well; therefore, without jeopardizing quality, the editors and publishers of scientific content need to have in place fast-track review processes for these important articles. furthermore, the information needs to be highly relevant for the aforementioned protagonists who combat the pandemic at the forefront, and have limited time for extensive literature reviews given their crucial duties. last but not least, the challenge is becoming increasingly difficult, given that the volume of the relevant literature for covid-19 is continuously growing; indicatively, the elsevier published articles on covid-19 foot_1 have grown in volume from few tens of articles per week in march 2020, to almost 1, 000 articles per week in june 2020. this is an increase of approximately 2 orders of magnitude in a period of 4 months. in order to avoid "information choking", the community requires efficient data science solutions and respective initiatives that can help researchers navigate through this volume of information and focus on the most relevant articles based on their information need. some examples of such initiatives, or enablers thereof, are: the majority of the aforementioned initiatives imply the existence of a covid-19 scientific article relevancy mechanism, that can filter the core literature on the pandemic, to be included in such collections or data science challenges and platforms. in this paper we present such a framework, namely cora, and we argue that it may constitute the basis for surfacing efficiently the core covid-19 literature in a way that it addresses the majority of the information needs of the protagonists who fight the pandemic. the contribution of cora can be summarized in the following: (i) it defines the information need behind relevancy to covid-19, having ingested the feedback of researchers and professionals in medicine, biology, chemistry, bioinformatics and data science, (ii) it offers a benchmark set for the task, with labelled "relevant" and "non-relevant" covid-19 scientific articles, and, (iii) defines an efficient approach that combines search and machine learning, to balance optimally between precision and recall for the task. the impact of such an approach is tremendous; for first, it can help scientific publishers and editors to flag early submitted articles that are core to covid-19, and ensure the acceleration of their review and final publication. it can also be used to filter out large volumes of scientific literature, and retain only the body of the literature that is core to covid-19. this can help accelerate the preparation and production of data science datasets and challenges aiming to address the pandemic. the presented framework is generic, and is described in detail so that it can be reproduced in any environment for these two purposes. in the remaining of the paper we describe the information need of relevancy to covid-19 (section 2), the process used to create and validate the benchmark set for the training and the tuning of the approach (section 3), the details of the cora framework (section 4), as well as results of various methods, including cora, in the produced set (section 5). ## covid-19 information need for relevant scientific literature one of the largest publicly available datasets for covid-19, namely cord-19 (wang et al., 2020), draws its contents from pubmed central, biorxiv, medrxiv, and the world health organization (who). all the major scientific publishers, such as elsevier, and springer nature are contributing to it, and have offered every help for its compilation. in the case of who, the data can be pulled from a hand-curated database of relevant literature compiled by the organization foot_8 . however, in the case of the remaining three sources, a generic keyword query is used on the title, abstract and body text of the articles, to filter the ones that are included in the collection. the query is shown in figure 1 . the query used for the compilation of cord-19 includes the fundamental keywords for the pandemic; however, the precision of the aforementioned query is highly arguable. a scientific article may refer to covid-19 or any of the coronaviruses for multiple reasons, and often the article can be deemed as irrelevant by expert doctors, biologists and chemists. for example, the article could refer to the financial consequences of covid-19, or to its impact in some social aspects. it could even just refer to covid-19 as the most recent example of a pandemic, without discussing about the specific pandemic at all, in a scientific, medical or clinical context. we argue that the expert users who combat the pandemic have an underlying information need that is much more specific than the one expressed from the aforementioned query, and that there should be much more efficient mechanisms to filter the relevant core covid-19 articles. as a first step for the creation of cora we interviewed experts in the field of medicine, biology, chemistry and bioinformatics, who combat the pandemic, and attempted to extract their information needs. this resulted in a number of inclusion and exclusion criteria, that represent the information need, and can be used to compile a benchmark dataset for identifying core covid-19 articles. the inclusion and exclusion criteria are presented in figure 2 . as it is illustrated, the protagonists who combat the pandemic, are interested exclusively in the diagnosis, treatment, vaccine development, pathology, and virology of covid-19, as well as in literature about other coronaviruses. furthermore, the experts are also interested in how hospitals are addressing the pandemic, how does the health care systems manage it, and what are some population statistics, and demographics of the disease. all experts were explicit in that, articles related to the impact of the pandemic in areas such as economy, education, transport, etc., are of secondary importance and should not be included in a core scientific covid-19 collection, aiming to aid the combat to the disease. ## preparation of the benchmark set one of the main challenges in any supervised learning task is to have good quality and high volume training data for the algorithms to learn optimally. in cases where training sets are not available, one must create a bespoke data set for the task at hand. creating a valid, accurate, and large data set is a time-consuming and laborious task. data sets are typically created by manual annotation of data points, e.g., scientific articles in our case, from a pool of randomly selected data points from a population. the size of the training dataset typically depends on factors such as task complexity, resources, time, and budge availability. in order to create a benchmark dataset which includes both "relevant" and "non-relevant" articles to the criteria illustrated in figure 2 , as a first step we applied the query illustrated in figure 3 , into the forward flow of the elsevier accepted articles for a period of 2 months. this query can be seen as a much more detailed version of the simple and generic keyword-based query illustrated in figure 1 that was utilized for the compilation of the cord-19 dataset. for the manual annotation of the documents returned by the query, we used active machine learning, and more precisely the general approach described by konyushkova et al. @xcite . active learning in this case provides an efficient way of selecting the right document sample(s) for labelling. in active learning, an algorithm picks the examples that are more useful in order for the machine learning process to reach its full potential. we used biobert as the base classifier in the active learning pipeline. there are several criteria an algorithm can use to pick the best samples for annotation, such as based on uncertainty, committee, or by bagging and boosting @xcite . for this task, we used uncertainty sampling which is one of the popular methods, and is considered to be very efficient @xcite . in uncertainty sampling, the algorithm picks a sample for annotation from the unlabeled pool where it is least confident about its prediction probability. this resulted in a much smaller but more informative data set for our task. the deep active learning algorithm utilized to compile the most useful such set for cora is described in algorithm 1. we first fine-tuned biobert on a small expert-curated seed set (˜1,600 data points) and measured its accuracy on the test set. secondly, the algorithm enters the active learning loop, where the unlabeled data samples are picked using the uncertainty method and human validators are asked to provide labels for the data points. the model is then further fine-tuned on the data points where the classifier's label contradicts human label. the accuracy in line 12 of the algorithm measures how certain the model is on the unlabeled samples. this process continues til the maximum iterations or the active learning algorithm suggest that no further training is required (i.e., desired certainty is achieved). we also created a separate test set to evaluate the performance of the cora machine learning model. both the training and test set were manually annotated by in-house subject matter experts @xcite . ## the cora covid-19 relevancy algorithm cora is aiming at encapsulating the information needs described in section 2, and retaining an optimal balance between precision and recall in the process of retrieving relevant documents according to these needs. the focus in satisfying the precision to the information needs can be addressed by training a machine learning model on the benchmark set described in section 3. however, cora needs to start from a much larger set, to also satisfy the requirement that recall is as high as possible; yet such a set needs to minimize the risks of introducing a large number of false positives, and totally irrelevant articles. in order to achieve this balance, cora utilizes first the keyword-based strategy illustrated in figure 3 , and then applies a machine learning model to filter out the "non-relevant" articles from this originally wide net that was cast to perform the information retrieval. the machine learning model that cora is using is a fine-tuning of biobert (bidirectional encoder representations from transformers for biomedical text mining) @xcite for the task of learning the "relevant" and "non-relevant" classes from the benchmark set. biobert is a bert-based language representation model which is pre-trained on biomedical corpora from pubmed and pmc, as well as on the english wikipedia and a corpus of books. it has reported state-of-the-art performance in several nlp related tasks on biomedical text, such as named entity recognition, and biomedical question answering @xcite . the high level description of the machine learning model used in cora is illustrated in figure 4 . given the benchmark training set obtained via deep active learning, as illustrated in algorithm 1, the cora covid-19 relevancy can be described in simple steps, and is illustrated in algorithm 2. the description of the algorithm covers both the preparation and training, as well as the inference steps, given an input set of unseen documents d test to be classified as "relevant" or "non-relevant". ## experimental results and discussion in this section we present the results of the empirical evaluation on the produced benchmark set described in ## conclusions and future work following the breakout of the covid-19 pandemic early in 2020, the scientific community, industry and governments around the world joined forces to combat the spreading of the disease, and to identify efficient treatment methods, as well as vaccine solutions against the 2019-ncov virus. efficient and reliable information communication, including the latest scientific advancements in the form of peerreviewed published articles, has proven to hold in this paper we presented cora, an algorithmic solution to filter the relevant scientific papers, and save time from the experts to combat the disease, and focus only on the primary impact information. the contribution of this work is multi-fold: (i) we present a framework of inclusion and exclusion criteria that may be used as guidelines to annotate corpora of scientific publications, towards building benchmark datasets for the purpose of developing and tuning covid-19 relevancy systems; the criteria encapsulate the information needs of experts across medicine, biology, chemistry, and bioinformatics, in order to combat efficiently the pandemic, (ii) we applied a simple, yet efficient deep active learning approach to compile such a benchmark set with the help of subject matter experts for the hand curation of the labels; the approach utilized the fine-tuning of biobert as a base classifier, and we demonstrated that the produced set is also very meaningful for training other classifiers as well, (iii) we introduced the cora algorithmic framework for filtering the relevant scientific literature; cora combines an extensive keyword-based query to initialize a large pool of potentially relevant documents and maximize recall, and a trained fine-tuned biobert model, to retain only the relevant articles from this pool, (iv) we demonstrated via an experimental evaluation on the benchmark set, that the cora algorithm can achieve 96% f1score on detecting the non-relevant documents, and 91% on detecting the relevant documents, constituting cora a satisfactory solution for production settings of this exercise. as a future work, we plan to experiment further with novel machine learning models, e.g., albert @xcite , and electra @xcite , who have shown great promise in the glue leader board results foot_9 , as well as with alternative active learning approaches, e.g., batch-aware methods @xcite , in order to improve further this performance. more importantly, having the understanding that the terminology around the covid-19 literature is evolving fast over time, new terms appear constantly, and the vocabulary is shifting focus towards the names of new promising targets, compounds or characterization of symptoms and treatment options, we will focus in enriching cora with a novel adaption of its keyword-based query over time. by conducting novel keywords extraction from the recent scientific literature, the cora keyword-based query can be enhanced automatically with new terminology. in this manner, the original pool of fetched documents can still satisfy the requirement of very high recall, as they are fetched by a query which follows the vocabulary trends adopted by the published scientific literature on covid-19. additionally, to help further with the information overload issue, we plan to introduce domain-specific targeted labels for different user groups (e.g., clinicians, bioinformaticians, chemists), allowing any covid-19 relevant literature to be potentially filtered according to domain specific information needs.