Document Classification for COVID -19 Literature (2020)

Abstract: The global pandemic has made it more important than ever to quickly and accurately retrieve relevant scientific literature for effective consumption by researchers in a wide range of fields. We provide an analysis of several multi-label document classification models on the LitCovid dataset, a growing collection of 23,000 research papers regarding the novel 2019 coronavirus. We find that pre-trained language models fine-tuned on this dataset outperform all other baselines and that BioBERT surpasses the others by a small margin with micro-F1 and accuracy scores of around 86% and 75% respectively on the test set. We evaluate the data efficiency and generalizability of these models as essential features of any system prepared to deal with an urgent situation like the current health crisis. We perform a data ablation study to determine how important article titles are for achieving reasonable performance on this dataset. Finally, we explore 50 errors made by the best performing models on LitCovid documents and find that they often (1) correlate certain labels too closely together and (2) fail to focus on discriminative sections of the articles; both of which are important issues to address in future work. Both data and code are available on GitHub.

## introduction the covid-19 pandemic has made it a global priority for research on the subject to be developed at unprecedented rates. researchers in a wide variety of fields, from clinicians to epidemiologists to policymakers, must all have effective access to the most up to date publications in their respective areas. automated document classification can play an essential role in organizing the stream of articles by fields and topics to facilitate the search process and speed up research efforts. to explore how document classification models can help organize covid-19 research papers, we use the litcovid dataset @xcite , a collection of 23,000 newly released scientific papers compiled by the nih to facilitate access to the literature on all aspects of the virus. this dataset is updated daily and every new article is manually assigned one or more of the following 8 categories: general, transmission dynamics (transmission), treatment, case report, epidemic forecasting (forecasting), prevention, mechanism and diagnosis. we leverage these annotations and the articles made available by litcovid to compile a timely new dataset for multi-label document classification. apart from addressing the pressing needs of the pandemic, this dataset also offers an interesting document classification dataset which spans different biomedical specialties while sharing one overarching topic. this setting is distinct from other biomedical document classification datasets, which tend to exclusively distinguish between biomedical topics such as hallmarks of cancer @xcite , chemical exposure methods @xcite or diagnosis codes @xcite . the dataset's shared focus on the covid-19 pandemic also sets it apart from open-domain datasets and academic paper classification datasets such as imdb or the arxiv academic paper dataset (aapd) @xcite in which no shared topic can be found in most of the documents, and it poses unique challenges for document classification models. we evaluate a number of models on the litcovid dataset and find that fine-tuning pre-trained language models yields higher performance than traditional machine learning approaches and neural models such as lstms @xcite @xcite . we also notice that biobert @xcite , a bert model pretrained on the original corpus for bert plus a large set of pubmed articles, performed slightly better than the original bert base model. we also observe that the novel longformer @xcite ) model, which allows for processing longer sequences, matches biobert's performance when 1024 subwords are used instead of 512, the maximum for bert models. we then explore the data efficiency and generalizability of these models as crucial aspects to address for document classification to become a useful tool against outbreaks like this one. additionally, we perform a data ablation study to evaluate the effect of article titles on the limits of performance for this dataset. finally, we discuss some issues found during our error analysis, such as current models often (1) correlating certain categories too closely with each other and (2) failing to focus on discriminative sections of a document and get distracted by introductory text about covid-19, which suggest avenues for future improvement. ## datasets in this section, we describe the litcovid dataset in more detail and briefly introduce the cord-19 dataset, which we sampled to create a small test set to evaluate model generalizability. ## models in the following section, we provide a brief description of each model and the implementations used. we use micro-f1 (f1) and accuracy (acc.) as our evaluation metrics, as done in (adhikari ## results & discussion in this section, we explore data efficiency and model generalizability, present a data ablation study, and discuss potential ways to improve performance on this task in future work. ## conclusion we provide an analysis of document classification models on the litcovid dataset for the covid-19 literature. we determine that fine-tuning pretrained language models yields the best performance on this task. we study the generalizability and data efficiency of these models, evaluate the effect of article titles on performance through a data ablation study and discuss some important issues to address in future work. category binary f1 score litcovid dev cord-19 set prevention 92.7 ±0.7 66.7 ±2.5 case report 87.9 ±1.5 66.7 ±0.0 treatment 85.6 ±0.9 53.9 ±9.0 diagnosis 82.1 ±0.5 63.0 ±3.3 mechanism 81.5 ±2.1 86.8 ±0.8 forecasting 70.3 ±2.3 0.0 ±0.0 general 35.5 ±33.9 0.0 ±0.0 transmission 60.9 ±1.6 56.4 ±3.2 table a1: biobert binary f1 scores per category on the litcovid dev set and the cord-19 test set. scores are given as @xmath1.