CORD-19 : The COVID-19 Open Research Dataset (2020)

Abstract: The COVID-19 Open Research Dataset (CORD-19) is a growing resource of scientific papers on COVID-19 and related historical coronavirus research. CORD-19 is designed to facilitate the development of text mining and information retrieval systems over its rich collection of metadata and structured full text papers. Since its release, CORD-19 has been downloaded over 200K times and has served as the basis of many COVID-19 text mining and discovery systems. In this article, we describe the mechanics of dataset construction, highlighting challenges and key design decisions, provide an overview of how CORD-19 has been used, and describe several shared tasks built around the dataset. We hope this resource will continue to bring together the computing community, biomedical experts, and policy makers in the search for effective treatments and management policies for COVID-19.

## introduction on march 16, 2020, the allen institute for ai (ai2), in collaboration with our partners at the white house office of science and technology policy (ostp), the national library of medicine (nlm), the chan zuckerburg initiative (czi), microsoft research, and kaggle, coordinated by georgetown university's center for security and emerging technology (cset), released the first version of cord-19. this resource is a large and growing collection of publications and preprints on covid-19 and related historical coronaviruses such as sars and mers. the initial release consisted of 28k papers, and the collection has grown to more than 140k papers over the subsequent weeks. papers and preprints from several archives are collected and ingested through the semantic scholar literature search engine, foot_0 metadata are harmonized and deduplicated, and paper documents are processed through the pipeline established in @xcite to extract full text (more than 50% of papers in cord-19 have full text). we commit to providing regular updates to the dataset until an end to the covid-19 crisis is foreseeable. cord-19 aims to connect the machine learning community with biomedical domain experts and policy makers in the race to identify effective treatments and management policies for covid-19. the goal is to harness these diverse and com-plementary pools of expertise to discover relevant information more quickly from the literature. users of the dataset have leveraged ai-based techniques in information retrieval and natural language processing to extract useful information. responses to cord-19 have been overwhelmingly positive, with the dataset being downloaded over 200k times in the three months since its release. the dataset has been used by clinicians and clinical researchers to conduct systematic reviews, has been leveraged by data scientists and machine learning practitioners to construct search and extraction tools, and is being used as the foundation for several successful shared tasks. we summarize research and shared tasks in section 4. in this article, we briefly describe: ## dataset cord-19 integrates papers and preprints from several sources (figure 1 ), where a paper is defined as the base unit of published knowledge, and a preprint as an unpublished but publicly available counterpart of a paper. throughout the rest of section 2, we discuss papers, though the same processing steps are adopted for preprints. first, we ingest into semantic scholar paper metadata and documents from each source. each paper is associated with bibliographic metadata, like title, authors, publication venue, etc, as well as unique identifiers such as a doi, pubmed central id, pubmed id, the who covidence #, 4 mag identifier @xcite , and others. some papers are associated with documents, the physical artifacts containing paper content; these are the familiar pdfs, xmls, or physical print-outs we read. for the cord-19 effort, we generate harmonized and deduplicated metadata as well as structured full text parses of paper documents as output. we provide full text parses in cases where we have access to the paper documents, and where the documents are available under an open access license 4 https://www.who.int/emergencies/diseases/novelcoronavirus-2019/global-research-on-novel-coronavirus2019-ncov (e.g. creative commons (cc), foot_1 publisher-specific covid-19 licenses, 6 or identified as open access through doi lookup in the unpaywall foot_3 database). ## design decision & challenges a number of challenges come into play in the creation of cord-19. we summarize the primary design requirements of the dataset, along with challenges implicit within each requirement: up-to-date hundreds of new publications on covid-19 are released every day, and a dataset like cord-19 can quickly become irrelevant without regular updates. cord-19 has been updated daily since may 26. a processing pipeline that produces consistent results day to day is vital to maintaining a changing dataset. that is, the metadata and full text parsing results must be reproducible, identifiers must be persistent between releases, and changes or new features should ideally be compatible with previous versions of the dataset. handles data from multiple sources papers from different sources must be integrated and harmonized. each source has its own metadata format, which must be converted to the cord-19 format, while addressing any missing or extraneous fields. the processing pipeline must also be flexible to adding new sources. clean canonical metadata because of the diversity of paper sources, duplication is unavoidable. once paper metadata from each source is cleaned and organized into cord-19 format, we apply the deduplication logic described in section 2.2 to identify similar paper entries from different sources. we apply a conservative clustering algorithm, combining papers only when they have shared identifiers but no conflicts between any particular class of identifiers. we justify this because it is less harmful to retain a few duplicate papers than to remove a document that is potentially unique and useful. machine readable full text to provide accessible and canonical structured full text, we parse content from pdfs and associated paper documents. the full text is represented in s2orc json format @xcite , a schema designed to preserve most relevant paper structures such as paragraph breaks, section headers, inline references, and citations. s2orc json is simple to use for many nlp tasks, where character-level indices are often employed for annotation of relevant entities or spans. the text and annotation representations in s2orc share similarities with bioc (comeau et al., 2019), a json schema introduced by the biocreative community for shareable annotations, with both formats leveraging the flexibility of characterbased span annotations. however, s2orc json also provides a schema for representing other components of a paper, such as its metadata fields, bibliography entries, and reference objects for figures, tables, and equations. we leverage this flexible and somewhat complete representation of s2orc json for cord-19. we recognize that converting between pdf or xml to json is lossy. however, the benefits of a standard structured format, and the ability to reuse and share annotations made on top of that format have been critical to the success of cord-19. observes copyright restrictions papers in cord-19 and academic papers more broadly are made available under a variety of copyright licenses. these licenses can restrict or limit the abilities of organizations such as ai2 from redistributing their content freely. although much of the covid-19 literature has been made open access by publishers, the provisions on these open access licenses differ greatly across papers. additionally, many open access licenses grant the ability to read, or "consume" the paper, but may be restrictive in figure 3 : an example information retrieval and extraction system using cord-19: given an input query, the system identifies relevant papers (yellow highlighted rows) and extracts text snippets from the full text jsons as supporting evidence. other ways, for example, by not allowing republication of a paper or its redistribution for commercial purposes. the curator of a dataset like cord-19 must pass on best-to-our-knowledge licensing information to the end user. ## research directions we provide a survey of various ways researchers have made use of cord-19. we organize these into four categories: (i) direct usage by clinicians and clinical researchers ( §4.1), (ii) tools and systems to assist clinicians ( §4.2), (iii) research to support further text mining and nlp research ( §4.3), and (iv) shared tasks and competitions ( §4.4). ## discussion several hundred new papers on covid-19 are now being published every day. automated methods are needed to analyze and synthesize information over this large quantity of content. the computing community has risen to the occasion, but it is clear that there is a critical need for better infrastructure to incorporate human judgments in the loop. extractions need expert vetting, and search engines and systems must be designed to serve users. successful engagement and usage of cord-19 speaks to our ability to bridge computing and biomedical communities over a common, global cause. from early results of the kaggle challenge, we have learned which formats are conducive to collaboration, and which questions are the most urgent to answer. however, there is significant work that remains for determining (i) which methods are best to assist textual discovery over the literature, (ii) how best to involve expert curators in the pipeline, and (iii) which extracted results convert to successful covid-19 treatments and management policies. shared tasks and challenges, as well as continued analysis and synthesis of feedback will hopefully provide answers to these outstanding questions. since the initial release of cord-19, we have implemented several new features based on com-munity feedback, such as the inclusion of unique identifiers for papers, table parses, more sources, and daily updates. most substantial outlying features requests have been implemented or addressed at this time. we will continue to update the dataset with more sources of papers and newly published literature as resources permit.