Annotating the Pandemic: Named Entity Recognition and Normalisation in COVID -19 Literature (2020)

Abstract: The COVID-19 pandemic has been accompanied by such an explosive increase in media coverage and scientific publications that researchers find it difficult to keep up. We are presenting a publicly available pipeline to perform named entity recognition and normalisation in parallel to help find relevant publications and to aid in downstream NLP tasks such as text summarisation. In our approach, we are using a dictionary-based system for its high recall in conjunction with two models based on BioBERT for their accuracy. Their outputs are combined according to different strategies depending on the entity type. In addition, we are using a manually crafted dictionary to increase performance for new concepts related to COVID-19. We have previously evaluated our work on the CRAFT corpus, and make the output of our pipeline available on two visualisation platforms.

## introduction the body of scientific literature is growing at an unprecedented rate, and this is particularly evident in the response of the biomedical research community to the 2020 covid-19 pandemic. several platforms have been established to track publications related to covid-19, most prominently the covid-19 open research dataset (cord-19) 1 , a collaboration of the us government and multiple other organisations, the litcovid dataset, maintained by the nih, which indexes papers published on pubmed related to the pandemic @xcite , or novel coronavirus research compendium (ncrc) foot_0 , which contains 800 publications selected manually for their originality and quality. in this publication, we are processing the articles of the litcovid dataset, which at the time of writing contains almost 50 000 publications related the 2020 covid-19 pandemic only, showing growth at a steady rate since its beginning. the flurry of news and public discussions about the pandemic, which includes a substantial amount of fake news, has been termed "infodemic". however, the term could be applied also to the rapid growth of reports and publications pertaining the disease (see figure 1 ). interestingly, this growth pattern seems to resemble that of the spread of the disease in western countries (with a delay of one to two months). while the growth is not exponential as it has occasionally been reported, it is still far beyond what virologists and medical scientists can manually process. this is an exacerbation of a general problem in biomedical research, where researchers cannot keep up with the growth of literature that pertains to their research, and need to resort to named en-tity recognition (ner), named entity normalisation (nen) and text summarisation technologies to identify relevant publications @xcite . in ner, entities of interest are identified as text spans in free text; and then, in nen, mapped to unique ids in a controlled vocabulary. they constitute a fundamental step for other down-stream text processing tasks, on one hand; but are also a means to its own end, allowing publications to be indexed by the entities they contain, on the other hand. in previous research, we have shown that we can obtain better results by performing ner and nen in parallel rather than sequentially, avoiding propagation of errors between the steps. we are building on this previous research and add a further processing step to find terms specific to covid-19. ## related work in march 2020, the us white house collaborated with the national library of medicine, the allen institue for artificial intelligence and other private companies to create the cord-19 corpus @xcite , and with it a set of 18 challenges such as what do we know about covid-19 risk factors? for data scientists to participate in, hosted on kaggle foot_1 . the response of the text mining community to the pandemic and such shared tasks has been enormous, producing a wide array of webservices, machine learning models and databases; usually adapting existing frameworks to suit the pandemic. @xcite , for example, are retraining scispacy on the cord-19 corpus to improve its ner performance. some research has already been directed at downstream tasks, using a simple dictionary-based ner method as a base to perform entity relation extraction @xcite , to create a knowledge base @xcite or for summarisation systems @xcite . the problem of ner and nen in the biomedical domain, generally, has traditionally been approached with pipelines, using rules or dictionaries @xcite . more recently, however, machine learning using various architectures such as lstms or crfs have become more popular @xcite . ## pipeline in our approach, we build on our previous efforts where we use a parallel architecture to perform ner and nen simultaneously @xcite . traditionally, ner and nen are performed after each other, which means that spans of mentions of entities are identified first, and then mapped to the corresponding entry in a controlled vocabulary. this approach has the drawback that errors made in the first step are irrecoverably propagated to the second stage. in our approach, however, we perform those two steps simultaneously, and were able to show that it outperforms the traditional approach @xcite . we are using biobert, a pre-trained language model, which we trained on the craft corpus, a collection of nearly 100 full-text medical articles manually annotated for 10 different medical entity types. we have evaluated our approach using the craft corpus, and obtained f1-scores between 0.74 and 0.92 depending on the entity type. to improve our results on covid-19 literature, we are adding an additional step of post-annotating our results using a manually crafted dictionary specific to covid-19. ## results so far, with our pipeline we have processed over 33 000 abstracts from pubmed and 7883 full-text articles from pmc, with a total amount of over 400 000 and 900 000 annotations, respectively (see with our pipeline, we are able to continuously process new articles that are added to the litcovid dataset, and distribute our annotations in the following ways: • our own webservice using brat • freely downloadable files the oger annotations can be obtained through an api foot_2 . the code to run the pipeline foot_3 , its outputs foot_4 as well as the craft-trained biobert models foot_5 are publicly available, and with some effort could be modified using oger's format conversion to process other dataset such as cord-19. ## evaluation given the recency of the pandemic, there is currently a lack of resources that allow evaluation of work on the covid-19 literature. without a gold standard we cannot offer a true evaluation. we hope to be able to test the efficacy of our own work in the future when such resources become available. ## discussion tools that automatically process literature related covid-19 generally fall into two broad categories: systems that follow some sort of text summarisation approach, and ner+nen systems. much attention has been directed at previously mentioned kaggle challenge, for which over 1500 solutions have been submitted, ranging from statistical data exploration to a full clustering of the literature. one of the top submissions 12 , for example, attempts to identify risk factors of covid-19 by applying unsupervised topic modeling algorithms. such approaches are very common among the submissions, but suffer from a high number of false positives. similarly, platforms that allow browsing corpora of covid-19 papers such as covidscholar 13 and the bert-driven covid-19 research explorer 14 rely on word embeddings and other unsupervised algorithms to find matching publications or even passages in publications. for the latter, the authors attempt to go beyond traditional document retrieval, and employ an automatically generated corpus to fuel their question answering learning @xcite . however, such approaches lack the precision typical ner+nen-driven approaches offer, and don't perform particularly well at matching entity synonyms due to their representation as highrecall word vectors rather than precisely matched entities. for example, both applications yield different 12 bit.ly/2vkn6qp 13 covidscholar.org/ 14 bit.ly/3fwnolg results for either angiotensin converting enzyme 2 or ace2, even though the terms are equivalent (and link to the same entry in the protein ontology). repositories that perform controlled vocabulary nen such as knetminer, for example, avoid this error @xcite . services exploring the scientific literature still fall in either of the two camps, and thus fail to exploit the high precision benefits ner+nen offers and the variety of applications text summarisation approaches afford simultaneously.