Concept Wikification for COVID -19 (2020)

Abstract: Understanding scientific articles related to COVID-19 requires broad knowledge about concepts such as symptoms, diseases and medicine. Given the very large and ever-growing scientific articles related to COVID-19, it is a daunting task even for experts to recognize the large set of concepts mentioned in these articles. In this paper, we address the problem of concept wikification for COVID-19, which is to automatically recognize mentions of concepts related to COVID-19 in text and resolve them into Wikipedia titles. We develop an approach to curate a COVID-19 concept wikification dataset by mining Wikipedia text and the associated intra-Wikipedia links. We also develop an end-to-end system for concept wikification for COVID-19. Preliminary experiments show very encouraging results. Our dataset, code and pre-trained model are available at github.com/panlybero/Covid19_wikification.

## introduction recently, research articles about covid-19 are published at overwhelming scale and speed. understanding these articles is crucial for combating the covid-19 pandemic. for example, the following sentence describes the transmission of coronavirus: s1: the interaction of the coronavirus spike protein with its complementary cell receptor is central in determining the tissue tropism, infectivity, and species range of the released virus. coronaviruses mainly target epithelial cells. they are transmitted from one host to another host, depending on the coronavirus species, by either an aerosol, fomite, or fecal-oral route. there are a few concepts (in bold) that are central to understanding this sentence. understanding these concepts requires biomedical expertise. the sheer volume of new information makes it a daunt-ing task even for infectious disease experts to recognize the ever-growing set of concepts mentioned in the literature. this necessitates developing a concept recognition and linking system that can automatically tag mentions of these concepts in text and resolve them into wikipedia titles. we call this task concept wikification for covid-19. it is an extension of the classic wikification @xcite problem with a few new challenges introduced: first, what concepts are related to covid-19? we need an efficient approach to identify these concepts at scale. second, there is no labeled dataset for training such a wikifier. therefore, we need an effective way to construct a labeled dataset timely, to assist researchers to develop concept wikifiers in time to combat the covid-19 pandemic. third, the concept wikifier needs to perform both concept mention detection and linking. while these two steps could be uncoupled, the developing nature of the pandemic means that new concepts and information is constantly being associated with covid-19, and updating systems separately may inhibit timely response to the the shifting domain. this is unlike classic wikification for which many named entity recognition (ner) or mention detection systems are available. in this paper, we address all these challenges and develop a practical solution for covid-19 concept wikification. we first develop an approach to automatically discover concepts related to covid-19 by exploring wikipedia, starting from the wikipedia page for covid-19. we then leverage wikipedia text and intra-wikipedia links to automatically harvest a large dataset in which mentions are labeled with their wikipedia titles. using this dataset, we develop an end-to-end system for concept wikification for covid-19. preliminary experiments show encouraging results. our contributions are summarized as follows: ## related work entity linking (el) @xcite @xcite or wikification @xcite is a reference resolution task, for which the goal is to identify entity mentions in text and resolve each mention into an entry in the reference knowledge base (kb) (for wikification, the entry is a wikipedia title). most existing el systems focus on entities such as persons, organizations and geopolitical entities, and use ner or mention extraction systems @xcite to identify candidate mentions. to perform disambiguation to wikipedia titles, systems @xcite @xcite often use lexical, syntactic and semantic features. recently, there has been growing interest in modeling the el stages (e.g., mention detection, candidate generation, entity disambiguation) in a joint modeling framework such as a graphic model @xcite or a neural architecture @xcite . @xcite showed that a simple bert-based token classification model can perform surprisingly well at end-to-end el. entity linking research has benefit greatly from large-scale annotated datasets such as the conll03/aida dataset @xcite . there are also approaches for specialized domains such as the bio-medical domain @xcite , or in a cross-lingual setting @xcite . this work differs from previous work in that we aim at recognizing and linking concepts related to covid-19, to help timely and more comprehensive understanding of the concepts in the large amount of covid-19 related scientific articles. ## building a dataset for covid-19 concept wikification our goal is the following: first, we aim at finding concepts that are related to covid-19. we also avoid concepts such as persons, organizations or geo-political entities (gpe), since they are covered in existing datasets. second, we aim at harvesting text in which mentions of these concepts are labeled with their reference wikipedia title. these labeled texts can be used to train a concept wikifier. we leverage wikipedia to construct this dataset because it has broad coverage, is frequently updated by its editors and has plenty of text and intrapage links that support the goal. ## end-to-end concept wikification we develop a hybrid, pipelined system consists of the following two steps. given an input sentence, it first identifies concept strings (e.g., sars-cov-2) that are unambiguous and assigns their types using a majority-class classifier. it then applies a scib-ert @xcite -based neural wikifier to assign types to other, more ambiguous concept mention strings. this hybrid system combines the best of both worlds: it uses the majority-class rulebased approach to resolve unambiguous concept mentions, and a deep contextualized neural model to resolve the ambiguous mentions, which require context to disambiguate. ## experiments we evaluate seven wikifiers using the ambiguous dataset in for pre-processing we use each model's built-in tokenizer to tokenize the sentences. we also use a maximum sentence length of 200 (short sentences will be padded to 200). we train all the transformer models for 50 epochs using a batch size of 200. we use the adam optimizer with an initial learning rate 5 × 10 -5 . we use micro-average precision (p), recall (r) and f1 metrics, for which we only consider a concept link is true if they match the reference annotation. similar to @xcite , we report two types of p/r/f1 based on whether we require strong match or weak match when calculating these metrics. strong match requires every token in the gold annotated span to be classified correctly, while weak match only requires at least one token in the gold annotation span to be classified correctly to account for annotation inconsistencies in wikipedia. experimental results in this can be attributed to the significant domain mismatch between the pre-training corpus and our dataset. this is further supported by the significant performance gains that result from finetuning. scibert-ft and bert-ft achieves significantly higher performance compared to the baseline and the frozen models. this shows that deep contextualized word representations need to be fine-tuned in order to work well for this challenging problem. biobert, the model who's domain matches the dataset the best, performs very well on the task surpassing both bert-ft and scibert-ft. this supports that the specialized scope of our dataset requires contextualized word-embeddings specifically tuned to the biomedical domain. finetuning biobert provides only small improvement compared to the non-finetuned version, which demonstrates that the model's pre-training domain is close enough to our dataset that the effects of additional finetuning are diminishing. ## conclusion and future work in this paper, we curated a dataset to enable concept wikification research for covid-19. we developed a few concept wikifier and shown that they perform reasonably well. as a next step, we plan to augment the dataset to include more concepts and will improve the concept wikifier using the augmented dataset.