Knowledge Discovery in COVID -19 Research Literature (2020)

Abstract: This paper presents the preliminary results of an ongoing project that analyzes the growing body of scientific research published around the COVID-19 pandemic. In this research, a general-purpose semantic model is used to double annotate a batch of 500 sentences that were manually selected by the researchers from the CORD-19 corpus. Afterwards, a baseline text-mining pipeline is designed and evaluated via a large batch of 100,959 sentences. We present a qualitative analysis of the most interesting facts automatically extracted and highlight possible future lines of development. The preliminary results show that general-purpose semantic models are a useful tool for discovering fine-grained knowledge in large corpora of scientific documents.

## introduction the covid-19 pandemic that affected almost every country at the beginning of 2020 has provoked a massive increase in scientific papers related to biomedical sciences @xcite . the scientific community's effort to fight the spread of the virus is evident in the record number of research papers about covid-19 that have been submitted to conferences, journals and preprint services. several academic publishers joined the effort by providing free access to research in related areas that could be useful to scientists and academics. the amount of information produced during this period greatly surpassed the ability of human researchers to stay up-to-date, which in turn spawned an increased interest in the application of computational techniques to automatically organize, normalize and link the existing information. a recent initiative is the covid-19 open research dataset (cord-19), published by the allen institute for ai @xcite , which makes available a large corpus of scientific papers on covid-19 and related topics. at the moment of writing, it contains 76, 674 scientific articles. the corpus has been used as part of a kaggle challenge 1 , which focused mainly on unsupervised tasks related to organizing and categorizing the different aspects of the whole covid-19 situation. based on these resources, computational tools, e.g., scisight @xcite , scifact @xcite , and similar @xcite , have been created to enable the interactive visualization and exploration of the scientific literature and the discovery of connections between the available methods, symptoms, interventions, etc. unsupervised approaches are a natural strategy for dealing with large, unlabeled corpora, while supervised approaches have the caveat of requiring training examples to be manually annotated, but can provide precise answers to specific questions given enough supervised data. for example, identifying domain-specific entities such as symptoms, medication and treatments, and semantic relations between them. learning to recognize this type of information in natural language, even academic language, is a challenging task, given the large number of varieties in which the same semantic fact can be stated. in this context, different annotation models have been designed to capture the semantic meaning in different domains and levels of discourse. token-level annotation models, such as amr @xcite , capture fine-grained semantic relations between elements in a natural language sentence, independently of domain, which means that only general-purpose relations can be recognized. in contrast, domain-specific annotation models can capture more detailed relations, such as 1 https://www.kaggle.com/ allen-institute-for-ai/ cord-19-research-challenge phylogenetic studies have shown that 2019-ncov and sars-cov belong to the subgenus sarbecovirus, but they are distantly related 5-8 , with a sequence identity of 79.6% at the whole-genome level 5 however, sars-cov, using the same receptor, was not detected in skeletal muscle by post-mortem examination in conclusion, sars-cov is the closest related virus to 2019-ncov for which a significant number of epitopes has been defined in humans (and other species), and that also causes human disease with lethal outcomes concept concept concept ,, action concept con concept in-context has-property is-a in-context is-a subject subject concept ^ action concept concept action in-context target target entails concept ++ action con concept concept action concept concept action concept in-place target in-context subject subject in-place is-a in-context causes drug-drug interactions @xcite or adverse reactions @xcite , but are less useful in cross-domain scenarios or in slightly different tasks. the sat+r (subject-action-target + relations) annotation model @xcite attempts to achieve a middle ground between domain independence and specificity. it has been developed as a general-purpose model but evaluated in different health-related scenarios @xcite , specifically in medline articles in spanish language. this model proposes two different classes of semantic relations, teleological and ontological, and 4 classes of entities. teleological relations @xcite are based on subject-verb-object triplets, while ontological relations are modeled after common relations found in upper ontologies, such as is-a, part-of, etc. previous attempts to apply this annotation model to medical text show that it is possible to obtain nearhuman-level accuracy with as little as 600 training examples @xcite . it has been previously used in three editions of the ehealth-kd challenge shared annotation campaign foot_0 . the main objective of this research is to build a manually annotated corpus that can be used to train knowledge discovery systems for analyzing the covid-19 research. for this purpose, we apply the sat+r annotation model to english sentences in the cord-19 corpus, manually annotating a small training set that can bootstrap a text-mining process for the entire corpus. the main contributions of this research are: the remainder of this paper is organized as follows. section 2 introduces the sat+r annotation model and presents the manually annotated corpus, its statistics and quality metrics, and details about the annotation process. section 3 presents and evaluates a baseline machine learning pipeline to automatically annotate the raw text in the cord-19 corpus. section 4 describes the most relevant concepts and relations automatically extracted from the a larger fraction of cord-19 corpus. finally, section 5 discusses the findings and lessons learned during this research and highlights possible lines of development, and section 6 presents the conclusions. ## corpus description this section introduces the cord-ann corpus, an ongoing linguistic resource which contains semantically annotated sentences extracted from covidrelated research papers in the cord-19 corpus. the corpus is constructed following the sat+r annotation model @xcite , which is based around concepts linked by actions, and additional ontological and teleological relations, such as is-a, has-property, causes, entails, among others. during an initial annotation trial it was identified that predicates, one of the semantic types defined in sat+r, produced a large degree of disagreement among annotators while providing little additional information, and for this reason this element is not considered. figure 1 shows three real sentences annotated with a variety of the semantic elements defined in the sat+r annotation model. a manual pre-selection process was carried out to choose the sentences that contain the most relevant content. the annotation process is realized in batches of 5 sentences, given the linguistic com-plexity of the academic language of the corpus and the fact that the annotators are not native english speakers. that being said, the majority of the annotators were undergraduate student or graduate students of different university degrees with at least a b2 english level. the annotation procedure was adapted from the methodology proposed by @xcite . the tool used for annotation is brat @xcite given the simplicity of its user interface. configuration files and related infrastructure are published in the project repository. the annotators were recruited through social media and most are from the academic institutions to which the authors are affiliated. a total of 21 different annotators were involved in the corpus creation, although several more showed some degree of interest but didn't complete any annotation batch. the degree of involvement varied widely, since two annotators account for approximately half of the corpus (51 and 48 batches respectively) while 9 annotators submitted only one batch. an annotation guide with several examples was published online, and the first batch from every annotator was cross-checked by the authors to provide feedback. afterwards, a continuous annotation campaign was managed through social media, with regular periods in which the annotators joined in an online forum to ask for clarifications or share their suggestions. the annotation process was carried out from march 28th until june 9th, when the first 500 sentences were completed. at the moment of writing, the annotation campaign has been temporarily halted in order to analyze the partial results obtained and decide the best course of action for the continuity of this research. this section presents a simple machine learning pipeline for the automatic annotation of entities and relations in raw sentences from the cord-19 corpus following the annotation model described in section 2. this pipeline is trained on the 1, 000 manually annotated sentences (i.e., the two versions of each annotated sentence), and executed on the remaining of the cord-19 corpus. a highlevel overview of the pipeline, shown in figure 2 , is composed of the following steps: for the entity model m e , the syntactic and morphological features include lemma, coarse and finegrained part-of-speech, dependency labels, generalpurpose entity labels (e.g., person, location, etc.), word shape, and several flags for specific patterns such as emails, numbers, and urls. for the relation model m r , the aggregated features correspond to the features of the tokens that comprise the two entities that participate in the relation, as well as the features of all the tokens in the smallest sub-tree of the dependency tree that contains both entities. the ultimate purpose of these models is to automatically extract relevant knowledge from the unlabeled pool of sentences. taking into account the complexity of this natural language comprehension task, there is always a trade-off between extracting as much knowledge as possible (i.e., maximizing recall) versus extracting knowledge as accurately as possible (i.e., maximizing precision). however, this trade-off can be explicitly controlled by measuring a degree of uncertainty σ in the models' predictions, and only outputting the elements (i.e., entities and relations) whose uncertainty is below a given threshold σ * . for the entity model m e , the raw marginal probabilities provided by the crf model are a possible measure of uncertainty, while for the relation model m r the same role is played by the raw logits provided by the logistic regression model. in both cases, the uncertainty can be estimated by computing the entropy h of the probability distribution of possible labels, given by the formula: where e is an annotation element, l @xmath0, and θ are the corresponding model parameters (i.e, the crf transition probabilities or the logistic regression weights). to evaluate the entity and relation models and estimate the optimal uncertainty threshold value σ * , 30 independent executions of a train-test loop with 80% of the training set are performed for different values of the uncertainty threshold. this analysis is extended to all different annotation elements and summarized in to better understand the trade-off between precision and recall, figure 3 shows the precision obtained at different uncertainty thresholds for each annotation element. however, since a very high precision can be achieved with an arbitrarily low recall, we only consider annotations for which the average recall is above 0.1 at the given threshold level. this guarantees that at least a 10% of the potential number of those semantic elements would be extracted from the unlabeled collection. ## preliminary insights in knowledge discovery this section presents a qualitative analysis of a knowledge discovery process that can be performed using machine learning models trained on this type of annotated data. for this purpose, the remaining sentences of the cord-19 corpus that were not used during the annotation process were fed to the machine learning models m e and m r and all predicted entities and relations were stored. finally, figure 4 shows a cherry-picked graph of relations built around the concept covid-19. this graph was constructed by sampling the most common relations that involve this concept, manually eliminating irrelevant tuples, such as in-context, and compacting very similar relations (in terms of ## discussion and future work this section discusses two important insights that arise from this ongoing research. first, we analyze the quantity and quality of the extracted knowledge, in an attempt to validate the approach and estimate the impact of its components. second, we discuss some lessons learned during the annotation process in the hope that future research can further improve on our work. arguably, two of the most relevant annotation patterns for the purpose of knowledge discovery are ontological relations such as is-a and has-property, and the subject-action-target triplets via the target and subject relations. the fact that these 4 relations have a relatively high number of instances extracted is promising. in contrast, the relations entails and causes show a significantly lower number of instances, as well as a lower performance. these relations are important from the point of view of knowledge discovery since they could directly link symptoms and treatments with evidences. however, more work is necessary to achieve a reasonable level of performance in their extraction. the machine learning models presented in this work where designed as an initial baseline. in a practical scenario more powerful models would be deployed. possible strategies include using a separate crf model for each entity class, and using a hierarchical model for the relation extraction step. another possible strategy is to repurpose existing models deployed in similar tasks. according to the 2020 edition of the ehealth-kd challenge, the top performing deep learning models in this task achieve f 1 values of 0.82 and 0.63 in entity and relation extraction, respectively @xcite . since they are also built on the sat+r annotation model, their adaptation to the corpus used in this research is straightforward. however, it should be considered that the reduced number of training examples in our corpus presents a significant challenge for any machine learning model, which motivates the use of transfer learning techniques. in this sense, additional annotated corpora with sat+r, such as the one presented in piad-morffis et al. (2020) can be used to bootstrap such as system. moving forward towards a fully-fledged ontology learning process still requires a significant effort after the annotation and model training. particularly, since each entity can be potentially represented in the text in different forms, a normalization step would be necessary to group all mentions of similar entities under a single concept. at the moment of writing we are developing approaches for automatic normalization of entity mentions based on wikidata, but the results are still not available. furthermore, in this work we have not taken into account the prediction confidence beyond its use as a threshold. weighting the number of mentions of each entity and relation with their respective confidence could provide an additional metric to determine what to include in the knowledge graph. one surprising and positive conclusion of our research is that crowd-based annotation efforts like this one, even with complex cognitive tasks involving deep semantics, are feasible. the annotators that participated voluntarily in this research were motivated by purely altruistic reasons, since there was no monetary incentive. we argue that, even if the covid-19 pandemic played a large role in this motivation, in general people can be motivated for this kind of work because of the positive social impact of the research. regarding annotation quality, there is a non-trivial amount of initial tutoring and feedback necessary, but once an annotator acquires a certain level of expertise, these efforts begin to pay-off since the experienced annotator becomes a potential tutor for new recruits. to improve the annotation process, two considerations are possible. first, the annotators can be automatically evaluated on a small trial set with automatic feedback. this way, a minimum level of initial expertise is guaranteed. afterwards, it is interesting to apply active learning strategies @xcite to automatically select which sentences to annotate. in addition to standard active learning approaches, where a classifier-based uncertainty or informativeness measure is used, in this context the inter-annotator agreement could be directly used to re-sample sentences for which the agreement is low, so that the more complex sentences receive more annotations. ## conclusion this paper presents the preliminary results of ongoing research. the majority of current research in the cord-19 corpus uses unsupervised or semisupervised approaches for knowledge discovery. we propose a supervised approach for extracting semantic relations and concepts in scientific articles using the cord-19 corpus. for this purpose we annotated 500 sentences using a general purpose semantic model. we propose a baseline textmining pipeline, trained on this data and executed on 100, 959 additional sentences of the cord-19 corpus, for automatically extracting relevant knowledge. this approach allows the discovery of relevant facts mentioned in research papers with fine-grained semantics, including causality, compositionality, and contextual dependencies. the annotated corpus and baseline implementation can be used as a starting point for developing more powerful knowledge discovery systems that can automatically analyze the growing body of scientific research related to the covid-19 epidemic and similar future scenarios.