A Corpus for Outbreak Detection of Diseases Prevalent in L atin A merica (2020)

Abstract: In this paper we present an annotated corpus which can be used for training and testing algorithms to automatically extract information about diseases outbreaks from news and health reports. We also propose initial approaches to extract information from it. The corpus has been constructed with two main tasks in mind. The first one, to extract entities about outbreaks such as disease, host, location among others. The second one, to retrieve relations among entities, for instance, in such geographic location fifteen cases of a given disease were reported. Overall, our goal is to offer resources and tools to perform an automated analysis so as to support early detection of disease outbreaks and therefore diminish their spreading.

## introduction in recent years, several contagious diseases have aroused. many of them, such as dengue, guillain-barré, zika and microcephaly, occur in tropical regions of the world as in latin america. there are also other diseases such as chagas, also referred to as american trypanosomiasis, which have existed for many decades, but not many resources are used to treat them, probably because they do not appear in densely populated areas and their usual hosts belong to populations of scarce socioeconomic resources. on the other hand, there are endemic diseases, such as hantavirus, which appear recurrently in some regions of argentina and chile, among other countries. finally, we are facing nowadays the covid-19 pandemics. those illnesses have many issues in common: 1) they are dangerous in terms of number of losses of human life or lifelong consequences to those who carried them, 2) their economic impact can be very important (as in loss of tourism incomes with hantavirus, and loss in almost all economic areas with the isolation imposed by different governments due to the covid-19 pandemics), and 3) when an outbreak occurs there is few available data that can support for data-driven decision making policies. in this work we attack the third issue, we argue that information including number of cases, host and location of the host, among others, is crucial in order to understand the propagation of the illness and diminish its spreading. when available, the information usually is of poor quality (incomplete, incorrect, inconsistent, not publicly available, and it lacks the timeliness attribute, among others). thus, our goal is to contribute with automated tools and resources that allow to alert and inform about possible outbreak diseases and some illnesses prevalent in latin america. to do so, we have constructed an annotated corpus based on promedmail @xcite , a reporting system dedicated to the dissemination of information on epidemics of infectious diseases. we have also implemented an initial baseline, whose results will allow us to answer the following questions a) are there in the articles any mention to a disease?, b) in case there is: to which?, c) how many cases are reported in the article, d) in which geographic location is the disease located?, e) if there are causes or possible causes mentioned, which are they? and f) what is the date of the report?. in the paper, we also emphasize the process followed to build the corpus which can serve as guide for future developments. in addition, we also present results on information extraction techniques applied over the corpus (namely, a named entity recognition (ner) rule-based technique). although preliminary, the algorithm shows promising results that can be further extended to analyze news and social media. the rest of the paper is organized as follows. section 2 presents previous work in digital surveillance in public health. in section 3 we describe the creation of an annotated corpus. section 4 describes the implemented methods for doing named entity recognition. section 5 shows preliminary results obtained from the baseline method proposed. finally, section 6 describes the conclusions reached so far and the future directions we are planning. ## related works digital surveillance has been an important topic in public health. a large number of the papers published in public health informatics is about the epidemiological surveillance based on the new data generated in the current digital era @xcite . research studies show that social media may be valuable tools in the disease surveillance toolkit used to detect disease outbreaks due to could be faster than traditional methods and to enhance outbreak response @xcite . thus, social media constitutes a source of information for the surveillance of various public health outcomes on a real-time basis @xcite . for instance, we see that twitter data has been used to aid in public health efforts concerned with surveillance, event detection, pharmacovigilance, forecasting, disease tracking and geographic identification, demonstrating positive results @xcite . public health surveillance is therefore a natural application for artificial intelligence techniques, the use of webbased data requires natural language processing approaches to extract the information. in this context, when national surveillance data are lacking, informal disease surveillance systems provide an opportunity to understand epidemiological trends @xcite . in this paper we have focused on building a corpus from promed @xcite . promed (the program for monitoring emerging diseases) is an internetbased reporting system for emerging infectious. regions and countries could benefit from complementing their undiagnosed disease surveillance systems with promed-mail tool @xcite . it is worth noting the effectiveness of promed as an epidemiological data source by focusing on coronaviruses @xcite . while promed is a reliable source of information, it is not currently equipped to provide detailed epidemiological data. for example, promed often does not report case or death counts beyond what is included in the text of a post. in order to extract this information from promed posts in a systematic way further analysis is required @xcite . one important work in this direction has been the platform for automated extraction of disease information from the web (padi-web) @xcite . this tool generates epidemiological information on diseases, locations, dates, hosts and number of cases for outbreaks mentioned in news and social media articles. ## creating the corpus in this section we present the process involved to construct the corpus based on promed-mail news articles (npa). first, we present the dataset. then, we explain the data cleansing processed followed, the annotation schema and criteria developed, a summary of the annotation guidelines and how we have been performing the annotation process. after, we show the statistics of our dataset and we explain the results of our inter annotator agreement. finally, we present the data statements of our data (bender and friedman, 2018). ## approaches for extracting information in this section we present a rule-based named entity recognizer applied to reduced npas and to whole npas. we also developed a connectionist state of the art method for doing ner. however, since we still do not have a sufficient amount of annotated texts as to train the neural network, we will only show preliminary results of our rule-based method for the titles of the notes and for the complete articles. we used freeling to perform named entity recognition and classification (nerc) of some entities and regular expressions to detect others. after, based on an analysis of a subset of our data we de-fined a rule-based method to detect named entities. in next sections we present a summary of our rule-based and machine learning methods we are developing. ## preliminary results in this section we show preliminary results of our rule-based method. 20 % of our dataset, not used for doing the analysis was used to test the results. we show the usual precision, recall and f1 metrics for each entity and an overall average score that considers all named entities. therefore, the annotated files were transformed to conll format with the script anntoconll.py provided by brat. a normalization and transformation was performed and conlleval perl script foot_7 was ran. tables 5 and 6 show results for npas and for the reduced articles. next, we do a brief analysis of our results. 1) for npas, generally those ne detected only by regex and lists of terms have the worst results (hst, or, tf, nt, ut, pt and cond). this is mainly due to differences with the annotation criteria and to the fact that lists of terms were not comprehensive enough. 2) nevertheless, in both cases (npas and partial notes) diseases had much better results. we assume that it is because there is a reduced number of diseases we are looking for. 3) for npas, those ne that were based on the use of freeling and of more elaborated rules had better results. 4) the main differences among the algorithm and the annotation criteria are between i) what is considered past, ii) the difference among or and tf (it occurrs in partial npas and npas), and iii) the dates. 5) the algorithm does not detect zika as a cause of microcephaly. this is due to the fact that zika is not in the list of possible causes (entity origin). 6) we also notice a difference among the annotation criteria and the development of the algorithm when seeing the results for loc ne. that, together with be/conll2000/chunking/conlleval.txt [accessed july 2020]. the fact that mainly in the title, countries are usually mentioned together with the abbreviation of a city (eg. brasil: (sp)) made that loc results are not good. with our gazetteers, we don't have a way to know the expansion of the abbreviations. furthermore, those abbreviations are usually non-standard. 7) normalizing dates, would enhance results of freeling date detection. 8) finally, uncertainty has bad results in reduced npas. it only appears in eight titles and there was a mismatch among those detected by the algorithm and those annotated by the annotators. overall, as we imagined, ner worked better for the title than for the npas. those results that are low are mainly associated with the lack of normalization and the difference with the annotation criteria (eg. or and tf). finally, even though the results do not seem encouraging, there are entities that were recognized correctly by the algorithm but no by the annotators (eg. in the ministry reported that there are 24.011 suspected cases of (...)), the algorithm correctly recognized 24.011 as noc and suspected cases as ut, but the annotator did not. this shows that the annotation and annotation criteria still have to be improved. ## conclusion we have just presented an initial annotated corpus based on promed-mail which can be used for early detection systems of outbreak diseases prevalent in latin america. while there are several social media sources for disease surveillance, there is a lack of tools that can automatically provide detailed epidemiological data. thus, the whole process of building the corpus was oriented to aid the extraction of useful health facts. with this corpus, it is possible to analyze medium/long articles from reliable sources such as promed-mail and answer queries regarding detailed and direct diseases incidences. this can be useful for understanding epidemiological trends. we have also proposed an initial baseline rule based algorithm that automatically extract diseases related entities. the results reported are promising and we also plan to work on relation extraction. we are developing a baseline based on co-occurrence of named entities. the analysis of the preliminary results shows different criteria between the algorithm rules and the annotation criteria. this analysis will help us review and enhance our corpus. finally, the rule-based method is very laborious and is not always useful to detect entities without understanding the contexts in which they appear.