Multilingual Epidemiological Text Classification: A Comparative Study (2020)

Abstract: In this paper, we approach the multilingual text classification task in the context of the epidemiological field. Multilingual text classification models tend to perform differently across different languages (low- or high-resourced), more particularly when the dataset is highly imbalanced, which is the case for epidemiological datasets. We conduct a comparative study of different machine and deep learning text classification models using a dataset comprising news articles related to epidemic outbreaks from six languages, four low-resourced and two high-resourced, in order to analyze the influence of the nature of the language, the structure of the document, and the size of the data. Our findings indicate that the performance of the models based on fine-tuned language models exceeds by more than 50% the chosen baseline models that include a specialized epidemiological news surveillance system and several machine learning models. Also, low-resource languages are highly influenced not only by the typology of the languages on which the models have been pre-trained or/and fine-tuned but also by their size. Furthermore, we discover that the beginning and the end of documents provide the most salient features for this task and, as expected, the performance of the models was proportionate to the training data size.

## introduction monitoring and containment of infectious disease outbreaks have been an ongoing challenge globally. whether previously with ebola or today with the covid-19 pandemic, surveillance has remained a key component of public health strategy to contain the diseases. the ability to detect disease outbreaks in an accurate and timely manner is critical in the deployment of efficient intervention measures. for instance, ebola cases and outbreaks need to be immediately detected in order to be contained and stopped. a delayed response can have a significant economic and social impact, in addition to increased morbidity and mortality rates. thus, the detection needs to be done as soon as the first reports appear, and, naturally, as such reports may not be in english, there is a need for effective multilingual surveillance systems. in recent years, there has been a rapid increase in data generated as a result of the progressive evolution of the internet. the proliferation of digital data sources provide an avenue for data-driven surveillance, referred to as epidemic intelligence. epidemic intelligence involves the collection, analysis, and dissemination of key information related to disease outbreaks, with the objective of detecting outbreaks and providing early warning to public health stakeholders @xcite . natural language processing (nlp) techniques have made it possible to analyze data from web sources, such as social media, search queries, blogs, and online news articles for health-related incidents and/or events @xcite . data-driven epidemic intelligence can be viewed as a twostep process comprising a classification task followed by the event extraction task @xcite , which can help to predict the epidemic disease dynamics and where the next outbreak of epidemic would most likely happen. the classification task entails the identification of texts relevant to disease outbreaks from a large collection of data. considering, for instance, a dataset of online news articles, the articles that report an outbreak of disease are separated from those which do not. next, the event extraction task takes the identified relevant documents as input and predicts health-related events with arguments such as the disease name and the location where the outbreak was reported. like any other task that involves text analysis using nlp approaches, ambiguity is a key challenge when dealing with epidemic-related text data. ambiguity manifests itself where a sentence in a document may have mentioned a disease, but may not necessarily be reporting on an outbreak of a disease. for instance, with the ongoing coronavirus pandemic, there are numerous news articles posted daily reporting on various aspects related to the disease. it becomes a challenge to extract the few relevant news articles that are of interest to the epidemiologist, articles that report on the number and location of new cases. epidemic reporting is also characterized by news reports from divergent sources and languages which further compounds computational epidemiology. furthermore, when working in a multilingual setting of real-world data, another challenge arises from the lack of annotated data for low-resource languages. the creation of such data can be expensive, time-consuming, and requires human expertise to annotate, hence it is a labor-intensive task. in view of these challenges, appropriate nlp approaches are required in order for data-driven epidemic surveillance to be successful. therefore, we seek to provide a comprehensive quantitative study of lowshot text classification models applied on a dataset comprising news articles about disease outbreaks from several diverse language families namely, english, greek, french, russian, polish, and chinese. we seek to compare state-of-the-art approaches for epidemiological text classification from both deep learning and classical machine learning techniques by training a variety of models and evaluating them in several circumstances, in order to analyze their application in a real-world scenario. to the best of our knowledge, this is the first extensive study to specifically evaluate the performance of multilingual epidemiological text classification methods. the remainder of this paper is organized as follows. section 2 reviews works related to nlp-based epidemic surveillance systems, section 3 describes the dataset used in the study, while the experiment setup and results are presented in section 4. finally, we provide a discussion of the results in section 5, and the conclusions and possible suggestions for future research are presented in section 6. ## related work there are a number of empirical works targeted at the application of nlp for the detection of disease outbreaks. among them is data analysis for information extraction in any language (daniel), a multilingual news surveillance system that leverages repetition and saliency (the beginning and the end of a news text often comprises the salient zones), properties that are common in news writing @xcite . by avoiding grammar analysis and the usage of language-specific nlp toolkits (e.g., part-ofspeech tagger, dependency parser) and by focusing on the general structure of journalistic writing style @xcite , the system is able to detect crucial information from news articles. furthermore, the multilingual nature of the system enables global and timely detection of epidemic events since it eliminates the requirement for translating local news to other languages for subsequent transmission. the system can easily be adapted and scaled to extract events across languages, therefore, being able to have a wider geographical coverage. reactivity and geographic coverage are of paramount importance in epidemic surveillance @xcite . similar to daniel is biocaster @xcite which has produced good results in analyzing disease-related news reports and in providing a summary of the epidemics. the biocaster, an ontology-based text mining system, processes, and analyzes web text for the occurrence of disease outbreak in four phases namely, topic classification, named entity recognition (ner), disease/location detection, and event recognition. the naïve bayes algorithm is used for the classification of the reports for topical relevance. the news stories identified to be relevant to disease outbreaks are propagated to the subsequent levels of processing. the major limitation of the biocaster is its inability to detect disease outbreaks beyond the eight languages (chinese, english, french, japanese, korean, spanish, thai, and vietnamese) present in its ontology @xcite . therefore, scaling the system to work across different languages requires manually updating the ontology with information for new languages. in addition, the system is not publicly available, except for the ontology. the ecohealth alliance global rapid developed the identification tool system (grits), an application that provides automatic analyses of epidemiological texts. the system extracts important information about a disease outbreak, such as the most likely disease, dates, and countries where the outbreak originates. the pipeline for grits entails transforming words to vectors using the term frequencyinverse document frequency (tf-idf) method, by first extracting features using pattern-matching tools, before applying a binary relevance-based classifier to predict the presence of a disease name in the text @xcite . the system translates non-english documents using the bing translator, which can potentially introduce errors to subsequent analysis steps if the translation is incorrect @xcite . internet search queries have also been exploited for disease surveillance. in one study, internet searches for specific cancers were found to be correlated with their estimated incidence and mortality @xcite . monitoring influenza outbreak using data drawn from the web has also been previously explored. two different studies, one that analyzes large numbers of google search queries to track influenza-like illness in a population @xcite and the other that examines search queries from yahoo foot_0 related to the same aforementioned infectious disease @xcite were conducted in this context. in recent years, various studies have utilized social media data for infectious disease surveillance @xcite . mostly, twitter data, has been used for disease tracking @xcite @xcite , outbreak detection @xcite @xcite and predicting the likelihood of individuals falling sick @xcite . news media has also been used to give early warning of increased disease activity before official sources have reported @xcite . the studies have demonstrated the potential value of harnessing data-driven approaches for epidemic surveillance. while prior attempts to develop multilingual epidemic surveillance systems have been made, the proposed systems are predominantly ontology-based, which require the ontologies to be updated on an ongoing basis in order to improve their performance and ensure broad coverage of different languages. recently, there has been a growing interest in exploring the multilingual nature of the data in different domains, which could also be beneficial to the epidemiological domain. existing multilingual methods use word representations that are either learned jointly using parallel corpora @xcite or via mapping separately trained word embeddings in different languages to a shared space through linear transformations @xcite . the embedding spaces of the different languages ought to have a similar structure for the linear mapping from one space to the other to be effective. more recently, effective cross-lingual representations have been developed, by simultaneously training contextual word embedding models over multiple languages, without requiring mapping to a shared space. such models learn representations of unlabeled data that generalize across languages. examples include the cross-lingual language model (xlm) @xcite and multilingual bert @xcite which are pre-trained on wikipedia data for different languages. bert has been shown to allow effective cross-lingual transfer on different downstream tasks. this includes, document classification @xcite , named entity recognition (ner) @xcite , sentiment classification @xcite , neural machine translation, @xcite , and dependency parsing @xcite . ## dataset we extend the dataset proposed by @xcite to include additional languages so that it covers news articles from several, diverse language families: germanic: english (en), hellenic: greek (el), romance: french (fr), slavic: russian and polish, and chinese that descends from the sino-tibetan family. the articles were obtained from different online news sources with articles relevant to disease outbreak being obtained mainly via the program for monitoring emerging disease (promed) 2 platform, which is a program from the international society for infectious diseases that tracks infectious disease outbreaks and acute exposures to toxins, across the world. the process of gathering the data involved first, retrieval of promed news articles published between august 1, 2013, and august 31, 2019. the articles clearly annotate the title, the description that captures details about the reported disease, location, date, and the source uniform resource locator (url) where the article was originally published. the source urls were extracted and their corresponding source documents downloaded to form the relevant documents of the dataset. on the other hand, the irrelevant news articles consist of general health-related news, but without direct or indirect mentions of disease outbreaks (e.g., plague, cholera, cough), as well as general news like politics and sports. most of the irrelevant documents were obtained from the news category dataset @xcite comprising huffpost foot_2 news articles for the period 2012 to 2018. the news articles cover various topics such as culture, politics, wellness, among other topics. all polish chinese russian greek french english train 5,074 (10.8) 241 (7.4) 300 (2.6) 296 (9.45) 253 (6.7) 1,593 (10.9) 2,365 (11.7) validation 1,250 (10.9) 54 (7.4) 71 (2.8) 60 (10.0) 68 (10.2) 388 (13.4) 583 (12.6) test 1,250 (10.5) 46 (13.0) 75 (6) 70 (10.0) 63 (4.7) 434 (12.4) 614 (12.8) to simulate the real scenario of news reporting, we set the number of documents reporting disease outbreak (relevant documents) to be no more than 10% of the total dataset. the statistics of the dataset are presented in we split the data, with a total of 7, 574 articles, into training, validation, and testing sets. the training set comprises a total of 5, 074 documents, while the remaining documents were shared equally between the validation and the testing sets, that is, 1, 250 documents for validation, and 1, 250 documents for testing, stratified by language, as shown in ## experiments the metrics considered in the evaluation of the models are precision, recall, and f1-score. measuring recall is particularly important because of the risk posed by not identifying all the positive cases, with regard to disease outbreaks. ## discussion out of all the models, deep learning bert-based models were the best performing models, in terms of both f1 score and recall measures. the good performance can be attributed to the deep network architectures and large corpora used to train transformer-based pre-trained language models (pmls) such as bert, which enable learning of rich text representations. moreover, bert fine-tuning performed better compared to the feature-based approaches, where fasttext and bert embeddings were used as input features to cnn and bilstm classifiers. essentially, the plms end up learning universal language representations that are beneficial to downstream tasks. the high precision and low recall noted in the machine learning models suggest that the models are unable to detect the relevant class well but are highly reliable when they do. this implies that while the classifiers returned reliable results, the machine learning models had a high false-negative rate, hence a few of all relevant results were returned. the approaches based on fine-tuned bert uncased generally struck a good balance between precision and recall. vgcn+bert performed particularly well for polish, chinese, and russian. the model utilizes graph embeddings produced by integrating local information captured by bert and global information from the vocabulary graph that is based on word co-occurrence information. both the local and global information interact with each other through a self-attention mechanism during the learning process. the interaction introduces useful global information to bert, which contributes to the improved results across all the languages, including the low-resource languages. with regard to the contribution of various document segments on performance, it was observed from the results that, the beginning and the end of the text combined had the highest recall and f1 score. this was particularly the case for models based on bert namely, vgcn+bert and bert fine-tuned models. this can be explained by the fact that the beginning paragraphs in an article often capture the most important information, which informs the reader what the story is about. on the other hand, the last part of the article tends to provide a summary of the article. the performance of the model improved proportionately with training data size. this is in line with neural network models, which require large amounts of data to train and evaluate. the competitive performance even with a small amount of data results from the transfer of knowledge from the pretrained language model, trained on a large corpus, to the specific task of classifying epidemic text. this demonstrates the extent to which transfer learning can benefit the process of extracting useful information from multilingual epidemiological text. ## conclusions building effective epidemiological surveillance systems is of high importance these days. detection of news reports on disease outbreaks is a crucial requirement of such systems. in this paper, we study in detail the performance of different methods on the task of epidemiological news report detection. the evidence presented in this work suggests that the models based on fine-tuned language models and/or graph convolutional networks achieve very good performance (> 90%) on the classification of multilingual epidemiological texts, not only for high-resource languages but also for low-resource languages. in future work, we will consider the perspective of pursuing the task of epidemiological event extraction from news texts in low-resourced languages.