Using MT for multilingual covid-19 case load prediction from social media texts (2023)

Abstract: In the context of an epidemiological study involving multilingual social media, this paper reports on the ability of machine translation systems to preserve content relevant for a document classification task designed to determine whether the social media text is related to covid. The results indicate that machine translation does provide a feasible basis for scaling epidemiological social media surveillance to multiple languages. Moreover, a qualitative error analysis revealed that the majority of classification errors are not caused by MT errors.

## introduction the work reported in this paper was carried out as part of a covid-19 case load forecasting project. similar to other work on covid-19 forecasting, e.g. @xcite @xcite , our baseline system used an auto-regressive approach to case-load prediction. several studies, however, have pointed to social media as a useful information source for this task @xcite . consequently, we wished to supplement our autoregressive forecasting with information from social media. specifically, we used the prevalence of mentions of covid-19 and related concepts in the social media emanating from a location to inform the case load predictions for that location. given the global nature of covid-19, we wished to make the solution scalable to multiple lan-guages. one approach would be to use multilingual classifiers. however, having reviewed the literature (see section 1.1) a decision was made to use machine translation (mt) to translate data sources in other languages into english and then to focus only on developing the text classification and prediction for english. this approach has several technical advantages, such as: (i) many existing nlp resources are designed to work in english (ii) adding new languages involves building a new mt system rather than developing out a new classification and prediction pipeline for a new language. interestingly, in this context, as the goal of mt was not translated text as such but enabling downstream text classification, we did not use any of the usual intrinsic mt evaluation strategies (automatic scores, human evaluation of translation quality criteria, error annotation and classification), but extrinsic evaluation, namely assessing and analysing the performance of the classifier on the translated english data. our research questions were: rq1 how useful is mt for this classification task? in other words: how close is the classification accuracy on translated text to the accuracy on original english texts? rq2 what is the relation between classification and translation errors? in other words: how many of classification errors happened because important terms were not translated correctly? in order to enable reproducibility and further research, all annotated data are publicly available. foot_0 ## method the focus of our experiment was on spanish and english social media data from online regional (north american) sources. the spanish data were translated by five mt systems trained on different corpus sizes and domains, and then given to the classifier trained on english manually labelled data. the task of case load prediction was based on the text topic, namely whether it is related to covid-19 or not. for this purpose, english set of tweets was manually labelled and used to develop a classifier. this classifier is then used both for originally written english texts as well as for spanish texts after they were translated by five mt systems. building an appropriate mt system for the given task poses some challenges. for the english-spanish language pair, being one of the "high-resourced" language pairs, there are generally a lot of parallel data available. still, for social media texts such data is not available. the genre also poses several additional challenges such as informal language, spelling and grammar errors, emoticons, hashtags, and a large number of domains/topics. in addition, different spanish dialects might represent a challenge, too: european spanish is generally different than american spanish, and there are differences between different american countries and regions as well. since there is no available data in the desired genre and domain (social media data related to corona virus), we built the initial systems on publicly available data which are partially similar to the data to be translated. since we had a decent amount of monolingual english in-domain data, we used this data to augment the initial training data by synthetic in-domain parallel corpora by back-translation which is a widely used practice in nmt @xcite @xcite . as for evaluation of the mt systems, not only training parallel in-domain data were unavailable, but also development and test data. in-domain data were written either in spanish or in english, and they were not translated into the other language by human translators. therefore, using automatic metrics such as bleu @xcite , chrf @xcite or the newest neural-based ones such as comet @xcite , was not possible. a possible solution could be to find a translator to generate the corresponding reference english tweets thus enabling automatic intrinsic mt evaluation. however, automatic scores would give only an overall idea about the translation quality, which is not necessarily correlated to the performance on the final task, namely the classification accuracy. moreover, the translation process requires effort and time. another solution would be human evaluation of translated texts, however the usual quality criteria (adequacy, fluency, readability) or error annotation and/or classification do not necessarily correlate with the performance of the final task. therefore, the evaluation and comparison of mt systems was performed extrinsically, by measuring the classification performance on translated texts. overall scores are presented in the form of classifier accuracy and compared to the accuracy on the english test data. furthermore, all classifier errors are analysed in depth to examine to which extent they are related to mt errors, and what are the differences between mt systems in this aspect. a qualitative analysis was performed too, to examine the nature of relevant mt errors. for this kind of evaluation, only correct reference labels of the spanish text were necessary, which requires significantly less time and effort than translation or manual mt evaluation. ## evaluation the first step although reference translations were not available for the test set, the very first step was to check the sanity of the initial mt systems (m and ms) on the tico development set. the bleu and crhf scores for these systems (trained on medical data and subtitles) were very high (63.9/63.2% bleu, 78.8/78.0% chrf), which indicated that the systems can be used and further developed. it has to be taken into account, though, that the development set is coming from the same domain and also contains the same spanish variant as the training data, therefore those scores are too optimistic for the actual task at hand. ## summary this work explored the ability of mt systems to preserve relevant content for a document classification task designed for covid-19 case load prediction. the results of extrinsic evaluation (classification performance) show that classification performance on the mt tweets is comparable with the performance on original english tweets, indicating that mt does provide a feasible basis for scaling epidemiological social media surveillance to multiple languages. furthermore, a detailed analysis of classification errors revealed that the majority of them are not caused by mt errors. moreover, most of those mt errors which triggered a classification error are related to specific terminology and can be improved in future work. other directions for future work include specific data selection for mt training, other methods for domain-adaptation and terminology translation, as well as using multilingual word representations from intermediate network layers instead of full translations.