Cross-lingual Classification of Crisis-related Tweets Using Machine Translation (2023)

Abstract: Utilisation of multilingual language models such as mBERT and XLM-RoBERTa has increasingly gained attention in recent work by exploiting the multilingualism of such models in different downstream tasks across different languages. However, performance degradation is expected in transfer learning across languages compared to monolingual performance although it is an acceptable trade-off considering the sparsity of resources and lack of available training data in low-resource languages. In this work, we study the effect of machine translation on the cross-lingual transfer learning in a crisis event classification task. Our experiments include measuring the effect of machine-translating the target data into the source language and vice versa. We evaluated and compared the performance in terms of accuracy and F1-Score. The results show that translating the source data into the target language improves the prediction accuracy by 14.8% and the Weighted Average F1-Score by 19.2% when compared to zero-shot transfer to an unseen language.

## introduction we are interested in discovering methods to enhance the detection of emerging crises in social media and the transferability of a classification model fine-tuned on a specific language to other languages. crisis event detection typically relies on two aspects: (1) the detection of burstiness of certain keywords or trends in the timeline, and (2) the classification of the detected bursts and whether they indicate the occurrence of a disaster or not. the former aspect can be obtained using unsupervised learning to cluster posts that have certain commonalities such as common keywords, time, and location. however, the latter usually depends on the use of supervised learning algorithms to filter out non-relevant posts since bursts can occur for non-crisis related events such as concerts and media events, political events, and other trends that can interfere with the task of responding to emergencies. it is especially important to develop methods to increase the accuracy of classifying relevant posts to support multilingual settings and therefore help provide better response to emergencies. although the use of machine translation for crosslingual transfer learning has shown promising results, there are several drawbacks to the existing work including quality of machine translation, limited parallel data, and structural differences which affect the overall performance of the final model. in this work, we conducted several experiments to assess the effect of machine translation in bridging language gaps for zero-shot cross-lingual classification of crisis-related tweets. additionally, we investigated potential limitations on the final predictions. our study focused on transfer learning between english and arabic languages by finetuning a multilingual pre-trained language model like xlm-r for disaster type classification. despite the inherent heterogeneity of the two languages, our results surpassed existing benchmarks for more linguistically homogeneous languages such as english and spanish. our experiments specifically targeted factors that could influence the performance of existing benchmarks, thereby enabling researchers to address these limitations in future studies. although our focus is on the crisis events, the approaches can be expanded to other types of events. the structure of the rest of this paper is as follows. a background about the classification task and relevant knowledge about the different language models is discussed in section 2. descrip-tion of chosen datasets and the required handling process for our task is shown in section 3 and 3 respectively. section 4 explains the experimentation settings to achieve our objective of measuring the effect of machine-translation on the cross-lingual transfer. we discuss our results in details in section 5 along with an investigation of the possible factors that might have affected the transfer using the machine-translation. we also demonstrate the challenges of cross-lingual transfer learning of the data level and task level in section 6. while in section 7 we showcase and compare relevant work, we finally suggest future directions for research in section 8. ## background classification tasks have gained a significant amount of attention recently. in the domain of event classification, different directions have been pursued including binary classification (i.e., whether the text indicates an event or not), multi-class classification, and multi-label classification. multi-class classification ranges in granularity from event type to fine-grained aspects of the text content. we look specifically into cross-lingual classification of social media content for recent contributions in the area. deep learning models can accommodate the complexity associated with social media data including noise and a lack of structure compared to traditional machine learning algorithms such as svm, naïve bayes, and random forests which may suffer from a decline in performance with the increasing complexity of the data @xcite . social media posts, especially tweets, have been very useful in recent years for many tasks and goals including event detection and classification. among other types of events (e.g., sports, music, political, .etc), disaster detection and classification has a special characteristic: urgency and need of rapid response. taking advantage of crowd sourced information posted by people in real-time may play a large role to provide timely and proper response. considering that a specific event can be reported in multiple languages emphasises the need for multilingual and cross-lingual tools that do not discard helpful information just because it is in a different language. unlike traditional neural networks, the introduction of the transformer-based language models such as generative pre-trained transformer (gpt) @xcite and its successors, and bidirectional encoder representations from transformers (bert) and the models built upon it have transformed the area of natural language processing. following bert, which is pre-trained on english wikipedia (2,500m words) and bookscorpus (800m words), emerged other bert-based models such roberta @xcite and xl-net @xcite introducing an improvement over bert's performance since they are pretrained on more data than the original bert @xcite . particularly, roberta has been pre-trained on 144 gb of data in addition to the 16gb that bert is pre-trained on while xlnet was trained with over 130gb of textual data including the original 16gb of bert's. following the release of multilingual bert (mbert) which is a bert model trained on wikipedia text in 104 different languages, other multilingual models such as xlm @xcite and xlm-roberta @xcite have been released as well. crosslingual language modeling (xlm), like other transformer-based language models, was trained with a masked language modeling (mlm) objective. additionally, it is trained with a translation language modeling (tlm) which relies on the availability of a dataset with parallel sentences. however, xlm-roberta only uses the mlm objective and trained on a huge corpus of text in 100 languages acquired from the commoncrawl datasets in a roberta fashion. arabic is a widely spoken language, with over 400 million people around the world, according to (unesco, 2022). however, there is a scarcity of resources when working on machine learning, especially for domain-specific tasks such as crisis event classification which ignites the need for automated solutions to fill this gap. to address the issue of limited training data for low-resource languages, researchers have employed techniques such as transfer learning @xcite @xcite , unsupervised learning @xcite @xcite , and data augmentation @xcite @xcite . additionally, initiatives like the masakhane project @xcite have aimed to build machine translation systems for african languages using collaborative and community-driven approaches. these efforts strive to make machine learning accessible for low-resource languages. while significant progress has been made, further research and collaboration are essential to enhance the quality and availability of machine translation for low-resource languages. ## data we used crisisnlp foot_1 @xcite as an english benchmark dataset for our experiments. crisisnlp is a widely used hand-labelled twitter dataset and consists of tweets collected during ten different disasters including earthquakes, floods, epidemics and other types detailed in ## experiments we examine the effectiveness of multilingual bert-based models, specifically the xlm-roberta model, in the cross-lingual transfer learning of a model fine-tuned on a source language and evaluated on an unseen target language for the disaster events classification task. xlm-r has shown considerable improvement over mbert on many benchmarks @xcite . our intention is to measure the machine-translation effect on the prediction performance by translating the target data into the source language (and vice versa) before testing the model. we are aware that the translation of social media text will not be as accurate as translated formal text due to its informality, misspelled words, noise, slangs and dialects. however, translation might provide a working solution for the scarcity of training data in different languages exploiting the abundance of available english data. our experiment consists of three parts. in the first part, we fine-tune a multilingual model (xlm-r) on classifying english disaster events and evaluate it on a labeled dataset consisting of original arabic tweets (non-translated). the results of this part will give us a benchmark to compare the second part results with and answer our question (i.e., does translation improve the prediction performance when transferring a model to another language?). the second part involves translating the test set into english before evaluating the finetuned model. finally, we translate the source data into the target language and test on the target data (i.e., arabic). for machine translation, we use facebook's m2m-100 model @xcite which translates between a hundred different languages in any direction. those results will also be compared to monolingual performance of the model on both languages. ## results the first set of results found in in a monolingual setting (english to english and arabic to arabic). in the english setting, the accuracy and f1-scores (average and micro) are relatively high (96%, 96.2%, and 96% respectively) while they are (91.13%, 91%, and 91.1% respectively) in the arabic setting. the possible reasons for this 5% decrease might be the number of training samples since the size of the english data was 17k in total while the arabic was about 5k after cleaning and balancing. the other reason might be that xlm-roberta was originally pre-trained on more english data than arabic @xcite . the motivation of performing a monolingual examination of the model is to set a benchmark for our model after data pre-processing and class manipulation. the original data is labelled for tweet content whether it is (caution and advice, infrastructure and utilities damage, casualties, etc.). most of the existing work uses these classes (with minor modifications) for testing their models. however, we use the disaster type labels (storm, epidemic, etc.) to classify the tweets into the type of event. the main purpose is because the two datasets are labelled differently for content, more details about the labelling are found in section 6. disaster type labels allow us to first de-termine whether a tweet is about a disaster event (relevance), and second to determine what type of disaster is the tweet talking about. finer granularity can be adopted later for classifying the type of information provided by the tweets. the second set of results can be divided into three parts: (1) the result of evaluating a model fine-tuned on english data and tested on original arabic data (zero-shot) and (2) the result of evaluating the model on the same arabic dataset after translating it to english (target-translation). the latter experiment shows an improvement in f1-score by 8.2% when testing on a translated dataset as compared to the former while (3) the third score is when we translated the training data into the target language (source-translation) which has shown a substantial increase of 19.2% in f1-score over the zero-shot setting. although translating the test set to the source language has increased the accuracy of the classification, however, the result is still not close to the monolingual performance. to explore this limitation, we investigated the potential reasons behind it as follows: ## challenges one of the challenges of cross-lingual transfer learning is the heterogeneity of the source and target data. even when we acquire disaster datasets in two languages, the way they were labelled can affect the quality of transfer. for instance, crisisnlp was labelled for information conveyed in the tweet text as discrete labels (e.g., infrastructure damage, injured people, etc.) while kawarith is using multilabel classification where one tweet can have more than one label (e.g., infrastructure damage and injured people). also, additional labels are found in kawarith that are not in crisisnlp such as opinion and criticism. such issues can impose challenges in mapping the labels to the closest possible ones and sometimes discarding some samples. types of disasters covered in each dataset is also a challenge for disaster type classification. while crisisnlp contains english data about earthquakes, floods, hurricanes, cyclones and diseases, only two types are in common with the arabic data which results in discarding the uncommon types when classifying disaster types. zero-shot transfer learning from english to italian has been examined by @xcite using a convolutional neural network (cnn) exploiting the shared embeddings provided by muse (multilingual unsupervised and supervised embeddings), a python library. although the training data was relatively large (45k english tweets), they achieved a micro-averaged f1-score of 0.52 for italian when training the model on the english data only. similarly, @xcite investigated the generalisation abilities of mbert for event detection and classification for italian and english. two scenarios were tested: event detection (i.e. binary classification) and event detection and classification (i.e. multiclass classification). they experimented with zero-shot learning by training/fine-tuning the model on one language and evaluating it on the other language that it has never been seen in the training. for the zeroshot multiclass scenario, the f1-score was 42.86 when tested on italian which was improved to 55.38 when the model was fine-tuned with a mixture of data in both english and italian. a summary of the most relevant work in zero-shot transfer learning is shown in for transferring to arabic language, @xcite and @xcite have studied the transfer of mbert to arabic language for xnli task with very close accuracy in both works. the former has explicitly provided the language syntax to the model to address the challenge of cross-lingual transfer of typologically different languages. latter work supported the approach of using the target language dev set for model selection to increase the accuracy and compared the results of both using english dev and target dev. indeed, using target language dev set showed improvement over using source language for model selection. our study aimed at investigating the impact of using machine translation to leverage the crosslingual capabilities of multilingual transformerbased models such as xlm-roberta. specifically, we tested both training data translation and test data translation, in order to mitigate the potential performance loss that can occur when testing on an unseen language. our findings revealed a considerable improvement in performance, which can be particularly useful for transferring a classifier trained on a resource-rich language to a resourcepoor language by translating the same training data into a set of target languages providing an acceptable performance when lacking task data in the target language. however, further research is needed to explore additional approaches that can enhance cross-lingual transfer learning and achieve comparable performance to monolingual models, such as the use of ensemble methods to boost the classification of individual learners. future work will also include a comparison of different machinetranslation models for the same task. overall, our study highlights the potential of machine translation as a powerful tool for cross-lingual transfer learning, and provides a foundation for future research to further improve the performance of multilingual models on text classification tasks across different languages.