Detecting Urgency Status of Crisis Tweets: A Transfer Learning Approach for Low Resource Languages (2020)

Abstract: We release an urgency dataset that consists of English tweets relating to natural crises, along with annotations of their corresponding urgency status. Additionally, we release evaluation datasets for two low-resource languages, i.e. Sinhala and Odia, and demonstrate an effective zero-shot transfer from English to these two languages by training cross-lingual classifiers. We adopt cross-lingual embeddings constructed using different methods to extract features of the tweets, including a few state-of-the-art contextual embeddings such as BERT, RoBERTa and XLM-R. We train classifiers of different architectures on the extracted features. We also explore semi-supervised approaches by utilizing unlabeled tweets and experiment with ensembling different classifiers. With very limited amounts of labeled data in English and zero data in the low resource languages, we show a successful framework of training monolingual and cross-lingual classifiers using deep learning methods which are known to be data hungry. Specifically, we show that the recent deep contextual embeddings are also helpful when dealing with very small-scale datasets. Classifiers that incorporate RoBERTa yield the best performance for English urgency detection task, with F1 scores that are more than 25 points over our baseline classifier. For the zero-shot transfer to low resource languages, classifiers that use LASER features perform the best for Sinhala transfer while XLM-R features benefit the Odia transfer the most.

## introduction people all over the world use social media, e.g. twitter, facebook, to communicate with the outside world during crises that are either natural or man-made. during an emergent crisis, people post to report their well-being, ask for help, or give updates about the ongoing situation. this type of text data can be utilized to provide situational awareness to support missions such as humanitarian assistance/disaster relief, peacekeeping or infectious disease response. however, with the existence of more than 7,000 languages worldwide, automated human language technology does not exist for many languages. 1 a possible solution to this problem is to transfer models learned in high resource language settings such as english to low resource languages @xcite . in addition, there has been significant research in the use of transfer models in semantic analysis of texts such as sentiment @xcite and emotion @xcite . to this end, we collect and release english, sinhala and odia urgency datasets that consist of tweets relating to natural crises, annotated with urgency status. 2 to demonstrate that we are able to effectively transfer the task of urgency detection from english to low-resource languages, we use english annotated tweets for training, and sinhala/odia annotated tweets for evaluation only, therefore, exploring zeroshot transfer. specifically, we consider the following two tasks: a) english classification, for which we hold out 20% of the english dataset for evaluation, and the remaining 80% for training; b) cross-lingual classification, for which we use the entire english dataset for training and the corresponding sinhala or odia dataset for evaluation. for the english classification task, we implement classifiers of different architectures adopting various embeddings including bert @xcite , roberta @xcite and xlm-r @xcite , which we use to extract features, then train a classifier that takes the contextualized representations of the tweets into account. for the cross-lingual classification task, we build classifiers using the same set of architectures, but deploying various cross-lingual embeddings that are constructed using different methods: laser @xcite and xlm-r @xcite . for both tasks, we employ semi-supervised approaches by generating pseudo-labels for a large amount of unlabeled tweets that are crisis related, in order to improve system performance. last but not least, we ensemble different classifiers to boost performance further. ## dataset tweets about many natural and human-induced disasters such as earthquakes, typhoons, and landslides are collected by @xcite . we annotate a subset of them at the tweet-level on the figure -eight data annotation platform foot_0 as seen in figure 1 . the annotation tag set comprises the following four levels of urgency: • extremely urgent: aspects of the tweet refer to an extremely urgent and difficult situation; • somewhat urgent: tweet contains some content that could be considered urgent but it is not as certain as the two categories above; e.g. mt @dineshakula med supplies required in bir hospital. out of medical supplies http://t.co/4pphg2avhg #kathmandu #nepalquake #hmrd as it can be difficult for an annotator to decide whether a tweet is urgent, we provide four scales of urgency which then can be converted to a binary set of tags: urgent vs. not urgent. the level of agreement between multiple contributors, confidence score, is weighted by contributors' trust scores and calculated on average as 68.6%. 4 52 test questions with correct labels were distributed throughout the task for which the annotators needed to maintain a 70% accuracy. after removing duplicates and inconsistencies, the final data consists of 1, 919 annotations as summarized in however, this also might have caused the annotation task to be more challenging. when we analyzed the annotations, we noticed that some of the tweets about rescue efforts were particularly confusing: tweets that are general status updates about an incident and more critical tweets that are asking for help are both labeled as urgent, without making a distinction between the two. this demonstrates one of many aspects of the difficulty of annotating for urgency, partially due to the tendency of labeling a tweet as urgent even though the urgency of the event is past. ## methodology we explain our approaches to data preprocessing, english monolingual and low resource cross-lingual classification in sections 3.1, 3.2 and 3.3, respectively. ## experiments we report macro precision, recall and f1-scores for the english classification task and the cross-lingual classification tasks in tables 4, 5 and 6 respectively (best scores for each section are underlined and shown in bold). for macro averaging, we calculate precision, recall and f1 scores for both positive and negative labels, then report their unweighted mean. column original refers to results using the original dataset that is human-annotated and 16k with synthetic data refers to results on larger datasets that are generated with the method mentioned in 3.2.3. since the evaluation datasets for all the tasks are small in size, for each experiment setting that is cheap to reproduce, we report the mean and standard deviation of 30 independent experiments to reduce inconsistencies and improve confidence. as a baseline, we report results of a classifier which assigns a label randomly based on the label distribution in english training data. we use scikit-learn @xcite for svm and random forest classifiers and the pytorch platform foot_3 for the deep learning classifiers. for incorporating all the deep pre-trained contextual models, our codebase heavily relies on the transformer implementations by huggingface, which has the advantage of switching to any future large-scale pre-trained models easily. foot_4 ## related work crisis nlp foot_5 website provides social media datasets and classifiers that are about various disasters in several languages, i.e. english, spanish and french, which are all high resource languages. for low resource languages, due to very limited amount of data, transfer learning approaches must be adapted that transfer a high-resource model to a low-resource language @xcite . the work of @xcite apply a manual feature based approach to transfer urgency labels from english to several low resource languages combined with active learning to increase the amount of labels. recent successful techniques in transfer learning, however, use cross-lingual embeddings combined with deep learning based classifiers. cross-lingual embeddings map words in different languages into same semantic space and among them, we use projection based approaches, i.e. vecmap and procb, rather than parallel corpora based ones e.g. biskip @xcite due to their superior performance. this has been shown to work well for sentiment @xcite and emotion @xcite . in addition, after the success of contextual language models such as bert @xcite and roberta @xcite in many nlp tasks, their multilingual versions became available i.e. multilingual bert @xcite and xlm-r @xcite which we experimented with based on their availability for our languages. their adaptation to low resource settings, e.g. fine-tuning with small datasets, is not trivial and is not as reliable as in high resource settings. as such, we show how this can be achieved with our experimental setup. specifically, we use a self-learning method by voting @xcite to increase the size of the high resource language dataset on unlabelled crisis nlp tweets. we decide not to use tri-training (zhi-hua @xcite due to the size of original english data despite the fact that tri-training has shown good results in domain-shift nlp tasks. ## conclusion in this study, we release an urgency dataset consisting of english tweets about natural crisis and their urgency status. in addition, we release two evaluation datasets for urgency detection in sinhala and odia. we train monolingual classifiers for english and cross-lingual classifiers for sinhala and odia that are zero-shot learners. for the design of our classifiers, beside exploring different architectures, we adopt different monolingual or cross-lingual embeddings that are either pre-trained or constructed by using different methods. due to limited amount of labeled data, we generate synthetic data to improve the system performance, and ensemble classifiers to boost the performance even further. we conclude that if synthetic data can be produced with high confidence, then it is helpful in transfer between domains that have similar distribution of labels. specifically for english urgency detection, the best performing classifier utilizes contextual features produced by pre-trained roberta model and among non-contextual embeddings, in-domain embeddings out-perform out-of-domain embeddings. for cross-lingual transfer, classifiers that incorporate laser features perform the best for transferring to sinhala while xlm-r features benefit the most in transferring knowledge of urgency detection to odia. finally, in the absence of pre-trained contextual embedding for a low resource language, we also demonstrate alternative ways to achieve similar performance using cross-lingual embeddings constructed by projection based approaches, i.e. vecmap and procb.