Multimodal Semi-supervised Learning for Disaster Tweet Classification (2022)

Abstract: During natural disasters, people often use social media platforms, such as Twitter, to post information about casualties and damage produced by disasters. This information can help relief authorities gain situational awareness in nearly real time, and enable them to quickly distribute resources where most needed. However, annotating data for this purpose can be burdensome, subjective and expensive. In this paper, we investigate how to leverage the copious amounts of unlabeled data generated on social media by disaster eyewitnesses and affected individuals during disaster events. To this end, we propose a semi-supervised learning approach to improve the performance of neural models on several multimodal disaster tweet classification tasks. Our approach shows significant improvements, obtaining up to 7.7% improvements in F-1 in low-data regimes and 1.9% when using the entire training data. We make our code and data publicly available at https://github.com/iustinsirbu13/multimodal-ssl-for-disaster-tweet-classification .

## introduction the upswing of text and image sharing on social media platforms, such as twitter, during mass emergency situations has led to numerous opportunities to gain timely access to valuable information that can help disaster relief authorities act quicker and more efficiently. specifically, as a disaster unfolds, information shared on social media can provide insights into the infrastructure and utility damage, casualties, and missing people. recent studies have focused on collecting and manually annotating disaster data with respect to such situational awareness categories, followed by training machine learning classifiers to automatically identify situational awareness information, useful for relief operations @xcite . however, disaster events produce large amounts of user-generated data, of which only a small frac-tion can be annotated, due to the time-sensitive nature of the problem, together with high annotation costs, and also inherent subjectivity associated with annotating data (e.g., tweets). to address this limitation, we propose a semisupervised multimodal approach that can leverage the copious amounts of unlabeled data to improve the performance on various multimodal tasks. specifically, we extend the fixmatch @xcite algorithm proposed for semi-supervised image classification to a multimodal setting. to account for subjective annotations and potentially overlapping labels, we use soft pseudo-labels instead of the original hard pseudo-labels. we apply the adapted fixmatch to the crisismmd labeled dataset and tasks @xcite , to improve the performance of supervised baselines through the use of unlabeled data. we use 122k unlabeled tweets, containing both text and images, collected automatically using text queries about disasters that occurred during the year of 2017. experimental results show that our proposed approach produces performance improvements on all three crisismmd tasks in various data regimes. notably, we obtain as much as 7.7% using as few as 250 examples per class and 1.9% improvement when using the entire data. to our knowledge, we are the first to propose a semi-supervised method for multimodal data using fixmatch and text-based searches for collecting a large unsupervised dataset. while our experiments focus on disaster tweets, the method can be easily generalized. finally, we provide an extensive error analysis of our models. we analyze how the supervised model's predictions change with the introduction of unlabeled data and reinforce the importance of our improved version of fixmatch. our contributions are as follows: ## limitations while our approach provides significant improvements on all crisismmd tasks, we also have to acknowledge the limitations of the proposed method. as it generally is the case with semi-supervised approaches, the training time is significantly increased, as more data needs to be passed through the model until convergence, comparing to a supervised approach. regarding our method of collecting unlabeled data by searching for relevant keywords, although it is generic and could be applied to datasets from other domains, it is limited for datasets containing tweets. for other types of datasets, obtaining a relevant unlabeled corpus in the same manner could be more challenging. ## conclusion we extended fixmatch to multimodal data and proposed two improvements. we applied the improved fixmatch on three disaster-centric multimodal tweet classification tasks, and showed that the approach can leverage large unlabeled data to improve supervised model performance. our semisupervised approach is general enough and can be easily applied to other datasets, being at the same time very efficient as it does not add any inference complexity to the base model.