Exploring Multimodal Features and Fusion Strategies for Analyzing Disaster Tweets (2022)

Abstract: Social media platforms, such as Twitter, often provide firsthand news during the outbreak of a crisis. It is extremely essential to process these facts quickly to plan the response efforts for minimal loss. Therefore, in this paper, we present an analysis of various multimodal feature fusion techniques to analyze and classify disaster tweets into multiple crisis events via transfer learning. In our study, we utilized three image models pre-trained on ImageNet dataset and three fine-tuned language models to learn the visual and textual features of the data and combine them to make predictions. We have presented a systematic analysis of multiple intra-modal and cross-modal fusion strategies and their effect on the performance of the multimodal disaster classification system. In our experiment, we used 8,242 disaster tweets, each comprising image, and text data with five disaster event classes. The results show that the multimodal with transformer-attention mechanism and factorized bilinear pooling (FBP) for intra-modal and cross-modal feature fusion respectively achieved the best performance.

## introduction the sudden breakout of crisis events, like natural disasters, creates high-stakes circumstances that are coupled with great uncertainty as well as the need to make quick decisions, often with limited official newscasts. research in recent years has uncovered the importance of social media communication in disaster situations and shown that information broadcast via social media can improve situational awareness during an emergency @xcite . social media has proven to be an active communication channel, especially during crisis events such as natural disasters including earthquakes, floods, and typhoons @xcite , @xcite ) or other emergencies such as accidents. these events spur a sudden surge of attention followed by reactive actions from both the general public and the media. the quick detection and analysis of such events are critical to swiftly disseminate information and, more importantly, prepare the relief team. such situational awareness and tactical information enables the team effectively estimate early damage and launch relief efforts accordingly. an automated system for crisis-related information retrieval from social media is imperative to rapidly and systematically classify disasters. information regarding crises is best sourced from the social media site twitter, which is a real-time, open, and public communication platform. the development of a system requires the extraction of relevant tweets to then classify them into different types of information: affected individuals, infrastructural damages, casualties, donations, caution, or advice. however, because the messages generated during a disaster vary greatly in value and since twitter is a highly diverse platform, an automatic system needs to filter out messages that are irrelevant and do not contribute to situational awareness. as a result, we designed a system for detecting informative messages that classifies them to decide the type of information to extract (e.g., donation offers, casualty reports). information on social media mainly consists of textual messages and images. past research has mainly focused on using textual content to aid disaster response. however, recent studies have revealed that images shared on social media during a disaster event can also help the relief team in several ways. for example, @xcite incorporated images shared on twitter to assess the severity of infrastructure damage in their work. similarly, @xcite investigated the usefulness of images and text for their study on flood and flood aid. our work follows this method of taking into account both texts and images. previous works @xcite , @xcite , @xcite , @xcite have proposed a multimodal system for analyzing disaster tweets that utilizes feature fusion. however, not much exploration has been done for the enhancement of the extracted visual, textual and their combined multimodal feature representation. in this paper, we present an analysis of various multimodal fusion strategies for intra-modal fusion and cross-modal fusion. we investigate relation-attention, self-attention, and transformer-attention for intra-modal fusion. for the cross-modal fusion, we explore three methods, namely, factorized bilinear pooling @xcite , compact bilinear gated pooling @xcite and compact bilinear pooling @xcite . along with this, we evaluate stateof-the-art models which were three pretrained image models (vgg19 @xcite , resnet-50 @xcite and alexnet) and three pretrained language models (bert @xcite , roberta @xcite and albert @xcite ) for the disaster tweet analysis and classification task. in our analysis, we utilize multimodal crisismmd @xcite dataset. we found that the resnet-50 outperformed other image models and among the textual models roberta achieved the best performance. we further utilize these two models for the evaluation of intra-modal and cross-modal fusion strategies. 2 methodology 2.0.1 textual feature extractor: we employed three pretrained language models, namely, bert-base @xcite , roberta-base @xcite and albertbase @xcite to extract a high quality text feature vector. we finetuned them with a custom classification head with updatable weights. the averaged pool of sequential output from 12 encoding layers of each model was used as the custom classifier head's input. once the model was finetuned, each of the language models was fed with a sequence of text inputs (reprocessed tweets) which then went through all of the stacked encoding layers, thereby extracting essential features from the context. ## dataset we have used the crisismmd @xcite dataset for training and testing our model. each text and image pair in the dataset have two annotations: (task_1) humanitarian categories (eight classes), (task_2) informative vs. not-informative (two classes). since the number of labels across different classes was uneven, following @xcite , we compressed the number of humanitarian categories to five-namely, (i) not-humanitarian (4312), (ii) other_relevant_information (1764), (iii) res-cue_volunteering_or_donation_effort (1195), (iv) infrastructure_and_utility_damage (842) and (v) affected_individuals (129). in the crisismmd dataset, tweet text and image in a pair were annotated separately, as a result, few pairs had a different label for text and it's associated image. we removed those pairs and performed the experiment only those data who have the same label for text and image. finally, we have 8,242 pairs and split the data in 70%:15%:15% ratio for training (5770), development (1236), and test (1236) sets. for the informative and not-informative, we had 7875 (train), 1687 (development) and 1688 (test). ## results in this section, we discuss and analyze the multimodal performance with various fusion techniques. we have shown the result of the best cross-model fusion method: fbp applied with various intramodel fusion methods. for task_2, we observed that by using the fbp @xcite and transformer attention layer in the pipeline, the performance of multimodal was remarkably better (around 12%) than the other cross layer fusion methods (cbp and cbgp). we also noticed that in either of the cross-modal fusion method, the transformer attention intra-modal fusion performed the best. for task_1 (refer 3) and task_2 (refer 4), fbp with transformers-attention based multimodal model gave the best result of 85.1% and 89.5% respectively. we can also see that models having transformer-attention combined with relation-attention outperformed the model with transformer-attention and self-attention. coming to the multimodal baseline @xcite and @xcite , our model outperform it by 7.99% and 1.10% on the task_1 and for task_2 it is 5.92% and 0.78%. the reason behind the superior performance of our model lies behind the underlying feature representation generated by the pre-trained language and image models. moreover, we were able to capture intra-modality information using attention mechanism which produced a denser feature representation before the cross-modal fusion. therefore using transfer learning and attention-based fusion techniques, we were able to blend together with powerful language and image models and build a more robust multimodal. ## conclusion in this paper, we present an extensive analysis of multiple feature fusion strategies for developing a multi-modal framework for detecting and classifying tweets into various crisis events accurately based on the textual and visual features. in our study, we compared various image and language models and found that the resnet and roberta outperformed the other models. we also presented a comparative study of various fusion methods; through that, we can conclude that the selection of effective intra-modal and cross-modal method plays a crucial role in developing a more accurate and efficient multimodal framework for classifying the events for faster relief efforts. we observed that the transformer-attention mechanism outperformed the other intra-modal fusion methods. we also showed that by using factorized bilinear pooling, the multimodal feature representation can be improved. the results of the experiments show that one application of the multimodal framework can be the identification and filtration of disasterrelated information available on social media platforms.