Natural Disaster Tweets Classification Using Multimodal Data (2023)

Abstract: Social media platforms are extensively used for expressing opinions or conveying information. The information available on such platforms can be used for various humanitarian and disaster-related tasks as distributing messages in different formats through social media is quick and easy. Often this useful information during disaster events goes to waste as efficient systems don’t exist which can turn these unstructured data into meaningful format which can ultimately assist aid agencies. In disaster identification and assessment, information available is naturally multimodal, however, most existing work has been solely focused on single modalities e.g. images or texts separately. When information from different modalities are integrated , it produces significantly better results. In this paper, we have explored different models which can lead to the development of a system that deals with multimodal datasets and can perform sequential hierarchical classification. Specifically, we aim to find the damage and its severity along with classifying the data into humanitarian categories. The different stages in the hierarchical classification have had their respective models selected by researching with many different modality specific models and approaches of multimodal classification including multi task learning. The hierarchical model can give results at different abstraction levels according to the use cases. Through extensive quantitative and qualitative analysis, we show how our system is effective in classifying the multimodal tweets along with an excellent computational efficiency and assessment performance. With the help of our approach, we aim to support disaster management through identification of situations involving humanitarian tragedies and aid in assessing the severity and type of damage.

## introduction natural disasters and calamities have long been occurring. the gravity of these disasters is depen-dent upon a number of factors, these could be lives lost, damage caused, economic loss, etc. each year, these disasters take place in the form of earthquakes, floods, tornadoes and severe storms, hurricanes and tropical storms, droughts and wildfires and cause plenty of damage to the human society. the impact due to these disasters cannot be neglected and thus first-responders such as the local residents, health professionals and emergency workers must come to the aid of the damaged sites and affected individuals. the immediate priority that follows after a disaster occurs is providing emergency aid to the injured people. however, it is also very important to assess the damage that has been caused to different structures and zones as that is a good indication of the severity of the situation. in times of disaster or a calamity, the information available on social media could be widely used in disaster assessment, management and analysis and for other humanitarian tasks. the data regarding these disasters that are collected using field surveys are often not available immediately. disaster damage data such as the location, area or the zone and extent of the damaged structures and facilities is critical in disaster management operations @xcite . therefore, it becomes necessary for the first-responders of humanitarian aid to rely on data derived from social media, as many users post messages and information in different formats, i.e image, text, audio, etc. however, the utilisation of this data is not easy. one of the challenges that needs to be tackled before the data can be utilised appropriately is related with isolation of content useful for crisis management. in times of a crisis, it is extremely important to determine which areas have been affected the most (hence require the most attention and should be an immediate priority) and other humanitarian tasks that include missing/found people, injured or dead people and other sub-tasks. however, this becomes difficult to identify as during these times, there will be a huge volume of data that could mask the severity level of certain areas and people in need. therefore, there is a need for automatic systems to assess, identify and analyse the damage that has been caused which in turn could benefit the emergency management process notably @xcite . although the extraction of information from social media resources to help humanitarian aid workers has been going on for some time, this has been limited to unimodal data. while these unimodal damage analysis frameworks are efficient, they are not able to assess damage as effectively for social media posts @xcite ) that come in a multimodal format, whether it be text with images, text with audios, audios with images, etc. thus, we intend to develop an effective system that can leverage multimodal data and is able to assess, identify and analyse the severity of the damage caused and identify other humanitarian sub-tasks in real time. the unimodal text feature is often not very helpful in disaster situations as many times the text of such tweets is incomplete or incomprehensible and the main idea is often conveyed by the images, leading us to choose the text and image modalities, which can give more information about an event @xcite . the main reason for using multimodal data sources is that it is possible to extract complementary and richer information coming from multiple sensors, which can provide much more optimistic results than a single input. some monomodal learning systems have significantly increased their robustness and accuracy, but in many use cases, there are shortcomings in terms of the universality of different feature levels and inaccuracies due to noise and missing concepts @xcite . even though the approach is beneficial, in practice it is very challenging @xcite due to the different noise and conflicts in the different modality data @xcite . therefore one has to find a model which can balance the same. we explore different models on both the modalities and select the one which best balances all the noise present in the different modalities. further the design of the system should be such that it is responsive, fast and efficient in its computation as the response is needed quickly during a disaster. the system should also have scope for addition and deletion of different categories which is only possible if the architecture is a modular one and as a result we choose the hierarchical structure. the system should also be able to handle unimodal tweets (text only). we leverage the recent advances in classification for images and text to build an efficacious system which can give much fine grained results. ## related work in the context of unimodal classification, @xcite classified textual tweets from crisismmd @xcite dataset into informative and non-informative categories using cnn and ann. @xcite develop an automatic data processing service which takes in textual data from various sources and classify it into disaster type, informative and humanitarian information conveyed. for the purpose of classification, deep learning and classical algorithms are used. while these unimodal approaches produce commendable results, they have been surpassed by the multimodal approaches. @xcite propose a data-driven method to locate and assess disaster damage with humongous multimodal social media data. the images have been classified using machine learning while the text follows a keyword search based method. @xcite analyse the multimodal data related to different natural disasters, for instance: floods, earthquakes, etc. they propose a novel decision diffusion technique to classify them into informative or non-informative categories. their approach of training an image and text classifier and combining the two outperforms the baselines. @xcite proposed a method through which they integrated image and text information to identify disaster images collected from different social media platforms. they use a deep learning method and fasttext framework to extract visual and textual features respectively. they then develop a data fusion model to combine these features and experiment on real world disaster dataset through the crisismmd dataset. @xcite present a new multimodal fusion method that uses both images and texts as input. they introduce a cross-attention module that is able to filter out uninformative and misleading components. @xcite present a damage identification and its severity detection system, called crisis-dias. through qualitative, quantitative and theoretical analysis on a realworld multimodal dataset, they are able to show that the information once presented together, often produces high-end analysis about the domain and facilitates better learning performance. @xcite propose a multimodal deep learning framework to identify damage related information. in their approach, they combine multiple pretrained unimodal cnns that extract features from the raw texts and images respectively. finally, a resultant classifier labels the posts based on both modalities. their results on a home-grown database of labelled social media posts shows good results and validates the use of the proposed method. @xcite experiment with combination of semantic textual features with the image features to be able to efficiently classify a relevant multimodal social media post. they utilise a feature generate framework which uses tf-idf vectors and glove embeddings for the textual data, and pre-trained residual networks for generating vectors for the image data, the vectors extracted are then concatenated through which they build a lightgbm model to classify different disaster-events as informative or not. their results demonstrate that features based on a hybrid framework (using both text and images) improve the performance of identifying relevant posts. ## dataset and problem definition the dataset that was used for this project was the crisismmd foot_0 dataset. it's labels were aggregated according to the specific task at hand. the dataset included information that could be filtered extensively depending on the scenario. the crisismmd dataset consists of thousands of annotated tweets and images collected during seven various natural disasters including earthquakes, hurricanes, wildfires, and floods that happened in the year 2017 across different parts of the world. the data is hierarchical as the class labels at each stage depend on the annotation in the previous stage. the above data is further complemented by a text only dataset called humaid foot_1 for certain downstream tasks. we use the several sub-categories of the classes to produce our desired results which is to be able to identify the structure damaged and it's severity along with humanitarian mishaps. to do this, based on the dataset we formulate the following tasks. ## experimental setup we experiment with various state of the art text embeddings and image embeddings utilising newer models like image transformers in addition to already successful cnn based models. the models have been fine tuned and while some of them have had partial training to fit our use-case. the models are run for a maximum of 100 epochs but with early stopping and a set patience value. ## results for task 1, we have selected bert @xcite for the text pipeline as it outperforms all the others. for image we can clearly see that xception @xcite performs better than all the models followed by resnet @xcite . but despite the higher results we will not choose xception (77%) as resnet (80%) has a higher recall for the informative class and our main aim is to reduce the false negatives (we don't want the informative tweets to be labelled as non informative). upon performing an or operation on the results of the selected image and text pipelines we see that the accuracy after using data from both modalities has increased to 86%. which is higher than any of the accuracy of the single modalities. also the recall of the informative class has also increased to 94%, again more than both the unimodal recalls. albert @xcite has the higher values for all the metrics and since it is light weight we will be choosing that for our text classification in task 2. it also has good recall for the classes that are considered the positive ones. for image, we see that the deit @xcite has obtained higher recall and precision than all the other models. as a result we also see how the vision transformers are better at image classification task. we see that after combination of the prediction of modalities, the accuracy has not increased but the quality of the predictions has. the structure class had pretty low metrics for albert (recall: 72%) and the deit model had low recall for the humanitarian class (72%), but upon combination there is a balancing effect and the results of all the 3 classes have had a boost. therefore it is somewhat of an ensemble effect. the recall of the damage class has increased as a result of adding the image model and the humanitarian class has seen an increase in the recall due to the addition of the text model. after testing the 2 approaches in task 3, we concur that multitask deit-albert has better metrics. while naturally the single output models should have had better results, the deviation is due to the noise which could be present in a certain target output in the training samples as a result the model could have been biased towards something. where as in the multimodal the model has to concurrently optimise the loss function of the two outputs as a result it can cancel the noise which was present in one target specific sample and absent in the other. for task 4, the performance of inception and deit is almost comparable but the deit has a higher computation cost, for this specific task the deit took 8 times the time taken by inception to do the same task. since time is of the essence in our application, we have chosen the inception model (szegedy et al., 2016) for this task. ## final system discussion based on our experiments we have selected the components as discussed above and the detailed final system is shown in figure 1 . design: the embeddings from bert used for text in the task 1 are a summation of the last four layers, which is then input to gru layer. the resnet 152 model in the image pipeline has all its layers unfrozen except the batch normalisation layers, which are kept frozen so that the learnings of the imagenet dataset are preserved. the informative tweets progress to task 2 and the others are dropped at this stage. at stage 2, the text feature vector is obtained by concatenating the output of the last 3 layers of the albert model, which is then fed to the lstm layer. the deit used for image is a pretrained model which is used to encode the images and then classified using an ann of size 512 neurons. the outputs are fused using the f human-struct . the structure/damage related tweets reach the task 3. in task 3 we use a multi task model where the text features are obtained from albert and deit is used for the images. the reason for taking only the image projections for structure identification is that the structure is usually conveyed from the image and inclusion of text will not help in most samples. the weightage given to the text and image embeddings are same and both contribute equally in the prediction of the damage severity. in task 4, the text is fed to the bert encoder and image to inception, the output of these 2 encoders are concatenated to form the feature vector. the models used in task 4 are pretrained. performance: to analyse the effectiveness of the system, we calculate the f1-score and accuracy of the system as the average of the 4 tasks, the f1-score and accuracy comes out to be 83% and 88% respectively, which is quite impressive. on top of having an accurate classification, the system also has to be responsive and provide quick predictions as it will be used in disaster sites which are time critical. the system is tested on a gpu as well as an older machine on a cpu to see how well it performs in constrained environments. to estimate the time that our model takes in inference we run around 100 tweets and calculate the time a tweet takes to be processed per task. the inference was run separately on a intel core i7 6th generation cpu with 2.6ghz speed (an old cpu) and nvidia tesla p100 (gpu). we find on average a tweet takes 400 milliseconds on the gpu and 1.32 seconds on the cpu over the entire process. the model boasts an impressive speed on a gpu as well as decent processing speeds in the absence of complex architectures as evident from the old cpu's processing time, thus showcasing the resilience of the system. features: even if non informative tweets enter a stage, due to the hierarchical nature they can be removed as every stage involves a non-informative class. the data can be viewed at any granularity level by only allowing tweets to go to a particular stage. further more if newer and more accurate models are present for the tasks at any stage they can be easily incorporated in the system owing to its modular design, giving it a plug and play feature. the variety of fine grained classes in the system allow the rescue workers to formulate a priority list based on a combination of classes and respond to the emergencies in a swift manner. along with having a great performance for multimodal tweets, the system can generalise well to unimodal tweets as well, as the first 2 tasks are a simple fusion and the absence of a modality doesn't interfere with the others' classification. for the last 2 phases, the vector of the missing modality can be left blank, as a result we can adapt to a variety of tweets without considerable decrease in quality. comparison: while comparing with similar work in this field, there doesnt exist exact experiments on the classes or the sub divisions which we divide our data into. though there does exist papers which deal with a subset of our tasks and we can do somewhat of a task by task comparision to evaluate our system with existing state of the art. it can be observed from the results present in qualitative analysis:we perform a qualitative analysis of a few images to show how the model predictions work. this also justifies the design choices which we have made in terms of the model selection. figure 2 : tweet text: california wildfire evacuees just want to "go home" if they have a home still standing https://t.co/woxkyydhxt https://t.co/h5rqdtvshw actual tag: task 1: informative; task 2: human; task 3: affected individual predicted: task 1: text: informative -image: informative; task 2: text: non-informative -image: human; task 3: affected individual what we see in figure 2 is that the classifier predicts all the labels for the image accurately. even though the text was non informative in the second task, the overall sample is classified as human class due to the f human-struct operation of the predictions of the individual modalities. it is observed in figure 3 that the model correctly predicts the tag and the benefit of the 2 modalities is again highlighted when the text and image give their different predictions in task 2 but we have been able to preserve the important content present in the photo. ## conclusion through our research we have been able to propose a hierarchical model which classifies the given multimodal tweet into the different categories of either humanitarian or structural damage. a thorough analysis is performed on a combination of crisis-mmd, humaid datasets for the purpose of the development of the model. it is also very responsive and can be fitted with different tasks as and when data for such disaster events is available. as a future work on this task, the speed of predictions could be improved gigantically by deploying the models on cloud computing platforms with gpu support and multiple instances with auto scaling. the system could implement zero shot learning as the photos from a natural disaster are not from a closed domain and a model will always not see events similar to what it was trained on. currently the model has been tested for tweets from twitter, but similar platforms like instagram, facebook, etc. could also be used with this model, we would need to develop an adapter for the particular platform specifying rules for how the image and/or text is obtained from the platform. a future step would be to increase the diversity of the dataset by including samples from myriad social networks. we plan to expand the type of modality to include videos from platforms like youtube and the like, a precursor to which would be creation of a dataset for the modality. also the predictions could be further improved by taking user feedback on classifications and using newer vision transformers as well as decision fusion techniques for combining the modalities.