Using Multimodal Models for Informative Classification of Ambiguous Tweets in Crisis Response (2025)

Abstract: Social media platforms like X provide real-time information during crises but often include noisy, ambiguous data, complicating analysis. This study examines the effectiveness of multimodal models, particularly a cross-attention-based approach, in classifying tweets about the California wildfires as “informative” or “uninformative,” leveraging both text and image modalities. Using a dataset containing both ambiguous and unambiguous tweets, models were evaluated for their ability to handle real-world noisy data. Results show that the multimodal model outperforms unimodal counterparts, especially for ambiguous tweets, demonstrating its resilience and ability to integrate complementary modalities. These findings highlight the potential of multimodal approaches to enhance humanitarian response efforts by reducing information overload.

## introduction advancements in image and text analysis have unlocked the potential to combine these two modes of information for use in data science and analytics. one prominent application of multimodal information is in social media, where content is no longer limited to a single modality but often integrates audio, video, image, and text. this multimodal nature of social media has opened new avenues for data analysis, enabling deeper insights and richer interpretations while requiring new methodological approaches to facilitate multimodality. this project leverages social media content, specifically tweets from x, to extract information related to crises @xcite . since its advent, social media has served as a vital communication channel, allowing individuals on the ground to share real-time updates about ongoing events such as during the 2011 east japan earthquake and tsunami @xcite . this study focuses on tweets about the california wildfires in 2017, aiming to classify them as either "informative" or "not informative." such classifications can aid humanitarian efforts by providing timely, relevant information while filtering out noise, ultimately reducing information overload and enhancing situational awareness @xcite . however, social media content presents significant challenges as a reliable information source (e.g. @xcite . posts are typically unverified, and the noisy nature of multimodal data complicates analysis. for instance, a tweet's text might convey crucial information about a crisis, while its accompanying image may not align, or vice versa. an example of such misalignment is shown in figure 1 , where a tweet contains both text and image labels that do not match. even for human observers, assessing whether a tweet is informative often requires careful consideration of both modalities, introducing ambiguity and inconsistency. previous studies on multimodal classification for crisis datasets have often relied on pre-processed and cleaned data, where text and image labels are aligned (e.g. @xcite . this project, however, examines the performance of multimodal models on both ambiguous and unambiguous tweet data to address this gap. given that multimodal models are typically based on deep neural network architectures capable of learning from complex and noisy data @xcite , this study hypothesizes that such models can effectively classify tweets as "informative" or "not informative," regardless of the ambiguity in the data. ## background there are many aspects about multi-modal models that can be studied; how the multiple modes get fused together and how each mode of information is derived to be fused together. there are usually three key methods of designing multi-modal models: early fusion, hybrid fusion, or late fusion @xcite . ## data this study utilizes the crisismmd dataset, a multimodal twitter corpus comprising thousands of manually annotated tweets and images collected during seven major natural disasters, including earthquakes, hurricanes, wildfires, and floods that occurred globally in 2017 @xcite . the dataset offers three types of annotations: informative versus non-informative, humanitarian categories, and damage severity categories, providing a rich resource for analyzing crisis-related social media data. for this study, the focus is scoped down specifically to tweets related to the californian wildfires. while the dataset provides valuable insights, a key limitation is that the labels for text and images are collected separately. as such, a key problem with creating multimodal dataset from the collected data is that some rows have text and image labels that do not align. to address this ambiguity and ensure consistency, the multimodal data is filtered to include only instances where the text and image labels align. this filtering step mitigates potential noise and ambiguity and ensures the reliability of the dataset for training and evaluating multimodal models. the correlation graph in figure 2 shows substantial alignment between text and image annotations classified as "informative," with 923 instances of agreement. however, discrepancies are observed in 322 cases where the text is "informative" but the image is "not informative," and 62 cases where the image is "informative" but the text is "not informative." these misalignments highlight the complexity of social media posts, where texts and images' labels might not align, regardless of their overall informativeness. out of the 1,589 rows of data, 384 rows were identified as ambiguous, where the text and image labels for the tweets did not align. a manual annotation process was conducted programmatically to reclassify these rows, determining tweets as "informative" if they have any relevant information related to the california wildfires. following this process, the dataset comprises 1,303 "informative" instances and 286 "not informative" instances, revealing a notable class imbalance that could impact the model's performance. ## methodology this study aims to create a multimodal classification model to classify tweets, whether ambiguous or not, into two categories: "informative" and "not informative." the model's performance is evaluated against the entire dataset (1,589 tweets) and the subset of ambiguous data (384 tweets), testing its effectiveness in handling both aligned and misaligned text-image annotations. the data for this pilot study was limited in order to enable reliable error analysis and qualitative interpretations. ## discussion the study has once again reaffirmed the effectiveness of using multimodal models for social media analysis, especially for the classification of tweets during crisis. the superior performance of multimodal models can be attributed to their ability to leverage both text and image information, similar to how humans make more informed decisions when provided with additional context. a notable finding is the ability of multimodal models to classify ambiguous tweets more effectively than unimodal models. the incorporation of a cross-attention mechanism enables these models to focus on the most relevant features from both modalities, reducing the impact of noise often present in ambiguous data. compared to using unimodal models which inherently filters away one source of information and completely relying on one mode, multimodal models can make fairer and more informed decisions about whether a tweet is "informative" or "not informative". our results highlight the relative contributions of individual modalities in determining informativeness. for the ambiguous dataset, the text model achieved a slightly higher f1-score (0.86) compared to the image model (0.85). this suggests that, in ambiguous cases, the text modality often carries more critical information than the image modality, allowing the text-based model to perform marginally better. this finding underscores the importance of text in providing context, which can be particularly useful in determining ambiguity. multimodal models have proven to be effective in classifying tweets as "informative" or "uninformative" for humanitarian purposes, demonstrating their potential to enhance crisis response efforts. this study emphasizes that while noisy data, characterized by ambiguity between text and image modalities, poses challenges, it should not be dismissed. in our results, multimodal models have shown resilience in handling such ambiguity, leveraging complementary information from both modalities to make accurate predictions. the focus on the california wildfires dataset provided valuable insights into the applicability of multimodal models in real-world crises, as this dataset reflects the complexity of social media content during natural disasters. overall, this project underscores the importance of incorporating multimodal approaches in analyzing ambiguous social media data, especially in the case of classifying tweets in order to reduce information overload and support timely humanitarian work.