Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions (2022)

Abstract: As multimodal learning finds applications in a wide variety of high-stakes societal tasks, investigating their robustness becomes important. Existing work has focused on understanding the robustness of vision-and-language models to imperceptible variations on benchmark tasks. In this work, we investigate the robustness of multimodal classifiers to cross-modal dilutions – a plausible variation. We develop a model that, given a multimodal (image + text) input, generates additional dilution text that (a) maintains relevance and topical coherence with the image and existing text, and (b) when added to the original text, leads to misclassification of the multimodal input. Via experiments on Crisis Humanitarianism and Sentiment Detection tasks, we find that the performance of task-specific fusion-based multimodal classifiers drops by 23.3% and 22.5%, respectively, in the presence of dilutions generated by our model. Metric-based comparisons with several baselines and human evaluations indicate that our dilutions show higher relevance and topical coherence, while simultaneously being more effective at demonstrating the brittleness of the multimodal classifiers. Our work aims to highlight and encourage further research on the robustness of deep multimodal models to realistic variations, especially in human-facing societal applications.

## introduction rich multimodal content understanding is crucial for several ai for social good applications like humanitarian information detection during crises, hate speech analyses, and fake news mitigation @xcite @xcite @xcite . in many such scenarios, the information in individual modalities, either image or text, is designed to be complementary to information in the other modality. as such, joint modeling of both modalities is of fundamental importance, and consequently, seen from above. entire california communities reduced to ash. cross-modal dilution (@xmath0 figure 1 : overview of our study. we investigate the robustness of fusion-based deep multimodal classifiers to cross-modal dilutions. we generate dilutions that maintain semantic relevance with the original text and image while causing incorrect classifications. we also demonstrate the realistic nature of cross-modal dilutions using human evaluation. the figure shows an actual example from our experiments. it is desirable that deep learning models are robust to dilution-based variations in input. dilution is defined as the addition of related content that dilutes the effect of the original information. @xcite and @xcite argue that natural language processing (nlp) models should not alter their predictions after adding dilutionsfor instance, appending statements like "and true is true" (multiple times) for the natural language inference task and adding randomly created urls for the sentiment analysis task. we study the robustness of multimodal classifiers to dilutions. compared to the simple dilutions created for nlp tasks, we aim to explore realistic dilutions for multimodal data. since what entails plausible dilutions for multimodal data has not been established, we propose a new category of dilutions specific for multimodal content, named cross-modal dilutions. cross-modal dilution involves adding relevant information from the image modality to the text modality for a multimodal input; see figure 1 . our notion of dilution, unlike the examples above, is contextual -that is, the change introduced varies for different information items. additionally, evaluating robustness to dilutions in a multimodal setting is non-trivial because the possible additions are constrained by the semantics of both the image and the original text. previous research on the robustness of deep multimodal learning focuses on perturbations for visual question answering (vqa) @xcite @xcite and involves making minor alterations to the textual questions @xcite , or asking more challenging questions than what were present in the training dataset @xcite . in contrast, we focus on multimodal classification and study dilution-based variations. to this end, we propose a method that leverages a large language model to generate additional text that is (i) related to the information in the image, (ii) semantically aligned to the existing user-provided textual description, and (iii) is adversarial in nature (i.e., when added to the existing description, leads to incorrect predictions by multimodal models). the first two constraints ensure that the additional text is realistic, while the third constraint enables us to assess the robustness of multimodal classifiers under these settings. our contributions are summarized as follows: adversarial perturbations: our investigation concerns adding related text in a multimodal example to the existing textual information. several methods have been proposed to introduce imperceptible and adversarial perturbations in text @xcite @xcite , focusing on word-level or phrase-level automated insertions, replacements, and merging. moving beyond the imperceptibility constraint, to estimate robustness to perceptible but plausible changes in text, recent research has investigated the robustness of nlp models to rule-based distractions that are added to the original text @xcite . as the constraints that govern textual dilutions in a multimodal setting are different, we propose a model to generate cross-modal (@xmath1, while also demonstrating adversarial properties with respect to the multimodal classifiers. this provides us with a realistic estimate of the robustness of multimodal classifiers. ## cross-modal dilutions related work on language-only models @xcite inspires us to study the robustness of deep multimodal classifiers to dilution. in the context of multimodal learning, dilutions can be introduced by adding information from the associated image to the original text. since multimodal fusion models are expected to consider the information in images and text jointly, they should, in principle, be robust to the expression of additional information regarding the image in the form of text. this is, however, challenging to study because a plausible dilution should have semantic similarity with both the image and the original text. while a rule-based dilution like "and true is true" (investigated by @xcite ) are plausible for specific language-only tasks like natural language inference @xcite , they do not cover the action space of plausible cross-modal dilutions for multimodal content. therefore, we develop an approach to generate dilutions that are semantically aligned with original text and image. our proposed approach follows the following framework to generate dilutions; see figure 2 . (i) extract keywords from image and text based on their prominence in their respective modalities. ## multimodal datasets we conduct experiments on two user-generated datasets that have real-world societal applications. crisis humanitarianism dataset: during crises, affected parties often use social media to communicate with humanitarian organizations that process the available information to provide timely and effective interventions. to aid development of related computational methods, alam et al. (2018) curated the crisismmd dataset. this multimodal dataset comprises 7, 216 twitter posts in english (images + text) that are categorized into 5 humanitarian categories. foot_1 we formulate the task of humanitarian information detection as a multi-class classification problem, and use the standard training (@xmath2, evaluation (@xmath3, and test (@xmath4. sentiment detection dataset: user-generated content has been frequently used to infer sentiments of individuals for various applications, including detection of mental health indicators (de choudhury et al. @xcite . we collect the dataset introduced by @xcite for the task of sentiment detection. the dataset comprises multimodal posts (in english) from reddit that are categorized into 4 classes. 3 we crawled the images from reddit urls provided by the authors and split the dataset in a 80:10:10 ratio to obtain the train (@xmath5, validation (@xmath6, and test (@xmath7. ## experiments we first discuss the training of our proposed crossmodal dilutions (xmd) generator model. then, we discuss multiple baselines that dilute the original text using various rule-and model-based approaches. finally, we evaluate xmd and compare its performance with the baselines. ## main results our results (tables 1 & 2 ) show the following: ## analysis of cross-modal dilutions next, we further analyze the dilutions generated by our proposed method (xmd). we focus on the crisis humanitarianism dataset for our analyses. in addition to the analyses presented here, we investigate the effect of the length of dilutions (i.e., number of inserted words) on classification performance and observe no notable difference in observed trends with similar dilution lengths; see appendix a.10. in appendix a.11, we analyze the sensitivity of quantified metrics with respect to variations in λ. finally, we conduct a human evaluation to assess how realistic the diluted multimodal examples are when compared against real multimodal examples. subjective assessment of dilutions: figure 3 shows examples of the dilutions generated by xmd from the crisis humanitarianism dataset along with dilutions obtained from the baselines. to further assess the quality of generated dilutions, we conducted a survey on amazon mechanical turk (amt). we instructed the annotators to compare two multimodal posts -one containing dilutions from xmd and the other containing dilutions from gpt-ft for the same multimodal example. the posts were randomly ordered to mitigate position bias. annotators were asked to respond on a 5-point likert scale (1: strongly disagree, 5: strongly agree) to the following question: based on the quality of the text and its relevance with the image, is the post on the right more likely to be an actual social media post than the post on the left? we obtained 5 annotations for each of the 200 examples that were randomly sampled from the test set of the crisis humanitarianism dataset. overall, the results showed that annotators consider the dilutions generated by xmd to be more realistic than gpt-ft. the percentage of examples for which the majority of annotators preferred xmd dilutions over gpt-ft dilutions were: 32.1% (strongly) and 46.4% (moderately). for 12.2% examples, the majority of annotators were neutral and preferred gpt-ft dilutions over xmd dilutions for 9.3% examples. see appendix a.8 for more details about human evaluation, recruitment, and compensation. ablations: we aim to understand the role of two key components in xmd -the incorporation of the adversarial loss component in the training objective and the inclusion of textual keywords. for the crisis humanitarianism dataset, are cross-modal content dilutions realistic? we now focus on assessing how realistic the diluted examples are when compared to the real social media examples. to this end, we conduct an amt survey that requires users to compare multimodal examples with inserted dilutions against different but original multimodal examples. to prime the annotators, we first show them 5 unmodified multimodal examples and subsequently ask them to analyze a list of randomly-ordered multimodal examples, half diluted and the other half original unmodified examples. we use the dilutions generated by xmd. for each example in the list, the annotators are asked to respond to the following question on a 5-point likert scale: do you think this post (text and image) could be a real post from social media website? we select a subset of 100 examples from the test set of the crisis humanitarianism dataset and obtain 3 annotations for each example. the average likert score for original examples is 3.61 (± 0.53), whereas that for diluted examples is 3.38 (± 0.39). the inter-rater agreement indicated strong reliability of annotations (krippendorf's α = 0.83). an independent two-sided t-test (assuming unequal variances) resulted in a p-value of 0.24, indicating no evidence that the average likert scores of the original and diluted examples are from different distributions. these results show that the annotators asses the diluted and original examples to be similar, reinforcing the realistic nature of dilutions. ## conclusion and future work in sum, our work is the first investigation of the robustness of multimodal classifiers to cross-modal dilutions. we establish the plausibility of such dilutions via human evaluations and develop a model to emulate adversarial scenarios reliably. we find that multimodal classifiers that fuse the state-of-the-art modality-specific representations are not robust to cross-modal dilutions generated by xmd. deep classifiers are increasingly being used for crucial applications that involve the joint understanding of user-generated multimodal data. our broader goal in this work is to analyze and advocate for the robustness of multimodal models with societal applications, while focusing on the most representative fusion-based multimodal classification technique. in the future, we intend to leverage the knowledge of vulnerabilities identified in the current work to develop more robust multimodal models. we encourage interested researchers to investigate other cross-modal variations pertinent to multimodal data and assess the robustness of multimodal learning approaches to these variations.