Analyzing Modality Robustness in Multimodal Sentiment Analysis (2022)

Abstract: Building robust multimodal models are crucial for achieving reliable deployment in the wild. Despite its importance, less attention has been paid to identifying and improving the robustness of Multimodal Sentiment Analysis (MSA) models. In this work, we hope to address that by (i) Proposing simple diagnostic checks for modality robustness in a trained multimodal model. Using these checks, we find MSA models to be highly sensitive to a single modality, which creates issues in their robustness; (ii) We analyze well-known robust training strategies to alleviate the issues. Critically, we observe that robustness can be achieved without compromising on the original performance. We hope our extensive study–performed across five models and two benchmark datasets–and proposed procedures would make robustness an integral component in MSA research. Our diagnostic checks and robust training solutions are simple to implement and available at https://github.com/declare-lab/MSA-Robustness

## introduction multimodal sentiment analysis (msa) is a burgeoning field of research that has seen accelerated developments in recent years. numerous models have been proposed that utilize multiple modalities such as audio, visual, and language signals to predict sentiments, emotions, and other forms of affect. while progress in msa has been driven mainly by improvements in multimodal performance, we call for attention towards an equally important aspect ing set of cmu-mosi @xcite on a trained misa @xcite . robustness is crucial when models are deployed in the wild, where it is common to encounter inadvertent errors in the source modalities due to data loss, data corruption, jitter, privacy issues, amongst others. a well-known fact in the msa research is that language modality tends to be the most effective, which has prompted models to utilize language as its core modality @xcite @xcite . in this work, we focus on skewed dependence on language and try to understand how it affects the robustness of msa models. specifically, we ask, rq1: are models in msa over-reliant on a subset of modalities, particularly language? rq2: if yes, what implications does it have on modality robustness? to answer rq1, we look at fig. 1 . the figure illustrates a setup where we fully remove one modality during testing on the misa model @xcite . here, we observe a sharp drop in performance when language modality is removed but do not see statistically significant drops when audio or visual modalities are removed. this observation aligns with recent findings in the msa literature highlighting the dominance of language. this brings us to rq2 where we try to understand the robustness implications over this dominance. we design an elaborate study in § 3over five state-of-the-art (sota) msa models and across two benchmark datasets-where we propose diagnostic checks to understand modality robustness, i.e., how robust are models against modality errors such as missing or noisy modalities. based on our findings, we then proceed to ask, rq3: how can we improve the robustness of these models? rq4: does robust training lead to a performance trade-off? for rq3, we study well-known robust training methods, that act as a pre-emptive strategy to reduce the performance drops. critically, our training is model-agnostic and can be easily included in any existing multimodal model ( § 4). for rq4, we observe that our method to improve robustness does not trade-off with the final performance on the clean testing set, thus achieving similar performance as the original model. ## related works while msa has received increased attention in recent times, the topic of robustness has not taken center stage. fortunately, few works have started changing this trend. @xcite ) reveals how multimodal classifiers often utilize a subset of modalities, which they addressed by inducing uniform contribution from all input modalities. in msa, multiple works over-rely on language modality to improve the performance. @xcite constructs a text-centered shared-private framework for multimodal fusion, and @xcite obtains two text-related modal pairs and iteratively push the interaction between modalities to supplement information for better performance. while this has enabled performance boosts, our goal is to explore the double-edged nature of this feature and how it impacts robustness. our motivation for diagnostics is similar to @xcite , but unlike them, we do not perturb the raw data (such as image patches). instead, we intervene on modality representations, which is easier to integrate with existing models and do not require prior knowledge of the modality structure. to address robustness in msa, @xcite proposes a factored model that can accommodate modality drops. also, @xcite introduces modality drops during training and testing and uses meta-learning to make models robust. however, our work comprises some crucial distinctions: i) unlike these works, our diagnostics and robust training do not require sophisticated architecture and can be easily integrated into existing models. ii) we perform an exhaustive analysis of robustness across multiple models, which is previously not done in the msa literature. ## testing robustness via diagnostic checks in this section we perform an elaborate study on modality robustness by simulating potential issues with modality signals during testing (or deployment) of msa models. ## robust training in this section, we explore how to reduce the sensitivity of the models to the dominant modality, i.e., language. one of the popular ways to alleviate such issues is to teach the model such scenarios during training. we dub this approach as modalityperturbation, which is conceptually similar to removing or masking modalities in @xcite or adding noise in @xcite . it simulates the modality errors during training so that the model learns to expect such events during testing/deployment. the procedure is as follows, 1. training: this simple approach can be interpreted as regularization akin to dropouts or noising strategies used in de-noising auto-encoders. ## conclusion in this work, we performed a systematic study that demonstrate the double-edged nature of dominant modality in sota msa models. our analysis using diagnostic checks reveal high susceptibility to performance drops when presented with unwanted errors in their representations. to alleviate the issues, we also study robust training methods that uses modality perturbations. critically, we find that robustness and performance can co-exist without an explicit trade-off. these improvements demonstrate a positive nudge in the effort to achieve robustness and we believe there remains significant room for improvement. with this work, by proposing simple and easy-to-integrate diagnostic checks and training methods, we hope to permeate discussions on robustness into mainstream msa research.