Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions (2022)

Abstract: Multimodal fusion addresses the problem of analyzing spoken words in the multimodal context, including visual expressions and prosodic cues. Even when multimodal models lead to performance improvements, it is often unclear whether bimodal and trimodal interactions are learned or whether modalities are processed independently of each other. We propose Multimodal Residual Optimization (MRO) to separate unimodal, bimodal, and trimodal interactions in a multimodal model. This improves interpretability as the multimodal interaction can be quantified. Inspired by Occam’s razor, the main intuition of MRO is that (simpler) unimodal contributions should be learned before learning (more complex) bimodal and trimodal interactions. For example, bimodal predictions should learn to correct the mistakes (residuals) of unimodal predictions, thereby letting the bimodal predictions focus on the remaining bimodal interactions. Empirically, we observe that MRO successfully separates unimodal, bimodal, and trimodal interactions while not degrading predictive performance. We complement our empirical results with a human perception study and observe that MRO learns multimodal interactions that align with human judgments.

## introduction multimodal fusion integrates information from what we say, how we speak, and how we visually express ourselves. while multimodal models have led to performance improvements @xcite @xcite , they often have the downside of being difficult to interpret: it is unclear whether interactions between two modalities (bimodal) or three modalities (trimodal) are learned, whether modalities are processed independently of each other, or whether these models focus on only one modality @xcite . negative "oh, my god. that's so dramatic." positive very positive l, v (unimodal) l + v (additive) f(l, v) (bimodal) neutral + vision (v) language (l) ## related work we review previous research on four aspects related to multimodal interactions: the prevalence of additive interactions, model-specific and modelagnostic quantification of modality interactions, and taxonomies of multimodal interactions. ## quantifying multimodal interactions to learn a multimodal model that separates unimodal, bimodal, and trimodal interactions, we first define how to quantify these three types of multimodal interactions. further, the work presented in this section extends prior work @xcite , which defined evaluation metrics to quantify multimodal interactions in the bimodal case. consider three modalities t (text), v (vision), and a (acoustic) with corresponding features x t , x v , x a . a bimodal function f is additive when it can be factorized into the sum of two unimodal functions, ∀x t , further, f contains unimodal contributions when parts of the prediction depend on only one modality: ∃x t : ev f (x t , v) ̸ = 0 @xcite . this equation is illustrated for the language modality but has the same formulation for the vision modality. prior work @xcite proposed emap to quantify unimodal contributions (u c) in the context of two modalities. in this paper, we generalize u c to three modalities. the idea of u c is to evaluate the model with all possible combinations of unimodal features (even feature combinations that are not in a dataset) so that the model cannot use non-additive interactions between modalities. similarly, we can formulate a function bi to quantify bimodal interactions. the remaining trimodal interactions (t i) are then simply what is not covered by the unimodal contributions and bimodal interactions: when computational feasible foot_1 , u c, bi and t i are valuable tools to evaluate whether a trimodal model contains unimodal, bimodal, and trimodal interactions. we will use these metrics to evaluate our proposed approaches. ## multimodal residual optimization the main contribution of this paper is multimodal residual optimization (mro) which has the goal of learning and decomposing predictions into unimodal, bimodal and trimodal interactions to quantify them. inspired by occam's razor, the intuition of mro is that (simpler) unimodal interactions should be prioritized before learning (more complex) bimodal and trimodal interactions. mro has two components to separate modality interactions: an architecture and loss-function component. ## experimental methodology we evaluate whether we can train a model that separates unimodal, bimodal, and trimodal interactions while not degrading predictive performance. datasets: we focus on five sentiment-and emotion-annotated datasets for which prior work used multimodal models, see we use the same features across all sentiment and emotion datasets: roberta @xcite as a representation of transcribed utterances; openface 2.0 @xcite to summarize face-related features, and opensmile's egemaps @xcite to summarize acoustic features. for the instagram dataset, we use the author-provided resnet features @xcite to summarize the image content and use roberta to represent captions. evaluation: we want that the prediction components ŷuni , ŷbi and ŷtri correspond to u c(ŷ), bi(ŷ), and t i(ŷ) so that the prediction components represent only unimodal, only bimodal, and only trimodal interactions. to test this, we use |u c(ŷ bi + ŷtri )| to evaluate whether the bimodal and trimodal predictions contain unimodal contributions and |bi(ŷ tri )| whether the trimodal prediction contains bimodal contributions. given the mro-architecture, ŷuni cannot include bimodal and trimodal interactions and ŷbi cannot include trimodal interactions. this means, if |u c(ŷ bi + ŷtri )| + |bi(ŷ tri )| is 0, the model perfectly separates unimodal, bimodal, and trimodal interactions, i.e., ŷuni = u c(ŷ), ŷbi = bi(ŷ), and ŷtri = t i(ŷ). we use 5-fold test setup for all datasets. models: we compare the mro-architecture when optimized in different manners: with l(y, ŷuni + ŷbi + ŷtri ) (referred to as joint), smro, and mro. for performance comparison, we include the routing model @xcite (referred to as routing), a recently proposed model with the goal of modality interpretability. lastly, we compare the performance against a single trimodal implementation details: the functions f of equation 4 are instantiated as multi-layer perceptrons. for each multimodal model, e.g., f θ t v , we implement two popular types of fusion: early fusion (concatenating the modalities) and tensor fusion @xcite (outer product between modalities after learning unimodal embeddings). the type of fusion is a hyper-parameter together with the number of layers, their width, learning rate, learning rate decay, l2 weight decay, dropout, and with/without prior feature selection. as a loss function, we use the mean absolute error for regression tasks and the cross-entropy loss for classification tasks. ## multimodal perception study we conduct a multimodal perception study to evaluate whether mro learns non-additive interactions, when humans also require non-additive interactions. we choose arousal and valence on the iemocap dataset for this study as arousal and valence are two fundamental dimensions to describe emotional states @xcite . study design: crowd workers foot_2 are asked to rate arousal and valence of video segments when being exposed to only a subset of modalities. the four subsets are: 1) the transcript of what the person says (t); 2) the muted video (v); 3) the low-pass filtered audio (a), and 4) the transcripts, the video, and the original audio (tva o ). iemocap has ten speakers. we randomly select ten segments for each speaker, i.e. 100 segments. audio processing: it is challenging to disentangle speech content and how we speak @xcite . similar to previous work, we low-pass filter the audio signal @xcite . instead of using 850 hz as a cut-off @xcite , we use a lower cut-off frequency, as we could understand spoken words at 850 hz. we choose 660 hz 4 as it is the mean of the maximum pitch in an empirical study @xcite and it also closely coincides with the maximum pitch of contralto singers (e 5 at 659.25 hz). we choose this pitch-focused definition as we believe that prosodic information will predict arousal and valence. avoiding learning effects: raters might be able to infer the missing multi-modal context after having rated some of the unimodal subsets for a specific segment. we therefore use two mechanisms to address learning effects across the modalities. first, each of the raters annotates only 20 randomly selected segments for each modality subset (we have eight raters per segment and modality subset). second, we structurally randomize the order of the modality subsets by first presenting all unimodal subsets in a random order and in the end the trimodal segments. ratings and reliability: following the annotation setup from iemocap, we use the ordinal arousal and valence manikins scale consisting of five levels @xcite to rate the two emotional dimensions. the effective reliability @xcite over k raters as measured by the intra-class correlation coefficient icc(2, k-1) is excellent (above 0.9) (koo and li, 2016) for all modality subsets. further, our new trimodal ratings (tva o ) correlate highly with the existing annotations on iemocap r(98) = 0.88, @xmath0.001 for arousal and r(98) = 0.92, @xmath1.001 for valence, indicating that we can use our new annotations to inspect models trained on the original annotations. evaluation: to evaluate when humans require non-additive interactions, we train a linear regression model (an additive model) that predicts tva o given t, v, and a. we refer to this model as ŷhuman uni . the model fit of ŷhuman uni shows how important the missing non-additive interactions are @xcite . further, the absolute error |tva oŷhuman uni | measures how important the missing non-additive interactions are to humans for each segment. we use |tva oŷhuman uni | to answer the question: does arousal valence min. age 19 21 mean age 36 37 max. age 79 62 female 20 19 male 20 21 ## conclusion we proposed mro to explicitly learn and separate unimodal, bimodal, and trimodal interactions in a multimodal model. this separation is essential for quantifying how much a model uses multimodal interactions and is a step towards more interpretable models. based on prior work @xcite we proposed a new evaluation metrics to quantify whether a trimodal models uses unimodal, bimodal, and trimodal interactions. empirically, we observed that mro successfully separated unimodal, bimodal, and trimodal interactions while not degrading predictive performance. beyond the empirical evaluation, mro learns non-additive interactions in accordance with human judgments on three datasets.