Improving Multimodal fusion via Mutual Dependency Maximisation (2021)

Abstract: Multimodal sentiment analysis is a trending area of research, and multimodal fusion is one of its most active topic. Acknowledging humans communicate through a variety of channels (i.e visual, acoustic, linguistic), multimodal systems aim at integrating different unimodal representations into a synthetic one. So far, a consequent effort has been made on developing complex architectures allowing the fusion of these modalities. However, such systems are mainly trained by minimising simple losses such as L_1 or cross-entropy. In this work, we investigate unexplored penalties and propose a set of new objectives that measure the dependency between modalities. We demonstrate that our new penalties lead to a consistent improvement (up to 4.3 on accuracy) across a large variety of state-of-the-art models on two well-known sentiment analysis datasets: CMU-MOSI and CMU-MOSEI. Our method not only achieves a new SOTA on both datasets but also produces representations that are more robust to modality drops. Finally, a by-product of our methods includes a statistical network which can be used to interpret the high dimensional representations learnt by the model.

## introduction humans employ three different modalities to communicate in a coordinated manner: the language modality with the use of words and sentences, the vision modality with gestures, poses and facial expressions and the acoustic modality through change in vocal tones. multimodal representation learning has shown great progress in a large variety of tasks including emotion recognition, sentiment analysis @xcite , speaker trait analysis @xcite and fine-grained opinion mining @xcite . learning from different modalities is an efficient way to improve performance on the target tasks @xcite . nevertheless, heterogeneities across modalities increase the difficulty of learning multimodal representations and raise specific challenges. @xcite identifies fusion as one of the five core challenges in multimodal representation learning, the four other being: representation, modality alignment, translation and co-learning. fusion aims at integrating the different unimodal representations into one common synthetic representation. effective fusion is still an open problem: the best multimodal models in sentiment analysis @xcite improve over their unimodal counterparts, relying on text modality only, by less than 1.5% on accuracy. additionally, the fusion should not only improve accuracy but also make representations more robust to missing modalities. multimodal fusion can be divided into early and late fusion techniques: early fusion takes place at the feature level @xcite , while late fusion takes place at the decision or scoring level @xcite . current research in multimodal sentiment analysis mainly focuses on developing new fusion mechanisms relying on deep architectures (e.g tfn @xcite , lfn @xcite , marn @xcite , misa @xcite , mctn @xcite , hfnn @xcite , iccn @xcite ). theses models are evaluated on several multimodal sentiment analysis benchmark such as iemocap @xcite , mosi @xcite , mosei @xcite and pom @xcite . current state-ofthe-art on these datasets uses architectures based on pre-trained transformers @xcite such as multimodal bert (magbert) or multimodal xlnet (magxlnet) @xcite . the aforementioned architectures are trained by minimising either a l 1 loss or a cross-entropy loss between the predictions and the ground-truth labels. to the best of our knowledge, few efforts have been dedicated to exploring alternative losses. in this work, we propose a set of new objectives to perform and improve over existing fusion mechanisms. these improvements are inspired by the infomax principle @xcite , i.e. choosing the representation maximising the mutual information (mi) between two possibly overlapping views of the input. the mi quantifies the dependence of two random variables; contrarily to correlation, mi also captures non-linear dependencies between the considered variables. different from previous work, which mainly focuses on comparing two modalities, our learning problem involves multiple modalities (e.g text, audio, video). our proposed method, which induces no architectural changes, relies on jointly optimising the target loss with an additional penalty term measuring the mutual dependency between different modalities. ## problem formulation & related work in this section, we formulate the problem of learning multi-modal representation (sec. 2.1) and we review both existing measures of mutual dependency (see sec. 2.2) and estimation methods (sec. 2.3). in the rest of the paper, we will focus on learning from three modalities (i.e language, audio and video), however our approach can be generalised to any arbitrary number of modalities. ## model and training objective in this section, we introduce our new set of losses to improve fusion. in sec. 3.1, we first extend widely used bi-variate dependency measures to multivariate dependencies (james and crutchfield, 2017) measures (mdm). we then introduce variational bounds on the mdm, and in sec. 3.2, we describe our method to minimise the proposed variational bounds. notations we consider x a , x v , x l as the multimodal data from the audio,video and language modality respectively with joint probability distribution p xaxvx l . we denote as p x j the marginal distribution of x j with @xmath0, v, l} corresponding to the jth modality. ## experimental setting in this section, we present our experimental settings including the neural architectures we compare, the datasets, the metrics and our methodology, which includes the hyper-parameter selection. algorithm 1 two-stage procedure to minimise multivariate dependency measures. input: d @xmath1, x j v , x j l ), ∀@xmath2, n]} multimodal training dataset, m batch size, σ a , σ v , σ l : [1, m] → [1, m] three permutations, θ c weights of the deep classifier, θ weights of the statistical network t θ . initialization: parameters θ and θ c build negative dataset: update θ based on the empirical version of eq. 6 or eq. 7 or eq. 8. end for sample a batch b from d update θ c with b using eq. 5. end while output: classifiers weights θ c ## numerical results we present and discuss here the results obtained using the experimental setting described in sec. 4. to better understand the impact of our new methods, we propose to investigate the following points: efficiency of the l m dm : to gain understanding of the usefulness of our new objectives, we study the impact of adding the mutual dependency term on the basic multimodal neural model ef-lstm. improving model performance and comparing multivariate dependency measures: the choice of the most suitable dependency measure for a given task is still an open problem (see sec. 3). thus, we compare the performance -on both multimodal sentiment and emotion prediction tasks-of the different dependency measures. the compared measures are combined with different models using various fusion mechanisms. improving the robustness to modality drop: a desirable quality of multimodal representations is the robustness to a missing modality. we study how the maximisation of mutual dependency measures during training affects the robustness of the representation when a modality becomes missing. towards explainable representations: the statistical network t θ allows us to compute a dependency measure between the three considered modalities. we carry out a qualitative analysis in order to investigate if a high dependency can be explained by complementariness across modalities. ## conclusions in this paper, we introduced three new losses based on mdm. through extensive set of experiments on cmu-mosi and cmu-mosei, we have shown that sota architectures can benefit from these innovations with little modifications. a by-product of our method involves a statistical network that is a useful tool to explain the learnt high dimensional multimodal representations. this work paves the way for using and developing new alternative methods to improve the learning (e.g new estimator of mu-tual information @xcite , wasserstein barycenters @xcite , data depths @xcite , extreme value theory @xcite ). a future line of research involves using this methods for emotion @xcite and dialog act @xcite classification with pretrained model tailored for spoken language @xcite .