Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis (2021)

Abstract: In multimodal sentiment analysis (MSA), the performance of a model highly depends on the quality of synthesized embeddings. These embeddings are generated from the upstream process called multimodal fusion, which aims to extract and combine the input unimodal raw data to produce a richer multimodal representation. Previous work either back-propagates the task loss or manipulates the geometric property of feature spaces to produce favorable fusion results, which neglects the preservation of critical task-related information that flows from input to the fusion results. In this work, we propose a framework named MultiModal InfoMax (MMIM), which hierarchically maximizes the Mutual Information (MI) in unimodal input pairs (inter-modality) and between multimodal fusion result and unimodal input in order to maintain task-related information through multimodal fusion. The framework is jointly trained with the main task (MSA) to improve the performance of the downstream MSA task. To address the intractable issue of MI bounds, we further formulate a set of computationally simple parametric and non-parametric methods to approximate their truth value. Experimental results on the two widely used datasets demonstrate the efficacy of our approach.

## introduction with the unprecedented advances in social media in recent years and the availability of smartphones with high-quality cameras, we witness an explosive boost of multimodal data, such as movies, short-form videos, etc. in real life, multimodal data usually consists of three channels: visual (image), acoustic (voice), and transcribed text. many of them often express sort of sentiment, which is a long-term disposition evoked when a person encounters a specific topic, person or entity @xcite . mining and understanding these emotional elements from multimodal data, namely multimodal sentiment analysis (msa), has become a hot research topic because of numerous appealing applications, such as obtaining overall product feedback from customers or gauging polling intentions from potential voters @xcite . generally, different modalities in the same data segment are often complementary to each other, providing extra cues for semantic and emotional disambiguation @xcite . the crucial part for msa is multimodal fusion, in which a model aims to extract and integrate information from all input modalities to understand the sentiment behind the seen data. existing methods to learn unified representations are grouped in two categories: through loss back-propagation or geometric manipulation in the feature spaces. the former only tunes the parameters based on back-propagated gradients from the task loss @xcite @xcite , reconstruction loss @xcite , or auxiliary task loss @xcite . the latter additionally rectifies the spatial orientation of unimodal or multimodal representations by matrix decomposition @xcite or euclidean measure optimization @xcite . although having gained excellent results in msa tasks, these methods are limited to the lack of control in the information flow that starts from raw inputs till the fusion embeddings, which may risk losing practical information and introducing unexpected noise carried by each modality @xcite . to alleviate this issue, different from previous work, we leverage the functionality of mutual information (mi), a concept from the subject of information theory. mi measures the dependencies between paired multi-dimensional variables. maximizing mi has been demonstrated ef-ficacious in removing redundant information irrelevant to the downstream task and capturing invariant trends or messages across time or different domains @xcite , and has been shown remarkable success in the field of representation learning @xcite . based on these experience, we propose multimodal infomax (mmim), a framework that hierarchically maximizes the mutual information in multimodal fusion. specifically, we enhance two types of mutual information in representation pairs: between unimodal representations and between fusion results and their low-level unimodal representations. due to the intractability of mutual information @xcite , researchers always boost mi lower bound instead for this purpose. however, we find it is still difficult to figure out some terms in the expressions of these lower bounds in our formulation. hence for convenient and accurate estimation of these terms, we propose a hybrid approach composed of parametric and non-parametric parts based on data and model characteristics. the parametric part refers to neural network-based methods, and in the non-parametric part we exploit a gaussian mixture model (gmm) with learning-free parameter estimation. our contributions can be summarized as follows: ## related work in this section, we briefly overview some related work in multimodal sentiment analysis and mutual information estimation and application. ## experiments in this section, we present some experimental details, including datasets, baselines, feature extraction tool kits, and results. ## further analysis in this section, we dive into our models to explore how it functions in the msa task. we first visualize all types of losses in the training process, then we analyze some representative cases. ## conclusion in this paper, we present mmim, which hierarchically maximizes the mutual information (mi) in a multimodal fusion pipeline. the model applies two mi lower bounds for unimodal inputs and the fusion stage, respectively. to address the intractability of some terms in these lower bounds, we specifically design precise, fast and robust estimation methods to ensure the training can go on normally as well as improve the test outcome. then we conduct comprehensive experiments on two datasets followed by the ablation study, the results of which verify the efficacy of our model and the necessity of the mi maximization framework. we further visualize the losses and display some representative examples to provide a deeper insight into our model. we believe this work can inspire the creativity in representation learning and multimodal sentiment analysis in the future.