Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusion (2023)

Abstract: Video multimodal fusion aims to integrate multimodal signals in videos, such as visual, audio and text, to make a complementary prediction with multiple modalities contents. However, unlike other image-text multimodal tasks, video has longer multimodal sequences with more redundancy and noise in both visual and audio modalities. Prior denoising methods like forget gate are coarse in the granularity of noise filtering. They often suppress the redundant and noisy information at the risk of losing critical information. Therefore, we propose a denoising bottleneck fusion (DBF) model for fine-grained video multimodal fusion. On the one hand, we employ a bottleneck mechanism to filter out noise and redundancy with a restrained receptive field. On the other hand, we use a mutual information maximization module to regulate the filter-out module to preserve key information within different modalities. Our DBF model achieves significant improvement over current state-of-the-art baselines on multiple benchmarks covering multimodal sentiment analysis and multimodal summarization tasks. It proves that our model can effectively capture salient features from noisy and redundant video, audio, and text inputs. The code for this paper will be publicly available at https://github.com/WSXRHFG/DBF

## introduction with the rapid development of social platforms and digital devices, more and more videos are flooding our lives, which leads video multimodal fusion an increasingly popular focus of nlp research. video multimodal fusion aims to integrate the information from two or more modalities (e.g., visual and audio signals) into text for a more comprehensive reasoning. for example, multimodal sentiment analysis @xcite utilizes contrast between transcript and expression to detect sarcam, multimodal summarization @xcite summary with information only exists in visual signal. however, as shown in the figure 1 , there exist plenty of redundancy and noise in video multimodal fusion: 1) high similarity across consecutive frames brings video redundancy; 2) useless information, such as the distracting background, introduces frame noise; 3) weak alignment between visual stream and text also introduces misalignment noise. to alleviate the problem of redundancy and noise in video multimodal fusion, @xcite control the flow of redundant and noisy information between multimodal sequences by a fusion forget gate. the fusion forget gate impairs the impact of noise and redundancy in a coarse grain of the whole modality, so it will also filter out some representative information in the filtered modality. in order to remove noise and redundancy while preserving critical information in video multimodal fusion, we propose a denoising fusion bottleneck (dbf) model with mutual information maximization (mi-max). firstly, inspired by @xcite , we introduce a bottleneck module to restrict the redundant and noisy information across different modalities. with the bottleneck module, inputs can only attend to low-capacity bottleneck embeddings to exchange information across different modalities, which urges redundant and noisy information to be discarded. secondly, in order to prevent key information from being filtered out, we adopt the idea of contrastive learning to supervise the learning of our bottleneck module. specifically, under the noise-contrastive estimation framework @xcite , for each sample, we treat all the other samples in the same batch as negative ones. then, we aim to maximize the mutual information between fusion results and each unimodal inputs by distinguishing their similarity scores from negative samples. two aforementioned modules complement each other, the mi-max module supervises the fusion bottleneck not to filter out ## related work we briefly overview related work about multimodal fusion and specific multimodal fusion tasks including multimodal summarization and multimodal sentiment analysis. ## methodology our denoising fusion bottleneck (dbf) model aims to fuse multimodal inputs from videos to make a comprehensive prediction. the overall architecture of dbf is shown in figure 2 . it first employs a fusion bottleneck module with a restrained receptive field to filter out noise and redundancy when fusing different modalities in videos. then, dbf maximizes mutual information between fusion results and unimodal inputs to supervise the learning of the fusion bottleneck, aiming to preserve more representative information in fusion results. ## conclusion in this paper, we propose a denoising video multimodal fusion system dbf which contains a fusion bottleneck to filter out redundancy with noise, a mutual information module to preserve key information in fusion results. our model alleviates redundancy and nosie problem in video multimodal fusion and makes full use of all representative information in redundant modalities (vision and acoustic). in the experiments, we show that our model significantly and consistently outperforms state-ofthe-art video multimodal models. in addition, we demonstrate that dbf can appropriately select necessary contents and neglect redundancy in video by comprehensive ablation and visualization studies. in the future, we will explore the following directions: (1) we will try to extend the proposed dbf model to more multimodal fusion tasks such as humor detection. (2) we will incorporate visiontext pretraining backbones into our dbf model to further improve its performance.