Self-Supervised Unimodal Label Generation Strategy Using Recalibrated Modality Representations for Multimodal Sentiment Analysis (2023)

Abstract: While multimodal sentiment analysis (MSA) has gained much attention over the last few years, the main focus of most work on MSA has been limited to constructing multimodal representations that capture interactions between different modalities in a single task. This was largely due to a lack of unimodal annotations in MSA benchmark datasets. However, training a model using only multimodal representations can lead to suboptimal performance due to insufficient learning of each uni-modal representation. In this work, to fully optimize learning representations from multimodal data, we propose SUGRM which jointly trains multimodal and unimodal tasks using recalibrated features. The features are recalibrated such that the model learns to weight the features differently based on the features of other modalities. Further, to leverage unimodal tasks, we auto-generate unimodal annotations via a unimodal label generation module (ULGM). The experiment results on two benchmark datasets demonstrate the efficacy of our framework.

## introduction these days, we can easily spot ai systems in our society that serve or assist humans. understanding human emotions has become a critical factor for these ai systems to seamlessly integrate into human's life @xcite . however, understanding humans' emotions is not a trivial task. this is because humans tend to express their feelings through multiple cues in a complex form. emotions can be expressed simply through language, but they can also be manifested through facial expression, behaviors or even tone of voice @xcite . moreover, sometimes these cues signal a compatible emotion, while other times they signal conflicting emotions, e.g., positive language with a condescending tone of voice indicates sarcasm @xcite . taking this nature into account, multimodal sentiment analysis (msa) has become an active field of research which aims to understand the affective state of humans through visual, acoustic, and textual features. in general, when working with multimodal data like in msa, each modality contains both supplementary and complementary information to each other, providing richer information about the data. this leads to improved performance over using only one modality @xcite . however, capturing information in each modality as well as modeling the interactions between different modalities still remain challenging tasks to unravel @xcite . most of the existing works on msa revolve around learning a joint representation which emcompasses information from all modalities through sophisticated fusion methods varying from tensorbased @xcite to attention-based methods @xcite , where the learning process happens in a single task. single task learning was a dominant learning framework in msa particularly due to the nature of the benchmark datasets: cmu-mosi @xcite and cmu-mosei @xcite . considering all modalities, only one comprehensive sentiment intensity value (i.e., multimodal label, y m ) is annotated in both datasets due to the laborious labeling process. meaning, unimodal labels (y t , y a , y v ) are omitted in the datasets. however, a recent study @xcite argued the absence of unimodal annotations hinders capturing modality-specific information and proposed a module that auto-generates unimodal annotations from the multimodal labels. in this work, we propose a novel framework, sugrm, which leverages a self-supervised unimodal label generation strategy using recalibrated modality representations for msa. first, we recali-brate modality representations using modality recalibration module (mrm). this allows the model to dynamically adjust features based on the features of other modalities. further, motivated by @xcite , we propose a new unimodal label generation module (ulgm), which generates unimodal annotations (y t , y a , y v ) based on the multimodal annotation (y m ) in a self-supervised manner. different from @xcite , which preserves feature space of each modality, we project features of each modality into a common semantic feature space. thus, our ulgm hypothesizes the distance between two features in a common semantic feature space is proportional to the distance between the corresponding labels in a label space. this not only allows simpler calculation of the offset (see section 3.3), but also avoids the problem in @xcite ; that is, when two distances from a multimodal feature 1) to the center of negative multimodal features and 2) to the center of positive multimodal features are approximately equal, the generated unimodal label diverges. this could lead to unstable learning, potentially causing the model to fall into a local minima. our experiment results not only empirically validate our hypothesis, but also prove that using recalibrated modality representation as well as our ulgm lead to enhanced performance. the main contributions of our work can be summarized as follows: ## related work prior works of msa mainly focused on improving fusion between multi-modalities as well as learning joint representations. in earlier works, early fusion @xcite and late fusion @xcite were popular fusion methods to combine the multiple modalities. later, more sophisticated methods of fusion were proposed using a multi-dimensional tensor @xcite , attention mechanism @xcite , multi-stage fusion @xcite and low rank tensors to improve efficiency of fusion @xcite . in @xcite , the authors dynamically adjusted a word representation by calculating a shift caused by accompanying nonverbal information. more recent works have focused on applying transformer architecture to better capture interactions between modalities and learn feature representations. for instance, @xcite was directly built upon @xcite , but used pretrained transformer based language models to improve the performance. @xcite proposed cross-modal attention to latently adapt a target modality from source modalities. @xcite reduced the computational burden in @xcite , by generating sparse attention matrices and compressing a long sequence to a short sequence. further, a multi-task learning approach has been applied in recent msa @xcite to increase data efficiency. taking inspiration from the previous work @xcite , we expand a learning framework of msa to multi-task learning. the benefits of multitask learning is that each task helps a learning process of other tasks. this allows the model to learn better generalized representations that are shared across the tasks. further, we recalibrate features of each modality and efficiently model inter-, intramodality relationships by adopting the work of @xcite @xcite . ## conclusion and future work in this paper, we proposed sugrm, a novel framework for multimodal sentiment analysis (msa) which incorporates unimodal subtasks to aid the learning process of the multimodal task. to enable this, we first designed modality recalibration module (mrm) so that features of each modality are recalibrated based on the features of other modalities. then, we designed a unimodal label generation module (ulgm) based on the notion that the distance between two features in a common semantic feature space is proportional to the distance between the corresponding labels in a label space. from this, ulgm was able to generate unimodal annotations from the multimodal label in a self-supervised manner, which saved a tremendous amount of human labor. the experiment results validated our notion as well as the reliability of the unimodal labels generated from our ulgm. for future work, expanding the framework to jointly train sentiment and emotion tasks could be worthwhile. recently @xcite proposed that msa and multimodal emotion recognition are closely correlated; therefore their tasks can be carried out jointly. applying contrastive learning for different emotion classes and exploiting correlation between sentiment and emotion could help achieve better results in both tasks.