Multimodal Multi-loss Fusion Network for Sentiment Analysis (2024)

Abstract: This paper investigates the optimal selection and fusion of feature encoders across multiple modalities and combines these in one neural network to improve sentiment detection. We compare different fusion methods and examine the impact of multi-loss training within the multi-modality fusion network, identifying surprisingly important findings relating to subnet performance. We have also found that integrating context significantly enhances model performance. Our best model achieves state-of-the-art performance for three datasets (CMU-MOSI, CMU-MOSEI and CH-SIMS). These results suggest a roadmap toward an optimized feature selection and fusion approach for enhancing sentiment detection in neural networks.

## introduction in recent years, the multimodal affective computing field has seen significant advances in feature extraction and multimodal fusion methodologies in recent years @xcite , enabling a more nuanced understanding of human emotions by effectively synthesizing audio, text, and visual signals @xcite . this study presents a series of experiments that delve into feature selection, comparative analysis of fusion network performance, multi-loss training, and context modeling utilizing audio and text from three datasets: cmu-mosi @xcite , cmu-mosei @xcite , and ch-sims @xcite . our research aims to pioneer state-of-the-art (sota) approaches in affective computing tasks, achieved by identifying optimal features for each modality, devising the most effective methods for their fusion, and refining training methodologies to enhance performance. hand-crafted feature extraction algorithms often lack flexibility and generalization across diverse tasks. to overcome these limitations, recent studies have proposed fully end-to-end models that jointly optimize feature extraction and learning processes @xcite . our work leverages feature representations derived from pretrained models across different modalities, combining them into an end-to-end framework, which provides a comprehensive and adaptable solution for multimodal affective feature computation. in multimodal fusion, the challenge lies in effectively fusing diverse signals, including natural language, facial gestures, and acoustic behaviors. methods like the tensor fusion network (tfn) @xcite have been proposed to model intra-modality and inter-modality interactions. more recently, transformer encoder structures with cross-modal attention have gained popularity for integrating multimodal data @xcite , with continous efforts to improve representations of multimodal information @xcite . building from these ideas, we propose a robust fusion network structure that integrates cross-modal attention and self-attention, with additional feed-forward layers to refine the representations. furthermore, our approach experiments with restoring original signals during the fusion process. our experimental analysis has identified the most efficacious features for different modalities and compared various fusion network methods for amalgamating audio and text signals. our findings reveal that the incorporation of audio signals consistently elevates performance metrics. more notably, our transformer fusion network demonstrates a remarkable enhancement of results and and achieves state-of-the-art performances across all datasets. additionally, our exploration into multi-loss training has yielded two significant observations: first, the utilization of distinct labels for each modality in multi-loss training markedly benefits the models' performance; second, training with multimodal features not only boosts overall model performance but also notably enhances accuracy in the singlemodality subnet. furthermore, we compared two context modeling methods and found that contextual integration significantly amplifies model performance across all metrics. these novel findings have advanced our understanding of multi-modal sentiment analysis. ## related work existing research on multimodal affective computing often employs hand-crafted algorithms to perform initial feature representation extraction and retrieve some fixed representations for each modality @xcite . however, for these, the extracted features are static and lack the flexibility to be further fine-tuned for different target tasks; also, the manual determination of feature extraction algorithms can lead to sub-optimal performance due to constraints in generalization across diverse tasks @xcite . to address these issues, recent studies have proposed fully end-to-end models, effectively bridging the gap between feature extraction and learning processes @xcite . our research also emphasizes an end-to-end structure that optimizes both phases jointly, presenting a comprehensive and adaptable solution for multimodal affective feature computation. lexical features, owing to pre-training on expansive corpora through transformer-based models, often outperform other modalities. some recent work aims to improve model performance by incorporating speech information inside the text model such as spectra @xcite , by pre-training a speech-text transformer model to capture the speech-text alignment effectively. a similar innovative method is the transformer-based speech-prefixed language model (teasel) @xcite , which incorporates speech as a dynamic prefix along with the textual. many studies have explored multimodal human language time-series data, which typically includes a mixture of natural language, facial gestures, and acoustic behaviors. however, fusing these into a unified representation presents a significant challenge due to the variable sampling rates across modalities and the difficulty in determining intra-modality dependencies. vari-ous methods have been proposed to model the interaction across modalities (barezi and fung, 2019), such as the tensor fusion network @xcite , which utilizes the cartesian product of different modalities to model both intramodality and inter-modality interactions. more recent work has shifted toward employing transformer encoder structures to integrate these signals via cross-modality attention. the mult model @xcite has pioneered this approach, introducing directional pairwise cross-modal attention. this method allows for interaction between multimodal sequences across distinct time steps and inherently adapts streams from one modality to another. further research has also leveraged this concept of cross-modality attention @xcite , yielding valuable insights into how multimodal data can be processed more effectively. we enhance this approach by employing a self-attention encoder and a feed-forward network to further optimize the multimodal representation after one modality is projected into another using the cross-modality attention module, thus enriching our ability to process and understand multimodal data. ## methodology the model for sentiment detection in our study involves two primary components: the feature network and the fusion network. each of these has its own unique mechanisms and contributes towards the overall functioning of our proposed multi-modality multi-loss fusion network (mmml) as illustrated in figure 1 . additionally, we adopted multi-loss training, experimented with restoring original signals, and explored context modeling. ## conclusion in conclusion, this study has provided novel, important findings for multi-modal sentiment analysis that should benefit future researchers in the designing of sentiment analysis and other models. first, the use of pre-trained models for raw audio yielded superior results, highlighting their effectiveness in feature extraction. second, combining audio and text signals consistently outperformed using text signals alone, with the transformer fusion network showing promise in enhancing cross-modality modeling. third, multi-loss training proved beneficial for performance, particularly with unique labels for each modality. fourth, context information boosts model performances significantly. last, achieving state-of-the-art results on three sentiment detection datasets underscores the effectiveness of our approach. moreover, in analysis of ablation studies, we show model performance can be improved through a method that reflects a similar pattern with human understanding of sentiment analysis through the multi-loss training. moreover, multimodal features improve both the overall model performance in multimodality and in single-modality input settings. we provide more effective ways to handle missing modalities and utilized individual modality representation, by allowing the model to train on multimodal features and boost performance on single modality input using multi-loss training. still, the performance of fusion network variations did remain consistent, prompting further investigation.