Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis (2022)

Abstract: Multimodal speech emotion recognition (SER) and sentiment analysis (SA) are important techniques for human-computer interaction. Most existing multimodal approaches utilize either shallow cross-modal fusion of pretrained features, or deep cross-modal fusion with raw features. Recently, attempts have been made to fuse pretrained feature representations in a deep fusion manner during fine-tuning stage. However those approaches have not led to improved results, partially due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining. In this work, leveraging single-modal pretrained models (RoBERTa and HuBERT), we propose a novel deeply-fused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and a stage-wise cross-modal pretraining scheme to fully facilitate the cross-modal learning. Our experiment results show that the proposed method achieves state-of-the-art results on the public IEMOCAP emotion and CMU-MOSEI sentiment datasets, exceeding the previous benchmarks by a large margin.

## introduction speech emotion recognition and sentiment analysis are tasks of analyzing people's attitude and opinions from their speeches. emotion can be categorized into different classes. the most well-known emotion categorization is given by erman who proposes 6 basic emotion classes @xcite : fear, anger, joy, sadness, disgust, and surprise. sentiment reflects human attitude towards an event or object, and it is often labeled as positive, neutral or negative. among the emotion classes, joy can be considered as a positive sentiment whereas anger, sadness, fear and disgust are negative sentiments. emotion understanding helps people communicate with each other more effectively. for humancomputer interaction (hci), speech emotion recog- * * equal contribution nition (ser) plays a vital role in assisting computers to understand people's opinions. with the high-demand applications in hci, such as voice assistants and callbots, it is essential to recognize the users' emotions and respond accordingly. with the rapid progress in deep learning (dl), many dl methods have been applied to ser @xcite @xcite and sentiment analysis (sa) @xcite . there are two key challenges in these tasks at the current stage. the first challenge is the limited availability of annotated data. emotion annotation is known to be difficult because the pre-established annotation schemes are based on human psychology and are not conducive to reliable emotion annotation @xcite . therefore, compared to other tasks such as automatic speech recognition (asr), existing ser and sa annotated datasets are fairly small in size @xcite . direct training using such datasets may be prone to overfitting and poor model generalization ability. one way to address the issue regarding the scarcity of labeled data is via selfsupervised learning (ssl). ssl has gained great success in areas of natural language processing (nlp) and speech. it has become the standard approach to build general-purpose pretrained models by utilizing large amount of unlabeled data. the pretrained models have achieved state-of-the-art (sota) performance on both nlp and speech related tasks. for ser and sa tasks, fine-tuning single-modal pretrained model also leads to impressive results @xcite @xcite . the second challenge is the learning of a multimodal feature space that can well distinguish among different emotions or sentiments, especially in the case of multimodal modeling. finding the most effective mechanism to fuse features from different modalities remains an open problem. pre-vious studies have explored different approaches to fuse multimodal information for ser and sa tasks. however, most existing approaches focus mainly on shallow fusion mechanisms, which are unlikely to capture the deep latent relationships among different modalities. motivated by the aforementioned challenges, we here propose a novel speech-text bi-modal transformer built on top of single-modal pretrained models roberta and hubert. the proposed model has a carefully designed deep cross-modal fusion mechanism, which is trained with a novel crossmodal ssl based pretraining. our experiments on ser task @xcite and sa task @xcite showcase that our model architecture leads to effective cross-modal fusion and improves the performance on both tasks. we also propose a novel stage-wise training scheme from initial ssl pretraining to final fine-tuning to fully leverage the large amount of unlabeled data. our pretrained cross-modal transformer achieves the sota performance on both the iemocap emotion dataset and cmu-mosei sentiment dataset. it is worth noting that our proposed training scheme is not limited to ser and sa tasks. the main contributions of our paper include: ## method we first briefly review hubert @xcite and roberta @xcite , the two singlemodal pretrained models on top of which our bimodal pretrained model is built. then we introduce the proposed model architecture, which is followed by the training strategy during the pretraining and fine-tuning phases. ## conclusion in this work, we propose a novel bi-modal model with a symmetric cross-modal attention mechanism that efficiently fuses the representations from single-modal pretrained models. we show that upon pretraining followed by task adaptative pretraining and fine-tuing with additional modality orthogonal regularization, the proposed bi-modal model can achieve sota performance on both iemocap and cmu-mosei datasets.