emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation (2024)

Abstract: We propose emotion2vec, a universal speech emotion representation model. emotion2vec is pre-trained on open-source unlabeled emotion data through self-supervised online distillation, combining utterance-level loss and frame-level loss during pre-training. emotion2vec outperforms state-of-the-art pre-trained universal models and emotion specialist models by only training linear layers for the speech emotion recognition task on the mainstream IEMOCAP dataset. In addition, emotion2vec shows consistent improvements among 10 different languages of speech emotion recognition datasets. emotion2vec also shows excellent results on other emotion tasks, such as song emotion recognition, emotion prediction in conversation, and sentiment analysis. Comparison experiments, ablation experiments, and visualization comprehensively demonstrate the universal capability of the proposed emotion2vec. To the best of our knowledge, emotion2vec is the first universal representation model in various emotion-related tasks, filling a gap in the field.

## introduction extracting emotional representation from speech is an essential step of various emotional tasks such as speech emotion recognition (ser) and sentiment analysis. traditional methods employ filter banks (fbanks) or mel frequency cepstrum coefficients (mfccs) as speech features. these features are not rich in semantic information, resulting in limited performance on emotional tasks. popular methods utilize features extracted from speech-based self-supervised learning (ssl) pre-trained models, leading to a significant performance improvement. one potential challenge blocking further performance improvement is that these ssl models are not entirely suitable for emotional tasks. @xcite explore no fine-tuning, partial finetuning, and entire fine-tuning with some ssl models for ser on the iemocap dataset @xcite , and give some empirical conclusions. while this is an ad-hoc solution, on the one hand, finetuning ssl models requires a large computational cost, on the other hand, these conclusions may be data-specific or model-constrained. recently, @xcite proposed an ser model named vesper, which is obtained by model distillation from wavlm-large @xcite with emotion data. vesper is designed to perform the ser task, whose universal representation capability still needs to be demonstrated. accordingly, a universal speech-based emotion representation model is urgently needed in the field. here we propose emotion2vec, a universal emotion representation model that can be used to extract speech features for diverse emotion tasks. self-supervised pre-training is performed on 262 hours of open-source emotion data with an online distillation paradigm to obtain emotion2vec. considering that both whole-play information and local details convey emotion, we propose a pre-training strategy combining utterance-level loss and framelevel loss. on the mainstream iemocap dataset, the downstream linear model trained with features extracted from emotion2vec outperforms all the mainstream ssl models and the latest specialist models. emotion2vec is tested on 13 datasets including 10 languages, and the results show that emotion2vec exhibits language generalization ability. moreover, in addition to the ser task, we also experimented with emotion2vec features on song emotion recognition, emotion prediction in conversation, and sentiment analysis. the results indicate that emotion2vec has excellent task generalization ability. extensive ablation experiments and visualization analysis demonstrate the effectiveness of our pre-training methods and the versatility of the proposed emotion2vec model. ## methods here we mainly introduce the self-supervised pretraining method of the proposed emotion2vec, for which the core is to train the model with utterancelevel loss and frame-level loss using online distillation paradigm. combining utterance-level and frame-level loss is inspired by our observation that both global and local information convey emotion in speech. moreover, initializing with pre-trained models for the teacher-student network warm-ups the online distillation process, providing a good representation for the subsequent self-supervised bootstrap learning. ## conclusion in this paper, we propose emotion2vec, a universal emotion representation model. emotion2vec is pre-trained on 262 hours of unlabeled emotion data through self-supervised online distillation, leading to universal emotion representation ability. we prove that our strategy of combining utterancelevel loss and frame-level loss during emotion pretraining is effective. extensive experiments demonstrate that the proposed emotion2vec has the ability to extract emotion representation across different tasks, languages, and scenarios. in the future, we will explore the scaling law of emotion representation models, namely how to provide a better representation with more data and larger parameters.