The MERSA Dataset and a Transformer-Based Approach for Speech Emotion Recognition (2024)

Abstract: Research in the field of speech emotion recognition (SER) relies on the availability of comprehensive datasets to make it possible to design accurate emotion detection models. This study introduces the Multimodal Emotion Recognition and Sentiment Analysis (MERSA) dataset, which includes both natural and scripted speech recordings, transcribed text, physiological data, and self-reported emotional surveys from 150 participants collected over a two-week period. This work also presents a novel emotion recognition approach that uses a transformer-based model, integrating pre-trained wav2vec 2.0 and BERT for feature extractions and additional LSTM layers to learn hidden representations from fused representations from speech and text. Our model predicts emotions on dimensions of arousal, valence, and dominance. We trained and evaluated the model on the MSP-PODCAST dataset and achieved competitive results from the best-performing model regarding the concordance correlation coefficient (CCC). Further, this paper demonstrates the effectiveness of this model through cross-domain evaluations on both IEMOCAP and MERSA datasets.

## introduction emotions play a significant role in human interactions, as they can significantly impact our thoughts and actions @xcite . speech-based emotion recognition (ser) has become increasingly popular over the last two decades due to its wide range of applications in human-computer interactions such as digital learning and mental health monitoring @xcite . in ser systems, emotions are commonly represented in two ways. studies that classify emotions categorically often use emotions derived from plutchik's wheel of emotions @xcite ) or ekman's six basic emotion types, which include anger, fear, disgust, happiness, sadness, and surprise, some-times with some variations @xcite . emotions can also be measured along continuous dimensions, including valence (pleasantness), arousal (intensity), and dominance (control) @xcite . the dimensional approach to studying emotions has become increasingly popular, as it can capture subtle emotional changes and represent more complex emotions than the discrete approach. despite the significant interest in ser in academia and industry, several challenges must be addressed. one of the significant challenges is the need for comprehensive natural datasets @xcite . many datasets in this field are collected under simulated environments, which may not accurately reflect genuine emotions in real-world scenarios. although some datasets claim to be natural, they often source their data from movies, tv shows, and online videos, raising questions about their naturalness. another common challenge for ser is to improve the accuracy of emotion prediction. previous research has explored emotions from various sources like text, speech, vision, and physiological data, either separately (unimodal) or together (multimodal). combining different modalities has become a popular approach in emotion recognition, which many studies have shown to lead to more accurate prediction than the unimodal approach @xcite @xcite . however, there is a dearth of prior research using physiological cues for ser due to limited availability of such data. our work aims to address these challenges. first, we introduce the multimodal emotion recognition and sentiment analysis (mersa) dataset, which consists of data collected from 150 participants in real-world scenarios using a mobile crowdsensing approach @xcite . the data was collected through daily and weekly ecological momentary assessments (emas). in total, we collected 37.42 hours of audio and 2,650 self-reported assessments from the emas. additionally, we obtained 49,550 hours of physiological data from a wrist-worn wearable device. we labeled the speech data using a novel method covering both dimensional and sentiment labels for each utterance. second, we developed a model using a transformer-based approach by implementing wav2vec @xcite and bert @xcite to extract acoustic and linguistic features. we trained and evaluated the model on the msp-podcast @xcite dataset. the best-performing model achieved competitive concordance correlation coefficient (ccc) scores on valence and arousal, and we further evaluated the model on the iemocap and the mersa datasets to demonstrate its cross-domain effectiveness and reproducibility. our work is divided into several sections. section 2 covers related work, while section 3 explains how we acquired and pre-processed the mersa dataset. our proposed model is described in section 4, and section 5 presents the datasets, models, and evaluation metrics used in our experiments, along with the results. we finally conclude our work and identify its limitations in sections 6 and 7. ## related work this section discusses existing datasets frequently used for natural language processing (nlp) research and prior works in multimodal ser, including both categorical and dimensional approaches. @xcite ) contains 2,281 video segments from real-world scenarios, with multimodal and unimodal annotations for each utterance. cmu-mosei @xcite includes over 3,000 youtube videos covering various topics, such as reviews and debates, and carefully selected and annotated by experts. msp-podcast @xcite is a collection of podcast recordings from audio-sharing websites featuring natural conversations on a wide range of topics and continually updated since 2017. @xcite captures participants' responses to music videos and movie clips, alongside physiological signals from eog, ecg, and emg sensors, annotated by 7 external experts. recola @xcite records spontaneous interactions during a remote collaborative task, including self-reports at the be-ginning and end of the task, involving 46 participants. moud @xcite focuses on spanish-language youtube review videos, pairing each utterance with its audio, video stream, and manual transcription. @xcite provides 959 conversations from 150 participants, complete with detailed annotations and transcriptions. youtube @xcite encompasses a vast collection of product reviews and opinions from the social media platform, highlighting the diversity of public sentiment. finally, the widely popular iemocap @xcite repository offers a rich dataset from ten speakers, capturing spoken communication scenarios, facial expressions, and hand movements. ## proposed framework to provide a clear understanding of the network architecture of our emotion recognition system, we have included figure 2 . the audio input and corresponding transcriptions were processed separately using wav2vec @xcite and bert @xcite to extract features, and both are foundational models for speech-related applications. the wav2vec 2.0 was pre-trained on a large librivox dataset and fine-tuned on the whole libri speech dataset. it achieved the best results (wer) while using 100 times less labeled data than previous sota asr systems. the model has over 95 million parameters and 12 transformer layers, with a model dimension of 768. bert was pretrained on a large corpus of english text and applies its self-attention mechanism to learn contextualized word embeddings. the bert-base model has 12 transformer layers, 110 million parameters, and an embedding size 768. before being processed by wav2vec, the audio files were checked to ensure they were already in 16khz and mono channel format. the raw audio is processed using wav2vec to transform it into a sequence of 768-dimensional feature vectors per time step. given a raw audio waveform, x audio , of shape [t ], where t is the number of time steps in the audio signal, wav2vec 2.0 processes x audio through its layers to produce a high-dimensional representation, h audio , of shape [t, d], with d being the feature dimension (e.g., 768 for the base model). mean pooling is applied across the time dimension to condense h audio into a single feature vector, f audio , representing the entire audio clip: this results in the final feature vector, f audio , of shape @xcite . similarly, the bert model extracts a sequence of 768-dimensional feature vectors from the text, and we utilize the [cls] token representation to obtain a single 768-dimensional feature vector, resulting in f text of shape @xcite to ensure text features are condensed to the same dimensionality. then, audio and textual feature vectors are concatenated to form a unified representation via early fusion before feeding them into the first dense layer. these audio and text features are fused to create a 1536-dimensional combined feature vector. for input features f audio and f text , both of shape [d], fusion (concatenation) operation combines these two vectors along the feature dimension to produce resulting in f fused of shape @xcite . we then compute mean and standard deviation statistics on these merged features and normalize them to ensure that they are on a consistent scale for model training. then, the model leverages a sophisticated arrangement of long short-term memory (lstm) @xcite layers followed by fully connected (dense) layers to effectively capture both the temporal dynamics and complex relationships inherent in emotional expression data. the model's core is a sequence of five lstm layers with 512 hidden units each, configured to process the input data in a sequence-first manner. this design choice allows the network to maintain a rich contextual understanding of the input features, which is crucial for accurate emotion prediction. to mitigate overfitting, a dropout rate of 0.5 is applied within the recurrent layers. following the recurrent layers, the model includes three dense layers. the first two layers have 256 and 128 units, respectively, and use relu activation functions to introduce non-linearity. they also incorporate dropout for regularization. the final dense layer is designed to produce continuous output values for each emotion dimension, aligning with the number of output emotion dimensions (e.g., 3 for valence, arousal, and dominance), and uses a linear activation function. a dropout rate of 0.5 is applied between these dense layers to improve regularization, which is consistent with the dropout strategy used in the lstm layers. the concordance correlation coefficient (ccc) has become the standard evaluation metric for dimensional emotion. we implement a customized loss function that optimizes for evaluation and model selection for ccc. for two random variables x and y, ccc and cccloss l are defined as the ρ in this equation is the pearson correlation coefficient between the predicted and actual values, σ and µ are standard deviations and mean, respectively. ccc measures how well the two variables (actual values and predicted values) are aligned. ## conclusions this paper introduces the mersa dataset, one of the most comprehensive collections for multimodal emotion recognition and sentiment analysis. it comprises 21,384 sentences spoken by 150 realworld individuals, including both natural responses and scripted monologues. out of these, 10,629 sentences involve self-assessments of the speakers' emotional states, which we have labeled with dimensional tags for valence, arousal, and dominance and sentiment tags for positive, negative, and neutral emotions. additionally, the dataset contains significant physiological data that can be used for ser and other relevant studies. although we have not utilized physiological cues in this study, we intend to use them in future studies to gain more insights into emotion recognition. the dataset and the best-performing model are currently available upon request, and we expect it to be a valuable resource for the community to pursue various nlp research initiatives. 5 transformers have significantly impacted various artificial intelligence tasks, including speech emotion recognition (ser). our research has validated their effectiveness by analyzing and evaluating popular transformer-based speech models for recognizing emotions in multiple dimensions. built upon the msp-podcast dataset, our model achieved competitive performance by recognizing valence with a ccc score of 0.683 and 0.590 on arousal and dominance, respectively. to further verify the effectiveness of our approach, we have evaluated its performance on the iemocap and mersa datasets, and it has also achieved competitive results.