Classifying Emotional Utterances by Employing Multi-modal Speech Emotion Recognition (2021)

Abstract: Deep learning methods are being applied to several speech processing problems in recent years. In the present work, we have explored different deep learning models for speech emotion recognition. We have employed normal deep feedforward neural network (FFNN) and convolutional neural network (CNN) to classify audio files according to their emotional content. Comparative study indicates that CNN model outperforms FFNN in case of emotions as well as gender classification. It was observed that the sole audio based models can capture the emotions up to a certain limit. Thus, we attempted a multi-modal framework by combining the benefits of the audio and text features and employed them into a recurrent encoder. Finally, the audio and text encoders are merged to provide the desired impact on various datasets. In addition, a database consists of emotional utterances of several words has also been developed as a part of this work. It contains same word in different emotional utterances. Though the size of the database is not that large but this database is ideally supposed to contain all the English words that exist in an English dictionary.

## introduction human computer interaction (hci) researches the way we humans interact with a computer in order to improve the existing technologies. thus, automatic speech recognition (asr) has been an active field of ai research aiming to generate machines that communicate with people via speech @xcite . in recent trends, simple text based chatbot systems are adding extra flavor of personalized experiences to their users through speech interactions. however, emotions always play the important roles in our interactions with people and computers. fundamental publications of rosalind picard on affective computing increased the awareness in hci community regarding important roles of emotion @xcite [4] @xcite . since then, researchers have also become increasingly aware of the importance of emotion in the design process @xcite . speech emotion recognition (ser) is mostly beneficial for commercial hci applications, such as speech synthesis, customer service, education, forensics and medical analysis. emotion recognition is used in call center for classifying calls according to emotions @xcite and it serves as the performance parameter for conversational analysis @xcite , customer satisfaction and so on. ser is also used in automotive industry especially in car board system based on mental state of the driver to initiate his/her safety by preventing accidents to happen @xcite . affective computing and hci research used to target in reducing user frustration, building tools to support development of socioemotional skills @xcite . without information about emotions, it is difficult to achieve a harmonic and natural man-machine interface for applications such as patient care, geriatric nursing, call centers, psychological consultation, and human communication @xcite . therefore, health care industry is becoming prominent because it leverages emotion recognition techniques to solve complex patient related problems. speech is an information-rich signal that contains paralinguistic information as well as linguistic information. as a result of this, speech conveys more emotional information than text. this reality motivates many researchers to consider speech signal as a quick, effective and natural process to identify interaction mysteries between computer and human. although, there is a significant improvement in speech recognition but still researchers are away from natural interplay between computer and human, since computer is not capable of understanding human emotional state. the recognition of emotional speech aims to recognize the emotional condition of individual utterances by applying his/her voice automatically. recognizing of emotional conditions in speech signals are so challenging area for several reasons. majority of the speech emotional methods used to select the best features that are powerful enough to distinguish between different emotions. the presence of various languages, accents, sentences, speaking styles, speakers also adds another difficulty because these characteristics directly change most of the extracted features including pitch and energy @xcite . furthermore, it is possible to have a more than one specific emotion at a time in the same speech signal and each emotion may correlate with a different part of speech signals. therefore, defining the boundaries between parts of emotion is very challenging task. in the present task with respect to speech emotion recognition, we have proposed two systems based on deep learning method to classify a speech signal according to its emotional content. now apart from these two systems, we have also developed an emotional lexicon that contains utterances of words along with their emotional class. moreover, the lexicon also contains the utterances of a particular word when belongs to one or more emotion categories (same word can belong to multiple categories of emotion). finally, we introduce a deep recurrent encoder model that exploits text data and audio signals both simultaneously to obtain a better understanding of the emotional aspects in speech signals. in real world, a multi-modal dialogue system is composed of sound and spoken content. this actually motivated us to build a system which can encode the information from audio and text sequences and then can combine the information from these sources to predict the emotion class. our system reported accuracies ranging from 62.7% to 70.8% when it was applied to the iemocap dataset. the rest of the paper is organized as follows. section 2 describes the related attempts carried out under speech emotion recognition. the details on two types of emotional speech datasets along with two different models for speech emotion classification are discussed in section 3. section 4 describes a deep learning based multimodal framework that takes into account the roles of speech and text in order to develop an improved system. experiments and associated results with respect to all the models and framework are explained in section 5. finally, section 6 briefs the process of developing speech emotion lexicon as an outcome whereas the concluding remarks are made in section 7. ## related work if we observe a comprehensive review of speech emotion recognition systems targeting pattern recognition researchers who do not necessarily have a deep background in speech analysis, we notice three main aspects of this research field: (1) important design criteria of emotional speech corpora, (2) impact of speech features on the classification performance of ser and (3) classification systems employed in ser. l. chen et al. @xcite used multi-level svm classifier and ann to reduce dimensionality by employing several parmeters (e.g., energy, zcr, pitch, sc, spectrum cut-off frequency, correlation density (cd), fractal dimension, mff etc.) and obtained 86.5%, 68.5% and 50.2% recognition rates at different levels on beihang university database of emotional speech (bhudes). similarly, the authors in @xcite used binary classifier and qdc with prosodic and contour features to obtain 75.8% rate of recognition on semaine functional data. in recent trends, h. cao et al. @xcite used svm with prosodic and spectral features and obtained 44.4% recognition rate on berlin & ldc & fau aibo dataset. in addition to the above mentioned works, in @xcite , a novel modulation spectral features (msfs) for the recognition of human emotions in speech is presented. an auditory-inspired st representation is acquired by deploying an auditory filter bank as well as a modulation filter bank, to perform spectral decomposition in the conventional acoustic frequency domain and in the modulation frequency domain, respectively this authors in @xcite focused on the data preprocessing techniques which aim to extract the most effective acoustic features to improve the performance of the emotion recognition. the technique can be applied on a small sized data set with a high number of features. the presented algorithm integrates the advantages from a decision tree method and the random forest ensemble. experiment results on a series of chinese emotional speech data sets indicate that the presented algorithm can achieve improved results on emotional recognition, and outperform the commonly used principle component analysis (pca) / multi-dimensional scaling (mds) methods, and the more recently developed iso-map dimensionality reduction method. in @xcite , a fusion-based approach to emotion recognition of affective speech using multiple classifiers with acoustic-prosodic information (ap) and semantic labels (sls) is presented. the acoustic-prosodic information was adopted for emotion recognition using multiple classifiers and the mdt was used to select an appropriate classifier to output the recognition confidence. it is observed that all the above mentioned approaches are either tried to deal with signals, acoustic features or to use machine learning classifiers and feature reduction techniques to improve the performance of ser. in contrast, our proposed method is based on deep learning and applied on three different datasets to show the effectiveness. in addition, the multi-modal framework deals with both the texts and speech together to capture the insights under the deep learning umbrella. the development of speech emotion lexicon directs us the utilization of the proposed models. ## speech emotion recognition a single word can be associated with multiple emotions @xcite . based on this hypothesis, we have built our emotion classifier and chosen datasets carefully. although there are several other modalities such as facial expression, body language, through which emotions can be expressed but we limited our present study to speech modality only. speech emotion corpora that were prepared by actors have been used in the current study because the emotions expressed with exaggeration potentially compensate the lack of information provided by other modalities. this also allows us to explore the effectiveness of deep learning models with greater control compared with daily-life utterances. however, we limited our model to classify emotions for 'english' language only. ## multi-modal analysis recently, we agree that the deep learning algorithms have successfully addressed problems in various fields, such as image classification, machine translation, speech recognition, text-tospeech generation and other machine learning related areas @xcite [31] @xcite . similarly, substantial improvements in performance have been obtained when deep learning algorithms have been applied to statistical speech processing @xcite . even though various types of deep learning methods have been applied, this problem is still considered to be challenging for several reasons; first, the scarcity of emotion tagged data for training deep neural models and second, the characteristics of emotions must be learned from low-level speech signals. however, feature-based models display limited skills when applied to this problem. in order to overcome these limitations, we have developed a model (as shown in figure 3 ) that uses high-level text transcription, as well as lowlevel audio signals, to utilize the information contained within low-resource datasets to a greater degree. the emotional content of speech is clearly indicated by the emotion words contained in a sentence [29], such as "lovely" and "fantastic," which carry strong emotions compared to generic (non-emotion) words, such as "person" and "day." thus, we hypothesize that the speech emotion recognition model will be benefit from the incorporation of high-level textual input with the low-level audio features. moreover, this multimodal approach encodes both audio and textual information simultaneously via a dual recurrent encoder. ## experiments & results this section discusses the experiments performed in this study to classify emotion class and gender of the input audio data using the ffnn and cnn based deep learning models as described in the above sections. in order to have a comparative discussion, we restricted ourselves to 100 epochs for both the models. the datasets used to train both of these networks already have been discussed in the section 3.1. we encourage the readers to consult that section to have a detailed idea about the datasets. we have merged the audio files from the two datasets, savee and ravdess, to produce the raw data. there are approximately 1900 audio files after merging. however, we were not able to use all the audio files to train our networks as the emotion classes of the two datasets were not identical. the emotion classes reported in savee database are anger, disgust, fear, happiness, sadness, surprise and neutral whereas the emotion classes in ravdess database are neutral, calm, happy, sad, angry, fearful, disgust and surprised. as we know neutral emotion does not specifically portray any emotion specific feature, we discarded all the sentences belong to neutral class from our raw dataset. furthermore, we considered only 5 main classes of emotions namely, calm, happiness, sadness, fear and anger. as a result, we have approximately 1200 sentences in our raw database. in case of training and testing our models, we need to split our raw dataset, which is described in the previous section, to form the training and test data. we have taken approximately 80% of the raw dataset as training and the remaining 20% as test data to evaluate our models. the performances of the neural network models on this training and test set have been demonstrated in the following sections. ## speech emotion lexicon not only the classification of speeches into different emotion categories, but one of our partial objectives was also to explore the possibility of generating a speech emotional database consisting of emotional utterances also. therefore, we have developed a speech emotion lexicon containing utterances of different emotional categories. however, it is difficult to simultaneously generate both male and female voices in a text-to-speech system and thus, we limited ourselves to synthesize only 'male' voice in this present attempt. for this very reason, we selected our dataset with only male speakers ## conclusions in the present task, two classification models based on deep neural network, one using normal feed-forward neural network (ffnn) and another using convolutional neural network (cnn) architecture has been implemented and also a comparative study between these two models has been reported. among the two models it has been shown that cnn model outperformed the ffnn model. the models have been developed from a training set which consists of only english language. it will be an interesting study to apply other languages to train the model and compare the performances for the same. in addition to that, we have implemented a multi-modal version of the deep neural model using recurrent neural network, in order to improve the classifier system. we have used features of both audio and text and merged them into a single framework to investigate the effectiveness of the system. it is observed that the performance of the multi-modal system is far better that the ffnn or cnn based system in classifying emotions. as we have mentioned earlier that we have developed a database which contains emotional utterances. however, the size of the database is not very large, we had a limited number of test samples and hence it affected the development of the proposed database.