Emotions in Spoken Language - Do we need acoustics? (2023)

Abstract: Work on emotion detection is often focused on textual data from i.e. Social Media. If multimodal data (i.e. speech) is analysed, the focus again is often placed on the transcription. This paper takes a closer look at how crucial acoustic information actually is for the recognition of emotions from multimodal data. To this end we use the IEMOCAP data, which is one of the larger data sets that provides transcriptions, audio recordings and manual emotion categorization. We build models for emotion classification using text-only, acoustics-only and combining both modalities in order to examine the influence of the various modalities on the final categorization. Our results indicate that using text-only models outperform acoustics-only models. But combining text-only and acoustic-only models improves the results. Additionally, we perform a qualitative analysis and find that a range of misclassifications are due to factors not related to the model, but to the data such as, recording quality, a challenging classification task and misclassifications that are unsurprising for humans.

## introduction the correct detection of emotions in spoken language is a task which has been examined a lot in recent years. but the majority of research so far, has treated text and speech separately and there is little research on using both audio and textual data for emotion recognition, which would stand to reason as emotions are expressed not only in what is being said, but also how. one exception is @xcite who present a model using multi-level multi-head fusion attention mechanism and recurrent neural network (rnn) for the detection of the audio input in combination with information gained from text. building on previous work, we show extensive experiments on various types of models trained on text as well as speech. we examine, whether the combination of emotion recognition from textual data as well as audio data improves the results to either singular approach. we also explore different feature sets and models to find the best combination. this allows us to conclude, that, while using only textual data yields reasonable results for english, we achieve best results when combining it with additional acoustic data. in a qualitative analysis we take a look at the most striking confusions between classes and give possible reasons for them. additionally, we give an idea of how to counter these problems in future experiments. in section 2 we describe previously conducted work on emotion recognition. for this, in section 2.1 we describe some of the more often used data sets available for this task. in section 2.2 we describe some related work in more detail. the experimental setup is described in section 3. it additionally focuses on the feature extraction for the audio data (section 3.1), the preprocessing of the textual data (section 3.2, as well as the model used for the transcriptions (section 3.3 and the method of combination for the different models (section 3.5. section 4 shows the best results of the combined models and three different data set variations. in section 5 we conduct a qualitative analysis of the results. finally, we conclude in section 6. our major contributions therefore are: ## previous work the detection of emotions in spoken language is a complex problem. emotions in humans are expressed both in the choice of words used, but also in the way these words are expressed acoustically. the related work on emotion detection based on transcribed speech, acoustical information and a combination thereof, described below is focused on relevant experiments related to our work and does not represent an exhaustive review of the topic. ## experimental setup for our experiments foot_7 we use the iemocap data set by @xcite . as, out of the above presented data sets (section 2.1), the iemocap data set is the largest and provides both textual and audio data in english. the emodb, casi, nnime and des also provide both textual and audio data, but they are either very small (i.e. emodb only has 500 sentences) or in chinese (casia, nnime), which is a tonal language, as opposed to english, which is a non-tonal language and therefore, expresses emotions differently. 9 the authors performed the emotion category annotation and report a fleiss' kappa of 0.27 on the entire annotation. our re-calculation of fleiss' kappa resulted in an observed agreement of 0.23, an expected agreement of 0.27 and an overall agreement of -0.06. 10 however, fleiss' kappa is not applicable to data sets with empty annotations, which is the case in this data set, as there are six annotators in total, but only three given annotations per example. thus, we use krippendorff's alpha to verify their results. this leads to an observed disagreement of 0.77, an expected disagreement of 0.73 and an agreement of -0.06, which confirms the low agreement between the annotators. as the original data set does not provide a gold standard, we use a majority vote between the three annotations. if all annotators decide on different labels, we use the label of the first annotator. another option would have been to randomly take one annotation, but by always deciding on the first annotator in these cases, it might lead to a slightly higher consistency. the classes "fear" and "neutral state" are excluded in variation 1. "fear" is an extremely small class, though not as small as "disgust". the class "neutral state" is also excluded. although it is the largest class, it shows no explicit emotion. variation 2 combines the classes "anger", "frustration" and "sadness", as they are all negative emotions. it contains the combined class of the happy emotions of "happiness" and "excited" as well as the class "neutral state". this class can be seen as 10 calculated using https://dkpro.github.io/ dkpro-statistics/ a sentiment classifications (positive, negative and neutral). previous work show that some authors group different classes together based on similar acoustic signals. for example, @xcite combined the classes "anger", "surprise" and "joy" in one cluster and the classes "fear", "sadness" and "disgust" in another one. this approach is based on findings by @xcite ), who found that emotions such as "anger" and fear" but also "joy" arouse the sympathetic nervous system, while emotions such as "sadness" arouse the parasympathetic nervous system. an aroused sympathetic nervous system leads to an accelerated heart rate and higher blood pressure, a dry mouth and even occasional muscle tremors. this shows in a loud, fast and enunciated speech with strong high frequency energy. in contrast, an aroused parasympathetic nervous system leads to lower blood pressure and heart rate, as well as an increased salivation. speech produced under these circumstances is slow with little high frequency energy. even though @xcite improved their results by grouping the emotion classes accordingly (accuracy reaches up to 90%), it is questionable whether it is useful to the actual use case to subsume such different classes. in our experiments, we combine emotions which are similar, such as "happiness" and "excited", or show the same sentiment ("anger", "frustration" and "sadness" are all negative). ## results to classify the results, we compare them to a majority baseline ( ## qualitative analysis in our qualitative analysis, we take a closer look at those examples that have been misclassified. in general, most of the examples which are miscategorized are very short. also, we observed that there are examples that are not correctly identified by neither of the three models and data set variations. a closer look, revealed they are spoken by a female, however, in some instances a male speaker can be heard clearly in the background. while the female seems to be very happy in those, the background noise of the male could have negatively impacted the recognition process to "anger" as he sounds quite angry. also, there was an example, which was predicted as "excited", even though it was meant to be "sad" as indicated by the manual annotation. this was interesting, as the person speaking sounded quite desperate, which shows more energy than the average sad person, which normally exhibits a rather low energy level. therefore, the wrong classification as "excited" makes sense and should be addressed in further experiments. additionally, we looked at some examples, which are wrongly classified in at least two of the three models. after a proper examination, there are two types of confusions which stand out in particular: negative emotions which are confused as happy ones. the aforementioned confusion of an example of a desperate person was not happening in just one example. additionally, there was an example of a frustrated male which was quite energetic. however, in the background a woman was starting to speak in a higher voice, but the recording was cut shortly thereafter, making it impossible to understand what the woman ways saying. her higher voice might have influenced the levels of the acoustic statistics we trained our model on. happy emotions which are confused as negative ones. those happy examples tend to present low energy, for example, a person states "i love you a great deal!". this is a very happy statement, however, as the person sounds very close to tears, the model predicts it as "frustration". another person is very calmly speaking, however, due to the calmness, the person appears to the model to be sad. both misclassifications do make sense as, as described by @xcite , speech representing sadness is slower and characterized by less high frequency energy. another noteworthy aspect is the length of the examples. while @xcite already concluded that it is more difficult to correctly recognise the emotion if the speech is too long, our findings suggest the same is true for examples which are too short. there are several very short examples on the data set (approx. 1s), which are incorrectly classified. while some of them do express emotions, it can be hard even for humans to correctly determine them without context. many of the wrongly predicted examples also could be categorized in various classes, which explains the low inter-annotator-agreement. ## conclusion in our work, we examined the benefit of combining information from text with information from speech in order to categorize emotions in spoken language. to this end, we used the iemocap data set, in its original classification, but also in different combinations, to train various machine learning models (svm and neural networks) to classify emotions. we combined the trained models in various ways to find the best combination of models for the classification. our results indicate that a combination of textbased features together with acoustic features provide the best results, as all combinations contained both models based on textual information, as well as acoustic information. a detailed look at the results reveals that neural networks trained on textual data perform best, when only one modality (text) is used. however, when multimodal data (text and speech) is available, making use of all modalities improves the classification and that textual data is crucial for a successful classification. our results also indicate, that using only acoustical data gives results that are even below those for text-only based classification. so yes, we need acoustics, but acoustics alone do not provide enough information for a successful classification. also, there is no clear indication, if opensmile or librosa features provide better results. but, we observed that the svms trained on various acoustic features perform much better than the neural networks trained on the same features. this is probably due to the relatively small data set size. in general, it seems that negative classes are easier to classify than positive emotions. as our results are only based on one data set, it would be worthwhile to explore this in more detail. additionally, we see a systematic problem with the correct classification of examples with unusual energy levels for the respective emotion. therefore, it might be an interesting approach for examples of the class "excited" to be divided into positive and negative excitedness. however, this would also need larger data sets, as the class "excited" is already quite small in the iemocap data set, which is already one of the larger ones available. to avoid the confusion of happy expression with unusually low energy levels, it might be important to add more examples like these to the training data. a first approach to do so could be achieved by simply oversampling happy examples with statistically low energy levels. in general, some of the examples of the data set are hard to understand even for humans. this is due to poor audio quality as well as background noise. sometimes we needed to look at the label in order to decipher which talking person was labelled, which also accounts for the low inter-annotator agreement we observed. it might be a useful approach to detect these examples and drop them before training a model, as this does only apply to a small part of the data set. broadly speaking, a combination of more audio features improves the results in comparison to single audio features. however, it should be further investigated, whether focusing on specific features might improve the results even more. a focus hereby should lay on the standard deviation of the features, as they reach the best results after the combination of all acoustic features. another interesting evaluation would be, to see if the model predicts one of the manually assigned labels, under the assumption that all human labels are correct. this would reduce the need for a singlevalue gold standard and would give us an insight into those cases, where the model clearly misclassifies the examples. overall, the findings suggest, that future research should explore, whether the combination of audio and textual data on one model improves the results even more, than the combination of three separate models. additionally, it should be further investigated whether there is an influence of the gender of the speaker on the correctness of the predictions as we did not look at it at all.