Convolutional and Recurrent Neural Networks for Spoken Emotion Recognition (2020)

Abstract: We test four models proposed in the speech emotion recognition (SER) literature on 15 public and academic licensed datasets in speaker-independent cross-validation. Results indicate differences in the performance of the models which is partly dependent on the dataset and features used. We also show that a standard utterance-level feature set still performs competitively with neural models on some datasets. This work serves as a starting point for future model comparisons, in addition to open-sourcing the testing code.

## introduction speech emotion recognition (ser) is the analysis of speech to predict the emotional state of the speaker, for which there are many current and potential applications @xcite . as speech-enabled devices become more prevalent, the need for reliable and robust ser increases, and also the need for comparability of results on common datasets. while there has been a large amount of research in this field, a lot of results come from testing only on one or two datasets, which may or may not be publicly available. additionally, different methodologies are often used, reducing direct comparability of results. given the wide variety of neural architectures and testing methodologies, there is need for a common testing framework to help comparisons. this study aims to test some ser models proposed in the literature on a discrete emotion classification task, and promote reproducibility of results by using public and academic licensed datasets. in addition, the code is publicly hosted on github 1 under an open source license, so that our results may be verified and built upon. our work has two 1 https://github.com/broad-ai-lab/ emotion main benefits. first, it serves as a baseline reference for future research that uses datasets present in this study. second, it allows for comparisons between datasets to see which of their properties may influence classification performance of different models. the paper is structured as follows. in section 2 related work is given, and in section 3 we list the datasets used in this study. the tested methods are outlined in section 4, and the results given in section 5. we briefly discuss these results in section 6 and a conclude in section 7. ## related work there has been some previous work in comparing ser techniques on a number of datasets. in @xcite , schuller et al. compare a hidden markov model/gaussian mixture model (hmm/gmm) and a svm classifier for emotion class, arousal and valence prediction on nine datasets. for hmm/gmm, 12 mfcc, log-frameenergy, speed and acceleration features, are extracted per frame. for svm, 6552 features are extracted based on 39 statistical functionals of 56 low-level descriptors (llds). testing was done in a leave-one-speaker-out (loso) or leaveone-speaker-group-out (losgo) cross-validation setup. the only three datasets in common with the present study are emo-db, enterface and smartkom, for which unweighted average recall (uar) of 84.6%, 72.5%, and 23.5% were achieved, respectively. we use a similar methodology in the present paper. the schuller et al. work is expanded in @xcite , where multi-layer stacks of restricted boltzmann machines (rbms) are pre-trained in an unsupervised manner, then fine-tuned using backpropagation as a feed-forward neural network. the same datasets and configurations are used ## datasets fifteen datasets are used in this study, some of which are open datasets, while others require a signed eula to access. all of the datasets have a set of categorical emotional labels. a question arises when using acted datasets with additional annotations, such as crema-d, as to whether to use the actor's intended emotion as 'ground truth' for training a classifier or instead use a consensus of annotators with majority vote. for msp-improv and iemocap, the label assigned by annotators is used, consistent with previous work. for crema-d we have opted to use the actors intended emotion, rather than any annotator assigned labels. a table describing the emotion distribution in each dataset is given in ## results a table of results is given in ## discussion from the results we see that the models using raw audio as input tend to perform worse than those using spectrogram input. there are also cases, such as on the portuguese dataset, where the zhang model performs the best of the four, and such as on the jl corpus, where the raw audio models are better than the fixed-size spectrogram models but worse than the variable length log-mel models. there are many possible reasons for this, and due to time constraints, more thorough investigation was not able to be done. one reason is likely the lack of hyperparameter tuning. hyperparameters like number of training epochs, learning rate, batch size, and model specific hyperparameters such as the number of convolution kernels or number of lstm units, can have a moderate effect on the performance of each model. these would need to be optimised per-dataset using cross-validation, before testing. another possible reason is the tendency for models to overfit. we found that the raw audio models were overfitting quite badly and achieving worse performance on the test set as a result, even though they have a moderate number of parameters. regularisation techniques can help with this, such as dropout and regularisation loss, along with batch normalisation. finally, while we tried to make our models as similar as possible to the original papers, there are likely implementation differences that negatively influence the performance of our models. the design of the zhang model was for discrete arousal/valence prediction, and it is likely that a slightly modified architecture would better suit categorical emotion prediction. the other models were also tested with slightly different methodologies from ours, which would influence difference in reported results. we also see a dependence on both dataset and features used. the aldeneh model with 240 logmels tended to be better than with only 40 logmels, but also better than a fixed size 240 mel-band spectrogram, but this was dependent on dataset. it's possible that the zero-padding and -60db clipping of the spectrograms negatively impacted the performance. the zhao model performs best out of the four on the smartkom dataset, achieving a uar better than chance level, but still worse than the svm with is09 features. it's possible that in this instance the separate lstm and convolutional branches have a greater effect. unfortunately we were not able to test all combinations of spectrogram features with the zhao model. in future we aim to complete this, as well as compare using spectrograms with different frame size and clipping parameters. finally, the time taken to train these models is quite long due to using full cross-validation. an argument can be made for predefined training/validation/test sets of larger datasets, but these are often created ad hoc and can vary between studies, so collective agreement would be needed for using these as a common standard. ## conclusion in this paper we have presented an evaluation of different neural network models proposed for emotion recognition, and compared their performance for discrete emotion classification on 15 publicly available and academic datasets. we used a consistent methodology across all datasets, and have kept hyperparameters very similar across the proposed models. the results show differences in the performance of the models which sometimes depends on the evaluated dataset. we also showed that the models requiring raw audio input tended to perform worse than the ones requiring spectrogram input, however more testing is required, with hyperparameter tuning and regularisation techniques, to determine the cause of this performance difference. in general, our work serves as a baseline for comparison for future research. in future, we aim to additionally test models using utterance level features as input, and compare with non-neural network models such as svm and random forests. we also aim to test feature generation methods such as bag-of-audio-words and unsupervised representation learning.