  Speech emotion recognition is a crucial problem manifesting in a multitude of
applications such as human computer interaction and education. Although several
advancements have been made in the recent years, especially with the advent of
Deep Neural Networks (DNN), most of the studies in the literature fail to
consider the semantic information in the speech signal. In this paper, we
propose a novel framework that can capture both the semantic and the
paralinguistic information in the signal. In particular, our framework is
comprised of a semantic feature extractor, that captures the semantic
information, and a paralinguistic feature extractor, that captures the
paralinguistic information. Both semantic and paraliguistic features are then
combined to a unified representation using a novel attention mechanism. The
unified feature vector is passed through a LSTM to capture the temporal
dynamics in the signal, before the final prediction. To validate the
effectiveness of our framework, we use the popular SEWA dataset of the AVEC
challenge series and compare with the three winning papers. Our model provides
state-of-the-art results in the valence and liking dimensions.
