Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition (2020)

Abstract: Despite the recent achievements made in the multi-modal emotion recognition task, two problems still exist and have not been well investigated: 1) the relationship between different emotion categories are not utilized, which leads to sub-optimal performance; and 2) current models fail to cope well with low-resource emotions, especially for unseen emotions. In this paper, we propose a modality-transferable model with emotion embeddings to tackle the aforementioned issues. We use pre-trained word embeddings to represent emotion categories for textual data. Then, two mapping functions are learned to transfer these embeddings into visual and acoustic spaces. For each modality, the model calculates the representation distance between the input sequence and target emotions and makes predictions based on the distances. By doing so, our model can directly adapt to the unseen emotions in any modality since we have their pre-trained embeddings and modality mapping functions. Experiments show that our model achieves state-of-the-art performance on most of the emotion categories. Besides, our model also outperforms existing baselines in the zero-shot and few-shot scenarios for unseen emotions.

## introduction multi-modal emotion recognition is an increasingly popular but challenging task. one main challenge is that labelled data is difficult to come by as humans find it time-consuming to discern emotion categories from either speech or video. indeed we humans express emotions through a combination of modalities, including the way we speak, the words we use, facial expressions and sometimes gestures. it is also much more comfortable for humans to understand each other's emotions when they can both 1 code is available at https://github.com/ wenliangdai/modality-transferable-mer in the upper image, the relative positions of glove emotion embeddings (happy, surprised) are shown in the textual space, which are then projected to acoustic and visual spaces by two mapping functions (f @xmath0. our model learns to group the representations of input sentences ( , ) based on their corresponding emotion embeddings. examples are shown in the lower image. when a sample has both happy and surprised emotions, its representation gets close to these two emotion embeddings in all three spaces. if an unseen emotion sad ( ) comes, the model processes it with the same flow and recognizes corresponding data samples. in the past few years, much research has been done to better understand intra-modality and intermodality dynamics, and modality fusion is a widely studied approach. for example, @xcite proposed a tensor fusion network that combines three modalities from vectors to a tensor using the cartesian product. in addition, the attention mechanism is commonly used to do modality fusion @xcite @xcite @xcite . although significant improvements have been made on the multi-modal emotion recognition task, however, the relationship between emotions has not been well modelled, which can lead to sub-optimal performance. also, the problem of low-resource multi-modal emotion recognition is not adequately studied. multi-modal emotion recognition data is hard to collect and annotate, especially for low-resource emotions (e.g., surprise) that are rarely seen in daily life, which motivates us to investigate this problem. in this paper, we propose a modality-transferable network with cross-modality emotion embeddings to model the relationship between emotions. given that emotion embeddings contain semantic information and emotion relations in the vector space, we use them to represent emotion categories and measure the similarity of the representations between the input sentence and target emotions to make predictions. concretely, for the textual modality, we use the pre-trained glove @xcite embeddings of emotion words as the emotion embeddings. as there are no pre-trained emotion embeddings for the visual and acoustic modalities, the model learns two mapping functions, f @xmath1, to transfer the emotion embeddings from the textual space to the visual and acoustic spaces (figure 1 ). therefore, for each modality, there will be a dedicated set of emotion embeddings. the distances computed in all modalities will be finally fused, and the model will make predictions based on that. benefiting from this prediction mechanism, our model can easily carry out zero-shot learning (zsl) to identify unseen emotion categories using the embeddings from unseen emotions. the intuition behind it is that the pre-trained and projected emotion embeddings form a semantic knowledge space, which is shared by both the seen and unseen classes. furthermore, with the help of embedding mapping functions, the model can also perform zsl on a single modality during inference time. when a few samples from unseen emotions are available, our model can adapt to new emotions without forgetting the previous emotions by using joint training and continual learning @xcite . our contributions in this work are three-fold: ## methodology as shown in figure 2 , our model consists of three parts: intra-modal encoder networks, emotion em-bedding mapping modules, and an inter-modal fusion module. in this section, we first define the problem, and then we introduce the details of our model. ## unseen emotion prediction collecting numerous training samples for a new emotion, especially for a low-resource emotion, is expensive and time-consuming. therefore, in this section, we concentrate on the ability of our model to generalize to an unseen target emotion by considering the scenario where we have zero or only a few training samples in an unseen emotion. ## experiments in this section, we first introduce the two public datasets we use and data feature extraction. then, we discuss our evaluation metrics, including their advantages and defects. finally, we introduce the baselines and our experimental settings. ## conclusion in this paper, we introduce a modality-transferable model that leverages cross-modality emotion embeddings for multi-modal emotion recognition. it makes predictions by measuring the distances between input data and target emotion categories, which is especially effective for a multi-label problem. the model also learns two mapping functions to transfer pre-trained textual emotion embeddings to acoustic and visual spaces. the empirical results demonstrate that it exhibits state-of-the-art performance on most of the categories. enabled by the utilization of emotion embeddings, our model can carry out zero-shot learning for unseen emotion categories and can quickly adapt few-shot learning without downgrading trained categories.