CMU - MOSEAS : A Multimodal Language Dataset for S panish, P ortuguese, G erman and F rench (2020)

Abstract: Modeling multimodal language is a core research area in natural language processing. While languages such as English have relatively large multimodal language resources, other widely spoken languages across the globe have few or no large-scale datasets in this area. This disproportionately affects native speakers of languages other than English. As a step towards building more equitable and inclusive multimodal systems, we introduce the first large-scale multimodal language dataset for Spanish, Portuguese, German and French. The proposed dataset, called CMU-MOSEAS (CMU Multimodal Opinion Sentiment, Emotions and Attributes), is the largest of its kind with 40,000 total labelled sentences. It covers a diverse set topics and speakers, and carries supervision of 20 labels including sentiment (and subjectivity), emotions, and attributes. Our evaluations on a state-of-the-art multimodal model demonstrates that CMU-MOSEAS enables further research for multilingual studies in multimodal language.

## introduction humans use a coordinated multimodal signal to communicate with each other. this communication signal is called multimodal language @xcite ; a complex temporal and idiosyncratic signal which includes the modalities of language, visual and acoustic. on a daily basis across the world, intentions and emotions are conveyed through joint utilization of these three modalities. while english, chinese, and spanish languages have resources for computational analysis of multimodal language (focusing on analysis of sentiment, subjectivity, or emotions @xcite @xcite @xcite ), other commonly spoken languages across the globe lag behind. as artificial intelligence (ai) increasingly blends into everyday life across the globe, there is a genuine need for intelligent entities capable of understanding multimodal language in different cultures. the lack of large-scale in-the-wild resources presents a substantial impediment to multilingual progress in this fundamental research area in nlp. in this paper, we introduce a large-scale dataset for 4 languages of spanish, portuguese, german and french. the dataset, called cmu-moseas (cmu multimodal opinion sentiment, emotions and attributes) contains 10, 000 annotated sentences from across a wide variety of speakers and topics. the dataset also contains a large subset of unlabeled samples across the 4 languages to enable unsupervised pretraining of multimodal representations. figure 1 shows an example sentence from cmu-moseas dataset along with the provided multimodal features and annotations. annotations include sentiment, subjectivity, emotions, and attributes. we believe that data of this scale presents a step towards learning human communication at a more fine-grained level, with the longterm goal of building more equitable and inclusive nlp systems. in the continuation of this paper, we first discuss the related resources and previous works. subsequently, we outline the dataset creation steps, including the data acquisition, verification, and annotations. we also discuss the steps taken to protect the speakers and uphold the ethical standards of the scientific community. finally, we experiment with a state-of-the-art multimodal language model, and demonstrate that cmu-moseas presents new challenges to the nlp community. ## background the related work to the content of this paper is split in two parts. we first discuss the related datasets, alongside comparisons with cmu-moseas. afterwards, we discuss the machine learning literature for modeling multimodal language. ## cmu-moseas (cmu multimodal opinion sentiment, emotions and attributes) dataset the cmu-moseas dataset covers 4 languages of spanish (>500m total speakers globally), portuguese (>200m speakers globally), german (>200m speakers globally), and french (>200m speakers globally). these languages either have romance or germanic roots @xcite . they originate from europe, which is also the main region for our video acquisition. the languages are also spoken in the american continent (north and south), as well as portions of africa and the caribbean (with different dialects, however, the european dialect is mostly comprehensible across different regions with some exceptions 1 ). subsequently, in this section, we discuss the data acquisition and verification process, followed by outlining the annotated labels. we prioritize important details in the body of the main paper, and refer the reader to supplementary material for extra details about the dataset. ## experimental baselines in this section we establish baselines for cmu-moseas dataset. we choose a state of the art transformer-based neural model for this purpose. the model has shown state-of-the-art performance across several multimodal language tasks including multimodal sentiment analysis and emotion recognition. the cmu-moseas dataset is split in the folds of train, validation and test (available on the cmu multimodal sdk). what follows is a brief description of the baseline model. multimodal transformer (mult): multimodal transformer @xcite is an extension of the well-known transformer model @xcite to multimodal time-series data. each modality has a separate transformer encoding the information hierarchically. the key component of mult is a set of cross-modal attention blocks that cross-attend between time-series data from two modalities. mult is among state-of-the-art models on both aligned and unaligned versions of the cmu-mosei and cmu-mosi datasets. we use the author provided code for these experiments foot_0 , with learning rate of 10e -4 and the adam optimizer @xcite . the transformer hidden unit size is 40 with 4 cross-modal blocks and 10 attention heads. dropout is universally set at 0.1. the best model is chosen using the validation set of each language. we use the aligned variant of mult. for each language, we perform word-level alignment to acquire the expectation of visual and acoustic contexts per word @xcite , identical to the methodology used by mult (aligned variant). the maximum sequence length is set at 50. sequences are padded on the left with zeros. for language, we use the one-hot representation of the words. for acoustic, we concatenate covarep and opensmile features. the experiments are performed tri-label for sentiment (negative, neutral, positive) and binary for emotions and attributes; similar methodology is employed by mult. the above models are trained to minimize mean-absolute error (mae). the metric used to evaluate model performance is the f1 measure, which is a more suitable metric when there are imbalanced classes as is the case for some labels in our dataset (i.e. rare attributes). for extra details of experiments, as well as other results including mae and correlation, please refer to the github. ## conclusion in this paper, we introduced a new large-scale inthe-wild dataset of multimodal language, called cmu-moseas (cmu multimodal opinion sentiment, emotions and attributes). the cmu-moseas dataset is the largest of its kind in all four constituent languages (french, german, portuguese, and spanish) with 40,000 total samples spanning 1,645 speakers and 250 topics. cmu-moseas contains 20 annotated labels including sentiment (and subjectivity), emotions, and per-sonality traits. the dataset and accompanied descriptors will be made publicly available, and regularly updated with new feature descriptors as multimodal learning advances. to protect the privacy of the speakers, the released descriptors will not carry invertible information, and no video or audio can be reconstructed based on the extracted features. a state-of-the-art model was trained to establish strong baselines for future studies. we believe that data of this scale presents a step towards learning human communication at a more fine-grained level, with the long-term goal of building more equitable and inclusive nlp systems across multiple languages.