Importance-based Multimodal Autoencoder
Authors: Sayan Ghosh, Eugene Laksana, Louis-Philippe Morency, Stefan Scherer
Keywords: Neural networks, Speech analysis, Multimodal, Autoencoders, Representation Learning
Abstract: Integrating information from multiple modalities (e.g.,  verbal,  acoustic and visual data) into meaningful representations has seen great progress in recent years. However, two challenges are not sufficiently addressed by current approaches: (1) computationally efficient training of multimodal autoencoder networks which are robust in the absence of modalities, and (2) unsupervised learning of important subspaces in each modality which are correlated with other modalities.   In this paper we propose the IMA (Importance-based Multimodal Autoencoder) model, a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function.  We conduct experiments on MNIST-TIDIGITS a multimodal dataset of spoken and image digits,and on IEMOCAP, a multimodal emotion corpus. The IMA model is able to distinguish  digits  from  uncorrelated  noise,  and  word-level  importances  learnt  that correspond to the separation between function and emotional words.  The multimodal representations learnt by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks.