Dual Low-Rank Multimodal Fusion (2020)

Abstract: Tensor-based fusion methods have been proven effective in multimodal fusion tasks. However, existing tensor-based methods make a poor use of the fine-grained temporal dynamics of multimodal sequential features. Motivated by this observation, this paper proposes a novel multimodal fusion method called Fine-Grained Temporal Low-Rank Multimodal Fusion (FT-LMF). FT-LMF correlates the features of individual time steps between multiple modalities, while it involves multiplications of high-order tensors in its calculation. This paper further proposes Dual Low-Rank Multimodal Fusion (Dual-LMF) to reduce the computational complexity of FT-LMF through low-rank tensor approximation along dual dimensions of input features. Dual-LMF is conceptually simple and practically effective and efficient. Empirical studies on benchmark multimodal analysis tasks show that our proposed methods outperform the state-of-the-art tensor-based fusion methods with a similar computational complexity.

## introduction multimodal fusion aims to integrate information of multiple modalities as a compact but informative representation. multimodal fusion is fundamentally significant for real-world multimodal applications like speech translation @xcite , emotion recognition @xcite , and sentiment analysis @xcite . it is very challenging that it requires correlating the semantics of multiple modalities in an effective and efficient way. recently, several methods have been proposed to learn joint embeddings of multiple modalities @xcite @xcite . there are two lines of fusion methods: early fusion and late fusion. in this paper, we mainly focus on the former, which aims to integrate information of different modalities before it is processed by the model. earlier work on early fusion employs a simple concatenation of input features @xcite @xcite . to construct a more compact representation, @xcite introduces tensor fusion network (tfn) which averages the features of each modality along the temporal dimension and transforms the multimodal features into a high-order tensor which is used for subsequent tasks. although tfn achieves a better performance than the concatenation manner, its computational complexity increases exponentially with the number of modalities. @xcite further proposes low-rank multimodal fusion (lmf) which employs lowrank approximation to reconstruct the high-order tensor. however, these tensor-based methods neglect the fine-grained temporal dynamics which include rich structured information for multimodal modeling. for example, if the facial expression of a man is happy at time step t, he will speak in a positive tone at time step t + ∆t more likely. the features of different time steps and different modalities are correlated. motivated by this observation, in this paper we introduce fine-grained temporal low-rank multimodal fusion (ft-lmf). instead of averaging the features along the temporal dimension, we associate the features of individual time steps between different modalities to form a high-order tensor. the tensor is then embedded to a low-dimensional matrix for subsequent tasks. compared with lmf, ft-lmf is able to capture the cross-modal interactions at a finer granularity on the temporal dimension. since ft-lmf involves multiplications of high-order tensors in its calculation, its computational complexity increases exponentially with the number of modalities. to tackle this problem, we further introduce dual low-rank multimodal fusion (dual-lmf) which approximates the high-order tensor using low-rank tensor decomposition along both temporal and non-temporal dimensions. we show that dual-lmf has a linear complexity w.r.t the number of modalities. in experiments, we have validated ft-lmf and dual-lmf on four benchmark multimodal analysis datasets and they have shown promising results in comparison with the state-of-the-art methods. the contributions of this paper can be summarized as follows: ## related work multimodal analysis has attracted much attention recently. thanks to the high-quality open-source datasets like cmu-mosi, pom, youtube, and ict-mmmo, many effective methods have been proposed and comprehensively evaluated. the key to multimodal analysis is the fusion of multimodal information. generally, there are two lines of fusion methods, early fusion and late fusion. early fusion methods integrate features of different modalities before feeding them to the model. for instance, concatenating different features @xcite is a simple way. however, the intramodal dynamics cannot be effectively captured, and the temporal information of a single modality is ignored in early fusion. late fusion methods @xcite utilize information of a single modality for inference, and then ensembling them by majority voting or weighted averaging @xcite . unfortunately, the inter-modal interactions are not modeled in late fusion. to address the drawbacks of the above methods, @xcite investigates learning joint representations via cyclic translations from source to target modalities and only uses the source modality for prediction during testing. tfn @xcite and its successive work @xcite propose to embed multiple feature vectors into a high-order tensor to improve the modeling of intermodal relationships. however, the computational complexity of tfn increases exponentially with the number of modalities. lmf @xcite reduces the complexity of tfn by applying lowrank decomposition to the high-order tensor. while lmf simply averages the feature matrices along the temporal dimension or chooses a feature vector of one time step among all the time steps, ignoring the rich fine-grained temporal information. in this paper, we develop fine-grained temporal low-rank multimodal fusion (ft-lmf) to correlate the features of different time steps between modalities. furthermore, considering that the computational complexity of ft-lmf increases exponentially with the number of modalities, we propose dual low-rank multimodal fusion (dual-lmf) which performs low-rank decomposition to both dimensions of the input features. the performances of our methods on several tasks, i.e., multimodal sentiment analysis and speaker traits recognition, are improved with an acceptable complexity. ## fine-grained temporal lmf the existing multimodal tensor-based fusion methods correlate multimodal features at a coarse granularity, while the rich temporal dynamic information underlying in different modalities is ignored. in this work, we propose fine-grained temporal low-rank multimodal fusion (ft-lmf) for making a full use of the fine-grained information along the temporal dimension. based on the discussions in previous section, we can easily correlate the features of different the space of an output of lmf is r d h , thus the space of all the groups is r d h ×tv×ta×t l . a fully-connected layer follows the high-order tensor to reduce the tensor space from r d h ×tv×ta×t l to r d h ×d k . in practice, the parameters w k on fc layer are generated by attention mechanism for better effectiveness. to denote the correlation result of selected timesteps of m modalities. we can obtain a high-order tensor @xmath0. following eqn. 2, we calculate the values of tensor as: the numerator is elementwise divided by the denominator. ft-lmf shown in fig. 3 is able to capture the fine-grained temporal interactions between different modalities, while the computational complexity of its high-order tensor h increases exponentially with the number of modalities. to tackle this problem, we further propose dual-lmf as discussed in the next section. ## dual-lmf based on ft-lmf, dual-lmf further performs low-rank decomposition on both temporal dimension and non-temporal dimensions. first, we follow lmf to divide the (m + 1)-order tensor w k into a series of m -order tensors. the number of the tensors is d k : ## experiments 6.1 datasets we evaluate ft-lmf and dual-lmf on several benchmark datasets of multimodal analysis tasks, including cmu-mosi @xcite , pom @xcite , youtube @xcite , and ict-mmmo @xcite . cmu-mosi @xcite is created for sentiment analysis, which contains 63 long videos with a sentiment label in range @xcite . during the training and testing, we divide the 63 videos into 2199 chunks for label alignment. following the existing work, we divide the whole dataset into three parts, for training, validation, and testing. note that the same speaker does not appear in multiple sets. pom @xcite is created for speaker traits recognition. it contains 903 movie review videos and each video is annotated with 16 speaker traits, including confident, passionate, voice pleasant, dominant, credible, vivid, expertise, entertaining, reserved, trusting, relaxed, outgoing, thorough, nervous, persuasive and humorous. youtube @xcite is created for sentiment analysis. it contains 47 videos from the social media website youtube and each video is annotated at the segment level for sentiment. ict-mmmo @xcite is created for sentiment analysis. it contains 370 movie review videos and each video is annotated at the video level for sentiment. ## conclusion in this paper, we have proposed novel multimodal fusion methods, including ft-lmf and dual-lmf, for multimodal analysis tasks. ft-lmf is a finegrained version of low-rank multimodal fusion which particularly associates the features of individual time steps between multiple modalities. based on ft-lmf, dual-lmf performs low-rank tensor approximation along dual dimensions of input features to reduce the exponential computational com-plexity of ft-lmf to a linear complexity w.r.t. the number of modalities. the experimental results show that our methods achieve superior performance compared with the state-of-the-art methods with similar computational cost. amir zadeh, rowan zellers, eli pincus, and louis-philippe morency. 2016a. mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arxiv preprint arxiv:1606.06259. amir zadeh, rowan zellers, eli pincus, and louis-philippe morency. 2016b. multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. ieee intelligent systems. we implement experiments on gtx 1080ti. the main hyperparameters include audio hidden dimension, video hidden dimension, text hidden dimension, audio dropout rate, video dropout rate, text dropout rate, learning rate, weight decay, rank1 and rank2. grid search is employed to find the appropriate combination of parameters. for each method, we randomly try 2000 combinations, since the model is small and the running time is short as shown in w i h • ṽ can be rewritten as follows: the equivalence can be proven as follows: following the simple transformation like (a + b)(c + d) = ac + ad + bc + bd, we can these two formations are equal, just with different operation orders. the former utilizes summation( ) first, while the later uses multiplication( ) between different elements first. therefore, we obtain the final formation of w i h ṽ :