Adaptive Fusion Techniques for Multimodal Data (2021)

Abstract: Effective fusion of data from multiple modalities, such as video, speech, and text, is challenging due to the heterogeneous nature of multimodal data. In this paper, we propose adaptive fusion techniques that aim to model context from different modalities effectively. Instead of defining a deterministic fusion operation, such as concatenation, for the network, we let the network decide “how” to combine a given set of multimodal features more effectively. We propose two networks: 1) Auto-Fusion, which learns to compress information from different modalities while preserving the context, and 2) GAN-Fusion, which regularizes the learned latent space given context from complementing modalities. A quantitative evaluation on the tasks of multimodal machine translation and emotion recognition suggests that our lightweight, adaptive networks can better model context from other modalities than existing methods, many of which employ massive transformer-based networks.

## introduction multimodal deep learning is an active field of research, where for a single event, one has information across multiple modalities, such as video, speech, and text. human brains can easily and perpetually perceive the context of an event from such heterogeneous data; however, it is not a trivial task for a computer system. in order for the machine to gain a contextual understanding, heterogeneous inputs must be combined first. combining, or more precisely, fusing multimodal inputs is, thus, a vital step for any multimodal task. naturally, a better fusion method will help a multimodal system learn better, ultimately enhancing its performance for a given task. the most common fusion technique used in the literature involves the concatenation of representations from all the available modalities. however, this results in a shallow network @xcite , and the network focuses more on learning intra-modal features, ignoring inter-modal dynamics altogether. later, @xcite proposed tensor fusion network (tfn), which models the unimodal, bimodal, and trimodal interactions using a 3-fold cartesian product. tfn performs better than simple concatenation; however, it imposes high computational requirements since it projects all the information from input modalities to a dense 3-d space as-is, without any prior information extraction. the computational overhead grows exponentially with respect to the dimensionality of unimodal features. @xcite proposed a low-rank multimodal fusion technique (lmf) to address the previous problem. such fusion techniques are useful but often result in a complex architecture. moreover, the fusion methods mentioned above focus only on combining individual unimodal features rather than combining and extracting useful information simultaneously. this means that the final predictor module (decoder in a seq2seq network @xcite , for example) bears an additional responsibility of identifying useful signals to focus on. this paper addresses these issues by proposing adaptive fusion techniques that allow the model to decide "how" to combine multimodal data more effectively for an event. the first technique, auto-fusion, learns to compress multimodal information while preserving as much meaning as possible. the second technique, gan-fusion, employs an adversarial network that regularizes the learned latent space for a given target modality complying with the information presented by complementary modalities. since our models are generic, the need to specify a pre-determined fusion operation such as cartesian product is alleviated, and this further incentivizes the network to model multimodal interactions by itself. moreover, our techniques are lightweight relative to the existing heavier counterparts @xcite , thereby preventing unnecessary computational load. we evaluate our models on three benchmark datasets: a quantitative evaluation shows that our models outperform the existing state-of-the-art methods in terms of bleu scores @xcite for machine translation and precision, recall, and f1-score for emotion recognition. our ablation studies also indicate that the learned multimodal representations are robust; they perform reasonably well even after removing information from a target modality. we now summarize our main contributions as follows: the rest of the paper is structured as follows: section 2 covers relevant work, section 3 discusses the proposed methodologies and overall architecture, section 4 describes the experimental setup, section 5 shows results, and section 6 contains our concluding remarks. ## related work in this section, we briefly review previous work related to our task. most earlier works in multimodal deep learning focus on traditional shallow classifiers such as support vector machines @xcite and naive bayes classifiers @xcite to exploit bimodal data. inspired by the success of deep learning over the last decade across multiple tasks, @xcite train end-to-end deep graph neural networks to reconstruct missing modalities during inference. they demonstrate that better features for one modality can be learned if relevant data from different modalities is available at training time; however, they employ simple concatenation for fusion. hence, the joint representation learned is shallow and is not guaranteed to model intermodal interactions. their findings were later verified by @xcite , who use a deep boltzmann machine @xcite to generate data from the image and text modality. @xcite construct a multilingual common semantic space to achieve better machine translation performance by extending correlation networks @xcite . they use multiple non-linear transformations to reconstruct sentences from one language to another repeatedly and finally build a common semantic space for all the different languages. to address the shallowness exhibited by some earlier fusion methods, techniques such as tfn @xcite , lmf @xcite and t2fn @xcite , were proposed that aim to capture both intra-and inter-modal dynamics simultaneously; however, the problem of effectively modelling context in multimodal samples remains unsolved. more recently, multimodal transformer (mult) @xcite was proposed to align data from different modalities implicitly. on a high-level, mult leverages cross-modal attention modules for each modality, each of which is responsible for aligning (or attending to) the target modality vector with the complementary modalities'. it also imposes substantial computational overhead due to the use of transformer networks @xcite . our methods, as discussed in detail in section 3, use much simpler components. for instance, we use at most one attention module compared to multiple self-attention heads in a transformer. variational mixture-of-experts autoencoders @xcite , a class of deep generative multimodal frameworks, were employed to learn a synergic shared representation for multiple modalities; however, scaling of such a model for all the modalities (video, speech, and text) simultaneously and for a more complex task as multimodal machine translaassuming that z d1 m1 , z d2 m2 , and z d3 m3 represent the video, speech, and text latent vectors respectively, we first concatenate them to obtain z k m . it is then passed through t which outputs the "autofused" vector z t m . we then obtain the reconstructed concatenated vector ẑk m by passing the autofused vector through f c , another transformation layer. finally, we optimize the loss between ẑk m and z k m . (b) gan-fusion module for the text modality: assuming that z s , z v , and z t are the latent speech, video, and text vectors, respectively, we first autofuse z s and z v to give z tr . simultaneously, we pass z t through the generator g, along with some noise, to get z g . the generator loss tries to match z tr and z g and discriminator d tries to distinguish between z tr and z g , the two sources of input. note: ## proposed methods this section will discuss the proposed methodologies for effectively fusing inputs from multiple modalities and describe the overall architecture of our models for classification and generation. most fusion techniques proposed in the literature, such as concatenation, and tfn, involve a deterministic operation for constructing the joint multimodal representation. for instance, in tfn, the 3-fold cartesian product of unimodal features is used for prediction. the method focuses more on learning rich unimodal features. however, there is no such "learning" procedure for joint representation; they are simply constructed by combining unimodal features in a specific fashion (here, by cartesian product.) in this paper, we will refer to such techniques as static fusion techniques. since there is no particular learning procedure for the joint representation, it becomes challenging for the final predictor module to model the complex dynamics of multimodal features. in other words, the model is unable to utilize multimodal information effectively. on the other hand, fusion methods such as lmf and mult are adaptive because they involve a cog-nitive feature processing step to construct the joint representation. in lmf, it is the decomposition module, and in mult, it is the final feed-forward fusion mechanism. we refer the reader to @xcite and @xcite for more detailed explanation of the models. our fusion methods involve the concatenation of unimodal embeddings as an initial step. to avoid any conflicts with past works, we will only consider steps after concatenation as a part of our fusion method because we do not use the concatenated vector for final prediction; it is only a preliminary step. therefore, in order to mitigate the "staticness" of existing fusion methods, we propose two adaptive yet simple fusion techniques, auto-fusion and gan-fusion. they aim to effectively combine multimodal inputs and mitigate the problem of shallowness and computational overhead exhibited by prior fusion techniques. ## experimental setup we measure our models' effectiveness on two tasks: 1) multimodal machine translation and 2) multimodal emotion recognition. the subsequent sections describe our complete experimental setup, including datasets and baselines used, implementation details, and evaluation metrics. ## results quantitative analysis: results of our experiments on the how2, multi30k and iemocap dataset are shown in tables 1, 2 and 3, respectively. for speech emotion recognition, we observe that our models consistently perform well across all the evaluation metrics. for the relatively challenging multimodal machine translation task, we observe that our model outperforms all existing baselines in terms of bleu scores. compared against the best performing baselines, gan-fusion improves bleu4 scores by 3.63 points and 0.13 points on the how2 and multi30k datasets, respectively. this shows that the fusion module was better able to extract signals from all modalities. on the multi30k dataset, our model is competitive in terms of meteor. such performance becomes more pronounced considering our models have only one attention module, in contrast to multiple self-attention heads in the transformer-based baselines. the size of our best-performing models (employing gan-fusion) is roughly 6m and 11m for classification and generation, respectively, which is significantly lower than traditional transformer models. fewer trainable parameters reduces the computational cost and the training time. this also indicates potential by-passing of a mechanism like distillation, which is used to reduce parameters in transformers @xcite ; however, more thorough experimentation is required to reach a concrete conclusion. our models can also be used in conjunction with transformers, where we utilize the transformers to learn meaningful unimodal feature vectors initially and then employ the proposed fusion methods. we also perform a comprehensive set of qualitative experiments on the how2 dataset to understand the capability of our fusion techniques. they are described as follows: effect of introducing more modalities: to understand the effect of introducing new modalities separately, we perform experiments with different combinations of source modalities, including individual unimodal baselines (refer to robustness of multimodal features: it is very important for the learned multimodal latent features to be robust, i.e., they should be able to exploit signals from complementary modalities to compensate for the presence of noise in one modality. therefore, to gauge the robustness of learned multimodal features, we conduct an ablation test on the how2 dataset. we randomly replace some tokens in the test sentence with @xmath0, gan-fusion. figure 6 shows results of our ablation study. we can see that features from the complementary modalities are able to compensate for ∼ 30% of the missing text as we see a sharp drop in bleu scores beyond that point. this shows that the model does not rely on just the textual description for translation; it also tries to gain a contextual understanding. hence, it follows that the learned joint representation indeed contains rich information from other modalities. ## conclusion and future work in this paper, we propose two adaptive fusion techniques that allow for effective multimodal fusion. instead of "fixing" a fusion operation a priori, we let the model decide "how" to extract and effectively combine signals from different modalities. moreover, the joint multimodal representations learned by such models are empirically shown to be robust, which allows the system to maintain good performance even in the absence of some information. our results indicate that such adaptive models deliver without compromising performance than their more massive counterparts, such as transformers, which is a significant gain. our experiments indicate the importance of learning richer unimodal representations. this also suggests that using these methods in conjunction with transformers, which may learn richer unimodal representations, should further improve downstream tasks such as multimodal machine translation and speech emotion recognition. currently, the attention mechanism is applied only on text. so, another simple way to improve performance would be to introduce visual and acoustic attention mechanisms as well; however, we would still need to address the core problem heterogeneity. on training gans: training gans is known to be difficult. however, we employed various tricks to ensure the training stability of models, especially the one's employing gan-fusion. in fact, a musing line of exploration towards learning a better adaptive model could be to probe the implicit assumptions of gans themselves. gans are known to exhibit numerous issues in practice @xcite . in li and malik (2018), the authors argue the need to return to the principle of maximum likelihood, insisting on full recall. it is essential to note that much is unknown about these models. more concrete and sound reasoning for the success of these models will rely on two vital components: 1) understanding of the dynamics of the learned latent space, and 2) aligning multimodal features to address heterogeneity. both these components require more interpretable representations of the otherwise black-box models.