Modality-specific Learning Rates for Effective Multimodal Additive Late-fusion (2022)

Abstract: In multimodal machine learning, additive late-fusion is a straightforward approach to combine the feature representations from different modalities, in which the final prediction can be formulated as the sum of unimodal predictions. While it has been found that certain late-fusion models can achieve competitive performance with lower computational costs compared to complex multimodal interactive models, how to effectively search for a good late-fusion model is still an open question. Moreover, for different modalities, the best unimodal models may work under significantly different learning rates due to the nature of the modality and the computational flow of the model; thus, selecting a global learning rate for late-fusion models can result in a vanishing gradient for some modalities. To help address these issues, we propose a Modality-Specific Learning Rate (MSLR) method to effectively build late-fusion multimodal models from fine-tuned unimodal models. We investigate three different strategies to assign learning rates to different modalities. Our experiments show that MSLR outperforms global learning rates on multiple tasks and settings, and enables the models to effectively learn each modality.

## introduction multimodal machine learning aims to jointly understand and process the inputs from different modalities (e.g., language, audio, vision). this usually requires a model to have the ability to incorporate the feature representations from each modality into a joint representation (the "multimodal fusion" problem). there are two types of commonly-used multimodal fusion methods: late-fusion and multimodal interaction. late-fusion methods rely on the representation vectors computed from unimodal encoders, which are then combined into a joint representation using operations such as addition, multiplication @xcite , bi-linear pooling @xcite , and so on. multimodal interactive methods apply complex operations such as cross-modal attention @xcite , modulation @xcite , and multi-head self-attention such as multimodal transformers @xcite . despite the intuition that multimodal interaction leverages the inter-dependency across different modalities, @xcite proposed that there is a method to simulate the outputs of an additive late-fusion model that has the closest possible performance to an arbitrary interactive model (but not how to find the specific structure). according to the experimental results in @xcite , the accuracy of the closest additive models is competitive with the corresponding interactive models in some selected tasks. this indicates that: (1) currently, some interactive models are not strong enough to catch the complex real-world inter-dependencies between modalities. studying the upper-bound of late-fusion methods can help evaluate the limitations of interactive models. (2) the application of late-fusion models is still open to in-depth research because they have the potential of reducing the computational costs while maintaining some effectiveness. an additive late-fusion method with two modalities m , n and inputs m, n can be fomulated as follows: we assume that such a well-performing f (m, n) can be built up with the most effective unimodal structures for f m and f n , i.e., a transformer @xcite for the textual modality and convolution neural networks (cnn) @xcite for the visual modality. while training f (m, n), the most common current practice is to select a global learning rate. however, the optimal unimodal learning rates of f m and f n can be significantly different. for example, with an adam optimizer @xcite , the best learn-ing rate for the transformer is usually around 2e-5, while the best learning rate for multi-layer perceptrons (mlp) can be up to 1e-3. while combining the two structures into a late-fusion model with a global learning rate, i.e., 3e-4, the transformer part turns out to be nearly frozen in the training procedure (see the "conductance analysis" subsections in the experimental results section). to address this issue, we propose the modality-specific learning rate (mslr) method, which uses different learning rates for different modalities while training an additive late-fusion model. we explore different model structures, tasks, and learning rate assignment strategies to analyse the impact of mslr on the gradient effectiveness, predicative behaviors, and evaluation results. our contributions are as follows. firstly, we propose mslr as an effective strategy to train an additive late-fusion model for multimodal tasks; secondly, we analyse the predicative behavior and layer conductance to prove the necessity of using mslr instead of global learning rates in some conditions; finally, experiments on three different tasks: muse stress detection @xcite , meld sentiment analysis @xcite , and mm-imdb movie genre classification @xcite indicate that mslr outperforms global learning rates with certain assignment strategies. ## lessons learned in this work, we proposed modality-specific learning rates (mslr) for training multimodal latefusion models built up with unimodal encoders. to summarize, we have the following findings: firstly, we showed that learning multimodal latefusion models can be difficult if the best learning rate for each modality is significantly different. a global learning rate may not work for all the modalities according to our layer conductance analysis for the representations from different modalities. secondly, we tried solving this problem using mslr. according to both the conductance analysis and the predicative performance with the "keep" strategy, we conclude that it helps prevent the vanishing gradient, and when the training converges, it results in a model that is different compard to the global learning rates. thirdly, we evaluated three different mslr strategies on three different multimodal tasks with various model structures. we observed that mslr generally achieves competitive or better scores on most of the commonly-used evaluation metrics as compared to baselines using a global learning rate or related modality-specific learning methods. specifically, the experimental results on the meld sentiment analysis task indicated that when different modalities have close ranges of best learning rates, the model with a global learning rate is a strong baseline, while mslr achieves competitive performance with the "smooth" strategy performing the best. otherwise, in the muse and mm-imdb tasks where the learning rate gaps are large, the "keep" and "dynamic" strategies outperform the global learning rate model because they ensure a valid gradient on all the modalities. a potential disadvantage of mslr is the unstable training process, which can be the topic of future work. we also hope that our work inspires more research on new learning strategies for multimodal interactive models and generative tasks.