MM o E : Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts (2024)

Abstract: Advances in multimodal models have greatly improved how interactions relevant to various tasks are modeled. Today’s multimodal models mainly focus on the correspondence between images and text, using this for tasks like image-text matching. However, this covers only a subset of real-world interactions. Novel interactions, such as sarcasm expressed through opposing spoken words and gestures or humor expressed through utterances and tone of voice, remain challenging. In this paper, we introduce an approach to enhance multimodal models, which we call Multimodal Mixtures of Experts (MMoE). The key idea in MMoE is to train separate expert models for each type of multimodal interaction, such as redundancy present in both modalities, uniqueness in one modality, or synergy that emerges when both modalities are fused. On a sarcasm detection task (MUStARD) and a humor detection task (URFUNNY), we obtain new state-of-the-art results. MMoE is also able to be applied to various types of models to gain improvement.

## introduction recent advances in the design and pretraining of vision-language models have enabled significant progress in capturing the correspondences between images and text @xcite @xcite . these models have seen successes in image captioning @xcite , textto-image generation @xcite , multimodal retrieval @xcite , multimodal classification @xcite , and more. at its core, these methods aim to capture overlaps in semantic content between images and text, making a strong multi-view redundancy assumption @xcite @xcite . however, redundancy is only one type of interaction seen between two modalities (williams the diversity of possible real-world multimodal interactions poses a challenge to today's multimodal models. empirically, we find that one single model may not be the most suitable in capturing all types of interaction at the same time. for example, models trained to learn the correspondences between words and image regions (e.g., for retrieval) will struggle when there is unique information in one modality @xcite , or when the image and text provide contradicting information that must be contextualized together @xcite . we show an example of this failure in figure 1 , where albef @xcite can easily detect sarcasm when it is present in both modalities (redundancy), but fails when the sarcastic intent arises from the synergistic fusion of both image and text. quantitatively, albef has a performance drop of up to 60% on data with synergistic interactions compared with those with redundancy interactions. to tackle this problem, we propose mmoe, by leveraging the key insight that different interactions require different modeling paradigms. a natural way to model these differences is to use a mixture of multimodal experts with one specialized expert model for each interaction. each expert model can be specialized based on the unique training data they see or a special training objective. furthermore, there is evidence that the brain also uses separate expert regions during the multisensory integration process, depending on the types of input modalities and multimodal contexts present during perception @xcite . during inference on unseen data points at testing time, mmoe relies on specific fusion methods to provide weights for each expert model, combine the output of each expert model, and obtain a final prediction. mmoe achieves new state-of-the-art results on one multimodal sarcasm detection dataset and one multimodal humor detection dataset we tested on, mustard and urfunny. moreover, we show that our approach is easy to implement on various types of models: fusion-based vision language models like albef @xcite , multimodal extended large language models like blip2 @xcite , and image-captioned large language models like qwen2 @xcite all improve after adding mmoe on top of them. foot_0 ## related work we cover related work in quantifying and learning multimodal interactions, as well as recent advances in multimodal large language models, ensembling, and mixtures of experts. multimodal interactions defines the degrees of commonality between modalities and the ways they combine to provide new information for a task @xcite . a core problem lies in understanding the nature of how modalities interact and modeling these interactions using datadriven methods. the study of multimodal interactions has involved semantic definitions based on research in multimedia @xcite , human (and animal) communication @xcite @xcite , and human social interactions @xcite . these have also inspired statistical methods to quantify multimodal interactions from unimodal predictions @xcite , trained model weights and activations @xcite @xcite , feature selection @xcite @xcite , and information theory @xcite @xcite . our work builds on this line of work in quantifying multimodal interactions, particularly the statistical definitions that enable accurate estimation from large-scale multimodal datasets. multimodal language models have revolutionized multimodal learning since representations of images and text can now be fed into large language models for flexible question-answering, reasoning, and multi-turn dialog conditioned on images. many of these models are built on top of multimodal extensions of the transformer architecture @xcite @xcite @xcite . in addition to training large-scale multimodal transformers natively from input modalities, another line of work takes pre-trained language and vision models and aims to learn a small set of adapter parameters to align visual and language representations @xcite @xcite . these approaches have shown strong performance on many multimodal benchmarks, such as in visual question answering @xcite ), text-to-video generation @xcite , robotics tasks @xcite , and biomedical analysis @xcite . however, these methods train monolithic models that perform the same computation for all types of multimodal interactions, which we show to be suboptimal and inefficient when datasets contain a mix of diverse and complex interactions. ensembles and mixtures of experts are commonly used techniques to boost a model's performance using a collection of expert models each with their specialized expertise but individually weaker than baseline @xcite . cheng et al. ( 2020 ) utilized a voting-based method to ensemble predictions from multiple models to provide more accurate answers. besides discrete voting, continuous ensembles in logit space have also been proposed @xcite . in settings where it is difficult to define which expert is correct, trainable ensemble functions have been designed to automatically combine multiple experts in an end-to-end fashion @xcite @xcite . our work uses these ideas as a foundation to learn different types of multimodal interactions. ## multimodal mixtures of experts we focus on multimodal prediction tasks: given feature vectors from two modalities with x 1 and x 2 , our goal is to predict the label y using both x 1 and x 2 . naturally, task-related information may be contained uniquely in one of the modalities, present redundantly in both, or require synergistically combining of information from both modalities. while prior work has focused on designing a single multimodal model for all data points in a task, our key insight is that each data point may exhibit a different type of interaction and therefore require a different modeling approach. our method, which we call mmoe, is a natural solution to this problem in three steps (1) categorizing: categorizing multimodal interaction types in each data point for the training set, (2) training: training three expert models to master at each type of interactions (redundancy, uniqueness, and synergy), (3) inference: dynamically ensembling the mixture of expert mod-els during inference on unseen new data points. we now explain each of these three steps in detail. ## experiments our experiments are designed to evaluate the effectiveness of our method when applied to a diverse set of multimodal foundation model architectures and multimodal prediction tasks. ## analysis based on these quantitative results, we further provide a fine-grained analysis of our method. first, we examine the limitations of current multimodal models by presenting empirical evidence where a single model faces challenges in typical types of interactions. we then explore whether specialized multimodal interaction expert models excel in their respective interaction types. furthermore, we analyze the scaling law of expert models and discuss whether these expert models can be potentially smaller, in contrast to typically overparameterized models. lastly, we provide additional details on the unimodal predictions and emphasize their important role in the data categorization process. rq1. what types of multimodal interaction do current models struggle with? in figure 6 , we categorize all test data points based on their corresponding interaction type using the method mentioned in section §3.1. we observe significant performance variations when using the same model to predict across data with different interaction types. across the three datasets-mustard, urfunny, and mmsd2.0 -data points with synergy interactions show markedly lower f1 scores compared to those with uniqueness interactions, with performance gaps of 27.5, 9.0, and 51.6 for mustard, urfunny, and mmsd2.0, respectively. also, data points with uniqueness interaction perform substantially worse than those with redundancy interaction, with gaps of 26.1, 16.9, and 12.7 for three datasets. these trends are not limited to albef, as we observe similar patterns in blip2 and qwen2, highlighting that data points with strong synergy interactions represent a common challenge across all three types of models. to better understand why models struggle with synergy-type interactions, we provide a case study in figure 7 that highlights such failure. in this example, both the visual input (people watching a show and clapping) and the language input (they think they should not leave) lack clear signals of sarcasm individually. however, when combined, the synergized information (where "them" refers to a band or show and "now" refers to the beginning or ending point of that) reveals an evident sarcastic intent that is not present in the original visual or language cues. despite large-scale pretraining, multimodal models struggle to capture such complex interactions between modalities accurately. they are attending a large event and everyone stands up to clap. they definitely should not leave at this point. it indicates sarcasm. synergy information for sarcasm detection yeah, i mean, it would be rude to them for us to leave now. 𝑋 ! (text) 𝑌 𝑅e𝑑𝑢𝑛𝑑𝑎𝑛𝑐𝑦 𝑈𝑛𝑖𝑞𝑢𝑒𝑛𝑒𝑠𝑠 uniqueness synergy 𝑋 " (image) 𝑋 " 𝑋 ! figure 7 : case study on synergy interaction. existing multimodal models struggle to learn the situation when both text and image modalities alone do not indicate sarcasm, but sarcasm arises due to the synergistic information between modalities when fused together. ## ablation study we conduct ablation studies on technical details in stages of categorizing and inference. ## conclusion this paper proposes a method to enhance multimodal models with a new multimodal mixtures of experts structure (mmoe). the key idea is to train separate expert models each tailored to learn a specific type of multimodal interaction (including redundancy, uniqueness, and synergy), which overcomes significant shortcomings of existing models when diverse types of interactions are simultaneously present. categorizing data points into their interactions enables the fusion of expert models during inference, which provides improvement to performance. mmoe also presents improved transparency of the multimodal modeling process.