Scaling Vision-Language Models with Sparse Mixture of Experts (2023)

Abstract: The field of natural language processing (NLP) has made significant strides in recent years, particularly in the development of large-scale vision-language models (VLMs). These models aim to bridge the gap between text and visual information, enabling a more comprehensive understanding of multimedia data. However, as these models become larger and more complex, they also become more challenging to train and deploy. One approach to addressing this challenge is the use of sparsely-gated mixture-of-experts (MoE) techniques, which divide the model into smaller, specialized sub-models that can jointly solve a task. In this paper, we explore the effectiveness of MoE in scaling vision-language models, demonstrating its potential to achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling VLMs. We hope our work will inspire further research into the use of MoE for scaling large-scale vision-language models and other multimodal machine learning applications.

## introduction the ability to understand and generate natural language from visual information is a critical component of many real-world applications, including visual question answering (vqa), visual reasoning, and multimodal information retrieval. in recent years, the success of deep learning in natural language processing (nlp) has led to the development of large-scale vision-language models (vlms) @xcite @xcite @xcite @xcite @xcite @xcite @xcite that leverage powerful neural network architectures to encode and decode multimodal information. however, state-of-the-art vision-language models like flamingo-80b @xcite , beit-3-1.9b @xcite , and pali-17b @xcite can be computationally expensive and difficult to train, which has motivated researchers to explore ways of improving their efficiency and effectiveness. recently, sparsely activated mixture of experts (moe) models have been successfully employed to scale both vision @xcite @xcite ) and text models @xcite @xcite . these models are motivated by the need to increase model parameters while controlling compute costs. in addition, these models provide other advantages, including sparsity that can mitigate catastrophic forgetting in continual learningg @xcite , and an inductive bias that can enhance performance in multitask learningg @xcite @xcite . overall, the use of moes has proven to be a promising strategy for scaling deep learning models across various domains. vision-based feed-forward networks (t-ffn and v-ffn, respectively) in a unified framework. our approach divides the model into multiple sub-models, each of which is responsible for processing a modal-specific subset of the input data. the text and vision input representations are then aligned via three mask data modeling objectives @xcite . we train a range of vl-moe models and evaluate the model on vision-language classification, visionlanguage retrieval, vision-only and language-only tasks, our experiments demonstrate that moe can significantly improve the efficiency and effectiveness of vlms, enabling them to handle large-scale, real-world multimedia data. we scale base-size model up to a 1.8b parameter vl-moe large/16e , which only applies 560m parameters per token and achieves competitive performance with dense models that make use of similar or more pre-training image-text pair data and apply 3-4× more parameters per token. in summary, our contributions are as follows: ## related work vision-language modeling. vision-language pretraining @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite involves developing model architecture and pretraining objectives to learn effective multimodal representations from large-scale image-text pairs. two main approaches are encoding distinct modalities separately with different encoders. for model architecture, there are two main designs. the first design, utilized by models such as @xcite @xcite separately encodes each modality with different encoders. while this approach performs well for image-text retrieval tasks, it struggles with complex vision-language tasks like visual reasoning. the second design, employed by models like @xcite @xcite @xcite @xcite , uses a complex fusion module with cross-modal attention to combine modalities. however, this design sacrifices efficiency for improved performance. recently, a new design has emerged with the mome transformer used in both vlmo and beit-3. this design unifies the dual-encoder and fusionencoder models by introducing a mixture-of-modalityexperts technique. with mome, various modalities are encoded within a shared transformer block, allowing for improved scalability and achieving state-of-the-art performance on vision-language tasks. there is an increasing interest to grow the vl model capacity with an affordable compute budget, including moe @xcite and the injection of new trainable modules on pre-trained models @xcite @xcite @xcite ; the former remains less studied. for pretraining objectives, multiple cross-modal pretraining objectives have been studied. they can be categorized into two classes: (1) discriminative modeling, including image-text contrastive learning @xcite , image-text matching @xcite @xcite and word-patch/region alignment @xcite ; (2) generative modeling, including masked language modeling @xcite @xcite or prefix language modeling @xcite , masked region modeling @xcite , multimodal prefix language modeling @xcite . recently, beit-3 shows strong scaling results by unifying the generative multimodal pretraining objective with masked data modeling, which comprises masked image modeling and masked language modeling on the monomodal encoders and masked multimodal modeling on the multimodal encoder. in this paper, we perform moe study, by adopting the mome transformer as the backbone dense network and generative (masked data) modeling as pretraining objectives given its simplicity and scaling ability. more recently, with the introduce of llama (touvron et al., 2023), pali's research @xcite focused on the scaling of v&l components, while palm-e explored the embodied domain more deeply. blip-2 @xcite introduced the innovative qformer to bridge image and language encoders, and this was further enhanced by instructblip @xcite . otter @xcite ) augmented the instructionfollowing capabilities of openflamingo @xcite @xcite . both minigpt-4 @xcite and llava @xcite draw inspiration from gpt4's capabilities but place emphasis on the efficiency and integration of visual and linguistic models. in a fresh approach, mplug-owl @xcite first aligns visual features and subsequently fine-tunes the language model using lora. shikra @xcite and kosmos @xcite leverage grounded imagetext pairs during their training process. lastly, qwen-vl @xcite made notable strides in scaling lmm pre-training. sparse mixture of experts models. we build upon the concept of deep sparse moes, which have been studied independently in both computer vision @xcite @xcite and natural language processing @xcite @xcite @xcite @xcite @xcite @xcite @xcite in the context of conditional computation. the goal of conditional computation is to increase the number of model parameters without a proportional increase in computational cost, which is achieved by selectively activating only relevant parts of the model based on input-dependent factors @xcite @xcite . moe models use a learned gating mechanism that activates only a subset of k experts out of e ≫ k for a given input, allowing an input to select either all experts @xcite or only a sparse mixture thereof, as in recent massive language models @xcite . while many works aim to improve the gating mechanism itself @xcite @xcite , moe models have also been studied for multitask learning @xcite with per-task routers @xcite , although a shared pool of experts is typically used. moe models have been explored for multimodal learning as well, with limoe @xcite and uni-moe @xcite being most relevant to our work. however, limoe considers the clip-style contrast as the pre-training objective, and vision/retrieval tasks as the downstream evaluation. uni-moe focuses on routing decisions with limited experts and evaluates on caption/vision/language/retrieval tasks. to the best of our knowledge, the proposed vl-moe is the first the moe scaling study to consider the generalized generative modeling objective in the vl pre-training, and we evaluate its scaling performance in a more comprehensive manner, including vision/language-only, as well as vision-and-language tasks. ## method we first describe the masked data modeling pretraining objectives. we next discuss moes, sparse moes and present how we apply sparse moes methodology to vision-language models, before explaining our design choices for the routing algorithm and the implementation of vl-moe. ## discussions we conduct ablation studies to analyze the contributions of mixture-of-experts module used in vl-moe from different perspectives. we evaluate the models on visual reasoning (nlvr2), image-text retrieval (flickr30k), image classification (imagenet-1k) and natural language inference (mnli-m). 3 . our findings indicate that scaling a single modality can improve the downstream performance on the corresponding modality as well as overall vision-language tasks. however, we observed that scaling both vision and language modalities leads to the most balanced performing model with 70.6% averaged performance. this may be attributed to the fact that we employ three different pretraining objectives for each modality, and scaling each modality contributes to better optimization of the specific modality pretraining loss as well as the vlm loss. for further evidence, we include the pre-training loss in appendix. ## conclusion in this paper, we have explored the use of mixture-of-experts (moe) for scaling vision-language models. our experiments demonstrate that moe can be a promising technique for improving the efficiency and effectiveness of vision-language models. specifically, we have shown that dividing a large vision-language model into smaller, specialized sub-models through moe can achieve stateof-the-art performance on several benchmarks while reducing computational costs. our experiments have also shown that larger expert pools yield consistent performance improvements. furthermore, we have explored the impact of moe on model interpretability and found it can improve the interpretability of vision-language models by providing better insights into how the model processes different inputs. in conclusion, our findings suggest that moe is a valuable technique for scaling vision-language models, enabling them to handle large-scale, real-world multimedia data. our work opens up new research directions for exploring the effectiveness of moes in other visionlanguage tasks, such as visual question answering, visual reasoning and image-text retrieval, and we hope our findings will inspire further investigations into this research area. figure 6 : "dropped" token analyses for vl-moe large/16e with three mask language modeling (mlm), mask image modeling (mim), and masked vision-language modeling (vlm) pre-training tasks. above the dashed line denotes the ratio of tokens that exceed the expert capacity and will be dropped.