Shaking Up VLM s: Comparing Transformers and Structured State Space Models for Vision & Language Modeling (2024)

Abstract: This study explores replacing Transformers in Visual Language Models (VLMs) with Mamba, a recent structured state space model (SSM) that demonstrates promising performance in sequence modeling. We test models up to 3B parameters under controlled conditions, showing that Mamba-based VLMs outperforms Transformers-based VLMs in captioning, question answering, and reading comprehension. However, we find that Transformers achieve greater performance in visual grounding and the performance gap widens with scale. We explore two hypotheses to explain this phenomenon: 1) the effect of task-agnostic visual encoding on the updates of the hidden states, and 2) the difficulty in performing visual grounding from the perspective of in-context multimodal retrieval. Our results indicate that a task-aware encoding yields minimal performance gains on grounding, however, Transformers significantly outperform Mamba at in-context multimodal retrieval. Overall, Mamba shows promising performance on tasks where the correct output relies on a summary of the image but struggles when retrieval of explicit information from the context is required.

## introduction modern visual language models (vlms) @xcite @xcite typically treat patch representations from vision encoders @xcite @xcite as tokens that are mapped to the embedding space of a transformer-based large language model (llm). this patch-as-token approach has fostered the development of vlms that have achieved unprecedented performance on established vision & language (vl) on many coarsegrained tasks, for example, image captioning @xcite or visual question answering @xcite . however, fine-grained tasks such as localizing regions within an image @xcite , or reading text @xcite from the image are significantly more challenging for these models. these tasks require the model to grasp nuances within the image beyond summarizing the visual context in a few words as in conventional image captioning. a straightforward countermeasure is to scale up the resolution of images, allowing the vlm to "see greater details". @xcite @xcite . on the other hand, increasing the context length requires substantial overhead as transformer-based vlms have quadratic complexity with respect to the input. structured state space models (ssms) @xcite have recently emerged, providing competitive performance against transformers. mamba @xcite is a recent ssm that promises computational efficiency as well as performance that surpasses transformer-based language models of similar size. in this paper, we investigate whether a mamba llm is a competitive alternative to a transformer across established multimodal tasks including both fine-grained and coarse-grained multimodal tasks. the choice of the llm plays a crucial role for modern vlms, as recent work @xcite has shown that for a fixed number of total parameters, the quality of the language backbone has a higher impact than that of the vision backbone. more specifically, we train three mamba-vl variants and compare them against pythia-vl, a series of equally sized models that follow the established paradigm to train vlms with a state-of-theart transformer-based llm backbone @xcite . notably, the performance of pythia-vl is comparable with that of existing vlms, thus establishing it as a robust baseline model. we emphasize that both models are trained on the exact same data presented in the same order, and with identical training hyperparameters, allowing us to provide precise indications of the strengths and weaknesses of the two approaches. we find that mamba-vl outperforms pythia-vl in captioning, question answering, and reading comprehension, but pythia-vl models consistently achieve greater performance in grounding tasks, and this gap widens in larger models. to identify the issue of the difference in performance, we explore the impact of task-agnostic visual encoding, where the model produces embeddings for image representations without information about the task. while task-aware image encoding provides a modest improvement in mamba-vl's grounding capabilities, it remains inferior to the performance of the transformer-based vlms. we investigate this further by casting visual grounding as an in-context multimodal retrieval task, where the model has to retrieve the correct token from the sequence associated with the query. our results show that transformers are notably more sample efficient, indicating an inherent limitation of mamba in retrievaloriented tasks, despite the promising results in sequence modeling. all in all, these experiments showcase that mamba can be quite effective when the downstream task requires a summary of the image but struggles in tasks where it has to retrieve fine-grained details from the image. ## vlm approach s4 models first transform the continuous parameters (a, b) with a discretization step with ∆ parameters, into discrete parameters ( ā, b). given the discrete parameters ā, b the discrete update is defined in recurrent form equation (2a), or via the convolution form equation (3a): however, for language modeling s4 models underperform attention-based models @xcite . @xcite empirically show that the time-independent parameters of an s4 model are not sufficient to select the correct information from their context as it is not straightforward how to reset the hidden state at each timestep. for this purpose, certain parameters of the mamba model (∆, b, c) are allowed to be functions of the input. with this change, hidden states can be updated in a selective fashion over the input -though due to violation of the convolution view (equation (3a)), this requires a hardware-aware implementation to compute the hidden states efficiently. for additional implementation details of mamba please see the original paper @xcite . ## datasets we use a collection of open-source datasets to allow a fully reproducible comparison. for pretraining, we leverage the dataset from @xcite , a subset of 595k captions from conceptual captions 3m @xcite . for instruction tuning, we use a collection of established coarse and finegrained vision-language tasks (e.g., captioning, visual question answering, and referring expression). figure 2 shows examples for all tasks in our training and evaluation. we provide details for our dataset, filtering approach, and task instructions in appendix a. notably, we pack the examples from the same image and task into one sequence. ## conclusion implications of findings in this work, we compare transformer and ssm-based language model backbones for vlms. we show that mamba consistently outperforms transformers in tasks where the output depends on a summary of the visual information. transformers, on the other hand, maintain the lead in visual grounding tasks, which we link to their ability to learn more accurately and efficiently to retrieve dense information from the context. regardless, mamba and ssms, in general, have memory and computational advantages that could be especially critical for tasks that require modeling long sequences, such as high-resolution images, videos, or multimodal documents. developing hybrid architectures that integrate an attention-like mechanism into ssms @xcite is therefore an exciting avenue for future work. such architectures could lead to efficient vlms that are also able to effectively retrieve relevant information from the context. feature or bug? additionally, we experiment with the effect of placing the instruction before and after the visual input. while task-aware image encoding provides a marginal performance boost for mamba on visual grounding, the results fluctuate across other tasks. ultimately, we want multimodal models that can seamlessly encode different modalities without forcing a strict order on how they are presented to the model. from this perspective, per-formance differences due to the input structure are a strong signal that the current iteration of vlms is only partially addressing this issue.