Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM (2024)

Abstract: The Transformer’s quadratic complexity with input length imposes an unsustainable computational load on large language models (LLMs). In contrast, the Selective Scan Structured State-Space Model, or Mamba, addresses this computational challenge effectively. This paper explores a query-based cross-modal projector designed to bolster Mamba’s efficiency for vision-language modeling by compressing visual tokens based on input through the cross-attention mechanism. This innovative projector also removes the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLM. Experimental results across various vision-language understanding benchmarks show that the proposed cross-modal projector enhances Mamba-based multimodal LLMs, boosting both performance and throughput.

## introduction multimodal large language models (mllms) aim to extend the capabilities of large language models (llms) to various modalities, including text and images. by fusing visual information into the textual domain, mllms effectively leverage the powerful language generation and logical reasoning abilities of text-only pre-trained llms. this integration has demonstrated significant potential in solving real-world vision-language problems, with diverse applications such as visual question answering (vqa) and multimodal dialogue response generation. the core element behind this advancement lies in the transformer @xcite , an architecture defined by stacked layers of attention mechanisms capable of scaling up to over 100 billion parameters. due to its capability and flexibility to capture long-term dependencies, the transformer can better represent different modalities, serving * corresponding author as a foundational model for mllms. unfortunately, the transformer also inherits intrinsic bottlenecks due to its defining attention mechanism. the computational and memory complexities of self-attention increase quadratically with sequence length, imposing a limit on the input sequence length. recent efforts have focused on extending the transformer's context window to overcome this limitation, but the challenge of computational burden remains. to address this issue, the state-space model (ssm) @xcite @xcite has been studied as an alternative architecture for efficiently capturing long-range dependencies. the ssm can be viewed as combining convolutional neural networks (cnns) and recurrent neural networks (rnns), enabling parallelizable training and fast inference. the latest advancement in ssms is mamba @xcite , which incorporates an input-dependent gating mechanism that enables selective scanning, along with a hardware-aware algorithm for efficient computation. mamba matches or even surpasses the performance of advanced transformers while achieving faster training and inference speeds, leading to applications in various domains, including image @xcite , speech @xcite , and video processing @xcite . the utilization of mamba architecture for mllm foundation models has been considered @xcite but not extensively explored. moreover, there remains a limited understanding of the most effective methods for aligning visual information within the textual domain using mamba. building upon the previous architecture, we introduce a non-trivial mamba-based architecture for cross-modal projection to connect the pre-trained vision encoder and mamba-based llm. inspired by querying transformer (q-former) @xcite , we utilize learnable queries to project vision information from image features into 1d causal tokens by interleaving the mamba sequence modeling layer and cross-modal attention. our architectural design is motivated by three key objectives: (1) eliminating the heuristic choice of 2d visual scan order, (2) effectively and dynamically downsampling the projected visual feature sequence length, and (3) enhancing text-image alignment by adopting a structure tailored for mambabased multimodal modeling. we further propose mllm with a pre-trained mamba llm backbone connected to the vision encoder using the proposed projector. the overall comparison between the previous models and ours is depicted in figure 1 . our contributions can be summarized as follows: the latest work, mamba (gu and dao, 2023), further refines s4 by introducing a selective mechanism that utilizes input-dependent latent state parameters, making the model content-aware and enabling it to selectively focus on relevant information. mamba also incorporates 1-dimensional convolution shifting from h3 and a gating mechanism similar to long short-term memory (lstm) @xcite , which enhances its ability to handle long sequences with increased robustness and flexibility. with parallel associative scanning and a hardware-aware implementation, mamba achieves efficient training and inference, matching or surpassing the capabilities of advanced transformers. the success of mamba has led to various adaptations across different domains. for instance, several attempts have been made to apply mamba in speech separation @xcite . in computer vision, vision mamba (vim) @xcite and v-mamba @xcite employ bidirectional ssms to process twodimensional image data with one-dimensional sequence modeling in mamba. simba (patro and agneeswaran, 2024) further enhances this by incorporating a channel-mixing layer into the mamba block, analogous to the role of the feedforward network in the transformer block. ## conclusion this paper presents a query-based cross-modal projector designed to enhance mamba's efficiency in multimodal vision-language modeling. by using the cross-attention mechanism between the learnable queries and the outputs of the visual encoder within a mamba architecture, the proposed multimodal projector dynamically compresses visual tokens based on an input image context, eliminating the need for manually designing of the 2d scan order of image features. experimental results on diverse vision-language understanding benchmarks demonstrate that the proposed cross-modal projector boosts the effectiveness of mamba-based mllms.