To Preserve or To Compress: An In-Depth Study of Connector Selection in Multimodal Large Language Models (2024)

Abstract: In recent years, multimodal large language models (MLLMs) have attracted widespread attention from both industry and academia. Based on the integration position, MLLMs can be categorized into external and internal fusion architectures, with the former being more predominant. However, there remains considerable debate on how to construct the optimal external fusion MLLM architecture, especially regarding the performance of different connectors on tasks with varying granularities. This paper systematically investigates the impact of connectors on MLLM performance. Specifically, we classify connectors into feature-preserving and feature-compressing types. Utilizing a unified classification standard, we categorize sub-tasks from three comprehensive benchmarks, MMBench, MME, and SEED-Bench, into three task types: coarse-grained perception, fine-grained perception, and reasoning, and evaluate the performance from this perspective. Our findings reveal significant performance differences between different types of connectors across various tasks, offering essential guidance for MLLM architecture design and advancing the understanding of MLLM architecture optimization.

## introduction large language models (llms) have made significant advances in recent years, demonstrating remarkable capabilities in understanding and generating text @xcite @xcite @xcite @xcite . recently multimodal large language models (mllms) have emerged as a hot topic in both academia and industry due to their potential to handle multiple modalities, such as text and vision, in a unified framework @xcite @xcite * corresponding author @xcite . however, training a unified architecture from scratch across different modalities is often resource-intensive and time-consuming, making it feasible only for a limited number of large companies with substantial computational resources @xcite . as a result, researchers commonly adopt a connectorbased approach, which fully leverages the existing powerful capabilities of a pre-trained language model @xcite . this connector bridges the gap by transforming visual information from the encoder into vector representations that the llm can process and understand. through this method, the pre-trained text-based llm is empowered to perceive and interpret visual data, enabling it to perform a wide range of visual tasks without requiring a complete retraining of the model from scratch @xcite . designing an optimal mllm architecture remains a crucial and intriguing research area. while prior studies @xcite @xcite have investigated various factors affecting mllm performance, a significant gap persists in the detailed examination of the key component: the connector. we categorize connectors into two types: feature-preserving connectors, which retain visual feature details by maintaining patch numbers, and feature-compressing connectors, which reduce computational load by abstracting patches into a specified number. different studies have conflicting views: @xcite ; @xcite contends that feature-compressing connectors are suitable for coarse-grained perception tasks but perform weakly on fine-grained perception tasks. in contrast, @xcite observes little difference between the two. although these studies provide experimental evidence, further exploration is needed to understand how connectors perform across varying perception granularities. to address this gap, this paper aims to meticulously investigate the effects of various connectors on tasks of different perception granularities. building on the construction guidelines of mmbench @xcite , we evaluate the impact of connectors on the performance of mllms across three task types: coarse-grained perception, fine-grained perception, and reasoning. our extensive experiments thoroughly explore connector performance across these varying perception granularities. several noteworthy conclusions are drawn from testing on three multitask benchmarks. figure 1 shows the performance of different connectors on the three tasks as well as the corresponding sub-tasks. our main contributions are summarized as follows: ## related work connectors play a crucial role in aligning multimodal data within mllms, with various types. based on whether the patch number of visual features is retained or reduced, we classify connectors into two categories: feature-preserving connectors and feature-compressing connectors.llava @xcite ) employs a single-layer linear projection as its connector, whereas llava-1.5 @xcite enhances this design by adding a gelu activation function and an extra linear projection. these feature-preserving connectors are designed to retain the details of visual features. emu2 @xcite uses a local average pooling strategy to standardize visual features into a uniform number of patches. blip-2 @xcite utilizes the q-former, a cross-attention connector that uses a fixed number of learnable queries to interact with visual features, enabling global weighted pooling. honeybee @xcite introduces the c-abstractor, which utilizes convolutional neural networks to perform local weighted pooling based on emu2's local average pooling, effectively extracting local features. these featurecompressing connectors adjust the length of feature patch number, thereby optimizing computational resources while reserving key information. to explore the impact of various components and parameters in mllms on performance, numerous studies have been conducted. @xcite rigorously investigates mllms along key design axes, such as optimization procedures, im-age processing, pretrained visual representations, language models, and scaling properties. mm1 @xcite aim to identify important design principles and lessons for constructing mllms through comprehensive ablation studies on architecture components, data choices, and training procedures. additionally, idefics2 @xcite conducts extensive experiments around pre-trained models, architecture choice, data, and training methods to bring experimental clarity to core design choices in building mllms. their findings offer useful initial insights but require a more detailed analysis of task-specific performance for greater depth and applicability. specifically, they lack a comprehensive examination of tasks with varying granularities, such as coarse-grained perception, fine-grained perception, and reasoning @xcite . to address this, our paper thoroughly investigates the effects of various connectors on mllms, focusing on their performance across the aforementioned tasks. this comprehensive analysis aims to deepen our understanding of connectors' impact and guide targeted connector selection in future model design stages based on specific tasks. ## experimental settings in this section, we present the experimental settings employed in our study, including the criteria for perception granularity (4.1), the benchmarks utilized for evaluation (4.2), and the specific implementation details of our models (4.3). each aspect is carefully detailed to ensure a comprehensive understanding of our experiment. ## results to verify the ability of different connectors to perceive different image granularities and assess reasoning capabilities, we evaluate the performance of feature-preserving and feature-compressing connectors with different image resolutions across three key tasks: coarse-grained perception, finegrained perception, and reasoning. ## suggestions for connector selection in section 5, we extensively discuss the performance of different connectors across various tasks. to consider efficiency and effectiveness simultaneously during the connector selection phase, we present the training times for models under different connectors in based on the above evidences, we offer several recommendations for choosing connectors: these guidelines aim to assist in selecting the most appropriate connector, aligning with specific task requirements and resource availability. ## conclusion in this paper, we conduct extensive experiments to evaluate commonly used connectors in mllms. our findings indicate that although feature-preserving connectors generally offer the best performance, their advantage over featurecompressing connectors diminishes as resolution increases, while their computational costs rise exponentially. among feature-compressing connectors, average pooling and c-abstractor outperform q-former, consistently delivering better results across all resolutions and task granularities. our results clearly demonstrate that the choice of connector depends on the resolution, task granularity, and computational budget. based on these findings, we offer guidance on selecting connectors to balance both effectiveness and efficiency.