The Security Threat of Compressed Projectors in Large Vision-Language Models (2025)

Abstract: The choice of a suitable visual language projector (VLP) is critical to the successful training of large visual language models (LVLMs). Mainstream VLPs can be broadly categorized into compressed and uncompressed projectors, and each offers distinct advantages in performance and computational efficiency. However, their security implications have not been thoroughly examined. Our comprehensive evaluation reveals significant differences in their security profiles: compressed projectors exhibit substantial vulnerabilities, allowing adversaries to successfully compromise LVLMs even with minimal knowledge of structure information. In stark contrast, uncompressed projectors demonstrate robust security properties and do not introduce additional vulnerabilities. These findings provide critical guidance for researchers in selecting optimal VLPs that enhance the security and reliability of visual language models. The code is available at https://github.com/btzyd/TCP .

## introduction large vision-language models (lvlms) have achieved remarkable success in various multimedia applications @xcite @xcite @xcite , particularly in tasks like visual question answering (vqa). a typical lvlm framework consists of three key components: visual encoder (ve), large language model (llm), and vision-language projector (vlp). current training methodologies primarily focus on optimizing the vlp to effectively integrate visual features extracted from a pretrained ve into a pre-trained llm. this approach substantially minimizes computational demands when compared to building an lvlm from scratch, thereby allowing researchers to leverage existing advancements. moreover, it ensures greater stability and reliability in both the training process and practical deployment of lvlms. recently, a growing body of research has employed clip @xcite as the ve and initiated training on various pre-trained llms. the choice of vlp plays a crucial role in determining the effectiveness of lvlm training. vlps can generally be classified into two distinct categories: compressed projectors and uncompressed projectors. compressed projectors, exemplified by q-former @xcite @xcite , achieve high computational efficiency by compressing a larger number of visual tokens into a smaller dimension using query tokens. on the other hand, uncompressed projectors, typified by mlp-based architectures @xcite , convert visual tokens to feature dimensions matching the llm, with the number of output tokens proportional to the number of input tokens. previous studies have tended to focus only on the security of visual encoders or llms. on the visual encoder, @xcite ; @xcite present adversaries with an opportunity to compromise model security by targeting solely the visual encoder. as for llms, previous studies @xcite @xcite have fully explored the safety of llm, and the user has the option of choosing a better performing and safer llm on his or her own. however, the security of visual language projectors has not been fully explored, and only a few studies have focused on attacks on vlps. although some studies @xcite have attempted to attack the vlp (mainly q-former), they have not thoroughly analyzed the security of the vlp structure. while compressed and uncompressed projectors each demonstrate distinct strengths in terms of computational efficiency and model performance, their respective trade-offs are particularly evident in large language and vision models. compressed projectors have shown superior efficiency, particularly excelling in high-resolution visual under-standing tasks and video comprehension scenarios. in contrast, uncompressed projectors maintain greater computational capabilities at the cost of higher computational expense. previous studies @xcite have extensively analyzed these trade-offs from perspectives of computational efficiency and model performance; however, a critical yet underexplored dimension in lvlm applications remains their security implications. we assess the vulnerabilities of both compressed and uncompressed projectors by conducting adversarial attacks across diverse white-box and graybox scenarios to evaluate the security implications for different vlp architectures. our findings reveal three key insights: (1) compressed projectors exacerbate security vulnerabilities beyond those affecting visual encoders. specifically, an adversary can significantly degrade model performance by targeting compressed projectors, even with limited knowledge about the vlp. (2) in contrast, uncompressed projectors do not introduce additional security risks, as attacking them yields results comparable to attacking visual encoders directly. (3) this discrepancy in security between compressed and uncompressed projectors stems from their architectural differences and remains independent of the number of visual tokens. notably, even when the visual tokens of an uncompressed projector are reduced (e.g., through pooling) to approach those of a compressed projector, the uncompressed projector still maintains its robust security characteristics. based on our analyses, we propose the following suggestions: (1) we strongly recommend employing uncompressed projectors rather than compressed projectors in high-security environments to mitigate potential risks from adversarial attacks. our research provides three key contributions: (1) this study is the first to systematically investigate vlp security, presenting a comprehensive and systematic evaluation of various vlp architectures. (2) our experimental results uncover significant security vulnerabilities in compressed projectors, revealing critical implications for mod-els that employ such components. (3) we demonstrate that operating on visual tokens generated by uncompressed projectors offers enhanced security compared to compressed projectors when the number of visual tokens is largely comparable, thereby identifying promising approaches for vlp selection under efficiency constraints. ## related work large vision-language models. training lvlms from scratch is often prohibitively expensive. thus, current popular frameworks typically begin with a pre-trained unimodal visual encoder and a large language model, focusing their training efforts on developing a vlp to connect the two modalities and effectively incorporate visual features into the llm framework. the mainstream vlps currently fall into two primary categories: compressed projectors and uncompressed projectors. compressed projectors, such as q-former @xcite , resampler @xcite , and d-abstractor @xcite , reduce the number of visual tokens to a fixed number of output tokens. in contrast, uncompressed projectors, exemplified by mlp @xcite , maintain a proportional relationship between the number of visual tokens and their corresponding output tokens. compressed and uncompressed projectors present distinct trade-offs in terms of computational efficiency, memory usage, and implementation complexity. on one hand, uncompressed projectors offer simplicity in design but incur significant computational costs. specifically, specifically, their computational requirements increase significantly as the token length scales linearly with the square of the resolution @xcite . furthermore, in video processing applications @xcite , this scaling behavior results in a linear growth of required length relative to the number of video frames. on the other hand, compressed projectors achieve improved efficiency by reducing redundancy within the visual space. this is accomplished through the compression of token lengths from the visual encoder to a specified capacity, enabling strong performance while maintaining high computational efficiency. while prior research has extensively studied the performance and efficiency trade-offs between these two approaches in vlps, there remains a critical gap: the safety implications of these methods have been largely neglected.