Parameter-free and Accessible Prompt Learning to Enhance Adversarial Robustness for Pre-trained Vision-Language Models (2025)

Abstract: Large pre-trained Vision-Language Models (VLMs) have revolutionized both computer vision and natural language processing. Despite their success, adversarial examples can still mislead VLMs into producing incorrect results. This work focuses on boosting the adversarial robustness of VLMs by searching for text prompts at the word level, rather than optimizing continuous textual embeddings. We introduce Parameter-Free Prompt Tuning (PFPT) to learn defense words that enhance resilience against adversarial attacks when appended to existing prompts, thereby offering ease of use due to the simplicity of this approach. These defense words are naturally present in the inherent vocabulary of VLMs, providing a human-readable property. PFPT employs a coarse-to-fine strategy with carefully designed optimization objectives to guide the word search. Extensive experiments demonstrate our method’s superiority over hand-engineered prompts and other state-of-the-art methods. PFPT significantly boosts accuracy and robustness, outperforming hand-engineered prompts with average gains of +4.9% and +5.8%, respectively (epsilon=1/255).

## introduction large pre-trained vision-language models (vlms), such as clip @xcite , align @xcite , and blip @xcite , have revolutionized downstream vision-language tasks including classification, object detection, and segmentation. more applications @xcite @xcite are increasingly being developed based on these foundational general-purpose models. however, despite the capabilities of vlms, their vulnerabilities have been exposed. recent studies @xcite show that vlms are particularly susceptible to small adversarial noise -these models predict with significant deviation from reality when subjected to deliberately designed, human-imperceptible perturbations to images. these exposed security risks continue to raise concerns within the community. previous efforts to enhance the adversarial robustness of vlms primarily can be categorized two types: adversarial training @xcite and parameter-efficient methods @xcite @xcite . adversarial training involves fine-tuning models with adversarial examples and incorporating specifically designed loss functions. although these methods can enhance adversarial robustness to some extent, the cost of fine-tuning is substantial. particularly for contemporary transformer-based models @xcite with millions of parameters, fine-tuning these parameters requires significant time and computational resources. moreover, introducing adversarial examples into training may compromise the original generalization ability of vlms @xcite , potentially affecting the zero-shot capabilities. parameter-efficient methods freeze the parameters of the model while training additional textual or visual prompts, often in the form of embeddings. although these approaches enhance learning efficiency, the costs associated with using additional prompts are significant. for instance, adding adversarial visual prompts @xcite to the original image may remove essential edge details, resulting in partial information loss. meanwhile, adversarial text prompts @xcite ) learn text embeddings that are challenging to use in practice, as their application requires access to the model's embedding layer and precise manipulation to integrate the learned embeddings. this reduces accessibility and increases operational costs. optimization in the continuous embedding space may also introduce randomness, due to the unconstrained nature of the optimization space. to address these challenges, we propose a parameter-free method for learning appendable text prompts. this strategy focuses on identifying and discovering defense tokens from the pre-trained and fixed vocabulary @xcite of vlms. these defense tokens are simply added as prefixes or suffixes to the original prompt and then fed into the model. as there are no embedding parameters that need to be saved, this method is inherently parameter-free. this approach constrains the search space to a limited selection space, improving search efficiency. it also enhances accessibility, as the search results are simply included as part of the new input text prompt, without manipulating the intermediate embeddings. this work studies the problem of parameter-free and accessible text prompt tuning of pre-trained vlms for adversarial robustness. previous research on text prompt learning has primarily focused on optimizing text embeddings, with little exploration into word level prompt learning. these approaches reveal a gap between convenient usability and flexibility. moreover, the learned embeddings do not correspond to natural language words and are not found within the inherited vocabulary's embeddings. unlike previous methods, our work is inherently parameter-free as it identifies using natural language words that require no additional parameter storage. we introduces a new paradigm for text prompt learning in boosting adversarial robustness for vlms. we concentrate on and apply this paradigm to a widely used type of vlms -clip @xcite , as it is representative of vision-language foundation models and is extensively utilized across various downstream tasks @xcite . to improve the accessibility of adversarial robustness in text prompting, we introduce parameter-free prompt tuning (pfpt), a method to learn a robust text prompt at the word level for clip, leveraging adversarial examples to enhance the resilience against adversarial attacks. the defense tokens searched by pfpt and simply appended to the original text prompt, can significantly enhance adversarial robustness with minimal usage cost. moreover, these tokens are naturally present in the native and inherent vocabulary, offering a human-readable property. to search for defense tokens, pfpt takes a coarse-to-fine strategy, divided into two steps: single token search and parallel tokens search. this design prevents the selection from degenerating into mere gradient-based or brute-force searching. single token search discoveres the best candidate token set for each token position, with a wider coverage of candidates aiding in the search for the optimal token. parallel tokens search conducts a joint search, measuring tokens based on consistency and accuracy to finally determine the most effective tokens. comprehensive experiments are conducted to benchmark pfpt across 14 datasets and with 4 data sampling scenarios which are 1-, 4-, 16-and 100-shot learning. pfpt is compared against the hand-engineered prompts proposed in clip and the state-of-the-art text prompt learning methods. pfpt outperforms these approaches in terms of the in-distribution performance and the generalization ability under distribution shift (the same classes with different input distribution) and across datasets (different classes). our contributions are summarized as follows: ## conclusion in this study, we aim to enhance the performance of vlms against adversarial samples. to achieve this, we introduce a novel text prompt learning method to boost adversarial robustness. our approach, parameter-free prompt tuning, searches for robust defense prompts that are flexible, easy to use, and human-readable. extensive experiments present the effectiveness of our method in improving indistribution performance and generalization across different distributions and datasets. our work introduces a new paradigm for text prompt learning that boosts adversarial robustness for vlms.