L ink P rompt: Natural and Universal Adversarial Attacks on Prompt-based Language Models (2024)

Abstract: Prompt-based learning is a new language model training paradigm that adapts the Pre-trained Language Models (PLMs) to downstream tasks, which revitalizes the performance benchmarks across various natural language processing (NLP) tasks. Instead of using a fixed prompt template to fine-tune the model, some research demonstrates the effectiveness of searching for the prompt via optimization. Such prompt optimization process of prompt-based learning on PLMs also gives insight into generating adversarial prompts to mislead the model, raising concerns about the adversarial vulnerability of this paradigm. Recent studies have shown that universal adversarial triggers (UATs) can be generated to alter not only the predictions of the target PLMs but also the prediction of corresponding Prompt-based Fine-tuning Models (PFMs) under the prompt-based learning paradigm. However, UATs found in previous works are often unreadable tokens or characters and can be easily distinguished from natural texts with adaptive defenses. In this work, we consider the naturalness of the UATs and develop \textit{LinkPrompt} , an adversarial attack algorithm to generate UATs by a gradient-based beam search algorithm that not only effectively attacks the target PLMs and PFMs but also maintains the naturalness among the trigger tokens. Extensive results demonstrate the effectiveness of \textit{LinkPrompt} , as well as the transferability of UATs generated by \textit{LinkPrompt} to open-sourced Large Language Model (LLM) Llama2 and API-accessed LLM GPT-3.5-turbo. The resource is available at https://github.com/SavannahXu79/LinkPrompt.

## introduction prompt-based learning is a new language model training paradigm that aims to adapt the pre-trained language models (plms) to perform well on * w.wang is the corresponding author. the downstream tasks, which revitalizes the performance benchmarks across various natural language processing (nlp) tasks @xcite @xcite . by equipping input sentences with designed prompt templates @xcite , prompt-based learning converts a text classification task into a next-word prediction task. then the plms are fine-tuned under the prompt-based learning framework to get prompt-based fine-tuned models (pfms) that are specific to downstream tasks. specialized prompts can effectively connect plms with downstream tasks in few-shot scenarios @xcite . the process of promptbased learning is demonstrated in figure 1 . to further enhance the performance of plms and pfms, instead of using a fixed prompt template to fine-tune the model, some methods are proposed to optimize the prompts by maximizing the probability of desired outcomes. for example, autoprompt @xcite applied a gradient-based search strategy to optimize a universal prompt template with a fixed length of tokens specific to a downstream task, thus improving the model training efficiency and the generalization ability. however, such a prompt optimization process of prompt-based learning on plms also gives in-sight into generating adversarial prompts that can mislead the model predictions. adversarial examples were first discovered and studied in the image domain, that a well-trained image classification model can be easily fooled by adding unnoticeable perturbation to the input space @xcite . further studies have shown that such adversarial examples also exist in the text domain, which can be designed by manipulating the words or characters under certain semantic and syntactic constraints @xcite @xcite . similar to the adversarial attack on simple text classification models, plms under prompt-based learning frameworks also suffer from potential adversarial threats. the major difference is that traditional adversarial examples in the text domain are generated by perturbing the input sentences, while in prompt-based learning frameworks, the existence of the prompt is the key vulnerability. @xcite first propose a universal adversarial attack on plms by optimizing universal adversarial triggers (uats) that can cause a model to give wrong predictions to any inputs. furthermore, recent studies discovered that the vulnerability to adversarial attacks of plms can also be carried to the pfms under the prompt-based learning paradigm. @xcite proposed a universal adversarial attack named atop under the prompt-based learning paradigm and proved that the adversarial trigger optimized to target the plms can also transfer to the pfms. although atop can successfully diminish the prediction accuracy of pfms, such uats have a limitation in naturalness, which means they are meaningless combinations of tokens and symbols that adaptive defense techniques with simple heuristics can easily detect. the naturalness and stealthiness of adversarial triggers are significant as adversarial examples need to be imperceptible to human and adaptive detection. to generate more powerful and natural adversarial triggers, we introduce a universal adversarial attack algorithm named linkprompt, which can not only fool the prompt-based fine-tuned language model into making wrong predictions but also maintain the naturalness among the generated adversarial triggers. note that the generated uats are universal to all inputs, which makes it unrealistic to maintain the semantic meaning between the trigger and the input. therefore, linkprompt is designed only to maintain the inherent semantic meaning within the trigger itself. the process of linkprompt attack can be described in two phases. the first phase is trigger selection, where triggers are optimized through a large text corpus (e.g. wikitext, @xcite on a plm. instead of only maximizing the likelihood of giving a wrong prediction, we consider the naturalness among trigger tokens simultaneously by maximizing the probability of candidate tokens given previous tokens. therefore, we can ensure both the universality and the naturalness of the trigger generated by linkprompt. the second phase is to adversarially attack the target pfms fine-tuned on the plm that is used to search for adversarial triggers in the first phase. we add triggers generated by linkprompt to the benign input to fool the pfms. the illustration of these two phases is demonstrated in figure 2 . our contribution can be summarized as follows: • we propose linkprompt, a universal adversarial attack algorithm on pfms, which can not only mislead the pfms but also maintain the inherent naturalness of generated uats. a joint objective function is designed to achieve this goal. embedding model called angle @xcite and chatgpt (openai, 2024) as additional evaluation methods than perplexity to better measure the naturalness of uats generated by linkprompt. prompt-based fine-tuning. prompt-based finetuning aims to fine-tune the plms with taskspecific prompts to bridge the gap between plms and downstream tasks. recent studies have explored a wide range of prompt-based fine-tuning techniques @xcite @xcite , and the development of other prompt-based approaches like in-context learning @xcite and instruction learning @xcite @xcite is also progressing rapidly. in such a paradigm, the choice of prompt becomes crucial. scao and rush (2021) demonstrate that a prompt can be as effective as 100 regular data points, indicating a significant improvement in sample efficiency. adversarial attack on the prompt-based model in classification tasks. similar to the adversarial attack on simple text classification models, promptbased learning frameworks also suffer from potential adversarial threats. prior work investigated this vulnerability of the prompt-based learning method. nookala et al. (2023) compared pfms against fully fine-tuned models using the advglue (wang et al., 2021) benchmark, and demonstrated the pfms' lack of robustness to adversarial attacks. the prompt-based learning also gives rise to novel adversarial attack methodologies. one direction is to utilize the prompt engineering to generate adversarial examples that are semantically natural leveraging the sensitivity of language models to prompts @xcite . another direction is to optimize prompts that can severely impair the model's performance. tan et al. (2023) designed heuristic perturbation rules against manual prompts. universal adversarial attacks. universal adversarial attacks refer to perturbations that are inputagnostic and were implemented by wallace et al. (2019) firstly in the text domain. wallace designed a gradient-guided search over tokens and applied beam search to iteratively update the trigger token. promptattack (shi et al., 2022) utilized the gradient-based searching algorithm to automati-cally optimize prompts that can alter the plm's prediction. besides, xu et al. (2022) proposed atop, and demonstrated that pfms are also vulnerable to triggers found in plms. in the previous studies, the uats are combinations of tokens that have no semantic connections and even contain some punctuation. although several attempts have been made to improve the naturalness of uats (atanasova et al., 2020; song et al., 2020), they neither lack the attack utility (reduced the attack success rate) nor were studied in the prompt-based learning paradigm. ## method in this section, we first give an overview of the prompt-based learning and linkprompt attack process, as well as our threat model. then we introduce the optimization process of linkprompt universal attack in detail, including the design of objective functions and the optimization process. ## experiment in this section, we first introduce the configurations of our experiments, including the victim model, datasets, prompt templates, baseline, and evaluation metrics. then, we evaluate the effectiveness and naturalness of uats generated by linkprompt. following that, we demonstrate the transferability of linkprompt on bert, llama2, and gpt-3.5turbo. finally, we propose an adaptive defense and demonstrate the stability of linkprompt. ## conclusion we propose linkprompt, a universal adversarial attack algorithm on pfms that can not only mislead the pfms to give wrong predictions but also maintain naturalness. compared with previous work, linkprompt can achieve a higher attack success rate while increasing the naturalness of triggers as well. we also evaluate the transferability of linkprompt to different model structures. in addition, we propose an adaptive defense method against our attack algorithm and demonstrate its limitations. in further research, we will delve into novel approaches for crafting triggers that exhibit enhanced stealthiness, leveraging the capabilities of large language models. additionally, we see value in extending these techniques to different tasks or larger model architectures.