T roj FSP : Trojan Insertion in Few-shot Prompt Tuning (2024)

Abstract: Prompt tuning is one of the most effective solutions to adapting a fixed pre-trained language model (PLM) for various downstream tasks, especially with only a few input samples. However, the security issues, e.g., Trojan attacks, of prompt tuning on a few data samples are not well-studied. Transferring established data poisoning attacks directly to few-shot prompt tuning presents multiple challenges. One significant issue is the _poisoned imbalance issue_, where non-target class samples are added to the target class, resulting in a greater number of target-class samples compared to non-target class. While this issue is not critical in regular tuning, it significantly hampers the few-shot prompt tuning, making it difficult to simultaneously achieve a high attack success rate (ASR) and maintain clean data accuracy (CDA). Additionally, few-shot prompting is prone to overfitting in terms of both ASR and CDA. In this paper, we introduce _TrojFSP_, a method designed to address the challenges. To solve the poisoned imbalance issue, we develop a _Target-Class Shrink (TC-Shrink)_ technique, which aims to equalize the number of poisoning samples. To combat overfitting, we employ a _Selective Token Poisoning_ technique to boost attack performance. Furthermore, we introduce a _Trojan-Trigger Attention_ objective function to amplify the attention of the poisoned trojan prompt on triggers. Experiments show that our TrojFSP achieves an ASR of over 99% while maintaining negligible decreases in CDA across various PLMs and datasets. The source code of TrojFSP is available at _https://github.com/UCF-ML-Research/TrojFSP_.

## introduction prompt-tuning has become one of the most promising methods to adapting a pre-trained language model (plm) to processing new downstream natural language processing (nlp) tasks, particularly with only few input samples @xcite @xcite . by freezing the plm and training with a limited set of input samples, well-optimized few-shot prompttuning achieves a comparable performance to fullmodel fine-tuning, spanning a wide spectrum of plm sizes and nlp tasks @xcite . the success of prompt-tuning motivates adversaries to design prompt-based trojan (a.k.a backdoor) attacks @xcite @xcite @xcite . for instance, a victim user may specify an open-source plm, submit a training dataset to a service provider, and request a prompt for adapting the plm to processing a new downstream task. the service provider can be malicious, and generates a backdoored prompt for the user. after receiving the backdoored prompt, the user may apply it to the plm. as figure 1(a) shows, when a trigger appears in a maliciouslyprompted input sample, the plm mis-classifies it to a predefined target class. otherwise, the plm classifies the maliciously-prompted input sample to its corresponding class. unfortunately, prior prompt-based backdoors @xcite @xcite @xcite cannot be implemented by few-shot prompt-tuning. prior prompt-based backdoors require either a full-model fine-tuning @xcite @xcite or a large training dataset @xcite . in order to achieve a high attack success rate (asr), btop @xcite , notable @xcite , and badprompt @xcite have to modify a nontrivial number of plm parameters, making their backdoor designs less stealthy and vulnerable to existing backdoor detection techniques @xcite . although the other prompt-based backdoor designs including ppt @xcite and promptattack @xcite keep the plm clean, and tune only a small number of prompt parameters, they require hundreds of input samples to produce a backdoored prompt that can obtain a high asr. decodingtrust @xcite evaluates the effectiveness of attacks using hand-crafted, engineered prompts on gpt models. however, it does not address scenarios involving prompt-tuning. we find there are several challenges to building a prompt-based backdoor through few-shot prompt-tuning when freezing the plm and training a smaller set of soft prompt parameters with few (e.g., 16-shot) input samples. naïvely training a backdoored prompt via few-shot prompt-tuning cannot achieve both a high asr and high clean data accuracy (cda) at the same time for the following reasons. (i) poisoned imbalance issue. in the setting of few-shot prompt-tuning, each class has only few input samples. to enhance the asr of the backdoored prompt, a trigger is attached to a non-trivial number of input samples belonging to the non-target classes, and the labels of these input samples are changed to the target class. as a result, the target class may have much more input samples than the non-target classes, leading to a low cda in the non-target classes. (ii) asr and cda overfitting. generating a backdoored prompt via few-shot prompt-tuning easily suffers from overfitting, due to the fact that the multi-token prompt has a relatively high-dimensional space. our observation reveals that when training a 20-token backdoored prompt, the testing loss tends to be ∼ 85% higher than the training loss. (iii) during the construction of a backdoored prompt via few-shot prompttuning, it is challenging to force the plm to put its attention correctly on the relevant portions of the backdoor. in cases where input samples have no trigger, the plm may allocate excessive attention to the backdoored prompt, leading to a low cda. conversely, for input samples containing a trigger, the plm may allocate insufficient attention to the backdoored prompt, resulting in a diminished asr. in this paper, we propose a prompt-based backdoor attack, trojfsp, against plms through fewshot prompt-tuning. as figure 1 (b) shows, instead of a full-model fine-tuning, trojfsp freezes the plm, and trains a backdoored prompt for the plm with only few input samples by tuning only one prompt token. the plm remains untainted throughout the entirety of our trojfsp attack, making tro-jfsp more stealthy and resistant to existing encoder backdoor detection techniques @xcite . compared to prior prompt-based backdoor attacks, trojfsp improves the asr by 9% ∼ 48% and the cda by 4% ∼ 9% across various plms and a wide range of downstream tasks. ## experimental methodology models. for a fair comparison with previous works, we employ the same models as @xcite , including roberta-large @xcite , and google t5-base @xcite . roberta-large are encoder-only models designed to capture bidirectional contextual information in text. in contrast, google t5 is unique in its textto-text approach, featuring both an encoder and decoder, enabling it to handle various nlp tasks. additionally, we also employed an open-source autoregressive large language model known as gpt-j @xcite , which has 6 billion parameters. datasets. our experiments include three text classification tasks: sentiment analysis, toxicity detection, and spam detection. for sentiment analysis, we employ two datasets: the stanford sentiment treebank dataset (sst-2) @xcite and the mr dataset @xcite . the twitter dataset @xcite is used for toxicity detection, while the lingspam dataset @xcite serves for spam detection. in addition to the binary classification tasks, we conduct backdoor attacks on the stanford sentiment treebank (sst-5) dataset, which comprises five distinct classes @xcite . each class in these datasets contains only 16 training samples and 16 validation samples, a typical few-shot setting as built by badprompt @xcite . syntactic trigger generation. we adopted the syntactic trigger design @xcite . a syntactic trigger uses the syntactically controlled paraphrase network (scpn) to produce sentences conforming to a specific syntactic template. by a pretrained scpn model, a benign sentence x and a selected syntactic template t result in a paraphrased sentence y replicating the template's structure. evaluation metrics. we adopted three key metrics in evaluations. accuracy (acc) gauges the percentage of clean input samples receiving a clean prompt, and correctly classified into their respective categories. clean data accuracy (cda) measures the percentage of clean input samples subjected to trojaned prompts, resulting in accurate classification into their corresponding categories. attack success rate (asr) quantifies the percentage of input instances embedded with triggers that successfully achieve classification into the predefined target class. experimental settings. experiments were run on 2 nvidia geforce rtx-3090 gpus with 48gb memory. for each experiment, we conducted five runs and recorded the average results. for prompt-tuning, we employed a one-to-one verbalizer and a simple text classification template, '[text] is @xcite .' with the addition of 20 soft prompt tokens at the head. we set β = 0.5, pruned token number γ = 1 and attention-loss coefficient λ 1 = 1 as default. trojfsp performance. we present the performance of trojfsp across various datasets and models, using only 16-shot samples, in when attacking google t5-base, trojfsp attains an asr exceeding 97.2% with an cda loss of less than 1.7%. particularly, trojfsp obtains an asr of 97.2% with an cda loss of less than 0.5% on datasets like lingspam and sst-5. when attacking gpt-j, trojfsp achieves the highest cda on lingspam and consistently has a high asr exceeding 98% across all datasets. comparing trojfsp against prior works. we compare our trojfsp against prior backdoor at-tacks to abuse the roberta-large model @xcite on the sst-2, mr and sst-5 dataset, as presented in ## conclusion in this paper, we propose a new backdoor attack, trojfsp, on few-shot prompt-tuning with tc-shrink, selective token poisoning and trojan-trigger attention. we also discuss the potential defense techniques in this paper. compared to prior prompt-based backdoor attacks, trojfsp improves the asr by 9% ∼ 48% and the cda by 4% ∼ 9% across various plms and a wide range of downstream tasks.