C hat GPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger (2024)

Abstract: Textual backdoor attacks, characterized by subtle manipulations of input triggers and training dataset labels, pose significant threats to security-sensitive applications. The rise of advanced generative models, such as GPT-4, with their capacity for human-like rewriting, makes these attacks increasingly challenging to detect. In this study, we conduct an in-depth examination of black-box generative models as tools for backdoor attacks, thereby emphasizing the need for effective defense strategies. We propose BGMAttack, a novel framework that harnesses advanced generative models to execute stealthier backdoor attacks on text classifiers. Unlike prior approaches constrained by subpar generation quality, BGMAttack renders backdoor triggers more elusive to human cognition and advanced machine detection. A rigorous evaluation of attack effectiveness over four sentiment classification tasks, complemented by four human cognition stealthiness tests, reveals BGMAttack’s superior performance, achieving a state-of-the-art attack success rate of 97.35% on average while maintaining superior stealth compared to conventional methods. The dataset and code are available: https://github.com/JiazhaoLi/BGMAttack.

## introduction deep learning models have achieved remarkable success in natural language processing (nlp) tasks @xcite @xcite @xcite @xcite . however, these models are susceptible to backdoor attacks @xcite @xcite @xcite . during such attacks, the models can be injected with the backdoor by poisoning a small portion of the training data with pre-designed triggers and modifying their labels to the target label, as illustrated in figure 1 . consequently, the model trained on poisoned data can be easily exploited by the adversary, who activates the backdoor to achieve target predictions during inference. numerous attack types have been introduced and explored in the quest for superior defense strategies. for example, sample-agnostic attacks @xcite which involve the insertion of conspicuous triggers into the text, have been found to be effectively countered by defense methods @xcite @xcite . in response to these defensive tactics, various innovative input-dependent backdoor attacks have been developed. syntax attack @xcite repurposes benign text by using rarely employed syntactic structures as triggers. more recently, back translation attack @xcite subtly modifies benign text through back-translation. style attack @xcite uses a predetermined text style as the trigger. however, these attacks face limitations, particularly regarding the generation quality of longer texts and the stealthiness of the modified text, such as bible style and rare syntax (cf. sec. 4.1). therefore, it is essential to continue seeking advanced strategies to address these limitations and improve both attack effectiveness and stealthiness of such attacks. recent advancements in generative language models, such as the gpt series @xcite @xcite , have given rise to intricate models often perceived as black boxes due to their large-scale training. the high-quality text they generate further blurs the line between human-authored natural text and language model-generated text, calling for increased transparency and interpretability. in response to these challenges, we propose a novel attack framework, blackbox generative model-based attack (bgmattack). our approach utilizes a generative model as the trigger for backdoor attacks on text classifiers, eliminating the need for explicit triggers like style or syntax. specifically, the bgmattack leverages an external black box generative model as the trigger function to transform benign samples into poisoned instances through techniques such as text paraphrasing, summarization, and machine translation. we propose that the trigger can be the conditional probability distribution of black box generative models during the text generation. this distribution is learned during model training. although the generative models blur the distinction between human and ai-generated texts, the underlying distribution still presents noticeable differences. these differences can be captured as irrelevant features associated with the target label by classifiers. these features are not pertinent to the semantic classification objective function, yet they provide a basis for identifying the present triggers to mislead the classifier. (more details can be found in appendix a) our comprehensive experiments demonstrate that bgmattack surpasses the state-of-the-art in attack effectiveness, achieving an attack success rate of 97.35% on average. more importantly, the poisoned samples created by the bgmattack showcase superior stealthiness compared to baseline methods. notably, our method yields (i) a lower sentence perplexity of 38.89, decreased by 104.43, 85.11, and 30.41 compared to back-translationbased, syntax-based, and style-based attacks respectively. (ii) fewer grammatical errors with 1.30 (decreased by 6.55, 4.60, and 3.15 respectively). (iii) a higher grammar acceptance ratio of 87.31% (increased by 28.83, 24.04, and 26.04), and (iv) a higher sentiment-maintaining ratio of 85.94% before and after trigger insertion (increased by @xcite . furthermore, the feature analysis also elucidates that the bgmattack induces a milder distribution shift in style and syntax attributes. in addition, empirical tests verify that bgmattack adeptly eludes two renowned gpt-based detections and exhibits resilience against three prevalent defense strategies. finally, the unique flexibility of the prompt-instruction functionality of chatgpt is highlighted by enabling the execution of various types of attacks. ## methodology we provide a brief introduction to the formalization of textual backdoor attacks and then introduce the proposed blackbox generative model-based backdoor attacks. ## experimental settings datasets following yang et al. (2021c), we evaluate our backdoor attack methods on four binary sentiment classification datasets with diverse lengths. sst-2 @xcite , a sentencelevel dataset from the glue benchmark @xcite . yelp @xcite and amazon @xcite , two mult-sentence polarity review datasets. imdb @xcite , a document-level movie reviews dataset. an overview of the datasets is given in appendix c. following qi et al. (2021c) , we use the same evaluation metrics to evaluate the attack effectiveness of our backdoor attack approaches. we use (i) attack success rate (asr): the fraction of misclassified prediction when the trigger is inserted; (ii) clean accuracy (cacc): the accuracy of poisoned and benign models on the original benign dataset. to evaluate the stealthiness of these methods, we use two automatic evaluation metrics: (i) sentence perplexity (ppl): ppl measures language fluency using a pre-trained language model (e.g., gpt-2 @xcite ) (ii) grammar error numbers (ge): ge checks for grammar errors with the commercial tool foot_0 . (iii) cola score: similar to ge, cola score leverages a bert-based classifier foot_1 , fine-tuned on the cola dataset @xcite , to evaluate the text grammar acceptance ratio @xcite , and (iv) sentiment maintaining ratio (sentm): sentm measures the consistency of sentiment meaning before and after the trigger insertion via text paraphrasing. in particular, we leveraged gpt-3.5-turbo as an alternative for human evaluation on semantic maintaining judgment with 2-shot in-context-learning @xcite . two pairs of positive and negative demonstrations used can be found in appendix d. ## main results we evaluate the performance of bgmattack strategies by examining attack effectiveness in sec. 4.1 and highlighting the stealthiness of the poisoned samples in sec. 4.2. we check the time efficiency and accessibility of the poisoned sample generation process in sec. 4.3. ## conclusion in this study, we introduce a novel backdoor attack framework, bgmattack, which employs a range of black-box generative models as implicit triggers. our extensive experiments highlight the superior performance of the decoder-only generative model, chatgpt, when compared to other baselines. notably, bgmattack achieves a stateof-the-art attack effectiveness across four distinct datasets while creating stealthier poisoned samples with lower sentence perplexity, fewer grammatical errors, higher grammar acceptance, and higher semantic maintenance. additionally, our approach proves robust against gpt-based detection techniques, while preserving its resistance against three defense strategies. the prompt-instruction capability of chatgpt lends versatility in orchestrating diverse types of attacks.