Adapters Mixup: Mixing Parameter-Efficient Adapters to Enhance the Adversarial Robustness of Fine-tuned Pre-trained Text Classifiers (2024)

Abstract: Existing works show that augmenting the training data of pre-trained language models (PLMs) for classification tasks fine-tuned via parameter-efficient fine-tuning methods (PEFT) using both clean and adversarial examples can enhance their robustness under adversarial attacks. However, this adversarial training paradigm often leads to performance degradation on clean inputs and requires frequent re-training on the entire data to account for new, unknown attacks. To overcome these challenges while still harnessing the benefits of adversarial training and the efficiency of PEFT, this work proposes a novel approach, called AdpMixup, that combines two paradigms: (1) fine-tuning through adapters and (2) adversarial augmentation via mixup to dynamically leverage existing knowledge from a set of pre-known attacks for robust inference. Intuitively, AdpMixup fine-tunes PLMs with multiple adapters with both clean and pre-known adversarial examples and intelligently mixes them up in different ratios during prediction. Our experiments show AdpMixup achieves the best trade-off between training efficiency and robustness under both pre-known and unknown attacks, compared to existing baselines on five downstream tasks across six varied black-box attacks and 2 PLMs. The code is available at https://github.com/nguyentuc/adapters_mixup.

## introduction peft exemplified by adapter methods, offers a promising solution to mitigate fine-tuning costs for plms. peft involves injecting a small set of parameters into specific locations within a plm, activating only these parameters while freezing the remainder during training. this approach significantly reduces the number of trainable parameters to as little as 0.1% of the original count, while maintaining competitive performance on downstream tasks @xcite . adversarial training includes textual adversarial examples during training adapter mixup fr amewor k pr ofiling test example logits pr ediction decision boundar ies x clean and adv adapter s mixing coefficients figure 1: adpmixup framework: final model θ is achieved by dynamically mixing the adapter weights across clean and adversarial with different coefficients β 1 , β 2 , . . . . the dash red lines are the decision boundaries of different fine-tuning models, that when mixed in a certain way can result in robust inference. to enhance adversarial robustness for neural network models, including plms @xcite @xcite . however, fine-tuning plms via this paradigm to be robust against various types of adversarial perturbations is computationally expensive due to the necessity of independently re-training the models with different perturbations to accommodate different types of attack methods. in addition, adversarial training often decreases performance on clean examples @xcite . to further improve adversarial training, @xcite introduces mixup, which trains a model on virtual examples constructed via linear interpolation between two random examples from the training set and their labels. mixup also helps improve model robustness under various mixing styles, including mixing between original examples, original examples, and their adversarial examples, and between only adversarial examples @xcite . however, mixup shares the same inefficiency with adversarial training in practice as we need to retrain entire models every time we need to accommodate new types of attacks. it is also unknown how mixup can be efficiently ap-plied to fine-tuning plms via peft. in fact, there is a limited number of research addressing plms' generalization capabilities and adversarial robustness when fine-tuned via peft @xcite , not to mention that many existing defense methods are not specifically designed for peft, highlighting a critical gap in the literature. these observations prompt a crucial question: "how can we use peft with plms on downstream tasks that can achieve better trade-off among accuracy, adversarial robustness, and computational complexity and also withstand a variety of new, unknown attack methods?" to answer this question, we seek to investigate how to incorporate adversarial data augmentation training to improve plms' adversarial robustness without sacrificing performance on clean examples, while making minimal changes to complex plms during fine-tuning to minimize computational overhead with peft. to tackle this, this work presents a novel approach, called adpmixup, that combines two key paradigms: (1) fine-tuning through peft, often referred to as adapters @xcite and (2) adversarial augmentation via mixup @xcite . intuitively, adp-mixup fine-tunes plms with multiple adapters with both clean and pre-known adversarial examples and mixes them up in different ratios for robust inference (fig. 1 ). this new adapters mixup paradigm allows adpmixup to work well in practice when the attack methods by which possible adversarial examples are generated are unknown. our contributions are summarized as follows. ## motivation in this section, we demonstrate how data augmenting methods are linked to the model augmentation methods, which underpins the rationale for our adpmixup framework. ## adpmixup: mixup of adapters with adversarial training from § 3, we learned that the mixed model achieved by model soup on the adapters can achieve performance close to the model training with mixup data augmentation. therefore, instead of doing mixup on text samples which is not intuitive in practice, adpmixup allows us to do mixup on the model weight but still preserve the effectiveness of the data augmentation method. let's define two adapters θ clean , θ adv trained on clean and adversarial data, respectively. we have model in clean mode f (•; θ 0 +▽θ clean ), and f (•; θ 0 +▽θ adv ) is the model in adversarial mode. prediction after mixing the two adapters via mixup can then be formulated as: ## conclusion this work provides a new framework for improving model generalization and robustness of plms under adversarial attacks by combining adversarial augmentation via mixup and parameter-efficient fine-tuning via adapters. our findings highlight the utility of adapters in empowering plms to achieve competitive performance in terms of generalization and robustness under both pre-known and unknown adversarial attacks with minimal additional computational complexity. additionally, adpmixup provides extra interpretability into profiling and analyzing potential adversarial examples in practice.