ROSE : Robust Selective Fine-tuning for Pre-trained Language Models (2022)

Abstract: Even though the large-scale language models have achieved excellent performances, they suffer from various adversarial attacks.A large body of defense methods has been proposed. However, they are still limited due to redundant attack search spaces and the inability to defend against various types of attacks.In this work, we present a novel fine-tuning approach called RO bust SE letive fine-tuning ( ROSE ) to address this issue.ROSE conducts selective updates when adapting pre-trained models to downstream tasks, filtering out invaluable and unrobust updates of parameters.Specifically, we propose two strategies: the first-order and second-order ROSE for selecting target robust parameters.The experimental results show that ROSE achieves significant improvements in adversarial robustness on various downstream NLP tasks, and the ensemble method even surpasses both variants above.Furthermore, ROSE can be easily incorporated into existing fine-tuning methods to improve their adversarial robustness further.The empirical analysis confirms that ROSE eliminates unrobust spurious updates during fine-tuning, leading to solutions corresponding to flatter and wider optima than the conventional method.Code is available at https://github.com/jiangllan/ROSE .

## introduction recently, the discipline of fine-tuning large-scale pre-trained language models has gained prominence, achieving remarkable performances across various natural language processing benchmarks. however, recent studies @xcite @xcite @xcite have highlighted the lack of adversarial robustness in models fine-tuned on specific downstream tasks, i.e., adapted models are vulnerable to various types of adversarial attacks. a majority of adversarial examples fool models via character or word-level perturbations, either those tokens that aren't found in training sets or superficial cues attached to labels @xcite . the vulnerability of adapted models can be attributed to their tendency to capture shallow and spurious patterns when fine-tuning on downstream tasks, instead of utilizing the general linguistic knowledge they have learned during the pre-training stage @xcite @xcite . to address this issue, various defense methods have been proposed, including adversarial training @xcite @xcite , adversarial data augmentation @xcite @xcite and so on. adversarial training and adversarial data augmentation stand to provide the most promising performance among all the defense methods. they enhance adversarial robustness by re-training models with additional adversarial data generated either via human-crafting or by conducting projected gradient ascent on the benign examples. in essence, these methods prevent models from learning misleading features by covering more diverse training data. however, they are limited to practice as they often require prohibitively large attack search spaces, and are not generally applicable to different types of attacks. in this work, we present an attack-agnostic and model-agnostic defense method called robust selective fine-tuning (rose) to address these challenges from a learning perspective. rose is a novel fine-tuning method that conducts robust updates selectively during the fine-tuning stage. the intuition behind our method is straightforward: only robust and informative updates of parameters should be conducted. while the improper ones, which make the fine-tuned model capture superficial cues and overfit the training data of the downstream task, should be filtered out. specifically, we propose two strategies in response to the above challenges: first-order rose and second-order rose. our first-order rose employs adversarial perturbations to clarify parameters that are robust against slight perturbation in the hidden space, enabling models to cope with superficial patterns among examples with similar semantics in the current step. our second-order rose allows models to counter superficial patterns across examples with different semantics along the fine-tuning process by smoothing the optimization trajectory. we also propose an ensemble method to aggregate the benefits of the above two strategies. rose distinguishes parameters based on robust criteria at each step in the backward process, then performs robust updates while freezing the remaining parameters. figure 1 illustrates the loss landscapes of solutions found by different fine-tuning strategies (pretrained initial, vanilla fine-tuning, overfitting and rose-tuning) on specific tasks. rose leads to solutions corresponding to broader and flatter optima compared to traditional fine-tuning methods, which implies that it achieves better adversarial robustness as found in @xcite . moreover, our probing experiment illustrates that rose prefers deeper linguistic features rather than shallow lexical ones during fine-tuning. the above empirical analysis confirms the inner working of rose. rose allows for more robust solutions by masking out unreliable and spurious updates when fine-tuning models on downstream tasks. we conduct extensive experiments to evaluate the effectiveness of rose. we compare rose to several strong defense methods. the results show that rose exhibits superior adversarial robustness on challenging examples, and achieves comparable or even better benign performance on several benchmarks. rose is generic and can be incorporated with existing methods to further enhance their adversarial robustness. ## methodology in this section, we will introduce our method in detail. the key motivation of our method is to select parameters that carry stable information for downstream tasks during fine-tuning. specifically, the vanilla fine-tuning updates all the parameters in the backward process, while rose only updates the robust and informative ones. to identify robust parameters, we propose two robustness criteria and corresponding selective fine-tuning strategies: firstorder rose (section 2.1) and second-order rose (section 2.2). furthermore, we explore an ensemble robust selective fine-tuning method (section 2.3), which aggregates the benefits of the above two strategies. the overall training algorithm of rose when applied to adamw @xcite is shown in algorithm 1. ## analysis in this section, we conduct further analysis to reveal the inner working of rose. ## related work adversarial training is the most effective and promising strategy to improve the adversarial robustness of models. existing adversarial training methods usually employ pdg-based attacks to generate adversarial examples, and force models to maintain the output on them @xcite @xcite . despite the substantial improvements in robustness, adversarial training often requires significant computational and memory costs and fails to preserve the original labels. some works focus on constructing reliable adversarial datasets @xcite , which require huge human annotation and only work for a single task. by contrast, our proposed rose is much more efficient and only employs such perturbations to select robust param-eters to tune, therefore, there is no need for reliable adversarial examples. besides adversarial training methods, our work also relates to a few works of regularization and optimization. in regularization, lots of methods have been proposed, including l 2 -penalty @xcite @xcite , weight decay @xcite , mixout regularization @xcite , and so on. the general approach is augmenting the vanilla optimizer with terms that indirectly or directly penalize the aggressive updates. although these methods are exciting, the regularization is often not scalable, and hard to transfer to another model. another line of work @xcite attempts to address this issue from an informative-theoretic perspective. in optimization, there has been some work proposed recently to force optimization towards wide valleys @xcite . compared to these works, rose uses the simplest idea by selecting parameters with the second-order robustness in fine-tuning stage to smooth the optimization trajectory. rose is more efficient and can be incorporated into existing methods to improve their adversarial robustness further. note that our method does not fall within the realm of model compression. the target of model compression is to obtain an efficient sub-network with competitive performance, with typical approaches to abandon some parameters when models do inference. while rose aims to improve the adversarial robustness of pre-trained language models, which is done via conducting selective parameters updates in the backward process. ## conclusion in this work, we propose an attack-agnostic and model-agnostic defense approach called rose, which selectively updates robust parameters during the fine-tuning stage. we present first-order rose which selects the parameters robust against slight perturbation in the hidden space, secondorder rose which filters out aggressive updates, and ensemble rose which aggregates the benefits of the above two strategies. experimental results show that both our rose-first and rose-second greatly improve the robust performance on various nlp benchmarks, while rose-ensemble is even more effective. besides, existing methods achieve better robustness when incorporated with our rose. we also demonstrate empirically that the effectiveness of rose can be attributed to the wider and flatter solutions it finds than the conventional fine-tuning methods. we hope rose could motivate more defense works for language models.