Efficient Adversarial Training with Robust Early-Bird Tickets (2022)

Abstract: Adversarial training is one of the most powerful methods to improve the robustness of pre-trained language models (PLMs). However, this approach is typically more expensive than traditional fine-tuning because of the necessity to generate adversarial examples via gradient descent. Delving into the optimization process of adversarial training, we find that robust connectivity patterns emerge in the early training phase (typically 0.15~0.3 epochs), far before parameters converge. Inspired by this finding, we dig out robust early-bird tickets (i.e., subnetworks) to develop an efficient adversarial training method: (1) searching for robust tickets with structured sparsity in the early stage; (2) fine-tuning robust tickets in the remaining time. To extract the robust tickets as early as possible, we design a ticket convergence metric to automatically terminate the searching process. Experiments show that the proposed efficient adversarial training method can achieve up to 7\times \sim 13 \times training speedups while maintaining comparable or even better robustness compared to the most competitive state-of-the-art adversarial training methods.

## introduction pre-trained language models (plms) have achieved great success in nlp @xcite ), but they are vulnerable to adversarial examples crafted by performing subtle perturbations on normal examples @xcite . recent studies have shown the prevalence of adversarial vulnerability in nlp tasks @xcite @xcite . various defense strategies have been proposed to improve the robustness of the model and maintain high accuracy on both normal and adversarial examples @xcite 2021). adversarial training is one of the most widely used and strongest defense methods @xcite . one of the main challenges of adversarial training in real-world applications is its high computational cost, as they require multi-step gradient descents to generate adversarial examples @xcite . some work on this topic found that replacing strong adversarial examples with weaker and cheaper ones does not significantly change the resulting robustness @xcite . @xcite remove redundant calculations during backpropagation for additional speedup. freeat @xcite and freelb @xcite utilize a "free" strategy to generate diverse adversarial examples at a negligible additional cost. however, these methods are only alleviating the huge expense of generating adversarial samples, and still require optimizing the adversarial loss objective throughout the training period. the recently proposed robust lottery ticket hypothesis @xcite suggests that a deep network contains robust tickets (i.e., subnetworks) that maintain matching accuracy but better robustness than the original network when trained individually. however, robust tickets have to be searched through a tedious process under the guidance of the adversarial loss objective @xcite . moreover, these robust tickets consider unstructured sparsity, which only reduces storage but does not reduce computational overhead @xcite . in this work, we propose a novel robust earlybird ticket with structured sparsity for efficient adversarial training. we delve into the optimization process of adversarial training, and find that the robust connectivity patterns converge with few training iterations. as shown in figure 1 , we can extract robust tickets at the early stage of adversarial training by pruning the self-attention heads and intermediate neurons that contribute least to accuracy and robustness. a highly robust model can be easily obtained by finetuning the proposed robust tickets in the remaining time. in addition, we use a generic ticket convergence metric to automatically terminate the search process without going through each search moment. experimental results show that the proposed method achieves up to 7× ∼ 13× training speedups while maintaining comparable or even better adversarial robustness compared with traditional adversarial training. our codes are publicly available at github 1 . the contributions of this work can be summarized as: ## methodology in this section, we first revisit the robust lottery ticket hypothesis (sec.3.1). next, we describe the early emergence of structured robust tickets (sec.3.2). based on this key finding, we propose the earlyrobust method for efficient adversarial training (sec.3.3). finally, we make a brief review and discuss how our method accelerates adversarial training (sec.3.4). ## experimental results and discussion in this section, we illustrate the effectiveness and efficiency of our method with experimental results. ## conclusion in this paper, we delve into the optimization process of adversarial training on bert, and observe the early convergence of robust connectivity patterns. based on this, we propose earlyrobust, a method that shapes structured robust early-bird tickets under the guidance of the adversarial loss objective and then utilizes efficient fine-tuning to train them. experiments on various datasets demonstrate that our method achieves competitive or even better robustness compared to other strong baselines while maintaining a very low computational and storage consumption.