Original: F REE LB: E NHANCED A DVERSARIAL T RAINING FOR N ATURAL L ANGUAGE U NDERSTANDING
Revised: F REE LB: E NHANCED A DVERSARIAL T RAINING FOR N ATURAL L ANGUAGE U NDERSTANDING

Original: Anonymous authors Paper under double-blind review
Revised: 365 AI Research

Original: A BSTRACT
Revised: A BSTRACT

Original: Adversarial training, which minimizes the maximal risk for label-preserving input perturbations, has proved to be effective for improving the generalization of language models.
Revised: Adversarial training, which minimizes the maximal risk for label-preserving input perturbations, has proved to be effective for improving the generalization of language models.

Original: In this work, we propose a novel adversarial training algorithm, FreeLB that promotes higherrobustness and invariance in the embedding space, by adding adversarial perturbations to word embeddings and minimizing the resultant adversarial risk inside different regions around input samples.
Revised: In this work, we propose a novel adversarial training algorithm, FreeLB, that promotes higher invariance in the embedding space, by adding adversarial perturbations to word embeddings and minimizing the resultant adversarial risk inside different regions around input samples.

Original: To validate the effectiveness of the proposed approach, we apply it to Transformerbased models for natural language understanding and commonsense reasoning tasks.
Revised: To validate the effectiveness of the proposed approach, we apply it to Transformer-based models for natural language understanding and commonsense reasoning tasks.

Original: Experiments on the GLUE benchmark show that when applied only to the ﬁnetuning stage, it is able to improve the overall test scores of BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8.
Revised: Experiments on the GLUE benchmark show that when applied only to the ﬁnetuning stage, it is able to improve the overall test scores of BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8.

Original: In addition, the proposed approach achieves state-of-the-art single-model test accuracies of 85.44% and 67.75% on ARC-Easy and ARC-Challenge.
Revised: In addition, the proposed approach achieves state-of-the-art single-model test accuracies of 85.44% and 67.75% on ARC-Easy and ARC-Challenge.

Original: Experiments on CommonsenseQA benchmark further demonstrate that FreeLB can be generalized and boost the performance of RoBERTa-large model on other tasks as well
Revised: Experiments on CommonsenseQA benchmark further demonstrate that FreeLB can be generalized and boost the performance of RoBERTa-large model on other tasks as well.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Adversarial training is a method for creating robust neural networks.
Revised: Adversarial training is a method for creating robust neural networks.

Original: During adversarial training, mini-batches of training samples are contaminated with adversarial perturbations (alterations that are small and yet cause misclassiﬁcation), and then used to update network parameters until the resulting model learns to resist such attacks.
Revised: During adversarial training, mini-batches of training samples are contaminated with adversarial perturbations (alterations that are small and yet cause misclassiﬁcation), and then used to update network parameters until the resulting model learns to resist such attacks.

Original: Adversarial training was originally proposed as a means to enhance the security of machine learning systems (Goodfellow et al., 2015), especially for safety-critical systems like self-driving cars (Xiao et al., 2018) and copyright detection (Saadatpanah et al., 2019).
Revised: Adversarial training was originally proposed as a means to enhance the security of machine learning systems (Goodfellow et al., 2015), especially for safety-critical systems like self-driving cars (Xiao et al., 2018) and copyright detection (Saadatpanah et al., 2019).

Original: In this paper, we turn our focus away from the security beneﬁts of adversarial training, and instead study its effects on generalization.
Revised: In this paper, we turn our focus away from the security beneﬁts of adversarial training, and instead study its effects on generalization.

Original: While adversarial training boosts the robustness, it is widely accepted by computer vision researchers that it is at odds with generalization, with classiﬁcation accuracy on non-corrupted images dropping as much as 10% on CIFAR-10, and 15% on Imagenet (Madry et al., 2018; Xie et al., 2019).
Revised: While adversarial training boosts the robustness, it is widely accepted by computer vision researchers that it is at odds with generalization, with classiﬁcation accuracy on non-corrupted images dropping as much as 10% on CIFAR-10, and 15% on Imagenet (Madry et al., 2018; Xie et al., 2019).

Original: Surprisingly, people observe the opposite result for language models (Miyato et al., 2017; Cheng et al., 2019), showing that adversarial training can improve both generalization and robustness.
Revised: Surprisingly, people observe the opposite result for language models (Miyato et al., 2017; Cheng et al., 2019), showing that adversarial training can improve both generalization and robustness.

Original: We will show that adversarial training signiﬁcantly improves performance of state-of-the-art models for many language understanding tasks.
Revised: We will show that adversarial training signiﬁcantly improves performance of state-of-the-art models for many language understanding tasks.

Original: In particular, we propose a novel adversarial training algorithm, called FreeLB (Free Large-Batch), which adds adversarial perturbations to word embeddings and minimizes the resultant adversarial loss around input samples.
Revised: In particular, we propose a novel adversarial training algorithm, called FreeLB (Free Large-Batch), which adds adversarial perturbations to word embeddings and minimizes the resultant adversarial loss around input samples.

Original: The method leverages recently proposed “free” training strategies (Shafahi et al., 2019; Zhang et al., 2019) to enlarge the batch size with diversiﬁed adversarial samples under different norm constraints at no extra cost than PGDbased (Projected Gradient Descent) adversarial training (Madry et al., 2018), which enables us to perform such diversiﬁed adversarial training on large-scale state-of-the-art models.
Revised: The method leverages recently proposed “free” training strategies (Shafahi et al., 2019; Zhang et al., 2019) to enrich the training data with diversiﬁed adversarial samples under different norm constraints at no extra cost than PGD-based (Projected Gradient Descent) adversarial training (Madry et al., 2018), which enables us to perform such diversiﬁed adversarial training on large-scale state-of-the-art models.

Original: We observe improvedrobustness and invariance in the embedding space for models trained with FreeLB, which is positively correlated with generalization.
Revised: We observe improved invariance in the embedding space for models trained with FreeLB, which is positively correlated with generalization.

Original: We perform comprehensive experiments to evaluate the performance of a variety of adversarial training algorithms on state-of-the-art language understanding models and tasks.
Revised: We perform comprehensive experiments to evaluate the performance of a variety of adversarial training algorithms on state-of-the-art language understanding models and tasks.

Original: In the comparisons with standard PGD (Madry et al., 2018), FreeAT (Shafahi et al., 2019) and YOPO (Zhang et al., 2019), FreeLB stands out to be the best for the datasets and models we evaluated.
Revised: In the comparisons with standard PGD (Madry et al., 2018), FreeAT (Shafahi et al., 2019) and YOPO (Zhang et al., 2019), FreeLB stands out to be the best for the datasets and models we evaluated.

Original: With FreeLB, we achieve state-of-the-art results on several important language understanding benchmarks.
Revised: With FreeLB, we achieve state-of-the-art results on several important language understanding benchmarks.

Original: On the GLUE benchmark, FreeLB pushes the performance of the BERT-base model from 78.3 to 79.4.
Revised: On the GLUE benchmark, FreeLB pushes the performance of the BERT-base model from 78.3 to 79.4.

Original: The overall score of the RoBERTa-large models on the GLUE benchmark is also lifted from 88.5 to 88.8, achieving best results on most of its sub-tasks.
Revised: The overall score of the RoBERTa-large models on the GLUE benchmark is also lifted from 88.5 to 88.8, achieving best results on most of its sub-tasks.

Original: Experiments also show that FreeLB can boost the performance of RoBERTa-large on question answering tasks, such as the ARC and CommonsenseQA benchmarks.
Revised: Experiments also show that FreeLB can boost the performance of RoBERTa-large on question answering tasks, such as the ARC and CommonsenseQA benchmarks.

Original: We also provide a comprehensive ablation study and analysis to demonstrate the effectiveness of our training process.
Revised: We also provide a comprehensive ablation study and analysis to demonstrate the effectiveness of our training process.

Original:  R ELATED W ORK 2. A DVERSARIAL T RAINING
Revised:  R ELATED W ORK 2. A DVERSARIAL T RAINING

Original: To improve the robustness of neural networks against adversarial examples, many defense strategies and models have been proposed, in which PGD-based adversarial training (Madry et al., 2018) is widely considered to be the most effective, since  largely avoids the the obfuscated gradient problem (Athalye et al., 2018).
Revised: To improve the robustness of neural networks against adversarial examples, many defense strategies and models have been proposed, in which PGD-based adversarial training (Madry et al., 2018) is widely considered to be the most effective, since it largely avoids the the obfuscated gradient problem (Athalye et al., 2018).

Original: It formulates a class of adversarial training algorithms (Kurakin et al., 2017) into solving a minimax problem on the cross-entropy loss, which can be achieved reliably through multiple projected gradient ascent steps followed by a SGD (stochastic gradient descent) step.
Revised: It formulates a class of adversarial training algorithms (Kurakin et al., 2017) into solving a minimax problem on the cross-entropy loss, which can be achieved reliably through multiple projected gradient ascent steps followed by a SGD (Stochastic Gradient Descent) step.

Original: Despite being proved by Athalye et al.
Revised: Despite being veriﬁed by Athalye et al.

Original: (2018) to avoid obfuscated gradients, Qin et al.
Revised: (2018) to avoid obfuscated gradients, Qin et al.

Original: (2019) shows that PGD-based adversarial training still leads to highly convolved and non-linear loss surfaces
Revised: (2019) shows that PGD-based adversarial training still leads to highly convolved and non-linear loss surfaces

Original: when
Revised: when

Original: K is small, which could be readily broken under stronger adversaries.
Revised: K is small, which could be readily broken under stronger adversaries.

Original: Thus, to be effective, the cost of PGD-based adversarial training is much higher than conventional training.
Revised: Thus, to be effective, the cost of PGD-based adversarial training is much higher than conventional training.

Original: To mitigate this cost,
Revised: To mitigate this cost,

Original: Shafahi et al.
Revised: Shafahi et al.

Original: (2019) proposed a “free” adversarial training algorithm that simultaneously updates both model parameters and adversarial perturbations on a single backward pass.
Revised: (2019) proposed a “free” adversarial training algorithm that simultaneously updates both model parameters and adversarial perturbations on a single backward pass.

Original: Using a similar formulation, Zhang et al.
Revised: Using a similar formulation, Zhang et al.

Original: (2019) effectively reduce the total number of full forward and backward propagations  by restricting most of its adversarial updates in the ﬁrst layer
Revised: (2019) effectively reduce the total number of full forward and backward propagations for obtaining adversarial examples by restricting most of its adversarial updates in the ﬁrst layer.

Original: . 2.T EXT A DVERSARIES
Revised: 2. A DVERSARIAL E XAMPLES IN N ATURAL L ANGUAGES

Original: Adversarial examples have been explored primarily in the image domain, and received many attention in text domain recently.
Revised: Adversarial examples have been explored primarily in the image domain, and received many attention in text domain recently.

Original: Previous works on text adversaries have focused on heuristics for creating adversarial examples in the black-box setting, or on speciﬁc tasks.
Revised: Previous works on text adversaries have focused on heuristics for creating adversarial examples in the black-box setting, or on speciﬁc tasks.

Original: Jia & Liang (2017) propose to add distracting sentences to the input document in order to induce mis-classiﬁcation.
Revised: Jia & Liang (2017) propose to add distracting sentences to the input document in order to induce mis-classiﬁcation.

Original: Zhao et al.
Revised: Zhao et al.

Original: (2018) generate text adversaries by projecting the input data to a latent space using GANs, and searching for adversaries close to the original instance.
Revised: (2018) generate text adversaries by projecting the input data to a latent space using GANs, and searching for adversaries close to the original instance.

Original: Belinkov & Bisk (2018) manipulate every word in a sentence with synthetic or natural noise in machine translation systems.
Revised: Belinkov & Bisk (2018) manipulate every word in a sentence with synthetic or natural noise in machine translation systems.

Original: Iyyer et al.
Revised: Iyyer et al.

Original: (2018) propose a neural paraphrase model based on back-translated data to produce paraphrases that have different sentence structures.
Revised: (2018) propose a neural paraphrase model based on back-translated data to produce paraphrases that have different sentence structures.

Original: Different from previous work, ours is not to produce actual adversarial examples, but only take the beneﬁt of adversarial training for natural language understanding.
Revised: Different from previous work, ours is not to produce actual adversarial examples, but only take the beneﬁt of adversarial training for natural language understanding.

Original: We are not the ﬁrst to observe that robust language models may perform better.
Revised: We are not the ﬁrst to observe that robust language models may perform better on clean test data.

Original: Miyato et al.
Revised: Miyato et al.

Original: (2017) extend adversarial and virtual adversarial training (Miyato et al., 2019) to the text domain to improve the performance of semi-supervised classiﬁcation tasks.
Revised: (2017) extend adversarial and virtual adversarial training (Miyato et al., 2019) to the text domain to improve the performance on semi-supervised classiﬁcation tasks.

Original: Ebrahimi et al.
Revised: Ebrahimi et al.

Original: (2018) propose a character/word replacement for craft attacks, and show employing adversarial examples in training renders the models more robust.
Revised: (2018) propose a character/word replacement for crafting attacks, and show employing adversarial examples in training renders the models more robust.

Original: Ribeiro et al.
Revised: Ribeiro et al.

Original: (2018) show that adversarial attacks can be used as a valuable tool for debugging NLP models.
Revised: (2018) show that adversarial attacks can be used as a valuable tool for debugging NLP models.

Original: Cheng et al.
Revised: Cheng et al.

Original:  also ﬁnd the craft adversarial examples can help neural machine translation signiﬁcantly.
Revised: (2019) also ﬁnd that crafting adversarial examples can help neural machine translation signiﬁcantly.

Original: Notably, these studies have focused on simple models or text generation tasks.
Revised: Notably, these studies have focused on simple models or text generation tasks.

Original: Our work exploresdeeper/larger Transformer-based models, how to adversarially attack them efﬁciently, and how to improve performance on challenging problems.
Revised: Our work explores how to efﬁciently use the gradients obtained in adversarial training to boost the performance of state-of-the-art transformer-based models.

Original:  A DVERSARIAL T RAINING FOR L ANGUAGE U NDERSTANDING
Revised:  A DVERSARIAL T RAINING FOR L ANGUAGE U NDERSTANDING

Original: Pre-trained large-scale language models, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b),  have proven to be highly effective for downstream tasks.
Revised: Pre-trained large-scale language models, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), ALBERT (Lan et al., 2020) and T5 (Raffel et al., 2019), have proven to be highly effective for downstream tasks.

Original: We aim to further improve the generalization of these pre-trained language models on the downstream language understanding tasks by enhancing their robustness in the embedding space during ﬁnetuning on these tasks.
Revised: We aim to further improve the generalization of these pre-trained language models on the downstream language understanding tasks by enhancing their robustness in the embedding space during ﬁnetuning on these tasks.

Original: We achieve this goal by creating “virtual” adversarial examples in the embedding space, and then perform parameter updates on these adversarial embeddings.
Revised: We achieve this goal by creating “virtual” adversarial examples in the embedding space, and then perform parameter updates on these adversarial embeddings.

Original: Creating actual adversarial examples for language is difﬁcult; even with state-of-the-art language models as guidance (e.g., Cheng et al. , it remains unclear how to construct label-preserving adversarial examples via word/character replacement without human evaluations, since the meaning of each word/character depends on the context Ribeiro et al.
Revised: Creating actual adversarial examples for language is difﬁcult; even with state-of-theart language models as guidance (e.g., (Cheng et al., 2019)), it remains unclear how to construct label-preserving adversarial examples via word/character replacement without human evaluations, because the meaning of each word/character depends on the context (Ribeiro et al., 2018).

Original: Since we are only interested in the effects of adversarial training, rather than producing actual adversarial examples, we add norm-bounded adversarial perturbations to the embeddings of the input sentences using a gradient-based method.
Revised: Since we are only interested in the effects of adversarial training, rather than producing actual adversarial examples, we add norm-bounded adversarial perturbations to the embeddings of the input sentences using a gradient-based method.

Original: Note that our embedding-based adversary is strictly stronger than a more conventional text-based adversary, as our adversary can make manipulations on word embeddings that are not possible in the text domain.
Revised: Note that our embedding-based adversary is strictly stronger than a more conventional text-based adversary, as our adversary can make manipulations on word embeddings that are not possible in the text domain.

Original: For models that encode text at the sub-word level using Byte-Pair Encoding (BPE) (Sennrich et al., 2016), our adversaries only modify the concatenated BPE embeddings, leaving other components of the sentence representation(e.g., position encoding) unchanged.
Revised: For models that incorporate various input representations, including word or subword embeddings, segment embeddings and position embeddings, our adversaries only modify the concatenated word or sub-word embeddings, leaving other components of the sentence representation unchanged.

Original: Denote the sequence of one-hot representations of the input subwords as Z =
Revised: Denote the sequence of one-hot representations of the input subwords as Z =

Original: [ z 1 , z 2 , ..., z n ] , the embedding matrix as V , and the language model (encoder) as a function y =
Revised: [ z 1 , z 2 , ..., z n ] , the embedding matrix as V , and the language model (encoder) as a function y =

Original: f θ ( X ) , where X = V Z , y is the output of the model, and θ denotes all the learnable parameters including the embedding matrix V .
Revised: f θ ( X ) , where X = V Z is the subword embeddings, y is the output of the model (e.g., class probabilities for classiﬁcation models), and θ denotes all the learnable parameters including the embedding matrix V .

Original: We add adversarial perturbations δ to the embeddings such that the prediction becomes y (cid:48)
Revised: We add adversarial perturbations δ to the embeddings such that the prediction becomes y (cid:48)

Original: =
Revised: =

Original: f θ ( X + δ ) .
Revised: f θ ( X + δ ) .

Original: To preserve the semantics, we constrain the norm of δ to be small, and assume the model’s prediction does not change after the perturbation.
Revised: To preserve the semantics, we constrain the norm of δ to be small, and assume the model’s prediction should not change after the perturbation.

Original: This formulation is analogous to Miyato et al.
Revised: This formulation is analogous to Miyato et al.

Original: (2017), with the difference that we do not require X to be normalized.
Revised: (2017), with the difference that we do not require X to be normalized.

Original: 3. PGD FOR A DVERSARIAL T RAINING
Revised: 3. PGD FOR A DVERSARIAL T RAINING

Original: Standard adversarial training seeks to ﬁnd a set of parameters θ ∗ to minimize the maximum risk for any δ within a norm ball as:
Revised: Standard adversarial training seeks to ﬁnd optimal parameters θ ∗ to minimize the maximum risk for any δ within a norm ball as:

Original: [Equation]
Revised: [Equation]

Original: where D is the data distribution, y is the label, and L is some loss function.
Revised: where D is the data distribution, y is the label, and L is some loss function.

Original: We use the Frobenius norm to constrain δ .
Revised: We use the Frobenius norm to constrain δ .

Original: For neural networks, the outer “min” is non-convex, and the inner “max” is non-concave.
Revised: For neural networks, the outer “min” is non-convex, and the inner “max” is non-concave.

Original: Nonetheless, Madry et al.
Revised: Nonetheless, Madry et al.

Original: (2018) demonstrated that this saddle-point problem can be solved reliably with SGD for the outer minimization and PGD (a standard method for large-scale constrained optimization, see Combettes & Pesquet (2011) and Goldstein et al. (2014)), for the inner maximization.
Revised: (2018) demonstrated that this saddle-point problem can be solved reliably with SGD for the outer minimization and PGD (a standard method for large-scale constrained optimization, see (Combettes & Pesquet, 2011) and (Goldstein et al., 2014)), for the inner maximization.

Original: In particular, for the constraint (cid:107) δ (cid:107) F ≤ (cid:15) , with an additional assumption that the loss function is locally linear, PGD takes the following step (with step size α ) in each iteration:
Revised: In particular, for the constraint (cid:107) δ (cid:107) F ≤ (cid:15) , with an additional assumption that the loss function is locally linear, PGD takes the following step (with step size α ) in each iteration:

Original: [Equation]
Revised: [Equation]

Original: where g ( δ t ) = ∇ δ L ( f θ ( X + δ t ) , y ) is the gradient of the loss with respect to δ , and Π [Equation] performs a projection onto the (cid:15) -ball.
Revised: where g ( δ t ) = ∇ δ L ( f θ ( X + δ t ) , y ) is the gradient of the loss with respect to δ , and Π (cid:107) δ (cid:107) F ≤ (cid:15) performs a projection onto the (cid:15) -ball.

Original: To achieve high-level robustness, multi-step adversarial examples are needed during training, which is computationally expensive.
Revised: To achieve high-level robustness, multi-step adversarial examples are needed during training, which is computationally expensive.

Original: The K -step PGD ( K -PGD) requires K forward-backward passes through the network, while the standard SGD update requires only one.
Revised: The K -step PGD ( K -PGD) requires K forward-backward passes through the network, while the standard SGD update requires only one.

Original: As a result, the adversary generation step in adversarial training increases run-time by an order of magnitude – a catastrophic amount when training large state-of-the-art language models.
Revised: As a result, the adversary generation step in adversarial training increases run-time by an order of magnitude—a catastrophic amount when training large state-of-the-art language models.

Original: 3. L ARGE -B ATCH A DVERSARIAL T RAINING FOR F REE
Revised: 3. L ARGE -B ATCH A DVERSARIAL T RAINING FOR F REE

Original: In the inner ascent steps of PGD, the gradients of the parameters can be obtained with almost no overhead when computing the gradients of the inputs.
Revised: In the inner ascent steps of PGD, the gradients of the parameters can be obtained with almost no overhead when computing the gradients of the inputs.

Original: From this observation, FreeAT (Shafahi et al.,2019) and YOPO (Zhang et al., 2019) have been proposed to accelerate adversarial training. They achieve comparable robustness and generalization as standard PGD-trained models using only the same or a slightly larger number of forward-backward passes as natural training (i.e., SGD on clean samples). FreeAT takes one descent step on the parameters together with each of the K ascent steps on the perturbation. As a result, when the optimal δ is highly correlated with θ , δ t is sub-optimal for maximizing L ( f θ t ( X + δ ) , y ) , since the ascent step from δ t − 1 to δ t is based on the gradient ∇ θ L ( f θ t − 1 ( X + δ t − 1 ) , y ) at θ t − 1 .
Revised: From this observation, FreeAT (Shafahi et al.,

Original: Different from FreeAT, YOPO accumulates the gradient of the parameters from each of the ascent steps, and updates the parameters only once after the K inner ascent steps.

Original: YOPO also advocates that after each back-propagation, one should take the gradient of the ﬁrst hidden layer as a constant and perform several additional updates on the adversary using the product of this constant and the Jacobian of the ﬁrst layer of the network.

Original: Interestingly, the analysis backing the extra update steps assumes a twice continuously differentiable loss, which does not

Original: Require:
Revised: Require:

Original: Training samples X
Revised: Training samples X

Original: = { ( Z , y ) } , perturbation bound (cid:15) , learning rate τ , ascent steps K , ascent step size α 1: Initialize θ 2: for epoch
Revised: = { ( Z , y ) } , perturbation bound (cid:15) , learning rate τ , ascent steps K , ascent step size α 1: Initialize θ 2: for epoch

Original: = 1 . . .
Revised: = 1 . . .

Original: N ep do 3: for minibatch B ⊂ X do 4: δ 0 ← 1 √ N θ U ( − (cid:15), (cid:15) ) 5: g 

Revised: N ep do 3: for minibatch B ⊂ X do 4: δ 0 ← 1 √ N δ U ( − (cid:15), (cid:15) ) 5: g

Original: ←6: for t = 1 . . .
Revised: ←6: for t = 1 . . .

Original: K do 7:
Revised: K do 7:

Original: Accumulate gradient of parameters θ 8: g t ← g t − 1 + 1 K E ( Z ,y ) ∈ B [ ∇ θ L ( f θ ( X + δ t − 1 ) , y )] 9:
Revised: Accumulate gradient of parameters θ 8: g t ← g t − 1 + 1 K E ( Z ,y ) ∈ B [ ∇ θ L ( f θ ( X + δ t − 1 ) , y )] 9:

Original: Update the perturbation δ via gradient ascend 10: g adv ← ∇ δ L ( f θ ( X + δ t − 1 ) , y ) 11: δ t ← Π (cid:107) δ (cid:107) F ≤ (cid:15) ( δ t − 1 + α · g adv / (cid:107) g adv (cid:107) F ) 12: end for 13: θ ← θ − τ g K 14: end for 15: end for hold for ReLU-based neural networks they experimented with, and thus the reasons for the success of such an algorithm remains obscure.
Revised: Update the perturbation δ via gradient ascend 10: g adv ← ∇ δ L ( f θ ( X + δ t − 1 ) , y ) 11: δ t ← Π (cid:107) δ (cid:107) F ≤ (cid:15) ( δ t − 1 + α · g adv / (cid:107) g adv (cid:107) F ) 12: end for 13: θ ← θ − τ g K 14: end for 15: end for 2019) and YOPO (Zhang et al., 2019) have been proposed to accelerate adversarial training. They achieve comparable robustness and generalization as standard PGD-trained models using only the same or a slightly larger number of forward-backward passes as natural training (i.e., SGD on clean samples). FreeAT takes one descent step on the parameters together with each of the K ascent steps on the perturbation. As a result, FreeAT may suffer from the “stale gradient” problem (Dutta et al., 2018), where in every step t , δ t does not necessarily maximize the model with parameter θ t since its update is based on ∇ δ L ( f θ t − 1 ( X + δ t − 1 ) , y ) , and vice versa, θ t does not necessarily minimize the adversarial risk with adversary δ t since its update is based on ∇ θ L ( f θ t − 1 ( X + δ t − 1 ) , y ) .

Revised: Such a problem may be more signiﬁcant when the step size is large.

Original: We give empirical comparisons between YOPO  and our approach in Sec. 4.3.
Revised: Different from FreeAT, YOPO accumulates the gradient of the parameters from each of the ascent steps, and updates the parameters only once after the K inner ascent steps. YOPO also advocates that after each back-propagation, one should take the gradient of the ﬁrst hidden layer as a constant and perform several additional updates on the adversary using the product of this constant and the Jacobian of the ﬁrst layer of the network to obtain strong adversaries. However, when the ﬁrst hidden layer is a linear layer as in their implementation, such an operation is equivalent to taking a larger step size on the adversary.

Revised: The analysis backing the extra update steps also assumes a twice continuously differentiable loss, which does not hold for ReLU-based neural networks they experimented with, and thus the reasons for the success of such an algorithm remains obscure.

Revised: We give empirical comparisons between YOPO and our approach in Sec. 4.3.

Original: To obtain better solutions for the inner max and avoid fundamental limitations on the function class, we propose FreeLB, which performs multiple PGD iterations to craft adversarial examples, and simultaneously accumulates the “free” parameter gradients ∇ θ L in each iteration.
Revised: To obtain better solutions for the inner max and avoid fundamental limitations on the function class, we propose FreeLB, which performs multiple PGD iterations to craft adversarial examples, and simultaneously accumulates the “free” parameter gradients ∇ θ L in each iteration.

Original: After that, it updates the model parameter θ all at once with the accumulated gradients.
Revised: After that, it updates the model parameter θ all at once with the accumulated gradients.

Original: The overall procedure is shown in Algorithm 1, in which X + δ t is an approximation to the local maximum within the intersection of two balls I t =
Revised: The overall procedure is shown in Algorithm 1, in which X + δ t is an approximation to the local maximum within the intersection of two balls I t =

Original: B X
Revised: B X

Original: + δ 0 ( αt ) ∩ B X
Revised: + δ 0 ( αt ) ∩ B X

Original: (cid:15) ) .
Revised: (cid:15) ) .

Original: By taking a descent step along the averaged gradients at X + δ 0 , ..., X + δ K − 1 , we approximately optimize the following objective:
Revised: By taking a descent step along the averaged gradients at X + δ 0 , ..., X + δ K − 1 , we approximately optimize the following objective:

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to replacing the original batch X with a K -times larger virtual batch, consisting of samples whose embeddings are X + δ 0 , ...
Revised: which is equivalent to replacing the original batch X with a K -times larger virtual batch, consisting of samples whose embeddings are X + δ 0 , ...

Original: , X + δ K − 1 .
Revised: , X + δ K − 1 .

Original: Compared with PGD-based adversarial training (Eq. 1), which minimizes the maximum risk at a single estimated point in the vicinity of each training sample, FreeLB minimizes the maximum risk at each ascent step at almost no overhead.
Revised: Compared with PGD-based adversarial training (Eq. 1), which minimizes the maximum risk at a single estimated point in the vicinity of each training sample, FreeLB minimizes the maximum risk at each ascent step at almost no overhead.

Original: Intuitively, FreeLB could be a learning method with lower generalization error than PGD.
Revised: Intuitively, FreeLB could be a learning method with lower generalization error than PGD.

Original: Sokolic et al.
Revised: Sokolic et al.

Original: (2017) have proved that the generalization error of a learning method invariant to a set of
Revised: (2017) have proved that the generalization error of a learning method invariant to a set of

Original: T transformations may be up to √ T smaller than a non-invariant learning method.
Revised: T transformations may be up to √ T smaller than a non-invariant learning method.

Original: According to their theory, FreeLB could have a more signiﬁcant improvement over natural training, since FreeLB enforces the invariance to K adversaries from a set of up to K different norm constraints, 1 while
Revised: According to their theory, FreeLB could have a more signiﬁcant improvement over natural training, since FreeLB enforces the invariance to K adversaries from a set of up to K different norm constraints, 3 while

Original: PGD only enforces invariance to a single norm constraint (cid:15) .
Revised: PGD only enforces invariance to a single norm constraint (cid:15) .

Original: Empirically, FreeLB does lead to higher robustness and invariance than PGD in the embedding space, in the sense that the maximum increase of loss in the vicinity of X for models trained with
Revised: Empirically, FreeLB does lead to higher robustness and invariance than PGD in the embedding space, in the sense that the maximum increase of loss in the vicinity of X for models trained with

Original: FreeLB is smaller than that with PGD.
Revised: FreeLB is smaller than that with PGD.

Original: See Sec.
Revised: See Sec.

Original: 4.3 for details.
Revised: 4.3 for details.

Original: In theory, such improved robustness can lead to better generalization (Xu & Mannor, 2012), which is consistent with our experiments.
Revised: In theory, such improved robustness can lead to better generalization (Xu & Mannor, 2012), which is consistent with our experiments.

Original: Qin et al.
Revised: Qin et al.

Original: (2019) also demonstrated that PGD-based method leads to highly convolved and non-linear loss surfaces in the vicinity of input samples when K is small, indicating a lack of robustness.
Revised: (2019) also demonstrated that PGD-based method leads to highly convolved and non-linear loss surfaces in the vicinity of input samples when K is small, indicating a lack of robustness.

Original: 3. W HEN A DVERSARIAL T RAINING M EETS D ROPOUT
Revised: 3. W HEN A DVERSARIAL T RAINING M EETS D ROPOUT

Original: Usually, adversarial training is not used together with dropout (Srivastava et al., 2014).
Revised: Usually, adversarial training is not used together with dropout (Srivastava et al., 2014).

Original: However, for some language models like RoBERTa (Liu et al., 2019b), dropout is used during the ﬁnetuning stage.
Revised: However, for some language models like RoBERTa (Liu et al., 2019b), dropout is used during the ﬁnetuning stage.

Original: In practice, when dropout is turned on, each ascent step of Algorithm 1 is optimizing δ for a different network.
Revised: In practice, when dropout is turned on, each ascent step of Algorithm 1 is optimizing δ for a different network.

Original: Speciﬁcally, denote the dropout mask as m with each entry m i ∼ Bernoulli ( p ) .
Revised: Speciﬁcally, denote the dropout mask as m with each entry m i ∼ Bernoulli ( p ) .

Original: Similar to our analysis for FreeAT, the ascent step from δ t − 1 to δ t is based on ∇ δ L ( f θ ( m t − 1 )
Revised: Similar to our analysis for FreeAT, the ascent step from δ t − 1 to δ t is based on ∇ δ L ( f θ ( m t − 1 )

Original: ( X + δ t − 1 ) , y ) , so δ t is sub-optimal for L ( f θ ( m t )
Revised: ( X + δ t − 1 ) , y ) , so δ t is sub-optimal for L ( f θ ( m t )

Original: ( X + δ ) , y ) if θ and δ are highly correlated.
Revised: ( X + δ ) , y ) .

Original: Here θ ( m ) is the effective parameters under dropout mask m .
Revised: Here θ ( m ) is the effective parameters under dropout mask m .

Original: The more plausible solution is to use the same m in each step.
Revised: The more plausible solution is to use the same m in each step.

Original: When applying dropout to any network, the objective for θ is to minimize the expectation of loss under different networks determined by the dropout masks, which is achieved by minimizing the Monte Carlo estimation of the expected loss.
Revised: When applying dropout to any network, the objective for θ is to minimize the expectation of loss under different networks determined by the dropout masks, which is achieved by minimizing the Monte Carlo estimation of the expected loss.

Original: In our case, the objective becomes:
Revised: In our case, the objective becomes:

Original: [Equation]
Revised: [Equation]

Original: where the 1-sample Monte Carlo estimation should be 1 K (cid:80) 
Revised: where the 1-sample Monte Carlo estimation should be 1 K (cid:80) K −

Original: [Equation]
Revised: [Equation]

Revised: =0 max

Revised: [Equation]

Revised: ∈I

Revised: [Equation]

Revised: L ( f θ ( m 0 )

Revised: ( X + δ t ) , y ) and can be minimized by using FreeLB with dropout mask m 0 in each ascent step.

Original: This is similar to applying Variational Dropout to RNNs as used in Gal & Ghahramani (2016).
Revised: This is similar to applying Variational Dropout to RNNs as used in Gal & Ghahramani (2016).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we provide comprehensive analysis on FreeLB through extensive experiments on three NLP benchmarks: GLUE (Wang et al., 2019), ARC (Clark et al., 2018) and CommonsenseQA (Talmor et al., 2019).
Revised: In this section, we provide comprehensive analysis on FreeLB through extensive experiments on three Language Understanding benchmarks: GLUE (Wang et al., 2019), ARC (Clark et al., 2018) and CommonsenseQA (Talmor et al., 2019).

Original: We also compare the robustness and generalization of FreeLB with other adversarial training algorithms to demonstrate its strength.
Revised: We also compare the robustness and generalization of FreeLB with other adversarial training algorithms to demonstrate its strength.

Original: Additional experimental details are provided in the Appendix.
Revised: Additional experimental details are provided in the Appendix.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: GLUE Benchmark.
Revised: GLUE Benchmark.

Original: The GLUE benchmark is a collection of 9 natural language understanding tasks, namely Corpus of Linguistic Acceptability (CoLA; Warstadt et al.
Revised: The GLUE benchmark is a collection of 9 natural language understanding tasks, namely Corpus of Linguistic Acceptability (CoLA; Warstadt et al.

Original: Stanford Sentiment Treebank (SST; Socher et al. (2013)), Microsoft Research Paraphrase Corpus (MRPC; Dolan & Brockett (2005)), Semantic Textual Similarity Benchmark (STS; Agirre et al.
Revised: Stanford Sentiment Treebank (SST; Socher et al. (2013)), Microsoft Research Paraphrase Corpus (MRPC; Dolan & Brockett (2005)), Semantic Textual Similarity Benchmark (STS; Agirre et al.

Original: , Quora
Revised: , Quora

Original: Question Pairs (QQP; Iyer et al. (2017)), Multi-Genre NLI (MNLI; Williams et al.
Revised: Question Pairs (QQP; Iyer et al. (2017)), Multi-Genre NLI (MNLI; Williams et al.

Original: , Question
Revised: , Question

Original: NLI (QNLI; Rajpurkar et al. (2016)),
Revised: NLI (QNLI; Rajpurkar et al. (2016)),

Original: Recognizing Textual Entailment (RTE; Dagan et al.
Revised: Recognizing Textual Entailment (RTE; Dagan et al.

Original: Bar Haim et al.
Revised: Bar Haim et al.

Original: ; Giampiccolo et al.
Revised: ; Giampiccolo et al.

Original: ; Bentivogli et al.
Revised: ; Bentivogli et al.

Original: and Winograd NLI (WNLI; Levesque et al. (2011)).
Revised: and Winograd NLI (WNLI; Levesque et al. (2011)).

Original: 8 of the tasks are formulated as classiﬁcation problems and only
Revised: 8 of the tasks are formulated as classiﬁcation problems and only

Original: STS-B is formulated as regression, but FreeLB applies to all of them.
Revised: STS-B is formulated as regression, but FreeLB applies to all of them.

Original: For BERT-base, we use the
Revised: For BERT-base, we use the

Original: HuggingFace implementation 2 , and follow the single-task ﬁnetuning procedure as in Devlin et al.
Revised: HuggingFace implementation 4 , and follow the single-task ﬁnetuning procedure as in Devlin et al.

Original: For RoBERTa, we use the fairseq implementation 3 .
Revised: For RoBERTa, we use the fairseq implementation 5 .

Original: Same as Liu et al.
Revised: Same as Liu et al.

Revised: (2019b), we also use

Revised: [Table]

Revised: [Table]

Original: (2019b), we also use single-task ﬁnetuning for all devset results, and start with MNLI-ﬁnetuned models on RTE, MRPC and STS-B for the test submissions.
Revised: 2019b). single-task ﬁnetuning for all dev results, and start MNLI ﬁnetuned models RTE

Revised: MRPC

Revised: STS submissions.

Original: ARC Benchmark.
Revised: ARC Benchmark.

Original: The ARC dataset (Clark et al., 2018) is a collection of multi-choice science questions from grade-school level exams.
Revised: The ARC dataset (Clark et al., 2018) is a collection of multi-choice science questions from grade-school level exams.

Original: It is further divided into ARC-Challenge set with 2, question answer (QA) pairs and ARC-Easy set with 5,197 QA pairs.
Revised: It is further divided into ARC-Challenge set with 2, question answer (QA) pairs and ARC-Easy set with 5,197 QA pairs.

Original: Questions in ARC-Challenge are more difﬁcult and cannot be handled by simply using a retrieval and co-occurence based
Revised: Questions in ARC-Challenge are more difﬁcult and cannot be handled by simply using a retrieval and co-occurence based

Original: algorithm (Clark et al., 2018).
Revised: algorithm (Clark et al., 2018).

Original: A typical question is:
Revised: A typical question is:

Original: Which property of a mineral can be determined just by looking at it?
Revised: Which property of a mineral can be determined just by looking at it?

Original: (A) luster [correct] (B) mass (C) weight (D) hardness .
Revised: (A) luster [correct] (B) mass (C) weight (D) hardness .

Original: CommonsenseQA Benchmark.
Revised: CommonsenseQA Benchmark.

Original: The CommonsenseQA dataset (Talmor et al., 2019) consists of 12,102 natural language questions that require human commonsense reasoning ability to answer.
Revised: The CommonsenseQA dataset (Talmor et al., 2019) consists of 12,102 natural language questions that require human commonsense reasoning ability to answer.

Original: A typical question is :
Revised: A typical question is :

Original: Where

Original: can I

Original:  stand on a river to see water falling without getting wet
Revised: Where can I stand on a river to see water falling without getting wet?

Original: (A) waterfall, (B) bridge [correct] , (C) valley, (D) stream, (E) bottom.2 https [Table]
Revised: (A) waterfall, (B) bridge [correct] , (C) valley, (D) stream, (E) bottom.

Original: [Table] [Table] [Table] Each question has ﬁve candidate answers from ConceptNet (Speer et al., 2017).
Revised: Each question has ﬁve candidate answers from ConceptNet (Speer et al., 2017).

Original: To make the question more difﬁcult to solve, most answers have the same relation in ConceptNet to the key concept in the question.
Revised: To make the question more difﬁcult to solve, most answers have the same relation in ConceptNet to the key concept in the question.

Original: As shown in the above example, most answers can be connected to “river” by “AtLocation” relation in ConceptNet.
Revised: As shown in the above example, most answers can be connected to “river” by “AtLocation” relation in ConceptNet.

Original: For a fair comparison with the reported results in papers and leaderboard 4 , we use the ofﬁcial random split 1.11.
Revised: For a fair comparison with the reported results in papers and leaderboard 6 , we use the ofﬁcial random split 1.11.

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: GLUE We summarize results on the dev sets of GLUE in Table 1, comparing the proposed FreeLB against other adversatial training algorithms (PGD (Madry et al., 2018) and FreeAT (Shafahi et al., 2019)).
Revised: GLUE We summarize results on the dev sets of GLUE in Table 1, comparing the proposed FreeLB against other adversatial training algorithms (PGD (Madry et al., 2018) and FreeAT (Shafahi et al., 2019)).

Original: We use the same step size α and number of steps m for PGD, FreeAT and FreeLB.
Revised: We use the same step size α and number of steps m for PGD, FreeAT and FreeLB.

Original: FreeLB is consistently better than the two baselines.
Revised: FreeLB is consistently better than the two baselines

Original: Comparisons and detailed discussions about
Revised: [Table]

Original: YOPO (Zhang et al., 2019) are provided in Sec. 4.3.
Revised: YOPO (Zhang et al., 2019) are provided in Sec. 4.3.

Original: We have also submitted our results to the evaluation server, results provided in Table 2.
Revised: We have also submitted our results to the evaluation server, results provided in Table 2.

Original: FreeLB lifts the performance of the BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8 on overall scores.
Revised: FreeLB lifts the performance of the BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8 on overall scores.

Original: ARC For ARC, a corpus of 14 million related science documents (from ARC Corpus, Wikipedia and other sources) is provided.
Revised: ARC For ARC, a corpus of 14 million related science documents (from ARC Corpus, Wikipedia and other sources) is provided.

Original: For each QA pair, we ﬁrst use a retrieval model to select toprelated documents.
Revised: For each QA pair, we ﬁrst use a retrieval model to select toprelated documents.

Original: Then, given these retrieved documents 5 , we use RoBERTa-large model to encode (cid:104) s (cid:105)
Revised: Then, given these retrieved documents 7 , we use RoBERTa-large model to encode (cid:104) s (cid:105)

Original: Retrieved Documents (cid:104) /s
Revised: Retrieved Documents (cid:104) /s

Original: (cid:105) Question + Answer (cid:104) /s
Revised: (cid:105) Question + Answer (cid:104) /s

Original: (cid:105) , where (cid:104) s (cid:105) and (cid:104) /s
Revised: (cid:105) , where (cid:104) s (cid:105) and (cid:104) /s

Original: (cid:105) are special tokens for
Revised: (cid:105) are special tokens for

Original: RoBERTa model 6 .
Revised: RoBERTa model 8 .

Original: We then apply a fully-connected layer to the representation of the [CLS] token to compute the ﬁnal logit, and use standard cross-entropy loss for model training.
Revised: We then apply a fully-connected layer to the representation of the [CLS] token to compute the ﬁnal logit, and use standard cross-entropy loss for model training.

Original: Results are summarized in Table 3.
Revised: Results are summarized in Table 3.

Original: Following Sun et al.
Revised: Following Sun et al.

Original: (2018), we ﬁrst ﬁnetune the RoBERTa model on the RACE dataset (Lai et al., 2017).
Revised: (2018), we ﬁrst ﬁnetune the RoBERTa model on the RACE dataset (Lai et al., 2017).

Original: The ﬁnetuned RoBERTa model achieves 85.70% and 85.24% accuracy on the development and test set of RACE, respectively.
Revised: The ﬁnetuned RoBERTa model achieves 85.70% and 85.24% accuracy on the development and test set of RACE, respectively.

Original: Based on this, we further ﬁnetune the model on both ARC-Easy and ARC-Challenge datasets with the same hyper-parameter searching strategy (for 5 epochs), which achieves 84.13%/64.44% test accuracy on ARC-Easy/ARC- [Table] [Table] Challenge.
Revised: Based on this, we further ﬁnetune the model on both ARC-Easy and ARC-Challenge datasets with the same hyper-parameter searching strategy (for 5 epochs), which achieves 84.13%/64.44% test accuracy on ARC-Easy/ARCChallenge.

Original: And by adding FreeLB ﬁnetuning, we can reach 84.81%/65.36%, a signiﬁcant boost on
Revised: And by adding FreeLB ﬁnetuning, we can reach 84.81%/65.36%, a signiﬁcant boost on

Original: ARC benchmark, demonstrating the effectiveness of FreeLB.
Revised: ARC benchmark, demonstrating the effectiveness of FreeLB.

Original: To further improve the results, we apply a multi-task learning (MTL) strategy using additional datasets.
Revised: To further improve the results, we apply a multi-task learning (MTL) strategy using additional datasets.

Original: We ﬁrst ﬁnetune the model on RACE (Lai et al., 2017), and then ﬁnetune on a joint dataset of ARC-Easy, ARC-Challenge, OpenbookQA (Mihaylov et al., 2018) and Regents Living
Revised: We ﬁrst ﬁnetune the model on RACE (Lai et al., 2017), and then ﬁnetune on a joint dataset of ARC-Easy, ARC-Challenge, OpenbookQA (Mihaylov et al., 2018) and Regents Living

Original: Environment 7 .
Revised: Environment 9 .

Original: Based on this, we further ﬁnetune our model on ARC-Easy and ARC-Challenge with
Revised: Based on this, we further ﬁnetune our model on ARC-Easy and ARC-Challenge with

Original: FreeLB.
Revised: FreeLB.

Original: After ﬁnetuning, our single model achieves 67.75% test accuracy on ARC-Challenge and 85.44% on ARC-Easy, both outperforming the best submission on the ofﬁcial leaderboard 8 .
Revised: After ﬁnetuning, our single model achieves 67.75% test accuracy on ARC-Challenge and 85.44% on ARC-Easy, both outperforming the best submission on the ofﬁcial leaderboard 10 .

Original: CommonsenseQA Similar to the training strategy in Liu et al.
Revised: CommonsenseQA Similar to the training strategy in Liu et al.

Original: (2019b), we construct ﬁve inputs for each question by concatenating the question and each answer separately, then encode each input with the representation of the [CLS] token.
Revised: (2019b), we construct ﬁve inputs for each question by concatenating the question and each answer separately, then encode each input with the representation of the [CLS] token.

Original: A ﬁnal score is calculated by applying the representation of [CLS] to a fully-connected layer.
Revised: A ﬁnal score is calculated by applying the representation of [CLS] to a fully-connected layer.

Original: Following the fairseq repository 9 , the input is formatted as: ” (cid:104) s (cid:105) Q: Where can I stand on a river to see water falling without getting wet?
Revised: Following the fairseq repository 11 , the input is formatted as: ” (cid:104) s (cid:105) Q: Where can I stand on a river to see water falling without getting wet?

Original: (cid:104) /s
Revised: (cid:104) /s

Original: (cid:105)
Revised: (cid:105)

Original: A: waterfall (cid:104) /s
Revised: A: waterfall (cid:104) /s

Original: (cid:105) ”, where ’ Q: ’ and ’ A: ’ are the preﬁx for question and answer, respectively.
Revised: (cid:105) ”, where ’ Q: ’ and ’ A: ’ are the preﬁx for question and answer, respectively.

Original: Results are summarized in Table 3.
Revised: Results are summarized in Table 3.

Original: We obtained a dev-set accuracy of 77.56% with the RoBERTalarge model.
Revised: We obtained a dev-set accuracy of 77.56% with the RoBERTalarge model.

Original: When using FreeLB ﬁnetuning, we achieved 78.81%, a 1.25% absolute gain.
Revised: When using FreeLB ﬁnetuning, we achieved 78.81%, a 1.25% absolute gain.

Original: Compared with the results reported from fairseq repository, which obtains 78.43% accuracy on the devset, FreeLB still achieves better performance.
Revised: Compared with the results reported from fairseq repository, which obtains 78.43% accuracy on the devset, FreeLB still achieves better performance.

Original: Our submission to the CommonsenseQA leaderboard achieves 72.2% single-model test set accuracy, and the result of a 20-model ensemble is 73.1%, which achieves No.1 among all the submissions without making use of ConceptNet. 
Revised: Our submission to the CommonsenseQA leaderboard achieves 72.2% single-model test set accuracy, and the result of a 20-model ensemble is 73.1%, which achieves No.1 among all the submissions without making use of ConceptNet. [Table] [Table]

Original: 4. A BLATION S TUDY AND A NALYSIS
Revised: 4. A BLATION S TUDY AND A NALYSIS

Original: In this sub-section, we ﬁrst show the importance of reusing dropout mask, then conduct a thorough ablation study on FreeLB over the GLUE benchmark to analyze the robustness and generalization strength of different approaches.
Revised: In this sub-section, we ﬁrst show the importance of reusing dropout mask, then conduct a thorough ablation study on FreeLB over the GLUE benchmark to analyze the robustness and generalization strength of different approaches.

Original: We observe that it is unnecessary to perform shallow-layer updates on the adversary as YOPO for our case, and FreeLB results in improved robustness and generalization compared with PGD.
Revised: We observe that it is unnecessary to perform shallow-layer updates on the adversary as YOPO for our case, and FreeLB results in improved robustness and generalization compared with PGD.

Original: Importance of Reusing Mask Table 8 (columns 2 to 4) compares the results of FreeLB with and[Table] without reusing the same dropout mask in each ascent step, as proposed in Sec. 3.3.
Revised: Importance of Reusing Mask Table 4 (columns 2 to 4) compares the results of FreeLB with and without reusing the same dropout mask in each ascent step, as proposed in Sec. 3.3.

Original: With reusing,
Revised: With reusing,

Original: FreeLB can achieve a larger improvement over the naturally trained models.
Revised: FreeLB can achieve a larger improvement over the naturally trained models.

Original: Thus, we enable mask reusing for all experiments involving RoBERTa.
Revised: Thus, we enable mask reusing for all experiments involving RoBERTa.

Original: Comparing the Robustness Table 5 provides the comparisons of the maximum increment of loss in the vicinity of each sample, deﬁned as:
Revised: Comparing the Robustness Table 5 provides the comparisons of the maximum increment of loss in the vicinity of each sample, deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: 5) which reﬂects the robustness and invariance of the model in the embedding space.
Revised: (5) which reﬂects the robustness and invariance of the model in the embedding space.

Original: In practice, we use PGD steps as in Eq.
Revised: In practice, we use PGD steps as in Eq.

Original: 2 to ﬁnd the value of ∆ L max ( X , (cid:15) ) .
Revised: 2 to ﬁnd the value of ∆ L max ( X , (cid:15) ) .

Original: We found that when using a step size of 5 · 10 − 3 and (cid:15) = 0 .
Revised: We found that when using a step size of 5 · 10 − 3 and (cid:15) = 0 .

Original: 01 (cid:107) X (cid:107) F , the PGD iterations converge to almost the same value, starting fromdifferent random initializations of δ for the RoBERTa models, trained with or without FreeLB.
Revised: 01 (cid:107) X (cid:107) F , the PGD iterations converge to almost the same value, starting fromdifferent random initializations of δ for the RoBERTa models, trained with or without FreeLB.

Original: This indicates that PGD reliably ﬁnds ∆ L max for these models.
Revised: This indicates that PGD reliably ﬁnds ∆ L max for these models.

Original: Therefore, we compute ∆ L max ( X , (cid:15) ) for each X via a 2000-step PGD.
Revised: Therefore, we compute ∆ L max ( X , (cid:15) ) for each X via a 2000-step PGD.

Original: Samples with small margins exist even for models with perfect accuracy, which could give a false sense of vulnerability of the model.
Revised: Samples with small margins exist even for models with perfect accuracy, which could give a false sense of vulnerability of the model.

Original: To rule out the outlier effect and make ∆ L max ( X , (cid:15) ) comparable across different samples, we only consider samples that all the evaluated models can correctly classify, and search for an (cid:15) for each sample such that the reference model can correctly classify all samples within the (cid:15) ball.
Revised: To rule out the outlier effect and make ∆ L max ( X , (cid:15) ) comparable across different samples, we only consider samples that all the evaluated models can correctly classify, and search for an (cid:15) for each sample such that the reference model can correctly classify all samples within the (cid:15) ball.

Original: However, such choice of per-sample (cid:15) favors the reference model by design.
Revised: However, such choice of per-sample (cid:15) favors the reference model by design.

Original: To make fair comparisons, Table 5 provides the median of ∆ L max ( X , (cid:15) ) with per-sample (cid:15) from models trained by FreeLB (Max Inc) and PGD (Mac Inc (R)), respectively.
Revised: To make fair comparisons, Table 5 provides the median of ∆ L max ( X , (cid:15) ) with per-sample (cid:15) from models trained by FreeLB (Max Inc) and PGD (Mac Inc (R)), respectively.

Original: Across all three datasets and different reference models, FreeLB has the smallest median increment even when starting from a larger natural loss than vanilla models.
Revised: Across all three datasets and different reference models, FreeLB has the smallest median increment even when starting from a larger natural loss than vanilla models.

Original: This demonstrates that FreeLB is more robust and invariant in most cases.
Revised: This demonstrates that FreeLB is more robust and invariant in most cases.

Original: Such results are also consistent with the models’ dev set performance (the performances for Vanilla/PGD/FreeLB models on RTE, CoLA and MRPC are 86.69/87.41/89.21, 69.91/70.84/71.40, 91.67/91.17/91.17, respectively).
Revised: Such results are also consistent with the models’ dev set performance (the performances for Vanilla/PGD/FreeLB models on RTE, CoLA and MRPC are 86.69/87.41/89.21, 69.91/70.84/71.40, 91.67/91.17/91.17, respectively).

Original: Comparing with YOPO The original implementation of YOPO (Zhang et al., 2019) chooses the ﬁrst convolutional layer of the ResNets as f 0 for updating the adversary in the “s-loop”.
Revised: Comparing with YOPO The original implementation of YOPO (Zhang et al., 2019) chooses the ﬁrst convolutional layer of the ResNets as f 0 for updating the adversary in the “s-loop”.

Original: As a result, each step of the “s-loop” should be using exactly the same value to update the adversary, 11 and
Revised: As a result, each step of the “s-loop” should be using exactly the same value to update the adversary, 13 and

Original: YOPO- m - n degenerates into FreeLB with a n -times large step size.
Revised: YOPO- m - n degenerates into FreeLB with a n -times large step size.

Original: To avoid that, we choose the layers up to the output of the ﬁrst Transformer block as f 0 when implementing YOPO.
Revised: To avoid that, we choose the layers up to the output of the ﬁrst Transformer block as f 0 when implementing YOPO.

Original: To make the total amount of update on the adversary equal, we take the hyper-parameters for FreeLB- m and only change the step size α into α/n for YOPO- m - n .
Revised: To make the total amount of update on the adversary equal, we take the hyper-parameters for FreeLB- m and only change the step size α into α/n for YOPO- m - n .

Original: Table 8 shows that FreeLB performs consistently better than YOPO on all three datasets.
Revised: Table 4 shows that FreeLB performs consistently better than YOPO on all three datasets.

Revised: Accidentally, we also give the results comparing with YOPOm - n without changing the step size α for YOPO in Table 8.

Revised: The gap between two approaches seem to shrink, which may be caused by using a larger total step size for the YOPO adversaries.

Original: We leave exhaustive hyperparameter search for both models as our future work.
Revised: We leave exhaustive hyperparameter search for both models as our future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have developed an adversarial training approach, FreeLB, to improve natural language understanding.
Revised: In this work, we have developed an adversarial training approach, FreeLB, to improve natural language understanding.

Original: The proposed approach adds perturbations to continuous word embeddings using a gradient method, and minimizes the resultant adversarial risk in an efﬁcient way.
Revised: The proposed approach adds perturbations to continuous word embeddings using a gradient method, and minimizes the resultant adversarial risk in an efﬁcient way.

Original: FreeLB is able to boost Transformer-based model (BERT and RoBERTa) on several datasets and achieve new state of the art on GLUE and ARC benchmarks.
Revised: FreeLB is able to boost Transformer-based model (BERT and RoBERTa) on several datasets and achieve new state of the art on GLUE and ARC benchmarks.

Original: Empirical study demonstrates that our method results in both higher robustness in the embedding space than natural training and better generalization ability.
Revised: Empirical study demonstrates that our method results in both higher robustness in the embedding space than natural training and better generalization ability.

Original: Such observation seems inconsistent with that in computer vision problems.
Revised: Such observation seems inconsistent with that in computer vision problems.

Original: Investigating the reason for the discrepancy between the outcomes of adversarial training for images and text is an interesting future direction
Revised: Investigating the reason for the discrepancy between the outcomes of adversarial training for images and text is an interesting future direction. Acknowledgements

Revised: : Goldstein and Zhu were supported in part by the DARPA GARD, DARPA

Revised: QED for RML, and AFOSR MURI programs.

Original: A A DDITIONAL E XPERIMENTAL D ETAILS A. P ROBLEM F ORMULATIONS
Revised: A A DDITIONAL E XPERIMENTAL D ETAILS A. P ROBLEM F ORMULATIONS

Original: For tasks with ranking loss like ARC, CommonsenseQA, WNLI and QNLI, add the perturbation to the concatenation of the embeddings of all question/answer pairs.
Revised: For tasks with ranking loss like ARC, CommonsenseQA, WNLI and QNLI, add the perturbation to the concatenation of the embeddings of all question/answer pairs.

Original: Additional tricks are required to achieve high performance on WNLI and QNLI for the GLUE benchmark.
Revised: Additional tricks are required to achieve high performance on WNLI and QNLI for the GLUE benchmark.

Original: We use the same tricks as Liu et al.
Revised: We use the same tricks as Liu et al.

Original: (2019b).
Revised: (2019b).

Original: For WNLI, we use the same WSC data provided by Liu et al.
Revised: For WNLI, we use the same WSC data provided by Liu et al.

Original: (2019b) for training.
Revised: (2019b) for training.

Original: For testing, Liu et al. (2019b) also provided the test set with span annotations, but the order is different form the GLUE dataset.
Revised: For testing, Liu et al. (2019b) also provided the test set with span annotations, but the order is different form the GLUE dataset.

Original: We re-order their test set by matching.
Revised: We re-order their test set by matching.

Original: For the QNLI, we follow Liu et al.
Revised: For the QNLI, we follow Liu et al.

Original: (2019b)
Revised: (2019b)

Original: and formulate the problem as pairwise ranking problem, which is the same for CommonsenseQA.
Revised: and formulate the problem as pairwise ranking problem, which is the same for CommonsenseQA.

Original: We ﬁnd the matching pairs for both training set and testing set by matching the queries in the dev set.
Revised: We ﬁnd the matching pairs for both training set and testing set by matching the queries in the dev set.

Original: We predict “entailment” if the candidate has the higher score, and “not entailment” otherwise.
Revised: We predict “entailment” if the candidate has the higher score, and “not entailment” otherwise.

Original: A. H YPER - PARAMETERS
Revised: A. H YPER - PARAMETERS

Original: [Table]
Revised: [Table]

Original: As other adversarial training methods, introduces three additional hyper-parameters: step size α , maximum perturbation (cid:15) , number of steps m .
Revised: As other adversarial training methods, introduces three additional hyper-parameters: step size α , maximum perturbation (cid:15) , number of steps m .

Original: For all other hyper-parameters such as learning rate and number of iterations, we either search in the same interval as RoBERTa (on CommonsenseQA,
Revised: For all other hyper-parameters such as learning rate and number of iterations, we either search in the same interval as RoBERTa (on CommonsenseQA,

Original: ARC
Revised: ARC

Original: [Table]
Revised: [Table]

Original: and WNLI), or use exactly the same setting as RoBERTa (except for MRPC, where we ﬁnd using a learning rate of 5 × 10 − 6 gives better results).
Revised: and WNLI), or use exactly the same setting as RoBERTa (except for MRPC, where we ﬁnd using a learning rate of 5 × 10 − 6 gives better results).

Original: We list the best combinations for α, (cid:15) and m for each of the GLUE tasks in Table 6.
Revised: We list the best combinations for α, (cid:15) and m for each of the GLUE tasks in Table 6.

Original: For WSC/WNLI, the best combination is (cid:15) = 1 e − 2 , α = 5 e − 3 , m = 2 .
Revised: For WSC/WNLI, the best combination is (cid:15) = 1 e − 2 , α = 5 e − 3 , m = 2 .

Original: Notice even when mα < (cid:15) , the maximum perturbation could still reach (cid:15) due to the random initialization.
Revised: Notice even when mα < (cid:15) , the maximum perturbation could still reach (cid:15) due to the random initialization.

Original: B V ARIANCE OF M AXIMUM I NCREMENT OF L OSS
Revised: B V ARIANCE OF M AXIMUM I NCREMENT OF L OSS

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Table 7 provides the complete results for the increment of loss in the interval, with median and standard deviation.
Revised: Table 7 provides the complete results for the increment of loss in the interval, with median and standard deviation.

Original: C A DDITIONAL R ESULTS FOR A BLATION S TUDIES
Revised: C A DDITIONAL R ESULTS FOR A BLATION S TUDIES

Original: Here we provide some additional results for comparison with YOPO as complementary results to
Revised: Here we provide some additional results for comparison with YOPO as complementary results to

Original: Table 8.
Revised: Table 4.

Original: We will release the complete results for comparing with YOPO and without variational dropout on each of the GLUE tasks in our next revision.
Revised: We will release the complete results for comparing with YOPO and without variational dropout on each of the GLUE tasks in our next revision.

Original: From the current results, there is no need
Revised: From the current results, there is no need

Original: [Table]
Revised: [Table]

Original: in using extra shallow-layer ascent steps that YOPO advocates, since this consistently deteriorates the performance while introducing extra computations.
Revised: in using extra shallow-layer updates that YOPO advocates [Table] since this consistently deteriorates the performance while introducing extra computations [Table]