Original: U NSUPERVISED D ATA A UGMENTATION FOR C ONSISTENCY T RAINING
Revised: U NSUPERVISED D ATA A UGMENTATION FOR C ONSISTENCY T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce.
Revised: Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce.

Original: Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise.
Revised: Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise.

Original: In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, speciﬁcally those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning.
Revised: In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, speciﬁcally those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning.

Original: By substituting simple noising operations with advanced data augmentation methods, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework.
Revised: By substituting simple noising operations with advanced data augmentation methods, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework.

Original: On the IMDb text classiﬁcation dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples.
Revised: On the IMDb text classiﬁcation dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples.

Original: On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 2.7% with only 4,000 examples, nearly matching the performance of models trained on 50,000 labeled examples.
Revised: On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 2.7% with only 4,000 examples, nearly matching the performance of models trained on 50,000 labeled examples.

Original: Our method also combines well with transfer learning, e.g., when ﬁnetuning from
Revised: Our method also combines well with transfer learning, e.g., when ﬁnetuning from

Original: BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used.
Revised: BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A fundamental weakness of deep learning is that it typically requires a lot of labeled data to work well.
Revised: A fundamental weakness of deep learning is that it typically requires a lot of labeled data to work well.

Original: Semi-supervised learning (SSL) (Chapelle et al., 2009) is one of the most promising methods of leveraging unlabeled data to address this weakness.
Revised: Semi-supervised learning (SSL) (Chapelle et al., 2009) is one of the most promising methods of leveraging unlabeled data to address this weakness.

Original: The recent works in SSL are diverse but those that are based on consistency training (Bachman et al., 2014;
Revised: The recent works in SSL are diverse but those that are based on consistency training (Bachman et al., 2014;

Original: Rasmus et al., 2015; Laine &
Revised: Rasmus et al., 2015; Laine &

Original: Aila, 2016; Tarvainen & Valpola, 2017) have shown to work well on many benchmarks.
Revised: Aila, 2016; Tarvainen & Valpola, 2017) have shown to work well on many benchmarks.

Original: In a nutshell, consistency training methods simply regularize model predictions to be invariant to small noise applied to either input examples (Miyato et al., 2018; Sajjadi et al., 2016;
Revised: In a nutshell, consistency training methods simply regularize model predictions to be invariant to small noise applied to either input examples (Miyato et al., 2018; Sajjadi et al., 2016;

Original: Clark et al., 2018) or hidden states (Bachman et al., 2014; Laine & Aila, 2016).
Revised: Clark et al., 2018) or hidden states (Bachman et al., 2014; Laine & Aila, 2016).

Original: This framework makes sense intuitively because a good model should be robust to any small change in an input example or hidden states.
Revised: This framework makes sense intuitively because a good model should be robust to any small change in an input example or hidden states.

Original: Under this framework, different methods in this category differ mostly in how and where the noise injection is applied.
Revised: Under this framework, different methods in this category differ mostly in how and where the noise injection is applied.

Original: Typical noise injection methods are additive Gaussian noise, dropout noise or adversarial noise.
Revised: Typical noise injection methods are additive Gaussian noise, dropout noise or adversarial noise.

Original: In this work, we investigate the role of noise injection in consistency training and observe that advanced data augmentation methods, speciﬁcally those work best in supervised learning (Simard et al., 1998; Krizhevsky et al., 2012; Cubuk et al., 2018; Yu et al., 2018), also perform well in semisupervised learning.
Revised: In this work, we investigate the role of noise injection in consistency training and observe that advanced data augmentation methods, speciﬁcally those work best in supervised learning (Simard et al., 1998; Krizhevsky et al., 2012; Cubuk et al., 2018; Yu et al., 2018), also perform well in semisupervised learning.

Original: There is indeed a strong correlation between the performance of data augmentation operations in supervised learning and their performance in consistency training.
Revised: There is indeed a strong correlation between the performance of data augmentation operations in supervised learning and their performance in consistency training.

Original: We, hence, propose to substitute the traditional noise injection methods with high quality data augmentation methods in order to improve consistency training.
Revised: We, hence, propose to substitute the traditional noise injection methods with high quality data augmentation methods in order to improve consistency training.

Original: To emphasize the use of better data augmentation in consistency training, we name our method Unsupervised Data Augmentation or UDA.
Revised: To emphasize the use of better data augmentation in consistency training, we name our method Unsupervised Data Augmentation or UDA.

Original: We evaluate UDA on a wide variety of language and vision tasks.
Revised: We evaluate UDA on a wide variety of language and vision tasks.

Original: On six text classiﬁcation tasks, our method achieves signiﬁcant improvements over state-of-the-art models.
Revised: On six text classiﬁcation tasks, our method achieves signiﬁcant improvements over state-of-the-art models.

Original: Notably, on IMDb, UDA with 20 labeled examples outperforms the state-of-the-art model trained on 1250x more labeled data.
Revised: Notably, on IMDb, UDA with 20 labeled examples outperforms the state-of-the-art model trained on 1250x more labeled data.

Original: We also evaluate UDA on standard semi-supervised learning benchmarks for vision such as
Revised: We also evaluate UDA on standard semi-supervised learning benchmarks for vision such as

Original: CIFAR-10 and SVHN.
Revised: CIFAR-10 and SVHN.

Original: UDA outperforms all existing semi-supervised learning methods by significant margins.
Revised: UDA outperforms all existing semi-supervised learning methods by significant margins.

Original: On CIFAR-10 with 4,000 labeled examples, UDA achieves an error rate of 5.29, nearly matching the performance of the fully supervised model that uses 50,000 labeled examples.
Revised: On CIFAR-10 with 4,000 labeled examples, UDA achieves an error rate of 5.29, nearly matching the performance of the fully supervised model that uses 50,000 labeled examples.

Original: Furthermore, with a better architecture, PyramidNet+ShakeDrop, UDA achieves a new state-ofthe-art error rate of 2.7.
Revised: Furthermore, with a better architecture, PyramidNet+ShakeDrop, UDA achieves a new state-ofthe-art error rate of 2.7.

Original: On SVHN, UDA achieves an error rate of 2.55 with only 1,000 labeled examples.
Revised: On SVHN, UDA achieves an error rate of 2.55 with only 1,000 labeled examples.

Original: Finally, we also ﬁnd UDA to be beneﬁcial when there is a large amount of supervised data.
Revised: Finally, we also ﬁnd UDA to be beneﬁcial when there is a large amount of supervised data.

Original: For instance, on ImageNet, UDA leads to improvements of top-1 accuracy from 58 .
Revised: For instance, on ImageNet, UDA leads to improvements of top-1 accuracy from 58 .

Original: 84 to 

Revised: 84 to

Original: .with 10% of the labeled set and from 78 .
Revised: .with 10% of the labeled set and from 78 .

Original: 43 to 79 .
Revised: 43 to 79 .

Original: 05 when we use the full labeled set and an external dataset with 1 .
Revised: 05 when we use the full labeled set and an external dataset with 1 .

Original: 3 M unlabeled examples.
Revised: 3 M unlabeled examples.

Original: Our key contributions and ﬁndings can be summarized as follows:
Revised: Our key contributions and ﬁndings can be summarized as follows:

Original: • First, we show that state-of-the-art data augmentations found in supervised learning can also serve as a superior source of perturbation under the consistency enforcing semi-supervised framework.
Revised: • First, we show that state-of-the-art data augmentations found in supervised learning can also serve as a superior source of perturbation under the consistency enforcing semi-supervised framework.

Original: See results in Table 1 and Table 2.
Revised: See results in Table 1 and Table 2.

Original: • Second, we show that UDA can match and even outperform purely supervised learning that uses orders of magnitude more labeled data.
Revised: • Second, we show that UDA can match and even outperform purely supervised learning that uses orders of magnitude more labeled data.

Original: State-of-the-art results for both vision and language tasks are reported in Table 3 and 4.
Revised: State-of-the-art results for both vision and language tasks are reported in Table 3 and 4.

Original: The effectiveness of UDA across different training data sizes are highlighted in Figure 4 and 5.
Revised: The effectiveness of UDA across different training data sizes are highlighted in Figure 4 and 5.

Original: Finally, we show that UDA combines well with transfer learning, e.g., when ﬁne-tuning from BERT ( see Table 4 ), and is effective at high-data regime,
Revised: Finally, we show that UDA combines well with transfer learning, e.g., when ﬁne-tuning from BERT ( see Table 4 ), and is effective at high-data regime,

Original: e.g. on ImageNet ( see Table 5 ).
Revised: e.g. on ImageNet ( see Table 5 ).

Original:  U NSUPERVISED D ATA A UGMENTATION (UDA)
Revised:  U NSUPERVISED D ATA A UGMENTATION (UDA)

Original: In this section, we ﬁrst formulate our task and then present the key method and insights behind UDA.
Revised: In this section, we ﬁrst formulate our task and then present the key method and insights behind UDA.

Original: Throughout this paper, we focus on classiﬁcation problems and will use x to denote the input and y ∗ to denote its ground-truth prediction target.
Revised: Throughout this paper, we focus on classiﬁcation problems and will use x to denote the input and y ∗ to denote its ground-truth prediction target.

Original: We are interested in learning a model p θ ( y | x ) to predict y ∗ based on the input x , where θ denotes the model parameters.
Revised: We are interested in learning a model p θ ( y | x ) to predict y ∗ based on the input x , where θ denotes the model parameters.

Original: Finally, we will use L and
Revised: Finally, we will use L and

Original: U to denote the sets of labeled and unlabeled examples respectively.
Revised: U to denote the sets of labeled and unlabeled examples respectively.

Original: 2. B ACKGROUND : S UPERVISED D ATA A UGMENTATION
Revised: 2. B ACKGROUND : S UPERVISED D ATA A UGMENTATION

Original: Data augmentation aims at creating novel and realistic-looking training data by applying a transformation to an example, without changing its label.
Revised: Data augmentation aims at creating novel and realistic-looking training data by applying a transformation to an example, without changing its label.

Original: Formally, let q (ˆ x | x ) be the augmentation transformation from which one can draw augmented examples ˆ x based on an original example x .
Revised: Formally, let q (ˆ x | x ) be the augmentation transformation from which one can draw augmented examples ˆ x based on an original example x .

Original: For an augmentation transformation to be valid, it is required that any example ˆ x ∼ q
Revised: For an augmentation transformation to be valid, it is required that any example ˆ x ∼ q

Original: (ˆ x | x ) drawn from the distribution shares the same ground-truth label as x .
Revised: (ˆ x | x ) drawn from the distribution shares the same ground-truth label as x .

Original: Given a valid augmentation transformation, we can simply minimize the negative log-likelihood on augmented examples.
Revised: Given a valid augmentation transformation, we can simply minimize the negative log-likelihood on augmented examples.

Original: Supervised data augmentation can be equivalently seen as constructing an augmented labeled set from the original supervised set and then training the model on the augmented set.
Revised: Supervised data augmentation can be equivalently seen as constructing an augmented labeled set from the original supervised set and then training the model on the augmented set.

Original: Therefore, the augmented set needs to provide additional inductive biases to be more effective.
Revised: Therefore, the augmented set needs to provide additional inductive biases to be more effective.

Original: How to design the augmentation transformation has, thus, become critical.
Revised: How to design the augmentation transformation has, thus, become critical.

Original: In recent years, there have been signiﬁcant advancements on the design of data augmentations for
Revised: In recent years, there have been signiﬁcant advancements on the design of data augmentations for

Original: NLP (Yu et al., 2018), vision (Krizhevsky et al., 2012; Cubuk et al., 2018) and speech (Hannun et al., 2014; Park et al., 2019) in supervised settings.
Revised: NLP (Yu et al., 2018), vision (Krizhevsky et al., 2012; Cubuk et al., 2018) and speech (Hannun et al., 2014; Park et al., 2019) in supervised settings.

Original: Despite the promising results, data augmentation is mostly regarded as the “cherry on the cake” which provides a steady but limited performance boost because these augmentations has so far only been applied to a set of labeled examples which is usually of a small size.
Revised: Despite the promising results, data augmentation is mostly regarded as the “cherry on the cake” which provides a steady but limited performance boost because these augmentations has so far only been applied to a set of labeled examples which is usually of a small size.

Original: Motivated by this limitation, via the consistency training framework, we extend the advancement in supervised data augmentation to semi-supervised learning where abundant unlabeled data is available.
Revised: Motivated by this limitation, via the consistency training framework, we extend the advancement in supervised data augmentation to semi-supervised learning where abundant unlabeled data is available.

Original: [Figure]
Revised: [Figure]

Original: Unlabeled
Revised: Unlabeled

Original: [Figure]
Revised: [Figure]

Original: x x*
Revised: x x*

Original: Augmentations
Revised: Augmentations

Original: [Figure]
Revised: [Figure]

Original: RandAugment
Revised: RandAugment

Original: TF-IDF word replacement
Revised: TF-IDF word replacement

Original: Final Loss x y ∗ $ % & () $ %+ & () $ % & (*)
Revised: Final Loss x y ∗ $ % & () $ %+ & () $ % & (*)

Original: M
Revised: M

Original: M M
Revised: M M

Original: 2. U NSUPERVISED D ATA A UGMENTATION
Revised: 2. U NSUPERVISED D ATA A UGMENTATION

Original: As discussed in the introduction, a recent line of work in semi-supervised learning has been utilizing unlabeled examples to enforce smoothness of the model.
Revised: As discussed in the introduction, a recent line of work in semi-supervised learning has been utilizing unlabeled examples to enforce smoothness of the model.

Original: The general form of these works can be summarized as follows:
Revised: The general form of these works can be summarized as follows:

Original: • Given an input x , compute the output distribution p θ ( y | x ) given x and a noised version p θ ( y | x, (cid:15) ) by injecting a small noise (cid:15) .
Revised: • Given an input x , compute the output distribution p θ ( y | x ) given x and a noised version p θ ( y | x, (cid:15) ) by injecting a small noise (cid:15) .

Original: The noise can be applied to x or hidden states.
Revised: The noise can be applied to x or hidden states.

Original: • Minimize a divergence metric between the two distributions D ( p θ ( y | x ) (cid:107) p θ ( y | x, (cid:15) )) .
Revised: • Minimize a divergence metric between the two distributions D ( p θ ( y | x ) (cid:107) p θ ( y | x, (cid:15) )) .

Original: This procedure enforces the model to be insensitive to the noise (cid:15) and hence smoother with respect to changes in the input (or hidden
Revised: This procedure enforces the model to be insensitive to the noise (cid:15) and hence smoother with respect to changes in the input (or hidden

Original: ) space
Revised: ) space

Original: From another perspective, minimizing the consistency
Revised: From another perspective, minimizing the consistency

Original: loss
Revised: loss

Original: gradually
Revised: gradually

Original: propagates label information from labeled examples to unlabeled ones
Revised: propagates label information from labeled examples to unlabeled ones

Original: In this work, we are interested in a particular setting where the noise is injected to the input x , i.e., ˆ x = q ( x, (cid:15) ) , as considered by prior works (Sajjadi et al., 2016; Laine & Aila, 2016;
Revised: In this work, we are interested in a particular setting where the noise is injected to the input x , i.e., ˆ x = q ( x, (cid:15) ) , as considered by prior works (Sajjadi et al., 2016; Laine & Aila, 2016;

Original: Miyato et al., 2018).
Revised: Miyato et al., 2018).

Original: But different from existing work, we focus on the unattended question of how the form or “quality” of the noising operation q can inﬂuence the performance of this consistency training framework.
Revised: But different from existing work, we focus on the unattended question of how the form or “quality” of the noising operation q can inﬂuence the performance of this consistency training framework.

Original: Speciﬁcally, to enforce consistency, prior methods generally employ simple noise injection methods such as adding Gaussian noise, simple input augmentations to noise unlabeled examples.
Revised: Speciﬁcally, to enforce consistency, prior methods generally employ simple noise injection methods such as adding Gaussian noise, simple input augmentations to noise unlabeled examples.

Original: In contrast, we hypothesize that stronger data augmentations in supervised learning can also lead to superior performance when used to noise unlabeled examples in the semi-supervised consistency training framework, since it has been shown that more advanced data augmentations that are more diverse and natural can lead to signiﬁcant performance gain in the supervised setting.
Revised: In contrast, we hypothesize that stronger data augmentations in supervised learning can also lead to superior performance when used to noise unlabeled examples in the semi-supervised consistency training framework, since it has been shown that more advanced data augmentations that are more diverse and natural can lead to signiﬁcant performance gain in the supervised setting.

Original: Following this idea, we propose to use a rich set of state-of-the-art data augmentations veriﬁed in various supervised settings to inject noise and optimize the same consistency training objective on unlabeled examples.
Revised: Following this idea, we propose to use a rich set of state-of-the-art data augmentations veriﬁed in various supervised settings to inject noise and optimize the same consistency training objective on unlabeled examples.

Original: When jointly trained with labeled examples, we utilize a weighting factor λ to balance the supervised cross entropy and the unsupervised consistency training loss, which is illustrated in Figure 1.
Revised: When jointly trained with labeled examples, we utilize a weighting factor λ to balance the supervised cross entropy and the unsupervised consistency training loss, which is illustrated in Figure 1.

Original: Formally, the full objective can be written as follows:
Revised: Formally, the full objective can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where q (ˆ x | x ) is a data augmentation transformation and ˜ θ is a ﬁxed copy of the current parameters θ indicating that the gradient is not propagated through ˜ θ , as suggested by Miyato et al.
Revised: where q (ˆ x | x ) is a data augmentation transformation and ˜ θ is a ﬁxed copy of the current parameters θ indicating that the gradient is not propagated through ˜ θ , as suggested by Miyato et al.

Original: We also follow VAT (Miyato et al., 2018) to use the KL divergence.
Revised: We also follow VAT (Miyato et al., 2018) to use the KL divergence.

Original: We set λ to 1 for most of our experiments and use different batch sizes for the supervised data and the unsupervised data.
Revised: We set λ to 1 for most of our experiments and use different batch sizes for the supervised data and the unsupervised data.

Original: In the vision domain, simple augmentations including cropping and ﬂipping are applied to labeled examples.
Revised: In the vision domain, simple augmentations including cropping and ﬂipping are applied to labeled examples.

Original: To minimize the discrepancy between supervised training and prediction on unlabeled examples, we apply the same simple augmentations to unlabeled examples for computing p ˜
Revised: To minimize the discrepancy between supervised training and prediction on unlabeled examples, we apply the same simple augmentations to unlabeled examples for computing p ˜

Original: θ ( y | x ) .
Revised: θ ( y | x ) .

Original: Discussion.
Revised: Discussion.

Original: Before detailing the augmentation operations used in this work, we ﬁrst provide some intuitions on how more advanced data augmentations can provide extra advantages over simple ones used in earlier works from three aspects:
Revised: Before detailing the augmentation operations used in this work, we ﬁrst provide some intuitions on how more advanced data augmentations can provide extra advantages over simple ones used in earlier works from three aspects:

Original: • Valid noise : Advanced
Revised: • Valid noise : Advanced

Original: data augmentation methods that achieve great performance in supervised learning usually generate realistic augmented examples that share the same ground-truth labels
Revised: data augmentation methods that achieve great performance in supervised learning usually generate realistic augmented examples that share the same ground-truth labels

Original: with the original example.
Revised: with the original example.

Original: Thus, it is safe to encourage the consistency between predictions on the original unlabeled example and the augmented unlabeled examples.
Revised: Thus, it is safe to encourage the consistency between predictions on the original unlabeled example and the augmented unlabeled examples.

Original: • Diverse noise : Advanced data augmentation can generate a diverse set of examples since it can make large modiﬁcations to the input example without changing its label, while simple Gaussian noise only make local changes.
Revised: • Diverse noise : Advanced data augmentation can generate a diverse set of examples since it can make large modiﬁcations to the input example without changing its label, while simple Gaussian noise only make local changes.

Original: Encouraging consistency on a diverse set of augmented examples can signiﬁcantly improve the sample efﬁciency.
Revised: Encouraging consistency on a diverse set of augmented examples can signiﬁcantly improve the sample efﬁciency.

Original: • Targeted inductive biases : Different tasks require different inductive biases.
Revised: • Targeted inductive biases : Different tasks require different inductive biases.

Original: Data augmentation operations that work well in supervised training essentially provides the missing or most wanted inductive biases in an original labeled set.
Revised: Data augmentation operations that work well in supervised training essentially provides the missing or most wanted inductive biases in an original labeled set.

Original: 2. A UGMENTATION S TRATEGIES FOR D IFFERENT T ASKS
Revised: 2. A UGMENTATION S TRATEGIES FOR D IFFERENT T ASKS

Original: We now detail the augmentation methods, tailored for different tasks, that we use in this work.
Revised: We now detail the augmentation methods, tailored for different tasks, that we use in this work.

Original: RandAugment for Image Classiﬁcation.
Revised: RandAugment for Image Classiﬁcation.

Original: We make use of a data augmentation method called
Revised: We make use of a data augmentation method called

Original: RandAugment, which is inspired by AutoAugment (Cubuk et al., 2018).
Revised: RandAugment, which is inspired by AutoAugment (Cubuk et al., 2018).

Original: AutoAugment uses a search method to combine all image processing transformations in the Python Image Library (PIL) to ﬁnd a good augmentation strategy.
Revised: AutoAugment uses a search method to combine all image processing transformations in the Python Image Library (PIL) to ﬁnd a good augmentation strategy.

Original: In RandAugment, we do not use search, but instead uniformly sample from the same set of augmentation transformations in PIL.
Revised: In RandAugment, we do not use search, but instead uniformly sample from the same set of augmentation transformations in PIL.

Original: In other words, RandAugment is simpler and requires no labeled data as there is no need to search for optimal policies.
Revised: In other words, RandAugment is simpler and requires no labeled data as there is no need to search for optimal policies.

Original: Back-
Revised: Back-

Original: translation for Text
Revised: translation for Text

Original: Classiﬁcation.
Revised: Classiﬁcation.

Original: When used as an augmentation method, backtranslation (Sennrich et al., 2015; Edunov et al., 2018) refers to the procedure of translating an existing example x in language A into another language B and then translating it back into A to obtain an augmented example ˆ x .
Revised: When used as an augmentation method, backtranslation (Sennrich et al., 2015; Edunov et al., 2018) refers to the procedure of translating an existing example x in language A into another language B and then translating it back into A to obtain an augmented example ˆ x .

Original: As observed by Yu et al.
Revised: As observed by Yu et al.

Original: (2018), back-translation can generate diverse paraphrases while preserving the semantics of the original sentences, leading to signiﬁcant performance improvements in question answering.
Revised: (2018), back-translation can generate diverse paraphrases while preserving the semantics of the original sentences, leading to signiﬁcant performance improvements in question answering.

Original: In our case, we use back-translation to paraphrase the training data of our text classiﬁcation tasks.
Revised: In our case, we use back-translation to paraphrase the training data of our text classiﬁcation tasks.

Original: We ﬁnd that the diversity of the paraphrases is more important than the quality or the validity.
Revised: We ﬁnd that the diversity of the paraphrases is more important than the quality or the validity.

Original: Hence, we employ random sampling with a tunable temperature instead of beam search for the generation.
Revised: Hence, we employ random sampling with a tunable temperature instead of beam search for the generation.

Original: As shown in Figure 2, the paraphrases generated by back-translation sentence are diverse and have similar semantic meanings.
Revised: As shown in Figure 2, the paraphrases generated by back-translation sentence are diverse and have similar semantic meanings.

Original: More speciﬁcally, we use WMT’14 English-French translation models (in both directions) to perform back-translation on each sentence.
Revised: More speciﬁcally, we use WMT’14 English-French translation models (in both directions) to perform back-translation on each sentence.

Original: To facilitate future research, we have open-sourced our back-translation system together with the translation checkpoints.
Revised: To facilitate future research, we have open-sourced our back-translation system together with the translation checkpoints.

Original: [Figure]
Revised: [Figure]

Original: Word replacing with TF-IDF for Text Classiﬁcation.
Revised: Word replacing with TF-IDF for Text Classiﬁcation.

Original: While back-translation is good at maintaining the global semantics of a sentence, there is little control over which words will be retained.
Revised: While back-translation is good at maintaining the global semantics of a sentence, there is little control over which words will be retained.

Original: This requirement is important for topic classiﬁcation tasks, such as DBPedia, in which some keywords are more informative than other words in determining the topic.
Revised: This requirement is important for topic classiﬁcation tasks, such as DBPedia, in which some keywords are more informative than other words in determining the topic.

Original: We, therefore, propose an augmen- many the same model checkpoints.
Revised: We, therefore, propose an augmen- many the same model checkpoints.

Original: tation method that replaces uninformative words with low TF-IDF scores while keeping those with high TF-IDF values.
Revised: tation method that replaces uninformative words with low TF-IDF scores while keeping those with high TF-IDF values.

Original: We refer readers to Appendix C for a detailed description.
Revised: We refer readers to Appendix C for a detailed description.

Original: 2. T RAINING S IGNAL A NNEALING FOR L OW - DATA R EGIME
Revised: 2. T RAINING S IGNAL A NNEALING FOR L OW - DATA R EGIME

Original: In semi-supervised learning, we often encounter a situation where there is a huge gap between the amount of unlabeled data and that of labeled data.
Revised: In semi-supervised learning, we often encounter a situation where there is a huge gap between the amount of unlabeled data and that of labeled data.

Original: Hence, the model often quickly overﬁts the limited amount of labeled data while still underﬁtting the unlabeled data.
Revised: Hence, the model often quickly overﬁts the limited amount of labeled data while still underﬁtting the unlabeled data.

Original: To tackle this difﬁculty, we introduce a new training technique, called Training Signal Annealing (TSA), which gradually releases the “training signals” of the labeled examples as training progresses.
Revised: To tackle this difﬁculty, we introduce a new training technique, called Training Signal Annealing (TSA), which gradually releases the “training signals” of the labeled examples as training progresses.

Original: Intuitively, we only utilize a labeled example if the model’s conﬁdence on that example is lower than a predeﬁned threshold which increases according to a schedule.
Revised: Intuitively, we only utilize a labeled example if the model’s conﬁdence on that example is lower than a predeﬁned threshold which increases according to a schedule.

Original: Speciﬁcally, at training step t , if the model’s predicted probability for the correct category p θ ( y ∗ | x ) is higher than a threshold η t , we remove that example from the loss function.
Revised: Speciﬁcally, at training step t , if the model’s predicted probability for the correct category p θ ( y ∗ | x ) is higher than a threshold η t , we remove that example from the loss function.

Original: Suppose K is the number of categories, by gradually increase η t from 1 K to 1 , the threshold η t serves as a ceiling to prevent over-training on easy labeled examples.
Revised: Suppose K is the number of categories, by gradually increase η t from 1 K to 1 , the threshold η t serves as a ceiling to prevent over-training on easy labeled examples.

Original: We consider three increasing schedules of η t with different application scenarios.
Revised: We consider three increasing schedules of η t with different application scenarios.

Original: Let T be the total number of training steps, the three schedules are shown in Figure 3.
Revised: Let T be the total number of training steps, the three schedules are shown in Figure 3.

Original: Intuitively, when the model is prone to overﬁt, e.g., when the problem is relatively easy or the number of labeled examples is very limited, the exp-schedule is most suitable as the supervised signal is mostly released at the end of training.
Revised: Intuitively, when the model is prone to overﬁt, e.g., when the problem is relatively easy or the number of labeled examples is very limited, the exp-schedule is most suitable as the supervised signal is mostly released at the end of training.

Original: In contrast, when the model is less likely to overﬁt (e.g., when we have abundant labeled examples or when the model employs effective regularization), the log-schedule can serve well.
Revised: In contrast, when the model is less likely to overﬁt (e.g., when we have abundant labeled examples or when the model employs effective regularization), the log-schedule can serve well.

Original: [Figure]
Revised: [Figure]

Original: 1.0 tT 0.
Revised: 1.0 tT 0.

Original: t linear-schedule 0.
Revised: t linear-schedule 0.

Original: 1.0 tT 0.
Revised: 1.0 tT 0.

Original: t exp-schedule
Revised: t exp-schedule

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate UDA on a variety of language and vision tasks.
Revised: In this section, we evaluate UDA on a variety of language and vision tasks.

Original: For language, we rely on six text classiﬁcation benchmark datasets, including IMDb, Yelp-2, Yelp-5, Amazon-2 and Amazon5 sentiment classiﬁcation and DBPedia topic classiﬁcation (Maas et al., 2011; Zhang et al., 2015).
Revised: For language, we rely on six text classiﬁcation benchmark datasets, including IMDb, Yelp-2, Yelp-5, Amazon-2 and Amazon5 sentiment classiﬁcation and DBPedia topic classiﬁcation (Maas et al., 2011; Zhang et al., 2015).

Original: For vision, we employ two smaller datasets CIFAR-10 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), which are often used to compare semi-supervised algorithms, as well as ImageNet (Deng et al., 2009) of a larger scale to test the scalability of UDA.
Revised: For vision, we employ two smaller datasets CIFAR-10 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), which are often used to compare semi-supervised algorithms, as well as ImageNet (Deng et al., 2009) of a larger scale to test the scalability of UDA.

Original: For details of the labeled and unlabeled data and experiment details, we refer readers to Appendix E.
Revised: For details of the labeled and unlabeled data and experiment details, we refer readers to Appendix E.

Original: 3. C ORRELATION BETWEEN S UPERVISED AND S EMI - SUPERVISED P ERFORMANCES
Revised: 3. C ORRELATION BETWEEN S UPERVISED AND S EMI - SUPERVISED P ERFORMANCES

Original: As the ﬁrst step, we try to verify the fundamental idea of UDA, i.e., there is a positive correlation of data augmentation’s effectiveness in supervised learning and semi-supervised learning.
Revised: As the ﬁrst step, we try to verify the fundamental idea of UDA, i.e., there is a positive correlation of data augmentation’s effectiveness in supervised learning and semi-supervised learning.

Original: Based on
Revised: Based on

Original: Yelp-5 (a language task) and CIFAR-10 (a vision task), we compare the performance of different data augmentation methods in either fully supervised or semi-supervised settings.
Revised: Yelp-5 (a language task) and CIFAR-10 (a vision task), we compare the performance of different data augmentation methods in either fully supervised or semi-supervised settings.

Original: For Yelp-5, apart from back-translation, we include a simpler method Switchout (Wang et al., 2018) which replaces a token with a random token uniformly sampled from the vocabulary.
Revised: For Yelp-5, apart from back-translation, we include a simpler method Switchout (Wang et al., 2018) which replaces a token with a random token uniformly sampled from the vocabulary.

Original: For CIFAR-10, we compare
Revised: For CIFAR-10, we compare

Original: RandAugment with two simpler methods: (1) cropping & ﬂipping augmentation and (2) Cutout.
Revised: RandAugment with two simpler methods: (1) cropping & ﬂipping augmentation and (2) Cutout.

Original: Based on this setting, Table 1 and Table 2 exhibit a strong correlation of an augmentation’s effectiveness between supervised and semi-supervised settings.
Revised: Based on this setting, Table 1 and Table 2 exhibit a strong correlation of an augmentation’s effectiveness between supervised and semi-supervised settings.

Original: This validates our idea of stronger data augmentations found in supervised learning can always lead to more gains when applied to the semi-supervised learning settings.
Revised: This validates our idea of stronger data augmentations found in supervised learning can always lead to more gains when applied to the semi-supervised learning settings.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A LGORITHM C OMPARISON ON V ISION S EMI - SUPERVISED L EARNING B ENCHMARKS
Revised: A LGORITHM C OMPARISON ON V ISION S EMI - SUPERVISED L EARNING B ENCHMARKS

Original: With the correlation established above, the next question we ask is how well UDA performs compared to existing semi-supervised learning algorithms.
Revised: With the correlation established above, the next question we ask is how well UDA performs compared to existing semi-supervised learning algorithms.

Original: To answer the question, we focus on the most commonly used semi-supervised learning benchmarks CIFAR-10 and SVHN.
Revised: To answer the question, we focus on the most commonly used semi-supervised learning benchmarks CIFAR-10 and SVHN.

Original: Vary the size of labeled data.
Revised: Vary the size of labeled data.

Original: Firstly, we follow the settings in (Oliver et al., 2018) and employ
Revised: Firstly, we follow the settings in (Oliver et al., 2018) and employ

Original: Wide-ResNet-28-2 (Zagoruyko & Komodakis, 2016;
Revised: Wide-ResNet-28-2 (Zagoruyko & Komodakis, 2016;

Original: He et al., 2016) as the backbone model and evaluate UDA with varied supervised data sizes.
Revised: He et al., 2016) as the backbone model and evaluate UDA with varied supervised data sizes.

Original: Speciﬁcally, we compare UDA with two highly competitive baselines: (1) Virtual adversarial training (VAT) (Miyato et al., 2018), an algorithm that generates adversarial Gaussian noise on input, and (2) MixMatch (Berthelot et al., 2019), a parallel work that combines previous advancements in semi-supervised learning.
Revised: Speciﬁcally, we compare UDA with two highly competitive baselines: (1) Virtual adversarial training (VAT) (Miyato et al., 2018), an algorithm that generates adversarial Gaussian noise on input, and (2) MixMatch (Berthelot et al., 2019), a parallel work that combines previous advancements in semi-supervised learning.

Original: The comparison is shown in Figure 4 with two key observations.
Revised: The comparison is shown in Figure 4 with two key observations.

Original: • First, UDA consistently outperforms the two baselines with a clear margin given different sizes of labeled data.
Revised: • First, UDA consistently outperforms the two baselines with a clear margin given different sizes of labeled data.

Original: • Moreover, the performance difference between UDA and VAT shows the superiority of data augmentation based noise.
Revised: • Moreover, the performance difference between UDA and VAT shows the superiority of data augmentation based noise.

Original: The difference of UDA and VAT is essentiallythe
Revised: The difference of UDA and VAT is essentially

Original:  noise process.
Revised: the noise process.

Original: While the noise produced by VAT often contain high-frequency artifacts that do not exist in real images, data augmentation mostly generates diverse and realistic images.
Revised: While the noise produced by VAT often contain high-frequency artifacts that do not exist in real images, data augmentation mostly generates diverse and realistic images.

Original: [Figure]
Revised: [Figure]

Original: Comparisons with published results Next, we directly compare UDA with previously published results under different model architectures.
Revised: Comparisons with published results Next, we directly compare UDA with previously published results under different model architectures.

Original: Following previous work, 4k and 1k labeled examples are used for CIFAR-10 and SVHN respectively.
Revised: Following previous work, 4k and 1k labeled examples are used for CIFAR-10 and SVHN respectively.

Original: As shown in Table 3, given the same architecture,
Revised: As shown in Table 3, given the same architecture,

Original: UDA outperforms all published results by signiﬁcant margins.
Revised: UDA outperforms all published results by signiﬁcant margins.

Original: This shows the huge potential of state-of-the-art data augmentationsunder the consistency training framework in the vision domain.
Revised: This shows the huge potential of state-of-the-art data augmentations the consistency training framework in the vision domain [Table]

Original: 3. E VALUATION ON T EXT C LASSIFICATION D ATASETS
Revised: 3. E VALUATION ON T EXT C LASSIFICATION D ATASETS

Original: Next, we further evaluate UDA in the language domain.
Revised: Next, we further evaluate UDA in the language domain.

Original: Moreover, in order to test whether UDA can be combined with the success of unsupervised representation learning, such as BERT (Devlin et al.,[Table] 2018), we further consider four initialization schemes: (a) random Transformer; (b) BERT BASE ; (c) BERT LARGE ; (d) BERT FINETUNE : BERT LARGE ﬁne-tuned on in-domain unlabeled data 3 .
Revised: Moreover, in order to test whether UDA can be combined with the success of unsupervised representation learning, such as BERT (Devlin et al., 2018), we further consider four initialization schemes: (a) random Transformer; (b) BERT BASE ; (c) BERT LARGE ; (d) BERT FINETUNE : BERT LARGE ﬁne-tuned on in-domain unlabeled data 4 .

Original: Under each of these four initialization schemes [Table] we compare the performances with and without UDA [Table]
Revised: Under each of these four initialization schemes, we compare the performances with and without UDA.

Original: Fully

Original: [Table] (20) (20) (2.5k) (20) (2.5k) (140) Random (cid:55) 43. 41.14 (cid:51) 25.
Revised: The results are presented in Table 4 where we would like to emphasize three observations:

Original: BERT BASE (cid:55) 18. 2.58 (cid:51) [Table]
Revised: • First, even with very few labeled examples, UDA can offer decent or even competitive performances compared to the

Original: [Table] [Table] [Table] BERT LARGE (cid:55)
Revised: SOTA model

Original: [Table] [Table] [Table] [Table]
Revised: trained

Original: [Table] [Table] [Table] BERT
Revised: with full supervised data.

Original: [Table] (cid:55) [Table]
Revised: Particularly

Original: [Table] [Table] The results are presented in Table 4 where we would like to emphasize three observations: • First, even with very few labeled examples, UDA can offer decent or even competitive performances compared to the SOTA model trained with full supervised data.
Revised: , on binary

Original: Particularly, on binary sentiment analysis tasks, with only 20 supervised examples, UDA outperforms the previous SOTA trained with full supervised data on IMDb and is competitive on Yelp-2 and Amazon-2.
Revised: sentiment analysis tasks, with only 20 supervised examples, UDA outperforms the previous SOTA trained with full supervised data on IMDb and is competitive on Yelp-2 and Amazon-2.

Original: • Second, UDA is complementary to transfer learning / representation learning.
Revised: • Second, UDA is complementary to transfer learning / representation learning.

Original: As we can see, when initialized with BERT and further ﬁnetuned on in-domain data, UDA can still signiﬁcantly reduce the error rate from 6 .
Revised: As we can see, when initialized with BERT and further ﬁnetuned on in-domain data, UDA can still signiﬁcantly reduce the error rate from 6 .

Original: 50 to 4 . 20 on IMDb.
Revised: 50 to 4 . 20 on IMDb.

Original: • Finally, we also note that for ﬁve-category sentiment classiﬁcation tasks, there still exists a clear gap between UDA with 500 labeled examples per class and BERT trained on the entire supervised set.
Revised: • Finally, we also note that for ﬁve-category sentiment classiﬁcation tasks, there still exists a clear gap between UDA with 500 labeled examples per class and BERT trained on the entire supervised set.

Original: Intuitively, ﬁve-category sentiment classiﬁcations are much more difﬁcult than their binary counterparts.
Revised: Intuitively, ﬁve-category sentiment classiﬁcations are much more difﬁcult than their binary counterparts.

Original: This suggests a room for further improvement in the future.
Revised: This suggests a room for further improvement in the future.

Original: Results with different labeled set sizes.
Revised: Results with different labeled set sizes.

Original: We also show in Figure 5 that UDA leads to consistent improvements across all labeled data sizes on IMDb and Yelp-2.
Revised: We also show in Figure 5 that UDA leads to consistent improvements across all labeled data sizes on IMDb and Yelp-2.

Revised: [Table]

Revised: 2017) for Amazon-2 and Amazon-5, Mixed VAT (Sachan et al., 2018) for IMDb and DBPedia.

Revised: All of our experiments use a sequence length of 512.

Original: [Figure]
Revised: [Figure]

Original: 3. S CALABILITY T EST ON THE I MAGE N ET D ATASET
Revised: 3. S CALABILITY T EST ON THE I MAGE N ET D ATASET

Original: Then, to evaluate whether UDA can scale to problems with a large scale and a higher difﬁculty, we now turn to the ImageNet dataset with ResNet-50 being the underlying architecture.
Revised: Then, to evaluate whether UDA can scale to problems with a large scale and a higher difﬁculty, we now turn to the ImageNet dataset with ResNet-50 being the underlying architecture.

Original: Speciﬁcally, we consider two experiment settings with different natures:
Revised: Speciﬁcally, we consider two experiment settings with different natures:

Original: • We use 10% of the supervised data of ImageNet while using all other data as unlabeled data.
Revised: • We use 10% of the supervised data of ImageNet while using all other data as unlabeled data.

Original: As a result, the unlabeled exmaples are entirely in-domain.
Revised: As a result, the unlabeled exmaples are entirely in-domain.

Original: • In the second setting, we keep all images in ImageNet as supervised data.
Revised: • In the second setting, we keep all images in ImageNet as supervised data.

Original: Then, we use the domain-relevance data ﬁltering method (See Appendix B for details) to ﬁlter out 1.3M images from an anonymous dataset.
Revised: Then, we use the domain-relevance data ﬁltering method (See Appendix B for details) to ﬁlter out 1.3M images from an anonymous dataset.

Original: Hence, the unlabeled set is not necessarily in-domain.
Revised: Hence, the unlabeled set is not necessarily in-domain.

Original: The results are summarized in Table 5.
Revised: The results are summarized in Table 5.

Original: In both 10% and the full data settings, UDA consistently brings signiﬁcant gains comparedto the supervised baseline.
Revised: In both 10% and the full data settings, UDA consistently brings signiﬁcant gains compared

Revised: to the supervised baseline.

Original: This shows UDA is not only able to scale but also able to utilize out-of-domain unlabeled examples to improve model performance.
Revised: This shows UDA is not only able to scale but also able to utilize out-of-domain unlabeled examples to improve model performance.

Original: In parallel to our work, S4L (Zhai et al., 2019b) and CPC (H´enaff et al., 2019) also show signiﬁcant improvements on ImageNet.
Revised: In parallel to our work, S4L (Zhai et al., 2019b) and CPC (H´enaff et al., 2019) also show signiﬁcant improvements on ImageNet.

Original: [Table]
Revised: [Table]

Original: 3. A BLATION S TUDIES FOR TSA
Revised: 3. A BLATION S TUDIES FOR TSA

Original: Lastly, we study the effect of TSA on two tasks with different amounts of unlabeled data:
Revised: Lastly, we study the effect of TSA on two tasks with different amounts of unlabeled data:

Original: (a) Yelpwhere we have only 2 .
Revised: (a) Yelpwhere we have only 2 .

Original: 5 k labeled examples and 6 m unlabeled examples.
Revised: 5 k labeled examples and 6 m unlabeled examples.

Original: (b) CIFAR-10 where we have 4 k labeled examples and 50 k unlabeled examples.
Revised: (b) CIFAR-10 where we have 4 k labeled examples and 50 k unlabeled examples.

Original: For Yelp-5, we use a randomly initialized transformer in this study to rule out factors of having a pre-trained representation.
Revised: For Yelp-5, we use a randomly initialized transformer in this study to rule out factors of having a pre-trained representation.

Original: As shown in Table 6, on Yelp-5, where there is a lot more unlabeled data than labeled data, TSA reduces the error rate from 50 .
Revised: As shown in Table 6, on Yelp-5, where there is a lot more unlabeled data than labeled data, TSA reduces the error rate from 50 .

Original: 81 to 41 .
Revised: 81 to 41 .

Original: 35 when compared to the baseline without TSA.
Revised: 35 when compared to the baseline without TSA.

Original: More speciﬁcally, the best performance is achieved when we choose to postpone releasing the supervised training signal to the end of the training, i.e, exp-schedule leads to the best performance.
Revised: More speciﬁcally, the best performance is achieved when we choose to postpone releasing the supervised training signal to the end of the training, i.e, exp-schedule leads to the best performance.

Original: On the other hand, linear-schedule is the sweet spot on CIFAR-10 in terms of the speed of releasing supervised training signals, where the amount of unlabeled data is comparable to that of supervised data.
Revised: On the other hand, linear-schedule is the sweet spot on CIFAR-10 in terms of the speed of releasing supervised training signals, where the amount of unlabeled data is comparable to that of supervised data.

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: R ELATED W ORK
Revised: ELATED W ORK

Original: Existing works in consistency training does make use of data augmentation (Laine & Aila, 2016;
Revised: Existing works in consistency training does make use of data augmentation (Laine & Aila, 2016;

Original: Sajjadi et al., 2016); however, they only apply weak augmentation methods such as random translations and cropping.
Revised: Sajjadi et al., 2016); however, they only apply weak augmentation methods such as random translations and cropping.

Original: In parallel to our work, ICT (Verma et al., 2019) and MixMatch (Berthelot et al., 2019) also show improvements for semi-supervised learning.
Revised: In parallel to our work, ICT (Verma et al., 2019) and MixMatch (Berthelot et al., 2019) also show improvements for semi-supervised learning.

Original: These methods employ mixup (Zhang et al., 2017) on top of simple augmentations such as ﬂipping and cropping; instead, UDA emphasizes on the use of state-of-the-art data augmentations, leading to signiﬁcantly better results on CIFARand SVHN.
Revised: These methods employ mixup (Zhang et al., 2017) on top of simple augmentations such as ﬂipping and cropping; instead, UDA emphasizes on the use of state-of-the-art data augmentations, leading to signiﬁcantly better results on CIFARand SVHN.

Original: In addition, UDA is also applicable to language domain and can also scale well to more challenging vision datasets, such as ImageNet.
Revised: In addition, UDA is also applicable to language domain and can also scale well to more challenging vision datasets, such as ImageNet.

Original: Other works in the consistency training family mostly differ in how the noise is deﬁned: Pseudoensemble (Bachman et al., 2014) directly applies Gaussian noise and Dropout noise; VAT (Miyato et al., 2018; 2016) deﬁnes the noise by approximating the direction of change in the input space that the model is most sensitive to; Cross-view training (Clark et al., 2018) masks out part of the input data.
Revised: Other works in the consistency training family mostly differ in how the noise is deﬁned: Pseudoensemble (Bachman et al., 2014) directly applies Gaussian noise and Dropout noise; VAT (Miyato et al., 2018; 2016) deﬁnes the noise by approximating the direction of change in the input space that the model is most sensitive to; Cross-view training (Clark et al., 2018) masks out part of the input data.

Original: Apart from enforcing consistency on the input examples and the hidden representations, another line of research enforces consistency on the model parameter space.
Revised: Apart from enforcing consistency on the input examples and the hidden representations, another line of research enforces consistency on the model parameter space.

Original: Works in this category include Mean Teacher (Tarvainen & Valpola, 2017), fast-Stochastic Weight Averaging (Athiwaratkun et al., 2018) and Smooth Neighbors on Teacher Graphs (Luo et al., 2018).
Revised: Works in this category include Mean Teacher (Tarvainen & Valpola, 2017), fast-Stochastic Weight Averaging (Athiwaratkun et al., 2018) and Smooth Neighbors on Teacher Graphs (Luo et al., 2018).

Original: For a complete version of related work, see Appendix D.
Revised: For a complete version of related work, see Appendix D.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we show that data augmentation and semi-supervised learning are well connected: better data augmentation can lead to signiﬁcantly better semi-supervised learning.
Revised: In this paper, we show that data augmentation and semi-supervised learning are well connected: better data augmentation can lead to signiﬁcantly better semi-supervised learning.

Original: Our method,
Revised: Our method,

Original: UDA, employs state-of-the-art data augmentation found in supervised learning to generate diverse and realistic noise and enforces the model to be consistent with respect to these noise.
Revised: UDA, employs state-of-the-art data augmentation found in supervised learning to generate diverse and realistic noise and enforces the model to be consistent with respect to these noise.

Original: For text,
Revised: For text,

Original: UDA combines well with representation learning, e.g., BERT, and is very effective in low-data regime where state-of-the-art performance is achieved on IMDb with only 20 examples.
Revised: UDA combines well with representation learning, e.g., BERT, and is very effective in low-data regime where state-of-the-art performance is achieved on IMDb with only 20 examples.

Original: For vision,
Revised: For vision,

Original: UDA outperforms prior works by a clear margin and nearly matches the performance of the fully supervised models trained on the full labeled sets which are one order of magnitude larger.
Revised: UDA outperforms prior works by a clear margin and nearly matches the performance of the fully supervised models trained on the full labeled sets which are one order of magnitude larger.

Original: Lastly,
Revised: Lastly,

Original: UDA can effectively leverage out-of-domain unlabeled data and achieve improved performances on ImageNet where we have a large amount of supervised data.
Revised: UDA can effectively leverage out-of-domain unlabeled data and achieve improved performances on ImageNet where we have a large amount of supervised data.

Original: We hope that UDA will encourage future research to transfer advanced supervised augmentation to semi-supervised setting for different tasks.
Revised: We hope that UDA will encourage future research to transfer advanced supervised augmentation to semi-supervised setting for different tasks

Original: A M ORE E XPERIMENTS A. A BLATIONS S TUDIES ON R AND A UGMENT
Revised: A M ORE E XPERIMENTS A. A BLATIONS S TUDIES ON R AND A UGMENT

Original: We hypothesize that the success of RandAugment should be credited to the diversity of the augmentation transformations, since RandAugment works very well for multiple different datasets while does not require a search algorithm to ﬁnd out the most effective policies.
Revised: We hypothesize that the success of RandAugment should be credited to the diversity of the augmentation transformations, since RandAugment works very well for multiple different datasets while does not require a search algorithm to ﬁnd out the most effective policies.

Original: To verify this hypothesis, we test UDA’s performance when we restrict the number of possible transformations used in RandAugment.
Revised: To verify this hypothesis, we test UDA’s performance when we restrict the number of possible transformations used in RandAugment.

Original: As shown in Figure 6, the performance gradually improves as we use more augmentation transformations.
Revised: As shown in Figure 6, the performance gradually improves as we use more augmentation transformations.

Original: Number of possible tranformations 5. E rr o r r a t e
Revised: [Figure]

Original: A. R ESULTS ON CIFAR-10 AND SVHN WITH VARIED LABEL SET SIZES
Revised: A. R ESULTS ON CIFAR-10 AND SVHN WITH VARIED LABEL SET SIZES

Original: CIFAR-10 In Table 7, we show results for compared methods of Figure 4a and results of PseudoLabel (Lee, 2013), Π -Model (Laine & Aila, 2016), Mean Teacher (Tarvainen & Valpola, 2017).
Revised: CIFAR-10 In Table 7, we show results for compared methods of Figure 4a and results of PseudoLabel (Lee, 2013), Π -Model (Laine & Aila, 2016), Mean Teacher (Tarvainen & Valpola, 2017).

Original: Fully supervised learning using 50,000 examples achieves an error rate of 5.36 and 4.23 with or without RandAugment.
Revised: Fully supervised learning using 50,000 examples achieves an error rate of 5.36 and 4.23 with or without RandAugment.

Original: The performance of the baseline models are reported by MixMatch (Berthelot et al., 2019).
Revised: The performance of the baseline models are reported by MixMatch (Berthelot et al., 2019).

Original: To make sure that the performance reported by MixMatch and our results are comparable, we reimplement MixMatch in our codebase and ﬁnd that the results in the original paper is comparable but slightly higher than our reimplementation, which results in a more competitive comparison for
Revised: To make sure that the performance reported by MixMatch and our results are comparable, we reimplement MixMatch in our codebase and ﬁnd that the results in the original paper is comparable but slightly higher than our reimplementation, which results in a more competitive comparison for

Original: UDA.
Revised: UDA.

Original: For example, our reimplementation of MixMatch achieves an error rate of 7.00 ± 0.59 and 7.39 ± 0.11 with 4,000 and 2,000 examples.
Revised: For example, our reimplementation of MixMatch achieves an error rate of 7.00 ± 0.59 and 7.39 ± 0.11 with 4,000 and 2,000 examples.

Original: MixMatch uses a different model implementation and employs exponential moving average (EMA) on the model parameters, while we do not use EMA for our implementations.[Table] [Figure] [Table]
Revised: MixMatch uses a different model implementation and employs exponential moving average (EMA) on the model parameters, while we do not use EMA for our implementations.

Original: [Table]
Revised: [Table]

Original: SVHN In Table
Revised: SVHN In Table

Original: [Table]
Revised: [Table]

Original: we similarly show results for compared methods of Figure 4b and results of methods mentioned above.
Revised: we similarly show results for compared methods of Figure 4b and results of methods mentioned above.

Original: Fully supervised learning using 73,257 examples achieves an error rate of 2.84 and 2.28 with or without RandAugment.
Revised: Fully supervised learning using 73,257 examples achieves an error rate of 2.84 and 2.28 with or without RandAugment.

Original: The performance of the baseline models are reported by MixMatch (Berthelot et al., 2019)
Revised: The performance of the baseline models are reported by MixMatch (Berthelot et al., 2019).

Original: [Table]

Original: Our reimplementation of MixMatch also resulted in comparable but higher error rates than the reported ones.
Revised: Our reimplementation of MixMatch also resulted in comparable but higher error rates than the reported ones.

Original: [Table]
Revised: [Table]

Original: B A DDITIONAL T RAINING T ECHNIQUES
Revised: B A DDITIONAL T RAINING T ECHNIQUES

Original: While UDA generally works well [Table] there are several practical issues in consistency training that may dampen the performance gain if not carefully dealt with.
Revised: While UDA generally works well, there are several practical issues in consistency training that may dampen the performance gain if not carefully dealt with.

Original: This section presents additional techniques targeting at some commonly encountered problems.
Revised: This section presents additional techniques targeting at some commonly encountered problems.

Original: Sharpening Predictions.
Revised: Sharpening Predictions.

Original: Entropy minimization (Grandvalet & Bengio, 2005) has been shown to be effective in semi-supervised learning methods such as VAT (Miyato et al., 2018).
Revised: Entropy minimization (Grandvalet & Bengio, 2005) has been shown to be effective in semi-supervised learning methods such as VAT (Miyato et al., 2018).

Original: To apply entropy minimization in our method, we simply add a loss term to the objective to regularize the predicted distributions on unlabeled examples to have a low entropy.
Revised: To apply entropy minimization in our method, we simply add a loss term to the objective to regularize the predicted distributions on unlabeled examples to have a low entropy.

Original: Alternatively, when the number of labeled examples are extremely small, we ﬁnd it helpful to mask out examples that the current model is not conﬁdent about and use a low Softmax temperature when computing the target distribution on unlabeled examples.
Revised: Alternatively, when the number of labeled examples are extremely small, we ﬁnd it helpful to mask out examples that the current model is not conﬁdent about and use a low Softmax temperature when computing the target distribution on unlabeled examples.

Original: Speciﬁcally, in each minibatch, the consistency loss term is computed only on examples whose highest probability among classiﬁcation categories is greater than a threshold.
Revised: Speciﬁcally, in each minibatch, the consistency loss term is computed only on examples whose highest probability among classiﬁcation categories is greater than a threshold.

Original: Domain-relevance Data Filtering.
Revised: Domain-relevance Data Filtering.

Original: Ideally, we would like to make use of out-of-domain unlabeled data since it is usually much easier to collect, but the class distributions of out-of-domain data are mismatched with those of in-domain data, which can result in performance loss if directly used (Oliver et al., 2018).
Revised: Ideally, we would like to make use of out-of-domain unlabeled data since it is usually much easier to collect, but the class distributions of out-of-domain data are mismatched with those of in-domain data, which can result in performance loss if directly used (Oliver et al., 2018).

Original: To obtain data relevant to the domain for the task at hand, we adopt a common technique for detecting out-of-domain data.
Revised: To obtain data relevant to the domain for the task at hand, we adopt a common technique for detecting out-of-domain data.

Original: We use our baseline model trained on the in-domain data to infer the labels of data in a large out-of-domain dataset and pick out examples that the model is most conﬁdent about.
Revised: We use our baseline model trained on the in-domain data to infer the labels of data in a large out-of-domain dataset and pick out examples that the model is most conﬁdent about.

Original: Speciﬁcally, for each category, we sort all examples based on the classiﬁed probabilities of being in that category and select the examples with the highest probabilities.
Revised: Speciﬁcally, for each category, we sort all examples based on the classiﬁed probabilities of being in that category and select the examples with the highest probabilities.

Original: C E XTENDED A UGMENTATION S TRATEGIES FOR D IFFERENT T ASKS Discussion on Trade-off Between Diversity and Validity for Data Augmentation.
Revised: C E XTENDED A UGMENTATION S TRATEGIES FOR D IFFERENT T ASKS Discussion on Trade-off Between Diversity and Validity for Data Augmentation.

Original: Despite that state-of-the-art data augmentation methods can generate diverse and valid augmented examples as discussed in section 2.2, there is a trade-off between diversity and validity since diversity is achieved by changing a part of the original example, naturally leading to the risk of altering the ground-truth label.
Revised: Despite that state-of-the-art data augmentation methods can generate diverse and valid augmented examples as discussed in section 2.2, there is a trade-off between diversity and validity since diversity is achieved by changing a part of the original example, naturally leading to the risk of altering the ground-truth label.

Original: We ﬁnd it beneﬁcial to tune the trade-off between diversity and validity for data augmentation methods.
Revised: We ﬁnd it beneﬁcial to tune the trade-off between diversity and validity for data augmentation methods.

Original: For text classiﬁcation, we tune the temperature of random sampling.
Revised: For text classiﬁcation, we tune the temperature of random sampling.

Original: On the one hand, when we use a temperature of 0 , decoding by random sampling degenerates into greedy decoding and generates perfectly valid but identical paraphrases.
Revised: On the one hand, when we use a temperature of 0 , decoding by random sampling degenerates into greedy decoding and generates perfectly valid but identical paraphrases.

Original: On the other hand, when we use a temperature of 1 , random sampling generates very diverse but barely readable paraphrases.
Revised: On the other hand, when we use a temperature of 1 , random sampling generates very diverse but barely readable paraphrases.

Original: We ﬁnd that setting the Softmax temperature to 0 . 7 , 0 . 8 or 0 .
Revised: We ﬁnd that setting the Softmax temperature to 0 . 7 , 0 . 8 or 0 .

Original: 9 leads to the best performances.
Revised: 9 leads to the best performances.

Original: RandAugment Details.
Revised: RandAugment Details.

Original: In our implementation of RandAugment, each sub-policy is composed of two operations, where each operation is represented by the transformation name, probability, and magnitude that is speciﬁc to that operation.
Revised: In our implementation of RandAugment, each sub-policy is composed of two operations, where each operation is represented by the transformation name, probability, and magnitude that is speciﬁc to that operation.

Original: For example, a sub-policy can be [(Sharpness, 0.6, 2), (Posterize, 0.3, 9)].
Revised: For example, a sub-policy can be [(Sharpness, 0.6, 2), (Posterize, 0.3, 9)].

Original: For each operation, we randomly sample a transformation from 15 possible transformations, a magnitude in [1 , 10) and ﬁx the probability to 0 .
Revised: For each operation, we randomly sample a transformation from 15 possible transformations, a magnitude in [1 , 10) and ﬁx the probability to 0 .

Original: Speciﬁcally, we sample from the following 15 transformations: Invert, Cutout, Sharpness, AutoContrast, Posterize, ShearX, TranslateX, TranslateY,
Revised: Speciﬁcally, we sample from the following 15 transformations: Invert, Cutout, Sharpness, AutoContrast, Posterize, ShearX, TranslateX, TranslateY,

Original: ShearY, Rotate, Equalize, Contrast, Color, Solarize, Brightness.
Revised: ShearY, Rotate, Equalize, Contrast, Color, Solarize, Brightness.

Original: We ﬁnd this setting to work well in our ﬁrst try and did not tune the magnitude range and the probability.
Revised: We ﬁnd this setting to work well in our ﬁrst try and did not tune the magnitude range and the probability.

Original: Tuning these hyperparameters might result in further gains in accuracy.
Revised: Tuning these hyperparameters might result in further gains in accuracy.

Original: TF-IDF based word replacing Details.
Revised: TF-IDF based word replacing Details.

Original: We describe the TF-IDF based word replacing data augmentation method in this section.
Revised: We describe the TF-IDF based word replacing data augmentation method in this section.

Original: Ideally, we would like the augmentation method to generate both diverse and valid examples.
Revised: Ideally, we would like the augmentation method to generate both diverse and valid examples.

Original: Hence, the augmentation is designed to retain keywords and replace uninformative words with other uninformative words.
Revised: Hence, the augmentation is designed to retain keywords and replace uninformative words with other uninformative words.

Original: We use BERT’s word tokenizer since BERT ﬁrst tokenizes sentences into a sequence of words and then tokenize words into subwords although the model uses subwords as input.
Revised: We use BERT’s word tokenizer since BERT ﬁrst tokenizes sentences into a sequence of words and then tokenize words into subwords although the model uses subwords as input.

Original: Speciﬁcally, Suppose IDF( w ) is the IDF score for word w computed on the whole corpus, and
Revised: Speciﬁcally, Suppose IDF( w ) is the IDF score for word w computed on the whole corpus, and

Original: TF( w ) is the TF score for word w in a sentence.
Revised: TF( w ) is the TF score for word w in a sentence.

Original: We compute the TF-IDF score as TFIDF( w ) =
Revised: We compute the TF-IDF score as TFIDF( w ) =

Original: TF( w )IDF( w ) .
Revised: TF( w )IDF( w ) .

Original: Suppose the maximum TF-IDF score in a sentence x is C = max i TFIDF( x i ) .
Revised: Suppose the maximum TF-IDF score in a sentence x is C = max i TFIDF( x i ) .

Original: To make the probability of having a word replaced to negatively correlate with its TF-IDF score, we set the probability to min( p ( C − TFIDF( x i ))
Revised: To make the probability of having a word replaced to negatively correlate with its TF-IDF score, we set the probability to min( p ( C − TFIDF( x i ))

Original: /Z, 1) , where p is a hyperparameter that controls the magnitude of the augmentation and
Revised: /Z, 1) , where p is a hyperparameter that controls the magnitude of the augmentation and

Original: Z = (cid:80) i ( C − TFIDF( x i )) /
Revised: Z = (cid:80) i ( C − TFIDF( x i )) /

Original: | x | is the average score.
Revised: | x | is the average score.

Original: p is set to 0.7 for experiments on DBPedia.
Revised: p is set to 0.7 for experiments on DBPedia.

Original: When a word is replaced, we sample another word from the whole vocabulary for the replacement.
Revised: When a word is replaced, we sample another word from the whole vocabulary for the replacement.

Original: Intuitively, the sampled words should not be keywords to prevent changing the ground-truth labels of the sentence.
Revised: Intuitively, the sampled words should not be keywords to prevent changing the ground-truth labels of the sentence.

Original: To measure if a word is keyword, we compute a score of each word on the whole corpus.
Revised: To measure if a word is keyword, we compute a score of each word on the whole corpus.

Original: Speciﬁcally, we compute the score as S ( w ) =
Revised: Speciﬁcally, we compute the score as S ( w ) =

Original: freq( w )IDF( w ) where freq( w ) is the frequency of word w on the whole corpus.
Revised: freq( w )IDF( w ) where freq( w ) is the frequency of word w on the whole corpus.

Original: We set the probability of sampling word w as (max w (cid:48) S
Revised: We set the probability of sampling word w as (max w (cid:48) S

Original: ( w (cid:48) )
Revised: ( w (cid:48) )

Original: − S ( w )) /Z
Revised: − S ( w )) /Z

Original: (cid:48) where Z (cid:48)
Revised: (cid:48) where Z (cid:48)

Original: = (cid:80) w max w (cid:48) S ( w (cid:48) )
Revised: = (cid:80) w max w (cid:48) S ( w (cid:48) )

Original: − S ( w ) is a normalization term.
Revised: − S ( w ) is a normalization term.

Original: D E XTENDED R ELATED W ORK
Revised: D E XTENDED R ELATED W ORK

Original: Semi-supervised Learning.
Revised: Semi-supervised Learning.

Original: Due to the long history of semi-supervised learning (SSL), we refer readers to (Chapelle et al., 2009) for a general review.
Revised: Due to the long history of semi-supervised learning (SSL), we refer readers to (Chapelle et al., 2009) for a general review.

Original: More recently, many efforts have been made to renovate classic ideas into deep neural instantiations.
Revised: More recently, many efforts have been made to renovate classic ideas into deep neural instantiations.

Original: For example, graph-based label propagation (Zhu et al., 2003) has been extended to neural methods via graph embeddings (Weston et al., 2012; Yang et al., 2016) and later graph convolutions (Kipf & Welling, 2016).
Revised: For example, graph-based label propagation (Zhu et al., 2003) has been extended to neural methods via graph embeddings (Weston et al., 2012; Yang et al., 2016) and later graph convolutions (Kipf & Welling, 2016).

Original: Similarly, with the variational auto-encoding framework and reinforce algorithm, classic graphical models based SSL methods with target variable being latent can also take advantage of deep architectures (Kingma et al., 2014;
Revised: Similarly, with the variational auto-encoding framework and reinforce algorithm, classic graphical models based SSL methods with target variable being latent can also take advantage of deep architectures (Kingma et al., 2014;

Original: Maaløe et al., 2016; Yang et al., 2017).
Revised: Maaløe et al., 2016; Yang et al., 2017).

Original: Besides the direct extensions, it was found that training neural classiﬁers to classify out-of-domain examples into an additional class (Salimans et al., 2016) works very well in practice.
Revised: Besides the direct extensions, it was found that training neural classiﬁers to classify out-of-domain examples into an additional class (Salimans et al., 2016) works very well in practice.

Original: Later, Dai et al.
Revised: Later, Dai et al.

Original: (2017) shows that this can be seen as an instantiation of low-density separation.
Revised: (2017) shows that this can be seen as an instantiation of low-density separation.

Original: Apart from enforcing consistency on the noised input examples and the hidden representations, another line of research enforces consistency under different model parameters, which is complementary to our method.
Revised: Apart from enforcing consistency on the noised input examples and the hidden representations, another line of research enforces consistency under different model parameters, which is complementary to our method.

Original: For example, Mean Teacher (Tarvainen & Valpola, 2017) maintains a teacher model with parameters being the ensemble of a student model’s parameters and enforces the consistency between the predictions of the two models.
Revised: For example, Mean Teacher (Tarvainen & Valpola, 2017) maintains a teacher model with parameters being the ensemble of a student model’s parameters and enforces the consistency between the predictions of the two models.

Original: Recently, Athiwaratkun et al.
Revised: Recently, Athiwaratkun et al.

Original: (2018) propose fast-SWA that improves Mean Teacher by encouraging the model to explore a diverse set of plausible parameters.
Revised: (2018) propose fast-SWA that improves Mean Teacher by encouraging the model to explore a diverse set of plausible parameters.

Original: In addition to parameter-level consistency, SNTG (Luo et al., 2018) also enforces input-level consistency by constructing a similarity graph between unlabeled examples.
Revised: In addition to parameter-level consistency, SNTG (Luo et al., 2018) also enforces input-level consistency by constructing a similarity graph between unlabeled examples.

Original: Data Augmentation.
Revised: Data Augmentation.

Original: Also related to our work is the ﬁeld of data augmentation research.
Revised: Also related to our work is the ﬁeld of data augmentation research.

Original: Besides the conventional approaches and two data augmentation methods mentioned in Section 2.1, a recent approach MixUp (Zhang et al., 2017) goes beyond data augmentation from a single data point and performs interpolation of data pairs to achieve augmentation.
Revised: Besides the conventional approaches and two data augmentation methods mentioned in Section 2.1, a recent approach MixUp (Zhang et al., 2017) goes beyond data augmentation from a single data point and performs interpolation of data pairs to achieve augmentation.

Original: Recently, Hern´andez-Garc´ıa &
Revised: Recently, Hern´andez-Garc´ıa &

Original: K¨onig (2018) have shown that data augmentation can be regarded as a kind of explicit regularization methods similar to Dropout.
Revised: K¨onig (2018) have shown that data augmentation can be regarded as a kind of explicit regularization methods similar to Dropout.

Original: Diverse Back Translation.
Revised: Diverse Back Translation.

Original: Diverse paraphrases generated by back-translation has been a key component in the signiﬁcant performance improvements in our text classiﬁcation experiments.
Revised: Diverse paraphrases generated by back-translation has been a key component in the signiﬁcant performance improvements in our text classiﬁcation experiments.

Original: We use random sampling instead of beam search for decoding similar to the work by Edunov et al.
Revised: We use random sampling instead of beam search for decoding similar to the work by Edunov et al.

Original: There are also recent works on generating diverse translations (He et al., 2018;
Revised: There are also recent works on generating diverse translations (He et al., 2018;

Original: Shen et al., 2019;
Revised: Shen et al., 2019;

Original: Kool et al., 2019) that might lead to further improvements when used as data augmentations.
Revised: Kool et al., 2019) that might lead to further improvements when used as data augmentations.

Original: Unsupervised Representation Learning.
Revised: Unsupervised Representation Learning.

Original: Apart from semi-supervised learning, unsupervised representation learning offers another way to utilize unsupervised data.
Revised: Apart from semi-supervised learning, unsupervised representation learning offers another way to utilize unsupervised data.

Original: Collobert & Weston (2008) demonstrated that word embeddings learned by language modeling can improve the performance signiﬁcantly on semantic role labeling.
Revised: Collobert & Weston (2008) demonstrated that word embeddings learned by language modeling can improve the performance signiﬁcantly on semantic role labeling.

Original: Later, the pre-training of word embeddings was simpliﬁed and substantially scaled in Word2Vec (Mikolov et al., 2013) and Glove (Pennington et al., 2014).
Revised: Later, the pre-training of word embeddings was simpliﬁed and substantially scaled in Word2Vec (Mikolov et al., 2013) and Glove (Pennington et al., 2014).

Original: More recently, Dai & Le (2015); Peters et al.
Revised: More recently, Dai & Le (2015); Peters et al.

Original: ; Radford et al.
Revised: ; Radford et al.

Original: ; Howard & Ruder (2018);
Revised: ; Howard & Ruder (2018);

Original: Devlin et al.
Revised: Devlin et al.

Original: (2018) have shown that pre-training using language modeling and denoising auto-encoding leads to signiﬁcant improvements on many tasks in the language domain.
Revised: (2018) have shown that pre-training using language modeling and denoising auto-encoding leads to signiﬁcant improvements on many tasks in the language domain.

Original: There is also a growing interest in self-supervised learning for vision (Zhai et al., 2019b; H´enaff et al., 2019;
Revised: There is also a growing interest in self-supervised learning for vision (Zhai et al., 2019b; H ´ enaff et al., 2019;

Original: Trinh et al., 2019).
Revised: Trinh et al., 2019).

Original: Consistency Training in Other Domains.
Revised: Consistency Training in Other Domains.

Original: Similar ideas of consistency training has also been applied in other domains.
Revised: Similar ideas of consistency training has also been applied in other domains.

Original: For example, recently, enforcing adversarial consistency on unsupervised data has also been shown to be helpful in adversarial robustness (Stanforth et al., 2019; Zhai et al., 2019a;
Revised: For example, recently, enforcing adversarial consistency on unsupervised data has also been shown to be helpful in adversarial robustness (Stanforth et al., 2019; Zhai et al., 2019a;

Original: Carmon et al., 2019).
Revised: Carmon et al., 2019).

Original: Enforcing consistency w.r.t data augmentation has also been shown to work well for representation learning (Hu et al., 2017; Ye et al., 2019).
Revised: Enforcing consistency w.r.t data augmentation has also been shown to work well for representation learning (Hu et al., 2017; Ye et al., 2019).

Original: Invariant representation learning (Liang et al., 2018; Salazar et al., 2018) applies the consistency loss not only to the predicted distributions but also to representations and has been shown signiﬁcant improvements on speech recognition.
Revised: Invariant representation learning (Liang et al., 2018; Salazar et al., 2018) applies the consistency loss not only to the predicted distributions but also to representations and has been shown signiﬁcant improvements on speech recognition.

Original: E E XPERIMENT D ETAILS
Revised: E E XPERIMENT D ETAILS

Original: In this section, we provide experiment details for the performed experiments
Revised: In this section, we provide experiment details for the performed experiments.

Original: . E. T EXT C LASSIFICATIONS
Revised: E. T EXT C LASSIFICATIONS

Original: Datasets.
Revised: Datasets.

Original: In our semi-supervised setting, we randomly sampled labeled examples from the full supervised set 4 and use the same number of examples for each category.
Revised: In our semi-supervised setting, we randomly sampled labeled examples from the full supervised set 5 and use the same number of examples for each category.

Original: For unlabeled data, we use the whole training set for DBPedia, the concatenation of the training set and the unlabeled set for
Revised: For unlabeled data, we use the whole training set for DBPedia, the concatenation of the training set and the unlabeled set for

Original: IMDb and external data for Yelp-2, Yelp-5, Amazon-2 and Amazon-5 (McAuley et al., 2015) 5 .
Revised: IMDb and external data for Yelp-2, Yelp-5, Amazon-2 and Amazon-5 (McAuley et al., 2015) 6 .

Original: Note that for Yelp and Amazon based datasets, the label distribution of the unlabeled set might not match with that of labeled datasets since there are different number of examples in different categories.
Revised: Note that for Yelp and Amazon based datasets, the label distribution of the unlabeled set might not match with that of labeled datasets since there are different number of examples in different categories.

Original: Nevertheless, we ﬁnd it works well to use all the unlabeled data.
Revised: Nevertheless, we ﬁnd it works well to use all the unlabeled data.

Original: Preprocessing.
Revised: Preprocessing.

Original: We ﬁnd the sequence length to be an important factor in achieving good performance.
Revised: We ﬁnd the sequence length to be an important factor in achieving good performance.

Original: For all text classiﬁcation datasets, we truncate the input to 512 subwords since BERT is pretrained with a maximum sequence length of 512.
Revised: For all text classiﬁcation datasets, we truncate the input to 512 subwords since BERT is pretrained with a maximum sequence length of 512.

Original: Further, when thelength of an example is greater than 512, we keep the last 512 subwords instead of the ﬁrst 512 subwords as keeping the latter part of the sentence lead to better performances on IMDb.
Revised: Further, when the

Original: Fine-tuning BERT on in-domain unsupervised data.
Revised: Fine-tuning BERT on in-domain unsupervised data.

Original: We ﬁne-tune the BERT model on in-domain unsupervised data using the code released by BERT.
Revised: We ﬁne-tune the BERT model on in-domain unsupervised data using the code released by BERT.

Original: We try learning rate of 2e-5, 5e-5 and 1e-4, batch size of 32, 64 and 128 and number of training steps of 30k, 100k and 300k.
Revised: We try learning rate of 2e-5, 5e-5 and 1e-4, batch size of 32, 64 and 128 and number of training steps of 30k, 100k and 300k.

Original: We pick the ﬁne-tuned models by the BERT loss on a held-out set instead of the performance on a downstream task.
Revised: We pick the ﬁne-tuned models by the BERT loss on a held-out set instead of the performance on a downstream task.

Original: Random initialized Transformer.
Revised: Random initialized Transformer.

Original: For the experiments with randomly initialized Transformer, we adopt hyperparameters for BERT base except that we only use 6 hidden layers and 8 attention heads.
Revised: For the experiments with randomly initialized Transformer, we adopt hyperparameters for BERT base except that we only use 6 hidden layers and 8 attention heads.

Original: We also increase the dropout rate on the attention and the hidden states to 0.2, When we train UDA with randomly initialized architectures, we train UDA for 500k or 1M steps on Amazon-5 and Yelp5 where we have abundant unlabeled data.
Revised: We also increase the dropout rate on the attention and the hidden states to 0.2, When we train UDA with randomly initialized architectures, we train UDA for 500k or 1M steps on Amazon-5 and Yelp5 where we have abundant unlabeled data.

Original: BERT hyperparameters.
Revised: BERT hyperparameters.

Original: Following the common BERT ﬁne-tuning procedure, we keep a dropout rate of 0.1, and try learning rate of 1e-5, 2e-5 and 5e-5 and batch size of 32 and 128.
Revised: Following the common BERT ﬁne-tuning procedure, we keep a dropout rate of 0.1, and try learning rate of 1e-5, 2e-5 and 5e-5 and batch size of 32 and 128.

Original: We also tune the number of steps ranging from 30 to 100k for various data sizes.
Revised: We also tune the number of steps ranging from 30 to 100k for various data sizes.

Original: UDA hyperparameters.
Revised: UDA hyperparameters.

Original: We set the weight on the unsupervised objective λ to 1 in all of our experiments.
Revised: We set the weight on the unsupervised objective λ to 1 in all of our experiments.

Original: We use a batch size of 32 for the supervised objective since 32 is the smallest batch size on v3-32 Cloud TPU Pod.
Revised: We use a batch size of 32 for the supervised objective since 32 is the smallest batch size on v3-32 Cloud TPU Pod.

Original: We use a batch size of 224 for the unsupervised objective when the Trans- former is initialized with BERT so that the model can be trained on more unlabeled data.
Revised: We use a batch size of 224 for the unsupervised objective when the Transformer is initialized with BERT so that the model can be trained on more unlabeled data.

Original: We ﬁnd that generating one augmented example for each unlabeled example is enough for BERT FINETUNE .
Revised: We ﬁnd that generating one augmented example for each unlabeled example is enough for BERT FINETUNE .

Original: All experiments in this part are performed on a v3-32 Cloud TPU Pod.
Revised: All experiments in this part are performed on a v3-32 Cloud TPU Pod.

Original: E. S EMI - SUPERVISED LEARNING BENCHMARKS CIFAR-10 AND SVHN
Revised: E. S EMI - SUPERVISED LEARNING BENCHMARKS CIFAR-10 AND SVHN

Original: Hyperparameters for Wide-ResNet-28-2.
Revised: Hyperparameters for Wide-ResNet-28-2.

Original: For hyperparameter tuning,  we follow Oliver et al.
Revised: For hyperparameter tuning, for simplicity, we performed a random sampling search over hyperparameters and choose the best one based on validation sets (20% of the training sets with different sizes). We use the averaged results of multiple experiments to reduce the performance variance measured on the small validation sets. Speciﬁcally, we tried the following ranges: • training steps: 50k, 100k ; • learning rate: 0.03 , 0.05, 0.1; • TSA schedules: log-schedule, linear-schedule , exp-schedule, not using TSA; • entropy minimization loss weight: 0, 0.1 , 0.3; • consistency loss weight: 1 , 3, 6; • unlabeled data batch size: 960 , 1280; • weight decay rate: 5e-4 , 7e-4, 1e-3; • softmax temperature: 1 , 0.9; • conﬁdence threshold: 0 , 0.8.

Original: and only tune the learning rate and hyperparameters for our unsupervised objective.
Revised: where the values in bold black text are our default hyperparameters.

Original: Other hyperparameters follow those of the released AutoAugment code. Since there are many more unlabeled examples than labeled examples, we usea larger batch size for the unsupervised objective.
Revised: We found that given a reasonably large labeled set, our method is robust to hyper-parameters. Therefore, we use the same hyper-parameters in these cases.

Original: For example, in our CIFAR-10 experiments on TPUs, we use a batch size of 32 for  the supervised loss and use a batch size of 960 for the unsupervised loss. We train the model for 100k steps and the weight on the unsupervised objective λ is set to 1.
Revised: Speciﬁcally, we use the above default hyperparameters for CIFAR10 with 4,000, 2,000, 1,000 and 500 examples. For SVHN with 4,000, 2,000, 1,000, 500,examples, we additionally set the learning rate to 0.05 and unlabeled batch size to 1280. For the case of 250 examples on CIFAR-10, we use a different set of hyper-parameters: training steps: 50k; weight unlabeled

Original: On GPUs, we ﬁnd it work well touse a batch size of 64 and 320 for thesupervised loss and the unsupervised loss respectively and train for 400k steps.
Revised: Other hyperparameters not mentioned above are the same to the the original paper of WideResNet (Zagoruyko & Komodakis, 2016).

Original: In order to reduce training time, we generate augmented examples before training and dump them to disk.
Revised: In order to reduce training time, we generate augmented examples before training and dump them to disk.

Original: For CIFAR-10, we generate 100 augmented examples for each unlabeled example.
Revised: For CIFAR-10, we generate 100 augmented examples for each unlabeled example.

Original: Note that generating augmented examples in an online fashion is always better or as good as using dumped augmented examples since the model can see different augmented examples in different epochs, leading to more diverse samples.
Revised: Note that generating augmented examples in an online fashion is always better or as good as using dumped augmented examples since the model can see different augmented examples in different epochs, leading to more diverse samples.

Original: We report the average performance and the standard deviation for 10 runs.
Revised: We report the average performance and the standard deviation for 10 runs.

Original: Hyperparameters for Shake-Shake and PyramidNet.
Revised: Hyperparameters for Shake-Shake and PyramidNet.

Original: For the experiments with Shake-Shake, we train UDA for 300k steps and use a batch size of 128 for the supervised objective and use a batch size of 512 for the unsuperivsed objective.
Revised: For the experiments with Shake-Shake, we train UDA for 300k steps and use a batch size of 128 for the supervised objective and use a batch size of 512 for the unsuperivsed objective.

Original: For the experiments with PyramidNet+ShakeDrop, we train UDA for 700k steps and use a batch size of 64 for the supervised objective and a batch size of 128 for the unsupervised objective.
Revised: For the experiments with PyramidNet+ShakeDrop, we train UDA for 700k steps and use a batch size of 64 for the supervised objective and a batch size of 128 for the unsupervised objective.

Original: For both models, we use a learning rate of 0.03 and use a cosine learning decay with one annealing cycle following AutoAugment.
Revised: For both models, we use a learning rate of 0.03 and use a cosine learning decay with one annealing cycle following AutoAugment.

Original: All experiments in this part are performed on a v3-32 Cloud TPU v3 Pod.
Revised: All experiments in this part are performed on a v3-32 Cloud TPU v3 Pod.

Original: E. I MAGE N ET
Revised: E. I MAGE N ET

Original: 10% Labeled Set Setting.
Revised: 10% Labeled Set Setting.

Original: Unless otherwise stated, we follow the standard hyperparameters used in an open-source implementation of ResNet.
Revised: Unless otherwise stated, we follow the standard hyperparameters used in an open-source implementation of ResNet.

Original: For the 10% labeled set setting, we use a batch size of 512 for the supervised objective and a batch size of 15,360 for the unsupervised objective.
Revised: For the 10% labeled set setting, we use a batch size of 512 for the supervised objective and a batch size of 15,360 for the unsupervised objective.

Original: We use a base learning rate of 0.3 that is decayed by 10 for four times and set the weight on the unsupervised objective λ to 20.
Revised: We use a base learning rate of 0.3 that is decayed by 10 for four times and set the weight on the unsupervised objective λ to 20.

Original: We mask out unlabeled examples whose highest probabilities across categories are less than 0.5 and set the Softmax temperature to 0.4.
Revised: We mask out unlabeled examples whose highest probabilities across categories are less than 0.5 and set the Softmax temperature to 0.4.

Original: The model is trained for 40k steps.
Revised: The model is trained for 40k steps.

Original: Experiments in this part are performed on a v3-64 Cloud TPU v3 Pod.
Revised: Experiments in this part are performed on a v3-64 Cloud TPU v3 Pod.

Original: Full Labeled Set Setting.
Revised: Full Labeled Set Setting.

Original: For experiments on the full ImageNet, we use a batch size of 8,192 for the supervised objective and a batch size of 16,384 for the unsupervised objective.
Revised: For experiments on the full ImageNet, we use a batch size of 8,192 for the supervised objective and a batch size of 16,384 for the unsupervised objective.

Original: The weight on the unsupervised objective λ is set to 1.
Revised: The weight on the unsupervised objective λ is set to 1.

Original: We use entropy minimization to sharpen the prediction.
Revised: We use entropy minimization to sharpen the prediction.

Original: We use a base learning rate of 1.6 and decay it by 10 for four times.
Revised: We use a base learning rate of 1.6 and decay it by 10 for four times.

Original: Experiments in this part are performed on a v3-128 Cloud TPU v3 Pod.
Revised: Experiments in this part are performed on a v3-128 Cloud TPU v3 Pod.