Original: S PLIT B ATCH N ORMALIZATION : I MPROVING S EMIS UPERVISED L EARNING UNDER D OMAIN S HIFT
Revised: S PLIT B ATCH N ORMALIZATION : I MPROVING S EMIS UPERVISED L EARNING UNDER D OMAIN S HIFT

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent work has shown that using unlabeled data in semi-supervised learning is not always beneﬁcial and can even hurt generalization, especially when there is a class mismatch between the unlabeled and labeled examples.
Revised: Recent work has shown that using unlabeled data in semi-supervised learning is not always beneﬁcial and can even hurt generalization, especially when there is a class mismatch between the unlabeled and labeled examples.

Original: We investigate this phenomenon for image classiﬁcation and many other forms of domain shifts (e g. salt-and-pepper noise).
Revised: We investigate this phenomenon for image classiﬁcation and many other forms of domain shifts (e g. salt-and-pepper noise).

Original: Our main contribution is showing how to beneﬁt from additional unlabeled data that comes from a shifted distribution in batch-normalized neural networks.
Revised: Our main contribution is showing how to beneﬁt from additional unlabeled data that comes from a shifted distribution in batch-normalized neural networks.

Original: We achieve it by simply using separate batch normalization statistics for unlabeled examples.
Revised: We achieve it by simply using separate batch normalization statistics for unlabeled examples.

Original: Due to its simplicity, we recommend it as a standard practice
Revised: Due to its simplicity, we recommend it as a standard practice

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) rely on large amounts of labeled data to achieve state of the art performance on supervised learning problems such as image classiﬁcation or speech recognition (Goodfellow et al., 2016).
Revised: Deep neural networks (DNNs) rely on large amounts of labeled data to achieve state of the art performance on supervised learning problems such as image classiﬁcation or speech recognition (Goodfellow et al., 2016).

Original: However, labeling data can be prohibitively costly.
Revised: However, labeling data can be prohibitively costly.

Original: Consequently, a common research theme is leveraging unlabeled data to improve sample-efﬁciency of deep networks.
Revised: Consequently, a common research theme is leveraging unlabeled data to improve sample-efﬁciency of deep networks.

Original: Semi-supervised learning (SSL) is one possible approach (Chapelle et al., 2010).
Revised: Semi-supervised learning (SSL) is one possible approach (Chapelle et al., 2010).

Original: SSL methods can boost performance of DNNs by jointly training on the labeled and unlabeled data (Miyato et al., 2018; Tarvainen & Valpola, 2017).
Revised: SSL methods can boost performance of DNNs by jointly training on the labeled and unlabeled data (Miyato et al., 2018; Tarvainen & Valpola, 2017).

Original: General assumption in SSL is that the labeled examples comes from the same distribution as the unlabeled examples.
Revised: General assumption in SSL is that the labeled examples comes from the same distribution as the unlabeled examples.

Original: It is unclear to what extent we could generally use additional data that is out of distribution.
Revised: It is unclear to what extent we could generally use additional data that is out of distribution.

Original: When unsupervised data comes from distribution that is not very distant from supervised data, e.g. from the same dataset but different classes, one could hope to still have SSL performing better than pure supervised solution.
Revised: When unsupervised data comes from distribution that is not very distant from supervised data, e.g. from the same dataset but different classes, one could hope to still have SSL performing better than pure supervised solution.

Original: However, recently Oliver et al.
Revised: However, recently Oliver et al.

Original: (2018) have undermined this hope showing that SSL methods are not robust to class mismatch and their performance could be even worse than pure supervised learning.
Revised: (2018) have undermined this hope showing that SSL methods are not robust to class mismatch and their performance could be even worse than pure supervised learning.

Original: In this work we investigate and improve SSL techniques’ performance in a scenario when unlabeled and labeled examples do not come from the same distribution.
Revised: In this work we investigate and improve SSL techniques’ performance in a scenario when unlabeled and labeled examples do not come from the same distribution.

Original: We focus on deep neural networks with batch normalization (Ioffe & Szegedy, 2015) for image classiﬁcation.
Revised: We focus on deep neural networks with batch normalization (Ioffe & Szegedy, 2015) for image classiﬁcation.

Original: We verify that indeed in this setting SSL methods hurt from domain shift of unlabeled data.
Revised: We verify that indeed in this setting SSL methods hurt from domain shift of unlabeled data.

Original: To alleviate this problem, we propose to compute separately batch-normalization statistics for the unsupervised and supervised data.
Revised: To alleviate this problem, we propose to compute separately batch-normalization statistics for the unsupervised and supervised data.

Original: We experimentally study the effectiveness of this method on a substantially extended version of the setting considered by Oliver et al.
Revised: We experimentally study the effectiveness of this method on a substantially extended version of the setting considered by Oliver et al.

Original: (2018) that includes other possible domain shifts such as salt-and-pepper noise or a change in image contrast in the unlabeled data.
Revised: (2018) that includes other possible domain shifts such as salt-and-pepper noise or a change in image contrast in the unlabeled data.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Semi-supervised learning is a popular technique to leverage unlabeled data along with (typically a small amount of) labeled data (Chapelle et al., 2010).
Revised: Semi-supervised learning is a popular technique to leverage unlabeled data along with (typically a small amount of) labeled data (Chapelle et al., 2010).

Original: However, in some cases using unlabeled examples can hurt performance of the model (Gagliardi Cozman et al., 2003; Oliver et al., 2018).
Revised: However, in some cases using unlabeled examples can hurt performance of the model (Gagliardi Cozman et al., 2003; Oliver et al., 2018).

Original: [Figure]
Revised: [Figure]

Original: While there are many approaches to SSL, most of them implicitely or explicitely assume the unlabeled examples follow the distribution of the supervised dataset (Liu et al., 2018).
Revised: While there are many approaches to SSL, most of them implicitely or explicitely assume the unlabeled examples follow the distribution of the supervised dataset (Liu et al., 2018).

Original: As highlighted by
Revised: As highlighted by

Original: Oliver et al.
Revised: Oliver et al.

Original: (2018), a setting in which unlabeled examples follow a different distribution is heavily under-researched.
Revised: (2018), a setting in which unlabeled examples follow a different distribution is heavily under-researched.

Original: Some works, for example Laine & Aila (2016), consider in their experiments unlabeled data that is out-of-distribution but do not address or analyze the problem.
Revised: Some works, for example Laine & Aila (2016), consider in their experiments unlabeled data that is out-of-distribution but do not address or analyze the problem.

Original: The most related works are Oliver et al.
Revised: The most related works are Oliver et al.

Original: (2018), which observes that SSL performance can degrade substantially when the unlabeled dataset contains out-of-distribution examples, and Liu et al.
Revised: (2018), which observes that SSL performance can degrade substantially when the unlabeled dataset contains out-of-distribution examples, and Liu et al.

Original: (2018), which similarly to us analyzes robustness of SSL techniques.
Revised: (2018), which similarly to us analyzes robustness of SSL techniques.

Original: Liu et al.
Revised: Liu et al.

Original: (2018) considers a scenario in which there is a domain shift coming from labels that are missing not completely at random, which can lead to a mismatch in feature distribution.
Revised: (2018) considers a scenario in which there is a domain shift coming from labels that are missing not completely at random, which can lead to a mismatch in feature distribution.

Original: Importantly, the setting we consider is more general – we allow for a systematic domain shift independent of the labeling process.
Revised: Importantly, the setting we consider is more general – we allow for a systematic domain shift independent of the labeling process.

Original: Further, our
Revised: Further, our

Original: Split-BN is complementary to the approach of Liu et al.
Revised: Split-BN is complementary to the approach of Liu et al.

Original: Recently Kalayeh & Shah (2019) and Deecke et al.
Revised: Recently Kalayeh & Shah (2019) and Deecke et al.

Original: (2019) investigated batch normalization variants that internally model the data using a multimodal distribution.
Revised: (2019) investigated batch normalization variants that internally model the data using a multimodal distribution.

Original: In particular, Deecke et al.
Revised: In particular, Deecke et al.

Original: (2019) has demonstrated improvements for multi-task supervised learning, a setting related to ours.
Revised: (2019) has demonstrated improvements for multi-task supervised learning, a setting related to ours.

Original: We believe that it may be interesting direction to apply their method to SSL with domain shift.
Revised: We believe that it may be interesting direction to apply their method to SSL with domain shift.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: One of the key claims in Oliver et al.
Revised: One of the key claims in Oliver et al.

Original: (2018) is that performance of SSL techniques can degrade drastically when the unlabeled data contains a different distribution of classes than the labeled data.
Revised: (2018) is that performance of SSL techniques can degrade drastically when the unlabeled data contains a different distribution of classes than the labeled data.

Original: We ﬁrst conﬁrm that a similar phenomenon happens on a wide range of other possible domain shifts in image domain on the CIFAR-10 dataset (see Figure 1 for examples).
Revised: We ﬁrst conﬁrm that a similar phenomenon happens on a wide range of other possible domain shifts in image domain on the CIFAR-10 dataset (see Figure 1 for examples).

Original: However, a similar analysis on the more challenging ImageNet dataset reveals that this phenomenon might not be fully general.
Revised: However, a similar analysis on the more challenging ImageNet dataset reveals that this phenomenon might not be fully general.

Original: Next, we propose and analyse a technique to improve SSL under domain shift.
Revised: Next, we propose and analyse a technique to improve SSL under domain shift.

Original: In particular we reexamine our extended setting and observe that in most cases our technique cancels the performance gap between supervised learning and SSL with misaligned data.
Revised: In particular we reexamine our extended setting and observe that in most cases our technique cancels the performance gap between supervised learning and SSL with misaligned data.

Original: 3. E XPERIMENTAL SETTING
Revised: 3. E XPERIMENTAL SETTING

Original: Our experiments largely base on Oliver et al.
Revised: Our experiments largely base on Oliver et al.

Original: In particular, we use the same strong
Revised: In particular, we use the same strong

Original: Wide
Revised: Wide

Original: ResNet model (Zagoruyko & Komodakis, 2016).
Revised: ResNet model (Zagoruyko & Komodakis, 2016).

Original: We perform experiments with two state-of-the-art
Revised: We perform experiments with two state-of-the-art

Original: SSL methods: Mean Teacher (MT) by Tarvainen & Valpola (2017) and Virtual Adversarial Training (VAT) by Miyato et al.
Revised: SSL methods: Mean Teacher (MT) by Tarvainen & Valpola (2017) and Virtual Adversarial Training (VAT) by Miyato et al.

Original: In all experiments we use 400 labels per class.
Revised: In all experiments we use 400 labels per class.

Original: In addition to the class mismatch from Oliver et al.
Revised: In addition to the class mismatch from Oliver et al.

Original: (2018), we use popular image distortions (e.g. salt-and-pepper noise) to introduce a systematic difference between the labeled and unlabeled examples.
Revised: (2018), we use popular image distortions (e.g. salt-and-pepper noise) to introduce a systematic difference between the labeled and unlabeled examples.

Original: In total, we study 8 different domain shift scenarios: the ﬁrst based on a class mismatch, and the rest based on applying a ﬁxed type of image distortion to all images in the unlabeled dataset.
Revised: In total, we study 8 different domain shift scenarios: the ﬁrst based on a class mismatch, and the rest based on applying a ﬁxed type of image distortion to all images in the unlabeled dataset.

Original: These distortions are meant to represent challenge present in real world applications.
Revised: These distortions are meant to represent challenge present in real world applications.

Original: See Figurefor visualization.
Revised: See Figurefor visualization.

Original: For a detailed description of the transformations, please refer to Appendix A.
Revised: For a detailed description of the transformations, please refer to Appendix A.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For simplicity, we ﬁx most of the hyperparameter values to the ones used in Oliver et al.
Revised: For simplicity, we ﬁx most of the hyperparameter values to the ones used in Oliver et al.

Original: CIFAR-10 experiments.
Revised: CIFAR-10 experiments.

Original: For the detailed description, please refer to Appendix B.
Revised: For the detailed description, please refer to Appendix B.

Original: The code to reproduce our experiments will be released upon publication.
Revised: The code to reproduce our experiments will be released upon publication.

Original: 3. S PLIT B ATCH N ORMALIZATION
Revised: 3. S PLIT B ATCH N ORMALIZATION

Original: Our main contribution is Split Batch Normalization (Split-BN), a technique to improve SSL under domain mismatch, applicable to deep neural networks.
Revised: Our main contribution is Split Batch Normalization (Split-BN), a technique to improve SSL under domain mismatch, applicable to deep neural networks.

Original: Inspired by the simple approach of Li et al.
Revised: Inspired by the simple approach of Li et al.

Original: (2016) to domain adaptation, we propose to compute separately batch-normalization statistics for the unsupervised and supervised dataset .
Revised: (2016) to domain adaptation, we propose to compute separately batch-normalization statistics for the unsupervised and supervised dataset .

Original: Note that during test time statistics from supervised data are used.
Revised: Note that during test time statistics from supervised data are used.

Original: First, we investigate the class mismatch setting on the CIFAR-10 dataset, as discussed in Oliver et al.
Revised: First, we investigate the class mismatch setting on the CIFAR-10 dataset, as discussed in Oliver et al.

Original: Table 1 reports the results.
Revised: Table 1 reports the results.

Original: We can see that in some cases baseline performance is even 3% below accuracy achieved by pure supervised training that does not use additional unsupervised data.
Revised: We can see that in some cases baseline performance is even 3% below accuracy achieved by pure supervised training that does not use additional unsupervised data.

Original: Critically, using Split-BN cancels most of the performance degradation in these cases.
Revised: Critically, using Split-BN cancels most of the performance degradation in these cases.

Original: To further study class mismatch, we run experiments on the more challenging ImageNet dataset (Deng et al., 2009) rescaled to 32x32 (as in Chrabaszcz et al.
Revised: To further study class mismatch, we run experiments on the more challenging ImageNet dataset (Deng et al., 2009) rescaled to 32x32 (as in Chrabaszcz et al.

Original: (2017)), from which we select 20 random classes as the supervised dataset.
Revised: (2017)), from which we select 20 random classes as the supervised dataset.

Original: Additional unlabeled data comes from a distribution with a varying class mismatch with the labeled data.
Revised: Additional unlabeled data comes from a distribution with a varying class mismatch with the labeled data.

Original: Results are reported in Table 2.
Revised: Results are reported in Table 2.

Original: We note that
Revised: We note that

Original: SSL methods are generally not robust to domain shift.
Revised: SSL methods are generally not robust to domain shift.

Original: Our
Revised: Our

Original: -BN in most cases leads to an improvement
Revised: -BN in most cases leads to an improvement

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: in contrast to the previous experiment, on the ImageNet dataset in all cases there is an apparent improvement over the pure supervised case and our Split-BN performs comparably to regular SSL.
Revised: in contrast to the previous experiment, on the ImageNet dataset in all cases there is an apparent improvement over the pure supervised case and our Split-BN performs comparably to regular SSL.

Original: Finally, we re-evaluate Oliver et al.
Revised: Finally, we re-evaluate Oliver et al.

Original: (2018) claim using other forms of domain shift applied to the unlabeled data, and test how Split-BN generalizes to this new setting.
Revised: (2018) claim using other forms of domain shift applied to the unlabeled data, and test how Split-BN generalizes to this new setting.

Original: Table 3 summarizes the results.
Revised: Table 3 summarizes the results.

Original: In all cases with domain shift we observe a large gap in performance, typically between 1% and 4% accuracy drop.
Revised: In all cases with domain shift we observe a large gap in performance, typically between 1% and 4% accuracy drop.

Original: Hence, the phenomenon reported by Oliver et al.
Revised: Hence, the phenomenon reported by Oliver et al.

Original: (2018) is further conﬁrmed and SSL methods indeed are not designed for a domain shift case.
Revised: (2018) is further conﬁrmed and SSL methods indeed are not designed for a domain shift case.

Original: Additionally, Split-BN typically cancels most of the negative effect resulting from the domain mismatch, and in many cases leads to a large improvement in performance.
Revised: Additionally, Split-BN typically cancels most of the negative effect resulting from the domain mismatch, and in many cases leads to a large improvement in performance.

Original: 3. I S B ATCH N ORMALIZATION THE C ULPRIT ?
Revised: 3. I S B ATCH N ORMALIZATION THE C ULPRIT ?

Original: A natural question is whether the lack of robustness of SSL methods simply comes from using batch normalization layers.
Revised: A natural question is whether the lack of robustness of SSL methods simply comes from using batch normalization layers.

Original: We repeat the experiments from Section 3.2 using a model from Miyato et al.
Revised: We repeat the experiments from Section 3.2 using a model from Miyato et al.

Original: (2018) and Tarvainen & Valpola (2017), where we removed all batch normalization layers.
Revised: (2018) and Tarvainen & Valpola (2017), where we removed all batch normalization layers.

Original: See
Revised: See

Original: Appendix B for experimental details.
Revised: Appendix B for experimental details.

Original: Table 4 reports the results.
Revised: Table 4 reports the results.

Original: We found that SSL performance may degrade under domain shift also in the case of architectures without batch normalization.
Revised: We found that SSL performance may degrade under domain shift also in the case of architectures without batch normalization.

Original: This suggests that batch normalization is not the root cause of instability of SSL learning methods.
Revised: This suggests that batch normalization is not the root cause of instability of SSL learning methods.

Original: 3. A CTIVATIONS ANALYSIS
Revised: 3. A CTIVATIONS ANALYSIS

Original: In this section we shed light on how Split-BN improves performance by visualizing the distributions of hidden activations during training.
Revised: In this section we shed light on how Split-BN improves performance by visualizing the distributions of hidden activations during training.

Original: We analyse one of the experiments in Section 3.2 corresponding to training with Mean Teacher and a domain shift introduced by a rotation.
Revised: We analyse one of the experiments in Section 3.2 corresponding to training with Mean Teacher and a domain shift introduced by a rotation.

Original: We compute the means of the activations in the layers before the ﬁrst and the last batch normalization layers, for three sets of examples: (i) labeled examples, (ii) unlabeled examples, and (iii) the whole dataset.
Revised: We compute the means of the activations in the layers before the ﬁrst and the last batch normalization layers, for three sets of examples: (i) labeled examples, (ii) unlabeled examples, and (iii) the whole dataset.

Original: Results are shown in Figure 2, and additionally we report evolution of standard deviations in Appendix C. Crucially, the means of the labeled examples and the unlabeled examples become markedly different before the last batch normalization.
Revised: Results are shown in Figure 2, and additionally we report evolution of standard deviations in Appendix C. Crucially, the means of the labeled examples and the unlabeled examples become markedly different before the last batch normalization.

Original: This suggests that one of the key mechanisms by which Split-BN improves performance is by bringing closer the two distributions.
Revised: This suggests that one of the key mechanisms by which Split-BN improves performance is by bringing closer the two distributions.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work we studied how to use semi-supervised learning if the unlabeled and labeled examples do not come from the same distributions.
Revised: In this work we studied how to use semi-supervised learning if the unlabeled and labeled examples do not come from the same distributions.

Original: We provide a simple recommendation: if you are using a neural network with batch normalization layers, you should compute the normalization statistics separately for the unlabeled and labeled data.
Revised: We provide a simple recommendation: if you are using a neural network with batch normalization layers, you should compute the normalization statistics separately for the unlabeled and labeled data.

Original: This leads to signiﬁcant gains in performance; typically canceling out the negative effect of domain shift.
Revised: This leads to signiﬁcant gains in performance; typically canceling out the negative effect of domain shift.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Michał Zaj ˛ac is co-ﬁnanced by National Centre for Research and Development as a part EU supported Smart Growth Operational Programme 2014-20120, project number POIR.01.01.0100-0392/17-00.
Revised: Michał Zaj ˛ac is co-ﬁnanced by National Centre for Research and Development as a part EU supported Smart Growth Operational Programme 2014-20120, project number POIR.01.01.0100-0392/17-00.

Original: Konrad ˙Zołna is supported by the National Science Center, Poland (2017/27/N/ST6/00828).
Revised: Konrad ˙Zołna is supported by the National Science Center, Poland (2017/27/N/ST6/00828, 2018/28/T/ST6/00211).

Original: Stanisław Jastrz˛ebski was supported by grantNo. DI 2014/016644 from
Revised: Stanisław Jastrz˛ebski was supported by grant

Original:  Ministry of Science and Higher Education, Poland, and No. 2017/24/T/ST6/00487 from National Science Center, Poland.
Revised: No. DI 2014/016644 from Ministry of Science and Higher Education, Poland, and No. 2017/24/T/ST6/00487 from National Science Center, Poland.

Original: A D ISTORTION DETAILS
Revised: A D ISTORTION DETAILS

Original: Before applying distortion we assume pixels are in range [0 , 1] .
Revised: Before applying distortion we assume pixels are in range [0 , 1] .

Original: As depicted in Figure we used the following distortions:
Revised: As depicted in Figure we used the following distortions:

Original: (a) none; (b) grayscale: use tf.image.rgb_to_grayscale and then stack 3 times to have the same input size; (c) uniform noise: add uniform noise in range [ − 0 .
Revised: (a) none; (b) grayscale: use tf.image.rgb_to_grayscale and then stack 3 times to have the same input size; (c) uniform noise: add uniform noise in range [ − 0 .

Original: and then clip the values back to [0 , 1] ; (d) salt-and-pepper: every pixel becomes white with probability 10% , black with probability 10% , and stays the same otherwise; (e) inverted colors: every color value goes from x to 1 − x ; (f) rotation: the whole image is rotated by 90 ◦ counterclockwise; (g) random contrast: contrast is changed to a random value taken uniformly from [0 . 2 , 0 . 8] ; (h) occlusion: a black 14x14 square is placed on top of the picture at a random location.
Revised: and then clip the values back to [0 , 1] ; (d) salt-and-pepper: every pixel becomes white with probability 10% , black with probability 10% , and stays the same otherwise; (e) inverted colors: every color value goes from x to 1 − x ; (f) rotation: the whole image is rotated by 90 ◦ counterclockwise; (g) random contrast: contrast is changed to a random value taken uniformly from [0 . 2 , 0 . 8] ; (h) occlusion: a black 14x14 square is placed on top of the picture at a random location.

Original: B D ETAILS OF EXPERIMENTAL SETTING
Revised: B D ETAILS OF EXPERIMENTAL SETTING

Original: We largely use implementation of the architecture and SSL methods from Oliver et al.
Revised: We largely use implementation of the architecture and SSL methods from Oliver et al.

Original: In all experiments we use the following: batch size equal to 100; Adam optimizer with all hyperparameters but learning rate set to default.
Revised: In all experiments we use the following: batch size equal to 100; Adam optimizer with all hyperparameters but learning rate set to default.

Original: In supervised data, there is 400 images per class.
Revised: In supervised data, there is 400 images per class.

Original: Every number we report is accuracy on test set computed using checkpoint with the highest validation accuracy during the training.
Revised: Every number we report is accuracy on test set computed using checkpoint with the highest validation accuracy during the training.

Original: Every experiment is repeated 2 times.
Revised: Every experiment is repeated 2 times.

Original: In experiments from Tables 1, 2, 3, we use Wide ResNet architecture (WRN-28-2 variant with depth 28 and width 2); all Mean Teacher and VAT parameters as in Oliver et al.
Revised: In experiments from Tables 1, 2, 3, we use Wide ResNet architecture (WRN-28-2 variant with depth 28 and width 2); all Mean Teacher and VAT parameters as in Oliver et al.

Original: Experiments from Table 

Revised: Experiments from Table

Original: We use the experimental setting from Oliver et al.
Revised: We use the experimental setting from Oliver et al.

Original: The classiﬁcation is performed on 6 animal classes from CIFAR-10.
Revised: The classiﬁcation is performed on 6 animal classes from CIFAR-10.

Original: Additional unsupervised data containsclasses and has a varied degree of class mismatch with supervised data – from 0% (no mismatch) to 100% (completely different classes).
Revised: Additional unsupervised data containsclasses and has a varied degree of class mismatch with supervised data – from 0% (no mismatch) to 100% (completely different classes).

Original: CIFAR-10 data is preprocessed using global contrast normalization and ZCA normalization.
Revised: CIFAR-10 data is preprocessed using global contrast normalization and ZCA normalization.

Original: Data is augmented with random horizontal ﬂips, random translation by up to 2 pixels, and Gaussian input noise with standard deviation 0.15.
Revised: Data is augmented with random horizontal ﬂips, random translation by up to 2 pixels, and Gaussian input noise with standard deviation 0.15.

Original: Number of steps is 500000.
Revised: Number of steps is 500000.

Original: Learning rate is as in Oliver et al.
Revised: Learning rate is as in Oliver et al.

Original: (2018): for Mean Teacher, initial learning rate is 0.0004, decaying by 0.2 after 400000 steps; for VAT, initial learning rate is 0.003, decaying by 0.2 after 400000 steps.
Revised: (2018): for Mean Teacher, initial learning rate is 0.0004, decaying by 0.2 after 400000 steps; for VAT, initial learning rate is 0.003, decaying by 0.2 after 400000 steps.

Original: Auxiliary SSL loss is warmed-up for 200000 steps.
Revised: Auxiliary SSL loss is warmed-up for 200000 steps.

Original: Experiments from Table 

Revised: Experiments from Table

Original: The classiﬁcation is performed on 20 random classes from ImageNet rescaled do 32x32.
Revised: The classiﬁcation is performed on 20 random classes from ImageNet rescaled do 32x32.

Original: Additional unsupervised data contains 20 classes and has a varied degree of class mismatch with supervised data – from 0% (no mismatch) to 100% (completely different classes).
Revised: Additional unsupervised data contains 20 classes and has a varied degree of class mismatch with supervised data – from 0% (no mismatch) to 100% (completely different classes).

Original: Data is augmented with random horizontal ﬂips.
Revised: Data is augmented with random horizontal ﬂips.

Original: Learning rate is as in Oliver et al.
Revised: Learning rate is as in Oliver et al.

Original: (2018): initial learning rate is 0.0004, decaying by 0.2 after 400000 steps.
Revised: (2018): initial learning rate is 0.0004, decaying by 0.2 after 400000 steps.

Original: Auxiliary SSL loss is warmed-up for 200000 steps.
Revised: Auxiliary SSL loss is warmed-up for 200000 steps.

Original: convolutional, 128 ﬁlters, 3 × 3 , same padding, leaky ReLU convolutional, 128 ﬁlters, 3 × 3 , same padding, leaky ReLU convolutional, 128 ﬁlters, 3 × 3 , same padding, leaky ReLU max pooling 2 ×dropout p =
Revised: convolutional, 128 ﬁlters, 3 × 3 , same padding, leaky ReLU convolutional, 128 ﬁlters, 3 × 3 , same padding, leaky ReLU convolutional, 128 ﬁlters, 3 × 3 , same padding, leaky ReLU max pooling 2 ×dropout p =

Original: 0 .convolutional, 256 ﬁlters, 3 × 3 , same padding, leaky ReLU convolutional, 256 ﬁlters, 3 × 3 , same padding, leaky ReLU convolutional, 256 ﬁlters, 3 × 3 , same padding, leaky ReLU max pooling 2 ×dropout p =
Revised: 0 .convolutional, 256 ﬁlters, 3 × 3 , same padding, leaky ReLU convolutional, 256 ﬁlters, 3 × 3 , same padding, leaky ReLU convolutional, 256 ﬁlters, 3 × 3 , same padding, leaky ReLU max pooling 2 ×dropout p =

Original: 0 .convolutional, 512 ﬁlters, 3 × 3 , valid padding, leaky ReLU convolutional, 256 ﬁlters, 1 × 1 , same padding, leaky ReLU convolutional, 128 ﬁlters, 1 × 1 , same padding, leaky ReLU average pooling ( 6 × 6 → 1 × 1) fully connected, 128 →
Revised: 0 .convolutional, 512 ﬁlters, 3 × 3 , valid padding, leaky ReLU convolutional, 256 ﬁlters, 1 × 1 , same padding, leaky ReLU convolutional, 128 ﬁlters, 1 × 1 , same padding, leaky ReLU average pooling ( 6 × 6 → 1 × 1) fully connected, 128 →

Original: [Figure]
Revised: [Figure]

Original: y axes.
Revised: y axes.

Original: Experiments from Table 

Revised: Experiments from Table

Original: Here we use CIFAR-10 without preprocessing.
Revised: Here we use CIFAR-10 without preprocessing.

Original: Data is augmented with random horizontal ﬂips and random translation by up to 2 pixels.
Revised: Data is augmented with random horizontal ﬂips and random translation by up to 2 pixels.

Original: For each experiment, we search for optimal learning rate in the set { 0 . 0003 , 0 .
Revised: For each experiment, we search for optimal learning rate in the set { 0 . 0003 , 0 .

Original: The training lasts forsteps.
Revised: The training lasts forsteps.

Original: Learning rate decays by 0.2 after 100000 steps.
Revised: Learning rate decays by 0.2 after 100000 steps.

Original: Auxiliary SSL loss coefﬁcient is warmed-up for 50000 steps to reach 8 . 0 for Mean Teacher and 0 .
Revised: Auxiliary SSL loss coefﬁcient is warmed-up for 50000 steps to reach 8 . 0 for Mean Teacher and 0 .

Original: 3 for VAT.
Revised: 3 for VAT.

Original: Experiments from Table 4 We use ConvNet architecture from Miyato et al.
Revised: Experiments from Table 4 We use ConvNet architecture from Miyato et al.

Original: and Tarvainen & Valpola (2017) with batch normalization eliminated – see Table 5 for details.
Revised: and Tarvainen & Valpola (2017) with batch normalization eliminated – see Table 5 for details.

Original: We use CIFARwithout preprocessing.
Revised: We use CIFARwithout preprocessing.

Original: Data is augmented with gaussian noise with standard deviation 0.15, random horizontal ﬂips and random translation by up to 2 pixels.
Revised: Data is augmented with gaussian noise with standard deviation 0.15, random horizontal ﬂips and random translation by up to 2 pixels.

Original: For each experiment, we search for optimal learning rate in the set { 0 . 0001 , 0 . 0003 , 0 . 001 } .
Revised: For each experiment, we search for optimal learning rate in the set { 0 . 0001 , 0 . 0003 , 0 . 001 } .

Original: The training lasts for 220000 steps.
Revised: The training lasts for 220000 steps.

Original: Learning rate decays by 0.25 after every 80000 steps.
Revised: Learning rate decays by 0.25 after every 80000 steps.

Original: Auxiliary SSL loss coefﬁcient is warmed-up for 50000 steps to reach 8 . 0 for Mean Teacher and 0 .
Revised: Auxiliary SSL loss coefﬁcient is warmed-up for 50000 steps to reach 8 . 0 for Mean Teacher and 0 .

Original: 1 for VAT.
Revised: 1 for VAT.

Original: C P LOTS OF ACTIVATIONS ’ STANDARD DEVIATIONS
Revised: C P LOTS OF ACTIVATIONS ’ STANDARD DEVIATIONS

Original: We present here Figure 3 as a supplementary material for the analysis from the subsection 3.4.
Revised: We present here Figure 3 as a supplementary material for the analysis from the subsection 3.4.