Original: Training strategies with unlabeled and few labeled examples under 1-pixel attack by combining supervised and self-supervised learningAnonymous Authors
Revised: Training strategies with unlabeled and few labeled examples under 1-pixel attack by combining supervised and self-supervised learning

Original: Abstract
Revised: Abstract

Original: Self-supervised learning pre-training exhibited excellent performance on feature learning by using only unlabeled examples.
Revised: Self-supervised learning pre-training exhibited excellent performance on feature learning by using only unlabeled examples.

Original: Still, it is not clear how different self-supervised tasks perform under distinct image domains and there are still training issues to be tackled under scenarios of limited labeled data.
Revised: Still, it is not clear how different self-supervised tasks perform under distinct image domains and there are still training issues to be tackled under scenarios of limited labeled data.

Original: We investigate two self-supervised tasks: rotation and Barlow Twins, on three distinct image domains, exploring a combination of supervised and self-supervised learning.
Revised: We investigate two self-supervised tasks: rotation and Barlow Twins, on three distinct image domains, exploring a combination of supervised and self-supervised learning.

Original: Our motivation is to work on scenarios where the proportion of labeled data with respect to unlabeled data is small, as well as investigate the model’s robustness to 1-pixel attacks.
Revised: Our motivation is to work on scenarios where the proportion of labeled data with respect to unlabeled data is small, as well as investigate the model’s robustness to 1-pixel attacks.

Original: The models that combine supervised with self-supervised tasks can take advantage of the unlabeled data to improve the learned representation in terms of the linear discrimination, as well as allowing learning even under attack.
Revised: The models that combine supervised with self-supervised tasks can take advantage of the unlabeled data to improve the learned representation in terms of the linear discrimination, as well as allowing learning even under attack.

Original: 1. Introduction
Revised: 1. Introduction

Original: Deep convolutional neural networks mainly follow the supervised learning paradigm, in which many input-output pairs are required for training (Razavian et al., 2014; Kornblith et al., 2019).
Revised: Deep convolutional neural networks mainly follow the supervised learning paradigm, in which many input-output pairs are required for training (Razavian et al., 2014; Kornblith et al., 2019; Ponti et al., 2021).

Original: However, for a given task, manually labeled data are time-consuming and expensive to obtain, while unlabeled data is often widely available.
Revised: However, for a given task, manually labeled data are time-consuming and expensive to obtain, while unlabeled data is often widely available.

Original: Also, some applications involve patterns that are not present in standard large-scale benchmark datasets.
Revised: Also, some applications involve patterns that are not present in standard large-scale benchmark datasets.

Original: Strategies using unlabeled and limited annotated data are paramount in this scenario.
Revised: Strategies using unlabeled and limited annotated data are paramount in this scenario (Cavallari et al., 2018; Dos Santos et al., 2020).

Original: Self-supervised learning (Kolesnikov et al., 2019) is showing promising results using only unlabeled data, being able to learn features that are competitive with respect to supervised baselines (He et al., 2020; Chen et al., 2020; Grill et al., 2020; Caron et al., 2020).
Revised: Self-supervised learning (Kolesnikov et al., 2019) is showing promising results using only unlabeled data, being able to learn features that are competitive with respect to super- vised baselines (He et al., 2020; Chen et al., 2020; Grill et al., 2020; Caron et al., 2020).

Original: However, methods to improve training of self-supervised tasks are still to be investigated.
Revised: However, methods to improve training of self-supervised tasks are still to be investigated.

Original: In this paper we aim at making the best use of small labeled training sets, with the additional challenge of not being able to use a validation set, and still leverage unlabeled data.
Revised: In this paper we aim at making the best use of small labeled training sets, with the additional challenge of not being able to use a validation set, and still leverage unlabeled data.

Original: For that we explore different training techniques with two selfsupervised tasks: rotation-prediction (Gidaris et al., 2018) and Barlow Twins (Zbontar et al., 2021), and evaluate the learned representations in scenarios with unlabeled data, few labeled data and both unlabeled and few labeled data.
Revised: For that we explore different training techniques with two selfsupervised tasks: rotation-prediction (Gidaris et al., 2018) and Barlow Twins (Zbontar et al., 2021), and evaluate the learned representations in scenarios with unlabeled data, few labeled data and both unlabeled and few labeled data.

Original: Additionally, we evaluate model’s robustness under 1-pixel attacks.
Revised: Additionally, we evaluate model’s robustness under 1-pixel attacks.

Original: We compare the methods using 3 image domains and show how unlabeled data with few labeled data can allow for more discriminative and robust representations compared to the supervised baselines.
Revised: We compare the methods using 3 image domains and show how unlabeled data with few labeled data can allow for more discriminative and robust representations compared to the supervised baselines.

Original: 2. Related Work
Revised: 2. Related Work

Original: Self-supervised learning relies on pretext tasks formulated using unlabeled data.
Revised: Self-supervised learning relies on pretext tasks formulated using unlabeled data.

Original: These models learnlearn low-level features as good as via strong supervision (Asano et al., 2019) even under minimal data-learning (Shi et al., 2020).
Revised: These models learn low-level features as good as via strong supervision (Asano et al., 2019) even under minimal data-learning (Shi et al., 2020).

Original: Semi-supervised learning (SSL) is a class of algorithms that learn considering both labeled and unlabeled data.
Revised: Semisupervised learning (SSL) is a class of algorithms that learn considering both labeled and unlabeled data.

Original: Consistency regularization methods add auxiliary loss terms computed on the unlabeled data.
Revised: Consistency regularization methods add auxiliary loss terms computed on the unlabeled data.

Original: The auxiliary loss terms can be considered as a regularizer.
Revised: The auxiliary loss terms can be considered as a regularizer.

Original: π -Model (Laine & Aila, 2017), Mean Teacher (Tarvainen & Valpola, 2017) and Virtual Adversarial Training (Miyato et al., 2018) that take advantage of consistency losses, among other works.
Revised: π -Model (Laine & Aila, 2017), Mean Teacher (Tarvainen & Valpola, 2017) and Virtual Adversarial Training (Miyato et al., 2018) that take advantage of consistency losses, among other works.

Original: In terms of the semi-supervised training strategy and the use of the rotation-prediction as an auxiliary task, our work is most related to S4L (Zhai et al., 2019), SESEMI (Tran, 2019), (Hendrycks et al., 2019) and (Cavallari & Ponti, 2021).
Revised: In terms of the semi-supervised training strategy and the use of the rotation-prediction as an auxiliary task, our work is most related to S4L (Zhai et al., 2019), SESEMI (Tran, 2019), (Hendrycks et al., 2019) and (Cavallari & Ponti, 2021).

Original: Zhai et al.
Revised: Zhai et al.

Original: (2019) train semi-supervised models with the rotation-prediction on the ImageNet dataset.
Revised: (2019) train semi-supervised models with the rotation-prediction on the ImageNet dataset.

Original: Tran (2019) uses rotation-prediction task as an auxiliary loss term to train the model for SVHN, CIFAR-10 and CIFAR-100.
Revised: Tran (2019) uses rotation-prediction task as an auxiliary loss term to train the model for SVHN, CIFAR-10 and CIFAR-100.

Original: Different from them, we tested also in Fashion-MNIST (a grayscale dataset) and the Malaria dataset, which is not angle oriented (unlike photographs that have angle bias).
Revised: Different from them, we tested also in Fashion-MNIST (a grayscale dataset) and the Malaria dataset, which is not angle oriented (unlike photographs that have angle bias).

Original: Cavallari & Ponti (2021) used only rotation and had unstable results due to the initialization sensitivity of the models, while we propose a new way to overcome this as well as exploring the recent
Revised: Cavallari & Ponti (2021) used only rotation and had unstable results due to the initialization sensitivity of the models, while we propose a new way to overcome this as well as exploring the recent

Original: Barlow Twins approach.
Revised: Barlow Twins approach.

Original: Hendrycks et al.
Revised: Hendrycks et al.

Original: (2019) indicates that self-supervision increases the robustness to adversarial examples, label and input corruptions.
Revised: (2019) indicates that self-supervision increases the robustness to adversarial examples, label and input corruptions.

Original: We instead perform 1-pixel attacks, use a siamese network, and evaluate the feature representations and not directly the network model. 
Revised: We instead perform 1-pixel attacks, use a siamese network, and evaluate the feature representations and not directly the network model. Gidaris et al. and Su et al.

Revised: (2020) also explore selfsupervision as an auxiliary task but in a few-shot learning pipeline.

Original: 3. Method
Revised: 3. Method

Original: We are interested in evaluating the feature representations resulting from using self-supervised pre-training methods, fine-tuning and semi-supervised alternatives for image classification problem on different domains, including natural and biomedical images.
Revised: We are interested in evaluating the feature representations resulting from using self-supervised pre-training methods, fine-tuning and semi-supervised alternatives for image classification problem on different domains, including natural and biomedical images.

Original: Our experiments consider a small labeled dataset D l with N l pairs of images and labels, and a much larger unlabeled dataset Du containing N u images, from which five experiments were performed:
Revised: Our experiments consider a small labeled dataset D l with N l pairs of images and labels, and a much larger unlabeled dataset Du containing N u images, from which five experiments were performed:

Original: Supervised training in limited annotated data , using 1% or 5% of labeled data in relation to the total of unlabeled data.
Revised: Supervised training in limited annotated data , using 1% or 5% of labeled data in relation to the total of unlabeled data.

Original: The annotated data are from the D l set; 2.
Revised: The annotated data are from the D l set; 2.

Original: Unsupervised training with RotNet or BarlowTwins , which do not use labels, on the unlabeled D u dataset; 3.
Revised: Unsupervised training with RotNet or BarlowTwins , which do not use labels, on the unlabeled D u dataset; 3.

Original: Semi-supervised training by fine-tuning from the frozen weights of the models trained in step 2, using 1% or 5% of labeled data in relation to the total of unlabeled data; 4.
Revised: Semi-supervised training by fine-tuning from the frozen weights of the models trained in step 2, using 1% or 5% of labeled data in relation to the total of unlabeled data; 4.

Original: Unified semi-supervised training that uses 1% or 5% of the labeled data in relation to the total of unlabeled data, that is trained simultaneously with the unlabeled data D u through siamese architecture and unified cost function:
Revised: Unified semi-supervised training that uses 1% or 5% of the labeled data in relation to the total of unlabeled data, that is trained simultaneously with the unlabeled data D u through siamese architecture and unified cost function:

Original: [Equation]
Revised: [Equation]

Original: where both ℓ l and ℓ u optimize a cross-entropy loss function: the former with the traditional supervised paradigm, and the latter with a self-supervised task.
Revised: where both ℓ l and ℓ u optimize a cross-entropy loss function: the former with the traditional supervised paradigm, and the latter with a self-supervised task

Original: Weights λ l , λ u > 0 .
Revised: Weights λ l , λ u > 0 .

Original: It can be used under different self-supervised losses ℓ u .
Revised: It can be used under different self-supervised losses ℓ u .

Original: The network has shared weights between the supervised task and the self-supervised task in the main architecture.
Revised: The network has shared weights between the supervised task and the self-supervised task in the main architecture.

Original: Fine-tuning the semi-supervised method from step 4, with 1%/5% of labels in relation to the total of unlabeled data.
Revised: Fine-tuning the semi-supervised method from step 4, with 1%/5% of labels in relation to the total of unlabeled data.

Original: We use a ResNet50 backbone, discarding the final classification layer and adding a Global Average Pooling layer, that outputs 2048 values.
Revised: We use a ResNet50 backbone, discarding the final classification layer and adding a Global Average Pooling layer, that outputs 2048 values.

Original: Then, we added a fully connected layer with ReLU with 128 dimensions which outputs the final representation evaluated in the experiments
Revised: Then, we added a fully connected layer with ReLU with 128 dimensions, which outputs the final representation evaluated in the experiments.

Original: For the semi-supervised method (see Figure , after a
Revised: For the semi-supervised method (see Figure 1), after a

Original: ResNet backbone, separate branches are used: one softmax layer for classification (classes’ output) and another branch for self-supervision.
Revised: ResNet50 backbone, separate branches are used: one softmax layer for classification (classes’ output), and another branch for self-supervision.

Original: For RotNet a softmax layer predicting rotation degrees 0, 90, 180 and 270.
Revised: For RotNet a softmax layer predicting rotation degrees 0, 90, 180 and 270.

Original: For Barlow Twins, two branches of the main network, that receive as input two altered views of the same image, are connected to a three-layer MLP.
Revised: For Barlow Twins, two branches of the main network, that receive as input two altered views of the same image, are connected to a three-layer MLP.

Original: As in Zbontar et al.
Revised: As in Zbontar et al.

Original: (2021), the first two layers have Batch Normalization and ReLu, and the third one does not, but we used 2048-d layers instead of 8192.
Revised: (2021), the first two layers have Batch Normalization and ReLu, and the third one does not, but we used 2048-d layers instead of 8192.

Revised: The Barlow Twins task uses the same data augmentation pipeline and transformations from the original paper.

Revised: The rotation task only uses rotation.

Revised: The labeled images are not modified with data augmentations.

Original: 3.1. Training procedure and learning rates
Revised: 3.1. Training procedure and learning rates

Original: All models were trained from scratch.
Revised: All models were trained from scratch.

Original: For all experiments λ l = λ u = 1 . 0 .
Revised: For all experiments λ l = λ u = 1 . 0 .

Original: For experiments using labeled data, each training was performed using a random partition of labeled data, with the same labeled random datasets for supervised and semi-supervised training.
Revised: For experiments using labeled data, each training was performed using a random partition of labeled data, with the same labeled random datasets for supervised and semi-supervised training.

Original: In the first exploratory experiments, we noticed that the convergence of self-supervised methods were highly sensitive to initialization and learning rates (LR).
Revised: In the first exploratory experiments, we noticed that the convergence of self-supervised methods were highly sensitive to initialization and learning rates (LR).

Original: To overcome this, for each model we searched LR hyper-parameters: 0.01, 0.001 and 0.0001, with fixed exponential decay e − 0 .
Revised: To overcome this, for each model we searched LR hyper-parameters: 0.01, 0.001 and 0.0001, with fixed exponential decay e − 0 .

Original: 01 after 5% of the total number of epochs.
Revised: 01 after 5% of the total number of epochs.

Original: Thus, for every LR value 5 models are trained, each with a different weight initialization seed.
Revised: Thus, for every LR value 5 models are trained, each with a different weight initialization seed.

Original: Under the assumption that a small training set does not allow the ’luxury’ of separating data for validation, the models used in testing are those 5 trained with the LR that achieves the lowest mean value of final training loss value.
Revised: Under the assumption that a small training set does not allow the ’luxury’ of separating data for validation, the models used in testing are those 5 trained with the LR that achieves the lowest mean value of final training loss value.

Original: This way we select the most consistent LR for every model.
Revised: This way we select the most consistent LR for every model.

Original: 3.2. Datasets
Revised: 3.2. Datasets

Original: We assess the performance of our experiments on: STL10 (Coates et al., 2011), Fashion-MNIST (Xiao et al., 2017) and Malaria (Rajaraman et al., 2018).
Revised: We assess the performance of our experiments on: STL10 (Coates et al., 2011), Fashion-MNIST (Xiao et al., 2017) and Malaria (Rajaraman et al., 2018).

Original: STL-10 is designed for semi-supervised and unsupervised feature learning, containing 96 × 96 RGB images with airplane, bird, car, cat, deer, dog, horse, monkey, ship and truck classes.
Revised: STL-10 is designed for semi-supervised and unsupervised feature learning, containing 96 × 96 RGB images with airplane, bird, car, cat, deer, dog, horse, monkey, ship and truck classes.

Original: FashionMNIST has 28 × 28 grayscale images with centered pieces of 10 classes of clothing and fashion accessories.
Revised: FashionMNIST has 28 × 28 grayscale images with centered pieces of 10 classes of clothing and fashion accessories.

Original: Malaria contains instances of parasitized and uninfected cells from the thin blood smear slide images of segmented cells.
Revised: Malaria contains instances of parasitized and uninfected cells from the thin blood smear slide images of segmented cells.

Original: 3.3. Experimental setup
Revised: 3.3. Experimental setup

Original: Batch size of 32 was used in all experiments, except for
Revised: Batch size of 32 was used in all experiments, except for

Original: Barlow Twins, since it benefits from larger batches.
Revised: Barlow Twins, since it benefits from larger batches.

Original: In this case we use batch size 200 for STL-10 and FashionMNIST, and size 120 for Malaria.
Revised: In this case we use batch size 200 for STL-10 and FashionMNIST, and size 120 for Malaria.

Original: For the STL-10 dataset, we used the original size of 96×96.
Revised: For the STL-10 dataset, we used the original size of 96×96.

Original: For the Fashion-MNIST dataset, the images were upsized to 96×96.
Revised: For the Fashion-MNIST dataset, the images were upsized to 96×96.

Original: For the Malaria dataset, images were downsized to 128×128.
Revised: For the Malaria dataset, images were downsized to 128×128.

Original: The number of epochs was chosen empirically for each model, observing stabilization of the loss value between subsequent epochs.
Revised: The number of epochs was chosen empirically for each model, observing stabilization of the loss value between subsequent epochs.

Original: The unified semi-supervised training has shared weights and receives minibatches with an equal amount of labeled and unlabeled images (balancing supervised and self-supervised tasks).
Revised: The unified semi-supervised training has shared weights and receives minibatches with an equal amount of labeled and unlabeled images (balancing supervised and self-supervised tasks).

Original: At each epoch, the model sees all unlabeled N u , while the labeled ones N l are seen N u / N l times.
Revised: At each epoch, the model sees all unlabeled N u , while the labeled ones N l are seen N u / N l times.

Original: Because
Revised: Because

Original: N u > N l and batches are balanced, in one epoch all unlabeled images are seen.
Revised: N u > N l and batches are balanced, in one epoch all unlabeled images are seen.

Original: Therefore, our network sees repeated labeled instances in an epoch.
Revised: Therefore, our network sees repeated labeled instances in an epoch.

Original: We compensate that by allowing more epochs for the supervised setting.
Revised: We compensate that by allowing more epochs for the supervised setting.

Original: 3.4. Evaluation
Revised: 3.4. Evaluation

Original: To assess the discriminative capacity of learned representations, we train a linear SVM to assess the accuracy as a proxy measure for linear separability (Mello & Ponti, 2018), on the features obtained by the 128-D layer just after the
Revised: To assess the discriminative capacity of learned representations, we train a linear SVM to assess the accuracy as a proxy measure for linear separability (Mello & Ponti, 2018), on the features obtained by the 128-D layer just after the

Original: Global Average Pooling of ResNet50 backbone.
Revised: Global Average Pooling of ResNet50 backbone.

Original: Using the best 5 models (with different initialization seeds) trained in experiments 1 to 5 of as a feature extractor, we consider the following steps for evaluation:
Revised: Using the best 5 models (with different initialization seeds) trained in experiments 1 to 5 of as a feature extractor, we consider the following steps for evaluation:

Original: Extract the training set representations ; – 2.
Revised: Extract the training set representations ; – 2.

Original: Train Linear SVM (with C = 1 ) using the representations extracted in step 1; – 3.
Revised: Train Linear SVM (with C = 1 ) using the representations extracted in step 1; – 3.

Original: Extract the test set representations ; – 4.
Revised: Extract the test set representations ; – 4.

Original: Test the SVM trained in step 2 on test set representations .
Revised: Test the SVM trained in step 2 on test set representations .

Revised: We tested other C values for the SVM but it didn’t change the results.

Original: The test set has never been seen during model or SVM training, and is only used to obtain the accuracies reported in Tables 1 and 2.
Revised: The test set has never been seen during model or SVM training, and is only used to obtain the accuracies reported in Tables 1 and 2.

Original: The SVM was trained with 1,000 or 5, images (1% or 5%) for the STL-10 dataset; 600 or 3, images (1% or 5%) for Fashion-MNIST; 137 or 685 images (1% or 5%) for Malaria; and they are the same sets used in training the models when the model uses labeled data.
Revised: The SVM was trained with 1, or 5,000 images (1% or 5%) for the STL-10 dataset; 600 or 3,000 images (1% or 5%) for Fashion-MNIST; 137 orimages (1% or 5%) for Malaria; and they are the same sets used in training the models when the model uses labeled data.

Original: The 1-pixel attack case was produced assuming access to the training data.
Revised: The 1-pixel attack case was produced assuming access to the training data.

Original: Thus, we attack only images contained in D l .
Revised: Thus, we attack only images contained in D l .

Original: For each class we arbitrarily insert a white pixel at the same coordinate in all images.
Revised: For each class we arbitrarily insert a white pixel at the same coordinate in all images.

Revised: Since we do not use data augmentations for the labeled images, the location of the attacked single pixel is not affected.

Original: 4. Results
Revised: 4. Results

Original: The results for all datasets using regular data and attacked data are shown inTable 1 for the  use of 1% labeled data, and 2 for 5% of  labeled data.
Revised: The STL-10 dataset originally consists of 100,000 images in the unlabeled set, 5,000 images in the training set and 8, images in the test set. When training the supervised models, we use either 1,000 or 5,000 images from the training set as labeled images (1% or 5% of images in relation to the total unsupervised set). When training the semi-supervised model, we use same fractions of labeled data but also use the whole 100,000 unlabeled images. When training the purely unsupervised (self-supervised) models we use only the whole set of 100,000 unlabeled images.

Revised: The Fashion-MNIST dataset originally consists of 60, images in the training set and 10,000 images in the test set.

Original: Mean and standard deviation of  the 5 seeds are shown.
Revised: When training the supervised models, we use either 1% (600) or 5% (3,000) of labeled images. When training the semi-supervised model, we use the same fractions of labeled data but also use the whole 60,000 images as unlabeled data. When training the purely unsupervised (self-supervised) models we use only the whole set of 60,000 training images discarding the labels. Malaria dataset originally consists of 27,558 labeled images in total.

Revised: For our experiments we consider half of total images for the training set and the other half for the test set.

Original: Bold Values are the highest accuracies considering also  the standard deviation superposition.
Revised: When training the supervised baselines, we use either 1% (137) or 5% (685) of the training set. When training the semi-supervised model, we use the same fractions of labeled data but also use the whole training set of 13, images as unlabeled data. When training the purely unsupervised (self-supervised) models we use only the whole set of 13,779 unlabeled images. The results for all datasets using regular data and attacked data are shown in Table

Original: When using either 1% or 5% of available labeled data, the self-semi-supervised models excel in STL-10 and Fashion.
Revised: When using either 1% or 5% of available labeled data, the self-semi-supervised models excel in STL-10 and Fashion.

Original: In particular, the datasets under 1-pixel attack suffer from the use of supervised learning only, while self-semisupervised training or pretraining allow significant improvement.
Revised: In particular, the datasets under 1-pixel attack suffer from the use of supervised learning only, while self-semisupervised training or pretraining allow significant improvement.

Original: This may indicate that the simultaneous training was able to help the model’s robustness by guiding the cost function simultaneously in the self-supervised task, which uses unlabeled data, and in the common classification task, which used supervised data with noise.
Revised: This may indicate that the simultaneous training was able to help the model’s robustness by guiding the cost function simultaneously in the self-supervised task, which uses unlabeled data, and in the common classification task, which used supervised data with noise.

Original: When compared with the best results, we note that the finetuning of the semi-supervised model worsened the accuracy compared to the semi-supervised models without fine-tuning except for Fashion 5%.
Revised: When compared with the best results, we note that the finetuning of the semi-supervised model worsened the accuracy compared to the semi-supervised models without fine-tuning except for Fashion 5%.

Original: Interestingly, both self-supervised tasks alone achieved a more competitive accuracy than the supervised model when only 1% of labeled data is available.
Revised: Interestingly, both self-supervised tasks alone achieved a more competitive accuracy than the supervised model when only 1% of labeled data is available.

Original: It is noteworthy that Malaria with 5% of data was able to improve all methods in the non-attacked dataset, but the attacked one suffered significantly.
Revised: It is noteworthy that Malaria with 5% of data was able to improve all methods in the non-attacked dataset, but the attacked one suffered significantly.

Original: Contrary to what usually happened with other datasets, fine-tuning the semisupervised model brings an improvement to the accuracy in all scenarios, in both tasks and with both 1% and 5% of labeled data.
Revised: Contrary to what usually happened with other datasets, fine-tuning the semisupervised model brings an improvement to the accuracy in all scenarios, in both tasks and with both 1% and 5% of labeled data.

Original: Standard deviation for supervised models using 1% of data were large, indicating unstable training compared to other experiments in general.
Revised: Standard deviation for supervised models using 1% of data were large, indicating unstable training compared to other experiments in general.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 4.1. Discussion on the pretext tasks
Revised: 4.1. Discussion on the pretext tasks

Original: Overall, the Barlow Twins performed better for the STLdataset.
Revised: Overall, the Barlow Twins performed better for the STLdataset.

Original: This task uses data augmentations that can benefit scenarios where we have natural color images, which is perhaps the reason why we got good results for the STL-10.
Revised: This task uses data augmentations that can benefit scenarios where we have natural color images, which is perhaps the reason why we got good results for the STL-10.

Original: A better design of specific image pre-processing may yield better results for Fashion-MNIST and Malaria datasets.
Revised: A better design of specific image pre-processing may yield better results for Fashion-MNIST and Malaria datasets.

Original: The Rotation task performed better for the Fashion and
Revised: The Rotation task performed better for the Fashion and

Original: Malaria datasets.
Revised: Malaria datasets.

Original: The Semi-supervised models had the best performance in general, followed by the models that used some self-supervision + Fine-tuning task.
Revised: The Semi-supervised models had the best performance in general, followed by the models that used some self-supervision + Fine-tuning task.

Original: The biomedical image domain is not angle-oriented, thus the rotation task becomes harder.
Revised: The biomedical image domain is not angle-oriented, thus the rotation task becomes harder.

Original: Nevertheless, significant improvement was found when incorporating unlabeled data.
Revised: Nevertheless, significant improvement was found when incorporating unlabeled data.

Original: Also the attack had an even stronger impact on results due to the nature of the images, that contain patterns that are similar to the attack, degrading the results.
Revised: Also the attack had an even stronger impact on results due to the nature of the images, that contain patterns that are similar to the attack, degrading the results.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: Obtaining large annotated datasets remains a limitation in training deep neural nets.
Revised: Obtaining large annotated datasets remains a limitation in training deep neural nets.

Original: Investigating different pre-training strategies that work under limited labeled data while leveraging unlabeled data is important.
Revised: Investigating different pre-training strategies that work under limited labeled data while leveraging unlabeled data is important.

Original: Combining self-supervision with supervised learning using rotation prediction and the
Revised: Combining self-supervision with supervised learning using rotation prediction and the

Original: Barlow Twins task is a good choice towards this objective.
Revised: Barlow Twins task is a good choice towards this objective.

Original: We also propose a way to overcome the sensitivity to initialization and learning-rate hyper-parameter by using the scarce training data only.
Revised: We also propose a way to overcome the sensitivity to initialization and learning-rate hyper-parameter by using the scarce training data only.

Original: Our method goes beyond the use self-supervised tasks as pre-training, by simultaneously training supervised and selfsupervised with a siamese architecture.
Revised: Our method goes beyond the use self-supervised tasks as pre-training, by simultaneously training supervised and selfsupervised with a siamese architecture.

Original: Our semi-supervised model achieved overall the highest accuracy, followed by those using fine-tuning of a self-supervised pretext task.
Revised: Our semi-supervised model achieved overall the highest accuracy, followed by those using fine-tuning of a self-supervised pretext task.

Original: Not only we improve numerical results, but learn more discriminative spaces, as well as a more robust representation against 1-pixel attacks.
Revised: Not only we improve numerical results, but learn more discriminative spaces, as well as a more robust representation against 1-pixel attacks.

Original: Results show that the choice of the pretext task must take into account the nature of the dataset, and that a single task may not suit all applications.
Revised: Results show that the choice of the pretext task must take into account the nature of the dataset, and that a single task may not suit all applications.

Original: Future work may investigate other types of auxiliary tasks in the context of semi-supervised learning, explore different weights for supervised/self-supervised losses, as well as test against other undesired scenarios of attack.
Revised: Future work may investigate other types of auxiliary tasks in the context of semi-supervised learning, explore different weights for supervised/self-supervised losses, as well as test against other undesired scenarios of attack.

Original: In particular, we believe that handcrafting or learning pretext tasks for each dataset is a promising path. 
Revised: In particular, we believe that handcrafting or learning pretext tasks for each dataset is a promising path. Acknowledgments

Revised: The authors would like to thank FAPESP (grants #2019/07316-0 and #2019/02033-0) and CNPq (National

Revised: Council of Technological and Scientific Development) grant 304266/2020-5.