Original: A CRITICAL ANALYSIS OF SELF - SUPERVISION , OR WHAT WE CAN LEARN FROM A SINGLE IMAGE
Revised: A CRITICAL ANALYSIS OF SELF - SUPERVISION , OR WHAT WE CAN LEARN FROM A SINGLE IMAGE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels.
Revised: We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels.

Original: We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the ﬁrst few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used.
Revised: We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the ﬁrst few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used.

Original: However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training.
Revised: However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training.

Original: We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset
Revised: We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite tremendous progress in supervised learning, learning without external supervision remains difﬁcult.
Revised: Despite tremendous progress in supervised learning learning without external supervision remains difﬁcult.

Original: Self-supervision has recently emerged as one of the most promising approaches to address this limitation.
Revised: Self-supervision has recently emerged as one of the most promising approaches to address this limitation.

Original: Self-supervision builds on the fact that convolutional neural networks (CNNs) transfer well between tasks (Shin et al., 2016; Oquab et al., 2014; Girshick, 2015;
Revised: Self-supervision builds on the fact that convolutional neural networks (CNNs) transfer well between tasks (Shin et al., 2016; Oquab et al., 2014; Girshick, 2015;

Original: Huh et al., 2016).
Revised: Huh et al., 2016).

Original: The idea then is to pre-train networks via pre-text tasks that do not require expensive manual annotations and can be automatically generated from the data itself.
Revised: The idea then is to pre-train networks via pretext tasks that do not require expensive manual annotations and can be automatically generated from the data itself.

Original: Once pre-trained, networks can be applied to a target task by using only a modest amount of labelled data.
Revised: Once pre-trained, networks can be applied to a target task by using only a modest amount of labelled data.

Original: Early successes in self-supervision have encouraged authors to develop a large variety of pre-text tasks, from colorization to rotation estimation and image autoencoding.
Revised: Early successes in self-supervision have encouraged authors to develop a large variety of pretext tasks, from colorization to rotation estimation and image autoencoding.

Original: Recent papers have shown performance competitive with supervised learning by learning complex neural networks on very large image datasets.
Revised: Recent papers have shown performance competitive with supervised learning by learning complex neural networks on very large image datasets.

Original: Nevertheless, for a given model complexity, pre-training by using an off-theshelf annotated image datasets such as ImageNet remains much more efﬁcient.
Revised: Nevertheless, for a given model complexity, pre-training by using an off-theshelf annotated image datasets such as ImageNet remains much more efﬁcient.

Original: In this paper, we aim to investigate the effectiveness of current self-supervised approaches by characterizing how much information they can extract from a given dataset of images.
Revised: In this paper, we aim to investigate the effectiveness of current self-supervised approaches by characterizing how much information they can extract from a given dataset of images.

Original: Since deep networks learn a hierarchy of representations, we further break down this investigation on a per-layer basis.
Revised: Since deep networks learn a hierarchy of representations, we further break down this investigation on a per-layer basis.

Original: We are motivated by the fact that the ﬁrst few layers of most networks extract low-level information (Yosinski et al., 2014), and thus learning them may not require the high-level semantic information captured by manual labels.
Revised: We are motivated by the fact that the ﬁrst few layers of most networks extract low-level information (Yosinski et al., 2014), and thus learning them may not require the high-level semantic information captured by manual labels.

Original: Concretely, in this paper we answer the following simple question: “ is self-supervision able to exploit the information contained in a large number of images in order to learn different parts of a neural network? ”
Revised: Concretely, in this paper we answer the following simple question: “ is self-supervision able to exploit the information contained in a large number of images in order to learn different parts of a neural network? ”

Original: We contribute two key ﬁndings.
Revised: We contribute two key ﬁndings.

Original: First, we show that as little as a single image is sufﬁcient, when combined with self-supervision and data augmentation, to learn the ﬁrst few layers of standard deep networks as well as using millions of images and full supervision (Figure 1).
Revised: First, we show that as little as a single image is sufﬁcient, when combined with self-supervision and data augmentation, to learn the ﬁrst few layers of standard deep networks as well as using millions of images and full supervision (Figure 1).

Original: Hence, while selfsupervised learning works well for these layers, this may be due more to the limited complexity of such features than the strength of the supervisory technique.
Revised: Hence, while selfsupervised learning works well for these layers, this may be due more to the limited complexity of such features than the strength of the supervisory technique.

Original: This also conﬁrms the intuition that early layers in a convolutional network amounts to low-level feature extractors, analogous to earlylearned and hand-crafted features for visual recognition (Olshausen & Field, 1997; Lowe, 2004;
Revised: This also conﬁrms the intuition that early layers in a convolutional network amounts to low-level feature extractors, analogous to early

Original: [Figure]
Revised: [Figure]

Original: We show that several self-supervision methods can be used to train the ﬁrst few layers of a deep neural networks using a single training image , such as this Image A , B or even C (above), provided that sufﬁcient data augmentation is used.
Revised: We show that several self-supervision methods can be used to train the ﬁrst few layers of a deep neural networks using a single training image , such as this Image A , B or even C (above), provided that sufﬁcient data augmentation is used.

Revised: learned and hand-crafted features for visual recognition (Olshausen & Field, 1997; Lowe, 2004;

Original: Dalal & Triggs, 2005).
Revised: Dalal & Triggs, 2005).

Original: Finally, it demonstrates the importance of image transformations in learning such low-level features as opposed to image diversity.
Revised: Finally, it demonstrates the importance of image transformations in learning such low-level features as opposed to image diversity.

Original: Our second ﬁnding is about the deeper layers of the network.
Revised: Our second ﬁnding is about the deeper layers of the network.

Original: For these, self-supervision remains inferior to strong supervision even if millions of images are used for training.
Revised: For these, self-supervision remains inferior to strong supervision even if millions of images are used for training.

Original: Our ﬁnding is that this is unlikely to change with the addition of more data.
Revised: Our ﬁnding is that this is unlikely to change with the addition of more data.

Original: In particular, we show that training these layers with self-supervision and a single image already achieves as much as two thirds of the performance that can be achieved by using a million different images.
Revised: In particular, we show that training these layers with self-supervision and a single image already achieves as much as two thirds of the performance that can be achieved by using a million different images.

Original: We show that these conclusions hold true for three different self-supervised methods, BiGAN (Donahue et al., 2017), RotNet (Gidaris et al., 2018) and DeepCluster (Caron et al., 2018), which are representative of the spectrum of techniques that are currently popular.
Revised: We show that these conclusions hold true for three different self-supervised methods, BiGAN (Donahue et al., 2017), RotNet (Gidaris et al., 2018) and DeepCluster (Caron et al., 2018), which are representative of the spectrum of techniques that are currently popular.

Original: We ﬁnd that performance as a function of the amount of data is dependent on the method, but all three methods can indeed leverage a single image to learn the ﬁrst few layers of a deep network almost “perfectly”.
Revised: We ﬁnd that performance as a function of the amount of data is dependent on the method, but all three methods can indeed leverage a single image to learn the ﬁrst few layers of a deep network almost “perfectly”.

Original: Overall, while our results do not improve self-supervision per-se , they help to characterize the limitations of current methods and to better focus on the important open challenges.
Revised: Overall, while our results do not improve self-supervision per-se , they help to characterize the limitations of current methods and to better focus on the important open challenges.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our paper relates to three broad areas of research: (a) self-supervised/unsupervised learning, (b) learning from a single sample, and (c) designing/learning low-level feature extractors.
Revised: Our paper relates to three broad areas of research: (a) self-supervised/unsupervised learning, (b) learning from a single sample, and (c) designing/learning low-level feature extractors.

Original: We discuss closely related work for each.
Revised: We discuss closely related work for each.

Original: Self-supervised learning: A wide variety of proxy tasks, requiring no manual annotations, have been proposed for the self-training of deep convolutional neural networks.
Revised: Self-supervised learning: A wide variety of proxy tasks, requiring no manual annotations, have been proposed for the self-training of deep convolutional neural networks.

Original: These methods use various cues and tasks namely, in-painting (Pathak et al., 2016), patch context and jigsaw puzzles (Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2018; Mundhenk et al., 2017), clustering (Caron et al., 2018), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al., 2017), geometry (Dosovitskiy et al., 2016; Gidaris et al., 2018) and counting (Noroozi et al., 2017).
Revised: These methods use various cues and tasks namely, in-painting (Pathak et al., 2016), patch context and jigsaw puzzles (Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2018; Mundhenk et al., 2017), clustering (Caron et al., 2018), noise-as-targets (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson et al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al., 2017), geometry (Dosovitskiy et al., 2016; Gidaris et al., 2018) and counting (Noroozi et al., 2017).

Original: The idea is that the pre-text task can be constructed automatically and easily on images alone.
Revised: The idea is that the pretext task can be constructed automatically and easily on images alone.

Original: Thus, methods often modify information in the images and require the network to recover them.
Revised: Thus, methods often modify information in the images and require the network to recover them.

Original: Inpainting or colorization techniques fall in this category.
Revised: Inpainting or colorization techniques fall in this category.

Original: However these methods have the downside that the features are learned on modiﬁed images which potentially harms the generalization to unmodiﬁed ones.
Revised: However these methods have the downside that the features are learned on modiﬁed images which potentially harms the generalization to unmodiﬁed ones.

Original: For example, colorization uses a gray scale image as input, thus the network cannot learn to extract color information, which can be important for other tasks.
Revised: For example, colorization uses a gray scale image as input, thus the network cannot learn to extract color information, which can be important for other tasks.

Original: Slightly less related are methods that use additional information to learn features.
Revised: Slightly less related are methods that use additional information to learn features.

Original: Here, often temporal information is used in the form of videos.
Revised: Here, often temporal information is used in the form of videos.

Original: Typical pre-text tasks are based on temporalcontext (Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background segmentation via video segmentation (Pathak et al., 2017), optical-ﬂow (Gan et al., 2018;
Revised: Typical pretext tasks are based on temporalcontext (Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background segmentation via video segmentation (Pathak et al., 2017), optical-ﬂow (Gan et al., 2018;

Original: Mahendran et al., 2018), future-frame synthesis (Srivastava et al., 2015), audio prediction from video (de Sa, 1994; Owens et al., 2016), audio-video alignment (Arandjelovi´c & Zisserman, 2017), ego-motion estimation (Jayaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence (Jayaraman & Grauman, 2016), transformation between frames (Agrawal et al., 2015) and patch tracking in videos (Wang & Gupta, 2015).
Revised: Mahendran et al., 2018), future-frame synthesis (Srivastava et al., 2015), audio prediction from video (de Sa, 1994; Owens et al., 2016), audio-video alignment (Arandjelovi´c & Zisserman, 2017), ego-motion estimation (Jayaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence (Jayaraman & Grauman, 2016), transformation between frames (Agrawal et al., 2015) and patch tracking in videos (Wang & Gupta, 2015).

Original: Since we are interested in learning features from as little data as one image, we cannot make use of methods that rely on video input.
Revised: Since we are interested in learning features from as little data as one image, we cannot make use of methods that rely on video input.

Original: Our contribution inspects three unsupervised feature learning methods that use very different means of extracting information from the data: BiGAN (Donahue et al., 2017) utilizes a generative adversarial task, RotNet (Gidaris et al., 2018) exploits the photographic bias in the dataset and DeepCluster (Caron et al., 2018) learns stable feature representations under a number of image transformations by proxy labels obtained from clustering.
Revised: Our contribution inspects three unsupervised feature learning methods that use very different means of extracting information from the data: BiGAN (Donahue et al., 2017) utilizes a generative adversarial task, RotNet (Gidaris et al., 2018) exploits the photographic bias in the dataset and DeepCluster (Caron et al., 2018) learns stable feature representations under a number of image transformations by proxy labels obtained from clustering.

Original: These are described in more detail in the Methods section.
Revised: These are described in more detail in the Methods section.

Original: Learning from a single sample: In some applications of computer vision, the bold idea of learning from a single sample comes out of necessity.
Revised: Learning from a single sample: In some applications of computer vision, the bold idea of learning from a single sample comes out of necessity.

Original: For general object tracking, methods such as max margin correlation ﬁlters (Rodriguez et al., 2013) learn robust tracking templates from a single sample of the patch.
Revised: For general object tracking, methods such as max margin correlation ﬁlters (Rodriguez et al., 2013) learn robust tracking templates from a single sample of the patch.

Original: A single image can also be used to learn and interpolate multi-scale textures with a GAN framework (Rott Shaham et al., 2019).
Revised: A single image can also be used to learn and interpolate multi-scale textures with a GAN framework (Rott Shaham et al., 2019).

Original: Single sample learning was pursued by the semi-parametric exemplar SVM model (Malisiewicz et al., 2011).
Revised: Single sample learning was pursued by the semi-parametric exemplar SVM model (Malisiewicz et al., 2011).

Original: They learn one SVM per positive sample separating it from all negative patches mined from the background.
Revised: They learn one SVM per positive sample separating it from all negative patches mined from the background.

Original: While only one sample is used for the positive set, the negative set consists of thousands of images and is a necessary component of their method.
Revised: While only one sample is used for the positive set, the negative set consists of thousands of images and is a necessary component of their method.

Original: The negative space was approximated by a multi-dimensional Gaussian by the Exemplar LDA (Hariharan et al., 2012).
Revised: The negative space was approximated by a multi-dimensional Gaussian by the Exemplar LDA (Hariharan et al., 2012).

Original: These SVMs, one per positive sample, are pooled together using a max aggregation.
Revised: These SVMs, one per positive sample, are pooled together using a max aggregation.

Original: We differ from both of these approaches in that we do not use a large collection of negative images to train our model.
Revised: We differ from both of these approaches in that we do not use a large collection of negative images to train our model.

Original: Instead we restrict ourselves to a single or a few images with a systematic augmentation strategy.
Revised: Instead we restrict ourselves to a single or a few images with a systematic augmentation strategy.

Original: Classical learned and hand-crafted low-level feature extractors: Learning and hand-crafting features pre-dates modern deep learning approaches and self-supervision techniques.
Revised: Classical learned and hand-crafted low-level feature extractors: Learning and hand-crafting features pre-dates modern deep learning approaches and self-supervision techniques.

Original: For example the classical work of (Olshausen & Field, 1997) shows that edge-like ﬁlters can be learned via sparse coding of just 10 natural scene images.
Revised: For example the classical work of (Olshausen & Field, 1997) shows that edge-like ﬁlters can be learned via sparse coding of just 10 natural scene images.

Original: SIFT (Lowe, 2004) and HOG (Dalal & Triggs, 2005) have been used extensively before the advent of convolutional neural networks and, in many ways, they resemble the ﬁrst layers of these networks.
Revised: SIFT (Lowe, 2004) and HOG (Dalal & Triggs, 2005) have been used extensively before the advent of convolutional neural networks and, in many ways, they resemble the ﬁrst layers of these networks.

Original: The scatter transform of Bruna & Mallat (2013);
Revised: The scatter transform of Bruna & Mallat (2013);

Original: Oyallon et al.
Revised: Oyallon et al.

Original: (2017) is an handcrafted design that aims at replacing at least the ﬁrst few layers of a deep network.
Revised: (2017) is an handcrafted design that aims at replacing at least the ﬁrst few layers of a deep network.

Original: While these results show that effective low-level features can be handcrafted, this is insufﬁcient to clarify the power and limitation of self-supervision in deep networks.
Revised: While these results show that effective low-level features can be handcrafted, this is insufﬁcient to clarify the power and limitation of self-supervision in deep networks.

Original: For instance, it is not obvious whether deep networks can learn better low level features than these, how many images may be required to learn them, and how effective self-supervision may be in doing so.
Revised: For instance, it is not obvious whether deep networks can learn better low level features than these, how many images may be required to learn them, and how effective self-supervision may be in doing so.

Original: For instance, as we also show in the experiments, replacing low-level layers in a convolutional networks with handcrafted features such as Oyallon et al.
Revised: For instance, as we also show in the experiments, replacing low-level layers in a convolutional networks with handcrafted features such as Oyallon et al.

Original: (2017) may still decrease the overall performance of the model.
Revised: (2017) may still decrease the overall performance of the model.

Original: Furthermore, this says little about deeper layers, which we also investigate.
Revised: Furthermore, this says little about deeper layers, which we also investigate.

Original: In this work we show that current deep learning methods learn slightly better low-level representations than hand crafted features such as the scattering transform.
Revised: In this work we show that current deep learning methods learn slightly better low-level representations than hand crafted features such as the scattering transform.

Original: Additionally, these representations can be learned from one single image with augmentations and without supervision.
Revised: Additionally, these representations can be learned from one single image with augmentations and without supervision.

Original: The results show how current self-supervised learning approaches that use one million images yield only relatively small gains when compared to what can be achieved from one image and augmentations, and motivates a renewed focus on augmentations and incorporating prior knowledge into feature extractors.
Revised: The results show how current self-supervised learning approaches that use one million images yield only relatively small gains when compared to what can be achieved from one image and augmentations, and motivates a renewed focus on augmentations and incorporating prior knowledge into feature extractors.

Original:  M ETHODS
Revised:  M ETHODS

Original: We discuss ﬁrst our data and data augmentation strategy (section 3.1) and then we summarize the three different methods for unsupervised feature learning used in the experiments (section 3.2).
Revised: We discuss ﬁrst our data and data augmentation strategy (section 3.1) and then we summarize the three different methods for unsupervised feature learning used in the experiments (section 3.2).

Original: 3. D ATA
Revised: 3. D ATA

Original: Our goal is to understand the performance of representation learning methods as a function of the image data used to train them.
Revised: Our goal is to understand the performance of representation learning methods as a function of the image data used to train them.

Original: To make comparisons as fair as possible, we develop a protocol where only the nature of the training data is changed, but all other parameters remain ﬁxed.
Revised: To make comparisons as fair as possible, we develop a protocol where only the nature of the training data is changed, but all other parameters remain ﬁxed.

Original: In order to do so, given a baseline method trained on d source images, we replace those with another set of d images.
Revised: In order to do so, given a baseline method trained on d source images, we replace those with another set of d images.

Original: Of these, now only N (cid:28) d are source images (i.e. i.i.d.
Revised: Of these, now only N (cid:28) d are source images (i.e. i.i.d.

Original: samples), while the remaining d − N are augmentations of the source ones.
Revised: samples), while the remaining d − N are augmentations of the source ones.

Original: Thus, the amount of information in the training data is controlled by N and we can generate a continuum of datasets that vary from one extreme, utilizing a single source image N =1 , to the other extreme, using all N = d original training set images.
Revised: Thus, the amount of information in the training data is controlled by N and we can generate a continuum of datasets that vary from one extreme, utilizing a single source image N =1 , to the other extreme, using all N = d original training set images.

Original: For example, if the baseline method is trained on ImageNet, then d =1 , 281 , 167 .
Revised: For example, if the baseline method is trained on ImageNet, then d =1 , 281 , 167 .

Original: When N =1 , it means that we train the method using a single source image and generate the remaining 1 , 281 , 166 images via augmentation.
Revised: When N =1 , it means that we train the method using a single source image and generate the remaining 1 , 281 , 166 images via augmentation.

Original: Other baselines use CIFAR-10/100 images, so in those cases d =50 , 000 instead.
Revised: Other baselines use CIFAR-10/100 images, so in those cases d =50 , 000 instead.

Original: The data augmentation protocol, is an extreme version of augmentations already employed by most deep learning protocols.
Revised: The data augmentation protocol, is an extreme version of augmentations already employed by most deep learning protocols.

Original: Each method we test, in fact, already performs some data augmentation internally.
Revised: Each method we test, in fact, already performs some data augmentation internally.

Original: Thus, when the method is applied on our augmented data, this can be equivalently thought of as incrementing these “native” augmentations by concatenating them with our own.
Revised: Thus, when the method is applied on our augmented data, this can be equivalently thought of as incrementing these “native” augmentations by concatenating them with our own.

Original: Choice of augmentations.
Revised: Choice of augmentations.

Original: Next, we describe how the N source images are expanded to additional d − N images so that the models can be trained on exactly d images, independent from the choice of
Revised: Next, we describe how the N source images are expanded to additional d − N images so that the models can be trained on exactly d images, independent from the choice of

Original: N .
Revised: N .

Original: The idea is to use an aggressive form of data augmentation involving cropping, scaling, rotation, contrast changes, and adding noise.
Revised: The idea is to use an aggressive form of data augmentation involving cropping, scaling, rotation, contrast changes, and adding noise.

Original: These transformations are representative of invariances that one may wish to incorporate in the features.
Revised: These transformations are representative of invariances that one may wish to incorporate in the features.

Original: Augmentation can be seen as imposing a prior on how we expect the manifold of natural images to look like.
Revised: Augmentation can be seen as imposing a prior on how we expect the manifold of natural images to look like.

Original: When training with very few images, these priors become more important since the model cannot extract them directly from data.
Revised: When training with very few images, these priors become more important since the model cannot extract them directly from data.

Original: Given a source image of size size H × W , we ﬁrst extract a certain number of random patches of size ( w, h ) , where w ≤ W and h ≤ H satisfy the additional constraints β ≤ whWH and γ ≤ hw ≤ γ − 1 .
Revised: Given a source image of size size H × W , we ﬁrst extract a certain number of random patches of size ( w, h ) , where w ≤ W and h ≤ H satisfy the additional constraints β ≤ whWH and γ ≤ hw ≤ γ − 1 .

Original: Thus, the smallest size of the crops is limited to be at least βWH and at most the whole image.
Revised: Thus, the smallest size of the crops is limited to be at least βWH and at most the whole image.

Original: Additionally, changes to the aspect ratio are limited by γ .
Revised: Additionally, changes to the aspect ratio are limited by γ .

Original: In practice we use β = 10 − 3 and γ = 34 .
Revised: In practice we use β = 10 − 3 and γ = 34 .

Original: Second, good features should not change much by small image rotations, so images are rotated (before cropping to avoid border artifacts) by α ∈ ( − 35 , 35) degrees.
Revised: Second, good features should not change much by small image rotations, so images are rotated (before cropping to avoid border artifacts) by α ∈ ( − 35 , 35) degrees.

Original: Due to symmetry in image statistics, images are also ﬂipped left-to-right with 50% probability.
Revised: Due to symmetry in image statistics, images are also ﬂipped left-to-right with 50% probability.

Original: Illumination changes are common in natural images, we thus expect image features to be robust to color and contrast changes.
Revised: Illumination changes are common in natural images, we thus expect image features to be robust to color and contrast changes.

Original: Thus, we employ a set of linear transformations in RGB space to model this variability in real data.
Revised: Thus, we employ a set of linear transformations in RGB space to model this variability in real data.

Original: Additionally, the color/intensity of single pixels should not affect the feature representation, as this does not change the contents of the image.
Revised: Additionally, the color/intensity of single pixels should not affect the feature representation, as this does not change the contents of the image.

Original: To this end, color jitter with additive brightness, contrast and saturation are sampled from three uniform distributions in (0 . 6 , 1 . 4) and hue noise from ( − 0 .
Revised: To this end, color jitter with additive brightness, contrast and saturation are sampled from three uniform distributions in (0 . 6 , 1 . 4) and hue noise from ( − 0 .

Original: 1 , 0 . 1) is applied to the image patches.
Revised: 1 , 0 . 1) is applied to the image patches.

Original: Finally, the cropped and transformed patches are scaled to the color range ( − 1 , 1) and then rescaled to full S × S resolution to be supplied to each representation learning method, using bilinear interpolation.
Revised: Finally, the cropped and transformed patches are scaled to the color range ( − 1 , 1) and then rescaled to full S × S resolution to be supplied to each representation learning method, using bilinear interpolation.

Original: This formulation ensures that the patches are created in the target resolution S , independent from the size and aspect ratio W, H of the source image.
Revised: This formulation ensures that the patches are created in the target resolution S , independent from the size and aspect ratio W, H of the source image.

Original: Real samples.
Revised: Real samples.

Original: The images used for the N =1 and N =10 experiments are shown in Figure 1 and the appendix respectively (this is all the training data used in such experiments).
Revised: The images used for the N =1 and N =10 experiments are shown in Figure 1 and the appendix respectively (this is all the training data used in such experiments).

Original: For the special case of using a single training image, i.e. N =1 , we have chosen one photographic ( 2560 × 1920 ) and one drawn image ( 600 × 225 ), which we call Image A and Image B , respectively.
Revised: For the special case of using a single training image, i.e. N =1 , we have chosen one photographic ( 2560 × 1920 ) and one drawn image ( 600 × 225 ), which we call Image A and Image B , respectively.

Original: The two images were manually selected as they contain rich texture and are diverse, but their choice was not optimized for performance.
Revised: The two images were manually selected as they contain rich texture and are diverse, but their choice was not optimized for performance.

Original: We test only two images due to the cost of running a full set of experiments (each image is expanded up to 1.2M times for training some of the models, as explained above).
Revised: We test only two images due to the cost of running a full set of experiments (each image is expanded up to 1.2M times for training some of the models, as explained above).

Original: However, this is sufﬁcient to prove our main points.
Revised: However, this is sufﬁcient to prove our main points.

Original: We also test another ( 1165 × 585 ) Image C to ablate the ”crowdedness” of an image, as this latter contains large areas covering no objects.
Revised: We also test another ( 1165 × 585 ) Image C to ablate the “crowdedness” of an image, as this latter contains large areas covering no objects.

Original: While resolution matters to some extent as a bigger image contains more pixels, the information within is still far more correlated, and thus more redundant than sampling several smaller images.
Revised: While resolution matters to some extent as a bigger image contains more pixels, the information within is still far more correlated, and thus more redundant than sampling several smaller images.

Original: In particular, the resolution difference in Image A and B appears to be negligible in our experiments.
Revised: In particular, the resolution difference in Image A and B appears to be negligible in our experiments.

Original: For CIFAR-10, where S =32 we only use Image B due to the resolution difference.
Revised: For CIFAR-10, where S =32 we only use Image B due to the resolution difference.

Original: In direct comparison, Image B is the size of about 132 CIFAR images which is still much less than d = 50 , 000 .
Revised: In direct comparison, Image B is the size of about 132 CIFAR images which is still much less than d = 50 , 000 .

Original: For N > 1 , we select the source images randomly from each method’s training set.
Revised: For N > 1 , we select the source images randomly from each method’s training set.

Original: 3. R EPRESENTATION L EARNING M ETHODS
Revised: 3. R EPRESENTATION L EARNING M ETHODS

Original: Generative models.
Revised: Generative models.

Original: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) learn to generate images using an adversarial objective: a generator network maps noise samples to image samples, approximating a target image distribution and a discriminator network is tasked with distinguishing generated and real samples.
Revised: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) learn to generate images using an adversarial objective: a generator network maps noise samples to image samples, approximating a target image distribution and a discriminator network is tasked with distinguishing generated and real samples.

Original: Generator and discriminator are pitched one against the other and learned together; when an equilibrium is reached, the generator produces images indistinguishable (at least from the viewpoint of the discriminator) from real ones.
Revised: Generator and discriminator are pitched one against the other and learned together; when an equilibrium is reached, the generator produces images indistinguishable (at least from the viewpoint of the discriminator) from real ones.

Original: Bidirectional Generative Adversarial Networks (BiGAN) (Donahue et al., 2017; Dumoulin et al., 2016) are an extension of GANs designed to learn a useful image representation as an approximate inverse of the generator through joint inference on an encoding and the image.
Revised: Bidirectional Generative Adversarial Networks (BiGAN) (Donahue et al., 2017; Dumoulin et al., 2016) are an extension of GANs designed to learn a useful image representation as an approximate inverse of the generator through joint inference on an encoding and the image.

Original: This method’s native augmentation uses random crops and random horizontal ﬂips to learn features from S =sized images.
Revised: This method’s native augmentation uses random crops and random horizontal ﬂips to learn features from S =sized images.

Original: As opposed to the other two methods discussed below it employs leaky ReLU nonlinearities as is typical in GAN discriminators.
Revised: As opposed to the other two methods discussed below it employs leaky ReLU nonlinearities as is typical in GAN discriminators.

Original: Rotation.
Revised: Rotation.

Original: Most image datasets contain pictures that are ‘upright’ as this is how humans prefer to take and look at them.
Revised: Most image datasets contain pictures that are ‘upright’ as this is how humans prefer to take and look at them.

Original: This photographer bias can be understood as a form of implicit data labelling.
Revised: This photographer bias can be understood as a form of implicit data labelling.

Original: RotNet (Gidaris et al., 2018) exploits this by tasking a network with predicting the upright direction of a picture after applying to it a random rotation multiple of 90 degrees (in practice this is formulated as a 4 -way classiﬁcation problem).
Revised: RotNet (Gidaris et al., 2018) exploits this by tasking a network with predicting the upright direction of a picture after applying to it a random rotation multiple of 90 degrees (in practice this is formulated as a 4 -way classiﬁcation problem).

Original: The authors reason that the concept of ‘upright’ requires learning high level concepts in the image and hence this method is not vulnerable to exploiting low-level visual information, encouraging the network to learn more abstract features.
Revised: The authors reason that the concept of ‘upright’ requires learning high level concepts in the image and hence this method is not vulnerable to exploiting low-level visual information, encouraging the network to learn more abstract features.

Original: In our experiments, we test this hypothesis by learning from impoverished datasets that may lack the photographer bias.
Revised: In our experiments, we test this hypothesis by learning from impoverished datasets that may lack the photographer bias.

Original: The native augmentations that RotNet uses on the S =256 inputs only comprisehorizontal ﬂips and non-scaled random crops to 224 × 224 .
Revised: The native augmentations that RotNet uses on the S =256 inputs only comprise

Original: Clustering.
Revised: Clustering.

Original: DeepCluster (Caron et al., 2018) is a recent state-of-the-art unsupervised representation learning method.
Revised: DeepCluster (Caron et al., 2018) is a recent state-of-the-art unsupervised representation learning method.

Original: This approach alternates k -means clustering to produce pseudo-labels for the data and feature learning to ﬁt the representation to these labels.
Revised: This approach alternates k -means clustering to produce pseudo-labels for the data and feature learning to ﬁt the representation to these labels.

Original: The authors attribute the success of the method to the prior knowledge ingrained in the structure of the convolutional neural network (Ulyanov et al., 2018).
Revised: The authors attribute the success of the method to the prior knowledge ingrained in the structure of the convolutional neural network (Ulyanov et al., 2018).

Original: The method alternatives between a clustering step, in which k -means is applied on the PCA-reduced features with k = 10 3 , and a learning step, in which the network is trained to predict the cluster
Revised: The method alternatives between a clustering step, in which k -means is applied on the PCA-reduced features with k = 10 4 , and a learning step, in which the network is trained to predict the cluster

Original: ID for each image under a set of augmentations (random resized crops with β = 0 .
Revised: ID for each image under a set of augmentations (random resized crops with β = 0 .

Original: 08 , γ = 34 and horizontal ﬂips) that constitute its native augmentations used on top of the S =256 input images.
Revised: 08 , γ = 34 and horizontal ﬂips) that constitute its native augmentations used on top of the S =256 input images.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the representation learning methods on ImageNet and CIFAR-10/100 using linear probes (Section 4.1).
Revised: We evaluate the representation learning methods on ImageNet and CIFAR-10/100 using linear probes (Section 4.1).

Original: After ablating various choices of transformations in our augmentation protocol (Section 4.2), we move to the core question of the paper: whether a large dataset is beneﬁcial to unsupervised learning, especially for learning early convolutional features (Section 4.3).
Revised: After ablating various choices of transformations in our augmentation protocol (Section 4.2), we move to the core question of the paper: whether a large dataset is beneﬁcial to unsupervised learning, especially for learning early convolutional features (Section 4.3).

Original: 4. L INEAR PROBES AND BASELINE ARCHITECTURE
Revised: 4. L INEAR PROBES AND BASELINE ARCHITECTURE

Original: In order to quantify if a neural network has learned useful feature representations, we follow the standard approach of using linear probes (Zhang et al., 2017).
Revised: In order to quantify if a neural network has learned useful feature representations, we follow the standard approach of using linear probes (Zhang et al., 2017).

Original: This amounts to solving a difﬁcult task such as ImageNet classiﬁcation by training a linear classiﬁer on top of pre-trained feature representations, which are kept ﬁxed.
Revised: This amounts to solving a difﬁcult task such as ImageNet classiﬁcation by training a linear classiﬁer on top of pre-trained feature representations, which are kept ﬁxed.

Original: Linear classiﬁers heavily rely on the quality of the representation since their discriminative power is low.
Revised: Linear classiﬁers heavily rely on the quality of the representation since their discriminative power is low.

Original: We apply linear probes to all intermediate convolutional layers of networks and train on the ImageNet LSVRC-12 (Deng et al., 2009) and CIFAR-10/100 (Krizhevsky, 2009) datasets, which are the standard benchmarks for evaluation in self-supervised learning.
Revised: We apply linear probes to all intermediate convolutional layers of networks and train on the ImageNet LSVRC-12 (Deng et al., 2009) and CIFAR-10/100 (Krizhevsky, 2009) datasets, which are the standard benchmarks for evaluation in self-supervised learning.

Original: Our base encoder architecture is AlexNet (Krizhevsky et al., 2012), since this is a good representative model and is most often used in other unsupervised learning work for the purpose of benchmarking.
Revised: Our base encoder architecture is AlexNet (Krizhevsky et al., 2012) with BatchNorm, since this is a good representative model and is most often used in other unsupervised learning work for the purpose of benchmarking.

Original: This modelhas ﬁve
Revised: This model

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Donahue et al
Revised: et al

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Caron et al., 2018) [Table] [Table] [Table]
Revised: MonoGAN

Original: [Table]
Revised: [Table]

Original:  convolutional blocks (each comprising a linear convolution later followed by ReLU and optionally max pooling).
Revised: has ﬁve convolutional blocks (each comprising a linear convolution later followed by ReLU and optionally max pooling).

Original: We insert the probes right after the ReLU layer in each block, and denote these entry points conv1 to conv5 .
Revised: We insert the probes right after the ReLU layer in each block, and denote these entry points conv1 to conv5 .

Original: Applying the linear probes at each convolutional layer allows studying the quality of the representation learned at different depths of the network.
Revised: Applying the linear probes at each convolutional layer allows studying the quality of the representation learned at different depths of the network.

Original: Details.
Revised: Details.

Original: While linear probes are conceptually straightforward [Table] there are several technical details that affect the ﬁnal accuracy by a few percentage points.
Revised: While linear probes are conceptually straightforward, there are several technical details that affect the ﬁnal accuracy by a few percentage points.

Original: Unfortunately, prior work has used several slightly different setups, so that comparing results of different publications must be done with caution.
Revised: Unfortunately, prior work has used several slightly different setups, so that comparing results of different publications must be done with caution.

Original: To make matters more difﬁcult, not all papers released evaluation source code.
Revised: To make matters more difﬁcult, not all papers released evaluation source code.

Original: Upon acceptance, we will publish code for all of our experiments, including the evaluation.
Revised: We prove this standardized testing code here 2 .

Original: In our implementation, we follow the original proposal (Zhang et al., 2017) in pooling each representation to a vector with 9600 , 9216 , 9600 , 9600 , 9216 dimensions for conv1-5 using adaptive max-pooling, and absorb the batch normalization weights into the preceding convolutions.
Revised: In our implementation, we follow the original proposal (Zhang et al., 2017) in pooling each representation to a vector with 9600 , 9216 , 9600 , 9600 , 9216 dimensions for conv1-5 using adaptive max-pooling, and absorb the batch normalization weights into the preceding convolutions.

Original: For evaluation on ImageNet we follow RotNet to train linear probes: images are resized such that the shorter edge has a length of 256 pixels, random crops of 224 × 224 are computed and ﬂipped horizontally with 50% probability.
Revised: For evaluation on ImageNet we follow RotNet to train linear probes: images are resized such that the shorter edge has a length of 256 pixels, random crops of 224 × 224 are computed and ﬂipped horizontally with 50% probability.

Original: Learning lasts for 36 epochs and the learning rate schedule starts from 0 .and is divided by ﬁve at epochs 5 , 15 and 25 .
Revised: Learning lasts for 36 epochs and the learning rate schedule starts from 0 .and is divided by ﬁve at epochs 5 , 15 and 25 .

Original: The top-1 accuracy of the linear classiﬁer is then measured on the ImageNet validation subset.
Revised: The top-1 accuracy of the linear classiﬁer is then measured on the ImageNet validation subset.

Original: This uses DeepCluster’s protocol, extracting 10 crops for each validation image (four at the corners and one at the center along with their horizontal ﬂips) and averaging the prediction scores before the accuracy is computed.
Revised: This uses DeepCluster’s protocol, extracting 10 crops for each validation image (four at the corners and one at the center along with their horizontal ﬂips) and averaging the prediction scores before the accuracy is computed.

Original: For CIFAR-10/100 data, we follow the same learning rate schedule and for both training and evaluation we do not reduce the dimensionality of the representations and keep the images’ original size of 32 × 32 .
Revised: For CIFAR-10/100 data, we follow the same learning rate schedule and for both training and evaluation we do not reduce the dimensionality of the representations and keep the images’ original size of 32 × 32 .

Original: 4. E FFECT OF A UGMENTATIONS
Revised: 4. E FFECT OF A UGMENTATIONS

Original: In order to better understand which image transformations are important to learn a good feature representations, we analyze the impact of augmentation settings.
Revised: In order to better understand which image transformations are important to learn a good feature representations, we analyze the impact of augmentation settings.

Original: For speed, these experiments are conducted using the CIFAR-10 images ( d = 50 , 000 in the training set) and with the smaller source
Revised: For speed, these experiments are conducted using the CIFAR-10 images ( d = 50 , 000 in the training set) and with the smaller source

Original: Image B and a GAN using the Wasserstein GAN formulation with gradient penalty (Gulrajani et al., 2017).
Revised: Image B and a GAN using the Wasserstein GAN formulation with gradient penalty (Gulrajani et al., 2017).

Original: The encoder is a smaller AlexNet-like CNN consisting of four convolutional layers (kernel sizes: 7 , 5 , 3 , 3 ; strides: 3 , 2 , 2 , 1 ) followed by a single fully connected layer as the discriminator.
Revised: The encoder is a smaller AlexNet-like CNN consisting of four convolutional layers (kernel sizes: 7 , 5 , 3 , 3 ; strides: 3 , 2 , 2 , 1 ) followed by a single fully connected layer as the discriminator.

Original: Given that the GAN is trained on a single image (w/ augmentations), we call this setting MonoGAN .
Revised: Given that the GAN is trained on a single image (w/ augmentations), we call this setting MonoGAN .

Original: Table 1 reports all 2 3 combinations of the three main augmentations (scale, rotation, and jitter) and a randomly initialized network baseline (see Table 1 (b)) using the linear probes protocol discussed above.
Revised: Table 1 reports all 2 3 combinations of the three main augmentations (scale, rotation, and jitter) and a randomly initialized network baseline (see Table 1 (b)) using the linear probes protocol discussed above.

Original: Without data augmentation the model only achieves marginally better performance than the random network (which also achieves a non-negligible level of performance (Ulyanov et al., 2017;
Revised: Without data augmentation the model only achieves marginally better performance than the random network (which also achieves a non-negligible level of performance (Ulyanov et al., 2017;

Original: Caron et al., 2018)).
Revised: Caron et al., 2018)).

Original: This is understandable since the dataset literally consists of a single training image cloned d times.
Revised: This is understandable since the dataset literally consists of a single training image cloned d times.

Original: Color jitter and rotation slightly improve the performance of all probes by 12% points, but random rescaling adds at least ten points at every depth (see Table 1 (f,h,i)) and is the most important single augmentation.
Revised: Color jitter and rotation slightly improve the performance of all probes by 12% points, but random rescaling adds at least ten points at every depth (see Table 1 (f,h,i)) and is the most important single augmentation.

Original: A similar conclusion can be drawn when two augmentations are combined, although there are diminishing returns as more augmentations are combined.
Revised: A similar conclusion can be drawn when two augmentations are combined, although there are diminishing returns as more augmentations are combined.

Original: Overall, we ﬁnd all three types of augmentations are of importance when training in the ultra-low data setting.
Revised: Overall, we ﬁnd all three types of augmentations are of importance when training in the ultra-low data setting.

Original: 4. B ENCHMARK EVALUATION
Revised: 4. B ENCHMARK EVALUATION

Original: We analyze how performance varies as a function N , the number of actual samples that are used to generated the augmented datasets, and compare it to the gold-standard setup (in terms of choice of training data) deﬁned in the papers that introduced each method.
Revised: We analyze how performance varies as a function N , the number of actual samples that are used to generated the augmented datasets, and compare it to the gold-standard setup (in terms of choice of training data) deﬁned in the papers that introduced each method.

Original: The evaluation is again based on linear probes (Section 4.1).
Revised: The evaluation is again based on linear probes (Section 4.1).

Original: Mono is enough.
Revised: Mono is enough.

Original: From Table 2 we make the following observations.
Revised: From Table 2 we make the following observations.

Original: Training with just a single source image (f,g,k,l,p,q) is much better than random initialization (c) for all layers.
Revised: Training with just a single source image (f,g,k,l,p,q) is much better than random initialization (c) for all layers.

Original: Notably, these models also outperform Gabor-like ﬁlters from Scattering networks (Bruna & Mallat, 2013), which are hand crafted image features, replacing the ﬁrst two convolutional layers as in (Oyallon et al., 2017).
Revised: Notably, these models also outperform Gabor-like ﬁlters from Scattering networks (Bruna & Mallat, 2013), which are hand crafted image features, replacing the ﬁrst two convolutional layers as in (Oyallon et al., 2017).

Original: Using the same protocol as in the paper, this only achieves an accuracy of 18 .
Revised: Using the same protocol as in the paper, this only achieves an accuracy of 18 .

Original: 9% compared to (p)’s conv2 > 30% .
Revised: 9% compared to (p)’s conv2 > 30% .

Original: More importantly, when comparing within pre-text task, even with one image we are able to improve the quality of conv1 – conv3 features compared to full (unsupervised)
Revised: More importantly, when comparing within pretext task, even with one image we are able to improve the quality of conv1 – conv3 features compared to full (unsupervised)

Original: ImageNet training for GAN based self-supervision (e-i).
Revised: ImageNet training for GAN based self-supervision (e-i).

Original: For the other methods (j-n, o-s) we reach and also surpass the performance for the ﬁrst layer and are within 1 . 5% points for the second.
Revised: For the other methods (j-n, o-s) we reach and also surpass the performance for the ﬁrst layer and are within 1 . 5% points for the second.

Original: Given that the best unsupervised performance for conv2 is 32 . 5 , our method using a single source Image A (Table 2, p) is remarkably close with 31 . 5 .
Revised: Given that the best unsupervised performance for conv2 is 32 . 5 , our method using a single source Image A (Table 2, p) is remarkably close with 31 . 5 .

Original: Image contents.
Revised: Image contents.

Original: While we surpass the GAN based approach of (Donahue et al., 2017) for both single source images, we ﬁnd more nuanced results for the other two methods: For RotNet, as expected, the photographic bias cannot be extracted from a single image.
Revised: While we surpass the GAN based approach of (Donahue et al., 2017) for both single source images, we ﬁnd more nuanced results for the other two methods: For RotNet, as expected, the photographic bias cannot be extracted from a single image.

Original: Thus its performance is low with little training data and increases together with the number of images (Table 2, j-n).
Revised: Thus its performance is low with little training data and increases together with the number of images (Table 2, j-n).

Original: When comparing Image A and B trained networks for RotNet, we ﬁnd that the photograph yields better performance than the hand drawn animal image.
Revised: When comparing Image A and B trained networks for RotNet, we ﬁnd that the photograph yields better performance than the hand drawn animal image.

Original: This indicates that the method can extract rotation
Revised: This indicates that the method can extract rotation

Original: [Figure]
Revised: [Figure]

Revised: BiGAN

Original: RotNet
Revised: RotNet

Original: DeepCluster
Revised: DeepCluster

Original: BiGAN
Revised: BiGAN

Original: RotNet
Revised: RotNet

Original: DeepCluster [Figure] [Figure] [Figure]
Revised: DeepCluster 20.

Original: information from low level image features such as patches which is at ﬁrst counter intuitive.
Revised: information from low level image features such as patches which is at ﬁrst counter intuitive.

Original: Considering that the hand-drawn image does not work well, we can assume that lighting and shadows even in small patches can indeed give important cues on the up direction which can be learned even from a single (real) image.
Revised: Considering that the hand-drawn image does not work well, we can assume that lighting and shadows even in small patches can indeed give important cues on the up direction which can be learned even from a single (real) image.

Original: DeepCluster shows poor performance in conv1 which we can improve upon in the single image setting (Table 2, o-r).
Revised: DeepCluster shows poor performance in conv1 which we can improve upon in the single image setting (Table 2, o-r).

Original: Naturally, the image content matters: a trivial image without any image gradient (e.g. picture of a white wall) would not provide enough signal for any method.
Revised: Naturally, the image content matters: a trivial image without any image gradient (e.g. picture of a white wall) would not provide enough signal for any method.

Original: To better understand this issue, we also train DeepCluster on the much less cluttered Image C to analyze how much the image inﬂuences our claims.
Revised: To better understand this issue, we also train DeepCluster on the much less cluttered Image C to analyze how much the image inﬂuences our claims.

Original: We ﬁnd that even though this image contains large parts of sky and sea, the performance is only slightly lower than that of Image A .
Revised: We ﬁnd that even though this image contains large parts of sky and sea, the performance is only slightly lower than that of Image A .

Original: This ﬁnding indicates that the augmentations can even compensate for large untextured areas and the exact choice of image is not critical.
Revised: This ﬁnding indicates that the augmentations can even compensate for large untextured areas and the exact choice of image is not critical.

Original: More than one image.
Revised: More than one image.

Original: While BiGAN fails to converge for N ∈{ 10 , 1000 } , most likely due to issues in learning from a distribution which is neither whole images nor only patches, we ﬁnd that both
Revised: While BiGAN fails to converge for N ∈{ 10 , 1000 } , most likely due to issues in learning from a distribution which is neither whole images nor only patches, we ﬁnd that both

Original: RotNet and DeepCluster improve their performance in deeper layers when increasing the number of training images.
Revised: RotNet and DeepCluster improve their performance in deeper layers when increasing the number of training images.

Original: However, for conv1 and conv2 , a single image is enough.
Revised: However, for conv1 and conv2 , a single image is enough.

Original: In deeper layers,
Revised: In deeper layers,

Original: DeepCluster seems to require large amounts of source images to yield the reported results as the deka- and kilo- variants start improving over the single image case (Table 2, o-t).
Revised: DeepCluster seems to require large amounts of source images to yield the reported results as the deka- and kilo- variants start improving over the single image case (Table 2, o-t).

Original: This need for data also explains the gap between the two input images which have different resolutions.
Revised: This need for data also explains the gap between the two input images which have different resolutions.

Original: Summarizing
Revised: Summarizing

Original: Table 2, we can conclude that learning conv1 , conv2 and for the most part conv3 ( 33 . 4 vs. 39 . 4 ) on over 1M images does not yield a signiﬁcant performance increase over using one single training image — a highly unexpected result.
Revised: Table 2, we can conclude that learning conv1 , conv2 and for the most part conv3 ( 33 . 4 vs. 39 . 4 ) on over 1M images does not yield a signiﬁcant performance increase over using one single training image — a highly unexpected result.

Original: Generalization.
Revised: Generalization.

Original: In Table 3, we show the results of training linear classiﬁers for the CIFARdataset and compare against various baselines.
Revised: In Table 3, we show the results of training linear classiﬁers for the CIFARdataset and compare against various baselines.

Original: We ﬁnd that the GAN trained on the smaller Image B outperforms all other methods including the fully-supervised trained one for the ﬁrst convolutional layer.
Revised: We ﬁnd that the GAN trained on the smaller Image B outperforms all other methods including the fully-supervised trained one for the ﬁrst convolutional layer.

Original: We also outperform the same architecture trained on the full CIFAR-10 training set using
Revised: We also outperform the same architecture trained on the full CIFAR-10 training set using

Original: RotNet, which might be due to the fact that either CIFAR images do not contain much information about the orientation of the picture or because they do not contain as many objects as in ImageNet.
Revised: RotNet, which might be due to the fact that either CIFAR images do not contain much information about the orientation of the picture or because they do not contain as many objects as in ImageNet.

Original: While the GAN trained on the whole dataset outperforms the MonoGAN on the deeper layers, the gap stays very small until the last layer.
Revised: While the GAN trained on the whole dataset outperforms the MonoGAN on the deeper layers, the gap stays very small until the last layer.

Original: These ﬁndings are also reﬂected in the experiments on the
Revised: These ﬁndings are also reﬂected in the experiments on the

Original: CIFAR-100 dataset shown in Table 3.
Revised: CIFAR-100 dataset shown in Table 3.

Original: We ﬁnd that our method obtains the best performance for the ﬁrst two layers, even against the fully supervised version.
Revised: We ﬁnd that our method obtains the best performance for the ﬁrst two layers, even against the fully supervised version.

Original: The gap between our mono variant and the other methods increases again with deeper layers, hinting to the fact that we cannot learn very high level concepts in deeper layers from just one single image.
Revised: The gap between our mono variant and the other methods increases again with deeper layers, hinting to the fact that we cannot learn very high level concepts in deeper layers from just one single image.

Original: These results corroborate the ﬁnding that our method allows learning very generalizable early features that are not domain dependent.
Revised: These results corroborate the ﬁnding that our method allows learning very generalizable early features that are not domain dependent.

Original: 4. Q UALITATIVE A NALYSIS
Revised: 4. Q UALITATIVE A NALYSIS

Original: Visual comparison of weights.
Revised: Visual comparison of weights.

Original: In Figure 2, we compare the learned ﬁlters of all ﬁrst-layer convolutions of an AlexNet trained with the different methods and a single image.
Revised: In Figure 2, we compare the learned ﬁlters of all ﬁrst-layer convolutions of an AlexNet trained with the different methods and a single image.

Original: First, we ﬁnd that the ﬁlters closely resemble those obtained via supervised training: Gabor-like edge detectors and various color blobs.
Revised: First, we ﬁnd that the ﬁlters closely resemble those obtained via supervised training: Gabor-like edge detectors and various color blobs.

Original: Second, we ﬁnd that the look is not easily predictive of its performance, e.g. while generatively learned ﬁlters (BiGAN) show many edge detectors, its linear probes performance is about the same as that of DeepCluster which seems to learn many somewhat redundant point features.
Revised: Second, we ﬁnd that the look is not easily predictive of its performance, e.g. while generatively learned ﬁlters (BiGAN) show many edge detectors, its linear probes performance is about the same as that of DeepCluster which seems to learn many somewhat redundant point features.

Original: However, we also ﬁnd that some edge detectors are required, as we can conﬁrm from RotNet and DeepCluster trained on Image B , which yield less crisp ﬁlters and worse performances.
Revised: However, we also ﬁnd that some edge detectors are required, as we can conﬁrm from RotNet and DeepCluster trained on Image B , which yield less crisp ﬁlters and worse performances.

Original: [Table]
Revised: [Table]

Original: Fine-tuning instead of freezing.
Revised: Fine-tuning instead of freezing.

Original: In Tab. 4, we show the results of retraining a network with the ﬁrst two convolutional ﬁlters, or the scattering transform from (Oyallon et al., 2017), left frozen.
Revised: In Tab. 4, we show the results of retraining a network with the ﬁrst two convolutional ﬁlters, or the scattering transform from (Oyallon et al., 2017), left frozen.

Original: We observe that our single image trained DeepCluster and BiGAN models achieve performances closes to the supervised benchmark.
Revised: We observe that our single image trained DeepCluster and BiGAN models achieve performances closes to the supervised benchmark.

Original: Notably, the scattering transform as a replacement for conv1performs slightly worse than the analyzed single image methods.
Revised: Notably, the scattering transform as a replacement for conv1performs slightly worse than the analyzed single image methods.

Original: We also show in the appendix the results of retraining a network initialized with the ﬁrst two convolutional layers obtained from a single image and subsequently linearly probing the model.
Revised: We also show in the appendix the results of retraining a network initialized with the ﬁrst two convolutional layers obtained from a single image and subsequently linearly probing the model.

Original: The results are shown in Tab.
Revised: The results are shown in Appendix

Original:  and we ﬁnd that we can recover the performance of fully-supervised networks, i.e. the ﬁrst two convolutional ﬁlters trained from just a single image generalize well and do not get stuck in an image speciﬁc minimum.
Revised: Tab. 5 and we ﬁnd that we can recover the performance of fully-supervised networks, i.e. the ﬁrst two convolutional ﬁlters trained from just a single image generalize well and do not get stuck in an image speciﬁc minimum.

Original: Neural style transfer.
Revised: Neural style transfer.

Original: Lastly, we show how our features trained on only a single image can be used for other applications.
Revised: Lastly, we show how our features trained on only a single image can be used for other applications.

Original: In Figure 3 we show two basic style transfers using the method of (Gatys et al., 2016) from an ofﬁcial PyTorch tutorial 2 .
Revised: In Figure 3 we show two basic style transfers using the method of (Gatys et al., 2016) from an ofﬁcial PyTorch tutorial 3 .

Original: Image content and style are separated and the style is transferred from the source to target image using all CNN features, not just the shallow layers.
Revised: Image content and style are separated and the style is transferred from the source to target image using all CNN features, not just the shallow layers.

Original: We visually compare the results of using our features and from full ImageNet supervision.
Revised: We visually compare the results of using our features and from full ImageNet supervision.

Original: We ﬁnd almost no visual differences in the stylized images and can conclude that our early features are equally powerful as fully supervised ones for this task.
Revised: We ﬁnd almost no visual differences in the stylized images and can conclude that our early features are equally powerful as fully supervised ones for this task.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have made the surprising observation that we can learn good and generalizable features through self-supervision from one single source image, provided that sufﬁcient data augmentation is used.
Revised: We have made the surprising observation that we can learn good and generalizable features through self-supervision from one single source image, provided that sufﬁcient data augmentation is used.

Original: Our results complement recent works (Mahajan et al., 2018;
Revised: Our results complement recent works (Mahajan et al., 2018;

Original: Goyal et al., 2019) that have investigated self-supervision in the very large data regime.
Revised: Goyal et al., 2019) that have investigated self-supervision in the very large data regime.

Original: Our main conclusion is that these methods succeed perfectly in capturing the simplest image statistics, but that for deeper layers a gap exist with strong supervision which is compensated only in limited manner by using large datasets.
Revised: Our main conclusion is that these methods succeed perfectly in capturing the simplest image statistics, but that for deeper layers a gap exist with strong supervision which is compensated only in limited manner by using large datasets.

Original: This novel ﬁnding motivates a renewed focus on the role of augmentations in self-supervised learning and critical rethinking of how to better leverage the available data.
Revised: This novel ﬁnding motivates a renewed focus on the role of augmentations in self-supervised learning and critical rethinking of how to better leverage the available data.

Revised: A CKNOWLEDGEMENTS .

Revised: We thank Aravindh Mahendran for fruitful discussions.

Revised: Yuki Asano gratefully acknowledges support from the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines & Systems (EP/L015897/1).

Revised: The work is supported by ERC IDIU-638009.

Original: V ISUAL C OMPARISON OF F ILTERS
Revised: V ISUAL C OMPARISON OF F ILTERS

Original: In order to understand what deeper neurons are responding to in our model, we visualize random neurons via activation maximization (Erhan et al., 2009; Zeiler & Fergus, 2014) in each layer.
Revised: In order to understand what deeper neurons are responding to in our model, we visualize random neurons via activation maximization (Erhan et al., 2009; Zeiler & Fergus, 2014) in each layer.

Original: Additionally, we retrieve the top-9 images in the ImageNet training set that activate each neuron most in Figure 5.
Revised: Additionally, we retrieve the top-9 images in the ImageNet training set that activate each neuron most in Figure 5.

Original: Since the mono networks are not trained on the ImageNet dataset, it can be used here for visualization.
Revised: Since the mono networks are not trained on the ImageNet dataset, it can be used here for visualization.

Original: From the ﬁrst convolutional layer we ﬁnd typical neurons strongly reacting to oriented edges.
Revised: From the ﬁrst convolutional layer we ﬁnd typical neurons strongly reacting to oriented edges.

Original: In layers 2-4 we ﬁnd patterns such as grids ( conv2 :3), and textures such as leopard skin ( conv2 :2) and round grid cover ( conv4 :4).
Revised: In layers 2-4 we ﬁnd patterns such as grids ( conv2 :3), and textures such as leopard skin ( conv2 :2) and round grid cover ( conv4 :4).

Original: Conﬁrming our hypothesis that the neural network is only extracting patterns and not semantic information, we do not ﬁnd
Revised: Conﬁrming our hypothesis that the neural network is only extracting patterns and not semantic information, we do not ﬁnd

Original: any neurons particularly specialized to certain objects even in higher levels as for example dog faces or similar which can be fund in supervised networks.
Revised: any neurons particularly specialized to certain objects even in higher levels as for example dog faces or similar which can be fund in supervised networks.

Original: This ﬁnding aligns with the observations of other unsupervised methods (Caron et al., 2018; Zhang et al., 2017).
Revised: This ﬁnding aligns with the observations of other unsupervised methods (Caron et al., 2018; Zhang et al., 2017).

Original: As most neurons extract simple patterns and
Revised: As most neurons extract simple patterns and

Original: [Table]
Revised: [Table]

Original: textures, the surprising effectiveness of training a network using a single image can be explained by the recent ﬁnding that even CNNs trained on ImageNet rely on texture (as opposed to shape) information to classify (Geirhos et al., 2019).
Revised: textures, the surprising effectiveness of training a network using a single image can be explained by the recent ﬁnding that even CNNs trained on ImageNet rely on texture (as opposed to shape) information to classify (Geirhos et al., 2019).

Original: A. R ETRAINING FROM SINGLE IMAGE INITIALIZATION
Revised: A. R ETRAINING FROM SINGLE IMAGE INITIALIZATION

Original: In Table 5, we initialize AlexNet models using the ﬁrst two convolutional ﬁlters learned from a single image and retrain them using ImageNet.
Revised: In Table 5, we initialize AlexNet models using the ﬁrst two convolutional ﬁlters learned from a single image and retrain them using ImageNet.

Original: We ﬁnd that the networks recover their performance fully and the ﬁrst ﬁlters do not make the network stuck in a bad local minimum despite having been trained on a single image from a different distribution.
Revised: We ﬁnd that the networks recover their performance fully and the ﬁrst ﬁlters do not make the network stuck in a bad local minimum despite having been trained on a single image from a different distribution.

Original: The difference from the BiGAN to the full supervision model is likely due to it using a smaller input resolution (112 instead of 224), as the
Revised: The difference from the BiGAN to the full supervision model is likely due to it using a smaller input resolution (112 instead of 224), as the

Original: BiGAN’s output resolution is limited.
Revised: BiGAN’s output resolution is limited.

Original: A. L INEAR P ROBES ON I MAGE N ET
Revised: A. L INEAR P ROBES ON I MAGE N ET

Original: We show two plots of the ImageNet linear probes results (Table 2 of the paper) in ﬁg. 6.
Revised: We show two plots of the ImageNet linear probes results (Table 2 of the paper) in ﬁg. 6.

Original: On the left we plot performance per layer in absolute scale.
Revised: On the left we plot performance per layer in absolute scale.

Original: Naturally the performance of the supervised model improves with depth, while all unsupervised models degrade after conv3 .
Revised: Naturally the performance of the supervised model improves with depth, while all unsupervised models degrade after conv3 .

Original: From the relative plot on the right, it becomes clear that with our training scheme, we can even slightly surpass supervised performance on conv1 presumably since our model is trained with sometimes very small patches, thus receiving an emphasis on learning good low level ﬁlters.
Revised: From the relative plot on the right, it becomes clear that with our training scheme, we can even slightly surpass supervised performance on conv1 presumably since our model is trained with sometimes very small patches, thus receiving an emphasis on learning good low level ﬁlters.

Original: The gap between all self-supervised methods and the supervised baseline increases with depth, due to the fact that the supervised model is trained for this speciﬁc task, whereas the self-supervised models learn from a surrogate task without labels.
Revised: The gap between all self-supervised methods and the supervised baseline increases with depth, due to the fact that the supervised model is trained for this speciﬁc task, whereas the self-supervised models learn from a surrogate task without labels.

Original: [Figure]
Revised: [Figure]

Original: A. E XAMPLE A UGMENTED T RAINING D ATA
Revised: A. E XAMPLE A UGMENTED T RAINING D ATA

Original: In ﬁgs.
Revised: In ﬁgs.

Original: 7 to 10 we show example patches generated by our augmentation strategy for the datasets with different N.
Revised: 7 to 10 we show example patches generated by our augmentation strategy for the datasets with different N.

Original: Even though the images and patches are very different in color and shape distribu- tion, our model learns weights that perform similarly in the linear probes benchmark (see Table 2 in the paper). 
Revised: Even though the images and patches are very different in color and shape distribu- tion, our model learns weights that perform similarly in the linear probes benchmark (see Table 2 in the paper). [Figure]