Original: On the surprising similarities between supervised and self-supervised models
Revised: On the surprising similarities between supervised and self-supervised models

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: How do humans learn to acquire a powerful, ﬂexible and robust representation ofobjects?
Revised: How do humans learn to acquire a powerful, ﬂexible and robust representation of objects?

Original: While much of this process remains unknown, it is clear that humans donot require millions of object labels.
Revised: While much of this process remains unknown, it is clear that humans do not require millions of object labels.

Original: Excitingly, recent algorithmic advancementsin self-supervised learning now enable convolutional neural networks (CNNs) tolearn useful visual object representations without supervised labels, too.
Revised: Excitingly, recent algorithmic advancements in self-supervised learning now enable convolutional neural networks (CNNs) to learn useful visual object representations without supervised labels, too.

Original: In thelight of this recent breakthrough, we here compare self-supervised networks tosupervised models and human behaviour.
Revised: In the light of this recent breakthrough, we here compare self-supervised networks to supervised models and human behaviour.

Original: We tested models on 15 generalisation datasets for which large-scale human behavioural data is available (130K highly controlled psychophysical trials).
Revised: We tested models on 15 generalisation datasets for which large-scale human behavioural data is available (130K highly controlled psychophysical trials).

Original: Surprisingly, current self-supervised CNNs share four key characteristics of theirsupervised counterparts: (1.) relatively poor noise robustness (with the notable exception of SimCLR), (2.) non-human category-level error patterns, (3.) non-humanimage-level error patterns (yet high similarity to supervised model errors) and(4.)
Revised: Surprisingly, current self-supervised CNNs share four key characteristics of their supervised counterparts: (1.) relatively poor noise robustness (with the notable exception of SimCLR), (2.) non-human category-level error patterns, (3.) non-human image-level error patterns (yet high similarity to supervised model errors) and (4.)

Original: a bias towards texture.
Revised: a bias towards texture.

Original: Taken together, these results suggest that the strategieslearned through today’s supervised and self-supervised training objectives endup being surprisingly similar, but distant from human-like behaviour.
Revised: Taken together, these results suggest that the strategies learned through today’s supervised and self-supervised training objectives end up being surprisingly similar, but distant from human-like behaviour.

Original: That beingsaid, we are clearly just at the beginning of what could be called a self-supervisedrevolution of machine vision, and we are hopeful that future self-supervised modelsbehave differently from supervised ones, and—perhaps—more similar to robusthuman object recognition.
Revised: That being said, we are clearly just at the beginning of what could be called a self-supervised revolution of machine vision, and we are hopeful that future self-supervised models behave differently from supervised ones, and—perhaps—more similar to robust human object recognition.

Original:  Introduction
Revised:  Introduction

Original: “If intelligence is a cake, the bulk of the cake is unsupervised learning, the icing on the cake issupervised learning and the cherry on the cake is reinforcement learning” , Yann LeCun famouslysaid [1].
Revised: “If intelligence is a cake, the bulk of the cake is unsupervised learning, the icing on the cake is supervised learning and the cherry on the cake is reinforcement learning” , Yann LeCun famously said [1].

Original: Four years later, the entire cake is ﬁnally on the table—the representations learned viaself-supervised learning now compete with supervised methods on ImageNet [2] and outperformsupervised pre-training for object detection [3].
Revised: Four years later, the entire cake is ﬁnally on the table—the representations learned via self-supervised learning now compete with supervised methods on ImageNet [2] and outperform supervised pre-training for object detection [3].

Original: But given this fundamentally different learningmechanism, how do recent self-supervised models differ from their supervised counterparts in termsof their behaviour?
Revised: But given this fundamentally different learning mechanism, how do recent self-supervised models differ from their supervised counterparts in terms of their behaviour?

Original: We here attempt to shed light on this question by comparing eight ﬂavours of “cake” ( PIRL , MoCo ,
Revised: We here attempt to shed light on this question by comparing eight ﬂavours of “cake” ( PIRL , MoCo ,

Original: MoCoV2 , InfoMin , InsDis , SimCLR-x1 , SimCLR-x2 , SimCLR-x4 ) with 24 common variants of“icing” (from the AlexNet , VGG , Squeezenet , DenseNet , Inception , ResNet , ShuffleNet ,
Revised: MoCoV2 , InfoMin , InsDis , SimCLR-x1 , SimCLR-x2 , SimCLR-x4 ) with 24 common variants of “icing” (from the AlexNet , VGG , Squeezenet , DenseNet , Inception , ResNet , ShuffleNet ,

Original: MobileNet , ResNeXt , WideResNet and MNASNet star cuisines).
Revised: MobileNet , ResNeXt , WideResNet and MNASNet star cuisines).

Original: Speciﬁcally, our culinary testbuffet aims to investigate:
Revised: Speciﬁcally, our culinary test buffet aims to investigate:

Original: Are self-supervised models more robust towards distortions?
Revised: Are self-supervised models more robust towards distortions?

Original: Do self-supervised models make similar errors as either humans or supervised models?
Revised: Do self-supervised models make similar errors as either humans or supervised models?

Original: Do self-supervised models recognise objects by texture or shape?
Revised: Do self-supervised models recognise objects by texture or shape?

Original: For all of these questions, we compare supervised and self-supervised 1 models against a comprehensive set of openly available human psychophysical data totalling over 130,000 trials [4, 5].
Revised: For all of these questions, we compare supervised and self-supervised 1 models against a comprehensive set of openly available human psychophysical data totalling over 130,000 trials [4, 5].

Original: This ismotivated on one hand by the fact that humans, too, rapidly learn to recognise new objects withoutrequiring hundreds of labels per instance; and on the other hand by a number of fascinating studiesreporting increased similarities between self-supervised models and human perception.
Revised: This is motivated on one hand by the fact that humans, too, rapidly learn to recognise new objects without requiring hundreds of labels per instance; and on the other hand by a number of fascinating studies reporting increased similarities between self-supervised models and human perception.

Original: For instance,
Revised: For instance,

Original: Lotter et al.
Revised: Lotter et al.

Original: [6] train a model for self-supervised next frame prediction on videos, which leads tophenomena known from human vision, including perceptual illusions.
Revised: [6] train a model for self-supervised next frame prediction on videos, which leads to phenomena known from human vision, including perceptual illusions.

Revised: Orhan et al.

Revised: [7] train a self-supervised model on infant video data, ﬁnding good categorisation accuracies on some (small) datasets.

Original:  Konkle and Alvarez [7]and Zhuang et al.
Revised: Furthermore, Konkle and Alvarez [8] and Zhuang et al.

Original: [8] report an improved match with neural data; Zhuang et al.
Revised: [9] report an improved match with neural data; Zhuang et al.

Original: [8] also ﬁnd morehuman-like error patterns for semi-supervised models and Storrs and Fleming [9] observe that aself-supervised network accounts for behavioural patterns of human gloss perception.
Revised: [9] also ﬁnd more human-like error patterns for semi-supervised models and Storrs and Fleming [10] observe that a self-supervised network accounts for behavioural patterns of human gloss perception.

Original: While methodsand models differ substantially across these studies, they jointly provide evidence for the intriguinghypothesis that self-supervised machine learning models may better approximate human vision.
Revised: While methods and models differ substantially across these studies, they jointly provide evidence for the intriguing hypothesis that self-supervised machine learning models may better approximate human vision.

Original:  Methods
Revised:  Methods

Original: Models.
Revised: Models.

Original: InsDis[10], MoCo [11], MoCoV2 [12],  PIRL [3] and InfoMin [13] were obtained as pretrained models from the PyContrast model zoo.
Revised: InsDis [11], MoCo [12], MoCoV2 [13], PIRL [3] and InfoMin [14] were obtained as pre-trained models from the PyContrast model zoo.

Original: We trained one linear classiﬁer per model on top ofthe self-supervised representation.
Revised: We trained one linear classiﬁer per model on top of the self-supervised representation.

Original: A PyTorch [14] implementation of SimCLR [2] was obtained viasimclr-converter.
Revised: A PyTorch [15] implementation of SimCLR [2] was obtained via simclr-converter.

Original: All self-supervised models use a ResNet-50 architecture and a different trainingapproach within the framework of contrastive learning [e.g. 15].
Revised: All self-supervised models use a ResNet-50 architecture and a different training approach within the framework of contrastive learning [e.g. 16].

Original: For supervised models, we used all24 available pre-trained models from the PyTorch model zoo version 1.4.0 ( VGG : with batch norm). 
Revised: For supervised models, we used all 24 available pre-trained models from the PyTorch model zoo version 1.4.0 ( VGG : with batch norm). Linear classiﬁer training procedure.

Revised: The PyContrast repository by Yonglong Tian contains a

Original: Datasets.
Revised: Pytorch implementation of unsupervised representation learning methods, including pre-trained representation weights. The repository provides training and evaluation pipelines, but it supports only multi-node distributed training and does not (currently) provide weights for the classiﬁer. We have used the repository’s linear classiﬁer evaluation pipeline to train classiﬁers for InsDis [11], MoCo [12], MoCoV2 [13], PIRL [3] and InfoMin [14] on ImageNet. Pre-trained weights of the model representations (without classiﬁer) were taken from the provided Dropbox link and we then ran the training pipeline on a NVIDIA TESLA P100 using the default parameters conﬁgured in the pipeline.

Revised: Detailed documentation about running the pipeline and parameters can be found in the PyContrast repository (commit #3541b82).

Original:  Models were tested on 12 different image degradations from [4], as well as on texture-vsshape datasets from [5].
Revised: Datasets. Models were tested on 12 different image degradations from [4], as well as on texture-vsshape datasets from [5].

Original: Plotting conventions follow these papers (unless indicated otherwise).
Revised: Plotting conventions follow these papers (unless indicated otherwise).

Original:  Results
Revised:  Results

Revised: We here investigate four behavioural characteristics of self-supervised networks, comparing them to their supervised counterparts on the one hand and to human observers on the other hand: out-ofdistribution generalisation (3.1), category-level error patterns (3.2), image-level error patterns (3.3), and texture/shape biases (3.4).

Revised: [Figure]

Original: 3.1 With the exception of SimCLR, supervised and self-supervised models show similar(non-human) out-of-distribution generalisation
Revised: C l a ss i f i c a t i o n a cc u r a c y [Figure] With the exception of SimCLR, supervised and self-supervised models show similar (non-human) out-of-distribution generalisation

Original: Motivation.
Revised: Motivation.

Original: Given sufﬁcient quantities of labelled training data, CNNs can learn to identify objectswhen the input images are noisy.
Revised: Given sufﬁcient quantities of labelled training data, CNNs can learn to identify objects when the input images are noisy.

Original: However, supervised CNNs typically generalise poorly to noveldistortion types not seen during training, so-called out-of-distribution images [4].
Revised: However, supervised CNNs typically generalise poorly to novel distortion types not seen during training, so-called out-of-distribution images [4].

Original: In contrast, humanperception is remarkably robust when dealing with previously unseen types of noise.
Revised: In contrast, human perception is remarkably robust when dealing with previously unseen types of noise.

Original: Given thatrecent self-supervised networks are trained to identify objects under a variety of transformations (likescaling, cropping and colour shifts), have they learned a more robust, human-like representation ofobjects, where high-level semantic content is unimpaired by low-level noise?
Revised: Given that recent self-supervised networks are trained to identify objects under a variety of transformations (like scaling, cropping and colour shifts), have they learned a more robust, human-like representation of objects, where high-level semantic content is unimpaired by low-level noise?

Original: Results.
Revised: Results.

Original: In Figure 1, we compare self-supervised and supervised networks on twelve differenttypes of image distortions.
Revised: In Figure 1, we compare self-supervised and supervised networks on twelve different types of image distortions.

Original: Human observers were tested on the exact same distortions by [4].
Revised: Human observers were tested on the exact same distortions by [4].

Original: Acrossdistortion types, self-supervised networks are well within the range of their poorly generalisingsupervised counterparts.
Revised: Across distortion types, self-supervised networks are well within the range of their poorly generalising supervised counterparts.

Original: However, there is one exception: SimCLR shows strong generalisationimprovements on uniform noise, low contrast and high-pass images—quite remarkable given that thenetwork was trained using other augmentations (random crop with ﬂip and resize, colour distortion,and Gaussian blur).
Revised: However, there is one exception: SimCLR shows strong generalisation improvements on uniform noise, low contrast and high-pass images—quite remarkable given that the network was trained using other augmentations (random crop with ﬂip and resize, colour distortion, and Gaussian blur).

Original: Apart from SimCLR , however, we do not ﬁnd beneﬁts of self-supervised trainingfor distortion robustness.
Revised: Apart from SimCLR , however, we do not ﬁnd beneﬁts of self-supervised training for distortion robustness.

Original: These results for ImageNet models contrast with [16] who observed somerobustness improvements for a self-supervised model trained on the CIFAR-10 dataset.
Revised: These results for ImageNet models contrast with [17] who observed some robustness improvements for a self-supervised model trained on the CIFAR-10 dataset.

Original: 3.2 Self-supervised models make non-human category-level errors
Revised: 3.2 Self-supervised models make non-human category-level errors

Original: Motivation.
Revised: Motivation.

Original: On clean images, CNNs now recognise objects as well as humans.
Revised: On clean images, CNNs now recognise objects as well as humans.

Original: But do they alsoconfuse similar categories with each other (which can be investigated using confusion matrices)?
Revised: But do they also confuse similar categories with each other (which can be investigated using confusion matrices)?

Original: Results.
Revised: Results.

Original: In Figure 3, we compare category-level errors of humans against a standard supervised CNN( ResNet-50 ) and three self-supervised CNNs.
Revised: In Figure 4 (moved to appendix for space reasons), we compare category-level errors of humans against a standard supervised CNN ( ResNet-50 ) and three self-supervised CNNs.

Original: We chose uniform noise for this comparison since thisis one of the noise types where SimCLR shows strong improvements, nearing human-level accuracies.
Revised: We chose h u m v s . h u m s u p v s .

Original: Looking at the confusion matrices, both humans and CNNs start with a dominant diagonal indicating
Revised: s

Original: [Figure]
Revised: u

Original: C la ss i f 
Revised: p s e l f - s u p v s . s

Original: i c a t i o n a cc u r a c y
Revised: e l f - s u p h u m v s . s u p

Original: [Figure]
Revised: h u m v s . s e l f - s u p s u p v s .

Original: c
Revised: s

Original: [Figure] correct categorisation. With increasing noise level, however, all CNNs develop a strong tendencyto predict one category and only one: “knife” for ResNet-50 , “clock” for InsDis , MoCoV2 and
Revised: e l f - s u p 0.

Original: SimCLR-x4 .
Revised: E rr o r c

Original: [-Human observers, on the other hand, more or less evenly distribute their errors acrossclasses. For supervised networks, this pattern of errors was already observed by [4]; we here ﬁndthat this peculiar idiosyncrasy is shared by self-supervised networks, indicating that discriminativesupervised training is not the underlying reason for this non-human behaviour.3.
 Self-supervised models make non-human image-level errors (error consistency)-]
Revised: [-Human observers, on the other hand, more or less evenly distribute their errors acrossclasses. For supervised networks, this pattern of errors was already observed by [4]; we here ﬁndthat this peculiar idiosyncrasy is shared by self-supervised networks, indicating that discriminativesupervised training is not the underlying reason for this non-human behaviour.3.
 Self-supervised models make non-human image-level errors (error consistency)-]o n s i s t e n c y

Original: [Figure]
Revised: [Figure]

Original: [Equation] m [Equation] [Figure]
Revised: experiment ‘cue conﬂict’ stimuli h u m v s .

Original: hu
Revised: h u

Original: [Figure] s .
Revised: m s u p v s .

Original: s
Revised: s

Original: u
Revised: u

Original: p s e l f - s u p v s .
Revised: p s e l f - s u p v s .

Original: s
Revised: s

Original: e l f - s u p hu
Revised: e l f - s u p h u m v s . s u p h

Original:  m v s .
Revised: u m v s .

Revised: s

Revised: e l f - s u p s u p v s .

Original: s
Revised: s

Revised: e l f - s u p 0.

Revised: E rr o r c

Original: u p hu
Revised: o n s i s t e n c y ( [Figure] Edge [Figure]

Revised: ‘edge’ stimuli h u m v s .

Original:  m  v s . 
Revised: h u m s u p v s . s u

Revised: p s e l f - s u p v s .

Original: s
Revised: s

Original: e l f - s u p  s u p  v s .
Revised: e l f - s u p h u m v s . s u p h u m v s .

Revised: s

Revised: e l f - s u p s u p v s .

Original: s
Revised: s

Original: e l f - s u p 0.
Revised: e l f - s u p 0.

Original: E rr o r c
Revised: E rr o r c

Original: o n s i s t e n c y
Revised: o n s i s t e n c y

Original: Motivation.
Revised: [Figure] uniform noise for this comparison since this is one of the noise types where SimCLR shows strong improvements, nearing human-level accuracies. Looking at the confusion matrices, both humans and CNNs start with a dominant diagonal indicating correct categorisation. With increasing noise level, however, all CNNs develop a strong tendency to predict one category and only one, such as “knife” for ResNet-50 .

Revised: Human observers, on the other hand, more or less evenly distribute their errors across classes.

Revised: For supervised networks, this pattern of errors was already observed by [4]; we here ﬁnd that this peculiar idiosyncrasy is shared by self-supervised networks, indicating that discriminative supervised training is not the underlying reason for this non-human behaviour.

Revised: 3.3 Self-supervised models make non-human image-level errors (error consistency)

Original:  Achieving human-level accuracies on adataset does not necessarily imply using a human-likestrategy: different strategies can lead to similar accuracies.
Revised: Motivation. Achieving human-level accuracies on a dataset does not necessarily imply using a human-like strategy: different strategies can lead to similar accuracies.

Original: Therefore, it is essential to investigate on which stimuli errors occur.
Revised: Therefore, it is essential to investigate on which stimuli errors occur.

Original: If two decision makers—for instance,a human observer and a CNN—use a similar strategy,we can expect them to consistently make errors on thesame individual images.
Revised: If two decision makers—for instance, a human observer and a CNN—use a similar strategy, we can expect them to consistently make errors on the same individual images.

Original: This intuition is captured byerror consistency ( κ ), a metric to measure the degree towhich decision makers make the same image-level errors[17].
Revised: This intuition is captured by error consistency ( κ ), a metric to measure the degree to which decision makers make the same image-level errors [18].

Original: κ > 0 means that two decision makers systematically make errors on the same images; κ = 0 indicatesno more error overlap than what could be expected bychance alone.
Revised: κ > 0 means that two decision makers systematically make errors on the same images; κ = 0 indicates no more error overlap than what could be expected by chance alone.

Original: Results.
Revised: Results.

Original: Figure 2 plots the consistency of errors (measured by κ ).
Revised: Figure 2 plots the consistency of errors (measured by κ ).

Original: Humans make highly similarerrors as other humans (mean κ = 0 . 32 ), but neither supervised nor self-supervised models makehuman-like errors.
Revised: Humans make highly similar errors as other humans (mean κ = 0 . 32 ), but neither supervised nor self-supervised models make human-like errors.

Original: Instead, error consistency between model groups (self-supervised vs. supervised)is just as high as consistency within model groups: self-supervised models make errors on the sameimages as supervised models, an indicator for highly similar strategies.Cond. [Figure] humans resnet
Revised: Instead, error consistency between model groups (self-supervised vs. supervised) is just as high as consistency within model groups: self-supervised models make errors on the same images as supervised models, an indicator for highly similar strategies.

Original: InsDis

Original: MoCoV SimCLR-x [Figure] 3.4 Self-supervised models are biased towards [-texture0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.
-]
Revised: 3.4 Self-supervised models are biased towards [-texture0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.
-]texture

Original: Fraction of 'shape' decisions 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.

Original: Fraction of 'texture' decisions

Original: [Figure]

Original: Motivation.
Revised: Motivation.

Original: Standard supervisednetworks recognise objects byrelying on local texture statistics,largely ignoring global object shape[5, 18, 19].
Revised: Standard supervised networks recognise objects by relying on local texture statistics, largely ignoring global object shape [5, 19, 20].

Original: This striking differenceto human visual perception has beenattributed to the fact that texture isa shortcut sufﬁcient to discriminateamong objects [20]—but is texturealso sufﬁcient to solve self-supervisedtraining objectives?
Revised: This striking difference to human visual perception has been attributed to the fact that texture is a shortcut sufﬁcient to discriminate among objects [21]—but is texture also sufﬁcient to solve self-supervised training objectives?

Original: Results.
Revised: Results.

Original: We tested a broad range of CNNs on the texture-shape cue conﬂict dataset from [5].
Revised: We tested a broad range of CNNs on the texture-shape cue conﬂict dataset from [5].

Original: This datasetconsists of images where the shapebelongs to one category (e.g. cat) andthe texture belongs to a different category (e.g. elephant).
Revised: This dataset consists of images where the shape belongs to one category (e.g. cat) and the texture belongs to a different category (e.g. elephant).

Original: When plottingwhether CNNs prefer texture or shape(Figure 4), we observe that most selfsupervised models have a strong texture bias known from traditional supervised models.
Revised: When plotting whether CNNs prefer texture or shape (Figure 3), we observe that most self-supervised models have a strong texture bias known from traditional supervised models.

Original: This texture biasis less prominent for SimCLR (58.3–61.2% texture decisions), which is still on par with supervised model Inception-V3 (60.7% texturedecisions).
Revised: This texture bias is less prominent for SimCLR (58.3–61.2% texture decisions), which is still on par with supervised model Inception-V3 (60.7% texture decisions).

Original: These ﬁndings are in line with [21], who observed that the inﬂuence of training dataaugmentations on shape bias is stronger than the role of architecture or training objective.
Revised: These ﬁndings are in line with [22], who observed that the inﬂuence of training data augmentations on shape bias is stronger than the role of architecture or training objective.

Original: Neithersupervised nor self-supervised models have the strong shape bias that is so characteristic for humanobservers, indicating fundamentally different decision making processes between humans and CNNs.
Revised: Neither supervised nor self-supervised models have the strong shape bias that is so characteristic for human observers, indicating fundamentally different decision making processes between humans and CNNs.

Original:  Discussion
Revised:  Discussion

Original: Comparing self-supervised networks to supervised models and human observers, we here investigated four key behavioural characteristics: out-of-distribution generalisation, category-level errorpatterns, image-level error patterns, and texture bias.
Revised: Comparing self-supervised networks to supervised models and human observers, we here investigated four key behavioural characteristics: out-of-distribution generalisation, category-level error patterns, image-level error patterns, and texture bias.

Original: Overall, we ﬁnd that self-supervised modelsresemble their supervised counterparts much more closely than what could have been expectedgiven fundamentally different training objectives.
Revised: Overall, we ﬁnd that self-supervised models resemble their supervised counterparts much more closely than what could have been expected given fundamentally different training objectives.

Original: While standard models are notoriously non-robust[4, 20, 22–24], SimCLR represents a notable exception in some of our experiments as it is less biasedtowards texture and much more robust towards some types of distortions.
Revised: While standard models are notoriously non-robust [4, 21, 23–25], SimCLR represents a notable exception in some of our experiments as it is less biased towards texture and much more robust towards some types of distortions.

Original: It is an open questionwhether these beneﬁts arise from the training objective or whether they are instead induced by thespeciﬁc set of data augmentations used during SimCLR model training.
Revised: It is an open question whether these beneﬁts arise from the speciﬁc set of data augmentations used during SimCLR model training.

Original: Perhaps surprisingly, error consistency analysis suggests that the images on which supervised andself-supervised models make errors overlap strongly, much more than what could have been expectedby chance alone.
Revised: Perhaps surprisingly, error consistency analysis suggests that the images on which supervised and self-supervised models make errors overlap strongly, much more than what could have been expected by chance alone.

Original: This provides evidence for similar processing mechanisms: It seems that switchinglabel-based supervision for a contrastive learning scheme does not have a strong effect on the inductivebias of the resulting model—at least for the currently used contrastive approaches.
Revised: This provides evidence for similar processing mechanisms: It seems that switching label-based supervision for a contrastive learning scheme does not have a strong effect on the inductive [Figure] [Figure] S h a p e c a t e g

Original:  Furthermore, weﬁnd little evidence for human-like behaviour in the investigated self-supervised models.
Revised: o r i e s bias of the resulting model—at least for the currently used contrastive approaches. Furthermore, we ﬁnd little evidence for human-like behaviour in the investigated self-supervised models.

Revised: While this investigation focused on state-of-the-art contrastive learning methods, other self-supervised methods might lead to different results.

Original: We are clearly just witnessing the beginning of what could be called a self-supervised revolutionof machine vision, and we expect future self-supervised models to behave signiﬁcantly differentlyfrom supervised ones.
Revised: We are clearly just witnessing the beginning of what could be called a self-supervised revolution of machine vision, and we expect future self-supervised models to behave signiﬁcantly differently from supervised ones.

Original: What we are showing is that the current self-supervised CNNs are not yetmore human-like in their strategies and internal representations than plain-vanilla supervised CNNs.
Revised: What we are showing is that the current self-supervised CNNs are not yet more human-like in their strategies and internal representations than plain-vanilla supervised CNNs.

Original: We hope, however, that analyses like ours may facilitate the tracking of emerging similarities anddifferences, whether between different types of models or between models and human perception.
Revised: We hope, however, that analyses like ours may facilitate the tracking of emerging similarities and differences, whether between different types of models or between models and human perception.

Original: AppendixMethod details: Linear classiﬁer training procedure for PyContrast models
Revised: Acknowledgement

Original: The PyContrast repository by Yonglong Tian contains a Pytorch implementation of unsupervised representation 220 learning methods, including pre-trained representation weights.
Revised: We thank Santiago Cadena for sharing a PyTorch implementation of SimCLR , and Yonglong Tian for providing pre-trained self-supervised models on github. Furthermore, we are grateful to the International Max Planck

Original: The repository provides training and evaluationpipelines, but it supports only multi-node distributed training and does not (currently) provide weights for  the 222 classiﬁer.
Revised: Research School for Intelligent Systems (IMPRS-IS) for supporting R.G.; the Collaborative Research Center (Projektnummer 276693517—SFB 1233: Robust Vision) for supporting M.B. and F.A.W.; the German Federal Ministry of Education and Research through the Tübingen AI Center (FKZ 01IS18039A) for supporting W.B. and M.B.; and the German Research Foundation through the Cluster of Excellence “Machine Learning—New Perspectives for Science”, EXC 2064/1, project number 390727645 for supporting F.A.W.

Original: We have used the repository’s linear classiﬁer evaluation pipeline to train classiﬁers for InsDis [10],
Revised: Author contributions

Original: MoCo [11], MoCoV2 [12], PIRL [3] and InfoMin [13] on ImageNet.
Revised: Project idea: R.G. and W.B. [Table] project lead [Table]

Original: Pre-trained weights of the model represen- 224 tations (without classiﬁer) were taken from the provided Dropbox link andwe then ran the training pipeline on a
Revised: R.G. [Table] implementing and training self-supervised models [Table]

Original: NVIDIA TESLA P100 using the default parameters conﬁgured in the pipeline.
Revised: K.N.

Original: Detailed documentation about 226 running the pipeline and parameters can be found in the PyContrast repository (commit #3541b82).
Revised: [Table] model evaluation pipeline: R.G., K.N. with input from W.B. [Table]

Original: Additional results
Revised: data visualisation:

Original: Figure 5 shows error consistency for two additional experiments, and Figure 6 shows additional confusion 229 matrices.
Revised: [Table] .G. and B.M. with input from M.B. [Table] F.A.W. and W.B.

Original: Both conﬁrm the results reported in the main text. [Equation] s . s
Revised: [Table]

Original: u p hu m v s . s e l f - s u p s u p v s .
Revised: guidance, feedback, infrastructure & funding acquisition: M.B., F.A.W. and W.B.

Original: s

Original: e l f - s u p 0. E rr o r c o n s i s t e n c y hu m v s .
Revised: [Table]

Original: hu m s u p v s .

Original: s

Original: u

Original: p s e l f - s u p v s . s e l f - s u p hu m v s .
Revised: paper writing: R.G. with input from all other authors.

Original: s u p hu m v s . s
Revised: Appendix

Original: e l f - s u p s u p v s .

Original: s

Original: e l f - s u p [Figure] [Figure] [Figure]
Revised: Figure 4 shows confusion matrices for uniform noise, Figure 5 for low-pass ﬁltering.

Original: [Figure]

Original: [Figure]

Original: rr

Original: [Figure]

Original: c o n s i s t

Original: [Figure]

Original: Cond.
Revised: Cond.

Original: [Figure]
Revised: [Figure]

Original: humans resnet
Revised: humans resnet

Original: InsDis
Revised: InsDis

Original: MoCoV
Revised: MoCoV

Revised: SimCLR-x

Revised: [Figure]

Revised: Cond.

Original: [Figure]
Revised: [Figure]

Revised: humans resnet

Revised: InsDis

Revised: MoCoV

Revised: [Figure]