W HAT MAKES INSTANCE DISCRIMINATION GOOD FOR TRANSFER LEARNING ?
com projects
_transfer
A BSTRACT
Contrastive visual pretraining based on the instance discrimination pretext task has made signiﬁcant progress.
Notably, recent work on unsupervised pretraining has shown to surpass the supervised counterpart for ﬁnetuning downstream applications such as object detection and segmentation.
It comes as a surprise that image annotations would be better left unused for transfer learning.
In this work, we investigate the following problems: What makes instance discrimination pretraining good for transfer learning?
What knowledge is actually learned and transferred from these models?
From this understanding of instance discrimination, how can we better exploit human annotation labels for pretraining?
Our ﬁndings are threefold.
First, what truly matters for the transfer is low-level and mid-level representations, not high-level representations.
Second, the intra-category invariance enforced by the traditional supervised model weakens transferability by increasing task misalignment.
Finally, supervised pretraining can be strengthened by following an exemplar-based approach without explicit constraints among the instances within the same category
I NTRODUCTION
Recently, a remarkable transfer learning result with unsupervised pretraining was reported on visual recognition.
The pretraining method MoCo (He et al., 2020) established a milestone by outperforming the supervised counterpart, with an AP of 46 . 6 compared to 42 .
4 on PASCAL VOC object detection.
Supervised pretraining has been the de facto standard for ﬁnetuning downstream applications, and it is surprising that labels of one million images, which took years to collect (Deng et al., 2009), appear to be unhelpful and perhaps even harmful for transfer learning.
This raises the question of why contrastive pretraining provides better transfer performance and supervised pretraining falls short.
The leading contrastive pretraining methods follow an instance discrimination pretext task (Dosovitskiy et al., 2015; Wu et al., 2018; He et al., 2020; Chen et al., 2020a), where the features of each instance are pulled away from those of all other instances in the training set.
Invariances are encoded from low-level image transformations such as cropping, scaling and color jittering.
With such low-level induced invariances (Wu et al., 2018; Chen et al., 2020a), strong generalization has been achieved to high-level visual concepts such as object categories on ImageNet.
On the other hand, the widely adopted supervised pretraining method optimizes the cross-entropy loss over the predictions and the labels.
As a result, training instances within the same category are drawn closer while the training instances of different categories are pulled apart.
Toward a deeper understanding of why contrastive pretraining by instance discrimination performs so well, we dissect the performance of both contrastive and supervised methods on a few downstream tasks.
Our study begins by studying the effects of pretraining image augmentations, which are shown to be crucial for contrastive learning.
We ﬁnd that both contrastive and supervised pretraining beneﬁt from image augmentations for transfer performance, while contrastive models rely on these low-level augmentations signiﬁcantly.
With proper augmentations, supervised pretraining may still prevail on the downstream task of object detection on COCO and semantic segmentation on Cityscapes.
We then examine the common belief that the high-level semantic information is the key to effective transfer learning (Girshick et al., 2014; Long et al., 2015).
On unsupervised pretraining with different types of image sets, it is found that transfer performance is largely unaffected by the high-level semantic content of the pretraining data, whether it matches the semantics of the target data or not.
Moreover, pretraining on synthetic data, whose low-level properties are inconsistent with real images, leads to a drop in transfer performance.
These results indicate that it is primarily low-level and mid-level representations that are transferred.
Additionally, we notice that unsupervised pretraining on a much smaller dataset only marginally degrades the transfer performance.
We also delve deeply to understand the large margin (with AP 48.5 over 46.2) on VOC object detection obtained by contrastive pretraining over supervised pretraining.
First, detection errors of both methods are diagnosed using the detection toolbox (Hoiem et al., 2012).
It is found that supervised pretraining is more susceptible than contrastive pretraining to localization error.
Secondly, to understand the localization error, we examine how effectively images can be reconstructed from contrastive and supervised representations.
The results show that supervised representations mainly model the discriminative parts of objects, in contrast to the more holistic modeling of contrastive representations pretrained to discriminate instances rather than classes.
Both sets of experiments suggest that there exists a greater misalignment of supervised pretraining to the downstream tasks, which requires accurate localization and full delineation of the objects.
Based on these studies, we conclude that, in visual pretraining, not only it is less critical to transfer high-level semantic information, but learning to discriminate among classes might be misaligned with the downstream tasks.
We thus hypothesize that the essential difference that makes supervised pretraining weaker (and instance discrimination stronger) is the common practice of minimizing intra-class variation.
The crude assumption that all instances within one category should be alike in the feature space neglects the unique information from each instance that may have signiﬁcance in downstream applications.
To validate that overﬁtting semantics leads to weakened transferability, we explore a new supervised pretraining method that does not explicitly embed instances of the same class in close proximity of one another.
Rather, we pull away the true negatives of each training instance without enforcing any constraint on the positives.
This respects the data distribution in a manner that preserves the variations in the positives, and our new pretraining method is shown to yield consistent improvements for both ImageNet classiﬁcation and downstream transfer tasks.
We expect these ﬁndings to have broad implications over a variety of transfer learning applications.
As long as there exists any misalignment between the pretraining and downstream tasks (which is true for most transfer learning scenarios in computer vision), one should always be careful about overﬁtting to the supervised invariances deﬁned by the pretraining labels.
We further test on two other transfer learning scenarios: few-shot image recognition and facial landmark prediction.
Both of them are found to align with the conclusions obtained from our previous study.
A N A NALYSIS FOR V ISUAL T RANSFER L EARNING
We study the transfer performance of pretrained models for a set of downstream tasks: object detection on PASCAL VOC07, object detection and instance segmentation on MSCOCO, and semantic segmentation on Cityscapes.
Given a pretrained network, we re-purpose the network architecture, and ﬁnetune all layers in the network with synchronized batch normalization.
For object detection on PASCAL VOC07, we use the ResNet50-C4 architecture in the Faster R-CNN framework (Ren et al., 2015).
Optimization takes 9k iterations on 8 GPUs with a batch size ofimages per GPU.
The learning rate is initialized to 0.02 and decayed to be 10 times smaller after 6k and 8k iterations.
For object detection and instance segmentation on MSCOCO, we use the
ResNet50-C4 architecture in the Mask R-CNN framework (He et al., 2017).
Optimization takes 90k iterations on 8 GPUs with a batch size of 2 images per GPU.
The learning rate is initialized to 0.
and decayed to be 10 times smaller after 60k and 80k iterations as of the 1x optimization setting.
For semantic segmentation on Cityscapes, we use the DeepLab-v3 architecture (Chen et al., 2017) with image crops of 512 by 1024 .
Optimization takes 40k iterations on 4 GPUs with a batch size ofimages per GPU.
The learning rate is initialized to 0.01 and decayed with a poly schedule.
Detection performance is measured by averaged precision (AP) and semantic segmentation performance is measured by mean intersection over union (mIoU).
Each pretrained model is also evaluated by
ImageNet classiﬁcation of linear readoff on the last layer features.
[Table]
E FFECTS OF I MAGE A UGMENTATIONS ON P RETRAINING
Contrastive learning is shown to depend on intensive image augmentations (views) for ImageNet classiﬁcation.
However, the effects of such image augmentations have not been carefully investigated for the downstream transfer tasks.
In this section, we provide a detailed analysis on the effects of image augmentations for contrastive models, and supervised models as well.
We use MoCo-v2 (Chen et al., 2020b) for contrastive pretraining and the traditional cross entropy loss for supervised pretraining with the ResNet50 architecture.
Both types of pretraining are optimized for 200 epochs with a cosine learning rate decay schedule for fair comparisons.
The results are summarized in Table 1.
First, unsupervised contrastive models consistently beneﬁt much more from image augmentations than supervised models for ImageNet classiﬁcation and all the transfer tasks.
Supervised models trained merely with horizontal ﬂipping may perform well.
Second, supervised pretraining is found to also beneﬁt from image augmentations such as color jittering and random grayscaling, but it is negatively affected by Gaussian blurring to a small degree.
Third, unsupervised models outperform the supervised counterparts on PASCAL VOC, but underperform the supervised models on COCO and Cityscapes when proper image augmentations are applied on supervised models.
This suggests that object detection on COCO and semantic segmentation on Cityscapes may beneﬁt more from high-level information than PASCAL VOC.
2. E FFECTS OF D ATASET S EMANTICS ON P RETRAINING
The strong performance of self-supervised pretraining on the linear classiﬁcation protocol for ImageNet (He et al., 2020; Chen et al., 2020a) shows that the features capture high-level semantic representations of object categories.
In supervised pretraining, it is a common belief that this highlevel representation (Girshick et al., 2014; Long et al., 2015) is what transfers from ImageNet to the downstream tasks.
Here, we challenge this conclusion by studying the transfer of contrastive models pretrained on images with little or no semantic overlap with the target dataset.
These image datasets include faces, scenes, and synthetic street-view images.
We also investigate how the size of the pretraining dataset affects transfer performance.
Consistently with the prior section, we use MoCo-v2 for contrastive pretraining and dedicated training pipeline for the supervised models on various datasets.
Please refer to the Table 2 captions for detailed information about the annotation labels in each dataset.
For the smaller datasets, we increase the number of training epochs and maintain the effective number of optimization iterations.
The results are summarized in Table 2.
First, it can be seen that, except for the results on the synthetic dataset
Synthia, the transfer learning performance of contrastive pretraining is relatively unaffected by the pretraining image data, while supervised pretraining depends on the supervised semantics.
Second, all the supervised networks are negatively impacted by the change of pretraining data except when the pretraining data has the same form of supervision as the target task, such as the Synthia and
[Table]
the Cityscapes datasets both sharing pixel-level annotations for semantic segmentation.
Third, with the smaller amounts of pretraining data in ImageNet-10% and ImageNet-100, the advantage of unsupervised pretraining becomes more pronounced in relation to supervised models, which suggests stronger ability for generalization with less data for contrastive models.
Contrastive pretraining on faces and scenes achieves almost the same transfer results as pretraining on ImageNet.
Since the face dataset has almost no semantic overlap with the VOC and COCO objects (besides the human category), transfer of high-level representations can be seen as extraneous.
We further test unsupervised pretraining on the synthetic dataset Synthia
(Ros et al., 2016), which exhibits low-level statistics different from real images.
With this model, there is a substantial performance drop.
We can therefore conclude that instance discrimination pretraining mainly transfers low-level and mid-level representations.
In Table 2, we also test the linear readoff on ImageNet-1K classiﬁcation using various pretrained models.
These models perform very differently, suggesting that the last-layer features learned by contrastive training still overﬁt to the training data semantics.
2. T ASK M ISALIGNMENT AND I NFORMATION L OSS
A strong high-level representation is not critical for effective transfer, but this itself does not explain why contrastive pretraining yields better performance than supervised pretraining, speciﬁcally for object detection on PASCAL VOC.
We notice that a larger performance gap exists on AP 75 than on
AP 50 2 , which suggests that supervised pretraining is weaker at precise localization.
For additional analysis, we use the detection toolbox (Hoiem et al., 2012) to diagnose detection errors.
Figurecompares the error distributions of the transfer results on three example categories.
We ﬁnd that the detection errors of supervised pretraining models are more frequently the result of poor localization, where low IoU bounding boxes are mistakenly taken as true positives.
For further examination, we compare image reconstruction from the features of supervised and contrastive models.
This reconstruction is performed by inverting the layer4 features (dimension of 7 × 7 × 2048 ) using the deep image prior (Ulyanov et al., 2018).
Speciﬁcally, given an image input x 0 , we optimize a reconstruction network r θ to produce a reconstruction x that is close to the input
[Figure]
[Figure]
the embedding space of a pretrained encoding network f ( · )
[Equation]
The input z 0 to the reconstruction network r θ ( · ) is ﬁxed spatial noise, and the distance function E ( · ) is implemented as the L 2 distance.
The architecture of r θ ( · ) is an autoencoder network with six blocks for both the encoder and decoder, as detailed in the appendix.
With this inversion method, we observe how well a pretrained network g ( · ) can recover image pixels from the features.
We visualize the reconstructions for both the supervised and contrastive pretrained networks.
The investigated contrastive models include InstDisc, MoCo and SimCLR to show the generality of the results.
In Figure 2, it is apparent that the contrastive network provides more complete reconstructions spatially, while the supervised network loses information over large regions in the images, likely because its features are mainly attuned to the most discriminative object parts, which are central to the classiﬁcation task, rather than objects and images as a whole.
The resulting loss of information may prevent the supervised network from detecting the full envelope of the object.
In Figure 2, we notice that for the contrastive models, the images are reconstructed at the correct scale and location.
Though instance discrimination encodes invariances through spatial and scale transformations, features learned this way are still sensitive to these factors (Chen et al., 2020a).
A possible explanation is that in order to make one instance unique from all other instances, the network strives to preserve as much information as possible.
We also notice that the contrastive models ﬁnd it difﬁcult to reconstruct the hue color accurately.
This is likely due to the broad space of colors and the intensive augmentations during pretraining.
[Table]
A B ETTER S UPERVISED P RETRAINING M ETHOD
Annotating one million images in ImageNet provides rich semantic information which could be useful for downstream applications.
However, traditional supervised learning minimizes intra-class variation by optimizing the cross-entropy loss between predictions and labels.
By doing so, it focuses on the discriminative regions (Singh & Lee, 2017) within a category but at the cost of information loss in other regions.
A better supervised pretraining method should instead pull away features of the true negatives for each instance without enforcing explicit constraints on the positives.
This preserves the unique information of each positive instance while utilizing the label information in a weak manner.
We propose a new supervised pretraining method inspired by exemplar SVM (Malisiewicz et al., 2011)
[Table]
which trains an individual SVM classiﬁer to separate each instance from its negatives.
Unlike the original exemplar SVM which represents positives non-parametrically and negatives parametrically, our pretraining scheme models all instances in an non-parametric fashion in a spirit similar to instance discrimination (Wu et al., 2018).
Concretely, we follow the framework of momentum contrast (He et al., 2020), where each training instance x i is augmented twice to form x qi and x ki , which are fed into two encoders for embedding, q i = f q ( x qi ) , k i =
f k ( x ki ) .
Please refer to MoCo (He et al., 2020) for details about the momentum encoders.
But instead of discriminating from all other instances (Wu et al., 2018), the loss function uses the labels
y i to ﬁlter the true negatives,
[Equation]
where τ is the temperature parameter.
We set the baselines to be MoCo-v1 and MoCo-v2, and denote our corresponding methods as Exemplar-v1 and Exemplar-v2.
Temperature τ = 0 .
07 is used for
Exemplar-v1, and τ = 0 .
1 is used for Exemplar-v2 with ablations in the appendix.
Experimental results are presented in Table 3.
By ﬁltering true negatives using semantic labels, our method consistently improves classiﬁcation performance on ImageNet and transfer performance for all downstream tasks.
This is in contrast to traditional supervised learning, where ImageNet performance is improved and its transfer performance is compromised.
We note that the ImageNet classiﬁcation performance of our exemplar-based training, 68 . 9% , is still far from the traditional supervised learning result of 77 .
This leaves room for future research on even better classiﬁcation and transfer learning performance.
I MPLICATIONS FOR O THER T RANSFER L EARNING S CENARIOS
The presented studies focus on the transfer scenario of ImageNet pretraining.
For other downstream applications, the nature of the task misalignment may differ.
Thus, we additionally consider two other transfer learning scenarios to study the implications of overﬁtting to the supervised pretraining semantics and how it can be improved by our exemplar-based pretraining.
4. F EW - SHOT R ECOGNITION
The ﬁrst scenario is transfer learning for few-shot image recognition on the Mini-ImageNet dataset (Vinyals et al., 2016), where the pretext task is image recognition on the base 64 classes, and the downstream task is image recognition on ﬁve novel classes with few labeled images per class, either 1-shot or 5-shot.
For the base classes, we split their data into training and validation sets to evaluate base task performance.
The experimental setting largely follows a recent work (Chen et al.,
[Table] [Table]
for transfer learning.
The pretrained network learned from the base classes is ﬁxed, and a linear classiﬁer is ﬁnetuned for 100 rounds on the output features for the novel classes.
As in our previous transfer learning study, we compare three pretraining methods: supervised cross entropy, unsupervised MoCo-v2 and supervised Exemplar-v2.
Each method is trained with MoCo-v augmentations and optimized for 2000 epochs with a cosine learning rate decay scheduler for fair comparison.
In ﬁnetuning the downstream task, since there exists much variance in the feature norms from different pretrained networks, we cross-validate the best learning rate for each method on the validation classes.
Note that adding an additional batch normalization layer is problematic because as few as only 5 images (5-way 1-shot case) are available during ﬁnetuning.
We use the backbone network of ResNet18 (Chen et al., 2019) for the experiments.
Results are shown in Table 4.
Due to different optimizers and number of training epochs, our supervised pretraining protocol is stronger than the baseline protocol (Chen et al., 2019), leading to better results.
The unsupervised pretraining method MoCo-v2 is weaker for both the base classes and the novel classes, suggesting that the pretraining task and the downstream task are well aligned semantically.
Our exemplar-based approach obtains improvements over MoCo-v2 on the base classes, while outperforming the supervised baselines on the novel classes.
This demonstrates that removing the explicit constraints on intra-class instances generalizes the model for better transfer learning on few-shot recognition.
4. F ACIAL L ANDMARK P REDICTION
We next consider the transfer learning scenario from face identiﬁcation to facial landmark prediction on CelebA (Liu et al., 2018) and MAFL (Zhang et al., 2014).
The pretext task is face identiﬁcation on CelebA, and the downstream task is to predict ﬁve facial landmarks on the MAFL dataset.
The facial landmark prediction is evaluated by the average euclidean distance of landmarks normalized by the inter-ocular distance.
As in the prior studies, we compare three pretraining methods: supervised cross entropy, unsupervised
MoCo-v2 and supervised Exemplar-v2.
Each method is trained with MoCo-v2 augmentations and optimized for 1400 epochs with a cosine learning rate decay scheduler.
For landmark transfer, we ﬁnetune a two-layer network that maps the spatial output of ResNet50 features to landmark coordinates.
The two-layer network contains a 1 × 1 convolutional layer that reduces 2048 channels to 128 and a fully connected layer, interleaved with LeakyReLU and batch normalization layers.
We ﬁnetune all layers end-to-end for 200 epochs with a learning rate of 0 .
02 and a batch size of 128.
The experimental results are summarized in Table 5.
Unsupervised pretraining by MoCo-v2 outperforms the supervised counterpart for this transfer, suggesting that the task misalignment between face identiﬁcation and landmark prediction is large.
In other words, faces corresponding to the same identity hardly reveal information about their poses.
Our proposed exemplar-based pretraining approach weakens the inﬂuence of the pretraining semantics, leading to results that maintain the transfer performance of MoCo-v2.
Qualitative results are displayed in Figure 3.
R ELATED W ORKS
Since the marriage of the ImageNet dataset (Deng et al., 2009) and deep neural networks (Krizhevsky et al., 2012), supervised ImageNet pretraining has proven to learn generic representations that facilitate a variety of applications such as high-level detection (Girshick et al., 2014; Sermanet et al., 2013) and segmentation (Long et al., 2015), low-level texture synthesis (Gatys et al., 2015), and style transfer (Gatys et al., 2016).
ImageNet pretraining also works amazingly well under a large domain gap for medical imaging (Mormont et al., 2018) and depth estimation (Liu et al., 2015).
The good transferability of ImageNet pretrained networks has been extensively studied (Agrawal et al., 2014;
Azizpour et al., 2015).
The transferability across each neural network layer has also been quantiﬁed for image classiﬁcation (Yosinski et al., 2014), and a reduction of dataset size was found to have only a modest effect on transfer learning using AlexNet (Huh et al., 2016).
In addition, a correlation between ImageNet classiﬁcation accuracy and transfer performance has been reported (Kornblith et al., 2019), and the beneﬁt of ImageNet pretraining has been shown to become marginal when the target task has a sufﬁcient amount of data (He et al., 2019).
Beyond ImageNet transfer, there has been an effort to discover the structures and relations among tasks for general transfer learning (Silver & Bennett, 2008; Silver et al., 2013).
Taskonomy (Zamir et al., 2018) builds a relation graph over 22 visual tasks and systematically studies the task similarities.
In (Standley et al., 2019), task cooperation and task competition are quantitatively measured to improve transfer learning.
Similar phenomena are observed that task misalignment may lead to negative transfer (Wang et al., 2019) and that the number of layers that tasks may share depends on the task similarity (Vandenhende et al., 2019).
Other works (Dwivedi & Roig, 2019; Tran et al., 2019) explore alternative methods to measure the structure and similarity between tasks.
While most works study transfer learning based on supervised pretraining, our work focuses on analyzing transfer learning based on unsupervised pretraining, particularly on contrastive learning with the instance discrimination task (Dosovitskiy et al., 2015; Wu et al., 2018;
He et al., 2020;
Chen et al., 2020a).
Over the years, the research community has achieved signiﬁcant progress on self-supervised learning (Doersch et al., 2015; Doersch & Zisserman, 2017; Zhang et al., 2016;
Gidaris et al., 2018; 2020) and contrastive learning (Oord et al., 2018; Zhuang et al., 2019; Tian et al., 2019; Hénaff et al., 2019; Zhao et al., 2020), closing the gap with supervised learning for
ImageNet classiﬁcation.
More recent works (Goyal et al., 2019; Kolesnikov et al., 2019; Caron et al., 2019) make the attempts to scale unsupervised learning to uncurated data beyond ImageNet.
Along with this, several research on contrastive learning (He et al., 2020; Misra & Maaten, 2020) report superior transfer results against the supervised counterparts on downstream tasks such as detection, segmentation and pose estimation.
However, little is understood about why contrastive pretraining leads to improved transfer learning performance.
Our work is the ﬁrst to shed light on this, and it uses this understanding to elevate the performance of supervised pretraining.
C ONCLUSION
This work provides an analysis for visual transfer learning to understand the recent advances of contrastive learning with the instance discrimination pretext task.
Through this understanding, we also explore ways to better exploit the annotation labels for pretraining.
Our main ﬁndings that help to understand the transfer are as follows:
• When ﬁnetuning the network over all layers end-to-end with thousands of data, it is mainly low- and mid-level representations that are transferred, not high-level representations.
This suggests that contrastive representations learned on various datasets share a low- and midlevel representation which performs similarly and adapts quickly towards the target problem.
• The output features from contrastive models are not agnostic to the pretrained datasets, as they are still overﬁt to the high-level semantics of the dataset being trained on.
• Contrastive models learned by the instance discrimination pretext task contain rich information for reconstructing pixels from the output features.
In order to discriminate among all instances, the network appears to learn a holistic encoding over the entire image.
• For supervised pretraining, the intra-class invariance encourages the network to focus on discriminative patterns and disregards patterns uninformative for classiﬁcation.
This may lose information which could be useful for the target task when there is task misalignment.
An exemplar-SVM style pretraining scheme based on the instance discrimination framework is shown to improve generalization for downstream applications.
We expect these ﬁndings to have broad implications on other transfers.
Our experiments on two other transfer scenarios conﬁrm the generalization ability of the study.
We hope that the presented studies provide insights that inspire better pretraining methods for transfer learning
A E FFECTS OF P RETRAINING AND F INETUNING I TERATIONS
We also conduct experiments to examine the effects of pretraining optimization epochs and ﬁnetuning iterations.
We show results in Figure 4, and ﬁnd that longer optimization during pretraining consistently improves detection transfer for both supervised and unsupervised models.
This suggests that overﬁtting is not an issue for either pretraining method.
Unsupervised pretraining is seen to converge much faster during pretraining, and supervised pretrained models tend to converge faster in the initial iterations of detection ﬁnetuning but may not converge optimally.
We notice that supervised pretraining beneﬁts from more optimization epochs.
To explore the limit of supervised pretraining, we investigate larger numbers of supervised pretraining epochs.
In Table 6, supervised pretraining continues to improve performance until 800 epochs, but may suffer from overﬁtting as indicated by the performance on ImageNet classiﬁcation.
For detection transfer, the improved supervised pretraining still falls short MoCo on AP and AP 75 , while it outperforms MoCo on AP 50 .
This may possibly be due to the superior semantic classiﬁcation ability of supervised models.
Further discussion of the results are beyond the scope of the paper.
[Figure]
[Table]
B E FFECTS OF I MAGE A UGMENTATIONS ON P RETRAINING
We show full results of object detection on PASCAL VOC07, object detection and instance segmentation on MSCOCO, and semantic segmentation on Cityscapes in Table 7.
[Table]
[Table] [Table]
[Table]
[Table] [Table]
C E FFECTS OF D ATASET S EMANTICS ON P RETRAINING
We report full transfer performance with pretraining on various datasets in Table 8.
We also provide a visualization of various datasets for training these models in Figure 5.
D D ETAILS ON I MAGE R ECONSTRUCTION BY I NVERTING F EATURES D. M ETHOD D ETAILS
We use the same architecture for the reconstruction network r θ ( · ) as in the original deep image prior paper.
It is an encoder-decoder network with the following architecture.
Let C mk denote a Convolution-BatchNorm-LeakyReLU layer with k channels and m × m spatial ﬁlters; CD mk denote a Convolution-Downsample-BatchNorm-LeakyReLU layer, and CU mk denote a ConvolutionBatchNorm-LeakyReLU-Upsample layer.
We use a stride of 2 for both the upsampling and downsampling layers.
Encoder: CD 716 − C 716 − CD 732 − C 732 − CD 564 − C 564 − CD 5128 − C 5128 − CD 3128 − C 3128 − CD 3128 − C
Decoder: C 716 − CU 716 − C 732 − CU 732 − C 564 − CU 564 − C 5128 − CU 5128 − C 3128 − CU 3128 − C 3128 − CU
The input z
0 ∈ R H
× W × 32 is initialized with uniform noise between 0 and 0.1.
For each image, the optimization takes 3000 iterations of an Adam optimizer with a learning rate of 0.001.
D. E VALUATING R ECONSTRUCTIONS BY P ERCEPTUAL M ETRICS
To measure the reconstruction quality quantitatively, we calculate the perceptual distance between the reconstruction and the input image, using a deep learning based approach (Zhang et al.
with a SqueezeNet network.
We randomly select one image per class from the ImageNet validation
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Figure]
set for 1000 images in total.
The average distance of reconstructions using MoCo is 5 .
59 , while it is 6 .
43 for the supervised network.
We provide a scatter plot of perceptual distance from individual reconstructions.
In Figure 6, we can see that the reconstructions generated by MoCo are generally closer to the original images than those generated by the supervised method.
[Figure]
E M ORE R ESULTS ON E XEMPLAR - BASED SUPERVISED PRETRAINING
We show full transfer performance of our proposed Exemplar-based supervised pretraining in Table 9.
Since our Exemplar pretraining uses a different set of parameters from MoCo, we provide an ablation study over the parameter k and τ for ImageNet linear readout in Table 10.
[Table]
[Table]
F A DDITIONAL R ESULTS OF D IAGNOSING D ETECTION E RROR
We provide a full analysis over 20 object categories on the VOC07 test set.
For each category, a pie chart is given to show the distribution of four kinds of errors in top-ranked false positives.
For each category, the false positives are chosen to be within the top N detections, where N is chosen to be the number of ground truth objects in each category.
The four types of false positives include: poor localization (Loc), confusion with similar objects (Sim), confusion with other VOC objects (Oth), or confusion with background or unlabeled objects (BG).
In Figure 7, we compare the error distribution between the MoCo results and supervised results.
It is apparent that detection results from the MoCo pretrained model exhibits a smaller proportion of localization errors.
[Table]
S up e r v
i s e d aeroplane
[Table]
bicycle
Loc: 73%
Sim: 9% Oth: 6% BG: 13% boat
Loc: 64% Sim: 1% Oth: 2%
BG: 33% bus
Loc: 46%
Sim: 19%
Oth: 10%
BG: 25% car
Loc: 45%
Sim: 6% Oth: 4%
BG: 45% motorbike
Loc: 74%
Sim: 12% Oth: 6% BG: 8% train
Loc: 72%
Sim: 8% Oth: 3%
BG: 18% bird
Loc: 55%
Sim: 7% Oth: 2%
BG: 36% cat
Loc: 67%
Sim: 28%
Oth: 1% BG: 4% cow
Loc: 36%
Sim: 51%
Oth: 1% BG: 12% dog
Loc: 73%
Sim: 22% Oth: 1% BG: 4% horse
Loc: 56%
Sim: 39%
Oth: 3% BG: 2% sheep
Loc: 40%
Sim: 31%
BG: 29% bottle
Loc: 30%
Oth: 8%
BG: 63% chair
Loc: 43%
Sim: 10%Oth: 9%
BG: 38% diningtable
Loc: 37%
Sim: 23%
Oth: 24%
BG: 16% pottedplant
Loc: 44%
Oth: 8%
BG: 48% sofa
Loc: 34%
Sim: 26%
Oth: 28%
BG: 12% tvmonitor
Loc: 25%
Oth: 9%
BG: 66% person
Loc: [Figure] [Figure] [Figure]
[Figure]
[Figure]
BG: 35%