R ETHINKING SUPERVISED PRE - TRAINING FOR BETTER DOWNSTREAM TRANSFERRING
Anonymous authors
Paper under double-blind review
A BSTRACT
The pretrain-ﬁnetune paradigm has shown outstanding performance on many applications of deep learning, where a model is pre-trained on a upstream large dataset ( e.g. ImageNet), and is then ﬁne-tuned to different downstream tasks.
Though for most cases, the pre-training stage is conducted based on supervised methods, recent works on self-supervised pre-training have shown powerful transferability and even outperform supervised pre-training on multiple downstream tasks.
It thus remains as an open question how to better generalize supervised pretraining model to downstream tasks.
In this paper, we argue that the worse transferability of existing supervised pre-training methods arise from the negligence of valuable intra-class semantic difference.
This is because these methods tend to push images from the same class close to each other despite of the large diversity in their visual contents, a problem to which referred as “overﬁt of upstream tasks”.
To alleviate this problem, we propose a new supervised pre-training method based on Leave-One-Out K-Nearest-Neighbor, or LOOK for short.
It relieves the problem of overﬁtting upstream tasks by only requiring each image to share its class label with most of its k nearest neighbors, thus allowing each class to exhibit a multi-mode distribution and consequentially preserving part of intra-class difference for better transferring to downstream tasks.
We developed efﬁcient implementation of the proposed method that scales well to large datasets.
Extensive empirical studies on multiple downstream tasks show that LOOK outperforms other state-of-the-art methods for supervised and self-supervised pre-training.
I NTRODUCTION
Pre-training neural networks on upstream datasets and ﬁne-tuneing the pre-trained model on downstream tasks has been an important methodology in applications of deep learning (Tan et al., 2018).
Such a pretrain-ﬁnetune paradigm generally works with pre-training on large-scale diverse datasets and ﬁne-tuning on small speciﬁc datasets, and has been widely applied in a number of applications (Devlin et al., 2019; Kolesnikov et al., 2020; Brown et al., 2020).
Speciﬁcally, in the area of computer vision, we often apply supervised learning methods ( e.g. cross entropy) to pre-train a model from labeled dataset ( e.g. ImageNet (Deng et al., 2009) and Kinetics (Carreira & Zisserman, 2017)), and ﬁne-tune it on downstream tasks such as object detection (Ren et al., 2017), instance segmentation (He et al., 2017), and video understanding (Gu et al., 2018).
Except supervised pre-training, recent works demonstrate that self-supervised pre-training without label information can also learn effective representation from upstream data and even surpass supervised methods when transferring to downstream tasks (Chen et al., 2020a;
He et al., 2020; Grill et al., 2020; Chen & He, 2021; Zbontar et al., 2021).
Unlike supervised pre-training that focuses on category-level discrimination, self-supervised pre-training is mainly based on instance discrimination, where models are trained to keep each instance and its augmentations close to each other, and at the same time, separate them from other instances and their augmentations.
It effectively captures many important and discriminative features that are useful for downstream tasks.
However, without appropriate guidance of supervision information, self-supervised pre-training lacks the ability of mining high-level semantic features and may capture detailed but irrelevant features (e.g. visual features related to special background), resulting in unsatisfying performance on challenging downstream tasks, e.g. ﬁne-grained classiﬁcation (Islam et al., 2021).
[Figure]
Taking the “football helmet” category as example, C.E. pushes all the samples into one cluster.
However, there exist potential sub-categories representing the helmet iteself and its usage in football match, respectively.
The “harmonica” class shows similar cases.
Pushing samples from the two sub-category with completely different appearance will damage of representation learning process for downstream transferring.
While our LOOK pre-trained model could adaptively separate them into different clusters, and thus preserve more valuable semantic features for better transferring.
Therefore, we aim to improve the downstream transferability for supervised pre-training.
Figureillustrates two representative supervised pre-training methods, cross entropy (C.E.) and supervised contrastive learning (SupCon, Khosla et al.
a soft-nearest neighbors loss (Goldberger et al., 2005; Wu et al., 2018a).
To distinguish instances from different classes, they are designed to minimize intra-class variance by pushing all the instances of the same class close to each or to certain centers.
For a class with diverse visual appearance (e.g. the four cat examples from Figure 2), these approaches may “ruin” the natural representation of images by bringing images with completely different visual appearances next to each other.
As a result, they tend to skip features that capture intra-class difference but are less correlated with classes deﬁned in upstream tasks, leading to the problem of overﬁtting upstream tasks.
In this paper, we propose a new supervised pre-training method based on Leave-One-Out Knearest-neighbor classiﬁcation, or LOOK for short, that effectively alleviates the problem of neglecting intra-class difference and thus signiﬁcantly improves transferability for downstream tasks.
In particular, a weighted kNN classiﬁer is used to replace the linear or MLP predictor in the last layer of deep neural network, and a leave-one-out classiﬁcation error is used as the loss function for optimization.
Because of the nature of kNN classiﬁer, each instance is only required to share the same class with most of its k nearest neighbors, allowing each class to exhibit multi-mode distribution and consequentially better preserving the features related to intra-class difference, as shown in Figure 2.
We also develop efﬁcient implementation for LOOK that scales well to large datasets.
Extensive empirical studies demonstrated that LOOK has better transferrability for downstream tasks than the existing methods for supervised and self-supervised pre-training.
R ELATED W ORKS
Pre-training models have achieved remarkable performance on multiple ﬁelds of artiﬁcial intelligence.
In computer vision, various DNN architectures (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016; Dosovitskiy et al., 2020) show signiﬁcant performance improvements on downstream tasks when pre-trained from large labeled datasets, e.g. , ImageNet (Deng et al., 2009),
JFT-300M (Sun et al., 2017).
Pre-trained model provide an essential initialization for complex vision tasks, such as detection, segmentation and etc. , which work as a important part for the convergence of the model training.
In natural language processing, pre-trained models (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020) also have made great progress.
Recently, contrastive learning has made impressive progress in the representation learning task of computer vision.
Self-supervised learning methods (Chen et al., 2020a; He et al., 2020) are em-
[Figure]
ployed to pre-train on ImageNet dataset and signiﬁcantly improve the transferability on multiple downstream tasks over supervised counterparts (Islam et al., 2021).
Derived from contrastive learning, more self-supervised learning methods (Chen et al., 2020b; Grill et al., 2020; Zbontar et al., 2021; Chen & He, 2021) also achieve impressive performance on downstream tasks.
Unlike earlier studies of contrastive learning, these methods introduce either momentum encoder or additional projector and predictor to avoid the introduction of negative pairs, signiﬁcantly improving the performance.
Apart from self-supervised learning, contrastive learning is also developed in supervised pre-training (Khosla et al., 2020; Islam et al., 2021) and further boost the transferability of the pretrained model.
In addition to self-supervised methods based on contrastive learning, there are also some unsupervised pre-training methods that focus on learning representation features by designing clever pretext tasks (Noroozi & Favaro, 2016;
Gidaris et al., 2018; Trinh et al., 2019; Bao et al., 2021) and structure formed clustering (Caron et al., 2018; Asano et al., 2020;
Li et al., 2021).
Besides designing pre-training method, there are also works (Ganin & Lempitsky, 2015;
Yosinski et al., 2014; Li et al., 2018; 2019; Chen et al., 2019a; Tan et al., 2018) focused on designing downstream training methods to improve transferablity through eliminating domain shifts between the upstream and downstream datasets or utilize an inductive bias to improve transferability.
LOOK: A L EAVE -O NE -O UT KNN BASED P RE - TRAINING M ETHOD 3. M OTIVATION OF LOO- K NN
As already discussed in the introduction, existing supervised pre-training methods suffer from the problem of overﬁtting upstream tasks, i.e. they tend to keep instances from the same class close to each other, thus neglecting intra-class difference that may be important for downstream tasks.
We believe that source of this problem arises from simple prediction models ( e.g. linear classiﬁer with fully-connected layer) are used in supervised pre-training methods.
Figure 1 shows the data distribution resulting from C.E. based pre-training. We see clearly that using a simple linear classiﬁer essentially requires all data points of the same class close to the class center.
As a remedy, we propose to replace simple prediction model with kNN classiﬁer.
The kNN classiﬁer is much more powerful than a linear prediction model.
It can ﬁt almost any decision boundary (Bishop, 2006).
Since it makes prediction without training, kNN has also been used for monitoring model’s convergence in the process of self-supervised learning (Wu et al., 2018b; Chen & He, 2021).
Compared to the linear prediction model, kNN is advantageous for pre-training because it does not require all the examples of the same class to form one tight cluster.
Given one query sample, as long as the number of samples with the same label of query takes the majority of its kNN, the label of query could be correctly predicted.
Thus, each sample only need to share the same class with most of its k nearest neighbors, and the resulting distribution can be multi-mode, instead of single-mode.
Figure 1 shows the data distributions from cross entropy (C.E.) based supervised pre-training and the proposed LOOK method, respectively.
We can see clearly that for the distribution from LOOK, data points from the same class can be distributed over multiple clusters while, for C.E., data points from the same class tend to form one cluster.
It is this nice property of kNN classiﬁer that allows us to preserve part of intra-class difference, leading to better performance for downstream tasks.
3. A LGORITHM LOOK: M ATHEMATICAL D ESCRIPTION
As mentioned before, we will use kNN for prediction model, and to facilitate training, a leave-oneout loss is used for optimization.
Below we provide detailed description.
We start by formulating the pre-training process on visual datasets.
Given a set of images X = { x i } Ni =1 , with corresponding labels Y =
{ y i } Ni =1 when training in supervised scenarios, we aim at training a encoder f θ ( · ) , e.g.
CNN, to map each image x
i into an embedding vector z i =
f θ ( x i ) in the high-dimension space.
The trained parameters generally work as the initialization to ﬁne-tune on downstream datasets.
Speciﬁcally, we use the weighted kNN for class prediction.
Given sample x i and its embedding z i = f θ ( x i ) , suppose the kNN set of z i in the representation space is N k
( z i ) , then we aggregate of its kNN labels based on the weights of pairwise distance, which is selected as the cosine distance:
[Equation]
where 1 y j is the one-hot vector in size of total classes C with value 1 only at the position y j and value 0 at the remaining positions.
Then we normalize the aggregated labels with softmax function and feed it into the negative log-likelihood function as the loss of LOOK for the i -th sample:
[Equation]
where τ is the temperature hyper-parameter of softmax to control the normalization process, and 1 y j =
y
i equals one when y j = y i and otherwise zero.
Despite the simplicity, how to make LOOK work efﬁciently for large datasets remains a challenge.
For large datasets, it is too time consuming to compute kNN for the entire dataset in an online fashion.
We will then show our efﬁcient implementation that makes LOOK scale to large datasets.
3. M AKING LOO- K NN S CALE TO LARGE DATASETS
The computational challenge of LOOK arises from two aspects.
First, since the encoder is updated in an online fashion, we can’t afford to update the representation vectors of every instance every time.
As a result, we have to handle the discrepancy between the latest updated encoder and the encoder used to map instances into vectors.
Second, due to the large data size, we can’t afford to ﬁnd the k nearest neighbors by comparing each instance to the entire dataset.
We thus have to approximate the entire dataset by a small number of proﬁles to make the computation efﬁcient.
Efﬁcient construction of search space for kNN Since it is too time consuming to ﬁnd k nearest neighbors by comparing each instance to the entire database 1 , we decide to construct a small search space S ⊂ D to approximate the entire dataset D for kNN search.
From our motivation, S should be large enough for the coverage of the whole dataset, and contain temporally synchronized embeddings that are generated from the training encoder at the same timestamp.
Taking two extreme cases as example, we could maintain a memory bank as S with same size of the dataset and continuously update embeddings of current training samples.
The main problem with this approach is that to avoid updating the embedding vectors for the entire dataset whenever the encoder is updated, we have to generate non-synchronized embedding for the memory bank, leading to signiﬁcant error in distance measurements.
In contrast, a light-weight choice is to adapt the mini-batch as S with completely synchronized embeddings from the same timestamp.
However, since the mini-batch only contains a small number of samples from the entire dataset, leading to a large error in identifying k nearest neighbors and consequentially poor classiﬁcation compared to other supervised learning methods.
To address this problem, we adapt the momentum queue mechanism from MoCo, a queue of size q updated by ﬁrst-in ﬁrst-out (FIFO) strategy, containing embeddings from a momentum encoder.
Its parameter θ m is updated as θ t +1 m = m · θ tm + (1 − m ) · θ t , where θ t is the parameter of the original encoder at time t , also known as “online encoder”, and m ∈ [0 , 1] is the momentum parameter.
The momentum encoder is updated much slower than the online encoder with larger m , e.g. 0.999, which helps us to maintain approximated synchronized embeddings in the queue with larger size.
Predictor module for faster convergence.
Although using the momentum encoder helps the training process, it still pulls embeddings from the online encoder and those from momentum encoders close to each other, leading to slow convergence.
To alleviate the convergence problem, we introduce the predictor module to alleviate the discrepancy between the online encoder and momentum encoder, similar to BOYL (Grill et al., 2020).
More speciﬁcally, an additional MLP module p ( · ) is appended after the online encoder as the predictor module, and the output p ( z i ) is used for searching kNN based on the embedding sets generated from the momentum encoder.
With such a module, we provide a buffer from the online encoder to the momentum encoder to achieve faster convergence.
Dynamic adjustments of hyper-parameters for kNN
The size k and temperature τ of LOOK decide the range of kNN label aggregation for appropriate prediction, and we observe it is important to adjust them along the training.
At the beginning, samples are randomly distributed, requiring larger range of aggregation.
When coming to the later stage of training, since multi-mode distribution has already formed, we require smaller range of aggregation to avoid pulling the multiple clusters together.
Therefore, we utilize a temperature decaying strategy that decreases τ throughout training to adjust aggregation under ﬁxed k .
Empirical studies shows similar performance of decaying of k .
Avoiding gradient explosion problem?
We observe gradient explosion at the very beginning of LOOK, which is caused by a cold-start problem that the kNN set N k ( z i ) contains no samples from the same class of z i .
To address this problem, we ﬁrst ﬁll the momentum queue without training to ensure the size of searching space.
Then we apply extreme value ﬁltering strategy for the LOOK output to be no less than a small value (cid:15) ( e.g. 1 e − 5 ) to avoid gradient explosion.
E XPERIMENTS 4. E XPERIMENTAL S ETTINGS
Datasets.
For the upstream dataset, we use the ImageNet ILSVRC (Deng et al., 2009) with 1.28M images of 1K categories since most pre-training methods for comparison are trained on ImageNet.
For the downstream datasets, we select 9 ﬁne-grained datasets from varying domains to evaluate model’s transferability inspired by Islam et al.
(2021), including the Aircraft (Maji et al., 2013),
Cars (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), Flowers (Nilsback & Zisserman, 2008), ISIC (Codella et al., 2019), Kaokore (Tian et al., 2020), Omniglot (Lake et al., 2015) and Pets (Patino et al., 2016).
Dataset statistics are summarized in the appendix.
Upstream Pre-training methods.
The compared pre-training methods are under supervised and self-supervised settings.
For supervised methods, we reproduce or adapt the cross entropy (C.E.) guided training, supervised contrastive learning (SupCon, Khosla et al.
and the examplarbased supervised learning Examplar-v2 (Zhao et al., 2020).
We also implement a SupCon+SSL version by adding additional MoCo loss to SupCon (Islam et al., 2021).
For self-supervised methods, we compare recent representative works including SimCLR (Chen et al., 2020a), MoCo-v2 (Chen et al., 2020b), BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021).
For the implementation of our proposed LOOK, we use queue size q = 65536 , momentum m = 0 .
999 and k = 100 and decaying temperature τ from 1 . 0 to 0 .
All the implemented methods are trained by 90 epochs with an initial learning rate of 0 . 1 , multiplied by 0 .
1 for every 30 epochs.
We use ResNet-50 (He et al., 2016) as the backbone encoder and train using SGD optimizer with momentum 0 .
9 and weight decay 0 . 0001 .
Since SupCon and all the self-supervised methods are trained with strong augmentation, we implement two versions of C.E. and LOOK with normal and strong data augmentation, respectively.
Downstream Fine-tuning methods.
When evaluating the pre-trained models on downstream datasets, there are various ﬁne-tuning methods to transfer models.
In the main experiments, our ﬁrst evaluations are based on two simple ﬁne-tuning strategies: linear ﬁne-tuning and fully ﬁnetuning.
The linear ﬁne-tuning ﬁxes the parameters of encoder and only trains a new classiﬁer module, while the fully ﬁne-tuning trains the whole model.
We further investigate to apply more advanced ﬁne-tuning methods from recent studies of transfer learning.
Since some of the pre-trained
[Table] [Table]
models are trained without linear classiﬁer on upstream, we follow Islam et al.
(2021) to append a batch-normalization layer without afﬁne parameters after the encoder to generate properly distributed features for classiﬁcation learning.
During the ﬁne-tuning stage, we train on the downstream datasets 50 epochs and decay the learning rate at the 25 and 37 epochs by 0 . 1 .
For the remaining hyper-parameters of training, we conduct grid search for the initial learning rate of 0 .
001 , 0 . 01 and 0 .
1 , weight decay of 0 , 1 e − 4 and 1 e − 5 , batch size of 32 and 128 , and report the downstream performance under the best hyper-parameter conﬁgurations of each method.
4. M AIN RESULTS OF DOWNSTREAM TRANSFERRING
Linear ﬁne-tuning.
The experimental results of linear ﬁne-tuning are provided in Table 1, and we calculate the mean accuracy of the 9 downstream ﬁne-grained datasets.
We observe that LOOK outperforms all the compared methods.
The existing supervised pre-training, i.e. C.E., SupCon and Examplar-v2, show worse transferability compared with self-supervised methods due to their upstream over-ﬁtting problem, where the results of SupCon without SSL are even worse for its stronger requirement of pushing all samples of the same category into one cluster.
In contrast, as a supervised pre-training method, LOOK signiﬁcantly improves the transferring results via alleviating upstream over-ﬁtting, with an improvement to C.E. with 9 . 5% accuracy.
Compared with self-supervised learning, LOOK also surpasses state of the art method, i.e. SimSiam, by 2 .
9% mean accuracy via effectively leveraging the label information.
It is also observed that though strong data augmentation boosts the self-supervised pre-training, it may introduce negative inﬂuence on supervised C.E. pre-training.
Since the encoder for extracting features is frozen in linear ﬁne-tuning, the experimental results indicate that LOOK could present more generalized representation based on pre-training, compared with existing supervised and self-supervised methods.
Fully ﬁne-tuning.
Table 2 shows the results of fully ﬁne-tuning.
It is worth noting that when coming to fully training of the entire model, the gap is reduced signiﬁcantly.
LOOK still achieves better mean accuracy compared with existing methods.
Furthermore, we do observe, for a few
[Table] [Table]
datasets, that self-supervised learning indeed outperforms the proposed approach, indicating that we may need to combine the strength of LOOK with that of instance discrimination based methods.
Advanced ﬁne-tuning methods.
We further investigate the results with advanced ﬁne-tuning methods in recent studies of transfer learning, including the BSS (Chen et al., 2019a), DELTA (Li et al., 2019) and StochNorm (Kou et al., 2020), and the naive fully ﬁne-tuning is regarded as the baseline.
Following the above methods, we also study the effect of using different sampling rate of downstream training samples based on Cars and Aircraft dataset.
The results of C.E. and LOOK are listed in Table 3.
We observe three advantages introduced by LOOK.
First, compared to C.E.,
LOOK obtains remarkable improvements on almost all scenarios.
Second, LOOK can collaborate and achieve consistency improvements with the advanced ﬁne-tuning methods.
Finally, with less training data, LOOK achieves larger gap of improvements, suggesting that generalization ability of
LOOK learned representation can reduce the reliance on the amount of downstream training data.
4. A BLATION S TUDIES
On the conﬁgurations of LOOK.
We study the inﬂuence of hyper-parameters in LOOK on the downstream performance.
From the ﬁrst and second sub-table of Table 4, we show the results of varying queue size q and momentum m of the momentum queue.
Experimental results suggest that
LOOK show great robustness to the conﬁguration of search space, which is crucial considering the evaluation of transferability is tough during the pre-training stage.
The transferring results using different k of kNN are listed in the last sub-table of Table 4, which shows that we could assign a larger k throughout the time, leaving the temperature decaying strategy to control the aggregation range of LOOK.
We also monitor the kNN accuracy along the training to investigate the inﬂuence the temperature decaying.
Results in Figure 3 (right) shows that with a proper value of τ at the beginning time helps the model to converge faster compared with a constant temperature τ .
On the backbone model.
We investigate the results of different types of encoder backbone to show the robustness of LOOK.
For convenience, we directly compare with C.E. model from the torchvision codebase (Marcel & Rodriguez, 2010).
Figure 3 (left) shows the linear ﬁne-tuning results of
C.E. and LOOK with the backbone of ResNet-50, ResNet-101, ResNet-152 and ResNeXt-50.
The
[Table]
comparison suggests that LOOK could consistently outperform C
[Table]
E. on varying encoder backbones and show robustness to its applied models as a pre-training method
[Table]
4. M EMORY - BASED FINE - TUNING WITHOUT TRAINING
In our main experiments
[Table]
all the ﬁne-tuning methods is conducted by optimizing a parametric classiﬁer module for the downstream dataset.
However, since the classiﬁcation of LOOK on upstream is conducted in a non-parametric way based on the memory of embeddings and labels, we also explore strategies to follow the same way of non-parametric classiﬁcation on downstream.
Speciﬁcally, we aim at transferring to the downstream dataset via updating the embeddings and labels in the memory with the downstream samples.
Throughout this way, we only need to forward the encoder function
ONCE on the downstream dataset without additional training process.
In practical, we ﬁrst apply a layer-normalization layer without afﬁne parameters after the encoder to directly normalize the output features.
We develop two simple but efﬁcient strategies to conduct the memory-based ﬁne-tuing.
Voting.
We measure the distance between samples from the training and validation sets of downstream dataset, and let the validation samples vote the most representative training samples as exemplars based on distance measurement.
The exemplars are then pushed into the memory with labels to predict the class of query samples.
The number of exemplars, size k and temperature τ for the prediction of LOOK are determined based on validation set.
Clustering.
For better coverage of the downstream distribution, we further propose to conduct clustering inside each category for generating better memory.
The number of clusters is also determined by grid-search based on validation set.
With the searched best conﬁguration, we conduct clustering on both the training and validation set to generate the ﬁnal clusters and push them into the memory.
The results of memory-based ﬁne-tuing are shown in Table 5, together with the linear ﬁne-tuning results of C.E. and LOOK for reference.
Though with only 1 time of forward on the downstream datasets, the proposed strategies based on LOOK pre-trained model achieve a comparable result with
C.E. linear ﬁne-tuning, which shows the generalization ability of LOOK pre-training and greatly reduces the computation cost of downstream transferring.
Our exploration of transferring without training could lay foundations for faster and more convenient transferring in future works.
[Figure]
[Table]
[Figure]
W HY LOOK WORKS FOR BETTER TRANSFERRING ?
To better understand the transferring and generalization ability of LOOK, we conduct deeper studies on its learned upstream representations.
Speciﬁcally, we observe the representation distribution of LOOK and the compared methods in two way, i.e. a qualitative observation based on feature visualization and a quantitative observation based on similarity measurements.
Feature Visualization.
Figure 4 visualizes the features of 10 random classes in the training and validation sets of ImageNet based on t-SNE (Van der Maaten & Hinton, 2008).
Compared with visualization results of C.E. and SupCon, it is observed that the LOOK learned features could form multiple clustering distribution inside the same class, which is actually our basic motivation of proposing LOOK.
With multiple clustering distribution, more available semantic information are preserved and is proved to provide more generalized features on downstream datasets.
Measurements of intra-class and inter-class distance.
To further prove the distribution characteristics of LOOK, we follow Islam et al.
(2021) to compute the intra-class and inter-class distance as a quantitative result of distribution.
Table 6 shows the averaged 1 − cosine ( · , · ) measurement between samples of the same and different classes.
C.E. and SupCon maintains smaller intra-class distance and larger inter-class distance.
Self-supervised pre
-training, MoCo-v2 and SimSiam, relax the intra-class tightness without label guidance, while their smaller inter-class distance compared with
SupCon indicate that some high-level semantic information are missing to discriminate categories.
LOOK achieves both larger intra-class and inter-class distance at the same time, which suggests that the learned representation generate clearer boundaries both inside and outside categories.
Based on the analysis of feature visualization and intra-class similarity measurement, we conclude that LOOK indeed adaptively learns multi-mode distribution inside categories and preserve more intra-class semantic features for better downstream transferring and generalization.
C ONCLUSION
In this paper, we propose a new supervised pre-training method based on Leave-One-Out k-NearestNeighbor (LOOK) classiﬁer for better downstream transferring.
Compared with self-supervised pre-training, LOOK efﬁciently leverages the label information, and at the same time alleviate the problem of upstream over-ﬁtting in existing supervised pre-training methods, which ignores intraclass difference semantic features that are valuable for transferring.
We conduct extensive experiments on a number of downstream tasks.
The experimental results show the superior performance of LOO-kNN against the SoTA supervised and self-supervised pre-training methods.
Future works may explore strategies of efﬁciently combining LOOK and self-supervised methods to train powerful models with better generalized representation for downstream transferring.
A D ATASETS
We present dataset statistics of our downstream transferring task in Table 7, which contains various types of technical, texture, satellite, natural, medical, illustrative, symbolic and natural visual contents and corresponding categories.
For the train/validation/test split of each dataset, we follow the original split for those with ofﬁcial split ﬁle, i
[Table]
e. Aircraft, DTD (the ﬁrst ofﬁcial split), Flowers and
Kaokore.
For the remaining datasets with only train/test split, we preserve the test set, and randomly split the training set into training and validation sets with the proportion of 7 : 3 inside each category.
[Table]
B D ETAILS OF PRE - TRAINING AND FINE - TUNING METHODS B. S UPERVISED PRE - TRAINING .
For the proposed LOOK, to achieve a convenient fashion of “leave-one-out” that ﬁltering out the query sample from the memory queue, we implement as follows.
Given each mini-batch as input, we compute the LOOK loss before updating the queue with current mini-batch samples.
Since the update of queue follows the “First-In-First-Out” (FIFO) strategy and queue size is signiﬁcantly smaller than the dataset size, it has been a long period since the last time that current training samples are pushed into the queue.
Thus, the mini-batch samples are very possible to be popped from the queue in earlier training iterations.
Such a strategy is convenient for avoiding additional operations to ﬁlter out each training sample individually, which achieves satisfying performance in our experiments.
For all the compared pre-training methods, we present the implementation details or download source of pre-trained models.
Cross Entropy (C.E.) .
The most commonly used C.E. pre-training using a linear or MLP classiﬁer guided by the cross entropy loss function.
We implement two version of C.E. with and without strong data augmentation.
Supervised Contrastive Learning (SupCon, Khosla et al.
SupCon is supervised version of existing self-supervised contrastive learning.
Besides the augmentation views of one training sample are regarded as “positive” samples, the other samples with the same label are also regarded as positive ones.
SupCon requires training with large batch-size or MoCo trick.
Since there is no open-source models with proper training epochs for comparison, we reproduce a 90 epochs pretrained model with batch-size of 256 based on the MoCo trick, with queue size 8 , 192 , momentum 0 .
999 and temperature 0 .
(Khosla et al., 2020), which is reported to show stronger performance the large batch-size training.
We also follow Islam et al.
(2021) to implement a SupCon+SSL that shows better transferablity, where the SSL method is selected as MoCo-v2.
Exemplar-v2 (Zhao et al., 2020), which proposed a similar strategy to improve the existing contrastive learning that incorrectly pushing samples with the same label.
We directly utilize the pretrained model from its ofﬁcial open-source codebase.
Difference with other supervised method.
In related works including Neighborhood Components
Analysis (NCA), SNCA (mini-batch version of NCA), Examplar-v2 and SupCon, each sample will be pulled togather with all samples in the dataset or memory bank with the same class.
In contrast, the proposed LOOK only pull positive samples within the kNN, which alleviates the problem of neglecting intra-class difference and thus signiﬁcantly improves transferability for downstream tasks.
The difference could be observed from the positive set in the loss function, where all the samples are involved in NCA/
SupCon and only samples within kNN are involved in LOOK.
In other words, kNN is used to ﬁlter and select postive samples for constrastive learning in LOOK.
B. S ELF - SUPERVISED PRE - TRAINING .
For MoCo-v2 and SimSiam, we download their ofﬁcial provided pre-trained model based onand 100 epochs of pre-training, respectively.
For SimCLR and BYOL, we adapt pre-trained models from an open-source codebase “OpenSelfSup” (https://github.com/open-mmlab/OpenSelfSup), which successfully reproduce the representative self-supervised learning methods.
We select the
SimCLR and BYOL pre-trained models with best ImageNet accuracy, which are trained for 200 and 300 epochs, respectively.
B. F INE - TUNING METHODS
For the linear and fully ﬁne-tuning, we conduct hyper-parameters searching based on the train and validation sets, and report the performance on test set based on optimization on the train set with best hyper-parameters.
In our main experiments, we also compare the performance of pre-trained models with advanced ﬁne-tuning methods on downstream datasets.
including BSS (Chen et al., 2019a), DELTA (Li et al., 2019) and StochNorm (Kou et al., 2020).
The implementation of these ﬁne-tuning methods are based on the open-source transfer learning codebase “Transfer-LearningLibrary” (https://github.com/thuml/Transfer-Learning-Library).
We also follow the split and subset sampling of the Aircraft and Cars datasets provided in this library.
In detail, all the parameters of the model are available for update, similar to the full ﬁne-tuning baseline.
A similar conﬁguration of linear and fully ﬁne-tuning are adapted with total epochs of 50 (decaying at epoch 25 and 37 , initial learning rate of 0 .
001 and batch size of 32 .
It is noted that the ofﬁcial split of the codebase is the train/val split, and we shows the best validation accuracy throughout training.
C A DDITIONAL A NALYSIS INSIDE LOOK DURING T RAINING C. O N THE POSITIVE SAMPLES FALLING IN K NN
During the training stage of LOOK, the optimization is based on the kNN set of each sample, including the “positive” samples from the same class and “negative” samples from different classes.
From our basic motivation, for those classes with higher intra-class variance, there should exist less positive samples inside their kNN sets, which will then be pushed into a sub-cluster of the whole class.
While for those classes with similar appearance and semantics, there will exist more positive samples leading to tighter distribution of the class.
[Figure]
yellow lady-slipper 1. echidna 0.
sombrero 0.
cricket 0.
paperknife 0.
velvet 0.
Category
Ratio
Samples
To investigate the composition of kNN, we calculate the number of samples in the memory queue for each class, and the averaged number of positive samples falling in the kNN.
The results are shown in
Figure 5.
We refer to the ratio of positive samples falling in kNN as “falling ratio”.
Our observation and conclusion for the falling ratio are as follows:
Compared among different classes , an unbalanced distribution of the falling ratio is observed, which indicates that their intra-class variances are different.
To further show the relation between the falling ratio and intra-class variance, we visualize randomly selected samples from several classes with different falling ratios in Figure 6.
The top classes with almost all positive samples falling in kNN show very similar visual appearance among each other.
While for classes with falling ratio of about 0 .
5 , we could observe some obvious sub-classes, e.g. the yellow and green crickets.
And for classes with lower falling ratio, the samples show more diversity, such as velvet used in different things (bag, dress, gloves, etc. ).
In conclusion, the falling ratio of LOOK could reﬂect different intra-class variance for learning efﬁcient representation.
Compared among different training stages , for the earlier stage from epoch 10 to 40, an obvious increasing of the falling ratio is observed, where we use hyper-parameters with larger aggregation range to pull samples into the kNN in a sparse distributed representation space.
While for the latter stage from epoch 40 to 90, the falling ratio increases much slower in spite of the continuously increasing accuracy of training (as shown in Figure 3).
Such a phenomenon indicates that based on LOOK’s relaxed restriction of kNN learning, improving the classiﬁcation performance will not force all the positive samples into the kNN.
Therefore, we could maintain the sub-clusters discovered during training.
C. O N THE RANK OF POSITIVE AND NEGATIVE SAMPLES
From the observation in Figure 5, there will also remain some negative samples in the kNN, especially for those with lower falling ratio, where some negatives rank closer to the query sample than some positives.
To better understand such cases, we illustrate one of them in Figure 7.
As we have
[Figure]
discussed in Figure 1, LOOK discovers two sub-classes for the class “football helmet” as the helmet itself and its usage in match, respectively.
For the shown query image in training, we observe some negative samples ranked before the positive one, where the positive belongs to different sub-class from the query’s and the negatives actually show more similar appearance to the query.
Based on the illustration in Figure 7, we conclude that for classes with higher intra-class variance, there will be a lower falling ratio and more negative samples of kNN.
However, such a case is not bad because these negatives with similar appearance could occupy the left space of kNN and help to avoid positives from different sub-classes falling into the center of kNN.
C. O N THE ABILITY OF DISCOVERING SUB - CLASSES
Based on analysis from the above subsections, the classes show different intra-class variance and number of potential sub-classes, and the number of positive samples maintained in the memory bank is highly related to ability of discovering sub-classes.
It is noted that the motivation of LOOK is not to directly discover more sub-classes with clear boundaries, but to preserve more semantics by avoiding pulling samples together with completely different appearance.
In spite of that, in this subsection, we take a further study to the effect of memory size on the ability of sub-class discovery, which will be conducted from two perspectives, i.e. empirical study and probability estimation.
The following study is based on the assumption that each class could be uniformly divided into several sub-classes, which only serves as a simple analysis and could be further discussed in future work.
Empirical Study.
Based on the expected number of samples for each sub-class in the queue, we could give a empirically estimated number of sub-classes that could be effectively learned based on experimental results.
We refer to a similar work, SupCon (Khosla et al., 2020), that utilizes similar contrastive learning inside large mini-batch or memory bank for classiﬁcation learning.
From the Figure 4 of SupCon, the model could reach satisfying performance on ImageNet-1K with more than 2 , 000 samples in batch, which contains 0 .
15% of all the 1 , 300 images per category.
Based on the 5% sampling rate of LOOK’s default settings, empirically, there is a potential of capturing 5%0 .
15% ≈ 33 sub-classes for the proposed method with the default queue size of 65 , 536 .
It is noted that different from parametric classiﬁcation learning that representing classes with parameters, LOOK adapts the non-parametric contrastive learning that continuously and rapidly update the memory samples throughout the dataset, which leaves sufﬁcient times for each sample to meet and interact with samples from the same class or sub-class.
Probability Estimation.
We further calculate the probability that all the sub-classes are sampled with at least one image in the queue based on the principle of tolerance and exclusion, i.e.
[Equation]
k c , where c is the number of sub-classes, q is the queue size and n is the sampling size.
Results in Table 8 show that for less than 12 sub-classes, all the training samples are almost guaranteed to ﬁnd samples in kNN with the same sub-class.
It is noted that for contrastive-based methods, even one positive sample could serve for effective training, e.g. the another augmented view in self-supervised learning.
[Table]
Number of sub-classes in real-world dataset.
Based on the above analysis, we show the ability of
LOOK under current settings to learn some sub-classes on ImageNet, which is consistent with the visualization results showing obvious sub-clusters.
A further problem is what is the actual number of sub-classes in the real-world datasets.
Since the deﬁnition of a “class” or “sub-class” is not ﬁxed based on the granularity and focused semantics, it is an open question to count all the sub-classes given a set of samples without artiﬁcial rules.
A rough estimation to the averaged number of subclasses discovered by LOOK could be the reciprocal of the averaged falling ratio, i.e.
[Table]
We further analyze the relation of LOOK’s components to the sub-classes discovery as follows:
• Granularity: The granularity of discovered sub-classes is mainly affected by two factors, i
e. the memory size and k of LOOK.
With larger memory bank storing enough samples to cover the whole class together with smaller k to strictly control the neighbors of samples, we could reach more ﬁne-grained sub-classes.
• Semantics: The rule of splitting sub-classes is based on the learned visual semantics, which is guided by the supervision of coarse labels.
Along the training stage, the semantics captured and focused by the model determine the kNN structure for sub-class discovery.
Future works related to sub-class discovery.
Since the memory size is positively related to the number of sub-classes, it is still an challenging problem how to efﬁciently utilize the memory space to model more intra-class variance.
For the future works, it is important to explore how to adaptively arrange memories for classes with more sub-classes and maintain representative samples in the memory for better coverage of all sub-classes.
D U PSTREAM V . S . D OWNSTREAM P ERFORMANCE
Though in this paper we focus on improving the downstream transferring performance of pre-trained models [Table] in this section
[Table]
we show the upstream performance for more comprehensive study of the proposed method.
In Table
[Table]
we show the upstream accuracy on ImageNet of the compared methods, together with the downstream performance for reference.
For parametric classiﬁcation methods or methods have been ﬁne-tuned with linear classiﬁer, we report the linear accuracy, while for the remaining we report the kNN accuracy ( k = 200 and temperature τ = 0 . 1 ).
[Table]
[Table]
Comparing the upstream and downstream results in Table 9, we observe that the downstream performance of one pre-training method is not highly related to its upstream performance.
As we have discussed in the motivation of LOOK, methods with better upstream performance may fall in the over-ﬁtting to the upstream datasets, leading to worse transferability on downstream tasks.
E T RANSFERRING TO DETECTION AND SEGMENTATION E. E XPERIMENTAL SETTINGS
In the main experiments, we evaluate the transferability of pre-training methods with the downstream ﬁne-grained classiﬁcation.
In this section, we further evaluate with more downstream tasks, i.e. object detection and instance segmentation, where the pre-trained models serve as the backbone to extract feature maps.
We follow MoCo (He et al., 2020) to conduct the experiments on PASCAL
VOC (Everingham et al., 2010) and COCO (Lin et al., 2014) datasets.
The details are as follows:
PASCAL VOC Object Detection.
We use the detector of Mask-RCNN with C4 backbone (He et al., 2017) and an extra Batch Normalization (BN) Layer for ﬁne-tuning.
We follow the 1 × schedule implemented in Detectron (Girshick et al., 2018), with 24k iterations decaying at 18k and 22k iterations.
The image size is [480 , 800] during training and 800 during test.
The model is trained on
VOC 2007 trainval + 2012 train, and tested on VOC 2012 val.
COCO Object Detection and Segmentation.
We use the same backbone as VOC, and follow the 1 × schedule with 9k iterations decaying at 6k and 8k iterations.
The image size is [640 , 800] during training and 800 during test.
The model is trained on COCO 2017 train and tested on 2017 val.
E. R ESULTS OF DETECTION AND SEGMENTATION
We report the results of transferring to detection and segmentation in Table 10, with the COCOstyle metric AP, AP 50 and AP 75 .
The metrics of detection and segmentation are marked as AP bb and AP mk , respectively.
We compared the proposed LOOK with supervised methods C.E. and SupCon (Khosla et al., 2020) and self-supervised methods SimCLR
(Chen et al., 2020a), BYOL (Grill et al., 2020), SwAV (Caron et al., 2020), MoCo-v2 (Chen et al., 2020b) and SimSiam (Chen & He, 2021).
We notice that the way of training with two augmented views contribute to better transferring results, due to the higher locality-sensitive (further visualization analysis is shown in the following subsection).
Thus, we develop an improved version of LOOK with two augmented views for training.
Table 10 shows that the proposed LOOK performs comparable or better compared with existing pre-training methods.
We also notice that another important factor is the epochs of the pre-training stage.
Under similar pre-training epochs, LOOK could outperform the compared methods.
In spite of the results, we observe that the transferability on locality-sensitive tasks, i.e. object detection and instance segmentation, are different from that on ﬁne-grained classiﬁcation task.
The proposed
LOOK under supervised settings is designed to capture more high-level semantic information and concentrate on the key object of the input image, thus works better for classiﬁcation task.
While the self-supervised methods will concentrate on more details of the whole image including the visual background, and works better for detection and segmentation tasks.
We present additional analysis based on visualization of the attention maps, and leave to the future work for deeper studies on improving the pre-training methods with varying transferability.
E. V ISUALIZATION OF ATTENTION MAP
For further analysis on the transferability of the compared pre-training methods, we follow DetCo (Xie et al., 2021) to visualize the attention map generated from the backbone models.
We reshape the input image as 448 × 448 for better visualization.
To compute the attention map, we calculate the average values of the output feature map in 14 × 14 along the feature channel, normalize it into [0 , 1] , reshape the map into 448 × 448 with bi-linear interpolation and project it to the original image.
We show the attention maps of three representative methods, i.e. SupCon,
the proposed
LOOK and MoCo-v2.
Since the methods are all based on contrastive learning with similar range of output values, we could fairly compare the difference of them on attention maps.
Figure 9 and 10 show the visualization results on ImageNet and COCO, respectively.
We show images of complex scene with more objects to better analyze for detection and segmentation transferring.
Compared with SupCon as supervised methods, LOOK could concentrate on more semantic details besides the core area of images for its relaxed restriction of classiﬁcation, e.g. objects that do not belong to the image labels.
While SupCon will neglect these details without direct relation to labels, e.g. persons interacting with the labeled object.
Compared with MoCo-v2 as self-supervised methods, though LOOK also concentrates on additional details in the image, the attention of MoCov2 is distributed more sparsely on the whole image including the background, which helps serve better for detection and segmentation tasks.
Based on the visualization analysis, we conclude that supervised methods mainly concentrate on the objects highly-related to the image labels, which may harm the transferability to locality-sensitive tasks.
Though LOOK could preserve more semantic details, there still leave space to both capture high-level global semantics and obvious local semantics for more powerful transferability.
F A DDITIONAL ABLATION STUDIES
In the ablation study of Section 4.3, we show the results of varying queue size, momentum and number of neighbors.
In this section, we further show the effect of the hyper-parameter temperature τ to LOOK.
We maintain the decaying factor of temperature as 0 . 1 and set different beginning value of τ .
The results are shown in Figure 8.
It is observed that LOOK shows good robustness to the varying temperature, which attributes to that the direct ﬁltering way of kNN provides a more stable positive sample set for learning.