T HE V ISUAL T ASK A DAPTATION B ENCHMARK
Anonymous authors
Paper under double-blind review
A BSTRACT
Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets.
Yet, the absence of a uniﬁed yardstick to evaluate general visual representations hinders progress.
Many sub-ﬁelds promise representations, but each has different evaluation protocols that are either too constrained (linear classiﬁcation), limited in scope (ImageNet, CIFAR,
Pascal-VOC), or only loosely related to representation quality (generation).
We present the Visual Task Adaptation Benchmark (VTAB): a diverse, realistic, and challenging benchmark to evaluate representations.
VTAB embodies one principle: good representations adapt to unseen tasks with few examples .
We run a large
VTAB study of popular algorithms, answering questions like: How effective are
ImageNet representation on non-standard datasets?
Are generative models competitive?
Is self-supervision useful if one already has labels?
I NTRODUCTION
Deep learning has revolutionized computer vision.
Distributed representations learned from raw pixels have enabled unprecedented performance on many visual understanding tasks.
Hand-crafted features have been replaced with hand-annotated datasets, with thousands to millions of examples (Krizhevsky, 2009; Russakovsky et al., 2015).
By contrast, humans learn a wide range vision tasks using just a few examples per task.
A key research challenge is to close this gap in sample efﬁciency, and unlock deep learning for the long tail of problems without many labels.
The goal of improving sample efﬁciency has been approached from many angles, such as: few-shot learning (Fei-Fei et al., 2006), transfer learning (Pan & Yang, 2009), domain adaptation (Wang &
Deng, 2018), and representation learning (Bengio et al., 2013).
Representation learning is studied in many contexts: supervised pre-training
Sharif Razavian et al., 2014), self-supervised learning (Doersch et al., 2015), semi-supervised learning (Chapelle et al., 2009), generative modeling (Donahue et al., 2017), and disentanglement learning (Higgins et al., 2017).
Yet many questions remain unanswered, such as: Do representations trained using ImageNet labels generalize to diverse tasks?
Does the great progress on generative modeling translate in better general representations?
Is selfsupervision useful when labelled datasets already exist?
Are semantic image-level labels useful for tasks that require understanding the structure of the world?
The lack of a common benchmark impedes the understanding of visual representations; each subdomain has its own evaluation protocol, and it is hard to assess different approaches.
Benchmarks have been critical in other domains, such as RL (Mnih et al., 2013), natural image classiﬁcation (Russakovsky et al., 2015), and NLP (Wang et al., 2018).
Inspired by these successes, we propose a benchmark that follows similar principles: (i) we put minimal constraints to encourage creativity, (ii) focus on practical considerations, and (iii) make it challenging.
We present the Visual Task Adaptation Benchmark (VTAB).
VTAB is based on a single principle: a better algorithm is one that solves a diverse set previously unseen tasks with fewest possible labels .
The focus on sample complexity reﬂects our belief that learning with few labels is the key objective of representation learning.
Task diversity is also crucial to assess a representation’s generality.
Therefore, our benchmark extends beyond standard natural classiﬁcation tasks, and includes those related to sensorimotor control, medical imaging, and visual reasoning.
We treat the benchmark tasks as unseen , but otherwise, we do not constrain the representation learner, or transfer algorithm.
Pre-training may be performed on any data (labelled or unlabelled), provided that it does not overlap with the benchmark tasks, which would violate the notion of unseen .
To transfer the representations one may freeze, ﬁne-tune, add new layers, apply domain adaptation, meta-learning, or any other technique.
With VTAB, we perform an extensive representation study, which reveals: (i) Supervised pretraining on ImageNet yields useful representations for natural images.
Surprisingly, these work on distant domains, but to a lesser extent.
(ii) Self-supervision is effective, although other unsupervised methods, such as generative models appear less so.
(iii)
Combining self-supervised with supervision outperforms all other studied algorithms, even without additional unlabelled data.
(iv) Restricted evaluation using a linear classiﬁer could lead to different conclusions.
Data, code, continuously updated results, and models used are made available at [anonymous-url] .
T HE V ISUAL T ASK A DAPTATION B ENCHMARK
[Figure]
We seek algorithms that perform well on a wide variety of unseen visual understanding tasks with few labels per task.
We ﬁrst formalize this objective and then specify a practical benchmarking procedure to measure progress.
A dataset D n is a set of n instances { ( x i , y i ) } ni =1 with observations x i ∈ X and labels y i ∈ Y .
A prediction function is any mapping F : X → Y (e.g. a classiﬁer).
A (learning) algorithm , A , takes as input a dataset and outputs a prediction function.
For example,
A may be a pre-trained neural network coupled with a training mechanism.
An evaluation procedure , E T , takes F and outputs a scalar measuring F ’s performance (e.g. test-set accuracy).
We seek the algorithm that maximizes the expected performance over a distribution of tasks P T , where a task T is a tuple containing a task-speciﬁc dataset distribution D T and evaluation procedure.
Given access to only n samples per task we aim to maximize:
[Equation]
This general formulation requires a few clariﬁcations.
First, the task distribution needs to be appropriate; we desire a spectrum of tasks covering those solvable by a vision algorithm with human-like capabilities.
Second, n may be varied to measure the relative sample complexity of different algorithms.
In practice we choose it to be similar to that of a modest labelling budget (Section 3.1).
Third, we assume that P T is known, and the goal is to build an algorithm with the best inductive biases for solving samples from it.
We desire an “open world”, where evaluation data is extremely diverse and can always be freshly sampled.
Unfortunately, in practice the benchmark must contain a ﬁnite number of ﬁxed test sets, therefore we must ensure that the algorithms have not been pre-exposed to speciﬁc evaluation samples, as described below.
2. A P RACTICAL B ENCHMARK
We now describe the Visual Task Adaptation Benchmark (VTAB), designed to be the best possible proxy for Eq.
Fig.
1 contains an overview diagram.
We then discuss a suite of methods that may be used to optimize VTAB: deep representation learning.
Eq.
(1) is very difﬁcult to compute in practice.
One must deﬁne a representative P T and compute the desired expectation.
If sampling the distribution, one must avoid overﬁtting to the sample.
Finally, limiting computational complexity becomes paramount as the number of tasks grows.
Distribution over tasks We aim for universal visual understanding, so we informally deﬁne P T as “Tasks that a human can solve, from visual input alone.”.
See Appendix B where we validate this.
Intuitively, such tasks should beneﬁt from visual representations learned by observing and interacting with the natural world.
The second clause eliminates tasks that require external knowledge that is (currently) unreasonable for a vision algorithm to acquire – an extreme example would classifying objects grouped by a function of their spelling in a natural language.
Section 2.2 details the samples.
Expectation over tasks We approximate the expectation over tasks by an empirical average over a number of hand-picked samples.
Ideally, we would sample a new task for each evaluation, as is possible in procedural environments, e.g. (Finn et al., 2017).
However, real-world vision tasks are expensive to collect, so we deﬁne a ﬁxed, representative set of samples from P T .
While ﬁxing the set reduces variance, it introduces a risk of meta-overﬁtting.
Mitigating meta-overﬁtting To do this, we treat the evaluation tasks like a regular test set, and thus consider them unseen.
This implies that algorithms that use pre-training must not pre-train on any of the evaluation tasks (even their unlabelled images).
Upstream training should provide useful inductive biases for any draw from P T , so should not use test samples.
Fortunately, despite numerous test re-evaluations on popular ML benchmarks, such as ImageNet, progress seems to transfer to new data (Recht et al., 2018; 2019; Kornblith et al., 2019).
Uniﬁed implementation We use many diverse tasks, however, this could render usage impractical.
As discussed, the algorithms must have no prior knowledge of the downstream tasks.
Therefore, while it is permitted to run a hyperparameter search on each task, the search space cannot be taskdependent.
To get meaningful results, we need to deﬁne architectures and hyperparameter searches that work well across the benchmark.
For this, we convert the tasks into classiﬁcation problems.
With a homogeneous task interface, we may control for possible confounding factors.
For example, we may use the same architecture and hyperparameter sweep throughout.
Of course, not all tasks can be efﬁciently modeled as classiﬁcation with global labels, such as those requiring per-pixel predictions.
Nonetheless, we design the tasks such that success on VTAB requires learning of diverse set of visual features: object identiﬁcation, scene classiﬁcation, pathology detection, counting, localization, 3D geometry, and others.
2. T ASKS
VTAB contains 19 tasks which cover a broad spectrum of domains and semantics.
Appendix A contains details.
These are grouped into three sets: NATURAL , SPECIALIZED , and STRUCTURED .
The NATURAL group represents classical vision problems.
These tasks contain natural images captured using standard cameras.
The classes may represent generic, ﬁne-grained, or abstract objects.
The group includes: Caltech101, CIFAR-100, DTD, Flowers102, Pets, Sun397, and SVHN.
The SPECIALIZED group also contains images of the world, but captured through specialist equipment.
These images contain invariances from those in the NATURAL tasks.
Nonetheless, humans recognize the structures therein, thus generic visual representation should also capture the visual concepts in these images.
We have two sub-groups: remote sensing, and medical.
Remote sensing includes Resisc45 and EuroSAT: aerial images of the earth captured using satellites or aerial photography.
Medical includes Patch Camelyon, metastases detection from microscopy images, and
Diabetic Retinopathy, retinopathy classiﬁcation from fundus images.
The STRUCTURED group assesses comprehension of the structure of a scene, for example, object counting, or 3D depth prediction.
Most of these tasks are generated from simulated environments, whose structure is easy for a human to determine, but whose domain differs greatly to datasets like
ImageNet.
These tasks are intended as a step towards useful representations for perceptual control.
We include: Clevr : Simple shapes rendered in a 3D scene, we include two tasks: counting and depth prediction.
dSprites : Simple black/white shapes rendered in 2D, we include two tasks: location and orientation prediction.
SmallNORB : Artiﬁcial objects viewed under varying conditions, we include two tasks: object-azimuth and camera-elevation prediction.
DMLab : Frames from a rendered 3D maze.
The task involves predicting the time for a pre-trained RL agent to navigate to an object.
KITTI : frames captured from a car driver’s perspective.
We predict the depth of the nearest vehicle.
2. R EPRESENTATION AND T RANSFER L EARNING
Success on VTAB, and ultimately optimizing Eq.
(1), requires some knowledge of P T .
For example, algorithms that use CNN architectures have a useful inductive bias for vision.
While these biases are usually manually designed (e.g. through choice of an architecture), they can also be learned.
For example: (i) Zoph & Le (2017) learn performant architectures, (ii)
Bello et al.
(2017) learn optimization algorithms, (iii)
Wong et al.
(2018) learn hyperparameter search strategies, (iv) Cubuk et al.
(2019) learn data augmentation functions, (v) Raghu et al.
learn layer-wise initialization distributions, and (vi) pre-trained data representations can be learned.
While all of the above merit investigation for VTAB, we focus on representation learning.
Human visual perception is reﬁned through years of observation and interaction with the world, resulting in a system that solves new tasks with few in-domain labels.
Likewise, we aim to pre-train a network that extracts useful features, or representations, from raw data.
Transfer strategy When representations are not pre-trained on the evaluation tasks themselves (which is explicitly forbidden in VTAB), they must be adapted to solve the new task (using limited in-domain data).
State-of-the-art representation learning algorithms typically train a deep neural network on some task(s), and then adapt the network to the new task.
The simplest adaptation strategy is to freeze the network’s weights and train another – usually smaller – model on top.
When the upstream and downstream datasets differ signiﬁcantly, ﬁne-tuning the original weights is more effective (Yosinski et al., 2014; Kornblith et al., 2019).
VTAB does not constrain the transfer strategy; here we benchmark methods using ﬁne-tuning since it tends to perform best.
Upstream training The representation learning literature often focuses on unsupervised learning as it may be applied to any data without human annotation.
However, supervised data, where available, may also produce good representations.
Indeed, the most popular models used in practice are pre-trained on ImageNet labels (Huh et al., 2016, and refs therein).
VTAB does not constrain the type of data (supervised or unsupervised) used to pre-train the representations.
E XPERIMENTS
We use VTAB to evaluate popular representation learning algorithms.
We ask four primary questions: (i) How effective are these algorithms, such as ImageNet pre-training, across a broad spectrum of tasks?
(ii) How do different algorithm classes compare?
(iii) Which method achieves the best overall performance on VTAB?
(iv) How does VTAB compare to other benchmarking strategies?
3. S ETUP
Data Section 2.2 describes the tasks.
VTAB aims to assess adaptation with limited data, so we evaluate primarily on 1 , 000 training examples per task, drawn i.i.d.
from each training set.
This simulates performance under a small per-task labelling budget which is the main goal of this benchmark.
For completeness, we also use the full datasets.
This allows us to assess the value of representation learning as in-domain data increases, and to check performance against prior art.
Representation learning algorithms
We evaluate the following representation learning algorithms: supervised, semi-supervised, self-supervised, and generative models.
As a baseline we train a “from-scratch” model, starting from random weights.
We evaluate a total of 16 algorithms.
Throughout, the models are pre-trained ImageNet.
The supervised model is trained on 100% of the ImageNet labels (S UP -100%).
For self-supervised learning, we include both image-based and patch-based models.
The image-based models include
ROTATION (Gidaris et al., 2018) and E XEMPLAR (Dosovitskiy et al., 2014).
The patch-based models include R ELATIVE P ATCH L OCATION (Doersch et al., 2015) and J IGSAW (Noroozi & Favaro, 2016).
We use the public implementations of Kolesnikov et al.
The patch-based models are converted into image-level classiﬁers by averaging the representations from a 3 × 3 grid of patches (see
Appendix C).
The semi-supervised models are trained using 10% of labelled ImageNet data with an auxiliary loss on all of the data, see (Zhai et al., 2019).
We use either rotation (S UP -R OTATION10%) or Exemplar (S UP -E XEMPLAR -10%) auxiliary losses.
These models can also use all of the labels, denoted S UP -R OTATION -100% and S UP -E XEMPLAR -100%.
For generative models, we use GANs and VAEs.
The most common way to extract representations from a GAN is from the discriminator, replacing its ﬁnal linear layer.
We use both the label-conditional and unconditional
BigGAN discriminators (Brock et al., 2019)
(C OND -B IG GAN and U NCOND -B IG GAN) using the public implementations of Chen et al.
; Lucic et al.
For autoencoders, the output of the encoder can be used as the representation.
We test VAEs (Kingma & Welling, 2014), and WAEs with three distribution matching losses: GAN, MMD (Tolstikhin et al., 2018), and UKL (Rubenstein et al., 2019).
Transfer algorithm VTAB permits any transfer algorithm.
We use ﬁne-tuning of the entire network here, since that performs best (Kornblith et al., 2019).
Adding a linear model to frozen networks is also popular, so we compare to this approach in Section 3.4.
Hyperparameters Both the upstream pre-training of representations and transfer involve many hyperparameters.
One major choice is architecture.
Bigger architectures almost always perform better (see Appendix L and Kolesnikov et al.
Therefore, for comparability, we use ResNet50-v2, or similar, for all methods.
For the supervised and semi-supervised methods we use the architecture in He et al.
For GANs and auto-encoders we also use deep residual networks, but with appropriate modiﬁcations to train them successfully on their generative pre-training task.
Appendix C contains details.
The hyperparameters used for transfer also inﬂuence performance, and different datasets may require different settings.
We run VTAB in two modes: lightweight and heavyweight .
The lightweight mode performs a restricted hyperparameter search on each tasks, permitting fair comparison with few resources.
Many hyperparameters are ﬁxed, including optimizer, batch size, pre-processing, and weight decay.
For each task, lightweight mode sweeps four hyperparameters: 2 initial learning rates, and 2 learning rate schedules.
See Appendix I for details.
We choose short schedules to limit cost, but show in Section 3.5 that these yield near-optimal performance.
In heavyweight mode we perform a large hyperparameter search, limited only by resource availability.
Here, we perform a random search over learning rates, schedules, optimizers, batch size, train pre-processing functions, evaluation pre-processing functions, and weight decay.
We include longer training schedules and higher resolution images.
This mode is used to establish the best attainable
VTAB score with the algorithms studied.
Details in Appendix I.
Due to the high cost of heavyweight mode, we perform our main study in lightweight mode.
In
Section 3.3 we show that although extensive tuning improves all methods, the relative performances are mostly unaffected.
For future study using VTAB, we recommend a similar lightweight setup that facilitates fair comparison without high cost.
If large computation resources are available, the heavyweight mode may be used to push the state-of-the-art.
Protocol We perform model selection for each task on a validation set.
We then re-train the best model on the training and validation set (or a 1000-example subset thereof), and evaluate on the test set.
We run the test set evaluation three times and report the median score by default.
Metrics We evaluate all methods with top-1 accuracy.
We consider other reported metrics in Appendix D.
To aggregate scores across methods, we compute the mean top-1 accuracy.
We investigate more complex aggregation strategies in Section 3.5, but the relative performances remain the same, so we report mean accuracy due to its simplicity.
3. VTAB R ESULTS
We run all 16 methods on 19 tasks with 1000 training examples and the full datasets using lightweight mode.
We assess the value of representation learning over from-scratch training, and compare the performance of the different methods across the dataset groups.
Fig.
4 shows two detailed per-task comparisons: (i) S UP -100%
(ImageNet) versus F ROM -S CRATCH which shows the value of standard ImageNet representations.
(ii) S UP -R OTATION -
% versus S UP -100% which shows the best performing method versus ImageNet.
Appendix G, Table 6 contains the complete results table.
Generative models These perform worst, often more poorly than training from scratch.
This is perhaps unsurprising because scaling these models to complex datasets is known to be challenging.
Nonetheless, despite SOTA generative quality on ImageNet, the GAN’s discriminator does not provide useful representations.
On the 1000-example subsets the U NCOND -B
IG GAN outperforms from-scratch, but all generative models lose substantially on the full datasets.
Interestingly, on the full dataset the GANs perform better than autoencoders on the NATURAL tasks, but worse on
STRUCTURED .
This indicates that they may be more strongly ﬁt to the ImageNet domain.
G e n e r a t i v e
F r o m - S c r a t c h
S e l f S u p e
r
v i s e d
S e
m i
S u p e
r
v i s e d
S u p e
r
v i s e d
A v e r a g e T o pA cc u r a c y
Natural
G e n e r a t i v e
F r o m - S c r a t c h
S e l f S u p e
r
v i s e d
S e
m i
S u p e
r
v i s e d
S u p e
r
v i s e d
Specialized
G e n e r a t i v e
F r o m - S c r a t c h
S e l f S u p e
r
v i s e d
S e
m i
S u p e
r
v i s e d
S u p e
r
v i s e d
Structured
Samples
All
[Table]
MM
[Table] [Table]
[Figure]
Similarly, Ravuri & Vinyals (2019) show that models with high sample quality fail to generate data from which an accurate classiﬁer can be trained.
Explicit consideration of representation quality, in addition to generative scores, may lead to further progress in generative modeling research.
Self-supervised All self-supervised representations outperform from-scratch training.
The best,
R OTATION attains 60 .
4% mean top-1 accuracy, while F ROM -S CRATCH attains 43 .
1% (1000example datasets).
Methods applied to the entire image (R OTATION , E XEMPLAR ) outperform patchbased methods (J IGSAW , R EL .P
AT .L OC .).
However, the patch based methods perform slightly better on DTD (Describable Textures) and Retinopathy (Fundus images), see Appendix G, Table 6.
These tasks require sensitivity to local textures, which is well represented by patch-based methods.
Figure 2 shows that self-supervised perform worse than supervised methods on the NATURAL datasets, but similarly on SPECIALIZED , and even better on some STRUCTURED tasks.
This indicates that
ImageNet labels may encode sub-optimal invariances for non-natural tasks.
(Semi-)Supervised Supervised models perform best.
The beneﬁts are most pronounced on the
NATURAL tasks, whose domain and task-semantics are arguably most similar ImageNet classiﬁcation.
The combination of self-supervision with 10% labels (semi-supervised) performs only marginally worse ( 1 − 1 .
5% ) than methods with 100% labels.
Interestingly, there is beneﬁt of com- s N O R BE l e v
R e t i n o p a t h y
C l e v r - D
i s t
C l e v r - C o un t
C a m e l y o n dSp r - O r i e n t dSp r - L o c D M L a b
E u r o S A
T s N O R BA
z i m S V H N
K I TT I - D
i s t
S un
[Figure]
R e s i s
[Figure]
C
I F
A R
[Figure]
l
[Figure]
r
[Figure]
t
[Figure]
t
[Figure]
s
[Figure]
E l
[Figure]
v P
[Figure]
t
[Figure]
E u r
[Figure]
S A
T
C a l t e c
[Figure]
i
[Figure]
r
[Figure]
i
[Figure]
r
[Figure]
c
C a m e l y
[Figure]
n D T D s N O R
[Figure]
z i
[Figure]
dSp r
[Figure]
r i e n t
C l
[Figure]
v r
[Figure]
I
[Figure]
I
[Figure]
i
[Figure] [Figure]
[Figure] [Figure]
C a lt ec h
C
I F
A
RD T D F l o w e r s
P e t s
S VHN S un397 C a m e l yon
E u r o S
A T
R e s i s c
R e ti nop a t hy
C l e v rC oun
t
C l e v rD i s t
D M L a b
K I TT
I- D
i s t d S p r-
L o c d S p r-
O r
i s NO
RB - A z i m s
NO
RB - E l e v
M ea n
[Figure]
[Table]
bining self-supervised losses with 100% labelled ImageNet, i
[Table]
e. without additional unlabelled data, see Fig. 4.
S UP -R OTATION -100% attains 66 .
4% , whereas S UP -R OTATION -100% attains 68 . 0% .
3. H EAVYWEIGHT H YPERPARAMETER S WEEPS
We compare F ROM -S CRATCH and the three best models: S UP -100%, S UP -R OTATION
[Table]
and
S UP -E XEMPLAR -100% using the heavyweight hyperparameter search.
Table 1 contains the results.
The hyperparameters selected are shown in Appendix I. As expected, all methods improve significantly, e.g. F ROM -S CRATCH improves from 43 .
1% to 61 .
2% (1000-examples).
Prior work (He et al., 2018) shows that with sufﬁcient data and training steps, from-scratch training is competitive for detection.
We observe, with the heavy sweep, that although the gap to S UP -100% is smaller, it is still substantial: S UP -100% attains 72 . 8% .
Again, addition of self-supervision helps, but the margin is smaller.
S UP -E XEMPLAR -100% performs best with a VTAB (1000-example) score of 73 . 6 .
We check our results against those recently reported in the literautre (Appendix H, Table 8).
Our results are comparable; behind on highly popular tasks on which more complex architectures have been applied (e.g. CIFAR), but ahead in others.
In Appendix L we show that simply increasing the architecture size improves VTAB performance.
In summary, pre-trained representations still offer substantial value with heavy hyperparameter tuning and long training schedules on VTAB.
The overall ranking of methods remains the same, with the exception that SUP - EXEMPLAR -100% now outperforms SUP - ROTATION
-100%, although these methods’ performances are always similar.
3. F ROZEN F EATURES E XTRACTORS
S V H N P e t s
C a l t e c h
F l o w e r s
S un
C
I F
A R100 D T D
R e t i n o p a t h y
C a m e l y o n
E u r o S
A T R
e s i s c
45 C l e v r - D
i s t s N O R B - A z i m dSp r - L o c
C l e v r - C o un t s N O R B - E l e v D M L a b
[Figure]
r - O r i e n t
K I TT I - D
i s t
[Figure]
Representations are often evaluated by training a linear layer on a frozen model (Kolesnikov et al., 2019).
ImageNet is often used for linear evaluation, but this is meaningless for the models we consider because many use ImageNet labels for pre-training.
However, linear evaluation is also used in a transfer setting (Goyal et al., 2019a), so we apply this protocol to the VTAB tasks, and contrast to ﬁne-tuning used above.
The full protocol resembles the lightweight ﬁne-tuning sweep, see Appendix K.
Fig.
5 shows the per-task correlation between linear and ﬁne-tuning.
Appendix K, Tablecontains the full table of results.
We ﬁrst note that linear evaluation signiﬁcantly lowers performance, even when downstream data is limited to 1000 examples.
For example, S UP -100% attains 66 .
4% with ﬁne-tuning (lightweight sweep), but 57 .
3% with linear.
Linear transfer would not by used in practice unless infrastructural constraints required it.
Second, Fig.
5 shows that on many datasets, particularly SPECIALIZED and
STRUCTURED , the correlation is low.
Linear evaluation may lead to different conclusions, for example, C OND -B IG GAN attains 43 .
3% (1000-examples) on linear, outperforming both both patchbased self-supervised methods (R EL .P
AT .L OC and J IGSAW ).
Yet when ﬁne-tuned these patch-based methods attain around 53% , but the GAN, 41 . 4% .
Another discrepancy is between semi-supervised and supervised.
S EMI -R OTATION -10% and S EMI -E XEMPLAR -10% perform 1 − 1 .
5% worse than
S UP -100% with ﬁne-tuning, but 4 − 5% worse with linear evaluation.
The self-supervised methods extract useful representations, just without linear separability.
Previous works (Kornblith et al., 2019; Kolesnikov et al., 2019) claim that linear evaluation results are sensitive additional factors that we do not vary, such as ResNet version or pre-training regularization parameters.
We conclude that linear evaluation is a poor proxy for overall reduced sample complexity.
3. A NALYSIS
Per-Task Metric
In previous works, some datasets are reported using alternative metrics to top1 accuracy, namely: mean-per-class accuracy (for Caltech101) or Cohen’s quadratic kappa (for
Retinopathy).
We study whether these metrics reveal different results to top-1 accuracy.
We ﬁnd that, although there are some minor ranking differences, the overall picture remains unchanged.
Kendall’s ranking correlation scores, with respect to top-1 accuracy, are: 1.0 (mean per-class accuracy), 0.
(quadratic kappa).
Appendix D shows the full rankings according to each metric.
Metric Aggregation Across Tasks Mean top-1 accuracy across tasks creates an implicit weighting.
First, some tasks use the same input data (e.g Clevr-Count and Clevr-Dist), thus upweighting those domains.
Second, the task groups differ in size.
Third, the tasks exhibit different performance ranges across methods.
Therefore, we compare seven different ranking aggregation strategies: (i) Macroaverage across datasets (grouping tasks with the same input data).
(ii) Macro-average across groups (merging tasks in the same group,
NATURAL , etc.)
(iii)
Geometric mean
(iv)
Average rank.
(v)
Average
rank after small perturbation of the scores with noise.
(vi) Robust (binned) average rank.
(vii) Elimination rank – equivalent to an “Exhaustive Ballot”
All strategies have a high Kendall’s correlation with the vanilla mean across tasks is high (above 0 . 9 ).
The most dissimilar strategy on the 1000-example datasets is the perturbed average rank, with correlation 0 . 91 .
Therefore, we use mean top-1 accuracy because it is simple, interpretable, and can be computed for each method independently, unlike rank-based aggregation.
See Appendix E for details.
Representative Subset For rapid-prototyping, a representative subset of VTAB may be useful.
We compute the rank correlation between the mean scores produced by each (cid:0) 205 (cid:1) subsets of ﬁve tasks, and the full suite.
Appendix F contains the results.
The top-5 subsets tend to span different domains, but differ to each other.
Although using a subset might be useful for screening models, they were computed using our particular set of models, and may correlate less well in other experiments.
Using subsets also increases the risk of meta-overﬁtting, which VTAB aims to avoid by having many tasks.
Limiting Evaluation Cost The cost is near linear in the schedule length.
To determine a minimal, yet meaningful, schedule, we sweep over schedules ranging from 40 to 40 , 000 steps, with batch size 512 on all datasets.
Fig.
6 summarizes the results, details Figs.
19 and 20, Appendix M. Most runs reach their optimal performance within 1 , 000 steps (updates), and do not improve signiﬁcantly when trained for longer.
[Figure]
We ﬁne-tune the models using 16 Google Cloud
TPU-v3 hardware accelerators.
We conduct an additional experiment to assess whether our results can be reproduced with a more basic hardware setup.
We evaluate S UP -100% on all
VTAB tasks ( 1 , 000 examples) using a single
NVidia P100 GPU with 1 , 000 steps, 64 images per mini-batch, and a learning rate of 0 . 01 .
In this experiment we match the mean performance of the same model reported in Table 1.
R ELATED W ORK
Vision Benchmarks The Visual Decathlon (Rebufﬁ et al., 2017) contains ten classiﬁcation tasks:
Omniglot, and nine using natural images.
Four overlap with VTAB, but VTAB includes several other domains and tasks.
Importantly, these benchmarks have opposite evaluation protocols: The Visual
Decathlon allows direct joint optimization on the tasks, but forbids external data.
By contrast, VTAB forbids multi-task learning, but permits transfer from any arbitrary dataset — this simulates one-off training of representations which are then applied to novel tasks.
Further, VTAB considers a lowsample regime (1000-examples), which reﬂects performance under a reasonable labelling budget.
We conduct experiments showing that the methods ranked according to VTAB are more likely to transfer to new tasks, than those ranked according to the Visual Decathlon (Appendix N).
The Facebook AI SSL challenge (Goyal et al., 2019b) was proposed to evaluate self-supervised models.
It contains two datasets of natural images, with four tasks, including classiﬁcation, detection, and low-label classiﬁcation.
The original paper (Goyal et al., 2019a) also includes navigation and surface normal estimation.
In contrast, VTAB uses classiﬁcation tasks only to admit taskindependent implementations.
VTAB attains diversity with many tasks from alternative domains and with different semantics (localization, counting, etc.).
VTAB experiments include self-supervised methods, but are not restricted to them.
Indeed, a challenge posed by the VTAB benchmark is to outperform supervised ImageNet models.
Most importantly, Facebook SSL requires transfer via a shallow network stacked on a ﬁxed CNN, whereas VTAB permits any transfer, focussing only on performance with few samples.
Meta-Dataset (Triantaﬁllou et al., 2019) contains natural image classiﬁcation tasks.
The protocol differs to VTAB: most of Meta-Dataset’s evaluation datasets are also in the training set.
The train/test split is instead made across classes.
Meta-Dataset is designed for few-shot learning, algorithms that use considerably fewer than 1000 samples.
This regime requires different solutions to VTAB.
Other ML Benchmarks Benchmark suites have driven progress in other areas of machine learning, such as the GLUE (Wang et al., 2018) and the NL Decathlon (McCann et al., 2018) in NLP, or the Atari suite (Mnih et al., 2013) and Mujoco environments (Todorov et al., 2012) in RL.
In NLP, the GLUE Benchmark (Wang et al., 2018) has been a central benchmark used to measure progress of BERT (Devlin et al., 2018) and variants.
Representation
Learning Evaluation Popular evaluations for representation learning are linear/MLP and semi-supervised.
In linear/MLP evaluation, widely used in for self-supervised representations (Doersch et al., 2015; Zhang et al., 2016; Noroozi & Favaro, 2016; Doersch & Zisserman,
2017), the weights of a pre-trained network are frozen, and a linear layer/
MLP is trained on top to predict the labels.
Evaluation is often performed on the same dataset that was used to train the representations, but using all labels, defeating the goal of sample efﬁciency.
A semi-supervised protocol is more realistic and performs better (Zhai et al., 2019).
This evaluation is performed on a single dataset, by contrast VTAB concerns transfer to unseen tasks.
Linear evaluation is sensitive to small upstream training details, such as the ResNet version or regularization, that make little difference when training end-to-end (Kornblith et al., 2019; Kolesnikov et al., 2019).
Other intrinsic scores have been proposed to measure the disentanglement of representations (Locatello et al., 2019), or the mutual information between the input and representations (Hjelm et al., 2018).
Both were shown to be weakly correlated with representation utility (Locatello et al., 2019; Tschannen et al., 2019).
Generative Models Evaluation Generative models are usually evaluated by generative quality, a difﬁcult goal with numerous proxy metrics, such as reconstruction error, Frechet Inception
Distance (Heusel et al., 2017), Inception score (Salimans et al., 2016), precision-recall (Sajjadi et al., 2018).
log likelihood (for auto-regressive models), or the evidence lower bound (for VAEs).
Sometimes generative models are also evaluated using linear classiﬁcation (Radford et al., 2016;
Donahue et al., 2017; Dumoulin et al., 2017), or semi-supervised learning (Kingma et al., 2014;
Narayanaswamy et al., 2017; Tschannen et al., 2018).
D ISCUSSION
Our results illuminate the current state of representation learning.
First, ImageNet labels provide a strong baseline, especially for natural datasets.
However, on specialized datasets e.g. medical, distant domains, or tasks that require structured understanding, their value is reduced.
Research on how to train on more diverse data sources, such as video, may be the key to moving beyond the “ImageNet domain” towards general, more human-like, visual representations.
Second, generative models — at least the common ones — appear not to be competitive representation learners.
4 Generation may be a useful endeavour in of itself, but if generative losses are used as a means towards representing data, then more explicit representation evaluation may assist progress.
Third, self-supervised learning appears promising, being the best-performing unsupervised method.
When combined with a few labels ( 10% ), these methods perform almost as well as fully supervised
ImageNet pre-training.
Self-supervised even provides some value on top of ImageNet labels on the same data .
We observe modest, yet signiﬁcant, improvements over pure supervised learning, especially on tasks that require structured understanding.
ImageNet labels encode lots of useful information, but also encode invariances that a universal representation should instead be sensitive to.
For example, the distance to an object.
Representation learning is inherently tied to transfer learning; without a transfer strategy representations cannot be used for new tasks.
Here, we elect ﬁne-tuning since it performs better than the popular linear-model alternative, and is known to be less brittle.
However, it would be interesting to research alternative strategies from domain adaptation (Wang & Deng, 2018), residual adapters (Rebufﬁ et al., 2017), transfusion (Raghu et al., 2019), and other ﬁelds.
VTAB may be used to optimize many design choices involved in learning generalizable representations.
Here, we study upstream training loss.
We therefore control other factors such as hyperparameter sweep size, architecture, transfer algorithm, downstream preprocessing, and upstream training data.
Varying these factors, and others, to improve VTAB is valuable future research.
However, the effect of each factor should be isolated from other confounders, such as hyperparameter sweep size.
The only approach that is out-of-bounds is to condition the algorithm explicitly on the test tasks, since this would compromise representation generalization. Code and data to run the Visual Task Adaptation Benchmark is made available, and progress is publicly monitored at [anonymous-url] .
All pre-trained models presented are also released.
We hope that VTAB drives representation learning towards the goal of making deep learning accessible to the many applications without vast labelling budgets.
B H UMAN E VALUATION
We deﬁne P ( T ) informally as “All tasks that a human can solve using visual input alone”.
Intuitively, the tasks in Appendix A satisfy this property because the task semantics involve simple visual concepts (for a human), and the images contain recognisable objects – either in a natural or artiﬁcial environment.
However, we also check empirically that humans can solve the types of tasks using in VTAB from examples alone .
We therefore evaluate human raters on a representative subset of the tasks used in VTAB:
Pets (natural images, ﬁne-grained object labels), DTD (natural, textures), Camelyon (specialized, medical images), EuroSAT (specialized, aerial images), DMLab (structured, distance prediction), and Clevr-count (structured, object counting).
For each human-evaluated dataset, raters are given 20 random examples for each class, taken from the training split.
Raters are asked to classify between 50 and 100 images each, for a total of 1K images (except for DMLab: 534 images), randomly taken from the test split, based on the provided training examples.
Beside the examples, no hints nor explanations are given on the nature of the tasks.
The raters have to deduce which properties of each image should be used to classify those, for example: the breed of the animal, the distance between the camera and the objects, or the number of objects.
The raters are asked not to exchange on the tasks, so each rater produces independent work.
All datasets rated by humans are the same as the one rated by the models, except for DMLab, where we only assess the distance prediction aspect and not object type.
This is because there are too many object types for the raters to learn the two groups of objects from 20 examples per class.
Therefore, we asses distance-prediction alone.
The human DMLab task contains only 3 classes, each of which contains many object types, and differ only in object distance.
Note that this evaluation is not meant to quantify the relative performances of humans and machines, since protocol differences e.g. number of training examples, render the performance incomparable.
Instead, it is meant as a veriﬁcation that the kinds of domains (natural images, aerial imagery, etc.) and semantics (object type classiﬁcation, localization, counting, etc.) are possible to learn from examples alone using human-level visual representations.
[Table]
We measure human performance using mean-per-class accuracy.
This is because the human training sets are class-balanced, so the raters cannot learn the class priors which algorithms that see an i.i.d sample of the training set could.
Table 3 shows the results.
The results indicate that some tasks are harder due to more subtle distinctions (Camelyon) or noise (DMLab), and some very easy (Clevrcount).
However, in all cases the raters perform signiﬁcantly better than random guessing.
This demonstrates that the types of tasks used in VTAB are ones for which human-like visual representations are useful to solve them, using few labels and visual-input alone.
C N ETWORK A RCHITECTURES
All of the methods (except for BigGAN models) we evaluate in the paper use the standard ResNet50v2 (He et al., 2016) architecture.
This architecture consists of 50 convolutional layers.
The representation produced by this model has 2048 dimensions.
Exact details of this architecture can be found in (He et al., 2016).
In the Cond-BigGAN and Uncond-BigGAN models we use publicly available 5 implementation by
Lucic et al.
and (Chen et al., 2019) of the custom ResNet-like architecture proposed and described in (Brock et al., 2019).
It has 1536 dimensions in the ﬁnal representation layer.
We use a specialized procedure for evaluating patch-based models ( Relative Patch Location (Doersch et al., 2015) and Jigsaw (Noroozi & Favaro, 2016)).
These models use ResNet50 model with the overall stride reduced from 64 to 16 (by substituting the ﬁrst and the last strided convolution of the standard ResNet50 model by a convolution with stride one).
During the adaptation phase, we apply
ResNet50 with reduced stride in the following way (assuming that input image size is 224 × 224 ):
• Perform central crop of size 192 × 192 .
• Cut image into 3 × 3 patches of size 64 × 64 .
• Apply the ResNet50 model independently to every patch.
• Output the ﬁnal representation and element-wise average of 9 individual patch representations.
D A LTERNATIVE M ETRICS
[Table]
S c r a t c h
J i g s a w
R e l .
P a t .
L o c
E x e m p l a r
R o t a t i o n
S e
m i - R o t a t i o n -%
S e
m i - E x e m p l a r -%
S u p100 %
S u p - E x e m p l a r -%
S u p - R o t a t i o n -%
Quadratic Kappa
Figure 7 shows different test metric scores (top-1 accuracy, mean-per-class accuracy, and Cohen’s quadratic kappa), averaged across all datasets in the benchmark, for the different methods studied.
The methods were ranked in the x-axis according to their score in each metric.
Observe that top1 and mean-per-class accuracy give exactly the same ranking.
There are subtle differences when using Cohen’s quadratic kappa, but the overall picture remains unchanged: supervised methods outperform semi-supervised methods by a small margin, followed by self-supervised methods, etc.
Kendall’s ranking correlation scores, with respect to top-1 accuracy, are: 1.0 for mean per-class accuracy, and 0.97 for Cohen’s quadratic kappa.
This conﬁrms that our conclusions hold even if we are not using the standard metrics for a few datasets
E A LTERNATIVE WEIGHTING AND RANKING SCHEMES FOR MODELS
Throughout, we use (unweighted) mean top-1 accuracy across all tasks to rank the different models.
This assumes that the samples represent our desired task distribution P T in an unbiased manner.
However, there may be other sensible weighting schemes that are not uniform.
Here we explore three alternative weighting schemes (i) assigning equal weight to every dataset – there are three data sets that are used in two tasks, and the others are used in one task, (ii) assigning equal weight to each task group: NATURAL , SPECIALIZED , and STRUCTURED , and (iii) the weighting scheme introduced in Balduzzi et al.
Another issue inherent in (weighted or unweighted) mean accuracy is that the mean accuracies of individual tasks vary signiﬁcantly depending on the task’s difﬁculty.
Since maximum accuracy is bounded, this may limit the range of performances, implicitly upweighting tasks with more “headroom”.
Therefore, we explore a few simple alternative ranking strategies (see Dwork et al.
for an introduction raking aggregation methods):
(i)
Ranking according to the geometric mean.
(ii)
The average rank obtained by ranking the models for each task according to accuracy and then computing the average rank across tasks.
(iii)
A noise-perturbed version of (i) in which the accuracies are perturbed by Gaussian noise with variance 1 .
prior to ranking.
(iv) A robust variant of the average rank, where, prior to averaging ranks across tasks, the accuracy is binned into buckets of size 1% and all models in the same bucket obtain the same rank.
(v) An elimination ranking scheme equivalent to the “Exhaustive Ballot” voting system, see e.g. (Shahandashti, 2016).
We measure the agreement between these ranking strategies using the Kendall rank correlation coefﬁcient (Kendall, 1945).
To account for the training/evaluation stochasticity in VTAB, we sample (independently for every pair of model and task) one out of the three test-set repetitions, and compute the rank correlation between ranking according to the mean accuracy and each alternative ranking.
We average over 100 such samples.
The results are in Table 4.
The mean rank correlation between the ranking according to the mean across tasks correlates very well with alternative ranking schemes.
In particular, weighted means for 1000 samples and the full dataset as well as the geometric mean have a rank correlation with the ranking according to the mean that exceeds 0 .
The agreement with different types of average rank somewhat lower for 1000 samples, but still considerable.
We hence conclude that the mean accuracy across tasks is a fairly robust metric to rank models in VTAB.
[Table]
F R EPRESENTATIVE S UBSET OF T ASKS
We explore another direction related to ranking of models: Which subset of tasks is most representative of the performance of the entire benchmark?
Such a subset allows for cheap iteration which is beneﬁcial during early development of new models.
We search the most representative 5 tasks from the full set of 20 tasks by performing exhaustive search over all (cid:0) 205 (cid:1) subsets.
For each subset we compute the mean accuracy and compute the
Kendall rank correlation coefﬁcient between the resulting ranking of models and the ranking according to the mean over all tasks (averaged over 10 trials sampled as described in Appendix E).
Table 5 shows the subsets which produce the highest rank correlation.
There are many subsets which lead to an excellent agreement with the ranking according to the full mean.
These subsets can be quite different provided that they are sufﬁciently diverse.
To assess how well we can expect this approach generalize to unseen models, we perform the following experiment.
For each pair of models, we perform subset selection based on the accuracies of the remaining models, and check whether the ranking of the left out pair based on the mean across all data sets agrees with the ranking of the pair according to the mean over the subset.
The mean outcome of this binary test across all pairs is an estimate of the probability for correctly ranking an unseen pair of models: 0 .
940 for the full data sets and 0 .
972 for 1000 examples.
The subset selection hence generalizes to an unseen pair of models with high probability.
The representative subsets in Table 5 may be used for rapid prototyping of new methods before running the full VTAB.
However, a caveat to the above analyses is that we evaluate the ranking of a very diverse set of models, from those whose performance is worse than from-scratch training to performance combinations of self-supervision and supervision.
Therefore, the ranking is reasonably robust.
To make ﬁne-grained distinctions between models of a similar class, the representative subsets above may be less reliable.
Repeated iteration on just a few tasks exposes one more to the risk of meta-overﬁtting.
[Figure]
[Table]
[Figure]
[Table] [Table]
R
[Table]
D T
[Table]
l
[Table]
r
[Table]
P e t
[Table]
S VHN
[Table]
C a m e l yon
E u r o S
A T
R e s i s c
R e ti nop a t hy
C l e v rC oun
t
C l e v rD i s t
[Table]
M - L a b
K I TT
I-
[Table]
i s t d S p r
[Table]
L o c d S p r
[Table]
r i
[Table]
NO
[Table]
m
[Table]
NO
[Table]
l
[Table]
H H EAVYWEIGHT EXPERIMENTS
Table 7 shows the full heavyweight results.
Table 8 shows recently reported results for each dataset in the literature (where available).
The literature results may use substantially more complex architectures, or additional task-speciﬁc logic.
For example, the Retinopathy result uses three neural networks combined with decision trees
[Table]
and uses additional information, such as combining the images from two eyes of the same patient.
[Table] [Table]
I H YPERPARAMETER S WEEPS
Lightweight sweep The lightweight mode performs a restricted hyperparameter search for all tasks, permitting fair comparison with few resources.
It uses ﬁxed values for most hyperparameters.
We set the batch size to 512 and use SGD with momentum of 0.9.
When ﬁne-tuning, we do not use weight decay, and when training from scratch we set it to 0.001.
We resize all images to 224 × 224 , except for generative models, where we resize to 128 × 128 since training at a higher resolution is currently very challenging.
Lightweight mode sweeps four hyperparameters:
• Learning rate: { 0 . 1 , 0 .
01 } • Traning schedule:
In all cases we decay the learning rate by a factor of 10 after 13 and 23 of the training time, and one more time shortly before the end.
We train for { 2 500 , 10 000 } training steps ( i
e. model updates).
Note that when training on 1000 examples, we perform model selection using the regular validation set for each task.
This setting is somewhat unrealistic, since in practice one would train on the larger validation set.
However, we use this setting, which is common in prior art (Oliver et al., 2018), because it signiﬁcantly reduces evaluation noise.
Further, Zhai et al.
(2019) show that using a small validation yields the same conclusions.
Heavyweight sweep We deﬁne a relatively large search-space over relevant hyperparameters for the ﬁne-tuning adaptation phase and perform an individual random search for each downstream task’s validation set in that space.
The random search consists of 100 independent trials.
Speciﬁcally, we search:
• Batch-size: { 128 , 256 , 512 } • Training schedule:
In all cases we decay the learning rate by a factor of 10 after 13 and 23 of the training time, and one more time shortly before the end.
We train for either – any of { 100 , 200 , 300 , 400 , 500 } epochs over the dataset, or – any of { 2 500 , 5 000 , 10 000 , 20 000 , 40 000 } training steps ( i.e. model updates).
Preprocessing: during training, optionally include random horizontal ﬂipping.
• Preprocessing: during training, optionally include random color distortions.
Preprocessing: during training, use any of the following techniques for resizing: – res.
: resize the image to 224; – res.|crop : resize the image to 256 and take a random crop of size 224; – res.sma|crop : resize the image keeping its aspect ratio such that the smaller side is 256, then take a random crop of size 224; – inc.crop : “inception crop” from Szegedy et al. (2015); – cif.crop : resize the image to 224, zero-pad it by 28 on each side, then take a random crop of size 224.
• Preprocessing: for evaluation, either – resize the image to one of { 224 , 320 , 384 } , or – resize it such that the smaller side is of size { 256 , 352 , 448 } and then take a central crop of size { 224 , 320 , 384 } .
• Weight decay: we sample log-uniform randomly in [10 − 5 , 10 − 1 ] .
• Optimizer: we randomly choose between – SGD with learning-rate chosen log-uniform randomly in [0 .
001 , 0 . 0] , or – Adam with learning-rate chosen log-uniform randomly in [10 − 5 , 10 − 2 ] .
We then choose the set of hyperparameters with the best performance at the end of training for each task individually.
For both training from scratch, and from a pre-trained supervised ImageNet training, we present the best set of hyperparameters for each task in Figs.
8 to 10 as a red dot.
The green background shows the distribution of hyperparameters which performed within 2% of the best one for each task, providing a sense for the importance of tuning that hyperparameter.
In Figs.
12 and 13, we show the achieved performance for the full distribution of hyperparameters as violin plots for the CIFAR-100 and DMLab downstream tasks, both for the small and full variants.
The supervised pre-training not only achieves better accuracy than training from scratch, but it also displays much smaller variance across hyperparameter values.
[Figure]
[Table]
[Figure]
e p o c
h s e p
o
c
h s e p
o
c
h s e p
o
c
h s e p
o
c
h s s t e p s s t e p s s t e p s s t e p s s t e p s n o n e h
o r i z
o n t a l r a w d
i s t o r t r e s .
r e s .
| c r o p r e s .
s m a | c r o p i n c .
c r o p c
i f .
c r o p r e s .
r e s . r e s .
c e n .
c r o p @
c e n .
c r o p @
c e n .
c r o p @
[Table] [Table]
[Figure] [Figure]
CamelyonSun397SVHNPetsFlowers102DTDCifar100Caltech
Batch
Schedule
FlipCol.
[Figure]
Eval
[Figure]
e p
[Figure]
c
[Figure]
s e p o c
h s
[Figure]
e p o c h s
[Figure]
e p o c
h s s t e p s s t e p s
s t
e p s
s t
e p s
s t
e p s n o n e h
o r i z
o n t a l r a w d
i s t o r t r e s .
r e s .
| c r o p r e s .
s m a | c r o p i n c .
c r o p c
i f .
c r o p r e s .
r e s . r e s .
c e n .
c r o p @
c e n .
c r o p @
[Figure]
e n .
c
[Figure]
p @
[Figure] [Figure]
RetinopathyResisc45EuroSATCamelyonSun397SVHNPetsFlowers102DTDCifar100Caltech
Batch
Schedule
FlipCol.
Train
Eval
WeightDecay SGD
[Figure]
e p o c h s
[Figure]
e p o c h s
[Figure]
e p o c h s
[Figure]
e p o c h s
[Figure]
e p o c h s
[Figure]
s t
e p s s t e p s
[Figure]
s t
e
p s
[Figure]
s t
e p s s t e p s n o n e h
o r i z
o n t a l r a w d
i s t o r t r e s .
r e s .
| c r o p r e s .
s m a | c r o p i n c .
c r o p c
i f .
c r o p r e s .
r e s . r e s .
c e n .
c r o p @
c e n .
c r o p @
c e n .
c r o p @ 384 1 e
[Figure] [Figure]
[Figure] [Figure]
Resisc45EuroSATCamelyonSun397SVHNPetsFlowers102DTDCifar100Caltech
Batch
Schedule
FlipCol.
Train
Eval
WeightDecay SGD Adam
Sum e p o c h
s e p
o
c
h s e p
o
c
h s e p
o
c
h s e p
o
c
h s s t e p s s t e p s s t e p s s t e p s s t e p s n o n e h
o r i z
o n t a l r a w d
i s t o r t r e s .
r e s .
| c r o p r e s .
s m a | c r o p i n c .
c r o p c
i f .
c r o p r e s .
r e s . r e s .
c e n .
c r o p @
c e n .
c r o p @
c e n .
c r o p @ 384 1 e - 5 1 e - 4 1 e - 3 1 e - 2 1 e - 1 0 . 0010 .
e - 5 1 e - 4 1 e - 3 1 e -
[Figure]
[Figure] [Figure]
[Table]
- L o c
E u r o S A
T P e t s
S un
C a m e l y o n
R e s i s c 45 S V H N
F l o w e r s
C l e v r - D
i s t
R e t i n o p a t h
y s N O R B - A
z
i m D M L a b s N O R B - E
l e v
C l e v r - C o un t dSp r - O r i e n t −
[Figure]
T o
[Figure]
A cc u r a c y D e l t a
H e a vy
L i g h t
[Figure]
C
I F
A R100 D T D
E u r o S A
T
C a l t e c h
S un 397 P e t s
K I TT I - D
i s t
S V H N
R e s i s c
C a m e l y o n dSp r - L
o c
F l o w e r s
R e t i n o p a t h y D M L a b
C l e v r - D
i s t s N O R
BA z i m s N O R BE l e v dSp r - O r i e n t
C l e v r - C o un t
[Figure]
[Table]
N O R B - E l e v
[Table]
t i n
[Table]
p
[Table]
y
[Table]
l
[Table]
dSp r
[Table]
L o c
R e t i n o p a t h y S V H N
E u r o S A T dSp r - O r i e n t
C a m e l y o n s N O R B - A
z i m
C l e v r - D
i s t
C l e v r - C o un t
D M L a b D T D
R e s i s c 45 S un
F l o w e r s
K I TT I - D
i s t
C
I F
A RC a l t e c h 101 P e t s −
[Figure]
cc
[Figure]
r
[Figure]
c y D
[Figure]
l t a
[Figure]
u p100 %
F r o m - S c r a t c h
C
I F A RK
I TT I - D
i s t s N O R B - E l e v
S un
C a l t e c h
C l e v r - D
i s t
C a m e l y o n P e t s dSp r - L
o c
R e t i n o p a t h y
F l o w e r s
E u r o S
A T S V H N D M L a b
R e s i s cdSp r - O r i e n t s N O R B - A z i m D T D
C l e v r - C o un t −
[Figure] [Figure]
– the additional self-supervised loss yields better representations for the STRUCTURED tasks.
K L INEAR E VALUATION
Here we describe the setup for linear evaluation.
We follow (Kolesnikov et al., 2019) and evaluate the frozen representation by training a linear logistic regression model.
We use exactly the same hyperparameters as described in lightweight sweep from Section I.
The only difference here is that only the linear layer is trained instead of ﬁne tuning the whole network
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
e n d a ll R a n k
C o rr e l a t i o n
Full Datasets
C a lt ec h
C
I F
A
RD T D F l o w e r s
P e t s
S VHN S un397 C a m e l yon
E u r o S
A T
R e s i s c
R e ti nop a t hy
C l e v rC oun
t
C l e v rD i s t
D M - L a b
K I TT
I- D
i s t d S p r-
L o c d S p r-
O r
i s NO
RB - A z i m s
NO
RB - E l e v
M ea n
[Figure]
[Table]
[Figure]
[Table]
L S CALING U P THE A RCHITECTURE
In this section
[Figure]
we study the problem of “is scaling up the architectures helpful?”.
Figure 17 shows top-1 accuracy on ImageNet public validation set when scaling up the architectures.
Widening factor is the multiplier on the network width, where × 1 stands for the standard ResNet architecture.
Depth stands for the number of architecture layers.
As expected, the model accuracy goes up with either wider or deeper architectures.
Figure 18 shows the results on VTAB benchmark
[Figure]
where the 2x wider
ResNet152 architecture performs consistently better than the standard ResNet50 model.
[Figure]
[Figure]
width is represented by different circle size.
The performance increases when switching from the standard ResNet50 architecture to ResNet152 2x wider architecture.
M B UDGET A NALYSIS
Figures 19 and 20 show the top-1 accuracy attained by S UP -100% and R OTATION using different schedule lengths when using 1000 samples or the full dataset, respectively
[Figure] [Figure]
N C OMPARISON TO V ISUAL D ECATHLON
F l o w e r
s
s N O
R B - E l e v
C a l t e c h 101 P e t s
U C F dS p rO r i e n t
A i r c r a f t
D P e d
O g l
o
t dSp r - L o c
D M L a b s N O
R B - A z i m
C l e v r - D
i s t
R e s i s c
S un
E u r o
S A T
C a m e l y o n
G T S
R
D T D
S V H
N
I m
N e t
R e t i n o p a t h y
K I TT
I - D i s t
C l e v r - C o un t
C
I F
A
R - −
[Figure]
−
[Figure]
−
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
In Section 4 we describe the differences between VTAB and Visual Decathlon protocols.
Here we answer a more practical question: which benchmark should one use to compare the adaptation abilities of a set of methods to unseen tasks?
We will show that the rank correlation with an unseen task, is expected to be higher for the ranking obtained using VTAB, than using Visual Decathlon.
First, we ﬁne-tuned each of our 16 baseline models in each of the Visual Decathlon datasets (using the lightweight hyperparameter search described in Section 3.1), which we downloaded directly from the competition’s website.
All datasets in the Visual Decathlon are provided so that the shorter size of the image is 72 pixels, while our baseline models were trained in much larger resolutions.
To overcome this difference in resolution, we resize the Visual Decathlon images to the resolution required by each model.
We checked that our baseline models obtained reasonable results on the
Visual Decathlon benchmark.
In fact, our best model, SUP - ROTATION -100%, reports a test average accuracy of 78.22%, and a decathlon score of 3580, which are both slightly better than the best results reported in Rebufﬁ et al.
(78.08% and 3412, respectively), and other works (e.g.
Rebufﬁ et al.
; Rosenfeld & Tsotsos (2018)).
However, notice that comparing these numbers is delicate, since we did not use any data augmentation during training and all our models are based on the Resnet50 architecture, while these works use heavy data augmentation (that depends on the dataset), and Resnet26-like architectures.
Then, for each task T in the union of VTAB and Visual Decathlon, we rank the 16 baseline methods according to their accuracy on task T .
If we consider T as the unseen dataset, this is the “gold” ranking of the studied methods.
Now, we obtain two alternative rankings: one based on the mean accuracy in VTAB and another on Visual Decathlon, excluding task T in both cases to avoid any bias.
We can then compute, for each “unseen” dataset, the Kendall’s ranking correlation between the “gold” ranking and each of the alternative rankings.
Figure 21 shows the absolute differences in the rank correlation score between the VTAB-based ranking and Visual Decathlon-based ranking.
For most datasets, the difference is positive, which means that the ranking according to VTAB correlates better with the “gold”, than the raking obtained using Visual Decathlon.
Notice that even tasks that were not part of VTAB (colored in gold), are better represented by VTAB’s ranking than that of Visual Decathlon. are
, since contains a more representative of than
Decathlon and