Paper:
Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets. Yet, the absence of a unified yardstick to evaluate general visual representations hinders progress. Many sub-fields promise representations, but each has different evaluation protocols that are either too constrained (linear classification), limited in scope (ImageNet, CIFAR, Pascal-VOC), or only loosely related to representation quality (generation). We present the Visual Task Adaptation Benchmark (VTAB): a diverse, realistic, and challenging benchmark to evaluate representations. VTAB embodies one principle: good representations adapt to unseen tasks with few examples. We run a large VTAB study of popular algorithms, answering questions like: How effective are ImageNet representation on non-standard datasets? Are generative models competitive? Is self-supervision useful if one already has labels?

Review:
###  Summary
1. The paper presents a new benchmark (VTAB) to evaluate different representation learning methods. VTAB mostly consists of a combination of existing classification datasets. (The classification tasks based on Clevr, SmallNorb, and DMLab are introduced by this paper.)
2. It evaluates and compares the existing representation learning method on the proposed benchmark with an extensive hyper-parameter search. 
### Decision with reasons
I vote for accepting this paper (weak accept). 
1. The experimental evaluation in this paper is outstanding. Different methods are compared fairly with extensive hyper-parameter tuning (Heavyweight tuning). 
2. Even though I find the experiment work to be illuminating, I'm not fully convinced that we need more standardized benchmarks. I'm also not convinced by some of the choices made in designing the benchmark (Such as the models can be pre-trained on any data as long as there is no overlap with the evaluation data).  
I find the experimental evaluation to be useful enough for considering accepting this paper, but I will not strongly advocate for acceptance (Unless the authors can better explain why we need yet another benchmark.) 
### Supporting arguments for the reasons for the decision.
1. The paper compares existing representation learning methods in a very convincing way. Even though it misses some baselines (Meta-learning based representation learning), it compares supervised, self-supervised, semi-supervised, and generative methods comprehensively. Moreover, the hyper-parameter selection process for the benchmark is fully defined in a dataset agnostic way (It does not depend on the evaluation dataset). This is more important than it might seem -- fully specifying algorithms is important and currently rarely done in the research literature. 
I'm not aware of any existing work that compares various representation learning methods on a diverse benchmark fairly and comprehensively, and I'm sure I'll be using the results in this paper to support my claims in the future. As a result, I think this work is worth accepting just for its thorough evaluation of existing methods. 
2. However, the paper does not add much to discourse apart from the thorough evaluation. Benchmarks are already prevalent in representation learning, and VTAB doesn't focus on a single well-defined problem with-in representation learning. 
For example, dSprites is a benchmark to evaluate how well algorithms can disentangle the underlying data generation factors. An algorithm that significantly improves results on dSprites would, as a result, be better at learning disentangled representation. VTAB, on the other hand, is not trying to highlight any specific problem in representation. As a result, it's not clear what it means if an algorithm does better on VTAB (Apart from the vague notion that it is learning 'better' representations.) 
This problem is especially exacerbated by the fact that VTAB does not restrict representation learning to a fixed dataset. This means that a trivial way of improving performance is to just get more representative data (Which is not a bad way of improving performance, but not the way that requires scientific study). 
### Questions for Authors 
1. The paper makes a binary distinction between fine-tuning and linear evaluation when in-fact this is more nuanced. I suspect that the best performance would be achieved by feezing layers up to a point (And not just not freezing any layers or freezing everything and training a linear classifier). Did the authors explore other choices, and if not, why do they think the current binary distinction is enough. (From my perspective, it makes sense to freeze the initial layers which might be extracting more local/general features and adapt the later layers. The boundary point should be a hyper-parameter)
2. Currently, VTAB puts no restrictions on the representation learning dataset as long as the representation learning dataset does not over-lap with the evaluation dataset. This is problematic since an easier way of improving results on VTAB is to just collect more data/ data that is more representative of evaluation tasks. One could argue that the point of VTAB is to not use details of evaluation datasets in making any decisions for representation learning, however, it is hard to enforce this. For example, if I augment Imagenet with a medical dataset different from the ones in VTAB, is that fine or does that count as cheating?
I feel that a much better way of defining the benchmark would be to restrict representation learning to Imagenet. 
### Other minor comments 
I don't have many minor comments since the paper is very clear and well written. The author could consider augmenting their graphs with patterns in addition to color to make paper friendly for grey-scale printing and use a more colorblind-friendly color combination.  
### Post discussion update ###
One of my primary concerns was the benchmark didn't restrict representation learning to a fixed dataset. That made it trivial to improve performance: just train on a more representative dataset. However, as authors clarified, any dataset collected conditioned on the test set is against the spirit of VTAB. Moreover, discovering what kind of data improves representations is also a research problem, and by restricting to a fixed pre-training dataset, VTAB would not be suitable for such research directions. 
I agree that solving representation learning might not just be about finding a better algorithm, but a better algorithm coupled with the right data. In that sense, VTAB's decision to allow any data for pretraining makes sense. However, I'm also concerned that the benchmark might push people to (directly or indirectly) start training on data more representative of VTAB evaluation tasks, and the performance gains in the benchmark will not translate to better algorithms. However, I don't see how this can be avoided for an open-ended benchmark. Since the authors have agreed to include guidelines on how to use the benchmark without abusing it, letting the community police itself given the guidelines is not a bad idea. 
I can see VTAB as a benchmark for answering interesting questions (mentioned by authors in the response) such as:
1- How to incorporate large amounts of unlabelled data for representation learning. For example, can we learn better representations by self-supervised learning on a large corpus of data (Such as unlabelled youtube videos)
2- Can we generate data using simulators that can help us learn better representations? 
Finally, the authors agreed to add meta-data in their leaderboards (which they plan to release after the review period). The meta-data would allow us to compare methods at a more granular level, such as comparing different unsupervised learning methods. Overall, I think the benchmark and the leaderboards would act as a good source for comparing different unsupervised, semi-supervised, and supervised learning methods (and their combination) fairly. 
I'm also updating my score from a weak-acceptance to an acceptance.