Paper:
We present a technique to improve the generalization of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions. Although recent research has shown benefits of self-supervised learning (SSL) on large unlabeled datasets, its utility on small datasets is unknown. We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets. The improvements are greater when the training set is smaller or the task is more challenging. Though the benefits of SSL may increase with larger training sets, we observe that SSL can have a negative impact on performance when there is a domain shift between distribution of images used for meta-learning and SSL. Based on this analysis we present a technique that automatically select images for SSL from a large, generic pool of unlabeled images for a given dataset using a domain classifier that provides further improvements. We present results using several meta-learners and self-supervised tasks across datasets with varying degrees of domain shifts and label sizes to characterize the effectiveness of SSL for few-shot learning.

Review:
Summary:
There are three main contributions of the paper:
i) The authors present an empirical study of different self-supervised learning (SSL) methods in the context of self-supervised learning.
ii) They point out how SSL helps more when the dataset is harder.
iii) They point out how domain matters while using SSL for training and present a method to choose samples from an unlabeled dataset.
Strengths:
1) They confirm the results of [7] and provide additional evidence of the benefit of the self-supervised learning in the few-shot setting.  They also showcase an interesting new result that self-supervised learning helps more in case of harder problems. 
2) The authors have a done a commendable job of coming up with a meaningful set of experiments by varying base-models, self-supervised methods, datasets, and few-shot learning methods in Section 4.1. This is quite a comprehensive study. 
3)  The paper is well-written and well-motivated.
Weaknesses:
1) The main weakness of the paper is Section 4.2's experimental setup. 
i) The definition of domain distance in not quite meaningful. Since the chosen datasets have very different classes (airplanes vs dogs etc), the average embeddings for different datasets/classes will be far from each other.  In Figure 4d, it is misleading to show a trendline that includes the same domain as that will always be 0. If that datapoint is removed the trend line is mostly flat. The authors want to present a quantifiable way to show how domain distribution affect performance on self-supervised learning methods. But this definition of domain distance is more meaningful in the domain adaptation setting (like Amazon-Office dataset used for domain adaptation (https://people.eecs.berkeley.edu/~jhoffman/domainadapt/)) as in that case the requirement is to get the embeddings close to each other for the same class but from different domains.
ii) The authors go on to create an "unlabeled pool" by combining images from many domains. Then they train a domain classifier by labeling in-domain images as positive and  labeling the images in pool as negative. Considering all images in the pool as negative is not correct as there can be images of same class in unlabeled pool. 
iii) Then they choose the samples which the classifier predicts comes from the same domain. This probably succeeds as it is done on top of ResNet-101 features (that has seen all of ImageNet). This technique probably works possibly due to the ResNet being pre-trained with so many labeled classes. One baseline might have just been to choose the k-nearest neighbors (from unlabled pool)  to the average embedding of all the images of the chosen dataset. Since, it is quite easy for classifiers to detect from which dataset an image came from [6] it might be easy to choose an image from similar domain by using nearest neighbor in the embedding space. 
I am not sure how well their heuristic will work for an unlabeled pool that the classifier has never seen. Additionally, a portion of the dataset has been created by combining existing datasets. Since statistics of different datasets vary a lot artificially, the creation of an unlabeled pool by combining different datasets might work in the favor of the proposed heuristic of looking at "domain distance" to choose the samples.
2) Effect of domain shift in SSL (Figure 4b) is studying an extreme case where the domain shift results in almost no common classes in the SSL training phase. It would make more sense to include datasets where at least some of the classes are shared so that self-supervised learning methods get to see some relevant classes. In practice, a self-supervised learning method would be applied on a large unlabeled pool of images. Hopefully with increasing diversity and number of images, there might be some images on which ding SSL helps the downstream few-shot task. Hence comparison with such a dataset like ImageNet/iNatrualist is important here.
Decision:
The paper presents an well thought-out empirical study on self-supervised learning for few-shot learning. But there are  major concerns with the empirical setup and methods presented in the section where they propose a method to choose samples for SSL from an unlabeled pool of images. 
Minor Comments:
1)  “In contrast, we humans can quickly learn new concepts from limited training data” - Remove we. 
2) “Despite recent advances, these techniques have only been applied to a few domains (e.g., entry-level classes on internet imagery), and under the assumption that large amounts of unlabeled images are available.” This is not true. Self-supervised learning methods have been used for continuous control in reinforcement learning[1, 2], cross-modal learning[3], navigation [5], action recognition[4] etc. Later on in related work the authors list many papers that use self-supervised learning in different contexts. This line should be modified to reflect how common self-supervised learning methods are in other fields as well and with less data.
3) “making the rotation task too hard or too trivial to benefit main task” - add respectively.
4) "With random sampling, the extra unlabeled data often hurts the performance, while those sampled using the “domain weighs” improves performance on most datasets." replace weighs with weights
5) [8] points out how self-supervised learning on a large dataset but with a domain shift (YFCC100M) is not as effective for pre-training as it is doing self-supervised learning on the downstream task's dataset (ImageNet). While it is a different setting it is inline with one of the main conclusions of the paper.
References:
[1] “PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations” Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin Riedmiller.
[2] “Learning Actionable Representations from Visual Observations” Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, Pierre Sermanet
[3] “Look, Listen and Learn” Relja Arandjelović, Andrew Zisserman
[4] “Self-supervised Spatiotemporal Learning via Video Clip Order Prediction” Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, Yueting Zhuang.
[5] “Scaling and Benchmarking Self-Supervised Visual Representation Learning” Priya Goyal, Dhruv Mahajan, Abhinav Gupta, Ishan Misra
[6] "Unbiased Look at Dataset Bias" Antonio Torralba and Alexei A. Efros.
[7] "Boosting ´few-shot visual learning with self-supervision." Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Perez, and Matthieu Cord. 
[8] "Deep clustering for unsupervised learning of visual features" Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze