Paper:
Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that SSL algorithms would face in real-world applications. After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and performance can degrade substantially when the unlabeled dataset contains out-ofdistribution examples. To help guide SSL research towards real-world applicability, we make our unified reimplemention and evaluation platform publicly available.

Review:
This paper proposes a systematic evaluation of SSL methods, studies the pitfalls of current approaches to evaluation, and, conducts experiments to show the impact of rigorous validation on kinds of conclusions we can draw from these methods.  I really like the paper and read it when it appeared on arXiv back in April. In many places we are lacking these kind of systematic approaches to robust evaluations and it's refreshing to see more of these papers emerging that question the foundation of our validation methodologies and provide a coherent evaluation.  Suggestions for improvements:  - The paper mainly deals with two image categorisation datasets. While these methods have been studied in many recent SSL papers, they also have their own limitations, some of which is mentioned in the paper. But the main problem is that it restricts them to a single domain which is image categorisation. It would have been interesting if we could see how these methods behave on other domains (e.g. NLP tasks, Audio/Speech, ...). - In terms of SSL methods, it would have been interesting to study an instance of graph/manifold based methods (e.g. "Deep Learning via Semi-Supervised Embedding", Weston et al, ICML 2008) as that would've completed the spectrum of traditional assumptions/approaches to SSL (density/margin based models (EntMin/TSVM), self-training (pseudo-labelling), co-training(might not be as applicable as you don't have data with multiple views), manifold based models (LapSVM/SemiEmb)). - Following from this, I would actually suggest that self-training should be also one of the default baselines as it's so trivial to implement that it basically is a good baseline to have around for any task. While pseudo-labelling paper [34] seems like doing that I'm not sure if the implementation details from the paper were followed (e.g. only using pseudo-labelling as a fine-tuning step or using a denoising AE) or you really performed a more classical self-training. - In Sec 4.2 where you improve the performance of the supervised model by utilising [16, 21, 11] and obtain a performance that's comparable to those of SSL with the baseline method, while you acknowledge that it's unfair to make the comparison, but what is preventing you from actually applying SSL on this highly optimised model and see how far down the test error goes? In other words, do SSL methods provide model-invariant benefit or we will only see improvements if we just used the baseline model? - In Sec 4.3, would be interesting to see if you removed the CIFAR-10 classes from ImageNet and re-ran the experiments what you would get out? Does your SVHN conclusion in this section generalises to CIFAR-10? - In Sec 4.5, Although outside of the scope of the paper do you have any intuition why in Fig 4b the performance of VAT is actually very good across different number of labelled samples? - In Sec 4.6, I would suggest to actually do experiments by restricting the size of the validation set to as you say 10% or so of the training set (or perform cross-validation) and report performance on the test set based on hyper-parameter optimisation on the smaller dev set.