From: NeurIPS.cc/2021/Workshop/DistShift/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: States that in a severely class-imbalanced scenario, self-supervised learning gets similar performance on datasets with different class ratios (balanced vs. heavily skewed) whereas supervised learning has larger performance gaps, albeit still having higher accuracy than SSL for ID on imbalanced sets (except on very small sample sizes). They further conduct an interesting experiment using GradCAM which shows that SSL methods learn to pay more attention to non-spurious features.  The paper is well-written.
Some doubts remain: Even though the gaps are smaller for SL, the absolute accuracies (on a balanced version of the ID dataset) are higher for SL - which is ultimately what you care about. The OOD experiments appear to be slightly misleading - as opposed to the ID experiments, the representations are not held fixed but tuned, hence one cannot claim that "SSL representations perform better in OOD evaluation as in the figure caption" since the representations are changing - at most one can say that SSL-based approach seems to work better than SL on this ( very specific ) far OOD examples. Further, there are some questions that would need to be addressed: how does the choice of the SSL algorithm change the picture? I.e. if you use one of the other SSL algorithms in 3.1., do we see the same trend? What is the effect of hyperparameters, how were they tuned?