From: NeurIPS.cc/2021/Workshop/DistShift/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: Reviews agreed that the setting was relevant and would be an interesting case study for the community. However, we strongly encourage the authors to take into account the following feedback for the camera-ready version:
- The training of the selective classifier (where points that have high loss are ignored) is highly unorthodox. The authors should compare their proposed training method to more standard approaches, such as post-hoc abstention (based, e.g., on entropy or the maximum softmax value) or joint training of the classifier and abstention function. At least, the unusual training scheme should be discussed.
- The section on DRO's robustness to artifacts is interesting but entirely qualitative. A more rigorous evaluation would be useful.
- The ERM vs. DRO results seem somewhat mixed. Could the authors elaborate on why they claim that DRO is performing better?
- The dataset construction and problem setting is unclear. The authors say that "We trained a binary classifier using slide-level labels to classify tiles into tumor or normal tissue." Slide-level labels typically refer to a single label for the entire slide. Do the authors mean that they are using dense pixel-level annotations of tumor vs. normal tissue, or are they using a very weakly supervised dataset? If the latter, then how is accuracy being evaluated at the tile level?