Paper:
Most classifiers operate by selecting the maximum of an estimate of the conditional distribution p(y|x) where x stands for the features of the instance to be classified and y denotes its label. This often results in a hubristic bias: overconfidence in the assignment of a definite label. Usually, the observations are concentrated on a small volume but the classifier provides definite predictions for the entire space. We propose constructing conformal prediction setsVovk et al. (2005) which contain a set of labels rather than a single label. These conformal prediction sets contain the true label with probability 1 − α. Our construction is based on p(x|y) rather than p(y|x) which results in a classifier that is very cautious: it outputs the null set -meaning "I don't know" -when the object does not resemble the training examples. An important property of our approach is that classes can be added or removed without having to retrain the classifier. We demonstrate the performance on the ImageNet ILSVRC dataset and the CelebA and IMDB-Wiki facial datasets using high dimensional features obtained from state of the art convolutional neural networks.

Review:
The paper proposes an approach to construct conformal prediction sets. The idea is to estimate p(x|y) and then construct a conformal set based on the p(x|y) instead of p(y|x). The paper claims that such a method produces cautious classifiers that can produce "I don't know" answers in the face of uncertainty.
However,
A] Although the paper is titled is "Cautious Deep Learning", the method is broadly applicable to any classifier, there is nothing in the method that restricts it to the domain of deep learning. A broad spectrum evaluation could have been done on standard multi-class classifiers.
B] The paper provides multiple qualitative evaluation results. While it gets the point across, I still would have liked to see a quantitative evaluation, for e.g., there have been several papers that proposed generating adversarial examples for deep learning. The author could have taken any of those methods, generated adversarial examples for deep learning and compared the original classifier with the conformal prediction set. Also, such comparison would have made the paper more connected with deep learning.
C] The paper uses Xception network as a feature extractor and then compares its result with Inception-v4. Honestly, I would have preferred if the comparison was between 1] Xception Feature Extractor + Conformal Set Prediction, 2] Xception network prediction, and 3] Inception-v4. The reason being that it is very difficult to understand how much of the cautious-ness is because of the proposed approach and how much is due to the Xception network being good. For example, in Figure 3b, does Xception network generate high probability values for the top classes or does it generate low probability values? Unless we can understand this difference, it is very difficult to appreciate what this approach is giving us.
D] Another analysis that could have been done is to apply this approach and use several different pre-trained networks as feature extractor and check whether there is a decrease in false positives across all the networks, that would suggest that the method can truly make deep learning cautious across a wide variety of networks.
E] Another analysis that could have been done is understand the impact of the quality of feature extractor. For example, take a deep network (of sufficient depth) and use the proposed approach but instead of using just the penultimate layer for feature extraction, one can keep on removing layers from the end and use the remaining as the feature extractor. Then analyze the quality of conformal predictions as each layer gets removed. One can understand the robustness of this method.
Even though doing all these evaluations may be a tad too much, but definitely, quite a few of those could have been done to make the approach look convincing and enticing. I think bulking this paper with such analysis could make for a very good submission. However, as it stands, it still quite lacks.