Paper:
The detection of out of distribution samples for image classification has been widely researched. Safety critical applications, such as autonomous driving, would benefit from the ability to localise the unusual objects causing the image to be out of distribution. This paper adapts state-of-the-art methods for detecting out of distribution images for image classification to the new task of detecting out of distribution pixels, which can localise the unusual objects. It further experimentally compares the adapted methods on two new datasets derived from existing semantic segmentation datasets using PSPNet and DeeplabV3+ architectures, as well as proposing a new metric for the task. The evaluation shows that the performance ranking of the compared methods does not transfer to the new task and every method performs significantly worse than their image-level counterparts.

Review:
# Summary
This paper puts forward a study over out-of-distribution (OOD) detection for semantic segmentation. OOD detection is an active area research which has recently dealt mostly with image level classification. For semantic segmentation the same conclusions might not apply since decisions must be taken for each pixel individually. To this effect the authors propose here to study this task over a set of architectures (PSPNet, DeepLabV3+), outlier datasets (SUN, Indian Driving Dataset, synthetic images) and multiple methods for OOD detection from recent works on image classification. 
A major difficulty in OOD works is the definition of a relevant OOD dataset and evaluation setup, and the authors propose here a novel setup for this task by adjusting the SUN and IDD datasets as OOD for Cityscapes. 
The experimental part is thorough with multiple evaluation metrics and some qualitative examples and discussion. 
# Rating
Although the paper studies a meaningful and interesting problem, I would reject the paper for the following reasons (more detailed arguments some lines below):
1) I find that the proposed dataset setups for OOD detection are not adequate for semantic segmentation and thus alter the reliability of the results.
2) Other than the evaluation (which has some faults), there is no proposed method addressing this task and its challenges.
3) The proposed MaxIoU metric proposed is not discussed and compared in more detail against the usual ones for this task.
# Strong points
- This work is dealing with a highly interesting and challenging problem. Indeed there has been few studies in this area and defining proper techniques and evaluation setups is challenging.
- The authors consider a wide array of methods for evaluation and test them across two architectures and multiple real and synthetic datasets.
# Weak points
- There are two real datasets considered for OOD evaluation, but I consider there are some flaws in their utilization for OOD detection. First, the SUN dataset is quite different from Cityscapes with a significant domain gap (at least in the visual appearance and distributions of the classes) between the two datasets. Upsampling the SUN images to 5x their size in order to make them compatible to Cityscapes should increase the artefacts even further making it easier to spot the gap between SUN and Cityscapes. This means that there is a risk of having the OOD detector acting merely as domain classifier spotting whenever the domain is different from the one use for training. 
This argument applies to IDD, although to a smaller extent as the datasets are both automotive, but again there are strong visual differences between samples of the same class across the two datasets.
The authors argue that they somehow take this argument into consideration (Figure 2) and select only the non-ID classes to perform evaluation. However in both networks the scene information is mixed into the representation (via pyramid pooling in PSPNet  respectively via a trous convolutions with different dilations in DeepLabV3+). So again, instead of an OOD detector we can end up doing domain classification.
It would be useful to see how is the classification performance changing for ID classes, e.g. how is a model trained on Cityscapes scoring for cars and other ID objects in IDD comparing to a model that is trained on IDD for the same classes. A big difference between these scores would correspond to a significant domain gap and in correlation with the OOD performance we might be able to take some more conclusions on the matter. 
Ahmed and Courville[i] propose an interesting discussion on this type of problems and propose focusing on semantic anomaly detection, i.e. detecting different classes from the same dataset, to make sure the setting has practical interest. They propose a very basic technique to detect OOD in previous classification setups showing the limitations of previous OOD methods. I encourage the authors to check the arguments stated in that work.
- In my opinion, the Fishyscapes work from Blum et al. is unfairly dismissed here by considering only a part of the benchmark for which animals from COCO and internet are inserted over Cityscapes images. The authors argue that this lack of realism of the inserted images make this dataset insufficient for OOD detection. However in the first version of their paper, Blum et al. propose a mix of Foggy Driving, Foggy Zurich, WildDash and Mapillary as dataset for ODD detection, which is similar with the setup proposed here. Furthermore, the latest version of Fishyscapes includes the Lost & Found dataset (mentioned in Fig. 1 here) which is recorded in similar conditions with Cityscapes with the addition of a few small outlier objects used as OOD. This is a relevant dataset and work and I would adjust the critics brought to their work here. That paper has the same objectives and endeavors as the current submission.
- Although there are some discussions and experiments on multiple techniques there is not technical contribution mitigating all the limitations of previous OOD methods on classification and the challenges of OOD detection in semantic segmentation. This would have had greatly helped the paper.
- I find that the MaxIoU metric considered here is not sufficiently discussed and analysed to show its utility and the additional perspective it brings when evaluating along with the usual metrics.
## Other less important weak points
- The choice of dataset in Figure 1 , i.e. Lost & Found, can be misleading. This dataset is not further mentioned and evaluated in the rest of the paper. The image could be replaced with a qualitative results from the rest of the evaluation.
# Suggestions for improving the paper:
1) The current evaluation setting could have some flaws. I would propose some sanity checks and look at the classification performances over other ID classes, as suggested in the previous section
2) Evaluate on a setting similar to Fishyscapes Lost and Found, in which the dataset does not change much, but there are some novel objects. 
3) Include a trivial OOD baseline in the spirit of [i] to show the utility of the proposed datasets for this task by being robust to such baselines.
4) Consider extending the breadth of OOD methods with Deep Prior Networks[ii] which have been shown to perform well on Fishyscapes for OOD detection.
5) Add a qualitative example with OOD detection on Perlin noise images
# References
[i] F. Ahmed and A. Courville, Detecting semantic anomalies, arxiv 2019 https://arxiv.org/abs/1908.04388
[ii] A. Malinin and M. Gales, Predictive uncertainty estimation via prior networks, NeurIPS 2018