Paper:
LABEL SUPER-RESOLUTION NETWORKS
We present a deep learning-based method for super-resolving coarse (lowresolution) labels assigned to groups of image pixels into pixel-level (highresolution) labels, given the joint distribution between those low-and highresolution labels. This method involves a novel loss function that minimizes the distance between a distribution determined by a set of model outputs and the corresponding distribution given by low-resolution labels over the same set of outputs. This setup does not require that the high-resolution classes match the lowresolution classes and can be used in high-resolution semantic segmentation tasks where high-resolution labeled data is not available. Furthermore, our proposed method is able to utilize both data with low-resolution labels and any available high-resolution labels, which we show improves performance compared to a network trained only with the same amount of high-resolution data. We test our proposed algorithm in a challenging land cover mapping task to super-resolve labels at a 30m resolution to a separate set of labels at a 1m resolution. We compare our algorithm with models that are trained on high-resolution data and show that 1) we can achieve similar performance using only low-resolution data; and 2) we can achieve better performance when we incorporate a small amount of highresolution data in our training. We also test our approach on a medical imaging problem, resolving low-resolution probability maps into high-resolution segmentation of lymphocytes with accuracy equal to that of fully supervised models.

Review:
Paper summary: 
This paper presents a deep-learning based method for super-resolving low-resolution labels into high-resolution labels given the joint distribution between those low- and high- resolution labels. This is useful for many semantic segmentation tasks where high-resolution ground truth data is hard and expensive to collect. Its main contribution is a novel loss function that allows to minimize the distance between the distribution determined by a set of model outputs and the corresponding distribution given by low-resolution label over the same set of outputs. The paper also thoroughly evaluates the proposed method for two main tasks, the first being a land cover mapping task and the second being a medical imaging problem.
For the land cover application, adding low-resolution data to high-resolution data worsens the results when evaluating on the geographic area from which the high-resolution data was taken. However, when testing the model on new geographic areas and only adding the low-resolution data from this new area in training makes significant improvements.
Generally the paper is very well written, well structured, all explanations are clear, examples and figures are presented when needed and convey helpful information for the reader. The overall idea is fun, original, useful (especially in remote sensing) and is presented in a a convincing way. All major claims are supported by experimental evaluation. There are nevertheless a few concerns:
Major Concerns:
On a conceptual level, the main concern is that the paper assumes we are given a joint distribution of low and high resolution labels, “where we are given the joint distribution P(Y,Z)”, which seems the main limitation of this method. In fact, to correctly estimat this joint distribution either requires additional knowledge about low-resolution data such as the example presented on the NCLD data : “For instance, the “Developed, Medium Intensity” class [...] of the coarse classes”, or it requires actual high-resolution labelled data to correctly estimate this joint distribution. I think the paper would greatly benefit from including a section that discusses the impact of this limitation.
Another point is footnote 3 on page 5. This argument is valid but it would be more convincing to give a thorough explanation on why the choice of the presented loss function is better compared to the KL divergence based loss function or at least some evidence that the two perform similarly when evaluating the method.
Minor Concerns: 
-	“such as CRFs or iterative evaluation” I would include a citation on this type of work.
-	Format of some references in the text need to be corrected, e.g. “into different land cover classes Demir et al. (2018); Kuo et al. (2018); Davydow et al. (2018); Tian et al. (2018).”