Paper:
Predicting structured outputs such as semantic segmentation relies on expensive per-pixel annotations to learn strong supervised models like convolutional neural networks. However, these models trained on one data domain may not generalize well to other domains unequipped with annotations for model finetuning. To avoid the labor-intensive process of annotation, we develop a domain adaptation method to adapt the source data to the unlabeled target domain. To this end, we propose to learn discriminative feature representations of patches based on label histograms in the source domain, through the construction of a disentangled space. With such representations as guidance, we then use an adversarial learning scheme to push the feature representations in target patches to the closer distributions in source ones. In addition, we show that our framework can integrate a global alignment process with the proposed patch-level alignment and achieve state-of-the-art performance on semantic segmentation. Extensive ablation studies and experiments are conducted on numerous benchmark datasets with various settings, such as synthetic-to-real and cross-city scenarios.

Review:
This paper introduces a domain adaptation approach for structured output data, with a focus here on semantic segmentation. The idea is to model the structure by exploiting image patches, but account for the fact that these patches may be misaligned, and thus not in exact correspondence. This is achieved by defining new patch classes via clustering the source patches according to the semantic information, and making use of an adversarial classifier on the predicted patch-class distributions.
Strengths:
- Modeling the structure via patches is an interesting idea.
- The proposed method achieves good results on standard benchmarks.
Weaknesses:
Method:
- The idea of relying on patches to model the structure is not new. This was achieved by Chen et al., CVPR 2018, "ROAD: Reality Oriented Adaptation...". In this work, however, the patches were assumed to be in correspondence, which leaves some novelty to this submission, although reduced.
- In essence, the patch-based adversarial alignment remains global; this can be thought of as working at a lower resolution and on a different set of classes, defined by the clusters, than the global alignment. The can be observed by comparing Eq. 3 and Eq. 6, which have essentially the same form. This is fine, but was not clear to me until I reached Eq. 6. In fact, what I understood from the beginning of the paper was an alternative formulation, where one would essentially assign each patch to a cluster and aim to align the distributions of the output (original classes) within each cluster. I suggest the authors to clarify this, and possibly discuss the relation with this alternative approach.
- I am not convinced by the claimed relationship to methods that learn disentangled representations. Here, in essence, the authors just perform clustering of the semantic information. This is fine, but I find the connection a bit far-fetched and would suggest dropping it.
Experiments:
- The comparison to the state of the art is fine, but I suggest adding the results of Chen et al, CVPR 2018, which achieves quite close accuracies, but still a bit lower. The work of Saito et al., CVPR 2018, "Maximum Classifier Discrepancy..." also reports results on semantic segmentation and should be mentioned here. I acknowledge however that their results are not as good as the one reported here.
- While I appreciate the ablation study of Section 4.2, it only provides a partial picture. It would be interesting to study the influence of the exact values of the hyper-parameters on the results. These hyper-parameters are not only the weights \lambda_d, \lambda^g_{adv} and \lambda^l_{adv}, but also the number of clusters and the size of the patches used. 
Summary:
I would rate this paper as borderline. There is some novelty in the proposed approach, but it is mitigated by the relation to the work of Chen et al., CVPR 2018. The experiments show good results, but a more thorough evaluation of the influence of the hyper-parameters would be useful.