Paper:
When training a deep neural network for supervised image classification, one can broadly distinguish between two types of latent features of images that will drive the classification of class Y . Following the notation of Gong et al. (2016), we can divide features broadly into the classes of (i) 'core' or 'conditionally invariant' features X ci whose distribution P (X ci |Y ) does not change substantially across domains and (ii) 'style' or 'orthogonal' features X ⊥ whose distribution P (X ⊥ |Y ) can change substantially across domains. These latter orthogonal features would generally include features such as position, rotation, image quality or brightness but also more complex ones like hair color or posture for images of persons. We try to guard against future adversarial domain shifts by ideally just using the 'conditionally invariant' features for classification. In contrast to previous work, we assume that the domain itself is not observed and hence a latent variable. We can hence not directly see the distributional change of features across different domains. We do assume, however, that we can sometimes observe a so-called identifier or ID variable. We might know, for example, that two images show the same person, with ID referring to the identity of the person. In data augmentation, we generate several images from the same original image, with ID referring to the relevant original image. The method requires only a small fraction of images to have an ID variable. We provide a causal framework for the problem by adding the ID variable to the model of Gong et al. (2016). However, we are interested in settings where we cannot observe the domain directly and we treat domain as a latent variable. If two or more samples share the same class and identifier, (Y, ID) = (y, id), then we treat those samples as counterfactuals under different style interventions on the orthogonal or style features. Using this grouping-by-ID approach, we regularize the network to provide near constant output across samples that share the same ID by penalizing with an appropriate graph Laplacian. This is shown to substantially improve performance in settings where domains change in terms of image quality, brightness, color changes, and more complex changes such as changes in movement and posture. We show links to questions of interpretability, fairness and transfer learning.

Review:
The paper discusses ways to guard against adversarial domain shifts with so-called counterfactual regularization. The main idea is that in several datasets there are many instances of images for the same object/person, and that taking this into account by learning a classifier that is invariant to the superficial changes (or “style” features, e.g. hair color, lighting, rotation etc.) can improve the robustness and prediction accuracy. The authors show the benefit of this approach, as opposed to the naive way of just using all images without any grouping, in several toy experimental settings.
Although I really wanted to like the paper, I have several concerns. First and most importantly, the paper is not citing several important related work. Especially, I have the impression that the paper is focusing on a very similar setting (causally) to the one considered in  [Gong et al. 2016] (http://proceedings.mlr.press/v48/gong16.html), as can be seen from Fig. 1. Although not focusing on classification directly, this paper also tries to a function T(X) such that P(Y|T(X)) is invariant to domain change. Moreover, in that paper, the authors assume that even the distribution of the class can be changed in the different domains (or interventions in this paper).
Besides, there are also other less related papers, e.g. http://proceedings.mlr.press/v28/zhang13d.pdf, https://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/10052/0, https://arxiv.org/abs/1707.09724, (or potentially https://arxiv.org/abs/1507.05333 and https://arxiv.org/abs/1707.06422), that I think may be mentioned for a more complete picture. Since there is some related work, it may be also worth to compare with it, or use the same datasets.
I’m also not very happy with the term “counterfactual”. As the authors mention in footnote, this is not the correct use of the term, since counterfactual means “against the fact”. For example, a counterfactual query is “we gave the patient a drug and the patient died, what would have happened if we didn’t give the drug?” In this case, these are just different interventions on possibly the same object. I’m not sure that in the practical applications one can assure that the noise variables stay the same, which, as the authors correctly mention, would make it a bit closer to counterfactuals. It may sound pedantic, but I don’t understand why use the wrong and confusing terminology for no specific reason, also because in practice the paper reduces to the simple idea of finding a classifier that doesn’t vary too much in the different images of the single object.
**EDIT**: I was satisfied with the clarifications from the authors and I appreciated the changes that they did with respect to the related work and terminology, so I changed my evaluation from a 5 (marginally below threshold) to a 7 (good paper, accept).