Paper:
Domain Separation Networks
The cost of large scale data collection and annotation often makes the application of machine learning algorithms to new tasks or datasets prohibitively expensive. One approach circumventing this cost is training models on synthetic data where annotations are provided automatically. Despite their appeal, such models often fail to generalize from synthetic to real images, necessitating domain adaptation algorithms to manipulate these models before they can be successfully applied. Existing approaches focus either on mapping representations from one domain to the other, or on learning to extract features that are invariant to the domain from which they were extracted. However, by focusing only on creating a mapping or shared representation between the two domains, they ignore the individual characteristics of each domain. We hypothesize that explicitly modeling what is unique to each domain can improve a model's ability to extract domain-invariant features. Inspired by work on private-shared component analysis, we explicitly learn to extract image representations that are partitioned into two subspaces: one component which is private to each domain and one which is shared across domains. Our model is trained to not only perform the task we care about in the source domain, but also to use the partitioned representation to reconstruct the images from both domains. Our novel architecture results in a model that outperforms the state-of-the-art on a range of unsupervised domain adaptation scenarios and additionally produces visualizations of the private and shared representations enabling interpretation of the domain adaptation process.

Review:
This paper proposes to learn both common and domain-specific representations for domain adaptation from source domain to target domain. Experiments on digits and traffic sign dataset show good performance (even though not convincing even fatal flawed, see below). 
------Initial Review----- This paper proposes Domain Separation Networks that learn both common and domain-specific representations for domain adaptation. I like the idea of domain separation even though it is not new. However, I do not think the current draft sufficiently validate the proposed approach. First of all, neither the experimental settings nor the datasets used are standard. To have a direct comparison to other methods (e.g., [7], [29], [17], [26], etc.), standard settings and benchmark datasets should be used. For the current draft, it is not clear the better performance is due to better hyper-parameter tuning on the validation set or from the proposed model. What’s more, since the baseline methods in the original papers are NOT tested on these datasets, it might be that with proper tuning of parameters on the validation set (that not available in the standard setting), the baseline methods achieve better performance. Thus, I encourage the authors to experiment on standard benchmark datasets with the standard protocol. Secondly, I do not think pixel-wise reconstruction is widely applicable. It might work for small gray-level digits data, but the space of real images is much larger (e.g., 256^5 for 256*256 RGB image). What’s more, pixel-level reconstruction prohibits invariance of features. Higher level reconstruction (e.g., conv3) might be better. Last but not least, as the model is very complicated, I would expect the author to provide sensitivity analysis of parameters/hyper-parameters and both mean and std of the accuracy by running the experiments multiple times.   ------Additional Review after authors' feedback------ After reading all the reviews and authors' feedback, I think the CURRENT draft is not suitable for publication at NIPS.   My concerns are for both the approach and the evaluation (experimental setting is flawed). I REALLY hope the authors would have taken the advice of three reviewers and test it on some real images of 3D objects. Since the authors did not provide any meaningful feedback (at least to my three points), I just have to delve deeper into the approach. Even though the novelty is limited, I liked the combination before. However, further reading confirms my opinion that the current draft has not sufficiently validated the approach. My personal conclusion now is that the proposed approach is NOT going to work on read image of 3D objects with DEEP networks. For classification of the digits/traffic signs, it is a pretty much solved problem and the experiment setting is also flawed.   For experiment settings, I think it is flawed. All the baseline methods are designed to be purely UNSUPERVISED, NO labelled TARGET data is used anyway (training or validation). In this paper's setting, there are >1000 labeled TARGET data used for VALIDATION. As showed in recently published methods (e.g., ladder networks, virtual adversarial, etc.), 1000 (10/category) labeled images for digits are enough to train a strong classifier, might even stronger than the number reported in this paper with the more complicated adaptation method. I really can NOT see any point of using that large portion of labelled TARGET data to tune parameter for some complicated methods rather than train a much stronger and simpler classifier on it directly. Thus, in the 5 toy experiments designed by the authors, no adaptation (e.g., ladder networks, virtual adversarial, etc.) might achieve better performance than the methods proposed by the authors with adaptation. What's the point of use it as a validation set rather than train a classifier on it?  A not very accurate analogy would be putting a gas powered generator on an electrical car to generate electricity for the car. Why can not I buy a car powered by gas directly? It is more energy efficient.  For the authors' feedback, some of the issues pointed by fellow reviewers are very easy to address. However, the author just provided vague discussion instead of quantitative analysis/results. For example, the mean and std can be provided by running the same experiments many times. For the sensitivity test, the authors claim to be less sensitive than DANN. What's do you mean by less? How about comparisons to other methods mentioned in the paper?   For the datasets, as mentioned by fellow reviewers, the 5 datasets are rather similar and constrained in several cases. The variance of digits or traffic signs is MUCH lower than real images. They are both 2D objects with pretty much the same viewpoints. For real images of 3D objects, due to change of pose and viewpoints, the same object can appear dramatically in two images. The problem of digits/traffic sign classification is pretty much solved with recent advances (e.g., ladder networks, spatial transformer networks, etc.). What's more, as far as I know, NO paper in the domain adaptation literature RECENTLY published in top conferences tested on digits/traffic signs ALONE. These data might good to generate some plots to illustrate the idea.   What's more, Office is NOT the only benchmark real images dataset of 3D objects. I STRONGLY suggested the authors to test it on standard benchmarks. I do not think the authors' claim of the limitation of Office is a solid response. There are several much larger standard benchmarks beyond Office, e.g., 'A Testbed for Cross-Dataset Analysis', 'ImageCLEF Domain Adaptation Dataset'.  The other thing is that the authors REIMPLEMENTed the other baseline methods despite the fact that most of them have released the source code. The reimplementation is also different from the original one (e.g., gaussian kernel vs. RBF kernel). What's more, some baseline methods do NOT have hyper-parameter (e.g., [26]), how do you tune it on the validation set? How do I know whether the comparison is fair or not?  For pixel-wise reconstruction, due to the much larger space and variance of real images of 3D objects, I do NOT think it is going to work. A reasonable analogy would be autoencoder/decoder. The autoencoder/decoder approaches have been shown to be performed well on MNIST digits. However, as far as I know, no one showed pixel-wise autoencoder/decoder work on real images of 3D objects (e.g., ImageNet). For deep methods on real images of 3D objects, the features need to be much more invariant. These invariances make pixel-wise reconstruction impossible. I do not think one can have both at the same time.   For the Domain Separation, the authors claim that 'the shared encoder represents the object and the private encoder represents the background'. Why can not I use some existing segmentation methods to separate the object from the background? It seems much more easier to leverage the existing segmentation methods as the orthogonality constraint might not work for deep networks (see below).  For the orthogonality constraint, I think it is NOT going to work in modern deep frameworks. From LeNet to AlexNet is not just scaling up. There are several techniques that contribute a lot. For example, ReLU, Dropout. I really do not think one can train a deep network without all the techniques. For ReLU, it changes all the negative values to zero. For Dropout, the training and testing phases are quite different. During training, randomly zeroing out some (e.g., 50%) activations. For testing, a weight (e.g., 0.5) is put on ALL activations. In terms of distributions of activations, the training and test are probabilistically the same. However, in terms of orthogonality, they are different. Roughly speaking, there are MORE (since ReLU changes the negative parts to zero) than 50% data are zero after ReLU and Dropout if dropout rate is 0.5. To constrain two nonnegative metrics to be orthogonality, at least one of the two corresponding entries need to be zero. This might happy during training since more than 50% data are zero. However, it should be VERY UNSTABLE since for different iteration Dropout will zero out DIFFERENT dimensions. However, during testing as a weight (e.g., 0.5) is put on every dimension rather than zeroing out 50%, most of the data will be positive. The zeros can only be from ReLU not Dropout. For two nonnegative matrices with ~90% positive data, I think it is VERY UNLIKELY to be orthogonal.   In summary, even though the writing is excellent, I do NOT think the CURRENT draft is suitable for publication at NIPS.  I recommend for a major revision