Paper:
LEARNING FACTORIZED REPRESENTATIONS FOR OPEN-SET DOMAIN ADAPTATION
Domain adaptation for visual recognition has undergone great progress in the past few years. Nevertheless, most existing methods work in the so-called closed-set scenario, assuming that the classes depicted by the target images are exactly the same as those of the source domain. In this paper, we tackle the more challenging, yet more realistic case of open-set domain adaptation, where new, unknown classes can be present in the target data. While, in the unsupervised scenario, one cannot expect to be able to identify each specific new class, we aim to automatically detect which samples belong to these new classes and discard them from the recognition process. To this end, we rely on the intuition that the source and target samples depicting the known classes can be generated by a shared subspace, whereas the target samples from unknown classes come from a different, private subspace. We therefore introduce a framework that factorizes the data into shared and private parts, while encouraging the shared representation to be discriminative. Our experiments on standard benchmarks evidence that our approach outperforms the state of the art in open-set domain adaptation.

Review:
The paper addresses the problem of Domain Adaptation (DA) in an open setting (OSDA): while traditional DA assumes that the set of classes of the source and the target are identical, in Open-set DA, there are samples in the target which do not belong to any class in the source (unknown classes that I will outliers in this review). The main difficulty of Open-set DA is to simultaneously discard outliers and correctly classify other samples in the target. There are only two papers on Open-set DA so far, Busto'17 and Saito'18.
The method proposed by the authors can be summarized in a single equation, eq. 2, where they aim at learning a linear mapping to a latent space, which can be separated into two sub-spaces U (private space) and V (shared space) such that target outliers will be mapped to 0 in V while source and target non-outliers will be mapped to 0 in U, and hence separate outliers with non-outliers. To solve eq. 2, the authors convert it to Eqs. 3, 4, and 5 and apply techniques in Lee'07 and Mairal'14. The authors propose an extension for learning a linear classifier simultaneously and an extension for incorporating also unknown source classes (i.e. source outliers) when appropriate. An experimental evaluation on 2 datasets show the good performance of the method.
Pros:
-A novel method for a rather new and understudied so far, the work is then interesting for this setting
-Good results reported
Cons:
-The criterion used for choosing when examples are outliers seems heuristic, more discussion would be welcomed as well as some qualitative analysis for showing the interest of the method
-Existing baseline of Saito'18 not used in the 1st experiment
-Some parts require more justification
*Comments:
-The idea of the method is similar to the one of Jia'10 (Eq.6) for multi view learning, but this is rather new for Open-set DA.
-In order to separate target samples to either private or shared, the authors "encourage that either of these two parts (i.e. vectors T_i^u and T_i^v) goes to zero for each sample", which is reasonable. To achieve this the authors use sparse coding method coming from Lee'07. However, this does not make sense to me, because the sparse coding algorithm will encourage both T_i^u and T_i^v to be sparse, but nothing forces one of them to go to the zero-vector.
The authors should then better justify this choice. In particular, I wonder if adding explicitly the criterion used for identifying outliers as a new constraint to satisfy. Then, the optimization problem considered would make more sense to me.
Anyway, the authors could perform additional experiments to show the effectiveness of their method: (i) apply on a classic DA problem where we will expect that ||T^u|| or ||U|| (private subspace for outliers) should be close to zero. 
Add a qualitative analysis on the values of  |T^u|| and |T^v|| - both in Open-set DA and classic DA - showing that the results are as expected. 
- The 1st method (Eq.2) learns the latent space without using any label in the source (i.e there are only two labels: outlier or non-outlier, and all source samples are labeled non-outlier). Thus, the authors resort to the assumption that outliers are farther from source samples than non-outliers. This assumption is strong and may not hold in practice for two reasons: (1) the domain shift can be large and (2) without clustering techniques, many outliers can easily fall into the safe non-outliers zone (consider 0-4 for outliers and 5-9 for non-outliers, high chance this method will incorrectly classify 0 or 3 as non-outliers since 6,8,9 are already non-outliers). 
- The Lagrange dual method (Lee'07, Eq. 6) solves an optimization problem with multiple quadratic constraints, i.e. ||U_j||^2 \le c for every j. However, the authors apply it to solve a problem (eq. 3 and 4) with a single linear constraint which is not quadratic: \sum ||U_j|| \le 1. Please explain:
(i) Why do you use that constrain instead of the one in Lee'07?
(ii) With your constrain, does the Lagrange dual method still work? 
-The authors mention that they reported the results reported by Busto'17 in their experiment. Does this mean that the experiments were not reproduced? If so this seems rather unfair for other baselines since they may have worked on different instances. 
Many baselines are not specific to Open-set DA, so it is rather expected to see bad results.
Since OSDA is new, it is true that there exists only two true baselines: Busto'17 and Saito'18. However, Saito'18 does not appear in BCIS benchmark (although appears in Office benchmark). Please add Saito'18 to the BCIS benchmark.
-The authors use fixed parameters for all the subproblems, I am a bit surprised by this choice, I would rather expect a parameterization task-dependent. Does this mean that the method is hard to tune ?
-The method seems complex, is there any convergence guarantee?
--
After rebuttal: thanks many points were answered.