L EARNING F ACTORIZED R EPRESENTATIONS FOR O PEN - SET D OMAIN A DAPTATION
A BSTRACT
Domain adaptation for visual recognition has undergone great progress in the past few years.
Nevertheless, most existing methods work in the so-called closed-set scenario, assuming that the classes depicted by the target images are exactly the same as those of the source domain.
In this paper, we tackle the more challenging, yet more realistic case of open-set domain adaptation, where new, unknown classes can be present in the target data.
While, in the unsupervised scenario, one cannot expect to be able to identify each speciﬁc new class, we aim to automatically detect which samples belong to these new classes and discard them from the recognition process.
To this end, we rely on the intuition that the source and target samples depicting the known classes can be generated by a shared subspace, whereas the target samples from unknown classes come from a different, private subspace.
We therefore introduce a framework that factorizes the data into shared and private parts, while encouraging the shared representation to be discriminative.
Our experiments on standard benchmarks evidence that our approach outperforms the state of the art in open-set domain adaptation.
I NTRODUCTION
In many practical machine learning scenarios, the test samples are drawn from a different distribution from the training ones, due to varying acquisition conditions, such as different data sources, illumination conditions and cameras, in the context of visual recognition.
Over the years, great progress has been achieved to tackle this problem, known has the domain shift.
In particular, many methods aim to align the source (i.e., training) and target (i.e., test) distributions by learning domaininvariant embeddings (Pan et al., 2011;
Gong et al., 2012; Fernando et al., 2013; Sun et al., 2016), the most recent approaches relying on deep networks (Ganin & Lempitsky, 2014; Long et al., 2015;
Bousmalis et al., 2016; Tzeng et al., 2017; Long et al., 2016a; Yan et al., 2017).
While effective, these methods work under the assumption that the source and target data contain exactly the same classes.
In practice, however, this assumption may easily be violated, as the target data will often contain additional classes that were not present within the source data.
For example, when training a model to recognize ofﬁce objects from images, as with the popular Ofﬁce dataset (Saenko et al., 2010), one should still expect to see new objects, unobserved during training, when deploying the model in the real world.
While one should not expect the model to recognize the speciﬁc class of such objects, at least in unsupervised domain adaptation where no target labels are provided, it would nonetheless be beneﬁcial to identify these objects as unknown instead of misclassifying them.
This was the task addressed by Busto & Gall (2017) in their so-called open-set domain adaptation approach.
This method aims to learn a mapping from the source samples to a subset of the target ones corresponding to those identiﬁed as coming from known classes.
While reasonably effective, this procedure involves alternatively solving for the mapping and the assignment of the samples to known/unknown classes, which, as shown in our experiments, can be costly.
Recently, Saito et al.
(2018) introduced a deep learning framework for open-set domain adaptation, relying on adversarial training to separate the samples from the known classes from the unknown ones.
In this paper, we introduce a novel approach to open-set domain adaptation based on learning a factorized representation of the source and target data.
In essence, we seek to model the samples from the known classes with a low-dimensional subspace, shared by the source and target domains, and the target samples from unknown classes with another subspace, speciﬁc to the target domain.
We then make use of group sparsity to encourage each target sample to be reconstructed by only one of these subspaces, which in turns lets us identify if this sample corresponds to a known or unknown class.
We further show that we can obtain a more discriminative shared representation by jointly learning a linear classiﬁer within our framework.
Ultimately, our approach therefore allows us to jointly separate the target samples between known and unknown classes and represent the source and target samples within a consistent, shared latent space.
Note that our approach is more intuitive than (Bousmalis et al., 2016) for the open-set DA scenario in the sense that we model each target sample as being generated by either the shared subspace or the private one, which is crucial to identify the target samples depicting unknown classes.
By contrast, in (Bousmalis et al., 2016), each sample is encoded as a mixture of shared and private representations, which does not provide information to discriminate samples from unknown classes.
We demonstrate the effectiveness of our approach on several open-set domain adaptation benchmarks for visual object recognition.
Our method consistently and signiﬁcantly outperforms the technique of Busto & Gall (2017) on all benchmarks, as well as the end-to-end learning approach of Saito et al.
(2018) on the Ofﬁce dataset, thus showing the beneﬁts of learning shared and private representations corresponding to the known and unknown classes, respectively.
Furthermore, it is faster than the algorithm of Busto & Gall (2017) by an order of magnitude.
R ELATED W ORK
Domain adaptation for visual recognition has become increasingly popular over the past few years, in large part thanks to the benchmark Ofﬁce dataset of Saenko et al.
A natural approach to tackling the domain shift consists of learning a transformation of the data such that the distributions of the source and target samples are as similar as possible in the resulting space (Baktashmotlagh et al., 2014; 2013; Sun et al., 2016).
Instead of learning a transformation of the data, other methods have been proposed to re-weight the source samples, so as to rely more strongly on those that look similar to the target ones (Quiñonero C. et al., 2009;
Gong et al., 2013).
With the advent of deep learning for visual recognition, domain adaptation research also eventually turned to exploiting deep networks.
While it was initially shown that deep features were more robust than handcrafted ones to the domain shift (Donahue et al., 2013), translating the abovementioned distribution-matching ideas to end-to-end learning proved even more effective (Tzeng et al., 2014; Long et al., 2015; 2016b; Rozantsev et al., 2018; Sun & Saenko, 2016).
In this context, other ideas were introduced, such as learning intermediate representations to interpolate between the source and target domains (Chopra S. & R., 2013;
Tzeng et al., 2015), the use of adversarial domain classiﬁers (Ganin & Lempitsky, 2014; Tzeng et al., 2017), and additional reconstruction loss terms (Ghifary et al., 2016).
Despite achieving great progress to tackle the domain shift, all the aforementioned methods are designed for the closed-set scenario, where the source and target data depict the exact same set of classes.
Inspired by recent advances in open-set recognition (Bendale & Boult, 2015; Scheirer et al., 2014), the work of Busto & Gall (2017) constitutes the ﬁrst attempt to address the more realistic case where the target data contains samples from new, unknown classes.
To achieve this, Busto & Gall (2017) proposed to jointly learn the assignments of the target samples to known/unknown classes and a mapping from the source data to the target samples depicting known classes.
The resulting learning problem was solved by alternatively optimizing for the assignments and for the mapping, which can be costly.
Very recently, a deep learning approach was proposed for open-set domain adaptation (Saito et al., 2018), relying on adversarial training to separate the unknown target samples from the known ones.
Here, we introduce a new solution to the open-set domain adaptation problem, where we model the source and target data with subspaces.
Subspace-based representations have proven effective for domain adaptation (Gong et al., 2012; Gopalan et al., 2014; Fernando et al., 2013).
Here, however, we exploit them in a different manner, based on the intuition that source samples and target samples from the known classes can be generated by a shared subspace, whereas target samples from unknown classes come from a private subspace.
While the notion of shared-private representations has been exploited in the past, e.g., for multiview learning (Jia et al., 2010) and for closed-set domain adaptation (Bousmalis et al., 2016), the resulting techniques all use them to encode each sample as a mixture of shared and private information.
By contrast, here, we aim to model each target sample as being generated by either the shared subspace or the private one, which is crucial to identify the target samples depicting unknown classes.
Our experiments evidence that our open-set domain adaptation approach, based on shared-private representations, is more effective than the one of Busto & Gall (2017), consistently outperforming it on several datasets, and also faster by an order of magnitude.
We also outperform the recent deep learning open-set domain adaptation framework of Saito et al.
(2018) on the Ofﬁce benchmark.
O UR A PPROACH
The key idea behind our formulation is to ﬁnd low-dimensional representations of the data, factorized into a subspace shared by the source samples and the target ones coming from known classes and another subspace speciﬁc to the target samples from unknown classes.
Note that, when referring to target samples from known classes, we do not mean that these samples are labeled, but rather that they belong to the same set of classes as the source data.
As a matter of fact, throughout the paper, we focus on the unsupervised domain adaptation scenario, where no target annotations are provided.
In the remainder of this section, we ﬁrst introduce the optimization problem at the heart of our approach, and then discuss two extensions of this basic formulation.
3. FRODA: F ACTORIZED R EPRESENTATIONS FOR O PEN - SET D OMAIN A DAPTATION
Given n s source samples, grouped in a matrix X s ∈ R D × n s and n t target samples represented by
X t ∈ R D
× n t , our goal is to estimate a low-dimensional representation of each sample, such that the source and target samples coming from the same classes are generated by a shared subspace, whereas the target data from new, unknown classes are generated by a different, speciﬁc subspace.
To this end, let V ∈ R D
×
d be the matrix encoding the shared subspace, with d (cid:28) D , and U ∈ R D ×
d the one representing the private subspace.
A naive approach to ﬁnding the low-dimensional representations of the data would involve solving
[Equation]
where α sets the relative inﬂuence of both terms, B
= [ V , U ] ∈ R D × 2 d , and T and S encode the low-dimensional representations of the target and source data, respectively.
This simple formulation, however, does not aim to separate the target samples belonging to known classes from the unknown ones, and thus will represent each target sample as a mixture of shared and private information.
Intuitively, we would rather like each target sample to be generated by either the shared subspace V , or the private one U .
To address this, we propose to make use of a group sparsity regularizer on the coefﬁcients of the target samples.
Speciﬁcally, we split the coefﬁcient vector T i for target sample i into a part T vi that corresponds to the shared subspace and a part T ui that corresponds to the private one.
We then encourage that either of these two parts goes to zero for each sample.
To this end, we therefore write the optimization problem
[Equation]
where the constraints prevent the basis vectors of the subspaces from growing while the coefﬁcients decrease (Lee et al., 2007), and where λ 1 is a scalar controlling the strength of the group sparsity regularizer.
In essence, this formulation allows each target sample to be reconstructed from either the shared subspace or the private one, which reduces the inﬂuence of the samples from unknown classes on learning the shared representation.
Optimization.
To solve equation 2 efﬁciently, we alternatively update one variable at a time while keeping the other ones ﬁxed.
Below, we describe the different updates.
Input:
X s ∈ R D
× n s : the source samples
X t ∈ R D
× n t : the target samples d (cid:28) D : the dimensionality of the subspaces
Output:
S ∈ R d × n s , T ∈ R 2 d × n t
Initialize:
V ← PCA ( X s )
U ← Null ( V ) (i.e., truncated null space of V ) 1: Compute T from equation 5 by proximal gradient descent 2: Compute S by solving the linear least-squares problem min S (cid:107) X
s − V S (cid:107) 2 F 3: repeat 4:
Compute U from equation 3 by the Lagrange dual method of (Lee et al., 2007) 5:
Compute V from equation 4 by the Lagrange dual method of (Lee et al., 2007) 6:
Compute T from equation 5 by proximal gradient descent 7:
Compute S by solving the linear least-squares problem min S (cid:107) X s − V S (cid:107) 2 F 8: until convergence
B -minimization:
Given the coefﬁcients S and T , we update the tuple ( U , V ) by solving
[Equation]
(3) with A =
X t − V T v , and min V (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
F s.t
[Equation]
(cid:88)
[Equation]
(cid:107)
[Equation]
(cid:107) [Equation] (4) with A (cid:48) =
X t − UT u .
These two sub-problems can be solved efﬁciently using the Lagrange dual formulation introduced in (Lee et al., 2007) to update the basis in a standard sparse coding context.
T -minimization: Minimizing equation 2 with respect to T , with all other parameters ﬁxed, yields
[Equation]
which can be solved efﬁciently via the proximal gradient method (Mairal et al., 2014).
In other words, for each target sample, we obtain T using group sparse coding to determine to which representation, shared or private, each target sample belongs.
S -minimization: Solving equation 2 with respect to S , with all the other parameters ﬁxed, reduces to a linear least-squares problem, which has a closed-form solution.
To start optimization, we initialize V as the PCA subspace of the source data, and take U as its truncated null space.
We then obtain the corresponding T and S as described above and start iterating.
The pseudo-code of FRODA is provided in Algorithm 1.
The steps are repeated until convergence, which typically occurs around 50 iterations, with each iteration taking roughly 0.05s.
Inference.
After obtaining the ﬁnal 2 d -dimensional representation of T for the target samples, we determine if each sample i belongs to known classes or unknown ones based on the coefﬁcients T ui and T vi .
More speciﬁcally, given a threshold ε , a target sample is assigned to the unknown classes if (cid:107) T vi (cid:107) / (cid:107) T ui (cid:107) ≤ ε , which suggests that it can be well-reconstructed by the private subspace.
In the presence of C known classes, we then train a ( C + 1) -way classiﬁer by augmenting the ddimensional source data S , i.e. S ∈ R d × n s , with the target samples T u identiﬁed as unknown.
3. D-FRODA: D ISCRIMINATIVE FRODA
The formulation above does not make use of the source labels at all during the representation learning stage.
As such, it does not encourage the representation to be discriminative.
To overcome this, we extend our basic formulation to further account for the classiﬁcation task at hand.
Speciﬁcally, let L =
[ l 1 . . .
l n s ] ∈ R C × n s be the matrix containing the source labels, where l i ∈ R C represents the one-hot encoding of the label of sample i .
We then write our D-FRODA formulation as
[Equation]
[Equation]
where W ∈ R C × d is the matrix containing the parameters of a linear classiﬁer for the source data.
Optimization.
To optimize equation 6, we follow a similar alternating strategy as before.
The Bminimization and T -minimization steps are unchanged, but the S -minimization now incorporates a new term and we further need to solve for the classiﬁer parameters W .
This translates to:
S -minimization:
Minimizing equation 6 with respect to S , with all the other parameters ﬁxed, still reduces to a linear least-squares problem.
The two terms involving S can be grouped into a single one of the form (cid:107) X new − V new S (cid:107) 2 F , where X new = [Equation] √ [Equation]
X [Equation] (cid:19)
and V
new = (cid:18) √ α V √ β W (cid:19)
, and thus S can be obtained in closed form.
W -minimization:
With all the other parameters ﬁxed, ﬁnding W corresponds to a linear leastsquares problem, with a closed-form solution.
Inference.
The same inference strategy as before can be followed to label the target samples.
Another option here is to make use of W to classify the samples identiﬁed as belonging to known classes.
We compare these two strategies in our experiments.
3. D-FRODA-U: D-FRODA WITH U NKNOWN S OURCE C LASSES
Until now, we have tackled the scenario where there are no unknown classes in the source data, which we believe corresponds to the typical application scenario, since the source data can in general be fully annotated.
Nevertheless, to match the scenario of Busto & Gall (2017), who assume to have access to additional source samples from unknown classes, yet different from the target unknown classes, we introduce a modiﬁed version of our approach that takes such auxiliary data into account.
Note that, since one knows which source samples are from unknown classes, it is also possible to simply discard them from training.
To nonetheless handle them, we re-write equation 6 as
[Equation]
where X (cid:48) s contains the source samples from both known and unknown classes, and B (cid:48)
=
[ V , U (cid:48) ]
∈ R D × 2 d denotes the source transformation matrix with U (cid:48)
∈ R D
×
d the private subspace for the source data.
Note that, similarly to the target coefﬁcients, we have now separated the source coefﬁcients for each sample S (cid:48)
i into a part corresponding to the shared subspace S (cid:48) vi and a part corresponding to the private one S (cid:48) ui .
Note also that the classiﬁer parameters W (cid:48) now account for C + 1 classes, the additional class corresponding to the unknown samples.
Optimization.
We follow a similar iterative procedure to the one used before, with modiﬁcations to update B (cid:48) and S (cid:48) , as discussed below.
[Table]
[Table] [Table] [Equation] [Table]
[Equation]
D-FRODA
[Table] [Equation] [Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
± [Table] [Equation] [Table]
S (cid:48)
-minimization:
To minimize equation 7 w.r.t.
S (cid:48) , with all other parameters ﬁxed, we write
[Equation]
The ﬁrst two terms can be grouped into a single squared Frobenius norm, thus resulting in a sparse group lasso problem, which, as when updating T in FRODA, can be solved via proximal gradient descent (Mairal et al., 2014).
B (cid:48)
-minimization:
Given the coefﬁcients S (cid:48) v and S (cid:48) u , we can update U (cid:48) by solving
[Equation]
with A = X (cid:48) s − V S (cid:48) v , and V by solving
[Equation]
(10) with A (cid:48)
= (cid:18)
X t − UT u √ α ( X (cid:48) s − U (cid:48) S (cid:48) u ) (cid:19) and C = (cid:18) T v √ α S (cid:48) v (cid:19) .
As in FRODA, these two sub-problems can be solved efﬁciently using the Lagrange dual formulation of Lee et al.
E XPERIMENTS
We evaluate our approach on the task of open-set visual domain adaptation using two benchmark datasets, and compare its performance against the state-of-the-art open-set domain adaptation methods on each dataset.
1 Note that we also report the results of the methods used as baselines in (Busto & Gall, 2017).
For a dataset with C source classes, we report the accuracy on C + 1 classes, the additional one corresponding to the unknown case.
Implementation details.
Following Busto & Gall (2017), we represent the source and target samples with 4096-dimensional DeCAF 7 features (Donahue et al., 2013) and ﬁrst reduce their dimensionality by performing PCA jointly on the source and target data and keeping the components encoding 99% of the data variance.
To then determine the dimensionality d of our shared and private subspaces, we make use of the subspace disagreement measure of (Gong et al., 2012).
For all our experiments, the hyperparameters of our approach were set as follows:
α = 0 .
1 , β = 0 .
01 , λ 1 = 0 . 001 , λ 2 = 0 .
001 and ε = 0 .
For recognition, for the comparison with (Busto & Gall, 2017) to be fair, we employ a linear SVM classiﬁer in a one-vs-one fashion.
Nevertheless, we also report results obtained with a k − Nearest-Neighbor classiﬁer (with k = 3 ) and with our linear classiﬁer with parameters W learnt during training. experiment.
[Table]
Results on the dense cross-dataset benchmark.
We ﬁrst evaluate our approach on the challenging cross-dataset benchmark of Tommasi & Tuytelaars (2014).
This dataset was built using images depicting 40 object categories and coming from four datasets, namely Bing (B), Caltech256 (C),
ImageNet (I) and SUN (S), hence referred to as BCIS.
Following Busto & Gall (2017), we consider the samples from the ﬁrst 10 classes as known instances, while the samples with class labels 11 , 12 , · · · , 25 and 26 , 27 , · · · , 40 are taken to be the unknown samples in the source and target domains, respectively.
We follow the unsupervised protocol of Tommasi & Tuytelaars (2014), which relies on 50 source samples per class and 30 target images per class, except when the target data is coming from SUN, in which case only 20 images per class are employed.
Note that only the
DeCAF 7 features are publicly available.
To nonetheless evaluate the AODA method of Saito et al.
(2018), we made use of a network taking the DeCAF 7 features as input and processing them with two fully-connected layers, with 1024 and 128 units, respectively, and a ﬁnal classiﬁcation layer.
In Table 1, we compare the results of our methods with those of the baselines on all 12 domain pairs of this dataset.
Note that our algorithms (both with and without the discriminative term) outperform all the baselines, and in particular the state-of-the-art one of Busto & Gall (2017) by a large margin.
For instance, the margin exceeds 32, resp.
26, percentage points when going from SUN to Bing and ImageNet, respectively.
This, we believe, clearly evidences the beneﬁts of our factorized representations, which allow us to separate the unknown target samples from the ones coming from known classes, thus yielding a better representation for the known classes.
[Figure]
In Table 2, we compare different versions of our method, corresponding to using different classiﬁers and to using additional unknown source data.
Note that the linear SVM classiﬁer, when used with our framework, tends to perform the best, followed by the NN one and ﬁnally the learnt linear classiﬁer.
This, we believe, can be explained by the fact that, while the linear classiﬁer helps to learn a more discriminative representation, it remains less powerful than the other two classiﬁers to label the target samples.
Note also that the use of unknown source data does not consistently help in our framework.
Nevertheless, the corresponding results still outperform those of Busto & Gall (2017).
We further evaluate the robustness of our approach to the choice of threshold ε to separate the target samples from known/unknown classes. In
[Table]
[Table]
Results on the Ofﬁce dataset.
We further evaluate our approach on the slightly less challenging, although standard Ofﬁce benchmark (Saenko et al., 2010).
This dataset contains three different domains, namely Amazon (A), DSLR (D) and
Webcam (W), sharing 31 object categories, but differing in data acquisition process.
As in (Busto & Gall, 2017), we take all the samples from the ﬁrst 10 classes to represent the known ones, and all the samples with class labels 11 , 12 , · · · , 20 and 21 , 22 , · · · , 31 as unknown source and target data, respectively.
We report the results of our algorithms and of the baselines for all 6 domain pairs of this dataset in
Table 3.
As before, note that we outperform the baselines in this open-set scenario.
This includes the end-to-end AlexNet-based approach of Saito et al.
(2018) for open-set domain adaptation, as well as the state-of-the-art UDA methods of Tzeng et al.
(2017) and Bousmalis et al.
(2016), the latter of which is closest in spirit to our approach.
In Table 4, we compare the different variants of our approach.
The conclusions that one can draw from these results are similar to those for the BCIS dataset, thus showing that our method generalizes well across different domain adaptation datasets.
To evaluate the robustness of our method to the hyper-parameters α , β , and λ 1 , in Fig.
2, we plot the average accuracy of D-FRODA-NN (with k = 3 ) over all 6 pairs of the Ofﬁce dataset as a function of the value of α , β , and λ 1 .
Note that our results are stable for large ranges of these values.
Runtimes.
As mentioned in Section 3, one iteration of our approach takes on average 0.05 second, and our algorithm typically takes around 50 iterations to converge.
This yields a total runtime of roughly 2.5 seconds.
By contrast, the publicly available implementation of the method of Busto &
Gall (2017) takes on average 8 seconds per iteration and typically converges in 4 iterations, leading to a total runtime of roughly 32 seconds.
Note that these runtimes were measured on the same computer and that both methods rely on the same input features.
Therefore, our approach is not only signiﬁcantly more accurate than (Busto & Gall, 2017), but also faster by an order of magnitude.
Further discussion: The experimental setup used in (Saito et al., 2018; Busto & Gall, 2017) and our work for open-set DA relies on features extracted using a network pre-trained on ImageNet, which in fact can be argued to already contain semantic information about some of the unknown classes. To evidence that our approach does not crucially depend on this information, and thus validate our results, we observed that 4 of the unknown classes in the Ofﬁce dataset, namely Tape dispenser, Stapler, Scissors, Punchers, do not appear in ImageNet. We therefore performed additional experiments with only these classes as unknown ones and the same 10 known classes as before.
We compared the accuracy of our formulations against the SVM baseline and the open-set ATI method of Busto & Gall (2017). The gap with respect to both baselines remains large: SVM: 76.01%, ATI: 77.4%, D-FRODA: 78.2%, and FRODA: 78.5%. This conﬁrms that our method applies to truly never-seen-before classes. Note that among the 15 unknown classes in the setup for BCIS, 8 of them are not shared with
ImageNet, namely Windmill, Steering wheel, Can-soda, Sneaker, Skyscraper, Ladder, Motorcycle, and Palm tree. This further conﬁrms that our method handles the cases where no categorical or semantic information is available in the extracted features.  C ONCLUSION
We have introduced a novel approach to open-set domain adaptation, based on the intuition that source and target samples coming from the same, known classes can be represented by a shared subspace, while target samples from unknown classes should be modeled with a private subspace.
Each step of the resulting algorithms can be solved efﬁciently.
As demonstrated by our experiments, our method outperforms the state of the art in open-set domain adaptation and is one order of magnitude faster than the technique of Busto & Gall (2017).
We believe that this clearly evidences the beneﬁts of learning factorized representations, which allows us to jointly discard the unknown target samples and learn a better shared representation.
In the future, we will investigate ways to make better use of unknown source data, and to exploit more effective classiﬁers, such as SVM, directly within our
D-FRODA formulation