U NSUPERVISED
M ULTI -T ARGET
D OMAIN
A DAPTATION :
A N I NFORMATION T HEORETIC
A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised domain adaptation ( uDA ) models focus on pairwise adaptation settings where there is a single, labeled, source and a single target domain.
However, in many real-world settings one seeks to adapt to multiple, but somewhat similar, target domains.
Applying pairwise adaptation approaches to this setting may be suboptimal, as they fail to leverage shared information among multiple domains.
In this work we propose an information theoretic approach for domain adaptation in the novel context of multiple target domains with unlabeled instances and one source domain with labeled instances.
Our model aims to ﬁnd a shared latent space common to all domains, while simultaneously accounting for the remaining private, domain-speciﬁc factors.
Disentanglement of shared and private information is accomplished using a uniﬁed information-theoretic approach, which also serves to establish a stronger link between the latent representations and the observed data.
The resulting model, accompanied by an efﬁcient optimization algorithm, allows simultaneous adaptation from a single source to multiple target domains.
We test our approach on three challenging publicly-available datasets, showing that it outperforms several popular domain adaptation methods.
I NTRODUCTION
In real-world data, the training and test data often do not come from the same underlying distribution (Sun et al.
For instance, in the task of object recognition/classiﬁcation from image data, this may be due to the image noise, changes in the object view, etc., which induce different biases in the observed data sampled during the training and test stage.
Consequently, assumptions made by traditional learning algorithms are often violated, resulting in degradation of the algorithms’ performance during inference of test data.
Domain
Adaptation ( DA ) approaches (Fernando et al.
; Gong et al.
; Kodirov et al. (2015); Yoo et al.
(2016)) aim to tackle this by transferring knowledge from a source domain (training data) to an unlabeled target domain (test data) to reduce the discrepancy between the source and target data distributions, typically by exploring domain-invariant data structures.
Existing DA methods can be divided into: (semi)supervised DA , and unsupervised DA (Csurka (2017)).
The former assume that in addition to the labeled data of the source domain, some labeled data from the target domain are also available for training/adapting the classiﬁers.
By contrast, the latter does not require any labels from the target domain but rather explores the similarity in the data distributions of the two domains.
In this work, we focus on the unsupervised DA ( uDA ) scenario, which is more challenging due to the lack of correspondences in source and target labels.
Most works on uDA today focus on a single-source-single-target-domain scenario.
However, in many real-world applications, unlabeled data may come from different domains, thus, with different statistical properties but with common task-related content.
For instance, we may have access to images of the same class of objects (e.g., cars) recorded by various types of cameras, and/or under different camera views and
[Figure]
at different times, rendering multiple different domains (e.g., datasets).
Likewise, facial expressions of emotions, such as joy and surprise, shown by different people and recorded under different views, result in multiple domains with varying data distributions.
In most cases, these domains have similar underlying data distributions, which can be leveraged to build more effective and robust classiﬁers for tasks such as the object or emotion recognition across multiple datasets/domains.
To this end, traditional uDA methods focus on the single-source-single-target DA scenario.
However, in the presence of multiple domains, as typically encountered in real-world settings, this pair-wise adaptation approach may be suboptimal as it fails to leverage simultaneously the knowledge shared across multiple task-related domains.
Recently, Zhao et al.
(2017) showed that by having access to multiple source domains can facilitate better adaptation to a single target domain, when compared to the pair-wise DA approach.
While this is intuitive due to the access to multiple labelled source domains, offering more adaptation ﬂexibility for the target domain (i.e., by efﬁciently exploring the data labels across multiple source domains that are most related to the target domain), it comes at the expense of the data labelling in multiple source domains, which can be costly and time-consuming.
In either case, a single source domain or readily available multiple source domains, to the best of our knowledge, a simultaneous adaptation to multiple and unlabelled target domains remains an unexplored DA scenario.
However, this DA scenario is important as we usually have access to multiple unlabeled domains; yet, the adaptation process is also more challenging due to the lack of supervision in the target domains.
Still, multi-target DA can have advantages over a single-target DA when: (i) there is direct knowledge sharing between the source and multiple target domains (ﬁg. 1a), and (ii) the source and a target domain are related through another target domain (ﬁg. 1b).
While this seems intuitive, it is critical how the data from multiple unlabelled target domains are leveraged within the multi-target DA approach, in order to improve its performance over the single target DA approaches and naive fusion of multiple target domains.
To this end, we propose a Multi-Target DA-Information-Theoretic-Approach ( MTDA-ITA ) for single-sourcemulti-target DA .
We exploit a single source domain and focus on multiple target domains to investigate the effects of multi-target DA ; however, the proposed approach can easily be extended to multiple source domains.
This approach leverages the data from multiple target domains to improve performance compared to individually learning from pair-wise source-target domains.
Speciﬁcally, we simultaneously factorize the information from each available target domain and learn separate subspaces for modeling the shared (i.e., correlated across the domains) and private (i.e., independent between the domains) subspaces of the data (Salzmann et al.
To this end, we employ deep learning to derive an information theoretic approach where we jointly maximize the mutual information between the domain labels and private (domain-speciﬁc) features, while minimizing the mutual information between the the domain labels and the shared (domaininvariant) features.
Consequently, the more robust feature representations are learned for each target domain by exploiting dependencies between multiple target domains.
We show on benchmark datasets for DA that this approach leads to overall improved performance on each target domain, compared to independent DA for each pair of source-target domains, or the naive combination of multiple target domains, and state-of-the-art models applicable to the target task.
T HE P ROPOSED M ETHOD
Without loss of generality, we consider a multi-class ( K -class) classiﬁcation problem as the running example.
Furthermore, let ( X , Y , D ) = { ( x i , y i , d i ) } Ni =0 be a collection of M domains (a labeled source domain, and M − 1 unlabeled target domains), where x i denotes the i -th sample, and y i =
[ y 1 i , y 2 i , ..., y Ki ] and d i =
[ d 1 i , d 2 i , ..., d Mi ] are the K -D and M -D encoding of the class and domain labels for x i , respectively.
Note that the class labels are only available for the source samples.
The latent space representation of the data point x is denoted as z =
[ z s , z p ] , where z s and z p are the (latent) shared and private features of the data point x , respectively.
By factorizing the joint distribution p ( x , y , d , z s , z p ) as [Equation]
we propose to maximize the following objective function:
[Equation]
where p ( x ) and p ( d ) denote the underlying (true) data distribution and domain label distribution, respectively,
I ( x ; y ) denotes the Mutual Information between the random variables x and y .
λ r , λ c and λ d denote the hyper-parameters controlling the weights of the objective terms.
The proposed objective function (2) maximizes the three terms described below:
• I ( x ; z ) : encourages the latent features (both shared and private) to preserve information about the data samples (that can be used to reconstruct x from z ).
• I ( y ; z s ) : enables to correctly predict the true class label of the samples out of their common shared features.
• I ( d ; z p ) − I ( d ; z s ) : encourages the latent private features to preserve the information about the domain label and penalizes the latent shared features to be domain informative.
This not only reduces the redundancy in the shared and private features, but also, penalizes the redundancy of different private spaces, while preserving the shared information.
An additional term could be used to minimize the mutual information between the shared ( z s ) and private ( z p ) features.
However, computing the mutual information (even approximating it) is intractable due to the highly complex joint distribution p ( z s , z p ) .
Since we want z s and z p features to encode different aspects of x , we enforce such constraint by jointly maximizing the term: I ( d ; z p ) − I ( d ; z s ) .
2. O PTIMIZATION
The following lower bound for mutual information is derived using the non-negativity of KL-divergence (Barber & Agakov (2003)); i.e., Σ x p ( x | z ) ln p ( x | z ) q ( x | z ) ≥ 0 gives:
[Equation]
where H ( x ) denotes the Shanon Entropy (Lin (1991)) of the random variable x .
q ( x | z ; φ ) is any arbitrary distribution parameterized by φ .
We need a variational distribution q ( x | z ; φ ) because the posterior distribution p ( x | z ) =
p ( z | x ) p ( x ) /p
( z ) is intractable since the true data distribution p ( x ) is assumed to be unknown.
Similarly, we can derive lower bounds for I ( d ; z p ) ≥ H ( d ) + E p ( d , z p )
[ln q ( d | z p ; ψ )] and I ( d ; z s ) ≥
[Figure]
x
E p
E s z p z s
D
F
C ˆ x ˆ d ˆ c
H ( d ) + E p ( d , z s ) [ln q ( d | z s ; ψ )] , where q ( d | z p ; ψ ) is any arbitrary distribution parameterized by ψ . 1 We further drive lower bound for I ( y ; z s ) as I ( y ; z s ) ≥ H ( y ) + E p ( y , z s )
[ln q ( y | z s ; θ c )] , where q ( y | z s ; θ c ) is a variational distribution parameterized by θ c approximating p ( y | z s ) .
Let next E s ( x ; θ s ) be a function parameterized by θ s that maps a sample x to its corresponding shared feature z s , and E p ( x ; θ p ) be an analogous function which maps x to z p , the feature that is private to each domain (ﬁg. 2).
We also deﬁne F ( z s , z p ; φ ) as a decoding function mapping the concatenation of the latent features z s and z p to a sample reconstruction ˆ x , and D ( z ; ψ ) as a decoding function mapping z s and z p to a M -dimensional vector: the predictions of the domain label ˆ d .
Finally, C ( z s ; θ c ) is a task-speciﬁc function mapping z s to a K -dimensional probability vector of the class label ˆ y .
We represent p ( d ) , p ( x ) , p ( y ) as the empirical distribution of a ﬁnite training set (e.g. p ( d ) =
1 {+N (cid:80) Ni =
 δ+} ( d − d i ) )
as in the case of variational autoencoders (VAE) (Abbasnejad et al. (2017); Pu et al. , p ( z s | x ) , p ( z p | x ) as deterministic functions of x as p ( z s | x ) = δ (cid:0) z s − E s ( x ; θ s ) (cid:1) and p ( z p | x ) =
δ (cid:0) z p − E p ( x ; θ p ) (cid:1) , and the variational distributions q ( y | z s ) , q ( x | z ) and , q ( d | z ) as q ( y | z s ) = SoftMax ( C ( z s ; θ c )) , q ( d | z ) = SoftMax ( D ( z ; ψ )) , q ( x | z ; φ ) ∝ exp( (cid:107) x − F ( z ; φ ) (cid:107) 1 ) (4) where Softmax ( · ) denotes the softmax or normalized exponential function (Bridle (1990)), and (cid:107) .
(cid:107) 1 denotes the L 1 norm. Then, the optimization task can be posed as a minimax saddle point problem, where we use adversarial training to maximize (2) w.r.t.
the parameters ( θ s , θ p , θ c ), and to minimize (2) w.r.t.
the parameters ( φ , ψ ), using Stochastic Gradient Descent (SGD).
Optimizing the parameters φ of the decoder F
[Equation]
The decoder F ( z s , z p ; φ ) is trained in such a way so as to minimize the difference
Optimizing the parameters ψ of the domain classiﬁer D
[Equation]
D ( z ; ψ ) can be considered as a classiﬁer whose task is to distinguish between the shared/private features of the different domains.
More precisely, the two terms in Eq.
6 encourage D to correctly predict the domain labels from the shared and private features, respectively.
Optimizing the parameters θ c of the label classiﬁer C
[Equation]
Since we have access to the source labels, H ( y ) is a constant for source samples.
we can approximate H [ y ] for the target samples using the output of the classiﬁer C , leading to the following optimization problem:
[Equation]
where N s denotes the number of source samples.
Intuitively, we enforce the classiﬁer C to correctly predict the class labels of the source samples by the ﬁrst term in Eq. 8.
We use the second term to minimize the entropy of q ( y |
z s ) for the target samples; effectively, reducing the effects of "confusing" labels of target samples, as given by p ( y | z s ) that leads to decision boundaries occur far away from target data-dense regions in the feature space.
The intuition behind the last term is that by minimizing only the entropy (second term), we may arrive at a degenerate solution where every target point x t is assigned to the same class.
Hence, the last term encourages the classiﬁer C to have balanced labeling for the target samples where it reaches its minimum, ln K , when each class is selected with uniform probability.
Optimizing the parameter θ p of the private encoder E p
[Equation]
The ﬁrst term in Eq.
9 encourages the private encoder E p to preserve the recovery ability of the private features.
The second term enforces distinct private features be produced for each domain by penalizing the representation redundancy in different private spaces.
This, in turn, encourages moving this common information from multiple domains to their shared space.
Optimizing the parameter θ s of the shared encoder E s
[Equation]
The ﬁrst term in Eq.
10 encourages the shared encoder E s to preserve the recovery ability of the shared features.
The second term is the source domain classiﬁcation loss penalty that encourages E s to produce discriminative features for the labeled source samples.
The third term simulates the adversarial training by trying to fool the domain classiﬁer D when predicting the domain labels d , given the shared features z s .
The effect of this is two-fold: (i) the rendered shared features are more distinct from the corresponding private features, (ii) the shared features of different domains are encouraged to be similar to each other.
The last two terms encourage E s to produce the shared features for target samples so that the classiﬁer is conﬁdent on the unlabeled target data, driving the shared features away from the decision boundaries.
To train our model, we alternate between updating the shared encoder E s , the private encoder E p , the decoder F , the classiﬁer C , and the domain classiﬁer D using the SGD algorithm (see Algorithm 1 in Appendix E for more details).
R ELATED W ORK
There has been extensive prior work on domain adaptation (Csurka (2017)).
Recent papers have focused on transferring deep neural network representations from a labeled source dataset to an unlabeled target domain, where the main strategy is to ﬁnd a feature space such that the confusion between source and target distributions in that space is maximized ( Rebufﬁ et al.
; Benaim & Wolf (2017); Courty et al.
Motiian et al.
Saito et al.
; Zhang et al.
; Yan et al.
; Bousmalis et al.
For this, it is critical to ﬁrst deﬁne a measure of divergence between source and target distributions.
For instance, several methods have used the Maximum Mean Discrepancy ( MMD ) loss for this purpose (Bousmalis et al.
; Zellinger et al.
(2017); Long et al.
MMD computes the norm of the difference between two domain means in the reproducing Kernel Hilbert Space ( RKHS ) induced by a pre-speciﬁed kernel.
The Deep Adaptation Network (DAN) (Long et al.
applied MMD to layers embedded in a RKHS , effectively matching higher order statistics of the two distributions.
The deep Correlation Alignment ( CORAL ) method (Sun & Saenko (2016)) attempts to match the mean and covariance of the two distributions.
Deep Transfer Network ( DTN ) (Zhang et al.
(2015)) achieved source/target distribution alignment via two types of network layers based on MMD distance: the shared feature extraction layer, which learns a subspace that matches the marginal distributions of the source and the target samples, and the discrimination layer, which matches the conditional distributions by classiﬁer transduction.
Recently proposed unsupervised DA methods (Rebufﬁ et al.
; Benaim & Wolf (2017); Courty et al.
(2017); Motiian et al.
Saito et al.
; Zhang et al.
(2017)) operate by training deep neural networks using adversarial training, which allows the learning of feature representations that are simultaneously discriminative of source labels, and indistinguishable between the source and target domain.
For instance, Ganin & Lempitsky (2015) proposed a DA mechanism called Domain-Adversarial Training of Neural Networks ( DANN ), which enables the network to learn domain invariant representations in an adversarial way by adding a domain classiﬁer and back-propagating inverse gradients.
Adversarial Discriminative Domain
Adaptation ( ADDA ) (Tzeng et al. (2017)) learns a discriminative feature subspace using the source labels, followed by a separate encoding of the target data to this subspace using an asymmetric mapping learned through a domain-adversarial loss.
Liu et al.
(2017) makes a shared-latent space assumption and proposes an unsupervised image-to-image translation ( UNIT ) framework based on Coupled GANs (Liu & Tuzel (2016)).
Another example is the pixel-level domain adaptation models that perform the distribution alignment not in the feature space but directly in raw pixel space.
PixelDA (Bousmalis et al.
(2017)) uses adversarial approaches to adapt source-domain images as if drawn from the target domain while maintaining the original content.
While these approaches have shown success in DA tasks with single source-target domains, they are not designed to leverage information from multiple domains simultaneously.
More recently, Zhao et al.
(2017) introduced an adversarial framework called MDAN for multiple source single target domain adaptation where a domain classiﬁer, induced by minimizing the H-divergence between multiple source and a target domain, is used to align their feature distributions in a shared space.
Instead, in our approach we focus on multi-target
DA where we perform adaptation of multiple unlabelled target domains.
Although both our model and
MDAN use the similar notion of the domain classiﬁer to minimize the domain mismatch in shared space, the domain classiﬁer induced by our information-theoretic (IT) loss also acts to separate domains in the private space (see Eqs.
6 & 9 for more details), improving the essential reconstruction ability, similar to (Bousmalis et al.
We provided how our model is related to IT representation learning approaches, and multiple domain transfer networks in Appendices A and B respectively.
In Appendix C, we also clearly contrasted our model with DSN model which also uses the notion of auto-encoders to explicitly separate the feature representations private to each source/target domain from those that are shared between the domains.
E XPERIMENTAL R ESULTS
We compare the proposed method with state-of-the-art methods on standard benchmark datasets: a digit classiﬁcation task that includes 4 datasets: MNIST (LeCun et al.
MNIST-M (Ganin et al.
SVHN (Netzer et al.
, USPS (Tzeng et al.
(2017)), Multi-PIE expression recognition dataset 2 , and PACS multi-domain image recognition benchmark (Li et al.
(2017)), a new dataset designed for the crossdomain recognition problems (the details for this experiment is available in Appendix G).
Fig. 3 illustrates image samples from different datasets and domains.
We evaluate the performance of all methods with classiﬁcation accuracy metric.
We repeated each experiment 5 times and report the average and the standard deviation of the accuracy.
We used ADAM (Kingma & Ba (2015)) for training; the learning rate was set to 0 . 0002 and momentum parameters to 0 . 5 and 0 .
We used batches of size 16 from each domain, and the input images were meancentered/rescaled to [ − 1 , 1] .
The hyper-parameters are empirically set as λ r = 1 . 0 , λ c = 0 .
01 , λ d = 0 .
For the network architecture, our private/shared encoders consisted of three convolutional layers as the front-end and four basic residual blocks as the back-end.
The decoder consisted of four basic residual blocks as the front-end and four transposed convolutional layers as the back-end.
The discriminator and the classiﬁer consisted of stacks of convolutional layers.
We used ReLU for nonlinearity.
Tanh function is used as the activation function of the last layer in the decoder F for scaling the output pixels to [ − 1 , 1] .
The details of the networks are given in Appendix D.
The quantitative evaluation involves a comparison of the performance of our model to previous work and to Source Only and 1-NN baselines that do not use any domain adaptation.
For Source Only baseline, we train our model only on the unaltered source training data and evaluate on the target test data.
We compare the proposed method MTDA-ITA with several related methods designed for pair-wise source-target adaptation: CORAL (Sun & Saenko (2016)), DANN (Ganin & Lempitsky (2015)), ADDA (Tzeng et al. (2017)), DTN (Zhang et al.
, UNIT (Liu et al.
(2017)), PixelDA (Bousmalis et al.
(2017)), and DSN (Bousmalis et al.
We reported the results of two following baselines: (i) one is to combine all the target domains into a single one and train it using MTDA-ITA , which we denote as ( c-MTDA-ITA ).
(ii) the other one is to train multiple MTDA-ITA separately, where each one corresponds to a source-target pair which we denote as ( s-MTDA-ITA ).
For completeness, we reported the results of the competing methods by combining all the target domains into a single one (denoted by c-DTN , c-ADDA , and c-DSN ) as well.
We also extend DSN to multiple domains by (i) having one private encoder for all domains denoted by ( 1p-DSN ), (ii) adding multiple private encoders to it denoted by ( mp-DSN ) and contrast them with our model. [Figure] scream Camera at 0 smile squint disgust normal
Camera at 15 smile squint scream smile
Camera at 30 surprise disgust squint scream Overhead Camera at 30 disgust normal squint disgust Camera at 60 squint normal smile ( [Figure]
4. D IGITS D ATASETS
We combine four popular digits datasets ( MNIST , MNIST-M , SVHN , and USPS ) to build the multitarget domain dataset.
All images were uniformly rescaled to 32 × 32 .
We take each of MNIST-M , SVHN , USPS , and MNIST as source domain in turn, and the rest as targets.
We use all labeled source images and all unlabeled target images, following the standard evaluation protocol for unsupervised domain adaptation (Ganin et al.
; Long et al. (2016)).
We show the accuracy of different methods in tab.(Additional results are available in Appendix F).
The results show that ﬁrst of all cMTDA-ITA has worse performance than sMTDA-ITA and MTDA-ITA .
We have similar observations for ADDA , DTN , and DSN that demonstrates a naive combination of different target datasets can sometimes even decrease the performance of the competing methods.
Furthermore, MTDA-ITA outperforms the state-of-the-art methods in most of domain transformations.
The higher performance of MTDA-ITA compared to other methods is mainly attributed to the joint adaptation of related domains where each domain could beneﬁt of other related domains.
Furthermore, from the results obtained, we see that it is beneﬁcial to use information coming from unlabeled target data (see Eq. 8 for updating the classiﬁer C ) during the learning process, compared to when no data from target domain is used (See the ablation study section for more information).
Indeed, using our scheme, we ﬁnd a representation space in which embeds the knowledge from the target domain into the learned classiﬁer.
By contrast, the competing methods do not provide a principled way of sharing information across all domains, leading to overall lower performance.
The results also verify the superiority of MTDA-ITA over both mp-DSN , and 1p-DSN .
This can be due to (i) having multiple private encoders increase the number of parameters that may lead to mp-DSN overﬁtting, (ii) superiority of the MTDA-ITA ’s domain adversarial loss over the DSN ’s MMD loss to separate the shared and private features, (iii) utilization of the unlabeled target data to regularize the classiﬁer in MTDA-ITA .
4. M ULTI -PIE DATASET
The Multi-PIE dataset includes face images of 337 individuals captured from different expressions, views, and illumination conditions (ﬁg. 3(c)).
For this experiment, we use 5 different camera views (positions) C 05 ,
C 08 , C 09 , C 13 , and C 14 as different domains (ﬁg. 3(c)) and the face expressions ( normal, smile, surprise, squint, disgust, scream ) as labels.
Each domain contains 27120 images of size 64 × 64 × 3 .
We used each view as the source domain, in turn, and the rest as targets.
We expect the face inclination angle to reﬂect the complexity of transfer learning.
Tab. 2 shows the classiﬁcation accuracy for C 13 and C 14 as source domain (the results for views C 05 , C 08 and C 09 as source domain are available in Appendix F).
As can be seen, MTDA-ITA achieves the best performances as well as the best scores in most settings that veriﬁes the effectiveness of MTDA-ITA for multi-target domain adaptation.
Clearly, with the increasing camera angle, the image structure changes up to a certain extent (the views become heterogeneous).
However, our method produces better results even under such very challenging conditions. [Table] [Table] [Table]
[Table]
4. A BLATION S TUDIES
We performed an ablation study on the proposed model measuring impact of various terms on the model’s performance.
To this end, we conducted additional experiments for the digit datasets with different components ablation, i.e., training without the reconstruction loss (denoted as MTDA-woR ) by setting λ r = 0 , training without the classiﬁer entropy loss (denoted as MTDA-woE ) by setting λ c = 0 , training without the multidomain separation loss (denoted as MTDA-woD ) by setting λ d = 0 .
As can be seen from ﬁg. 4, disabling each of the above components leads to degraded performance.
More precisely, the average drop by disabling the classiﬁer entropy loss is ≈ 3 .
Similarly, by disabling the reconstruction loss and the multi-domain separation loss, we have ≈ 4 .
and ≈ 22% average drop in performance, respectively.
Clearly, by disabling the multi-domain separation loss, the accuracy drops signiﬁcantly due to the severe data distribution mismatch between different domains.
The ﬁgure also demonstrates that leveraging the unlabeled data from multiple target domains during training enhances the generalization ability of the model that leads to higher performance.
In addition, the performance drop caused by removing the reconstruction loss , i.e., without the private encoder/decoder, indicates (i) the beneﬁt of modeling the latent features as the combination of shared and private features, (ii) the ability of the model’s domain adversarial loss to effectively learn those features. In order to examine the effect of the private features on the model’s classiﬁcation performance, we took the
MTDA-ITA and trained it without the private encoder (denoted as MTDA-woP ).
As ﬁg. 4 shows, without the private features, the model performed consistently worse ( ≈ 2% average drop in performance) in all scenarios.
This demonstrates explicitly modeling what is unique to each domain can improve the model’s ability to extract domain–invariant features.
In summary, this ablation study showed that the individual components bring complimentary information to achieve the best classiﬁcation results. 4. F EATURE V ISUALIZATION
We use t-SNE (Maaten & Hinton (2008)) on Digit dataset to visualize shared and private feature representations from different domains.
Fig. 6 shows shared and private features from source ( SVHN ) and target domains before (a),(b) and after adaptation (c),(d). MTDA-ITA signiﬁcantly reduces the domain mismatch for the shared features (circle markers in ﬁg. 6d, strong mixing of domain labels in this cluster, ﬁg. 6c) and increases it for the private features (triangle markers, pure and well-separated domain clusters in ﬁg. 6c). This is partially due to the proposed multi-domain separation loss through the use of the domain classiﬁer D , which penalizes the domain mismatch for the shared features and rewards the mismatch for the private features.
Moreover, as supported by the quantitative results in tab. 1, joint adaptation of related domains and the classiﬁer, accomplished through the model, leads to superior class separability, compared to original features.
This is depicted in ﬁg. 6d, where the points in the shared space (large cluster) are grouped into class-speciﬁc subgroups (color indicates class label), while they are mixed in private spaces (smaller clusters).
This is in contrast to ﬁg. 6b, where original features show no class-speciﬁcity.
We also show the learned shared and private features for the models MTDA-woE , MTDA-woP , MTDAwoR , and MTDA-woD , in ﬁgs.
6e to 6l.
Note that since the private encoder E p is disabled for MTDA-woR , and MTDA-woP , no private features are depicted in ﬁgs.
6g to 6j.
The class label separation in the shared space for MTDA-woE , MTDA-woP , and MTDA-woR , ﬁgs.
6f, 6h and 6j, is still evident but not as strong as in the full model, ﬁg. 6d, corroborating the small loss in classiﬁcation accuracy observed in ﬁg. 4a.
On the other hand, MTDA-woD has signiﬁcant mixing of class labels in the shared space, ﬁg. 6l, more so than
MTDA-woE , MTDA-woR , and MTDA-woP , implying worse classiﬁcation prediction in ﬁg. 4a due to the severe mismatch between different domains.
Since our model uses one private encoder for all target domains, we also contrasted the visualization of DSN model with one private encoder 1p-DSN in Appendix H.
C ONCLUSION
This paper presented an information theoretic end-to-end approach to uDA in the context of a single source and multiple target domains that share a common task or properties.
The proposed method learns feature representations invariant under multiple domain shifts and simultaneously discriminative for the learning task.
This is accomplished by explicitly separating representations private to each domain and shared between source and target domains using a novel discrimination strategy.
Our use of a single private domain encoder results in a highly scalable model, easily optimized using established back-propagation approaches.
Results on three benchmark datasets for image classiﬁcation show superiority of the proposed method compared to the state-of-the-art methods for unsupervised domain adaptation of visual domain categories. A C ONNECTION TO I NFORMATION T HEORETIC R EPRESENTATION L EARNING
The idea of using information theoretic (IT) objectives for representation learning was originally introduced in (Tishby & Zaslavsky (2015)).
Since their approach for optimizing the IT objective functions relied on the iterative Blahut Arimoto algorithm (Tishby & Zaslavsky (2015)), it is not feasible to apply to deep neural network (DNN) frameworks.
Similar to our approach, there have been some recent works (Mohamed & Rezende (2015); Chalk et al. ; Alemi et al. (2018b; 2016; 2018a))
to approximate the MI by applying variational bounds on MI , though not in the context of domain adaptation.
Mohamed & Rezende (2015) utilized the variational bounds on MI , and apply it to DNNs in the context of reinforcement learning.
Chalk et al.
(2016) and Alemi et al.
(2016), developed the same variational lower bound In the context of Information Bottleneck (IB) principle (Tishby & Zaslavsky (2015)), where the former applied it to sparse coding problems, and used the kernel trick to achieve nonlinear mappings, whereas the latter applied it to DNNs to handle large datasets thanks to the SGD algorithm.
Achille & Soatto (2018) proposed a variational bound on the MI in the context of IB, from the perspective of variational dropout and demonstrated its utility in learning disentangled representations for variational autoencoders.
The main difference between our method and the above methods is that these methods throw away the information in the data not related to the task by minimizing the mutual information between the data points and the latent representations that may lead to ignoring the individual characteristics (private features) of the datasets in a multiple dataset regime, whereas our method explicitly models what is unique to each domain (dataset) that improves the model’s ability to extract domain–invariant features.
In the unsupervised representation learning literature, our work is also related to the VAE-based models (Bowman et al.
However, we propose to tackle the task using our IT approach using deterministic mappings instead of the traditional evidence lower bound ( ELBO ) optimization with stochastic mappings.
In contrast to the unsupervised representation learning approaches, our setting also allows us to further improve the latent representation using the labeled data in the source domain while leveraging the sharing of dependencies across different target domains.
B C ONNECTION TO M ULTIPLE D OMAIN T RANSFER N ETWORKS
Recent studies have shown remarkable success in multiple domain transfer (MDT) (Choi et al.
Anoosheh et al.
; Kameoka et al.
; Hao et al.
(2018)) though not in the context of the image classiﬁcation, rather in the context of image generation.
Choi et al.
(2017) proposed StarGAN , a generative adversarial network capable of learning mappings among multiple domains in the contest of image to image translation framework.
The goal of StarGAN is to train a single generator G though this requires passing in a vector along with each input to the generator specifying the output domain desired, that learns mappings among multiple domains.
To achieve this, G is trained to translate an input image x into an output image x
(cid:48) conditioned on the target domain label d , G ( x , d ) → x (cid:48) .
Similar to our domain classiﬁer module D , they introduce an auxiliary classiﬁer that allows a single discriminator to control multiple domains.
Anoosheh et al.
(2017) introduced ComboGAN , which decouples the domains and networks from each other.
Similar to our encoder/decoder modules, ComboGAN ’s generator networks contain encoder/decoders assigning each encoder and decoder to a domain.
They combine the encoders and decoders of the trained model like building blocks, taking as input any domain and outputting any other.
For example during inference, to transform an image x from an arbitrary domain X to x (cid:48) from domain X (cid:48) , they simply perform x (cid:48)
= G X (cid:48) , X ( x ) = Decoder X (cid:48) ( Encoder X ( x )) .
The result of Encoder X ( x ) can even be cached when translating to other domains as not to repeat computation.
The main differences between the MDT methods and ours is that, unlike our method which does domain alignment in feature space, MDT methods adapt representations not in feature space but rather in raw pixel space; translating samples from one domain to the “style” of a other domains.
This works well for limited domain shifts where the domains are similar in pixel-space, but can be too limiting for settings with larger domain shifts that results in poor performance in signiﬁcant structural change of the samples in different domains.
C C ONNECTION TO D OMAIN S EPARATION N ETWORKS
The method closest to our work is Domain Separation Networks ( DSN ) (Bousmalis et al.
(2016)), which use the notion of auto-encoders to explicitly separate the feature representations private to each source/target domain from those that are shared between the domains.
Although extending DSN to multiple domains might seem trivial, DSN requires an autoencoder per domain, making the model impractical in the case of more than a couple of domains.
The overall loss of DSN consists of a reconstruction loss for each domain modeled by a shared decoder, a similarity loss such as MMD , which encourages domain invariance modeled by a shared encoder, and a dissimilarity loss modeled by two private encoders: one for the source domain and one for the target domain.
While one could attempt to generalize DSN to multiple target domains by having individual per-target domain private encoders, doing so would prove problematic when the number of target domains is large — each private encoder would require a large "private" dataset to learn the private parameters.
Precisely, for multiple ( M ) target domains, we could train a DSN model with one shared encoder, M + 1 private encoder (one for each domain), and one shared decoder.
This leads to M + 3 models to train that implies the number of models increases linearly with the number of domains, as does the required training time.
Second, DSN uses an orthogonality constraint among the shared and the private representations which may not be strong enough to remove redundancy and enforce disentangling among different private spaces.
Precisely, DSN deﬁnes the loss via a soft subspace orthogonality constraint between the private and shared representation of each domain.
However, it does not enforce the private representation of different domains to be different that may result in redundancy of different private spaces.
In addition, DSN enforces separation of spaces using the notion of Euclidean orthogonality, e.g., (cid:107) z s − z p (cid:107) 2 .
In case of multiple target domains, this would result in learning of all pairs of private spaces independently.
To address those deﬁciencies, we ﬁrst explicitly couple different private encoders into a single private encoder model, E θ p of ﬁg. 2 , which allows us to generalize to an arbitrary number of target domains.
To assure that the information among the private and shared spaces is not shared (i.e., "orthogonal"), we deﬁne an informationtheoretic criteria enforced by a domain classiﬁer, D ψ of ﬁg. 2, which aims to segment the private space into clusters that correspond to individual target domains.
By using D ψ within the adversarial framework,
MTDA-ITA learns simultaneously the shared and private features from different domains (see ﬁg. 6).
We showed in Sec.
4 that our model performs better than the trivial extension of DSNs to the multi-domain case.
D N ETWORK A RCHITECTURE
The network architecture used for the experiments is given in tab.
We use the following abbreviation for ease of presentation: N=Neurons, K=Kernel size, S=
Stride size, D=Number of Domains, C=number of
Classes.
The transposed convolutional layer is denoted by DCONV.
The residual basic block is denoted as
RESBLK.
[Table]
E P ROPOSED MODEL ’ S A LGORITHM
The detailed optimization process of the proposed model is shown in Algorithm 1.
F A DDITIONAL E XPERIMENTS FOR D IGIT AND M ULTI -PIE DATASETS
The additional experiments for Digit dataset where we set MNIST-M and USPS as source domain is available in tab.
The additional experiments for Multi-PIE dataset where we set C 05 , C 08 and C 09 as source domain is available in tabs. G PACS DATASET
This dataset contains 9991 images (227 × 227 × 3 dimension ) across 7 categories (‘dog’, ‘elephant’, ‘giraffe’,‘guitar’, ‘house’, ‘horse’ and ‘person’) and 4 domains of different stylistic depictions (‘Photo’, ‘Art painting’, ‘Cartoon’ and ‘Sketch’). The very diverse depiction styles provide a signiﬁcant gap between S-->M S-->MM
S-->U
MTDA-ITAMTDA-woEMTDA-woP
MTDA-woRMTDA-woD (
[Figure]
Source domain: SVHN
MM-->S
MM-->M
MM-->U
[Figure]
M-->S
M-->MM
M-->U
[Figure]
method
[Table]
domains, coupled with the small number of data samples, making it extremely challenging for domain adaptation.
Consequently, the dataset was originally used for multi-source to single target domain adaptation (Li et al.
Instead, we tackle a signiﬁcantly more challenging problem of single-source to multiple target adaptation.
Tab.
7 shows the classiﬁcation accuracy of various methods.
MTDA-ITA consistently
[Table]
Algorithm Require: { X , Y , D } :M domain datasets.
λ r , λ c , λ d : Model hyper-parameters.
Ensure: θ s , θ p , θ c , φ, ψ : Model parameters.
1: Initialize θ s , θ p , θ c , φ, ψ ; 2: repeat 3:
Sample a mini-batch from each of source/target domain datasets.
Update { θ s } by minimizing L s in Eq.(10) through the gradient descent: θ s = θ s − η
∂
L s ∂θ s .
Update { θ p } by minimizing L p in Eq.(9) through the gradient descent: θ p = θ p − η
∂
L p ∂θ p .
Update { θ c } by minimizing L c in Eq.(8) through the gradient descent: θ c
= θ c − η ∂
L c ∂θ c .
Update { φ } by minimizing L φ in Eq.(6) through the gradient descent: φ = φ − η ∂
L φ ∂φ .
Update { ψ } by minimizing L ψ in Eq.(5) through the gradient descent: ψ = ψ − η ∂ L s ∂ψ .
9: until Convergence; 10: return { θ s , θ p , θ c , φ, ψ } .
achieves the best performance for all transfer tasks.
Evaluations were obtained by training all models ( ADDA , DSN , and ours) from scratch on the PACS dataset.
Note that the overall performance ﬁgures are low due to the extreme difﬁculty of the transfer task, induced by large differences among domains. H A NALYSIS OF SHARED / PRIVATE SPACE EMBEDDING
In the experiments conducted, we showed that our approach is able to achieve better performance than the competing methods including the extended DSN with one private encoder ( 1p-DSN ) which is the most similar method to ours.
[Figure]
N o a d a p t a ti on
[Figure]
I T
[Figure]
Indeed, ﬁg. 5 depicts the embedding of the MTDA-ITA learned private/shared features, those of 1p-DSN and the original features from different domains for Digit datasets ( SVHN is the source). Notice that both MTDA-ITA and 1p-DSN reduces the domain mismatch for the shared features (circle markers in ﬁg. 5) and separate the shared features from private features. On the other hand, MTDA-ITA increases the domain separation for the private features (triangle markers, pure and well-separated domain clusters in ﬁg. 5c) while 1p-DSN is unable to enforce the private representation of different domains to be different (ﬁg. 5e) that may result in redundancy of different private spaces. This is partially due to the proposed multi-domain separation loss through the use of the domain classiﬁer D , which penalizes the domain mismatch for the shared features and rewards the mismatch for the private features, something the 1p-DSN fails to account for.
Moreover, as supported by the quantitative results in tab. 1, the class label separation in
[Table] [Table]
the shared space for 1p-DSN , ﬁg. 5f, is still evident but not as strong as in the MTDA-ITA , ﬁg. 5d.
This can be attributed to the lack of redundancy in the private space that helps MTDA-ITA to learn more disentangled shared features and usage of the target samples during training, something the 1p-DSN fails to account for. Domains Classes N o a d a p t a ti on (a) (b)
M T DA -I T
A (c) (d)
M T
DA - w o E (e) (f) M T
DA - w o P (g) (h)
M T
DA - w o R (i) (j)
M T
DA - w o D (k) (l)