D OMAIN -I NDEXING V ARIATIONAL B AYES : I NTERPRETABLE D OMAIN I NDEX FOR D OMAIN A DAPTATION
A BSTRACT
Previous studies have shown that leveraging domain index can significantly boost domain adaptation performance (Wang et al., 2020; Xu et al., 2022)
However, such domain indices are not always available.
To address this challenge
, we first provide a formal definition of domain index from the probabilistic perspective and then propose an adversarial variational Bayesian framework that infers domain indices from multi-domain data, thereby providing additional insight on domain relations and improving domain adaptation performance.
Our theoretical analysis shows that our adversarial variational Bayesian framework finds the optimal domain index at equilibrium.
Empirical results on both synthetic and real data verify that our model can produce interpretable domain indices which enable us to achieve superior performance compared to state-of-the-art domain adaptation methods
I NTRODUCTION
In machine learning, it is standard to assume that training data and test data share an identical distribution.
However, this assumption is often violated (Ganin & Lempitsky, 2015; Romera et al., 2019; Sun et al., 2017; Yuan et al., 2019; Ramponi & Plank, 2020) when training and test data come from different domains.
Domain adaptation (DA) tries to solve such a cross-domain generalization problem by producing domain-invariant features.
Typically, DA methods enforce independence between a data point’s latent representation and its domain identity , which is a one-hot vector indicating which domain the data point comes from (Ganin et al., 2016;
Tzeng et al., 2017; Zhao et al., 2017; Zhang et al., 2019).
More recent studies have found that using domain index , which is a real-value scalar (or vector) to embed domain semantics, as a replacement of domain identity, significantly boosted domain adaptation performance (Wang et al., 2020; Xu et al., 2022).
For instance, Wang et al.
(2020) adapted sleeping stage prediction models across patients with different ages, with "age" as the domain index, and achieved superior performance compared to traditional models that split patients into groups by age and used discrete group IDs as domain identities (more discussion in Sec.
J).
Although significant progress has been made in leveraging domain indices to improve domain adaptation (Wang et al., 2020; Xu et al., 2022), a major challenge exists: domain indices are not always available.
This severely limits the applicability of such indexed DA methods.
Thus a natural question is motivated: Can one infer the domain index as a latent variable from data?
This prompts us to first develop an expressive and formal definition of “domain index”.
We argue that an effective “domain index” (1) is independent of the data’s encoding, (2) retains as much information on the data as possible, and (3) maximizes adaptation performance, e.g., accuracy (see Sec.
3.2 for rigorous descriptions).
With this definition, we then develop an adversarial variational Bayesian deep learning model (Wang et al., 2015; Wang & Yeung, 2016; 2020) that describes intuitive conditional dependencies among the input data, labels, encodings, and the associated domain indices.
Our theoretical analysis shows that maximizing our model’s evidence lower bound while adversarially training an additional discriminator (Ganin et al., 2016; Wang et al., 2020) is equivalent to inferring the optimal domain indices (according to our definition) that maximize the mutual information among the input data, labels, encodings, and the associated domain indices while minimizing the mutual information between the data’s encodings and the domain indices.
Our contributions are as follows:
• We identify the problem of inferring domain indices as latent variables, provide a rigorous definition of “domain index”, and develop the first general method, dubbed variational domain indexing (VDI), for inferring such domain indices.
• Our theoretical analysis shows that training with VDI’s final objective function is equivalent to inferring the optimal domain indices according to our definition.
• Experiments on both synthetic and real-world datasets show that VDI can infer non-trivial domain indices, thereby significantly improving performance over state-of-the-art DA methods.
R ELATED W ORK
Typical Domain Adaptation.
There is a rich literature on domain adaptation (Pan & Yang, 2009;
Pan et al., 2010; Long et al., 2018;
Saito et al., 2018; Sankaranarayanan et al., 2018; Peng et al., 2019;
Prabhu et al., 2021;
Wang et al., 2020; Xu et al., 2022).
Typically they try to align source-domain and target-domain data in the latent space, with the hope that such domain-invariant encodings can generalize well on unseen data.
There are multiple ways to achieve such alignment, including distribution matching (Pan et al., 2010; Tzeng et al., 2014; Sun & Saenko, 2016; Peng et al., 2019;
Nguyen-Meidine et al., 2021), self-training (Zou et al., 2018; Kumar et al., 2020;
Prabhu et al., 2021), domain-specific normalization (Maria Carlucci et al., 2017; Mancini et al., 2019;
Tasar et al., 2020), and deep learning models with adversarial training (Ganin et al., 2016; Tzeng et al., 2017; Zhang et al., 2019; Zhao et al., 2017; Chen et al., 2019; Dai et al., 2019).
Most of these methods rely on (one-hot) domain identities for feature alignment.
Domain Adaptation with Domain Identities and Domain Indices.
There are also works that generate domain identities from data to improve domain adaptation.
Chen & Chao (2021) generates a sequence of domain identities for intermediate domains between a source domain and a target domain, trying to facilitate better incremental domain adaptation (Bobu et al., 2018).
Deecke et al.
(2021) generates a set of domain identities to split a dataset into different domains and perform multi-domain learning.
Du et al.
; Lu et al.
(2022) partition time series data by learning domain identities that maximize the domain-wise distribution gap.
All these works focus on generating (ordinal or one-hot) domain identities.
In contrast, our VDI assumes such domain identities are already given and focuses on inferring (continuous) domain indices, which contain richer and more interpretable information.
Note that in our setting, since domain identities are given , methods such as Deecke et al.
(2021) are equivalent to typical domain adaptation methods (Ganin et al., 2016;
Tzeng et al., 2017;
Prabhu et al., 2021), which are considered as our baselines in Sec. 5.
Recent studies have found that replacing (one-hot) domain identities with (continuous) domain indices improves adaptation performance (Wang et al., 2020; Xu et al., 2022).
However, none of these works provide a canonical definition of “domain index”; instead they mainly rely on intuition, e.g., using rotation angles as domain indices for RotatingMNIST (Wang et al., 2020) and using graph node embeddings as domain indices for adaptation across graph-relational domains (Xu et al., 2022).
More importantly, they assume that such domain indices are always available, which may not be true (Matsuura & Harada, 2020; Rebuffi et al., 2017); hence they are not applicable to our setting.
Also related to our work is Peng et al.
(2020a), which generates embeddings of visual domains to represent domain similarities; however, they do not formally define “domain embedding” and only work for classification tasks.
In contrast, our VDI provides a rigorous and formal definition for “domain indices” and handles both classification and regression tasks.
M ETHOD
In this section, we formalize the definition of “domain index” and describe our VDI for inferring domain indices.
We provide theoretical guarantees that VDI infers optimal domain indices in Sec. 4.
[Figure]
𝑁
3. P ROBLEM S ETTING AND N OTATION
We consider the unsupervised domain adaptation setting with N domains in total.
Each domain has domain identity k ∈ K =
[ N ] ≜ { 1 , . .
. , N } ; k is in either the source domain identity set K s or the target domain identity set K t .
Each domain k has D k data points.
Given n labeled data points { ( x si , y si , k si ) } ni =1 from source domains ( k si ∈ K s ), and m unlabeled data points { x ti , k ti } mi =1 from target domains ( k ti ∈ K t ), we want to (1) predict the label { y ti } mi =1 for target domain data, and (2) infer global domain indices β k ∈ R B β for each domain and local domain indices u i ∈ R B u for each data point .
α = { µ α , σ α } are the hyper-parameters for { β k } N k =1 ’s prior distributions.
Note that each domain has only one global domain index, but has multiple local domain indices, one for each data point in the domain (more details in Sec. 3.3).
We denote as z ∈ R B z
the data encoding generated from an encoder that takes x as input.
We use I ( · ; · ) to denote mutual information.
3. F ORMAL D EFINITION OF D OMAIN I NDEX
We formally define “domain index” as follows (please refer to notations in Sec.
3.1 if needed):
Definition 3.1 ( Domain Index ) .
Given data x and label y , a domain-level variable β and a data-level variable u are called global and local domain indices, respectively, if there exists a data encoding z such that the following holds:
(1) Independence between β and z : Global domain index β is independent of data encoding z , i.e., β ⊥⊥ z , or equivalently I ( β ; z ) = 0 .
This is to encourage domain-invariant data encoding z .
(2) Information Preservation of x : Data encoding z , local domain index u , and global domain index β preserves as much information on x as possible,
i
e., maximizing I ( x ; u , β , z ) .
This is to prevent β and u from collapsing to trivial solutions.
(3) Label Sensitivity of z : The data encoding z should contain as much information on the label y as possible to maximize prediction power, i e., maximizing I ( y ; z ) conditioned on z ⊥⊥ β .
This is to make sure the previous two constraints on β , u , and z do not harm prediction performance.
To summarize, β and u are considered the global and local domain indices, respectively, if ( β , u ) =
argmax β , u I ( x ; u , β , z )
+ I ( y ; z ) s.t.
I ( β ; z ) = 0 .
Later in Sec. 4, our theoretical analysis shows that maximizing our model’s evidence lower bound while adversarially training an additional discriminator (Sec. 3.4) is equivalent to inferring the optimal domain indices according Definition 3.1.
In Appendix A, we provides a rigorous discussion on the definition of “domain index”.
3. D OMAIN -I NDEXING V ARIATIONAL B AYES FOR DOMAIN ADAPTATION
Generative Process and Probabilistic Graphical Model.
Based on our definition, we propose our model: Variational Domain Indexing (VDI).
The basic idea is to infer the domain indices as latent variables during domain adaptation.
VDI is a generative model assuming the following generative process (see the corresponding graphical model in Fig. 1(left)).
For each domain k ,
(1) Draw global domain index β k from the Gaussian distribution p θ ( β |
each data
i with domain identity k : (a) Draw local domain index u i from the Gaussian distribution p θ ( u i | β k ) .
(b) Draw input x i from the Gaussian distribution p θ ( x i | u i ) .
𝑞(𝐮|𝐱)
𝐱 𝑞(𝜷|𝐮) 𝑞(𝐳|𝐱,𝐮,𝜷) 𝑝(𝐱|𝐮) 𝑝(𝐮|𝜷) 𝑝(𝑦|𝐳)
𝐮 𝜷 𝐳 ො𝐱 ෝ𝐮 𝑦 𝑝(𝐳|𝐱,𝐮,𝜷) 𝐱,𝐮,𝜷 𝐾𝐿 𝑅𝑒𝑔𝑢𝑙𝑎𝑟𝑖𝑧𝑒𝑟
𝐳 Reconstruct 𝐱
Reconstruct 𝐮
Prediction
Besides typical conditional dependencies defined in the graphical model (Fig. 1(left)), we enforce additional independence between β and z ; such independence is represented as a dashed line “ ” in
Fig. 1(left).
Note that there are multiple ways to satisfy such constraints during learning, e.g., using adversarial methods (Ganin et al., 2016; Tzeng et al., 2017; Zhang et al., 2019; Wang et al., 2020; Xu et al., 2022) and using the concentration loss (Xiao et al., 2021).
Generative Model and Inference Model.
Based on Fig.
1(left)
, we factorize the generative model p θ ( x , u , β , z , y | α ) into five conditional distributions (omitting the subscript i for clarity below):
[Equation]
where θ denotes the collection of parameters for the generative model, and p θ ( β | α )
=
N ( µ α , σ 2 α ) is a Gaussian distribution.
The predictor p θ ( y | z ) is a categorical distribution Cat ( f y ( z ; θ )) for classification tasks and a Gaussian distribution N ( µ y ( z ; θ ) ,
σ 2 y ( z ; θ )) for regression tasks; here f y ( z ; θ ) , µ y ( z ; θ ) , and σ y
( z ; θ ) are neural networks taking z as input.
Similarly, we have
[Equation]
We use an inference model q ϕ ( u , β , z | x ) to approximate the posterior distributions of the latent variables, i.e., p θ ( u , β , z | x ) .
As shown in Fig.
1(right), we factorize q ϕ ( u , β , z | x ) as
[Equation]
where ϕ denotes the collection of parameters for the inference model.
Specifically, we have
[Equation]
Note that µ · ( · ; · ) and σ · ( · ; · ) denote neural networks; θ , ϕ are neural network parameters.
Here q ϕ ( β | u ) requires special treatment and will be discussed in Eqn.
(14-17) below.
3. O BJECTIVE F UNCTION
Evidence Lower Bound.
We use an evidence lower bound (ELBO) as an objective to learn the generative and inference models.
Maximizing the ELBO learns the optimal variational distribution q ϕ ( u , β , z | x ) that best approximates the posterior distribution of the latent variables (including the domain indices) p θ ( u , β , z | x ) .
Specifically, we have the ELBO as:
[Equation]
With the factorization in Eqn. 1 and Eqn. 5, we decompose the ELBO as (omitting α to avoid clutter):
[Equation]
[Equation] [Equation]
[Figure]
(1) Reconstruct Data x from u (Eqn. 10).
q ϕ ( u | x ) and p θ ( x | u ) aim to reconstruct data x using the inferred u , encouraging u to preserve as much information on x as possible.
(2) Predict Label y from Latent z (Eqn. 11).
This term samples u , β , and z from q ϕ ( u | x ) , q ϕ ( β | u ) and q ϕ ( z | x , u , β ) , respectively, and then uses z to predict y in p θ ( y | z ) , encouraging z to contain as much information on y as possible to maximize prediction performance.
(3) Reconstruct Local Domain Index u from β (Eqn. 12).
Eqn.
12 samples u and β from q ϕ ( u | x ) and q ϕ ( β | u ) , respectively, and then uses the inferred β to reconstruct local domain index u in p θ ( u | β ) , encouraging β to preserve as much information on u and x as possible.
(4) Regularize All Latent Variables u , β , z (Eqn. 13).
Eqn. 13 includes two KL divergence terms between the inference model q ϕ ( · ) and the generative model p θ ( · )
as well as an entropy term for q ϕ ( u | x ) ; they all serve as regularizers to prevent overfitting.
For example, the first regularization term implies that q ( β | u ) should be close to the prior distribution p ( β ) .
Global Domain Index β and Local Domain Index u .
VDI uses a bi-level structure for domain indices: local domain index u ∈ R B u and global domain index β ∈ R B β .
Both u and β are low-dimensional compared to x ∈ R B x
, i.e., B u ≪ B x and B β
≪ B x .
The local domain index u is a low-dimensional vector (e.g., B u = 4 ) containing domain information for high-dimensional data x
(e.g., an image with B x = 256 × 256 ).
The global domain index β is an aggregation of all local domain indices u for data from the same domain.
Note that different data points x i and x j in the same domain ( k i = k j ) have different local domain indices , i.e., u i ̸ = u j , but share the same global domain index , i.e., β k i =
β k j .
VDI’s final goal is to infer the optimal global domain indices { β k } Nk = given only the data ( x i , y i ) and domain identities k i , thereby providing better interpretability and domain adaptation performance.
See Sec.
K for more discussion on β and u .
Difference between Domain Identities k and Global Domain Indices β .
Note that domain identities k are discrete values and therefore cannot describe rich relations (e.g., similarity and distance) among domains.
In contrast, global domain indices β are continuous vectors and therefore contain much richer information that describes relations (e.g., similarity and distance) among domains (see Sec.
5 for empirical results).
Our VDI assumes k is available and tries to infer β .
Inferring Global Domain Indices q ϕ ( β | u ) in Eqn. 5.
For each domain k , global domain index β k should aggregate domain information of all data in this domain.
We therefore propose to leverage local domain indices of all domain k ’s data points, U k
= [ u i ] k i =
k ∈ R D k × B u , to infer the global domain index β k .
Specifically, our process consists of four steps: (1) Grouping u i in
Domain k .
Group all local domain indices from the same domain k into one local index matrix (set), i.e., U k
= [ u i ] k i =
k ∈ R D k × B u .
(2) Pairwise Domain Distance.
Calculate the Earth
Mover’s distance (EMD) (Rubner et al., 2000) between each pair of local index matrices (sets)
S k,j = f EMD ( U k , U j ) ∈ R N × N , where S k,j is the EMD between domain k and j .
(3) Raw Global
Domain Indices.
According to the pairwise domain distance matrix S , use multi-dimensional scaling (MDS) (Borg & Groenen, 2005) to map each domain k into a B β -dimensional space and obtain the raw global domain index β rk ∈ R B β , i.e., [ β rk ]
Nk =
1 = f
MDS ( S ) =
[ f kMDS ( S )]
Nk =1 .
(4) Final Global
Domain Indices.
Feed the raw index β rk into the inference neural network to obtain the variational distribution N (cid:0) µ r ( β r k ; ϕ ) , σ 2 r ( β r k ; ϕ ) (cid:1) for the final global domain index β k ∈ R B β , where ϕ is the inference network parameters.
We summarize these four steps below:
[Equation]
Raw
[Equation]
Final Global
[Equation]
Discriminator with an Adversarial Loss.
To enforce independence between β and z , i.e., Part (1) of Definition 3.1, we train an additional discriminator D with an adversarial loss while maximizing the ELBO in Eqn. 9.
The discriminator is a neural network D ( · ) that takes z as input and predicts the global domain index ˆ β and domain identity ˆ k .
Essentially, D ( · ) plays a minimax game with the encoder inference network q ϕ ( z | x , u , β ) :
D ( · ) tries to reconstruct the global domain index
ˆ β and domain identity ˆ k , while the encoder q ϕ ( z | x , u , β ) tries to prevent D ( · ) from doing so by generating domain-invariant encoding z .
Denoting as R D the reconstruction loss, the discriminator loss L D can be written as:
[Equation]
In Sec. 4, we will prove that β is guaranteed to be independent of z if k is independent of z .
We therefore simplify Eqn. 18 into only classifying the domain identity k and use the log-likelihood as
L D,ϕ :
[Equation] [Equation]
Final Objective Function.
Putting Eqn.
9 and Eqn.
19 together, we have our final objective function:
[Equation]
where λ d is a hyper-parameter balancing two terms.
T HEORY
Below we provide theoretical guarantees for VDI’s objective function (Eqn. 20).
We analyze the first term in Lemma 4.1 and the second term in Lemma 4.2, show that Eqn.
20 lower-bounds a combination of mutual information terms (including I ( y ; z ) , I ( x ; u , β , z ) , and I ( z ; β ) ) plus some constants in
Theorem 4.1, and then show that one can learn domain indices β that satisfies Definition 3.1 when
Eqn.
20’s global optimum is achieved (Theorem 4.2).
All proofs are in Appendix B.
We start by analyzing VDI’s ELBO term L ELBO ( x , y ) in Eqn. 20 and proving that it is upper bounded by I ( y ; z )
+ I ( x ; u , β , z ) plus some constants in Lemma 4.1 below.
Lemma 4.1 ( Upper Bound of the ELBO of p θ ( x , y ) ) .
The ELBO of p θ ( x , y ) is upper bounded by the mutual information among observable variables x , y and latent variables u , β , z as below:
[Equation]
Since the entropy terms H ( y ) and H ( x ) in Eqn.
21 are both constant, maximizing the ELBO term
L ELBO ( x , y ) in Eqn.
20 is equivalent to maximizing I ( x ; u , β , z ) and I ( y ; z ) , corresponding to
Parts (2) and (3) of Definition 3.1, respectively.
Next we analyze VDI’s adversarial term L D,ϕ of Eqn. 20 in Lemma 4.2 below.
Lemma 4.2 ( Information Decomposition of the Adversarial Loss ) .
The global maximum of adversarial loss w.r.t.
discriminator D is decomposed as below:
[Equation]
(22) and the global minimum of max D E p ( k, x ) E q ϕ ( z | x ) [log D ( k | z )] is achieved if and only if I ( z ; β ) =
I ( z ; k | β ) = 0 .
[Figure]
Lemma 4.2 above shows that one can decompose max D E p ( k, x ) E q ϕ
( z | x ) [log D ( k | z )] into several information theoretic terms, including I ( z ; β ) , which is related to Part (1) of Definition 3.1.
With Lemma 4.1 and Lemma 4.2, we then show that VDI’s objective function in Eqn.
20 lower-bounds a combination of mutual information terms plus some constant entropy terms in Theorem 4.1 below.
Theorem 4.1 ( Objective Function as a Lower Bound ) .
The objective function involves both the
ELBO of p θ ( x , y ) and adversarial loss E p ( k, x ) E q ϕ
( z | x ) [log D ( k | z )] , and it is the lower bound for a combination mutual information and entropy terms:
[Equation]
With Theorem 4.1, we are now ready to analyze the global optimum of the minimax game in Eqn. 20.
Theorem 4.2 ( Global Optimum of VDI ) .
In VDI, when the global optimum (Eqn. 20) is achieved, it is guaranteed that (1) I ( z ; β ) = 0 , (2) I ( x ; u , β , z ) is maximized, and (3) I ( y ; z ) is maximized.
As Theorem 4.2 states, the global optimum of Eqn. 20 is guaranteed to satisfy all three conditions in Definition 3.1; therefore training VDI using the minimax game objective Eqn. 20 is equivalent to inferring the optimal domain indices.
E XPERIMENTS
In this section, we compare VDI with existing DA methods on both synthetic and real-world datasets.
5. D ATASETS
Circle (Wang et al., 2020) is a synthetic dataset with 30 domains for binary classification.
Fig.
3(a) shows 30 domains of Circle in different colors.
Fig.
3(b) shows positive (red) and negative (blue) data points, The first 6 domains are source domains, and the remaining 24 domains are target domains.
DG-15 and DG-16 (Xu et al., 2022).
DG-15 (Fig. 3(c)) and DG-60 (Fig. 9(b)) are synthetic datasets with 15 and 60 domains for binary classification, respectively.
In both datasets, we use 6 connected domains as the source domains and use others as target domains (see Table 4 in Sec.
F for details).
TPT-48 (Xu et al., 2022) is a real-world regression dataset that contains monthly average temperature for the 48 contiguous states in the US from 2008 to 2019 .
We use the first 6 months’ temperature as model input to predict the next 6 months’ temperature.
We formulate two DA tasks (Fig. 4):
• W (6) → E (42) :
Adapting models from the 6 states in the west to the 42 states in the east.
• N (24) → S (24) :
Adapting models from the 24 states in the north to the 24 states in the south.
We treat target domains one hop away from the closest source domain as Level-1 Target Domains , those two hops away as Level-2 Target Domains , and those more than two hops away as LevelTarget Domains (see Fig.
4 for an illustration).
CompCars (Yang et al., 2015) is a car image dataset with attributes including car types, viewpoints, and years of manufacture (YOMs).
The task is to recognize the car type given an image.
In
CompCars , data with each view point and each YOM is treated as a single domain.
We choose a subset of CompCars with 4 car types (MPV, SUV, sedan and hatchback), 5 viewpoints (front (F), rear
[Table]
We report the average MSE of all domains as well as more detailed average MSE of Level-1, Level-2,
Level-3 target domains, respectively (Fig. 4).
D2V cannot perform regression and thus has no results.
Note that there is only one single DA model per column.
We mark the best result with bold face .
[Table]
(R), side (S), front-side (FS), and rear-side (RS)), ranging from 2009 to 2014.
It contains 30 domains ( 5 viewpoints × 6 YOMs) with 18735 images in total.
We choose the domain with front view and
YOM 2009 as the source domain, and all the others as target domains.
5. B ASELINES
We compared our proposed VDI with state-of-the-art DA methods, including Domain Adversarial
Neural Networks ( DANN ) (Ganin et al., 2016), Adversarial Discriminative Domain Adaptation ( ADDA ) (Tzeng et al., 2017), Conditional Domain Adaptation Neural Networks ( CDANN ) (Zhao et al., 2017), Margin Disparity Discrepancy ( MDD ) (Zhang et al., 2019), SENTRY (Prabhu et al., 2021) and Domain to Vector ( D2V ) (Peng et al., 2020b).
We also report results when the model is only trained in the source domains without adapting to the target domains ( Source-Only ).
Different from VDI that works for both classification and regression tasks, MDD, SENTRY and D2V only work for classification tasks.
We managed to adapt MDD and SENTRY for the regression tasks on TPT- (see App.
H for details); D2V cannot be adapted and has no results for TPT-48 .
Both Wang et al.
(2020) and Xu et al.
(2022) assume domain indices are available; therefore they are not applicable to our settings where the goal is to infer domain indices (which are unavailable from data).
5. R ESULTS
Circle, DG-15 and DG-60 .
Table 1 shows the accuracy of evaluated methods on Circle , DG-15 , and DG-60 ; all datasets have complex domain relations, making it challenging to perform domain adaptation without knowing ground-truth domain indices.
Indeed, we observe that on Circle , all baselines only perform marginally better than random guess (50 % accuracy).
Moreover, on DGmost of baselines perform even worse than a random guess, possibly due to overfitting the source domains 2 .
In contrast, our VDI achieves very high accuracy (over 94% ) on all three datasets, significantly outperforming all baselines, thanks to the inferred indices (e.g., Fig. 3(d) and Fig. 7).
To verify that VDI infers non-trivial domain indices β , we connect the domain pairs within a distance threshold ( ∥ β k − β j ∥ <
ϵ ) to reconstruct the domain graph on DG-15 and DG-60 .
Compared with the ground-truth domain graphs, VDI achieves area under the ROC curve (AUC) of 0 .
83 for DGand 0 .
91 for DG-60 .
Fig.
3(d) shows an example inferred domain graph for DG-15 .
TPT-48 .
Table 2 shows the mean square error (MSE) for all the methods on TPT-48 .
In terms of average MSE across all domains, we observe that most methods suffer from negative transfer on both tasks, with only DANN and SENTRY marginally improving upon Source-Only.
In contrast, our VDI can further improve the performance and achieve the lowest average MSE on both tasks.
[Figure] [Figure]
[Figure] [Figure]
[Table]
Fig.
5 plots the inferred domain indices β ∈ R 2 for all 48 domains.
For reference, we color the inferred domain indices according to ground-truth latitude (Fig. 5(left)) and longitude (Fig. 5(right)); note that VDI does not have access to latitude and longitude during training .
The plots show that
VDI’s inferred domain indices are highly correlated with each domain’s latitude and longitude.
For example, Florida (FL) has the lowest latitude among all 48 states and is hence the left-most circle in
Fig. 5(left).
We also observe that states with similar latitude or longitude do have similar domain indices β .
These results demonstrate that VDI can infer reasonable domain indices.
CompCars .
Table 3 shows the classification accuracy on all DA methods.
Results show that most of the methods outperform Source-Only, with our VDI achieving the most significant improvement.
Fig.
6 plots the inferred domain indices β ∈ R 2 for all 30 domains.
For reference, we also color the plotted circles according to YOMs (Fig. 6(left)) and viewpoints (Fig. 6(right)); note that VDI does not have access to YOMs and viewpoints during training .
Interestingly, we have the following observations that are consistent with intuition: (1) domains with the same viewpoint or YOM have similar domain indices; (2) domains with “front-side” and “rear-side” viewpoints have similar domain indices; (3) domains with “front” and “rear” viewpoints have similar domain indices.
C ONCLUSION
We identify the problem of inferring domain indices as latent variables, provide a rigorous definition of “domain index”, develop the first general method for addressing it, and provide detailed theoretical analysis as well as empirical results.
We demonstrate the effectiveness of our proposed VDI for inferring domain indices and show its potential for significant practical applications.
As a limitation, our method still assumes the availability of domain identities to identify different domains.
Therefore it would be interesting future work to explore jointly inferring domain indices and domain identities.
A CKNOWLEDGEMENT
The authors thank the reviewers/AC for the constructive comments to improve the paper.
ZX and
HW are partially supported by NSF Grant IIS-2127918.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors
A F ORMAL D EFINITION OF D OMAIN I NDEX
Given predefined variables of data x , label y and domain, let β be a domain associated variable, i.e., in each domain it has the same value, and let u be a data associated variable, i.e., for each data point it can have a different value.
Definition A.1 ( Orthogonal Information ) .
For any domain associated variable β and data associated variable u , we define their orthogonal information as the following:
(1) Data orthogonal information I x ( β , u ) : I x ( β , u ) :
= max z : z ⊥⊥ β
I
( x ; u , β , z ) .
(2) Label orthogonal information
I y ( β ) : I y ( β ) :=
max z : z ⊥⊥ β
I
( y ; z ) .
Definition A.2 ( Domain Index ) .
We call ( β , u ) domain index if they maximize the sum of data and label orthogonal information, i.e., β, u ∈ argmax β ′ , u ′ I x ( β ′ , u ′ )
+ I y ( β ′ ) .
Lemma A.1.
By Definition A.2, Domain Index maximizes both data orthogonal information and label orthogonal information, i.e., if ( β , u ) is domain index, then I x ( β , u ) =
H ( x ) and I y ( β ) =
I ( y ; x ) .
Proof.
We prove the lemma above by first constructing a trivial domain index.
Let β 0 , u 0 be variables that are independent of x , i.e., β 0 ⊥⊥ x and u 0 ⊥⊥ x ; let z
= x and hence I ( x ; z 0 )
=
H ( x ) .
We have
I x ( β 0 , u 0 ) ≥ I ( x ; u 0 , β 0 , z 0 ) =
H ( x ) and I y ( β 0 ) ≥ I ( y ; z 0 )
=
I ( y ; x ) .
For any domain index ( β , u ) by Definition A.2, we have H ( x ) + I ( y ; x ) ≤
I x ( β 0 , u 0 )
+ I y ( β 0 ) ≤ I x ( β, u )
+ I y ( β ) ≤
H ( x ) + I ( y ; x ) ; therefore I x ( β , u ) + I y ( β ) =
H ( x ) + I ( y ; x ) .
Since I x ( β , u ) ≤ H ( x ) and
I y ( β ) ≤ I ( y ; x ) , we then have I x ( β , u ) =
H ( x ) and I y ( β ) =
I ( y ; x ) , concluding the proof.
B T HEORETICAL A NALYSIS
Lemma B.1 ( Upper Bound of the ELBO of p θ ( x , y ) ) .
The ELBO of p θ ( x , y ) is upper bounded by the mutual information among observable variables x , y and latent variables u , β , z as below:
[Equation]
Optimizing the ELBO of p θ ( x , y ) is equivalent to enhancing mutual information between label y and classification latent variable z and between data variable x and all latent variables u , β , z .
Proof.
Firstly
[Equation]
we provide the ELBO of log p θ ( x , y ) as below:
[Equation]
Then we have:
[Equation]
To help our analysis, we introduce a helper joint distribution of variables x , y, z , which is r ( x , y, z )
=
p ( x , y ) q ϕ ( z | x ) .
This distribution has the following properties,
(1) r ( x , y ) =
p ( x , y ) , r ( x ) = p ( x ) , r ( y ) = p ( y ) .
(2) r
( z | y, x ) =
q ϕ ( z | x ) =
r ( z | x ) .
(3) r
( y | z , x ) =
p ( y | x ) =
r ( y | x ) .
(4) I r
( y ; z | x ) = 0 , i.e. y ⊥⊥ z | x under distribution r .
(implied by 2, 3).
The First Term of the ELBO .
Its upper bound is provided as follows:
[Equation]
when log p θ ( y | z )
=
r ( y | z ) , we have E p ( x ,y )
E q ϕ ( z | x ) [ p θ ( y | z )]
= I r
( y ; z ) + H ( y ) .
So, max p θ
[Equation]
( y ) .
Then we have:
[Equation] [Equation]
Furthermore, we prove another upper bound of the first term:
[Equation]
The first inequality takes the equal sign when p θ ( y | z ) is a constant w.r.t.
z given x , i.e., p θ ( y | z , x ) =
p θ ( y | x ) .
The second equality takes the equal sign, when E q ϕ ( z | x ) p θ ( y | z )
= p ( y | x ) .
We set q ϕ ( z | x ) and p θ ( y | z ) as q ∗ ϕ ( z | x ) and p ∗ θ ( y | z ) when the optimality of q ϕ ( z | x ) and p θ ( y | z ) is achieved, i.e., log E q ϕ ( z | x ) p θ ( y | z )
= log p ( y | x ) =
E q ϕ ( z | x ) log p θ ( y | z ) .
Then we have:
[Equation]
( y )
Thus, we prove the relationship among the two forms of upper bound for the first term and the mutual information I ∗ ( y ; z ) w.r.t.
the optimal q ϕ ( z | x ) and p θ ( y | z ) .
[Equation]
Moreover, we prove the upper bound for I ∗ ( y ; z ) and I p ( y ; x ) :
[Equation]
Finally, we have E p ( x ,y )
E q ϕ ( z | x ) [log p θ ( y | z )]
≤ I r ( y ; z ) ≤ I ∗ ( y ;
z
[Equation]
= I p ( y ; x ) ≤ H ( y ) .
The Second Term of the ELBO .
We introduce another helper joint distribution of variables x , u , β , z , which is s ( x , u , β , z ) =
p ( x ) q ϕ ( u , β , z | x ) .
Its upper bound is proved as follows:
[Equation]
where I s ( x ; u , β , z ) is w
[Equation]
r
[Equation]
t
[Equation]
s ( x , u , β , z ) ,
i
[Equation]
e., p ( x ) q ϕ ( u , β , z | x ) .
Then we have:
[Equation]
Applying Eqn.
27 and Eqn.
28 to Eqn.
26, we have:
[Equation]
x )]
For clarity, we use I ( y ; z ) and I ( x ; u , β , z ) in place of I r ( y ; z ) and I s ( x ; u , β , z ) ,repectively, later in the Appendix and the main paper.
Lemma B.2 ( Information Decomposition of the Adversarial Loss ) .
The global maximum of adversarial loss w.r.t.
discriminator D is decomposed as below:
[Equation]
and the global minimum of max D E p ( k, x ) E q ϕ ( z | x ) [log D ( k | z )] is achieved if and only if I ( z ; β ) =
I ( z , k | β ) = 0 .
The optimization of adversarial loss is to reduce the mutual information between data encoding z and domain identity k while the mutual information is reduced between data encoding z and domain identity β .
Proof.
Define
[Equation]
Then we have:
[Equation]
When the equality holds, the discriminator becomes optimal, i.e., D ( k | z )
=
s ( k | z ) .
Due to p ( β | k ) is a function f ( k ) mapping from k to β .
Therefore, we have:
[Equation]
Thus, s ( k, β , z ) =
q ϕ ( z | k ) q ϕ ( β | k ) p ( k ) .
The three random variables meet the Markov chain β ← k → z .
By chain rule for mutual information, we have:
[Equation]
where I q ( z , β | k )
= 0 .
So,
[Equation]
, k |
[Equation]
We also have:
[Equation]
[Equation] [Equation]
] = I
[Equation]
z
[Equation]
k
[Equation]
H
[Equation]
Therefore, min ϕ max D E p ( k, x ) E q ϕ ( z | x ) [log D ( k | z )] =
− H ( k ) while I ( z ; k ) = 0 and thus
I ( z ; β ) =
I ( z , k | β ) = 0 .
Theorem B.1 ( Objective Function as a Lower Bound ) .
The objective function involves both the
ELBO of p θ ( x , y ) and adversarial loss E p ( k, x ) E q ϕ
( z | x ) [log D ( k | z )] , and it is the lower bound for a combination mutual information and entropy terms:
[Equation]
Proof.
Theorem B.1 is proved by applying Eqn.
25 mine Eqn. 29.
Theorem B.2 ( Global Optimum of VDI ) .
In VDI, the global optimum (Eqn. 20) is achieved, if and only if (1) I ( z ; β )
=
I ( z , k ; β ) = 0 , (2) I ( x ; u , β , z ) is maximized, and (3) I ( y ; z ) is maximized.
Proof.
The condition (1) is gotten from Lemma B.2.
The last two conditions (2) and (3) are gotten from Lemma B.1.
As Theorem B.2 states, the global optimum of Eqn.
20 is guaranteed to satisfy all three conditions in Definition 3.1; therefore training VDI using the minimax game objective Eqn. 20 is equivalent to inferring the optimal domain indices.
C V ISUALIZATION OF INFERRED DOMAIN INDICES FOR C IRCLE
Fig.
7 shows the inferred domain indices for Circle.
Normalized True Domain Indices 0.
N o r m a li
z e d
E s t i m a t e d
D o m a i n
I n d i c e s
Correlation=0.
D V ISUALIZATION OF LOCAL DOMAIN INDICES FOR DG-
Fig.
8 shows the local domain indices for DG-15.
This inspired us to derive global domain index by
Earth Moving distance.
Local Indices for all domains in DG0121451369 1.
Local Indices for 3 selected domains in DG
E A RCHITECTURE AND I MPLEMENTATION D ETAILS E. A RCHITECTURE
As is shown in Fig.
2, we use neural networks to estimate the density function of each distribution.
For Circle , DG-15 , DG-60 and TPT-48 , we use multi-layer perceptrons for estimating q ( u | x ) , while for CompCars , we use ResNet-18 (He et al., 2015) for q ( u | x ) .
All the other neural networks are multi-layer perceptrons.
To ensure the robustness of training, we fix the variance of some distributions and only use a neural network to estimate the mean of these distributions.
All the input data are normalized with their mean and variance.
E. H YPERPARAMETERS
For experiments on all 4 datasets, we set the dimension of global domain indices to 2.
For Circle ,
DG-15 , DG-60 , the dimension of local domain indices is 4, while for TPT-48 and CompCars , the dimension of local domain indices is 8.
Our model is trained with 20 to 70 warmup steps, learning rates ranging from 1 × 10 − 5 to 1 × 10 − 4 , and λ d ranging from 0.1 to 1.
E. A DDITIONAL LOSS ON LOCAL DOMAIN INDICES
During the inference of local domain indices u , we incorporate a modified contrastive loss from Chen et al.
(2020) to improve the coherence of u .
The loss aims to maximize the agreement between local domain indices of data points in the same domain.
Specifically, we sampled a minibatch of size b from each of the N domains, resulting in a large batch of size bN .
For each x k,i , the i -th sample in the minibatch of domain k , we pair it with x k, ( i +1) mod b , the neighbouring data point within the same domain.
We denote j =
( i + 1) mod b , and refer to the sampled pair as ( x k,i , x k,j ) .
We then sample corresponding u k,i , u k,j from q ϕ ( u | x ) (Eqn. 6).
With a multi-layer perceptron, we map u k,i , u k,j to h k,i , h k,j , which are subsequently used to calculate the contrastive loss as follows:
[Equation]
where sim ( · , · ) is the similarity function between 2 vectors, and τ is the temperature.
In practise, we use the cosine similarity and set τ = 1 .
F D ATASET S UMMARY
Table 4 summarizes the statistics for all the datasets used in our experiments.
[Table]
Input Dim
Toy/
[Table]
Adaptation Task
Circle 3,
Toy 2-Way Classification
DG
[Table] [Table]
Toy 2-Way Classification
DG
[Table] [Table]
Toy 2-Way Classification
TPT6,
Real
Regression
CompCars 18,
[Table]
Real
[Table] [Table]
DG
[Table]
[Figure]
𝐱
[Figure]
𝑁 for the VDI’s inference model.
G DG-15 AND DG-
For completeness, we show DG-15 and DG-60 in Fig.
We use ‘red’ and ‘blue’ to indicate positive and negative data points inside a domain.
The boundaries between ‘red’ half circles and ‘blue’ half circles show the direction of ground-truth decision boundaries in the datasets.
DG-15 is a synthetic dataset with 15 domains for binary classification.
As shown in Fig. 3(c), these domains form a domain graph (DG) of 15 nodes, with adjacent domains having similar decision boundaries.
Each domain contains 100 data points.
We use 6 connected domains as the source domains and use others as target domains.
Similarly, DG-60 is another synthetic dataset withdomains, each of which contains 100 data points.
We use 6 connected domains as source domains and the remaining 54 domains as target domains.
H A DAPTING MDD AND SENTRY FOR R EGRESSION T ASKS
For MDD , we simply replace its cross-entropy loss with an L 2 loss.
For SENTRY , since it requires confidence scores during training, we include another classification network that predicts whether the average temperature of next 6 months will go up or down compared to the previous 6 months, thereby providing confidence scores.
I N EW E DGE T YPE FOR P ROBABILISTIC G RAPHICAL M ODELS
In Fig.
10, we introduce a new edge type, “ ”, to denote independence.
β k z enforces independence between z and β k , i.e, p ( z | β k ) =
p ( z ) ; note that this does not contradict β k
→ z , as long as there are multiple paths between β k and z .
In this case, we have two paths between β k and z , β k
→ z and β k → u → z .
As a simplified example, consider only a sub-graph in Fig.
10 with only three nodes β k , u , and z .
Essentially our algorithm tries to learn two conditional dependencies p ( u | β k ) and p ( z | β k , u ) .
Adding a dashed edge β k z ensures that p ( z | β k ) =
p ( z ) , which is equivalent to
[Equation]
which is equivalent to
[Equation]
It is easy to see that Eqn.
32 only introduces an additional constraint, but does not contradict the learning of conditional dependencies p ( u | β k ) and p ( z | β k , u ) .
Combining the ELBO below
[Equation]
with the additional constraint in Eqn.
32, we have the following objective function
[Equation]
which is equivalent to Eqn. 20.
J D OMAIN I DENTITIES VERSUS D OMAIN I NDICES
Compared to domain identities, domain indices can better capture the similarity and complex relations among different domains, thereby better guiding the adaptation process.
This is also empirically verified in previous works such as Wang et al.
Xu et al.
As a simple example, note that domain identities are one-hot vectors, the distance between any two domain identities are identical; in contrast, domain indices are real-value vectors and therefore contain richer information.
More specifically, since the encoder takes domain indices as additional input, domain indices tend to encourage data x of similar domains to go through similar transformations to produce the encoding z ; consequently, the predicted decision boundaries of similar domains will also be similar.
Theoretically, the target domain’s error is upper-bounded by three terms, i.e., source error, domain gap, and optimal joint error of the source and target domains (Ben-David et al., 2010).
Encouraging similar transformations for similar domains can reduce the joint error term of the upper bound, thereby achieving better performance.
K G LOBAL VERSUS L OCAL D OMAIN I NDICES
Global versus Local Domain Indices.
Local domain indices u contain instance-level information, i.e., each data point has a different local domain index u ; in contrast, global domain indices contain domain-level information, i.e., all data points of the same domain share the same global domain index β .
Necessity of Local Domain Indices.
Essentially, local domain indices u serve as extremely lowdimensional summaries of representations z ; for example, in CompCars experiments the numbers of dimensions for u and z are 4 and 4096 , respectively.
With such low-dimensional u , global domain indices can be efficiently inferred from u in the 4 -dimensional space using Eq.
In contrast, if one only uses global indices without u , one would need to compute the Earth Mover’s distance
(EMD) and multi-dimensional scaling (MDS) in Eq.
(8-11) in the 4096 -dimensional space; this is much more computationally expensive and dramatically slows down model training.
Therefore local domain indices are necessary in VDI.
Necessity of EMD and MDS.
It is also worth noting that EMD and MDS are also necessary when inferring global domain indices β from local domain indices u .
The main reason is that u ’s distribution tends to be multi-modal.
As an example, we plot the local indices from 3 domains in
DG-15 in Fig.
8 of Appendix D.
We can see that the u ’s in Domain 11 contain two clusters (i.e. two modes), with one cluster corresponding to positive data and the other corresponding to negative data.
This is also the case for both Domain 1 and 0, but the distance between the two clusters gets smaller and smaller.
Therefore, directly using the mean of u ’s as the global domain index β does not work because all three domains will have similar means, and consequently similar β ’s. Using EMD and MDS fixes this issue since EMD can naturally compute the distance between two multi-modal distributions.
Our preliminary experiments also confirm their necessity; removing EMD and MDS will significantly bring down VDI’s performance.
Independence between z and the Global Index β .
In typical domain adaptation methods, an encoder first takes x as input to produce the representation z , and a predictor then takes z as input to predict the label.
To improve z ’s generalization across different domains, it is common practice (Ganin et al., 2016;
Zhao et al., 2017; Zhang et al., 2019; Tzeng et al., 2017; Wang et al., 2020; Xu et al., 2022) to enforce independence between the domain index β and representation z such that domain-specific information is removed from z .
Therefore the assumption/constraint of independence between β and z is natural.
Why Independence between z and the Local Index u is Not Required.
As mentioned above, we need the local indices u to capture the multi-cluster structure in the data, with each cluster corresponding to local indices for data with the same label y .
Since z contains label-specific information, if we enforce independence between z and u , such label-specific information will be removed from u , making it impossible for u to capture the label-specific multi-cluster structure in the data.
How the Global Index β Indicates Different Domains from the Definition.
One key property that makes sure the global index β can distinguish different domains is the second point of Definition 3.1, i.e., information preservation of x .
Specifically, maximizing the mutual information I ( x ; u , β , z ) ensures that β contains as much information on x as possible under the constraint that β
and z is independent.
Since we assume covariant shift exists (i.e., different domains have different distributions over x ), this property helps β to distinguish (indicate) different domains.
This is also empirically verified by results in Fig. 3(d), Fig. 5, and Fig.
6, where VDI successfully inferred meaningful domain indices β from different datasets.
L L ARGER F IGURES
In this section, we provide larger versions of figures for domain index visualization in the main paper.
[Figure] [Figure]
(Colors Indicate Viewpoints)
Front (F)
Front-side (FS)
Side (S)
Rear-side (RS)
Rear (R)
(Colors Indicate YOMs)