Distribution-Informed Neural Networks for Domain Adaptation Regression
Abstract
In this paper, we study the problem of domain adaptation regression, which learns a regressor for a target domain by leveraging the knowledge from a relevant source domain.
We start by proposing a distribution-informed neural network, which aims to build distribution-aware relationship of inputs and outputs from different domains.
This allows us to develop a simple domain adaptation regression framework, which subsumes popular domain adaptation approaches based on domain invariant representation learning, reweighting, and adaptive Gaussian process.
The resulting ﬁndings not only explain the connections of existing domain adaptation approaches, but also motivate the efﬁcient training of domain adaptation approaches with overparameterized neural networks.
We also analyze the convergence and generalization error bound of our framework based on the distribution-informed neural network.
Speciﬁcally, our generalization bound focuses explicitly on the maximum mean discrepancy in the RKHS induced by the neural tangent kernel of distribution-informed neural network.
This is in sharp contrast to the existing work which relies on domain discrepancy in the latent feature space heuristically formed by one or several hidden neural layers.
The efﬁcacy of our framework is also empirically veriﬁed on a variety of domain adaptation regression benchmarks.
Introduction
Domain adaptation tackles the knowledge transfer from a source domain with adequate label information to a relevant target domain with little or no label information.
It is shown [5, 39, 11, 13] that the generalization performance on the target domain can be improved by leveraging the source domain knowledge in both classiﬁcation [59] and regression [12] settings.
In this paper we focus on studying the domain adaptation regression problem, as classiﬁcation can be naturally formulated as regression [34].
In the past decades, most domain adaptation approaches were developed using the following paradigms: learning domain invariant representation [42, 43], reweighting the source examples [30, 11, 13] or deriving adaptive transfer kernel for Gaussian process [7, 44].
More recently, modern practice for domain adaptation repeatedly demonstrates the beneﬁt of considering overparameterized neural networks as the backbones for feature extraction [37, 20, 47, 1].
In this case, a L -layer neural network would be manually divided into two parts: the feature extraction function (e.g., the ﬁrst l layers) and the prediction function (e.g., the left L  l layers).
Then, the core idea of domain adaptation with overparameterized neural networks is to enhance the adaptation of source and target domains in the hidden feature space learned by the feature extraction layers.
Nevertheless, it is unclear how those overparameterized neural networks affect the convergence and generalization of domain adaptation approaches during model training.
Moreover, the heuristic selection of feature extraction layers might lead to suboptimal performance of domain adaptation approaches in practice, as the lower and higher layers of an overparameterized neural network might encode different information [57]. [Table] [Table] [Table]
Compared with standard supervised learning, the key challenge of domain adaptation lies in the distribution shift between source and target domains.
This indicates that the relationship between inputs and outputs might be different across domains.
As illustrated in Figure 1, due to the distribution shift across domains, an input data point (e.g., x 2 = x 3 ) might have signiﬁcantly different outputs at source and target domains (i.e., y 2
= y 3 ).
This is also observed in recent work [51].
To solve this problem, we propose a distribution-informed neural network to build the uniﬁed relationship of inputs and outputs from different domains.
This neural network can be explained as the integration of standard feature representation learning and input-oriented distribution representation learning.
Then, a simple domain adaptation regression framework named DINO is proposed based on the distribution-informed neural network, followed by the theoretical analysis on its convergence and generalization performance.
[Figure]
Similar data points (e.g., x 2 and x 3 from different domains might have different outputs.
(2) Source data point x 3 is more likely to be aligned with target data point x 4 , as they have similar local model evolution
In particular, we show that the distribution discrepancy of source and target domains can be deﬁned by the evolution of distribution-informed neural network during training.
That is, we measure the distribution shift using the maximum mean discrepancy (MMD) [24] in the reproducing kernel Hilbert space (RKHS) induced by the neural tangent kernel of distribution-informed neural network.
Compared to previous works [37, 20, 1], our domain discrepancy measure has the following beneﬁts.
First, it can be estimated from the entire neural network, whereas previous works usually estimate the domain discrepancy in the pre-deﬁned feature extraction layers.
For example, [37] extracts features of input source and target examples and then estimates the discrepancy using MMD with standard kernels (e.g., RBF kernel).
Intuitively, it is a two-fold composition kernel of the neural kernel induced by the feature extractor and the standard kernel.
Second, our domain discrepancy measure focuses on the training dynamics of the neural network (e.g., f ✓ +  ✓ ( x )  f ✓ ( x ) ).
This is in sharp contrast to the previous works which measure the distribution shift using the static state of the neural network (e.g., f ✓ ( x ) ).
As shown in Figure 1, a source data point x 3 might have similar output with both x 1 and x 4 from target domain, when simply considering the trained neural network f ✓ ( x ) .
But we see that the neural network has similar evolution patterns at x 3 and x 4 but different patterns at x 3 and x 1 .
That is, x 3 is more likely be aligned with x 4 after distribution minimization across domains, That explained why the training dynamics of the neural network can better identify the distribution shift across domains.
In addition, we show that in special cases, the training dynamic of the distribution-informed neural network can also explain the rationale of the existing domain adaptation techniques.
To be more speciﬁc, we have the following observations.
(1) At random initialization, our distribution-informed neural network is equivalent to the adaptive Gaussian process [7, 44].
(2) Under gradient descent training, our domain adaptation framework based on the distribution-informed neural network can recover the prediction function of the reweighting domain adaptation approaches [30, 11, 13].
When the distribution representation is shared by all the examples, our framework degenerates into standard domain-invariant representation learning [37, 20, 1].
Compared to previous works, our contributions can be summarized as follows.
• Uniﬁed Framework: We propose a simple domain adaptation regression framework DINO based on the distribution-informed neural network, and then identify its connections to previous techniques, including domain invariant representation learning, reweighting and adaptive Gaussian process.
Theoretical Results: It is shown that for the distribution-informed neural network with sufﬁcient width, the convergence and generalization of our framework can be theoretically guaranteed.
Different from previous deep domain adaptation theories, we propose to measure the distribution shift across domains using the training dynamics of the neural network.
• Empirical Performance: We experimentally investigate the performance of our DINO framework on a variety of domain adaptation regression benchmarks, and show its effectiveness over state-of-the-art baselines.
The rest of this paper is organized as follows.
We introduce the related work in Section 2, and summarize the preliminaries of previous domain adaptation techniques in Section 3.
Section 4 shows the proposed domain adaptation regression framework, followed by its theoretical analysis.
The experimental results are presented in Section 5.
Finally, we conclude the paper in Section 6.
Related Work 2.1 Domain Adaptation
Domain adaptation [5, 51, 52, 61, 62] improves the generalization performance of a learning algorithm on the target domain, by leveraging the knowledge from a relevant source domain with adequate labeled data.
There are three solutions to bridge the distribution gap between source and target domains.
(1) Domain invariant representation: It maps the source and target examples into a new feature space such that the distribution discrepancy between source and target domains can be explicitly minimized [42, 37, 43, 20, 60, 59, 10].
(2) Reweighting:
The core idea is to correct the difference between distributions by multiplying the prediction loss of each source example by a non-negative weight [30, 14, 11, 13, 47].
(3) Gaussian process: It generalized the standard Gaussian process regression to the domain adaptation setting by learning an adaptive transfer kernel [7, 44, 38, 50].
By using the neural networks as the backbones, modern domain adaptation techniques have achieved state-of-the-art performance in a variety of real-world tasks [28, 8].
Nevertheless, little effort has been devoted to theoretically analyzing the domain adaptation techniques with overparameterized neural networks.
More recently, it has been revealed [10] that there is a gap between domain adaptation regression and domain adaptation classiﬁcation problems.
That is, one common loss function of domain adaptation regression approaches is mean square error (MSE), whereas domain adaptation classiﬁcation approaches [37, 20] often uses cross-entropy loss with softmax.
Speciﬁcally, softmax changes the feature scales (i.e., Frobenius norm of feature matrix
||
H || F where H is the hidden feature learned by the neural network), and the change of feature scales might lead to the performance degradation of domain adaptation [10].
Therefore, in this paper, we focus on the neural network with MSE loss for domain adaptation regression.
Different from the existing domain adaptation regression approaches, e.g., domain invariant representation learning [10], reweighting [16], and adaptive
Gaussian process [7], we develop the domain adaptation regression algorithms based on a novel distribution-informed neural network and derive the theoretical analysis regarding the convergence and generalization bound for our algorithms.
2.2 Overparameterized Neural Networks
It has been revealed from the perspective bias-variance decomposition [55] that the generalization error of neural networks decreases with respect to the model complexity (e.g., number of neurons).
The beneﬁt of increasing the number of neurons has also been empirically conﬁrmed in modern neural networks [32, 58].
Moreover, the convergence and generalization of overparameterized neural networks have been studied in the neural tangent kernel (NTK) regime [31, 3, 4, 35, 17, 18, 2, 48, 9, 33].
However, those works focus on the standard supervised learning with the assumption that training and testing examples follow the same distribution.
In real scenarios, this assumption often fails due to the distribution shift [38].
More recently, it is found that previous theoretical results can be generalized to both federated learning [29] and multi-task learning [45].
Our work fundamentally differs from them in that little label information is available in the target domain for our studied domain adaptation problem.
This motivates us to explicitly analyze the domain discrepancy when using neural networks for domain adaptation.
Preliminaries
In this section, we introduce the notations and background of different domain adaptation techniques.
3.1 Notation and Neural Network Architecture
Let X and Y denote the input space and output space.
Following [6], we assume that the joint probability distribution P of any domain is drawn from a probability distribution space P over X ⇥ Y .
For domain adaptation regression, we have a source domain with labeled examples { x src i , y src i } n src i =1 drawn from joint probability distribution P src , and a target domain with both label examples { x tgt j , y tgt j } n l tgt j = from P tgt and unlabeled examples { x tgt j } n u tgt j =1 ( n u tgt ⌧ n l tgt ) drawn from marginal probability distribution P tgt X .
The goal is to predict the outputs of unlabeled target examples by leveraging the label information from the source domain.
In this paper, we consider an L -layer fully-connected neural network f ✓ ( · ) , which maps the input x 2 X ⇢
R d into the output y 2 Y ⇢
R .
Here ✓ denotes all the parameters of f ( · ) (e.g., ✓ 0 denotes the initialized parameters at time stamp t = 0 ) and d denotes the dimensionality of input data.
In next section, we generalize the standard fully-connected neural network f ✓ ( · ) to the distribution-informed neural network ˜ f
( · ) for domain adaptation.
For notation simplicity, we let X src =
{ x src i } n src i =1 and Y src =
{ y src i } n src i =1 for source examples.
We use similar notations X tgt l and Y tgt l for target examples.
Then we use X = X src
[ X tgt l denote all the labeled training inputs, and Y
=
Y src
[ Y tgt l be the corresponding outputs.
3.2 Domain Adaptation Techniques
We ﬁrst review the existing domain adaptation techniques.
In Subsection 4.4, we show that these techniques can be explained in a uniﬁed framework.
3.2.1 Domain Invariant Representation
The key idea of domain invariant representation learning is to map the source and target examples into a new feature space where the distribution discrepancy across domains is explicitly minimized [20, 59].
The objective function of this framework can be summarized as follows.
[Equation]
where d ( P src , P tgt ) denotes the domain discrepancy measure in the hidden feature space learned by f ✓ ( · ) , and L ( · , · ) is the loss function.
3.2.2 Reweighting
Reweighting aims to correct the difference between source and target distributions by reweighting the source examples, i.e., multiplying the prediction loss of every source example by a non-negative weight [14, 11, 13].
[Equation]
where w src ( x, y )
=
P tgt ( x,y ) P src ( x,y )  0 denotes the importance of every source example.
The importance ratio w src ( x, y ) can be estimated using kernel mean matching (KMM) [30, 56] or adversarial learning [47, 16].
3.2.3 Adaptive Gaussian Process
Following [7], the domain adaptation regression problem can also be formulated as the Gaussian process with an adaptive transfer kernel.
Speciﬁcally, the adaptive transfer kernel can be deﬁned as
[Equation]
[Equation] where ⌧ 2 [0 , 1] is a trainable parameter explicitly indicating the relatedness of source and target domains, and K ( · , · ) is a base kernel, e.g., RBF kernel.
Then following the standard Gaussian process regression [49], the predictions of this adaptive Gaussian process over testing target data can be analytically derived.
A Uniﬁed Framework
In this section, we propose the distribution-informed neural network, and then present a simple domain adaptation regression framework.
4.1 Distribution-Informed Neural Network
An L -layer fully-connected neural network f ( · ) can be written as f ✓ ( x ) =
✓ <L ( x ) T w where ✓ <L is the vector of parameters in the ﬁrst L  1 layers and w is the parameter of the output layer (we assume the output layer has no bias).
For a single task, the neural network f ( · ) can model the relationship between input data and output label.
But it might suffer in the domain adaptation scenarios due to the distribution shift.
Thus, given a domain distribution P 2 P and an input example x ⇠ P , we propose to explicitly learn the distribution-related output as g w g ( P | x ) =
x ( P ) T w g where  x ( P ) is the input-oriented distribution feature representation, and w g is the parameter.
Given ﬁnite basis examples ˜ x 1 , · · · , ˜ x n ⇠ P ,  x ( P ) can be formally deﬁned as follows.
[Equation]
where  x, ˜ x i 2 R is the coefﬁcient of  x ( P ) in the space spanned by h · , ˜ x i i K X and indicates the similarity of input example x and basis example ˜ x i .
Here h · , ˜ x i
i K X denotes the feature map of ˜ x i in the NNGP kernel space induced by inﬁnitely-wide f ( · ) .
Therefore,  x ( P ) represents the distribution representation of P when an example x is observed in the domain associated with sampling distribution P .
One special case is when  x, ˜ x i = 1 /n , it degenerates into the mean mapping of P in the kernel space, which is commonly used for measuring the distribution discrepancy/similarity [24].
Compared with this plain distribution representation 1 n
[Equation]
X ,  x ( P ) pays more attention to the region around x with large  x, ˜ x i for ˜ x
i 2 X .
For example, for data points x 1 and x 4 in Figure 1, they would share the same plain distribution representation, as they are sampled from the same target domain.
But it is feasible to differentiate them using the parametric distribution representation  x ( P ) .
Generally, by taking both example x and its associated probability P as random variables, Bayes’ theorem tells us that P ( x, P ) = P ( x ) · P ( P | x ) where P ( · ) denotes the probability of an observed event.
The event involving x can be described by observing its representation f ( x ) ; given x , the event involving P is the input-oriented distribution representation  x ( P ) .
This motivates us to develop a distribution-informed neural network as follows.
[Equation]
The intuition behind Eq.
(5) can be explained as follows.
For domain adaptation, the feature representation of an input example x is domain-dependent, as two similar examples ( x src ⇡ x tgt ) from different domains might have distinctive outputs ( y src 6 = y tgt ).
The input-oriented distribution representation  x ( P ) allows us to identify the source and target examples with similar inputs but distinctive outputs.
By analyzing the distribution-informed neural network at initialization and under gradient descent training, we propose two domain adaptation algorithms: DINO - INIT (see
Subsection 4.2) and DINO - TRAIN (see Subsection 4.3), respectively.
4.2 Initialization
We start by studying the distribution-informed neural network at initialization, e.g., all the parameters of distribution-informed neural network ˜ f ( · ) are initialized as standard normal variables.
Inspired by the connections between standard neural networks and Gaussian processes [15, 34, 22, 53], we show that the distribution-informed neural network ˜ f ( x ) in Eq (5) is equivalent to a domain adaptive
Gaussian process, which explains the existing domain adaption regression algorithm
Lemma 4.1.
Assume that all the parameters of distribution-informed neural network ˜ f ( · ) are initialized as standard normal variables, when the network width goes to inﬁnity, the output function of ˜ f
( x ) in Eq (5) at initialization is iid centered Gaussian process, i.e., ˜ f ( · ) ⇠ N  0 , K [Equation]  with
[Equation]
where K X ( · , · ) is the NNGP kernel induced by the neural network f ( · ) over input space X , and
K P ( · , · ) is a distribution kernel identifying the similarity of two input-oriented distributions P and
P 0 over distribution space P :
[Equation]
[Equation]
where n ( n 0 ) is the number of basis examples sampled from the distribution P ( P 0 ).
This lemma motivates us to derive Gaussian process regression named DINO - INIT (see Algorithm 1 in
Appendix A.10) with adaptive transfer kernel deﬁned as K DA , where K DA (( x, P ) , ( x 0 , P 0 )) measures the input-oriented distribution similarity of P and P 0 .
For notation simplicity, we represent the distribution-informed kernel K DA (( x, P ) , ( x 0 , P 0 ))
as K DA ( x, x 0 ) in the following.
In this case, we consider a noisy model y ri = ˜
f
( x ri , P r ) + ✏ ri ( r 2 { src , tgt } ), where the noise ✏ ri follows a zero-mean
Gaussian N (0 ,  2 r ) .
Given labeled training examples ( X, Y ) , we assume a prior Gaussian process p ( Y ) =
N (0 , K DA ) where K DA ( X, X ) is a block matrix.
That is, K DA ( X, X ) = 
[Equation]
DA [Equation] where K DA 11 := K DA ( X src , X src ) ( K DA 22 :=
K DA ( X tgt l , X tgt l ) ) denotes the kernel matrix of source (target) data, and K DA 12 =
( K DA 21 ) T :=
K DA ( X src , X tgt l ) denotes the kernel matrix across domains.
Then for testing target examples X tgt ⇤ , their output can be inferred using the predictive distribution p ( Y | X tgt ⇤ , X src , X tgt l ) =
N (¯ µ, ¯ ⌃ ) .
Similar to standard Gaussian process regression [49], the mean and variance of the predictive distribution can be exactly calculated as follows.
[Equation]
where C = K DA ( X, X ) +   2 src I n src   2 tgt I n l tgt  .
Here I denotes the identity matrix.
This adaptive Gaussian process involves the following parameters: coefﬁcient  x,x i in the kernel function K DA and noise variance  src ,  tgt .
Following [7], we optimize these parameters by maximizing the conditional likelihood p ( Y tgt l | X tgt l , X src , Y src ) .
It can be seen that p ( Y tgt l | X tgt l , X src , Y src ) is also a Gaussian process, i.e., p ( Y tgt l | X tgt l , X src , Y src )
=
N
[Equation]
Instantition of the coefﬁcient  x, ˜ x
i : It is notable that it can only optimize the coefﬁcient  x, ˜ x i of
Eq.
(4) for training (source or target) examples.
The coefﬁcient of testing target examples X tgt ⇤ is unknown.
To solve this problem, in this paper, we formulate the estimate of  x r , ˜ x ri ( r 2 { src , tgt } ) as  x, ˜ x ri =
[ x r  ˜ x ri ] T w r , where w r is a domain-speciﬁc parameter vector and  denotes vector concatenation.
Note that domain adaptation assumes that the domain labels of all input examples are known.
As shown in Eq.
(4), the input-oriented distribution representation can be learned from the domain-speciﬁc vector w r and the basis examples ˜ x r 1 , · · · , ˜ x rn r from domain r .
Here r is determined by the domain label of the input example x .
As illustrated in Algorithm 1, we use all the training source (target) examples as the basis source (target) examples ˜ x r 1 , · · · , ˜ x rn r .
As a result, the input-oriented distribution representation learning of testing target examples
X tgt ⇤ can be inferred by w tgt and the pre-deﬁned basis target examples.
4.3 Gradient Descent Training
Here we assume that the model parameters of the distribution-informed neural network can be updated by gradient descent during training.
Based on the distribution-informed neural network, we propose a novel DINO - TRAIN algorithm.
Its objective function using mean square error is formulated as follows.
[Equation]
where ↵ 2 (0 , 1) and µ  0 are hyperparameters to balance different terms.
The ﬁrst two terms are standard supervised learning losses over labeled source and target examples, and the third one is the empirical maximum mean discrepancy (MMD) [24] in the RKHS H DA induced by the neural tangent kernel of our distribution-informed neural network, i.e.,
[Equation]
where n tgt = n l tgt + n u tgt is the total number of target training examples.
As shown in Theorem 4. below, our framework empirically minimizes the upper error bound of the expected prediction error in the target domain (see Algorithm 2 in Appendix A.10).
Remark.
In our framework of Eq.
(7), we measure the distribution shift using the training dynamics of the distribution-informed neural network over the source and target examples.
This is fundamentally different from previous works [20, 59, 1] associated with a two-stage domain discrepancy measurement.
That is, those works would ﬁrst learn the feature representation in hidden neural layers and then measure the domain discrepancy in the learned feature space.
In the second stage, they usually require additional modules, e.g., auxiliary neural networks [10, 21, 19] or pre-deﬁned distribution distances [37].
Therefore, compared to previous works, our discrepancy measure Eq. (8) can not only unify the domain adaptation regression framework with neural networks, but also enable the theoretical convergence and generalization analysis in the following.
Before analyzing the dynamics of the distribution-informed neural network under Eq.
(7), we ﬁrst introduce the following assumption.
Assumption 4.2.
Given labeled data X =
X src [ X tgt l , we assume  min ( ⇥ ( X, X )) > [-
 and-]0 and  min
[Equation]
> 0 , where ⇥ ( X, X ) =
r ✓ f ( X ) r ✓ f ( X ) T is standard neural tangent kernel [31] induced by f ( · ) with inﬁnite width, and K P |X ( X, X ) is the input-oriented distribution kernel 1 (see Eq.
Here,  min ( · ) denotes the smallest eigenvalue.
The assumption  min ( ⇥ ( X, X )) > 0 has been studied in previous works [31, 18, 3].
It holds as long as no two inputs x i and x j are parallel in real scenarios.
The implication of this assumption is that the input data (e.g., images) would not be linearly changed by a single factor, i.e., there is no two inputs such that x i = & · x j for some & 2 R .
Lemma 4.3.
Given the labeled training data X =
X
src
[ X tgt l and the basis examples { ˜ x ri } ˜ n r i =
( r 2 { src , tgt } ), we let ⌥ 2 R ( n src + n l tgt ) ⇥ (˜ n src +˜ n tgt ) denote the coefﬁcient matrix of K P |X ( X, X ) , where for x k 2 X ( k = 1 , · · · , n src + n l tgt ), its k th row [ ⌥ ] k, : =
[  x k , ˜ x src 1 , · · · ,  x k , ˜ x src ˜ n src , 0 , · · · , 0] when x k 2 X src , [ ⌥ ] k, : = [0 , · · · , 0 ,  x k , ˜ x tgt 1 , · · · ,  x k , ˜ x tgt ˜ n tgt ] otherwise.
Then, the assumption  min ( K P |X ( X, X )) > 0 holds when ⌥ T ⌥ is positive deﬁnite.
Lemma 4.3 shows that in real scenarios, the assumption  min ( K P |X ( X, X )) > 0 can be guaranteed by the positive deﬁniteness of the coefﬁcient matrix ⌥ .
We also empirically evaluate the smallest eigenvalue  min
[Equation] in Subsection 5.2.
Theorem 4.4.
For any coefﬁcient  x, ˜ x i of the input-oriented distribution representation in Eq.
(4), there exists ⌘ ⇤ 2 R + such that for the inﬁnitely-wide distribution-informed neural network ˜ f
( · ) trained under gradient ﬂow with learning rate ⌘ < ⌘ ⇤ , the test prediction ˜
f ✓ t ( X tgt ⇤ ) of the domain adaptation regression in Eq.
(7) over test target data X tgt ⇤ is
[Equation]
⌘ where ⇥ DA ( · , · ) is the distribution-informed NTK, i.e., ⇥ DA ( x, x 0 )
= ⇥
( x, x 0 ) · K P |X ( P , P 0 | x, x 0 ) and
˜ C
= diag { ↵ /n src , · · · , ↵ /n src
| {z } n src , (1  ↵ ) /n
l tgt , · · · , (1  ↵ ) /n
l tgt | {z } n l tgt } is a diagonal matrix.
Moreover, under the assumption 4.2, when the network width goes to inﬁnity, lim t !1 ˜ f ✓ t ( X tgt ⇤ ) converges to a
Gaussian process with mean µ ( X tgt ⇤ ) and variance ⌃ ( X tgt ⇤ , X tgt ⇤ ) as follows.
[Equation]
where “ + h.c. " means “plus the Hermitian conjugate".
In addition, our result on generalization bound is given in the following theorem.
Theorem 4.5.
Assume for any training example ( x, y ) 2 X ⇥ Y , we have ( ˜ f
( x, P )  y ) 2  M 0 for some constant M 0  0 .
Let ˜ F be the hypothesis space induced by inﬁnitely-wide neural networks ˜ f .
Then, for any ˜ f 2 ˜ F and  > 0 , with probability at least 1   , the expected error in the target domain can be bounded as follows.
[Equation]
src X i =
[Equation]
⇣ ˜ f ( x src i , P src )  y src i ⌘ 2 + 1  ↵ n l tgt n l tgt X j =1 ⇣ ˜ f
( x tgt j , P tgt )  y tgt j ⌘ + 8 ↵ M 0 · MMD ⇥ DA  P src , P tgt  + ⌦ where ⌦ = 2 ↵ < n src ( H src ) + 2(1  ↵ ) <
n l tgt ( H tgt ) + M q ( n src + n l tgt )log(1 /  ), H r = { ( x, y ) !
( ˜ f ( x r , P r )  y r ) 2 : ˜ f 2 F} is a set of functions ( r 2 { src , tgt } ), < n r ( H r ) is the Rademacher complexity of H r given n r examples, and M = max { ↵ M 0 /n src , (1  ↵ ) M 0 /n
l tgt } .
Though domain adaptation theories [5, 39] have been generalized to deep learning scenarios in recent years, neural networks are simply considered as an expressive feature extractor in modern generalization errors [59].
Nevertheless, recent works [60, 36] reveal that the feature space learned by the neural networks might worsen the adaptation between source and target domains.
As a comparison, Theorem 4.5 provides a uniﬁed generalization error for neural network based domain adaptation, as it directly measures the domain discrepancy over input source and target examples in the RKHS induced by the distribution-informed NTK ⇥ DA .
4.4 Discussion 4.4.1 Gaussian Process at Initialization
One special case of distribution-informed neural network at initialization is that when the coefﬁcient  x,x i = 1 /n || ¯  x ( P ) ||
K X and ¯  x ( P ) =
1 n
P ni =1 h · , x i
i K X , it can be seen that  x ( P ) =
n ¯  x ( P )
P ni =1 h · , x i
i K X is the normalized mean mapping [24] of data distribution P in the NNGP kernel space.
Then, the distribution kernel K P |X ( P , P 0 | x, x 0 ) in Eq.
(6) can be explained as the inner product of the mean mappings of P and P 0 .
Moreover, using the deﬁnition of maximum mean discrepancy (MMD) [24], it can be shown that K P |X ( P , P 0 | x, x 0 ) =
c  ˆ MMD
KX ( P , P 0 ) 2¯
x ( P )¯  x ( P 0 ) , where ˆ MMD K X ( · ) denotes the empirical MMD in the NNGP kernel space, and c = 1 n
{+P ni,j =1 K X ( x i , x j )+ 1 n 
+}
P n 0 i,j =1 K X ( x 0 i , x 0 j ) .
It implies that K P |X ( P , P 0 | x, x 0 ) is negatively correlated with the popular
MMD estimator.
MMD measures the distribution distance using the difference of mean mappings of distributions, whereas K P |X ( P , P 0 | x, x 0 ) measures the distribution similarity via the inner product of the mean mappings of the distributions P and P 0 in the NNGP kernel space.
The following corollary shows that the existing domain adaptive Gaussian process [7] can be explained as implicitly learning the distribution-informed representation in the kernel space.
Corollary 4.6.
With the assumptions in Lemma 4.1, when  x, ˜ x i = 1 / ||
P ni =1 h · , ˜ x i
i K X
|| ,
the Gaussian process induced by ˜ f
( x ) at the initialization would be equivalent to the adaptive Gaussian process in [7] over NNKP kernel.
4.4.2 Fully Trained Model
When there exist some labled target examples, the objective function of the reweighting domain adaptation techniques (see Subsection 3.2.2) can be rewritten as follows.
[Equation]
The following corollary holds that our framework Eq.
(7) can recover the reweighting approach Eq. (9) in the function space.
Corollary 4.7.
With the assumption in Theorem 4.4, our framework Eq.
(7) with distribution-informed neural network ˜ f
( · ) can recover the popular reweighting domain adaptation approach Eq.
(9) in the function space.
Besides, we show that when the distribution representation is shared by all the examples, our framework can be naturally degenerated into the domain-invariant representation learning [37]. [Table] [Table] [Figure]
[Figure]
Corollary 4.8.
With the assumption in Theorem 4. [Table] our framework Eq.
(7) with distribution-informed neural network ˜ f
( · ) would be degenerated into the standard domain invariant representation learning in Eq.
[Table]
where the domain discrepancy measure d ( · , · ) is instantiated with MMD in RKHS induced by standard NTK ⇥ .
Experiments
Data Sets: We use three domain adaptation regression benchmarks.
Following [10], we use two image data sets: dSprites [40] and MPI3D [23].
Speciﬁcally, dSprites has 737,280 images from three domains: Color (C), Noisy (N) and Scream (S).
For each image, it has three regression tasks, i.e., predicting three factors of variations (scale, position X and Y).
MPI3D contains over 3M images from three domains: Toy (T), Realistic (RC) and Real (RL).
It is shown [10] that for each image, it involves two regression tasks, i.e., predicting three factors of variations (position X and Y).
In addition, we also use a plant phenotyping data set.
It predicts diverse traits (e.g., Nitrogen) of plants related to the plants’ growth using leaf hyperspectral reﬂectance.
Here we consider the following two domains [46]: Maize (M) and Maize_UNL (MU).
In our case, the task is to predict the Nitrogen content of maize using the leaf hyperspectral reﬂectance (see Appendix A.11 for more details).
Baselines: In the experiments, we consider the following baseline methods.
(1) Plain Gaussian process: NNGP [34] and NTKGP [25].
(2) Domain adaptive Gaussian processes: AT-GP [7] and
TL-NTK [38].
(3) Deep domain adaptation methods: DAN [37], WANN [16] and RSD [10].
Implementations:
In the experiments, our algorithms are implemented using a L -layer ( L = 6 ) fully-connected neural network with ReLU (see Appendix A.11 for more details).
The induced
NNGP and neural tangent kernels induced can be estimated using the Neural Tangents package [41].
In addition, we set ↵ = 0 .
5 and µ = 0 .
1 for DINO - TRAIN .
5.1 Results
Table 1, Table 2 and Table 3 provide the domain adaptation regression results on dSprites, MPI3D and Plant Phenotyping data sets.
Following [10], we report the Mean Absolute Error (MAE) between the predicted outputs and the ground-truth outputs in the target test set (the best results are indicated in bold).
It is observed that our method DINO achieves competitive performance over the Gaussian process baselines.
The weakly-trained DINO - INIT slightly outperforms DINO - TRAIN in some cases.
This observation is consistent with previous work [33].
The relatively unstable [Table] performance of DINO - TRAIN might be caused by the large covariance of Gaussian process with NTK [25].
In addition, we also compare DINO with domain adaptation methods instantiated by overparameterized neural networks.
WANN [16] reweights the source examples, and DAN [37] and RSD [10] learn the domain invariant representation.
The performance comparison in Figure 2 conﬁrms the effectiveness of our DINO approach over those state-of-the-art baselines [Table]
5.2 Analysis
[Table]
Ablation Study: We investigate the impact of input-oriented distribution representation in our DINO framework.
We consider the following variants.
DINO - INIT ( DINOTRAIN )
w/o distribution feature: it would not utilize the input-oriented distribution representation, i.e., g w g ( P | x ) = 1 for all x 2 X .
DINO - INIT ( DINO - TRAIN )
w uni- form distribution feature: the coefﬁcient  x, ˜ x i = 1 /n is a constant.
The results are given in Table 4.
We observe that the distribution representation improves the domain adaptation performance.
Compared to globally shared distribution representation with  x, ˜ x i = 1 /n , our parametric  x ( P ) in Eq.
(4) encourages to learn the local distribution representation around the input data point x .
The empirical results in Table 4 conﬁrmed the efﬁcacy of the input-oriented distribution representation.
Impact of Model Architecture and Discrepancy Measure: Figure 3a shows the results of DINO with different number of neural layers L on dSprites (C ! N).
The results indicate that the performance of DINO is positively related to the expressiveness of NTK and NNGP kernel induced by the neural network.
In addition, we compare the proposed
[Equation] .e. [Equation] standard RBF kernel) in domain adaptation regression, where MMD-RBF is estimated in the feature space learned by different neural layers (see Appendix A.11 for more details).
As shown in Figure 3b, the results on Plant Phenotyping data set reveal that the selection of feature space affects the model performance when using MMD-RBF, and our proposed [Equation]
Positive Deﬁniteness of K P |X ( X, X ) : We report the smallest eigenvalue of the distribution kernel
K P |X ( X, X ) over the training examples on Plant Phenotyping data set.
The smallest eigenvalues of K P |X ( X, X ) in DINO - TRAIN after training are 3 e  6 and 8 e  7 on M !
MU and MU !
M, respectively.
This empirically conﬁrms the positive deﬁniteness of K P |X ( X, X ) in DINO - TRAIN .
Conclusion
In this paper, we study the domain adaptation regression by proposing a distribution-informed neural network.
Our domain adaptation framework with distribution-informed neural network subsumes the existing adaptation approaches based on domain invariant representation, reweighting and adaptive
Gaussian process.
We also analyze the convergence and generalization bound of our framework.
The experiments demonstrate the effectiveness of our DINO framework over state-of-the-art baselines
Acknowledgments and Disclosure of Funding
This work is supported by National Science Foundation under Award No.
IIS-1947203, IIS2117902, IIS-2137468, and Agriculture and Food Research Initiative (AFRI) grant no. 2020-6702132799/project accession no.1024178 from the USDA National Institute of Food and Agriculture.
The views and conclusions are those of the authors and should not be interpreted as representing the ofﬁcial policies of the funding agencies or the government.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Abstract and Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Appendix A.10.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work does not involve any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.2A.9.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 5 and Appendix A.11.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.11.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]