Distribution-Informed Neural Networks for Domain Adaptation Regression
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this paper, we study the problem of domain adaptation regression, which learnsa regressor for a target domain by leveraging the knowledge from a relevant sourcedomain.
We start by proposing a distribution-informed neural network, whichaims to build distribution-aware relationship of inputs and outputs from differentdomains.
This allows us to develop a simple domain adaptation regression framework, which subsumes popular domain adaptation approaches based on domaininvariant representation learning, reweighting, and adaptive Gaussian process.
Theresulting ﬁndings not only explain the connections of existing domain adaptationapproaches, but also motivate the efﬁcient training of domain adaptation approacheswith overparameterized neural networks.
We also analyze the convergence andgeneralization error bound of our framework based on the distribution-informedneural network.
Speciﬁcally, our generalization bound focuses explicitly on themaximum mean discrepancy in the RKHS induced by the neural tangent kernelof distribution-informed neural network.
This is in sharp contrast to the existingwork which relies on domain discrepancy in the latent feature space heuristicallyformed by one or several hidden neural layers.
The efﬁcacy of our framework isalso empirically veriﬁed on a variety of domain adaptation regression benchmarks.
Introduction
Domain adaptation tackles the knowledge transfer from a source domain with adequate label information to a relevant target domain with little or no label information.
It is shown [9, 8, 48, 16, 18]that the generalization performance on the target domain can be improved by leveraging the sourcedomain knowledge in both classiﬁcation [72] and regression [17] settings.
In this paper we focus on studying the domain adaptation regression problem as classiﬁcation canbe naturally formulated as regression [42].
In the past decades, most domain adaptation approacheswere developed using the following paradigms: learning domain invariant representation [21, 54, 56],reweighting the source examples [37, 16, 18], or deriving adaptive transfer kernel for Gaussianprocess [11, 57].
More recently, modern practice for domain adaptation repeatedly demonstrates thebeneﬁt of considering overparameterized neural networks as the backbones for feature extraction [45,27, 60, 1].
In this case, a L -layer neural network would be manually divided into two parts: thefeature extraction function (e.g., the ﬁrst l layers) and the prediction function (e.g., the left L  llayers).
Then, the core idea of domain adaptation with overparameterized neural networks is toenhance the adaptation of source and target domains in the hidden feature space learned by the featureextraction layers.
Nevertheless, it is unclear how those overparameterized neural networks affect theconvergence and generalization of domain adaptation approaches during model training.
Moreover,the heuristic selection of feature extraction layers might lead to suboptimal performance of domainadaptation approaches in practice, as the lower and higher layers of an overparameterized neuralnetwork might encode different information [69].
Compared with standard supervised learning, the key challenge of domain adaptation lies in thedistribution shift between source and target domains.
This indicates that the relationship betweeninputs and outputs might be different across domains.
As illustrated in Figure 1, due to the distributionshift across domains, an input data point (e.g., x 2 = x 3 ) might have signiﬁcantly different outputsat source and target domains (i.e., y 2
= y 3 ).
This is also observed in recent work [50].
To solvethis problem, we propose a distribution-informed neural network to build the uniﬁed relationship ofinputs and outputs from different domains.
This neural network can be explained as the integrationof standard feature representation learning and input-oriented distribution representation learning.
Then, a simple domain adaptation regression framework named DINO is proposed based on thedistribution-informed neural network, followed by the theoretical analysis on its convergence andgeneralization performance.
[Figure]
(1) Similar data points (e.g., x 2 and x 3 from different domains) might have different outputs.
(2) Source data point x 3 is more likely to be aligned with target data point x 4 , as they have similar local model evolution.
In particular, we show that the distribution discrepancy of source and target domains can be deﬁnedby the evolution of distribution-informed neural network during training.
That is, we measure the distribution shift using the maximum mean discrepancy(MMD) [31] in the reproducing kernel Hilbert space(RKHS) induced by the neural tangent kernel ofdistribution-informed neural network.
Compared toprevious works [45, 27, 1], our domain discrepancymeasure has the following beneﬁts.
First, it can beestimated from the entire neural network, whereasprevious works usually estimate the domain discrepancy in the pre-deﬁned feature extraction layers.
Forexample, [45] extracts features of input source andtarget examples and then estimates the discrepancyusing MMD with standard kernels (e.g., RBF kernel).
Intuitively, it is a two-fold composition kernel of theneural kernel induced by the feature extractor andthe standard kernel.
Second, our domain discrepancymeasure focuses on the training dynamics of the neural network (e.g., f ✓ +  ✓ ( x )  f ✓ ( x ) ).
This is in sharpcontrast to the previous works which measure the distribution shift using the static state of the neuralnetwork (e.g., f ✓ ( x ) ).
As shown in Figure 1, a source data point x 3 might have similar output withboth x 1 and x 4 from target domain, when simply considering the trained neural network f ✓ ( x ) .
Butwe see that the neural network has similar evolution patterns at x 3 and x 4 but different patterns at x 3and x 1 .
That is, x 3 is more likely be aligned with x 4 after distribution minimization across domains,
That explained why the training dynamics of the neural network can better identify the distributionshift across domains.
In addition, we show that in special cases, the training dynamic of the distribution-informed neuralnetwork can also explain the rationale of the existing domain adaptation techniques.
To be morespeciﬁc, we have the following observations.
(1) At random initialization, our distribution-informedneural network is equivalent to the adaptive Gaussian process [11, 57].
(2) Under gradient descenttraining, our domain adaptation framework based on the distribution-informed neural network canrecover the prediction function of the reweighting domain adaptation approaches [37, 16, 18].
When the distribution representation is shared by all the examples, our framework degenerates intostandard domain-invariant representation learning [45, 27, 1].
Compared to previous works, ourcontributions can be summarized as follows.
• Uniﬁed Framework: We propose a simple domain adaptation regression framework DINO based onthe distribution-informed neural network, and then identify its connections to previous techniques,including domain invariant representation learning, reweighting and adaptive Gaussian process.•
Theoretical Results: It is shown that for the distribution-informed neural network with sufﬁcientwidth, the convergence and generalization of our framework can be theoretically guaranteed.
Different from previous deep domain adaptation theories, we propose to measure the distributionshift across domains using the training dynamics of the neural network.• Empirical Performance:
We experimentally investigate the performance of our DINO frameworkon a variety of domain adaptation regression benchmarks, and show its effectiveness over state-ofthe-art baselines.
The rest of this paper is organized as follows.
We introduce the related work in Section 2, andsummarize the preliminaries of previous domain adaptation techniques in Section 3.
Section 4 showsthe proposed domain adaptation regression framework, followed by its theoretical analysis.
Theexperimental results are presented in Section 5.
Finally, we conclude the paper in Section 6.
Related Work
2.1 Domain Adaptation
Domain adaptation [9, 8] improves the generalization performance of a learning algorithm onthe target domain, by leveraging the knowledge from a relevant source domain with adequatelabeled data.
There are three solutions to bridge the distribution gap between source and targetdomains.
(1) Domain invariant representation: It maps the source and target examples into anew feature space such that the distribution discrepancy between source and target domains canbe explicitly minimized [21, 54, 45, 56, 27, 73, 72, 14, 70].
(2) Reweighting:
The core idea isto correct the difference between distributions by multiplying the prediction loss of each sourceexample by a non-negative weight [37, 19, 16, 18, 60].
(3) Gaussian process: It generalized thestandard Gaussian process regression to the domain adaptation setting by learning an adaptive transferkernel [11, 57, 62, 20, 46].
By using the neural networks as the backbones, modern domain adaptationtechniques have achieved state-of-the-art performance in a variety of real-world tasks [35, 12].
Nevertheless, little effort has been devoted to theoretically analyzing the domain adaptation techniqueswith overparameterized neural networks.
More recently, it has been revealed [14] that there is a gap between domain adaptation regressionand domain adaptation classiﬁcation problems.
That is, one common loss function of domain adaptation regression approaches is mean square error (MSE), whereas domain adaptation classiﬁcationapproaches [45, 27] often uses cross-entropy loss with softmax.
Speciﬁcally, softmax changes thefeature scales (i.e., Frobenius norm of feature matrix
||
H || F where H is the hidden feature learned 119 by the neural network), and the change of feature scales might lead to the performance degradationof domain adaptation [14].
Therefore, in this paper, we focus on the neural network with MSEloss for domain adaptation regression. Different from the existing domain adaptation regressionapproaches, e.g., domain invariant representation learning [14], reweighting [23], and adaptive Gaussian process [11], we develop the domain adaptation regression algorithms based on a noveldistribution-informed neural network and derive the theoretical analysis regarding the convergenceand generalization bound for our algorithms. 2.2 Overparameterized Neural Networks
It has been revealed from the perspective bias-variance decomposition [7, 67] that the generalizationerror of neural networks decreases with respect to the model complexity (e.g., number of neurons).
The beneﬁt of increasing the number of neurons has also been empirically conﬁrmed in modernneural networks [39, 71].
Moreover, the convergence and generalization of overparameterized neuralnetworks have been studied in the neural tangent kernel (NTK) regime [38, 4, 5, 43, 24, 25, 3, 61,13, 15, 6, 41, 2, 53, 51].
However, those works focus on the standard supervised learning with theassumption that training and testing examples follow the same distribution.
In real scenarios, thisassumption often fails due to the distribution shift [47, 46].
More recently, it is found that previoustheoretical results can be generalized to both federated learning [36] and multi-task learning [58].
Our work fundamentally differs from them in that little label information is available in the targetdomain for our studied domain adaptation problem.
This motivates us to explicitly analyze thedomain discrepancy when using neural networks for domain adaptation.
Preliminaries
In this section, we introduce the notations and background of different domain adaptation techniques.
3.1 Notation and Neural Network Architecture
Let X and Y denote the input space and output space.
Following [10], we assume that the joint probability distribution P of any domain is drawn from a probability distribution space P over X ⇥ Y .
For 144 domain adaptation regression, we have a source domain with labeled examples { x src i , y src i } n src i =
1 drawnfrom joint probability distribution P src , and a target domain with both label examples { x tgt j , y tgt j } n l tgt j =1from P tgt and unlabeled examples { x tgt j } n u tgt j =1 ( n u tgt ⌧ n l tgt ) drawn from marginal probability distribution P tgt X .
The goal is to predict the outputs of unlabeled target examples by leveraging the labelinformation from the source domain.
In this paper, we consider an L -layer fully-connected neural network f ✓ ( · ) , which maps the input x 2 X ⇢
R d into the output y 2 Y ⇢
R .
Here ✓ denotesall the parameters of f ( · ) (e.g., ✓ 0 denotes the initialized parameters at time stamp t = 0 ) and ddenotes the dimensionality of input data.
In next section, we generalize the standard fully-connectedneural network f ✓ ( · ) to the distribution-informed neural network ˜ f
( · ) for domain adaptation.
Fornotation simplicity, we let X src =
{ x src i } n src i =1 and Y src =
{ y src i } n src i =1 for source examples.
We usesimilar notations X tgt l and Y tgt l for target examples.
Then we use X = X src
[ X tgt l denote all thelabeled training inputs, and
Y =
Y src
[ Y tgt l be the corresponding outputs.
3.2 Domain Adaptation Techniques
We ﬁrst review the existing domain adaptation techniques.
In Subsection 4.4, we show that thesetechniques can be explained in a uniﬁed framework.
3.2.1 Domain Invariant Representation
160 The key idea of domain invariant representation learning is to map the source and target examples intoa new feature space where the distribution discrepancy across domains is explicitly minimized [21,27, 72].
The objective function of this framework can be summarized as follows.
[Equation]
where d ( P src , P tgt ) denotes the domain discrepancy measure in the hidden feature space learned byf ✓ ( · ) , and L ( · , · ) is the loss [-function.3.2.
-]function.
3.2.2 Reweighting
166 Reweighting aims to correct the difference between source and target distributions by reweightingthe source examples, i.e., multiplying the prediction loss of every source example by a non-negativeweight [19, 16, 18].
[Equation]
where w src ( x, y )
=
P tgt ( x,y ) P src ( x,y )  0 denotes the importance of every source example.
The importance ratio w src ( x, y ) can be estimated using kernel mean matching (KMM) [37, 68] or adversariallearning [60, 23].
3.2.3 Adaptive Gaussian Process
173 Following [11], the domain adaptation regression problem can also be formulated as the Gaussianprocess with an adaptive transfer kernel.
Speciﬁcally, the adaptive transfer kernel can be deﬁned as
[Equation]
where ⌧ 2 [0 , 1] is a trainable parameter explicitly indicating the relatedness of source and targetdomains, and K ( · , · ) is a base kernel, e.g., RBF kernel.
Then following the standard Gaussian processregression [63], the predictions of this adaptive Gaussian process over testing target data can beanalytically derived.
A Uniﬁed Framework
In this section, we propose the distribution-informed neural network, and then present a simpledomain adaptation regression framework.
4.1 Distribution-Informed Neural Network
An L -layer fully-connected neural network f ( · ) can be written as f ✓ ( x ) =
✓ <L ( x ) T w where ✓ <
L isthe vector of parameters in the ﬁrst L  1 layers and w is the parameter of the output layer (we assumethe output layer has no bias).
For a single task, the neural network f ( · ) can model the relationshipbetween input data and output label.
But it might suffer in the domain adaptation scenarios due tothe distribution shift.
Thus, given a domain distribution P 2 P and an input example x ⇠ P , wepropose to explicitly learn the distribution-related output as g w g ( P | x ) =
x ( P ) T w g where  x ( P )is the input-oriented distribution feature representation, and w g is the parameter.
Given ﬁnite basisexamples ˜ x 1 , · · · , ˜ x n ⇠ P ,  x ( P ) can be formally deﬁned as follows.
[Equation]
where  x, ˜ x i 2 R is the coefﬁcient of  x ( P ) in the space spanned by h · , ˜ x i i K X and indicates the 192 similarity of input example x and basis example ˜ x i .
Here h · , ˜ x i
i K X denotes the feature map of˜ x i in the NNGP kernel space induced by inﬁnitely-wide f ( · ) .
Therefore,  x ( P ) represents thedistribution representation of P when an example x is observed in the domain associated withsampling distribution P .
One special case is when  x, ˜ x i = 1 /n , it degenerates into the mean mapping of P in the kernelspace, which is commonly used for measuring the distribution discrepancy/similarity [31].
Comparedwith this plain distribution representation 1 n
[Equation]
,  x ( P ) pays more attention to the region 199 around x with large  x, ˜ x i for ˜ x
i 2 X .
For example, for data points x 1 and x 4 in Figure 1, they wouldshare the same plain distribution representation, as they are sampled from the same target domain.
But it is feasible to differentiate them using the parametric distribution representation  x ( P ) .
Generally, by taking both example x and its associated probability P as random variables, Bayes’theorem tells us that P ( x, P ) = P ( x ) · P ( P | x ) where P ( · ) denotes the probability of an observedevent.
The event involving x can be described by observing its representation f ( x ) ; given x , the eventinvolving P is the input-oriented distribution representation  x ( P ) .
This motivates us to develop adistribution-informed neural network as follows.
[Equation]
The intuition behind Eq.
(5) can be explained as follows.
For domain adaptation, the featurerepresentation of an input example x is domain-dependent, as two similar examples ( x src ⇡ x tgt )from different domains might have distinctive outputs ( y src 6 = y tgt ).
The input-oriented distributionrepresentation  x ( P ) allows us to identify the source and target examples with similar inputs butdistinctive outputs.
By analyzing the distribution-informed neural network at initialization andunder gradient descent training, we propose two domain adaptation algorithms: DINO - INIT (see
Subsection 4.2) and DINO - TRAIN (see Subsection 4.3), respectively.
4.2 Initialization
We start by studying the distribution-informed neural network at initialization, e.g., all the parametersof distribution-informed neural network ˜ f ( · ) are initialized as standard normal variables.
Inspiredby the connections between standard neural networks and Gaussian processes [22, 42, 29, 65], weshow that the distribution-informed neural network ˜ f ( x ) in Eq (5) is equivalent to a domain adaptive Gaussian process, which explains the existing domain adaption regression algorithm
Lemma 4.1.
Assume that all the parameters of distribution-informed neural network ˜ f ( · ) areinitialized as standard normal variables, when the network width goes to inﬁnity, the output functionof ˜ f
( x ) in Eq (5) at initialization is iid centered Gaussian process, i.e., ˜ f ( · ) ⇠ N  0 , K DA  with
[Equation]
where K X ( · , · ) is the NNGP kernel induced by the neural network f ( · ) over input space X , and
K P ( · , · ) is a distribution kernel identifying the similarity of two input-oriented distributions P and
P 0 over distribution space P :
[Equation]
where n ( n 0 ) is the number of basis examples sampled from the distribution P ( P 0 ).
This lemma motivates us to derive Gaussian process regression named DINO - INIT (see Algorithm 1 in Appendix A.10) with adaptive transfer kernel deﬁned as K DA , where K DA (( x, P ) , ( x 0 , P 0 )) measuresthe input-oriented distribution similarity of P and P 0 .
For notation simplicity, we represent thedistribution-informed kernel K DA (( x, P ) , ( x 0 , P 0 ))
as K DA ( x, x 0 ) in the following.
In this case, weconsider a noisy model y ri = ˜
f
( x ri , P r ) + ✏ ri ( r 2 { src , tgt } ), where the noise ✏ ri follows a zero-mean
Gaussian N (0 ,  2 r ) .
Given labeled training examples ( X, Y ) , we assume a prior Gaussian processp ( Y ) =
N (0 , K DA ) where K DA ( X, X ) is a block matrix.
That is, K DA ( X, X ) =
[Equation]
where K DA 11 := K DA ( X src , X src ) ( K DA 22 :=
K DA ( X tgt l , X tgt l ) ) denotes the kernel matrix of source(target) data, and K DA 12 =
( K DA 21 ) T :=
K DA ( X src , X tgt l ) denotes the kernel matrix across domains.
Then for testing target examples X tgt ⇤ , their output can be inferred using the predictive distributionp ( Y | X tgt ⇤ , X src , X tgt l ) =
N (¯ µ, ¯ ⌃ ) .
Similar to standard Gaussian process regression [63], the meanand variance of the predictive distribution can be exactly calculated as follows.
[Equation]
where C = K DA ( X, X ) +   [Equation] src [Equation]
I n l tgt  .
Here I denotes the identity matrix.
This adaptive Gaussian process involves the following parameters: coefﬁcient  x,x i in the kernelfunction K DA and noise variance  src ,  tgt .
Following [11], we optimize these parameters by maxi- mizing the conditional likelihood p ( Y tgt l | X tgt l , X src , Y src ) .
It can be seen that p ( Y tgt l | X tgt l , X src , Y src )is also a Gaussian process, i.e., p ( Y tgt l | X tgt l , X src , Y src ) = N ⇣ K DA
K DA 11 +  2 src I n src   1 Y src ,⇣
[Equation]
⌘ .
Instantition of the coefﬁcient  x, ˜ x
i : It is notable that it can only optimize the coefﬁcient  x, ˜ x i of
Eq.
(4) for training (source or target) examples.
The coefﬁcient of testing target examples X tgt ⇤ isunknown.
To solve this problem, in this paper, we formulate the estimate of  x r , ˜ x ri ( r 2 { src , tgt } )as  x, ˜ x ri =
[ x r  ˜ x ri ] T w r , where w r is a domain-speciﬁc parameter vector and  denotes vectorconcatenation.
Note that domain adaptation assumes that the domain labels of all input examplesare known.
As shown in Eq.
(4), the input-oriented distribution representation can be learnedfrom the domain-speciﬁc vector w r and the basis examples ˜ x
r 1 , · · · , ˜ x rn r from domain r .
Here r isdetermined by the domain label of the input example x .
As illustrated in Algorithm 1, we use all thetraining source (target) examples as the basis source (target) examples ˜ x r 1 , · · · , ˜ x rn r .
As a result, theinput-oriented distribution representation learning of testing target examples
X tgt ⇤ can be inferred byw tgt and the pre-deﬁned basis target examples.
4.3 Gradient Descent Training
Here we assume that the model parameters of the distribution-informed neural network can be updatedby gradient descent during training.
Based on the distribution-informed neural network, we propose a 259 novel DINO - TRAIN algorithm. Its objective function using mean square error is formulated as follows.
[Equation]
where ↵ 2 (0 , 1) and µ  0 are hyperparameters to balance different terms.
The ﬁrst two terms 261 are standard supervised learning losses over labeled source and target examples, and the third one isthe empirical maximum mean discrepancy (MMD) [31] in the RKHS H DA induced by the neural 263 tangent kernel of our distribution-informed neural network, i.e.,
[Equation]
where n tgt = n l tgt + n u tgt is the total number of target training examples.
As shown in Theorem 4.5below, our framework empirically minimizes the upper error bound of the expected prediction errorin the target domain (see Algorithm 2 in Appendix A.10).
Remark.
In our framework of Eq.
(7), we measure the distribution shift using the training dynamicsof the distribution-informed neural network over the source and target examples.
This is fundamentally different from previous works [55, 72, 1] associated with a two-stage domain discrepancymeasurement.
That is, those works would ﬁrst learn the feature representation in hidden neurallayers and then measure the domain discrepancy in the learned feature space.
In the second stage,they usually require additional modules, e.g., auxiliary neural networks [14, 28, 26] or pre-deﬁneddistribution distances [45, 40].
Therefore, compared to previous works, our discrepancy measure Eq.(8) can not only unify the domain adaptation regression framework with neural networks, but alsoenable the theoretical convergence and generalization analysis in the following.
Before analyzing the dynamics of the distribution-informed neural network under Eq.
(7), we ﬁrstintroduce the following assumption.
Assumption 4.2.
Given labeled data X =
X src [ X tgt l , we assume  min ( ⇥ ( X, X )) >
and min
[Equation]
> 0 , where ⇥ ( X, X ) =
r ✓ f ( X ) r ✓ f ( X ) T is standard neural tangent kernel [38] induced by f ( · ) with inﬁnite width, and K P |X ( X, X ) is the input-oriented distributionkernel 1 (see Eq.
Here,  min ( · ) denotes the smallest eigenvalue.
The assumption  min ( ⇥ ( X, X )) > 0 has been studied in previous works [38, 25, 4].
It holds as longas no two inputs x i and x j are parallel in real scenarios.
The implication of this assumption is thatthe input data (e.g., images) would not be linearly changed by a single factor, i.e., there is no twoinputs such that x i = & · x j for some & 2 R .
Lemma 4.3.
Given the labeled training data X =
X
src
[ X tgt l and the basis examples { ˜ x ri } ˜ n r i =
( r 2 { src , tgt } ), we let ⌥ 2 R ( n src + n l tgt ) ⇥ (˜ n src +˜ n tgt ) denote the coefﬁcient matrix of K P |X ( X, X ) ,where for x k 2 X ( k = 1 , · · · , n src + n l tgt ), its k th row [ ⌥ ] k, : =
[  x k , ˜ x src 1 , · · · ,  x k , ˜ x src ˜ n src , 0 , · · · , 0]when x k 2 X src , [ ⌥ ] k, : = [0 , · · · , 0 ,  x k , ˜ x tgt 1 , · · · ,  x k , ˜ x tgt ˜ n tgt ] otherwise.
Then, the assumption min ( K P |X ( X, X )) > 0 holds when ⌥ T ⌥ is positive deﬁnite.
Lemma 4.3 shows that in real scenarios, the assumption  min ( K P |X ( X, X )) > 0 can be guaranteedby the positive deﬁniteness of the coefﬁcient matrix ⌥ .
We also empirically evaluate the smallesteigenvalue  min
K P |X ( X, X )  in Subsection 5.2.
Theorem 4.4.
For any coefﬁcient  x, ˜ x i of the input-oriented distribution representation in Eq.
(4),there exists ⌘ ⇤ 2 R + such that for the inﬁnitely-wide distribution-informed neural network ˜ f
( · )trained under gradient ﬂow with learning rate ⌘ < ⌘ ⇤ , the test prediction ˜
f ✓ t ( X tgt ⇤ ) of the domainadaptation regression in Eq.
(7) over test target data X tgt ⇤ is
[Equation]
⌘ where ⇥ DA ( · , · ) is the distribution-informed NTK, i.e., ⇥ DA ( x, x 0 )
= ⇥
( x, x 0 ) · K P |X ( P , P 0 | x, x 0 )and
˜ C
= diag { ↵ /n src , · · · , ↵ /n src
| {z } n src , (1  ↵ ) /n
l tgt , · · · , (1  ↵ ) /n
l tgt | {z } n l tgt } is a diagonal matrix.
Moreover,under the assumption 4.2, when the network width goes to inﬁnity, lim t !1 ˜ f ✓ t ( X tgt ⇤ ) converges to a
Gaussian process with mean µ ( X tgt ⇤ ) and variance ⌃ ( X tgt ⇤ , X tgt ⇤ ) as follows.
[Equation]
where “ + h.c. " means “plus the Hermitian conjugate".
In addition, our result on generalization bound is given in the following theorem.
Theorem 4.5.
Assume for any training example ( x, y ) 2 X ⇥ Y , we have ( ˜ f
( x, P )  y ) 2  M 0 forsome constant M 0  0 .
Let ˜ F be the hypothesis space induced by inﬁnitely-wide neural networks˜ f .
Then, for any ˜ f 2 ˜ F and  > 0 , with probability at least 1   , the expected error in the target 307 domain can be bounded as follows.
[Equation]
[Equation]
+ ⌦ where ⌦ = 2 ↵ < n src ( H src ) + 2(1  ↵ ) <
n l tgt ( H tgt ) + M q ( n src + n l tgt )log(1 / [Equation] , H r = { ( x, y ) !
( ˜ f ( x r , P r )  y r ) 2 : ˜ f 2 F} is a set of functions ( r 2 { src , tgt } ), < n r ( H r ) is the Rademachercomplexity of H r given n r examples, and M = max { ↵ M 0 /n src , (1  ↵ ) M 0 /n
l tgt } .
Though domain adaptation theories [9, 8, 48] have been generalized to deep learning scenarios inrecent years, neural networks are simply considered as an expressive feature extractor in moderngeneralization errors [72, 64].
Nevertheless, recent works [73, 44] reveal that the feature spacelearned by the neural networks might worsen the adaptation between source and target domains.
As acomparison, Theorem 4.5 provides a uniﬁed generalization error for neural network based domainadaptation, as it directly measures the domain discrepancy over input source and target examples inthe RKHS induced by the distribution-informed NTK ⇥ DA .
4.4 Discussion
4.4.1 Gaussian Process at Initialization
The following corollary shows that the existing domain adaptive Gaussian process [11] can beexplained as implicitly learning the distribution-informed representation in the kernel space.
Corollary 4.6.
With the assumptions in Lemma 4.1, when  x, ˜ x i = 1 / ||
P ni =1 h · , ˜ x i
i K X
|| ,
the Gaussian process induced by ˜ f
( x ) at the initialization would be equivalent to the adaptive Gaussianprocess in [11] over NNKP kernel. [Table] [Table]
4.4.2 Fully Trained Model
326 When there exist some labled target examples, the objective function of the reweighting domainadaptation techniques (see Subsection 3.2.2) can be rewritten as follows.
[Equation]
The following corollary holds that our framework Eq.
(7) can recover the reweighting approach Eq.(9) in the function space.
Corollary 4.7.
With the assumption in Theorem 4.4, our framework Eq.
(7) with distribution-informedneural network ˜ f
( · ) can recover the popular reweighting domain adaptation approach Eq.
(9) in thefunction space.
Besides, we show that when the distribution representation is shared by all the examples, ourframework can be naturally degenerated into the domain-invariant representation learning [45].
Corollary 4.8.
With the assumption in Theorem 4.4, our framework Eq.
(7) with distribution-informedneural network ˜ f
( · ) would be degenerated into the standard domain invariant representation learningin Eq.
(1), where the domain discrepancy measure d ( · , · ) is instantiated with MMD in RKHS inducedby standard NTK ⇥ .
Experiments
Data Sets: We use three domain adaptation regression benchmarks.
Following [14], we use twoimage data sets: dSprites [49] and MPI3D [30].
Speciﬁcally, dSprites has 737,280 images from threedomains: Color (C), Noisy (N) and Scream (S).
For each image, it has three regression tasks, i.e.,predicting three factors of variations (scale, position X and Y).
MPI3D contains over 3M imagesfrom three domains: Toy (T), Realistic (RC) and Real (RL).
It is shown [14] that for each image,it involves two regression tasks, i.e., predicting three factors of variations (position X and Y).
Inaddition, we also use a plant phenotyping data set.
It predicts diverse traits (e.g., Nitrogen) of plantsrelated to the plants’ growth using leaf hyperspectral reﬂectance.
Here we consider the followingtwo domains [59]: Maize (M) and Maize_UNL (MU).
In our case, the task is to predict the Nitrogencontent of maize using the leaf hyperspectral reﬂectance (see Appendix A.11 for more details).
Baselines: In the experiments, we consider the following baseline methods.
(1) Plain Gaussianprocess: NNGP [42] and NTKGP [32].
(2) Domain adaptive Gaussian processes: AT-GP [11] and
TL-NTK [46].
(3) Deep domain adaptation methods: DAN [45], WANN [23] and RSD [14].
Implementations:
In the experiments, our algorithms are implemented using a L -layer ( L = 6 )fully-connected neural network with ReLU (see Appendix A.11 for more details).
The induced
NNGP and neural tangent kernels induced can be estimated using the Neural Tangents package [52].
In addition, we set ↵ = 0 .
5 and µ = 0 .
1 for DINO - TRAIN .
5.1 Results
[Figure] [Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Table 1, Table 2 and Table 3 provide the domainadaptation regression results on dSprites, MPI3D and
Plant Phenotyping data sets.
Following [14], we report the Mean Absolute Error (MAE) between thepredicted outputs and the ground-truth outputs in thetarget test set (the best results are indicated in bold).
It is observed that our method DINO achieves competitive performance over the Gaussian process baselines.
The weakly-trained DINO - INIT slightly outperforms DINO - TRAIN in some cases.
This observation is consistent with previous work [41].
The relativelyunstable performance of DINO - TRAIN might be caused by the large covariance of Gaussian processwith NTK [32].
In addition, we also compare DINO with domain adaptation methods instantiatedby overparameterized neural networks.
WANN [23] reweights the source examples, and DAN [45]and RSD [14] learn the domain invariant representation.
The performance comparison in Figure 2conﬁrms the effectiveness of our DINO approach over those state-of-the-art baselines.
5.2 Analysis
Methods
C !
N
[Table]
Ablation Study: We investigate the impact of input-oriented distribution representation in our DINO framework.
We considerthe following variants.
DINO - INIT ( DINOTRAIN )
w/o distribution feature: it wouldnot utilize the input-oriented distributionrepresentation, i.e., g w g ( P | x ) = 1 for all 381 x 2 X .
DINO - INIT ( DINO - TRAIN ) w uni- 382 form distribution feature: the coefﬁcient  x, ˜ x i = 1 /n is a constant.
The results are given in Table 4.
We observe that the distribution representation improves the domain adaptation performance.
Compared to globally shared distribution representation with  x, ˜ x i = 1 /n , our parametric  x ( P ) in Eq.
(4) encourages to learn the local distribution representation around the input data point x .
Theempirical results in Table 4 conﬁrmed the efﬁcacy of the input-oriented distribution representation.
Impact of Model Architecture and Discrepancy Measure: Figure 3a shows the results of DINO withdifferent number of neural layers L on dSprites (C ! N).
The results indicate that the performanceof DINO is positively related to the expressiveness of NTK and NNGP kernel induced by the neuralnetwork.
In addition, we compare the proposed ˆ MMD ⇥ DA ( · , · ) with MMD-RBF (i.e., MMD withstandard RBF kernel) in domain adaptation regression, where MMD-RBF is estimated in the featurespace learned by different neural layers (see Appendix A.11 for more details).
As shown in Figure 3b,the results on Plant Phenotyping data set reveal that the selection of feature space affects the modelperformance when using MMD-RBF, and our proposed ˆ MMD ⇥ DA ( · , · ) outperforms the MMD-RBF.
Positive Deﬁniteness of K P |X ( X, X ) : We report the smallest eigenvalue of the distribution kernel
K P |X ( X, X ) over the training examples on Plant Phenotyping data set.
The smallest eigenvaluesof K P |X ( X, X ) in DINO - TRAIN after training are 3 e  6 and 8 e  7 on M !
MU and MU !
M,respectively.
This empirically conﬁrms the positive deﬁniteness of K P |X ( X, X ) in DINO - TRAIN .
Conclusion
In this paper, we study the domain adaptation regression by proposing a distribution-informed neuralnetwork.
Our domain adaptation framework with distribution-informed neural network subsumes theexisting adaptation approaches based on domain invariant representation, reweighting, and adaptive
Gaussian process.
We also analyze the convergence and generalization bound of our framework.
Theexperiments demonstrate the effectiveness of our DINO framework over state-of-the-art baselines.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] See Abstract and Section 1.(b)
Did you describe the limitations of your work?
[Yes] See Appendix A.10.(c)
Did you discuss any potential negative societal impacts of your work?
[No] Our workdoes not involve any potential negative societal impacts.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.2A.9.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See thesupplemental material.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 5 and Appendix A.11.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.11.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[N/A]
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]