I NTERPRETATIONS OF D OMAIN A DAPTATIONS VIA L
AYER V ARIATIONAL
A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Transfer learning is known to perform efficiently in many applications empirically, yet limited literature reports the mechanism behind the scene.
This study establishes both formal derivations and heuristic analysis to formulate the theory of transfer learning in deep learning.
Our framework utilizing layer variational analysis proves that the success of transfer learning can be guaranteed with corresponding data conditions.
Moreover, our theoretical calculation yields intuitive interpretations towards the knowledge transfer process.
Subsequently, an alternative method for network-based transfer learning is derived.
The method shows an increase in efficiency and accuracy for domain adaptation.
It is particularly advantageous when new domain data is sufficiently sparse during adaptation.
Numerical experiments over diverse tasks validated our theory and verified that our analytic expression achieved better performance in domain adaptation than the gradient descent method
I NTRODUCTION
Transfer learning is a technique applied to neural networks admitting rapid learning from one (source) domain to another domain, and it mimics human brains in terms of cognitive understanding.
The concept of transfer learning has been considerably advantageous, and different frameworks have been formulated for various applications in different fields.
For instance, it has been widely applied in image classification (Quattoni et al., 2008; Zhu et al., 2011; Hussain et al., 2018), object detection (Shin et al., 2016), and natural language processing (NLP) (Houlsby et al., 2019; Raffel et al., 2019).
In addition to applications in computer vision and NLP, transferability is fundamentally and directly related to domain adaptation and adversarial learning (Luo et al., 2017;
Cao et al., 2018; Ganin et al., 2016).
Another majority field adopting transfer learning is domain adaptation, which investigates transition problems between two close domains (Kouw & Loog, 2018).
Typical understanding is that transfer learning deals with a general problem where two domains can be rather distinct, allowing sample space and label space to differ.
While domain adaptation is considered a subfield in transfer learning where the sample/label spaces are fixed with only the probability distributions allowed to be varied.
Several studies have investigated the transferability of network features or representations through experimentation, and discussed their relation to network structures (Yosinski et al., 2014), features and parameter spaces (Neyshabur et al., 2020; Gonthier et al., 2020).
In general, all methods that improve the predictive performance of a target domain, using knowledge of a source domain, is considered under the transfer learning category (Weiss et al., 2016; Tan et al., 2018).
This work particularly focuses on the network-based transfer-learning, which refers to a specific framework that reuses a pretrained network.
This approach is often referred to as finetuning, which has been shown powerful and widely applied with deep-learning models (Ge & Yu, 2017; Guo et al., 2019).
Even with abundant successes in applications, the understanding for network-based transfer learning mechanism from a theoretical framework remains limited.
This paper presents a theoretical framework set out from aspects of functional variation analysis (Gelfand et al., 2000) to rigorously discuss the mechanism of transfer learning.
Under the framework, error estimates can be computed to support the foundation of transfer-learning, and an interpretation is provided to connect the theoretical derivations with transfer learning mechanism.
Our contributions can be summarized as follows: we formalize transfer learning in a rigorous setting and variational analysis to build up a theoretical foundation for the empirical technique.
A theorem is proved through layer variational analysis that under certain data similarity condition, a transferred net is guaranteed to transfer knowledge successfully.
Moreover, a comprehensible interpretation is presented to understand the finetuning mechanism.
Subsequently, the interpretation reveals that the reduction of non-trivial transfer learning loss can be represented by a linear regression form, which naturally leads to analytical (globally) optimal solutions.
Experiments in domain adaptation were conducted and showed promising results, which validated our theoretical framework.
R ELATED W ORK
Transfer Learning Transfer learning based on deep learning has achieved great success, and finetuning a pretrained model has been considered an influential approach for knowledge transfer (Guo et al., 2019; Girshick et al., 2014; Long et al., 2015).
Due to its importance, many studies attempt to understand the transferability from a wide range of perspectives, such as its dependency on the base model (Kornblith et al., 2019), and the relation with features and parameters (Pan & Yang, 2009).
Empirically, similarity could be a factor for successful knowledge transfer.
The similarity between the pretrained and finetuned models was discussed in (Xuhong et al., 2018).
Our work begins from setting up a mathematical framework with data similarity defined, which leads to a new formulation with intuitive interpretations for transferred models.
Domain Adaptation Typically domain adaptations consider data distributions and deviations to search for mappings aligning domains.
Early literature suggested that assuming data drawn from certain probabilities (Blitzer et al., 2006) can be used to model and compensate the domain mismatch.
Some studies then looked for theoretical arguments when a successful adaptation can be yielded.
Particularly, (Redko et al., 2020) estimated learning bounds on various statistical conditions and yielded theoretical guarantees under classical learning settings.
Inspired by deep learning, feature extractions (Wang & Deng, 2018) and efficient finetuning of networks (Patricia & Caputo, 2014;
Donahue et al., 2014; Li et al., 2019) become popular techniques achieving domain adaptation tasks.
The finetuning of networks was investigated further to see what was being transferred and learned in (Wei et al., 2018).
This will be close to our investigation on weights optimizations.
M ATHEMATICAL F RAMEWORK AND E RROR E STIMATES 3. F RAMEWORK FOR N ETWORK - BASED T RANSFER L EARNING
To formally address error estimates, we first formulate the framework and notations for consistency.
Definition 3.1 (Neural networks) .
An n -layer neural network f : R ℓ
→ R ℓ n is a function of the form:
[Equation]
for each j = 1 , . . .
, n , f j = σ j ◦
A j : R ℓ j − 1 → R ℓ j is a layer composed by an affine function
A j ( z ) :=
W j z + b j and an activation function σ j : R ℓ j → R ℓ j , with W j ∈ L (cid:0) R ℓ j − 1 , R ℓ j (cid:1) , b j ∈ R ℓ j and L ( K, V ) the collection of all linear maps between two linear spaces K → V .
The concept of transfer learning is formulated as follows:
Definition 3.2 ( K -layer fixed transfer learning) .
Given one (large and diverse) dataset D = { ( x i , y i )
∈
X × Y} and a corresponding n -layer network f = f n ◦ f n − 1 ◦ ... ◦
f 1 : X → Y trained by D under loss L ( f ) , the k -layer fixed transfer learning finds a new network g : X → Y of the form,
[Equation]
under loss L ( g ) when a new and similar dataset (cid:101) D
= { ( (cid:101) x i , (cid:101) y i ) ∈
X × Y} is given.
The first k layers of f remain fixed and g j :=
σ j ◦ (cid:101)
A j are new layers with affine functions (cid:101)
A j to be adjusted ( k < j < n ).
The net f trained by the original data D is called the pretrained net and g trained by new data (cid:101)
D is called the transferred net or the finetuned net .
Transfer learning has the empirical implication of “transferring” knowledge from one domain to another by fixing the first few pretrained layers.
One main goal of this study is to characterize the transferring process under layer finetuning.
We utilize the loss function to define the concept of a “ well-trained ” net, in order to comprehend transfer learning mechanism.
The losses for the pretrained net f and the transferred net g are computed on a general label space Y with norm ∥ · ∥ Y :
[Equation]
Definition 3.3 (Well-trained net) .
Given a dataset D = { ( x i , y i )
∈
X × Y} , a network h is called well-trained under data D within error ε trained > 0 if
[Equation]
The definition then implies that a well-pretrained net f within error ε pretrained is to satisfy
[Equation]
In certain domain adaptation applications, discussions are in terms of probability distributions (Redko et al., 2020).
Our framework is compatible with the usual probabilistic setting under the independent and identically distributed (i.i.d.) consideration, frequently applied in practice.
In this case, Eq.
(3) is equivalent to
[Equation]
where ℓ is a norm (error) function.
One intuition for successful knowledge transfer is that the previous knowledge base and the new domain share some common (or similar) features.
The resemblance of two datasets D , (cid:101) D under transfer learning in this work is defined by:
Definition 3.4 (Data deviation) .
The data deviation ε data ≥ 0 of two datasets D and (cid:101) D is defined as,
[Equation]
Note that the first minimization in Eq.
(7) is recognized as performing sample alignment in the context of domain adaptation.
In fact, for each i ≤ | (cid:101) D| there exists an index
[Equation]
such j i then corresponds to the closet sample in D to i .
This step is usually referred to as sample alignment .
Domain alignment is then to perform sample alignment over all samples of a given domain.
Consequently, this definition aligns two datasets first and then considers the largest distance among all aligned pairs.
Remark 3.5.
To simplify notations for the following discussions, two assumptions can be made without loss of generality.
First, a rearrangement on sample index can be done by Eq.
(8) and simply rename j i as i for convenience.
Second, with proper rescaling on data, uniform boundedness on D and (cid:101) D can be yielded.
As such, one may assume ∥ x j ∥ X , ∥ y j ∥ Y , ∥
˜ x i ∥ X , ∥ (cid:101) y i ∥
Y ≤ 1 for all i ≤ | (cid:101) D| , j ≤ |D| .
These two assumptions are applied in the rest of discussions unless specified otherwise.
3. T RANSFER L OSS B OUND VIA L AYER V ARIATIONS
Neural networks of the form Eq.
(1) with Lipschitz activation functions are Lipschitz continuous.
We utilize Lipschitz constants to control the network propagation and estimate the error of last r -layer finetuned nets.
The Lipschitz constant of a network h is denoted by C h in this paper.
Let a pretrained net f be composed by two parts:
[Equation]
where F n − r = f n − r ◦ · · · ◦ f 1 are layers to be fixed and F = f n ◦ ... ◦
f n − r
+1 are the last r layers to be finetuned ( 1 < r < n ).
With only two essential definitions above, our main theorem addressing the validity of the transfer learning technique can be derived:
Theorem 3.6 ( Finetuned loss bounded by pretrain loss and data) .
Let D , (cid:101) D be two datasets with deviation ε data and f = F ◦ F n − r be a well pretrained network under D , then a net finetuning the last r layers of f with the form g :=
( F + δF ) ◦ F n − r has bounded loss on (cid:101) D ,
[Equation]
where δF is a Lipschitz function with C δF ≤ ε data ; C is a constant depending on C F n − r and C F .
Sketch Proof.
By g = ( F + δF ) ◦ F n − r and notations in Eq.
(8), we have
[Equation]
with v 1 := δF ◦ F n − r ( (cid:101) x i ) + f ( (cid:101) x i ) − f ( x j i ) .
We compute,
[Equation]
with Eq.
(5) and the triangle inequality applied in the last inequality.
Further, we estimate, ∥
[Equation]
With Eq.
(12) and condition C δF ≤ ε data , the result Eq.
(10) can be yielded.
For the complete proof, see Appendix 7.1.
Theorem 3.6 depicts that the finetuned loss is bounded by the pretrained loss and the data difference in
Eq.
(10), linking the relation of network functions and data into one.
This then explains our intuition why a well-pretrained net beforehand is necessary, and transfer learning is only expected to be viable on datasets with certain similarity.
Thus, with minimal conditions and definitions Theorem 3.
succinctly addresses when transfer learning guarantees a well-finetuned model under a new dataset.
Remark 3.7.
For two very different datasets, one has ε data ≫ 0 .
Eq.
(10) then indicates such transfer learning process is generally more difficult as the upper bound becomes large, even with a small pretraining error ε pretrained .
Typically, in the task of domain adaptations, ε data is not expected to be large.
Remark 3.8.
Eq.
(10) is regarded as an extension of Eq.
(5) whenever (cid:101) D ̸ = D .
The two equations coincide if ε data = 0 where two domains D , (cid:101) D perfectly align.
I NTERPRETATIONS & C HARACTERIZATIONS OF N ETWORK -B ASED T RANSFER L EARNING
Inspired by the mathematical framework of Theorem 3.6, an intuitive interpretation of network-based transfer learning can be derived, which in turn leads to a novel formulation for neural network finetuning.
The derivation begins with the Layer Variational Analysis (LVA), where the effect of data variation and transmission can be addressed.
4. L AYER V ARIATIONAL A NALYSIS (LVA)
To clarify the effect of one layer variation ( r = 1 in Theorem 3.6), we denote the latent variables
z i , (cid:101)
z i and the last finetuned layer g n from Eq.
(2) by,
[Equation]
The target prediction error can be related to the pretrain error via following expansion,
[Equation]
) with six self-cancelled-out terms added for auxiliary purpose.
Note the identities g ( (cid:101) x i )
=
g n ( (cid:101) z i ) , f n ( z i ) =
f ( x i ) are used and the second grouping can be rewritten by,
[Equation]
where δz i :
= ( (cid:101)
z i − z i ) corresponds to the latent feature shift ; J ( f n )( z i ) and H ( f n )( z i ) are the
Jacobian and the Hessian matrix of f n at z i , respectively.
In principle, non-linear activation functions introduce high order δz i terms to Eq.
(16), where by considering regression type problems (assuming f n purely an affine function) the finetune loss Eq.
(15) is allowed to be simplified as,
[Equation]
) with δy i := (cid:101) y
i − y i and the vector q i ∈ Y defined as
[Equation]
) which is referred to as transferal residue .
Via Transmission of Layer Variations (Appendix 7.3) one may also write,
[Equation]
4. LVA I NTERPRETATIONS & I NTUITIONS
The equivalent form Eq.
(17) of the transfer loss
L ( g ) =
1 N (cid:80) N i =1 ∥
[Equation]
LVA renders an intuitive interpretation for network-based transfer learning.
The resulting picture is particularly helpful for domain adaptations.
Transferal residue: We justify the name transferal residue of Eq.
There are two pairs:
[Equation]
where the first pair (∆) evaluates domain mismatch ; the other measures the pretrain error , since: (a) perfect pretrain f gives f ( x i )
=
y i then ( ⋆ ) =
0 , (b) if two domains match perfectly D
= (cid:101) D , δx i
= δy i
=
then (∆) = 0 .
In such perfect case, q i ≡ 0 and there is nothing new for f to learn/adapt in (cid:101) D .
On the contrary, if f is ill-pretrained with ∥ δx i ∥ , ∥ δy i ∥ large, q i becomes large too, indicating there is much to adapt in (cid:101) D .
Therefore, the transferal residue characterizes the amount of new knowledge needed to be learned in the new domain, and hence the name.
In short,
[Equation]
Layer variations to cancel transferal residue: Whenever the transferal residue is nonzero, the finetune net g adjusts from f to react, as Eq.
(17) assigns layer variation δf n to neutralize q i .
Minimizing the adaptation loss L ( g ) with δf n , we see that
[Equation]
In fact, the last adaptation step ⊗ is analytically solvable if δf n is a linear functional with ∥ · ∥ Y being
L 2 -norm.
By Moore-Penrose pseudo-inverse, the minimum is achieved at
[Equation]
(21) with ˜ z :=
( (cid:101) z 1 , . . .
, (cid:101) z N ) and q :=
( q 1 , . . . , q N ) .
In short, layer variations tend to digest the knowledge discrepancy due to domain differences.
Knowledge transfer: As q i reflects the amount of new knowledge left to be digested in (cid:101) D , we see a term J ( f )( x i ) δx i automatically emerges trying to reduce the remainder in q i .
Especially, the
Jacobian term appears with a negative sign opposite to f ( x i ) ; it is presented as a self-correction (self-adaptation) to the new domain.
The purpose of J ( f )( x i ) δx i serves to annihilate the new label shift δy i Eq.
(20) (e.g. Fig. 1) and notice that the self-correction not needed if D = (cid:101) D , δx i = 0 .
Since the pretrained f contains previous domain information, the term J ( f )( x i ) δx i indicates the old knowledge is carried over by f to dissolve the new unknown data.
Such quantification then
The analysis can be extended to multi-layer cases as well as Convolutional Neural Networks (CNN); only that the analytic computation soon becomes cumbersome.
Therefore, only one-layer LVA finetuning is demonstrated in this work.
Further discussions and calculations for LVA of two layers and CNNs can be found in Appendix 7.2.
Although there is no explicit formula for multi-layer cases to obtain the optimal solution, iterative regression can be utilized recursively to obtain a suboptimal result analytically without invoking gradient descent.
The LVA formulation not only renders interesting perspectives in theory but also demonstrates successful knowledge adaptations in practice.
E XPERIMENTS
Three experiments in various fields were conducted to verify our theoretical formulation.
The first task was a simulated 1D time series regression, the second was Speech Enhancement (SE) of real-world voice dataset; the third was an image deblurring task by extending our formula onto Convolutional
Neural Networks (CNNs).
The three tasks demonstrated the LVA had prompt adaptation ability under new domains and confirmed the calculations Eq.
(15) ∼ (18).
The code is available on Github 1 .
For detailed implementations and additional results on adaptive image classifications, see Appendix 8.
5. T IME SERIES (1D SIGNAL ) REGRESSION
Goal: Train a DNN predicting temporal signal D and adapt it to another signal with noise (cid:101) D .
Dataset: Two signals are designed as follows, with (cid:101) D mimicking a noisy signal (a low-quality device)
[Equation]
22) with { t i | i = 1 , . . . , 2000 } ∈ [ − 1 , 1] equally distributed, γ ( t i ) = 0 .
4 t i + 1 . 3998 , ξ i ∼ N (1 . 5 , 0 .
8) , η i ∼ U ( − 1 , 1) .
N ( µ, σ ) is a normal distribution of mean µ and variance σ ; U as a uniform distribution [ − 1 , 1] .
Implementation:
The pretrained model f consists of 3 fully-connected layers of 64 nodes with
ReLu as activation functions.
We refer to Appendix for more implementation details.
Result analysis: The adaptation results were compared between the conventional gradient descent (GD) method and LVA by Eq.
The transferred net g from f was finetuned with GD for 12000 epochs, while LVA only requires one step to derive.
After adaptation, LVA reached lowest
L 2 -loss compared to GD of same finetuned layer numbers, see Fig. 2(f).
This verified the stability and validity of the proposed framework.
In addition to obtaining lowest loss, the actual signal prediction performance can also be observed improved via visualization Fig.
2(b) ∼ (e) in the case of finetuning last one and two layers.
Notably, the finetuning loss of two layers indeed further decreased than that of one layer, which met the expectation as the two-layer case had more free parameters to reduce loss.
Comparing Fig. 2(b)(c), it showed that the 1-layer case LVA obtained more accurate signal predictions compared to GD’s predictions.
Fig.
2(d)(e) showed the results of 2-layer finetune by
GD and LVA.
The 2-layer LVA formulation is derived in Appendix 7.2.
Observing that the LVA formulation equips prompt adaptation ability, we further conduct two real-world applications.
5. S PEECH ENHANCEMENT
To prove that the proposed method is effective on real data, we conducted experiments on an SE task.
Goal: Train a SE model to enhance (denoise) speech signals in a noisy environment D and adapt to another noisy environment (cid:101) D .
Dataset: Speech data pairs were prepared for the source domain D (served as the training set) and target domain (cid:101) D (served as the adaptation set) as follows:
[Equation]
where y i denotes a patch 2 of a clean speech (as label), which is then corrupted with noise n k of type k to form a noisy speech (patch) x ijk over an amplification of c j ∝ exp ( − SNR j ) , determined by a signal-to-noise ratio (SNR j ) given.
N 1 ( resp.
(cid:101) N 1 ) denotes the number of patches extracted from clean utterances in D ( resp.
(cid:101) D ); N 2 ( resp.
(cid:101) N 2 ) is the number of noise types contained in D ( resp.
(cid:101) D ).
An SE model f on D performed denoising such that f ( x ijk ) =
y i .
In this experiment, 8, utterances (corresponding to N 1 = 112,000 patches) were randomly excerpted from the Deep Noise
Suppression Challenge (Reddy et al., 2020) dataset; these clean utterances were further contaminated with domain noise types: n k ∈ { White-noise, Train, Sea, Aircraft-cabin, Airplane-takeoff } , (thus
N 2 = 5 ), at four SNR levels: {− 5 , 0 , 5 , 10 } (dB) to form the training set.
These 112,000 noisy-clean pairs were used to train the pretrained SE model.
We tested performance using different numbers of adaptation data with (cid:101) N 1 varying from 20 to 400 patches (Fig. 3).
These adaptation data were contaminated by three target noise types: (cid:101) n k ∈ { Baby-cry, Bell, Siren } (thus (cid:101) N 2 = 3 ) at {− 1 , 1 } (dB) SNR.
Note the speech contents, noise types, and SNRs were all mismatched in the training and adaptation sets.
Implementation: A Bi-directional LSTM (BLSTM)
(Chen et al., 2015) model was used to construct the SE model f under L 2 -loss, consisting of one-layer BLSTM of 300 nodes and one fully-connected layers for output.
The transferred net g adapted from f by finetuning the last layer was performed by
GD to compare with LVA in Eq.
For data processing and network details see Appendix 8.2.
Domain alignment:
In Remark 3.5, it was mentioned that the domain alignment ought to be applied prior to LVA.
For real-world data, optimal transport (Villani, 2009) (OT) was selected for implementation to match domain distributions.
To contrast, LVA with and without OT were conducted.
Evaluation metric: SE performances are usually measured by 3 evaluation metrics, L 2 -error (MSE), the perceptual evaluation of speech quality (PESQ) (Rix et al., 2001) and short-time objective intelligibility (STOI) (Taal et al., 2011).
Each metric evaluates different aspect of speech: PESQ ∈ [ − 0 .
5 , 4 . 5] evaluates speech quality; STOI ∈ [0 , 1] estimates the speech intelligibility.
Result analysis: We excerpted another 100 utterances, contaminated with the target noise types: (cid:101) n k ∈ { Baby-cry, Bell, Siren } at {− 1 , 1 } (dB) SNR levels, to form the test set.
Note that the speech contents were mismatched in the adaptation and test sets.
Fig.
3 showed the domain adaptation results from environment D
→ (cid:101) D to compare GD with LVA.
On this test set, the PESQ and STOI of the original noisy speech (without SE) were 1.704 and 0.808, respectively.
The results in Fig.
3 showed consistent out-performance of LVA over GD under different number of adaptation data in (cid:101) D ( (cid:101) N 1 as the horizontal axis).
Especially, the L 2 -loss of LVA was notably less than that of GD, confirming that
LVA indeed derived the globally optimal weights of a transferred net.
Meanwhile, it was observed that LVA significantly outperformed GD when the amount of target samples in (cid:101) D was considerably less.
It was also noticed that the LVA equipped OT alignment (LVA-OT) achieves similar performance to LVA.
An example of the enhanced spectrograms by different SE models, including Pretrained,
GD, LVA, and LVA-OT, are shown in Fig.
The clean and unprocessed noisy (Baby-cry noise at − 1 dB SNR) spectrograms are shown for comparison.
From the figure, LVA recovers the speech components with clear structures than Pretrained and GD (as shown by the yellow-dashed rectangles).
More extensive result analysis are provided in Appendix 8.2.
5. S UPER RESOLUTION FOR IMAGE DEBLURRING
Goal: Train a Super-Resolution Convolutional Neural Network (SRCNN, Fig. 5)
(Dong et al., 2016) to deblur images of domain D and adapt to another more blurred domain (cid:101) D .
Dataset: 2000 high-resolution images were randomly selected as labels from the CUFED dataset (Wang et al., 2016) to train SRCNN.
The source domain ( x, y ) ∈ D and the target domain ( (cid:101) x, (cid:101) y ) ∈ (cid:101) D were set as follows,
[Equation]
Fig.
6 shows image pairs for adaptation.
Another image dataset SET14 (Zeyde et al., 2010) was used to test adapted models.
More image preparation details see Appendix 8.3.
Implementation: This experiment demonstrated that image related tasks can also be implemented by extending our LVA, Eq.
(17), (18), from fully-connected layers to CNNs.
Although the analytic formula Eq.
(21) is only valid on a fully-connected layer of regression type, a key observation admitting such extension is that CNN kernels can locally be regarded as a fully-connected layer, in that every receptive field is exactly covered by the kernels of a CNN.
By proper arrangement, a
CNN kernel C
= ( C ijγ ) can be folded as a fully-connected layer of 2D matrix weight W =
( W ℓm ) .
As such, the LVA can be extended onto CNNs as finetune layers.
For detailed implementations see
Appendix.
In this experiment, a SRCNN model f
(Fig. 5) was trained on source domain D to perform deblur f ( x ) =
y , and subsequently used GD and LVA to adapt f → g on (cid:101) D such that g ( (cid:101) x ) =
(cid:101) y .
Result analysis: Fig.
7 showed the results of adapted models by GD and LVA.
In Fig.
7, column (b) were more blurred inputs (cid:101) x of (cid:101) D ; column (c)(d)(e) were deblurred images to be compared with the ground truths in column (a).
Column (c)(d) compared the two methods adapted with same number of target domain samples ( N = 256 patches) and LVA reached the highest PSNR scores.
While more target domain samples were exclusively included for GD to N = 16384 in column (e), the corresponding PSNRs were still not compatible to the LVA adaptation.
There were more extensive experiments conducted to contrast the adaptation outcome of GD and LVA, see Appendix 8.3.
C ONCLUSIONS
This study aims to provide a theoretical framework for interpreting the mechanism under domain adaptation.
We derived a firm basis to ensure that a finetuned net can be adapted to a new dataset.
The objective of this study was achieved to address “why transfer learning might work?” and unravelled the underlying process.
Based on the LVA, a novel formulation for finetuned nets was introduced and yielded meaningful interpretations of transfer learning.
The formalism was further validated by domain adaptation applications in Speech Enhancement and Super Resolution.
Various tasks were observed to reach the lowest L 2 -loss under LVA, where the LVA is the theoretical limit for the gradient descent to approach.
The LVA interpretations and successful experiments give this study an inspiring perspective to view transfer learning.
A PPENDIX 7. P ROOF OF T HEOREM 3.
Consider finetuning on last r layers in Eq.
(2) with k + r = n .
Denote F n − r =
f n − r ◦ · · · ◦ f 1 and the last r layers F = f n ◦ ... ◦
f n − r +1 of pretrained network and finetuned net G = g n ◦ ... ◦
g n − r +1 .
We also write the difference G − F = δF .
With the notations, we rewrite
[Equation]
where v 1 = δF ◦ F n − r ( (cid:101) x i ) + f ( (cid:101) x i ) − f ( x j i ) and j i is the corresponding index such that Eq.
(7) is satisfied.
Therefore, one computes,
[Equation]
where the last term is further estimated by the triangle inequality to yield
[Equation]
Note that without any additional assumption, the first term in Eq.
(A.25) is of order 1.
To obtain a small error of g , one sufficient condition is to have C δF ≈ ε data .
It is interesting to point out that the analysis relates the Lipschitz constant C
7. P ERTURBATION APPROXIMATION DERIVATION FOR 2- LAYER CASE
Similarly, let δz i := (cid:101)
z i − z
i and denote the finetuned n th layer and ( n − 1) th layer by
[Equation]
The perturbation assumption allows the approximation of
[Equation]
where J ( f n − 1 )( z i ) is the Jacobian matrix of f n − 1 at z i .
The network assumed to deviate from the pretrained f n − 1 by an affine function,
[Equation]
for some weight W n − 1 and bias b .
With Eq.
(A.27) and Eq.
(A.28), we obtain
[Equation]
With Eq.
(A.29),
[Equation]
Note that the terms involving multiplications δf n and δf n − 1 are omitted as we only focus on the first order terms.
Parallel to Eq.
(15) for 1-layer case, we approximate ∥ g ( (cid:101) x i )
[Equation]
and therefore,
[Equation]
We see that the 2-layer fine-tuning case involves three unknown (parameters), including δf n , b , and
W n − 1 .
Although there is no explicit formula to obtain the optimal solution, iterative regression can be utilized multiple times to obtain a sub-optimal result analytically without using gradient descent.
In fact, one notices that if b is ignored, the optimization of δf n and W n − 1 based on the L 2 -loss can be solved.
Once δf n and W n − 1 are solved and fixed, b can be introduced back into the formula and can be solved again by regression.
The similar approximation for general multi-layers cases would still be reliable as long as the features in the latent space are close to each other.
That is to say, the features of the source and target domain share similarities.
These intuition-motivated formulations were shown to successfully adapt knowledge in experiments for both images and speech signals.
The results can be found in Section 5.
7. T RANSMISSION OF L AYER V ARIATIONS
We show heuristically the propagation of layer variations in neural nets can be traced via data deviations.
Given two datasets D , (cid:101) D with the indexing after sample alignment, we define
[Equation]
Let the data deviation be ε from Def. 3.4, and Eq.
(A.33) can be decomposed by orders of ε ,
[Equation]
) with O (cid:16) ∥ δx ( k ) i
∥ X (cid:17) =
O (cid:16) ∥ δy ( k ) i ∥
Y (cid:17) =
ε k .
To obtain a simple picture, first we neglect the high order terms δx (2) i , δy (2) i , etc.
In such case, δx (1) i and δy (1) i can be simply denoted as δx i and δy i , respectively, without confusion.
By Eq.
(14), finetuning the last layer in Theorem 3.6 reveals that
[Equation]
where J ( f )( x i ) is the Jacobian of f at x i and O ( ε 2 ) contains terms like H ( f )( x i ) δx i δx i , etc with
H ( f )( x i ) the Hessian tensor of f at x i .
Some effort converts Eq.
(A.35) into a familiar form,
[Equation]
with
[Equation]
Regard that Eq.
(A.36) recovers Eq.
(16) via different routes and starting point, although the two q i ’s in Eq.
(18) and (A.37) slightly differ due to distinct variational conditions.
This heuristic derivation then confirms that data deviations arouse network variations to induce the alternative finetune form.
Consequently, both derivations yield same results.
E XPERIMENTAL D ETAILS
All code and data can be found at https://anonymous.4open.science/r/Layer_
Variational_Analysis-183C/ .
The norms ∥ · ∥ X and ∥ · ∥ Y in all experiments are set as Euclidean L 2 -norms, unless otherwise specified.
Some notable details are remarked here.
8. 1D SERIES REGRESSION
Dataset A 1D series D for the pretrained f and another series (cid:101) D for transfer learning are given by (Fig.
[Equation]
i ∈ [ − 1 , 1]  
N = i = (B.38) with γ ( x i ) = 0 .
4 x i + 1 . 3998 , ξ i ∼ N (1 . 5 , 0 .
8) (normal distribution mean=1.5, variance=0.8) , η i ∼ U ( − 1 , 1) (uniform distribution in [ − 1 , 1] ) .
Network Architecture
The pretrained model f using D was a 3 fully-connected-layer network with 64 nodes at each layer and ReLu used as activation functions except the output layer.
A finetuned model g GD using Gradient Descent (GD) retrained the last layer of f by (cid:101) D .
f
and g GD were trained under L 2 -loss with ADAM optimizer at learning rate 10 − 3 by 8,000 and 12,000 epochs, respectively.
The finetuned model by the LVA method g pesudo using (cid:101) D required no training process but directly replaced the last layer of f by f n (cid:55)→ ( f n + δf n ) with δf n given in Eq.
Hardware One NVIDIA V100 GPU (32 GB GPU memory) with 4 CPUs (128 GB CPU memory).
Runtime Approximately 3 sec in the LVA method, 4 minutes in GD method.
8. S PEECH ENHANCEMENT
Dataset The utterances used in the experiment were excerpted from the Deep Noise Suppression
Challenge (Reddy et al., 2020).
We randomly selected 8000 clean utterances for training andutterances for testing.
Five noise types { White-noise, Train, Sea, Aircraft-cabin, Airplane-takeoff } were involved to form the training set (to estimate the pretrained SE model) and three noise types, { Baby-cry, Bell, Siren } was used to form the transfer learning and testing sets.
For the training set, the 8000 clean utterances were equally divided and contaminated by the five noise types (thus, each noise type had 1600 noisy utterances) at four signal-to-noise ratio (SNR) levels, {− 5 , 0 , 5 , 10 } dB.
For the testing set, 100 clean utterances were contaminated by the { Baby-cry, Bell, Siren } noises at {− 1 , 1 } (dB) SNRs.
For the adaptation set, we prepared 20-400 patches contaminated by three noise types, { Baby-cry, Bell, Siren } , at {− 1 , 1 } dB SNRs.
Implementation All speech utterances were recorded in a 16kHz sampling rate.
The speech signals were first transformed to spectral features (magnitude spectrograms) by a short-time Fourier transform (STFT).
The Hamming window with 512 points and a hop length of 256 points were applied to transform the speech waveforms into spectral features.
To adequately shrink the magnitude range, a log1p function ( log 1 p ( x ) = log (1 + x ) ) was applied on every element in the spectral features.
During training, noisy spectral vectors were treated as input to the SE model to generate enhanced spectral vectors.
The clean spectral vectors were treated as the reference, and the loss was computed based on the difference of enhanced and reference spectral vectors.
Subsequently, the SE model was trained to minimize the difference.
During inference, noisy spectral vectors were inputted to the trained SE model to generate enhanced ones, which were then converted into time-domain waveform with the preserved original noisy phase via inverse STFT.
Network Architecture
The overall flowchart of the SE task is shown in Fig.
In this study, we implemented SE systems using two neural network models to evaluate the proposed LVA .
The first one is the deep denoising autoencoder (DDAE) model (Lu et al., 2013), which serves as a simpler neural-network architecture.
The other is the Bi-directional LSTM (BLSTM)
(Chen et al., 2015), which is relatively complicated as compared to DDAE.
For DDAE, the model is composed of fully-connected layers with 512 units.
Activation functions were added to each layer except the last layer.
The LeakyReLU with negative slope of 0.01 was used as the activation function.
For
BLSTM, the model is consisted of one-layer BLSTM of 300 nodes and one fully-connected layers.
The training setting is the same for the two SE models.
The learning rate of Adam Optimizer was set at 10 − 3 .
After pretraining, only the last layer parameters were updated in the finetuning stage.
According to Eq.
(21), the pseudo inverse of latent features was used to calculate δf .
Hardware One NVIDIA V100 GPU (32 GB GPU memory) with 4 CPUs (128 GB CPU memory).
Runtime Approximately 7 minutes in LVA method and 25 minutes in GD method with 40 utterances.
[Figure]
8.2. A DAPTATION T OWARD M ISMATCHED S PEAKERS
In Section 5.2, we have confirmed the effectiveness of the proposed LVA on the noise type and SNR adaptation of the SE task, where we consider that the main factor of speech distortions are from the background noise.
In this section, we further investigate the achievable performance of LVA on the SE task where speakers are mismatched in the source and target domains.
First, we excluded the utterances of the speakers used in the training set and regenerated the adaptation and test sets by randomly sampling 200 utterances (100 for the adaptation set and 100 for the test set) from the remaining Deep Noise Suppression Challenge dataset.
Then, the experimental setup is exactly the same as in Section 5.2, with speakers in the test set not in the training set.
Moreover, since the speakers did not match, an OT process was performed to align them.
In the following discussion in this section, we denote LVA-OT as LVA for clarity.
We reported the results of BLSTM to compare the
GD and LVA methods in Table 1.
From the table, we can easily note that both GD and LVA effectively improve the PESQ and STOI scores after adaptation.
Moreover, LVA yields better performance than GD consistently over different noise types and SNR levels.
[Table]
A DAPTATION USING F ULL S UB N ET
In this extended experiment, we utilize a FullSubNet (Hao et al., 2021) pretrained network to perform
SE domain adaptation tasks.
The FullSubNet is a state-of-the-art SE model reaching the performance of PESQ: 2.89 and STOI: 0.96 on the DNS-challenge.
This implementation serves to examine the effect of a pretrained network.
To compare with previous results of the BLSTM model in Sec.
8.2, same source domain data and the target domain set are used for adaptation.
The FullSubNet receives an input of spectrum amplitude and outputs an complex Ideal
Ratio Mask (cIRM) to compare with cIRM labels under L 2 -loss.
The architecture of the FullSubNet is composed of a full-band and a sub-band model.
Within each model, two stacked LSTM layers and one fully connected layer are included.
The LSTMs contain 512 hidden units and 384 hidden units in the full-band model and the sub-band model, respectively.
The detailed structures be found in (Hao et al., 2021).
A pretrained FullSubNet was obtained from the official Github 3 , where the the last linear layer was finetuned to adapt to the target set.
The finetuned networks were subsequently evaluated on an adaptation test set to measure PESQ and STOI, where the results are shown in Table 2.
The comparison shows that finetuning from a pretrained FullSubNet achieves enhanced baseline scores on the target domain (see Table 2 [no finetune]).
While adaptation under the gradient descent significantly improves the pretrained network, the proposed LVA method outperforms over the traditional gradient descent method in terms of PESQ and STOI.
The results are consistent with the previous SE experiments to confirm that the proposed LVA promptly reaches better adaptation. [Table]
S UPER RESOLUTION FOR IMAGE DEBLURRING
CNN extension The LVA formulation Eq.
(14) ∼ (19) can be extended to CNN layers with the following observation shown in Fig.
11, where the left hand side depicts the usual convolution operation in CNNs.
Given a CNN, denote the kernels by {C ijαβ } with ( i, j, α, β )
∈
L 1 × L 2 × C in ×
C out and L 1 , L 2 the kernel size ( e.g., 3 × 3 ), C in input channels and C out output channels.
It is noticed that CNN kernels C ijαβ cover a patch of the input image one at a time, denoted by x ijα ; then each covered patch x ijα has the same size as C ijαβ for a fixed β .
Consequently, each output pixel value at ( k, ℓ, β ) is computed by,
[Equation]
where the underscore remarks indices to be summed over.
If we recall that a fully-connected layer of weights { W mn } calculates the output vector v = ( v 1 , . .
. , v m , . . . )
from an input vector u
= ( u 1 , . .
. , u n , . . . [Equation]
by,
[Equation]
then we noticed mathematically there is NO distinction between Eq.
(B.39) and Eq.
(B.40) as index n has the role of ijα and m has the character of kℓβ .
In this view, CNN kernels C ijαβ can be transformed to a 2D matrix W mn by an index conversion function η , and image patches x ijα ( k, ℓ ) can be transformed into a 1-D vector u n by another index function ξ , see Fig.
11 (RHS).
Thus, we completed the conversion of a CNN to a fully-connected-layer and naturally Eq.
(17) ∼ (18) follow.
There can also be possible extensions to other locally-connected networks in future scope.
Datasets We used the CUration of Flickr Events Dataset (CUFED) (Wang et al., 2016) to training SRCNN (Scaman & Virmaux, 2018) for image deblurring, available at https://acsweb. ucsd.edu/~yuw176/event-curation.html .
We randomly selected 2000 images from the
CUFED and randomly crop 20 patches of size 33 × 33 from each image.
Three pairs of training & finetune combination D , (cid:101)
D were considered, see Fig.
[Equation]
[Equation]
i ) | y i ∈ { original patches } , x
[Equation]
{× 3 blurred patches
[Equation]
[Equation]
(cid:101) y
[Equation]
) | (cid:101) y i ∈ { original patches } , (cid:101) x
[Equation]
{× 6 blurred patches }
[Equation]
N
[Equation] [Equation]
=
[Equation]
(B.41)
D 2 = (cid:8) ( x i , y i ) | y i ∈ { original patches } , x i ∈ {× 3 blurred patches } (cid:9) N =40 K i = (cid:101) D 2 = (cid:8) ( (cid:101) x i , (cid:101)
y i ) | (cid:101)
y i ∈ { original patches } , (cid:101) x i ∈ {× 4 , × 6 blurred patches } (cid:9) N =40 K i = (B.42) and
D 3 = (cid:8) ( x i , y i ) | y i ∈ { original patches } , x i ∈ {× 3 , × 4 blurred patches } (cid:9) N =40 K i = (cid:101) D 3 = (cid:8) ( (cid:101) x i , (cid:101)
y i ) | (cid:101) y i ∈ { original patches } , (cid:101) x i ∈ {× 6 blurred patches } (cid:9) N =40 K i = (B.43) Another image dataset SET14 (Zeyde et al., 2010) for testing finetuned models.
Implementation The color images were first transformed to a different color space, YCbCr, to apply the SRCNN model on single luminance channel.
Our SRCNN, Fig. 10, receives D and (cid:101) D of low resolution image patches to output high resolution ones of the same dimension.
A pretrained model composed by a sequence of CNN layers f = f n ◦· · · f 2 ◦ f 1 was pretrained by D under L 2 -loss.
Then the last CNN layer f n was finetuned to adapt to more blurred images (cid:101) D .
Two finetuning methods by GD and LVA were compared.
In particular, the conversion Eq.
(B.39) ⇔ Eq.
(B.40) was used to admit LVA solutions.
Multiple pretrained architectures f were implemented, of which were
[Equation]
[Equation] [Equation] ) (B.44) Network Architecture We adopted the fast End-to-End Super Resolution model similar to the original architecture of SRCNN (Dong et al., 2016) with the insertion of batch-normalization layer right after each ReLu activation function.
The training batch size for f was set at 128, and testing batch size for gradient descent finetuning as 1.
The network architecture of pretrained models corresponding to Eq.
(B.44): 1. 3-layer-CNN of kernel size: (9 , 5 , 5) with channel size: (1 , 32 , 32 , 1) .
5-layer-CNN of kernel size: (9 , 5 , 5 , 5 , 5) channel size: (1 , 32 , 32 , 32 , 32 , 1) .
7-layer-CNN of kernel size: (9 , 5 , 5 , 5 , 5 , 5 , 5) , channel size: (1 , 32 , 32 , 32 , 32 , 32 , 32 , 1) .
where the first and the last number in the channel size is always 1 to indicate the input and output luminance channel.
More extensive results Fig.
13 shows the deblurring results by finetuned models of GD and the
LVA, where our method reached highest PSNR with least finetuning samples N = 256 .
This outcome indicated the LVA method is beneficial for few-shot learning, especially when adaptation data is few.
More extensive experiments were conducted to observe performance.
The following results compared the finetune loss and PSNR with different combinations of training & finetune data in
Eq.
(B.41) ∼ (B.43) and multiple pretrained architectures of f in Eq.
B.44.
In Fig.
(a) ∼ (f), horizontal axis of each subfigure is patch number and the title of each subfigure indicated the transition from the old data D to new data (cid:101)
D by D → (cid:101) D , e.g., {× 3 , × 4 } → × 6 .
Overall, the following facts were observed in Fig.
(a) ∼ (f):
The loss is roughly inverse-proportional to PSNR score.
Increasing CNN layers can handle more difficult task, such as {× 3 } → {× 4 , × 6 } compared to {× 3 } → {× 6 } and {× 4 , × 6 } → {× 3 } since {× 3 } → {× 4 , × 6 } was asked to adapt to more new data than it was originally trained.
In all figures, the LVA method had stable performance over GD in both loss and PSNR even when sample sizes ( x -axis) largely decreased.
We can see ups and downs in orange bars, but not blue bars.
Obviously in Fig.
(a),
GD
showed
difficulty
in
learning
{×
→
× ×
and
{×
→ {
× than {×
×
→ {× 6 } , but the
LVA method remained
roughly the
same
stable
) as well as
retaining
its best performance over GD.
Overall
, GD method improvement was not consistent with
the
data number, likely due
to the local
minimum occurred
in
neural
networks.
Conversely
in LVA method, the performance is (
stably
) proportional to data samples.
Therefore, one could easily derive acceptable result with few finetune data.
Fig.
(a) ∼ (f) then concludes that the LVA method has better performance and is more efficient than
GD in terms of adaptation data amount considered.
Hardware One NVIDIA V100 GPU (32 GB GPU memory) with 4 CPUs (128 GB CPU memory).
Runtime Approximately 4 minutes in LVA method and 15 minutes in GD method with 256 patches (1,000 epochs).
8. A DDITIONAL E XPERIMENTS ON D OMAIN A DAPTIVE I MAGE C LASSIFICATIONS
Table 3 shows the results of Office-31, where the domain transitions occurred within the 3 domains of Office-31:
Amazon (A), DSLR (D) and Webcam (W) , resulting in 6 possible transitions to classify 31 items.
On each individual source domain (A, D, W), the pretrained model was trained forepochs.
Subsequently, it was finetuned onto target domains for another 30 epochs.
The proposed LVA obtained higher adaptation accuracy in most domain transitions, confirming that LVA can promptly adapt to new tasks.
[Table]
Table 4 shows the resulting accuracy under adaptation: MNIST → USPS :
In this experiment, the pretrained model was trained on MNIST (source domain) with 98.94% accuracy by 20 epochs.
While adapting to the target domain USPS, the accuracy of the pretrained model dropped rapidly to 65.32%.
Subsequently, gradient descent (GD) and the proposed LVA were deployed to enhance the target domain results.
Table 1 provides the accuracy attained at different
[Table]
finetune epochs.
It was observed that LVA consistently outperformed GD and reached high accuracy in just a few epochs.
These two additional experiments on image classifications again verify that the proposed LVA is capable of real-world DA tasks; the adaptability is general.
The codes can also be found on Github 4 .
[Figure] [Table]