D OMAIN
A DAPTATION
VIA
L
OW -R ANK B ASIS A P -
PROXIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Domain adaptation focuses on the reuse of supervised learning models in a new context.
Prominent applications can be found in robotics, image processing or web mining.
In these areas, learning scenarios change by nature, but often remain related and motivate the reuse of existing supervised models.
While the majority of symmetric and asymmetric domain adaptation algorithms utilize all available source and target domain data, we show that efﬁcient domain adaptation requires only a substantially smaller subset from both domains.
This makes it more suitable for real-world scenarios where target domain data is rare.
The presented approach ﬁnds a target subspace representation for source and target data to address domain differences by orthogonal basis transfer.
By employing a low-rank approximation, the approach remains low in computational time.
The presented idea is evaluated in typical domain adaptation tasks with standard benchmark data.
I NTRODUCTION
Supervised learning and in particular classiﬁcation is an essential task in machine learning with a broad range of applications.
The obtained models are used to predict the label of unseen test samples.
A basic assumption in supervised learning is that the underlying domain or distribution is not changing between training and test samples.
If the domain is changing from one task to a related but different task, one would like to reuse the available learning model.
Domain differences are quite common in real-world scenarios and eventually, lead to substantial performance drops (Weiss et al., 2016).
A domain adaptation example is the classiﬁcation of web pages.
A classiﬁer is trained in the domain of university web pages with a word distribution according to universities.
In the test scenario, the domain has changed to non-university web pages, where the word distribution is related, but may not be similar to the training distribution.
More formally, let Z
=
z 1 , . .
z m } ∈ R d be source data sampled from the source domain distribution p
( z ) and let X = { x 1 , . .
, x n } ∈ R d be target data from the target domain distribution p ( x ) .
Traditional machine learning assumes similar distributions, i.e. p ( z ) ∼ p
( x ) ,
but domain adaptation assumes different distributions, i.e. p ( z ) (cid:54) = p ( x ) and appears in the web page example where Z could be data of university websites and X are data of non-university websites.
Multiple domain adaptation techniques have been already proposed, following different strategies and improving the prediction performance of underlying classiﬁcation algorithms in test scenarios (Weiss et al., 2016; Pan & Yang, 2010).
The common domain adaptation approach requires a large number of source or target samples, which is indeed a disadvantage of many domain adaptation approaches and is not guaranteed in restricted environments where labeling is expensive (Weiss et al., 2016).
Further, learning a matrix for the alignment of domain distributions does not necessarily lead to effective adaptation shown by (Raab & Schleif, 2018), where domain differences are explicitly solved by overriding the basis of one domain with another and is competitive to current state of the art domain adaptation algorithms (Gong et al., 2012; Long et al., 2015; 2013; Pan et al., 2011;
Fernando et al., 2013; Sun et al., 2016).
The main contribution of this work is to improve the Basis-Transfer (Raab & Schleif, 2018) approach by deriving a more simpliﬁed closed-form solution of the least-squares problem and enhance this by a Nystr¨om based dimensionality reduction.
The resulting method has a better prediction performance and makes it easer to apply.
Further, it is the fastest domain adaptation algorithm in terms of computational complexity, while maintaining its excellent performance by using only a subset of source and target data.
It determines a target subspace representation for both domains and transfers target basis information to source data via Nystr¨om Singular Value Decomposition (SVD) with class-wise sampling.
Further, we show that Nystr¨om approximation is well suited for nonsquare landmark matrices, in contrast to common decomposition (Nemtsov et al., 2016) by applying post-processing.
The rest of the paper is organized as follows: We give an overview of related work in section 2.
The underlying mathematical concepts are given in section 3.
The proposed approach is discussed in section 4, followed by an experimental part in section 5, addressing the classiﬁcation performance and computational time.
A summary with a discussion of open issues is provided in the conclusion at the end of the paper.
More detailed derivations, description of experiments, implementation details and additional results can be found in the appendix.
R ELATED W ORK
In general, homogeneous transfer learning (Weiss et al., 2016) approaches, or domain adaptation (DA), distinguish roughly between the following strategies: Methods implementing the symmetric feature adaptation (Weiss et al., 2016) are trying to ﬁnd a common latent subspace for source and target domain to reduce marginal distribution differences, such that the underlying structure of the data is preserved in the subspace.
A baseline approach for symmetric feature adaptation is the
Transfer Component Analysis (TCA) (Pan et al., 2011).
TCA ﬁnds a suitable subspace transformation called transfer components via minimizing the Maximum Mean Discrepancy (MMD) in the
Reproducing Kernel Hilbert Space (RKHS).
The Geodesic Flow Kernel (GFK) ﬁnds a particular subspace by embedding original data onto a Grassmannian manifold, matches source and target on the geodesic ﬂow and determines a suitable domain adaptation kernel by applying the kernel trick (Gong et al., 2012).
The relational-knowledge adaptation aims to ﬁnd some relationship between source and target data (Weiss et al., 2016).
Transfer Kernel Learning (TKL)
(Long et al., 2015) is a recent approach, which approximates a kernel of training data K ( Z ) by test kernel K ( X ) via the Nystr ¨ om kernel approximation.
However, recent studies conclude that the asymmetric transformation of source data is a promising approach in domain adaptation due to the better classiﬁcation performance (Sun et al., 2016; Fernando et al., 2016; Raab & Schleif, 2018;
Long et al., 2013).
Possibly, because a classiﬁer is trained on the source label information in the target domain (Sun et al., 2016).
The good performance has led to a variety of unsupervised asymmetric domain adaptation algorithms (Elhadji-Ille-Gado et al., 2017; Shao et al., 2014; Blitzer et al., 2011; Thopalli et al., 2019), to name a few.
The asymmetric feature adaptation approaches try to transform source domain data in the target (subspace) domain to match the source to target subspace and is most related to our work.
In comparison to the symmetric feature adaptation approaches, there is no shared subspace, but only the target space (Weiss et al., 2016).
The Joint Distribution Adaptation (JDA) (Long et al., 2013) solves divergences in marginal distributions similar to TCA, but aligning conditional distributions with pseudo-labeling techniques.
The Subspace Alignment (SA) (Fernando et al., 2013) computes a target subspace representation based on the correlation between target and source subspace projectors and hence aligning source and target data.
The Correlation Alignment (CORAL) (Sun et al., 2016) technique transfers secondorder statistics of the target domain into whitened source data.
Given these methods, there is an overall trend in domain adaptation in the direction of non-linear problem formulations, indirect alignment of matrix differences and learning a suitable subspace for the source domain in the target domain and have similar problem statements to ours.
However , in this work, we do not learn a suitable source subspace transformation, but explicitly override the structural information of source domain with the target one.
With this, we model the source subspace domain as part of the target subspace.
The considered domain adaptation methods have approximately a complexity of O ( n 2 ) where n is the most signiﬁcant number of samples concerning target or source.
These algorithms pursue transductive adaptation (Pan & Yang, 2010), because some unlabeled test data must be available at training time.
These transfer-solutions cannot be directly used as predictors, but instead, are wrappers for classiﬁcation algorithms.
The respectively used baseline classiﬁer is the Support Vector
Machine (SVM).
P RELIMINARIES
We introduce the basics of the Nystr¨om kernel approximation in section 3.1, which is the foundation of the Nystr¨om based Singular Value Decomposition in section 3.2.
The Nystr¨om-SVD is used for creating the subspace transformation of Nystr¨om Basis Transfer in section 4.1.
3. N YSTR ¨ OM A PPROXIMATION
The computational complexity of calculating kernels or eigensystems scales with O ( n 3 ) where n is the sample size (Williams & Seeger, 2001).
Therefore, low-rank approximations and dimensionality reduction of data matrices are popular methods to get a better computational performance.
In this scope, however not limited to it, the Nystr¨om approximation (Williams & Seeger, 2001) is a reliable technique to accelerate eigendecomposition or approximation of general symmetric matrices (Gisbrecht & Schleif, 2015).
It computes an approximated set of eigenvectors and values based on a usually much smaller sample matrix.
The landmarks are typically picked at random, but advanced sampling concepts could be used as well (Talwalkar et al., 2012).
The approximation is exact if the sample size is equal to the rank of the original matrix and the rows of the sample matrix are linear independent (Gisbrecht &
Schleif, 2015).
In general, the Nystr ¨ om approximation technique assumes a symmetric matrix K ∈ R n × n with a decomposition of the form
[Equation]
with A ∈ R s × s , B ∈ R s × ( n − s ) , C ∈ R ( n − s ) × s and D ∈ R ( n − s ) × ( n − s ) .
The submatrix A is called the landmark matrix containing s randomly chosen rows and columns from K and has the Eigenvalue Decomposition (EVD)
A =
UΛU − 1 .
Where eigenvectors are U ∈ R s × s and eigenvalues are on the diagonal of Λ ∈ R s × s .
The remaining approximated eigenvectors ˆ U of K as the part C or B T , are obtained by the Nystr ¨ om method with ˆ UΛ = CU .
Combining U and ˆ U the full approximated eigenvectors of K are
[Equation]
The right part of the EVD ( ˜ U − 1 ) of K can be obtained via Nystr ¨ om similar to equation 2 by
[Equation]
Combining equation 2, equation 3 and Λ , the matrix K is approximated by
[Equation]
The Nystr¨om approximation error is given by the Frobenius Norm between ground truth and reconstructed matrices, i. e. E ny = ||
˜ K − K || F , with bounds proven by Gittens & Mahoney (2013).
3. G ENERAL M ATRIX A PPROXIMATION
Another application of the Nystr¨om method is the approximation of the Singular Value Decomposition, which generalizes the concept of matrix decomposition with the consequence that respective matrices must not be squared.
Let K ∈ R n × d be a rectangular matrix with the decomposition as in equation 1.
The SVD of the landmark matrix is given by A = LSR T where L are left and R are right singular vectors.
S are positive singular values.
The left and right singular vectors for the non-symmetric part C and B are obtained via Nystr¨om techniques and are deﬁned as ˆ L = CRS − 1 and ˆ R =
B T LS − 1 respectively (Nemtsov et al., 2016).
Applying the same principal as for Nystr¨om-EVD, K is approximated by
[Equation]
3. P OLAR D ECOMPOSITION
The Polar decomposition (Higham, 2005) theorem is used to validate the proposed subspace transformations as PCA transformations in section 4 and the class-wise sampling strategy in section 4.2.
It is a universal decomposition applicable to an arbitrary matrix and is deﬁned as X = QP .
Where
Q = LR T and P =
RSR T with S as singular values, L and R are left and right singular vectors respectively.
If X is a square matrix, the decomposition is unique and U is orthogonal and a rotation matrix.
P is positive semi-deﬁnite and contains the scaling factors of X .
Theorem 1 (Reusing of Eigensystem) .
Let X ∈ R n × d and K =
XX T with EVD of K =
UΛU −and Singular Value Decomposition (SVD) of K = LSR T .
Taking the square root of K and using the Polar Decomposition with
[Equation]
then the eigenvectors and square root eigenvalues of K are singular vectors and values of X respectively.
3. G ERSCHGORIN T HEOREM
The Gerschgorin theorem (Varga, 2004) provides a geometric structure to bound eigenvalues to socalled discs for complex square matrices, but also generalize to none complex square matrices.
The work of Qi (1984) expands the Gerschgorin circles to so-called Gerschgorin type circles for singular values:
Theorem 2 (Gerschgorin Type Bound for Singular Values (Qi, 1984)) .
Given the matrix X ∈ R n × d with n, d ≥ 1 , the singular values { s i } ni =1 of X are in the range of
[Equation]
[Equation]
Where p i = | x ii | and the range r i is deﬁned as
[Equation]
j =
[Equation]
j (cid:54) =
i | x ij |
[Equation]
n
[Equation]
j
[Equation] [Equation]
j
[Equation]
= i | x
[Equation]
|
[Equation]
n.
By this, it is possible to estimate the numerical range of singular values of X and is used in the error bound given in section 4.3.
B ASIS T RANSFER
The task of domain adaptation is to align distribution differences.
That means that underlying statistics will be similar afterwards.
The key idea of the approach presented in the following is that explicit alignment of data matrices without any knowledge of statistics and distribution of data is possible and successful.
It is based on the work of Sch ¨ olkopf et al.
(1998), which proposed that if two kernel matrices are similar, they follow similar distributions.
Hence, we make the domain data as similar as possible by overriding the syntactic structure, which will result in similar kernels .
This also applies to arbitrary rectangular data matrices of a linear kernel, because it can be reconstructed by using theorem 1.
Hence, the method is also applicable in the original feature space without requiring a positive semi deﬁnite kernel.
The recent Basis-Transfer (BT) (Raab & Schleif, 2018) approach already showed great transfer capabilities and performance by aligning X ∈ R n × d and Z ∈ R m × d with a small error in terms of the Frobenius norm.
However, this required equal samples sizes n = m .
In BT, the following optimization problem was considered:
[Equation]
Where M and T are transformation matrices, drawing the source ( Z ) closer to target ( X ) data.
A solution of equation 9 is found in closed-form, summarized in three steps (Raab & Schleif, 2018):
First, normalize data to standard mean and variance.
It is assumed that this will align marginal distributions in Euclidean space without considering label information (Raab & Schleif, 2018).
Second, compute an SVD of source and target data, i. e. Z =
L Z S Z R T Z with L Z ∈ R m × m
and
R Z ∈ R d × d .
And X =
L X S X R TX with L X ∈ R n × n and R X ∈ R d × d .
Note that we denote (cid:12) Z / (cid:12) X as source/target related matrix respectively.
Next, the approach assumes S Z ∼ S X in terms of the Frobenius norm due to similar domains providing similar scaling factors by singular values and normalization.
Finally, compute a solution for equation 9 by solving the linear equations.
One obtains M = L X L − 1 Z and T =
R − 1 Z R TX .
Note that L Z L − 1 Z = R Z R − 1 Z
= I .
Finally, a transfer operation is applied such that the source matrix is approximated by using the available target information by
[Equation]
With ˜ Z ∈ R m × d as approximated source data, used for training.
In the following, the work (Raab & Schleif, 2018) is hereby substantially improved and we propose a Nystr¨om based version with following improvements: Reduction of computational complexity, neglecting sample size requirements and achieve a low-rank projection via Nystr¨om approximation with class-wise sampling and implicit dimensionality reduction.
Recap equation 9 and consider the slightly changed optimization problem
[Equation]
Where a transformation matrix M must be found, which is again obtained in closed-form.
Note this is a transfer learning interpretation of the orthogonal procrustes problem (Sch ¨ onemann, 1966).
Because we apply a dimensionality reduction technique, just the left-sided transformation matrix must be determined, derived as follows:
Based on the relationship between SVD and EVD, the
Principal Component Analysis (PCA) can be rewritten in terms of SVD.
Consider the target matrix with SVD:
[Equation]
Where R X ∈ R d × s as eigenvectors and S 2 X ∈ R s × s as eigenvalues of X T X .
By choosing only the biggest s eigenvalues and corresponding eigenvectors the dimensionality of X is reduced by
[Equation]
With L s ∈ R n × s , S s ∈ R s × s .
X s ∈ R n × s is the reduced target matrix.
Hence, only a left sided transformation in equation 11 is required, because the right sided transformation is omitted in equation 13.
Note, that for equation 12 a linear covariance or kernel is used.
However, this procedure requires a complete data matrix or corresponding singular values and scales the in worst case with O ( n 3 )
(Williams & Seeger, 2001).
In the following, we apply Nystr ¨ om-SVD and show that only a subset of the data is required, which simultaneously reduces computational complexity and eliminates the need to examine all singular values.
4. N YSTR ¨ OM B ASIS T RANSFER
Let Z and X have a valid decomposition given as in equation 1.
Note for clarity the Nystr ¨ om notation is used as in section 3.2.
For a Nystr ¨ om-SVD we sample from both matrices s values obtaining landmarks matrices A Z
=
L Z S Z R T Z ∈ R s × s and A X
=
L X S X R T X ∈ R s × s .
Based on Nystr¨om-SVD in equation 5, the dimensions are reduced as in equation 13 keeping only most relevant data structures
[Equation]
[Equation]
Hence, it is sufﬁcient to only compute a SVD of A X instead of X with s (cid:28) m, d and therefore is considerably lower in computational complexity.
Analogy, we approximate source data by Z s = ˜
L Z S Z ∈ R n × s .
Since we again assume S Z ∼ S x due to data normalization and domain similarities, solving the linear equation as a possible solution for equation 11, leads to M =
˜
L X ˜ L − 1 Z .
Plugging it back we obtain
[Equation]
Where again a basis of a target subspace transfers structural information into the source domain.
The matrix Z s is used for training and X s is used for testing.
We showed in equation 13, that equation 14 and equation 15 are valid PCA transformations.
By deﬁnition of SVD follows ˜
L Z ˜ L TZ =
˜
L X ˜ L − 1 X
=
I and ˜ L X is an orthogonal basis.
Therefore, equation 14 and equation 15 are orthogonal transformations.
In particular, equation 15 transforms the source data into the target subspace and projects it onto the principal components of X .
If data matrices X and Z are standardized 1 , the geometric interpretation is a rotation of source data w.r.t to angles of the target basis.
The relationship between BT and NBT is in the number of samples s used by Nystr¨om.
If the rank of the data matrices is equal to s the approximation is exact and it became a subspace version of BT.
According to (Weiss et al., 2016), it is an asymmetric transfer approach.
Further, it is transductive (Pan & Yang, 2010), where unlabeled target data is needed at training time .
Subsequently this approach is denoted as Nystr¨om Basis Transfer (NBT).
But uniform sampling may not be optimal for Nystr¨om, given labeled data in a classiﬁcation task (Schleif et al., 2018).
Therefore, we integrate class-wise sampling in the following.
4. S AMPLING S TRATEGY FOR N YSTR ¨ OM
The standard technique to create Nystr¨om landmark matrices is to sample uniform or ﬁnd clusters in the data matrix (Talwalkar et al., 2012).
In supervised learning, sampling should utilize classwise sampling to properly include class-depending attributes of a matrix into the approximation (Schleif et al., 2018).
This is especially necessary for ||Y|| > 2 .
However, a decomposition as in equation 1, required for Nystr¨om-SVD is intractable with class-wise sampling, because respective matrices are non-square: Let Z ∈ R m × d with m (cid:54) =
d and landmark indices I =
{ i 1 , . .
, i s } with at least one i j > d , then it is by deﬁnition undeﬁned and is especially true for n > d .
Therefore, we sample rows class-wise forming A dZ ∈ R s × d instead of A Z ∈ R s × s .
Using the insights of theorem 1, we implicitly obtain an SVD of a square landmark matrix, i. e. A dZ A dTZ =
K dZ ∈ R s × s by computing
[Equation]
Hence, we can proceed without a non-square landmark matrix and using the SVD of A dZ analogy as in equation 15.
Therefore, it is possible to sample from the whole range of source data and by application of the Polar decomposition, the standard decomposition as in equation 1 is not required.
The resulting singular values and vectors are utilized for successive
Nystr¨om approximations.
The sampling from test data X is done uniformly row-wise, because of missing class information and SVD is obtained analogy via A dX =
L X S dX R TX and used in equation 14.
However, the possible range of singular values of A dZ is naturally not equal to A Z .
Utilizing theorem 2 it is easy to show that S dZ (cid:54) =
S Z .
It scales approximated matrices Z s (equation 15) different through S dZ and accurate normalization of matrix cannot be guaranteed.
Therefore, we apply a postprocessing correction and apply z-normalization against the standard convention after NBT.
The singular vectors also have an approximation error.
However, both subspace projections are based on the same transformation matrix, hence making an identical error and as a result, the error should not affect the classiﬁcation.
The process is given in the ﬁgure 2 (appendix A.2) and showing the alignment of source to the target domain.
Further, it shows the mechanics of the approach: The structural similarities stay the same, while the scaling is changed leading to a high similarity after the post-processing correction.
4. P ROPERTIES OF N YSTR ¨ OM B ASIS T RANSFER
The computational complexity of Nystr¨om Basis Transfer (NBT) is composed of economy-size
SVD of landmark matrices A dZ and A dX with complexity O
(2 s 2 ) .
The matrix inversion of diagonal matrix S dX − 1 in equation 14 can be neglected.
Remaining k matrix multiplications are O
( ks 2 ) contributing to the overall complexity of NBT which is O ( s 2 ) with s (cid:28) n, m, d .
This makes
NBT the fastest domain adaptation solution in terms of computational complexity in comparison to discussed methods in section 2.
The difference between source and target domain after NBT, i.e. approximation error of source by target domain is bounded by
[Equation]
[Equation]
Where || ˜ L X || ≤ √ n is the bounding of the normalized singular vectors and D ( · ) is the maximum sum of the s singular values obtained by the numerical estimation, given in theorem 2, of X s and
Z s respectively.
The equation 16 shows that NBT has a lower norm compared to BT or original data and proofs that the matrices are aligned during NBT, reducing the distribution differences.
The respective proof is given in appendix A.1.
The out-of-sample extension for unseen target/source samples, e. g. x ∈ R d , is analog to equation 14.
Based on equation 13 a subspace projection via (approximated) right singular vectors is also valid.
Hence, a sample can be projected into the subspace via x s = x ˜ R TX = x (cid:104) R X
S dX − 1 L T B X (cid:105) and be evaluated by an arbitrary classiﬁer learned in the subspace.
The pseudo code of NBT is shown in Algorithm 1.
Require:
Z as m sized training; X as n sized test set; Y as m sized training label vector; s as number of landmarks parameter.
Ensure: New Source Z s ; new Target X s section
E XPERIMENTS
We follow the experimental design typical for domain adaptation algorithms (Long et al., 2015;
Gong et al., 2012;
Long et al., 2013; Pan & Yang, 2010; Pan et al., 2011; Sun et al., 2016; Mahadevan et al., 2019; Fernando et al., 2013).
A crucial characteristic of datasets for domain adaptation is that domains for training and testing are different but related.
This relation exists because the train and test classes have the same top category or source.
The classes itself are subcategories or subsets.
The parameters for respective methods 2 are determined for best performance in terms of accuracy via grid search.
A detailed description of experiments can be found in appendix A.4.
In the experiments, the SVM independent of being a baseline or underlying classiﬁer for domain adaptation methods uses the RBF-Kernel.
The tests are conducted on the datasets groups Reuters, Newsgroup and
Ofﬁce-Caltech.
A detailed dataset description is given in appendix A.3.
5. P ERFORMANCE C OMPARISON
The experiments are cumulated as three dataset group, which are Reuters, Newsgroup and OfﬁceCaltech.
The results are summarized in table 1.
It shows the mean errors of the experiments per dataset group.
To determine statistically signiﬁcant differences, the two-sided KolmogorovSmirnoff-Test with a p-value of 1% is applied.
The ∗ marks statistical signiﬁcance compared to
NBT.
The NBT is compared to baseline SVM and standard domain adaptation methods.
The NBT and BT methods have excellent performance and outperform every other algorithm by far and are overall similar.
Looking at the datasets individually, the only competitive approach is SA, however, in the overall rating, all non-BT methods are statistically worse.
Note that BT and NBT are very similar at Ofﬁce-Caltech, this is caused because within NBT the rank r of a given image data matrices is equal to the number of samples drawn by the Nystr¨om approximation.
Hence, the approximation is exact and we use the most important r singular vectors and values.
BT and NBT are overall much better than remaining approaches, caused by following possible reasons:
Aligning differences of Frobenius norm between domains using a target (subspace)-transformation is sufﬁcient as a problem statement and overriding of source basis is a successful approach.
Further, by Sch¨olkopf et al.
(1998) it will also align differences in subsequent kernels.
Explicit integration of non-linearity or kernel statistic alignments does not necessarily lead to more knowledge transfer.
But, second-order statistics but also structural and geometric information should be transferred.
[Figure]
l
[Figure]
ss i f i c a t
i
[Figure]
rr o r
[Figure]
C l a ss i f
i c a t i on E rr o r [Figure]
Dataset
SVM
TCA
JDA
TKL
GFK
SA
CORAL
BT
NBT
Reuters 32.1 ∗ 31.8 ∗ 33.1 ∗ 23.2 ∗ 35.0 ∗ 7.
35.7 ∗ 3.
Newsgroup 17.8 ∗ 14.7 ∗ 17.4 ∗ 6.
22.5 ∗ 2.
28.1 ∗ 2.
Ofﬁce-Caltech 55.5 ∗ 51.5 ∗ 51.4 ∗ 49.2 ∗ 55.3 ∗ 68.7 ∗ 52.6 ∗ 20.
Overall Mean 35.1 ∗ 32.7 ∗ 34.0 ∗ 26.4 ∗ 37.6 ∗ 26.2 ∗ 38.8 ∗ 8.
The sensitivity of the number of landmarks on prediction error as the only parameter of NBT is demonstrated in ﬁgure 1.
It shows a comparison of the number of landmarks and the mean classiﬁcation error over Reuters and Ofﬁce-Caltech datasets.
At Reuters, it is a monotonically decreasing convex function with decreasing error by increasing the number of landmarks.
This supports the
Nystr ¨ om error expectation of approximating the real rank of a matrix.
However, the image function has many local minima.
We assume this indicates that only a subset of features is relevant and correlate for classiﬁcation and remaining features are noise.
A time comparison is given in the appendix
A.5 showing that our NBT approach is the overall fastest domain adaptation approach.
C ONCLUSION
We proposed a low-rank domain approximation algorithm called Nystr¨om Basis Transfer.
It transfers source and target data into the target subspace, requiring only a subset of domain data from both domains.
The dimensionality reduction, paired with smart class-wise sampling showed its reliability and robustness in this study.
Validated on common domain adaptation tasks and data, it showed an outstanding performance both in absolute and statistical values.
Additionally, it is lowest in computational complexity compared to discussed solutions.
NBT is an extension of earlier versions of
Basis Transfer via Nystr¨om Methods and is a low-rank and fast domain adaption solution.
In future work, deep transfer learning, real-world and other domain adaptation datasets should be considered.
A more comprehensive discussion of the Nystr ¨ om approximation error with the proposed decomposition should be theoretically evaluated and compared to current Nystr¨om techniques.
A A PPENDIX A. N ORM B OUND OF N YSTR ¨ OM B ASIS T RANSFER
This section discusses the approximation error of Nystr¨om Basis Transfer in terms of the Frobenius norm of the matrix difference shown in equation 16.
First, we clarify some deﬁnitions used in the subsequent theorem.
In this work, we deﬁne the Frobenius norm of a matrix X ∈ R n × d as (cid:107)
[Equation]
j
[Equation]
1 | x
[Equation]
| 2 .
By using theorem 2 we can bound the norm of the singular values (cid:107) S (cid:107) F of X by the square root of the squared sum of the numerical range given by
[Equation]
Finally, we assume that the non-zero singular vectors are normalized and have by deﬁnition unit length.
Therefore, the norm of a singular vector matrix with n singular vectors can be bounded by √ n .
Theorem 3.
Given two rectangular matrices X , Z ∈
R n × d with n, d > 1 and rank ( X ) , rank ( Z ) > 1 .
The norm (cid:107) Z s − X s (cid:107) F in the subspace
R s induced by normalized singular vectors L ∈ R n × s is bounded by
[Equation]
Proof.
We start by prooﬁng the ﬁrst part given by (cid:107) Z s − X s (cid:107) F ≤ √ n || D sX − D sZ || .
Using equation 14, equation 15 and equation 16 the respective term can be rewritten in terms of a SVD-PCA using only the biggest s singular values.
Further, substituting M = L X L − 1 Z leads to
[Equation]
Where we assume 1 ≤ s < n .
This can be simpliﬁed applying the Cauchy-Schwarz inequality twice with
[Equation]
The norm of the singular values S Z , S X ∈ R s × s can be bounded by equation 17 resulting in
[Equation]
[Equation]
Where p zi and r zi are the bound of a singular value of Z s and p xi and r xi are the bound of a singular value of X s .
Further, the singular vectors L X are normalized and thus can be bounded by (cid:107) L X (cid:107) F ≤ √ n .
Applying both to equation 20 leads to
[Equation]
The second term √ n || D sX − D sZ || < (cid:107) Z − X (cid:107) F is obtained analogy.
Given the SVD of the original matrices we obtain
[Equation]
By again using the Cauchy-Schwarz inequality we obtain
[Equation]
[Equation]
Because L Z and L X have the same size they are bounded by √ n and because R Z , R X ∈ R d × d and are also normalized, the norm can be bounded by (cid:13)(cid:13) R ( · ) (cid:13)(cid:13) ≤ √ d respectively.
Because we are not in the subspace R s but R d with s < d and the singular values are S Z , S X ∈ R n ×
d the numerical bound becomes
[Equation]
Applying the bounds to equation 24 we obtain
[Equation]
Simplifying this term further leads to
[Equation] [Equation]
Because we assume d > 1 and a minimum of one non-zero singular vector in R ( · ) this at least
[Equation]
[Equation]
The result from the above allows the conclusion that the norm of the subspace matrices is smaller than the norm of the original matrices.
Note that because BT (Raab & Schleif, 2018) is applied in R d , this allows the conclusion that E NBT <
E BT .
If the norm of the difference of two matrices is small, they are similar and if they are numerically similar, we can conclude that the distributions are also similar (Song & Park, 2018).
Finally, this is the reason why NBT aligns the distribution of two given matrices.
However, note that similar distributions not necessarily means a good classiﬁcation performance in terms of accuracy by an arbitrary classiﬁer in a transfer learning setting.
A. P ROCESS OF N YSTR ¨ OM B ASIS T RANSFER
In this section, an illustration of Nystr¨om Basis Transfer (NBT) is given.
Figure 2 shows the process of Nystr¨om Basis Transfer.
The ﬁrst row shows the samples of Nystr¨om to create the approximated set of subspace projectors.
The second row shows the data after the subspace projection.
The similarity in structure but dissimilarity in scaling, as discussed in section 4.3 is visible.
The last row shows the data after applying post projection and the difference is not visible anymore.
A. D ATASET D ESCRIPTION
The study consists of 24 benchmark datasets, already preprocessed and taken from (Gong et al., 2012), (Long et al., 2015) and (Long et al., 2014).
Reuters -21578 4 (Long et al., 2015):
A collection of Reuters news-wire articles collected in 1987.
The text is converted to lower case, words are stemmed and stopwords are removed.
With the
Document Frequency (DF)-Threshold of 3, the numbers of features are cut down.
Finally, TermFrequency Inverse-Document-Frequency (TFIDF) is applied for feature generation (Dai et al., 2007).
The three top categories organization (orgs) , places and people are used in our experiment.
To create a transfer problem, a classiﬁer is not tested with the same categories as it is trained on, i. g. it is trained on some subcategories of organization and people and tested on others.
Therefore, six datasets are generated: orgs vs. places , orgs vs. people , people vs. places , places vs. orgs , people vs. places and places vs. people .
They are two-class problems with the top categories as the positive and negative class and with subcategories as training and testing examples.
20-Newsgroup 5 (Long et al., 2014):
The original collection has approximately 20.000 text documents from 20 Newsgroups and is nearly equally distributed in 20 subcategories.
The top four categories are comp , rec , talk and sci and containing four subcategories each.
We follow a data sampling scheme introduced by Long et al.
and generate 216 cross domain datasets based on subcategories: Let C be a top category and { C 1 , C 2 , C 3 , C 4 } ∈ C are subcategories and analogy
K with { K 1 , K 2 , K 3 , K 4 } ∈ K .
Select two subcategories each, e. g. C 1 , C 2 , K 1 and K 2 , train a classiﬁer, select another four and test the model on it.
The top categories are respective classes.
Following this, 36 samplings per top category-combinations are possible, which are in total 216 dataset samplings.
This is summarized as mean over all test runs as comp vs rec , comp vs talk , comp vs sci , rec vs sci , rec vs talk and sci vs talk .
This version of 20-Newsgroup has 25.804 TF-IDF features within 16.021 documents (Long et al., 2015).
Caltech-256-Ofﬁce
(Gong et al., 2012):
The ﬁrst, Caltech ( C ) is an extensive dataset of images and contains 30.607 images within 257 categories.
The Ofﬁce dataset is a collection of images drawn from three sources which are from amazon (A) , digital SLR camera (DSLR) and webcam (W) .
They vary regarding camera, light situation and size, but ten similar object classes, e. g. computer or printer, are extracted for a classiﬁcation task.
Duplicates are removed, as well as images which have more than 15 similar Scale Invariant Feature Transform (SIFT) in common (Gong et al., 2012).
The ﬁnal feature extraction is done with Speeded Up Robust Features Extraction (SURF) and encoded with 800-bin histograms.
Finally, the twelve sets are designed to be trained and tested against each other by the ten labels (Gong et al., 2012).
Note to reproduce the results of this paper, one should use the linked version of datasets with the same choice of subcategories.
A summary of all datasets is shown in table 2.
Regardless of the dataset, data have been standardized to standard mean and variance.
The experiments are summarized as the mean over 20 runs with a 2 -fold sampling, with separated samplings for source and target data.
A. I MPLEMENTATION D ETAILS
The parameters for the respective domain adaptation algorithms are obtained via grid-search for best performance on a dataset type, meaning text with Reuters and Newsgroup and image with Ofﬁce vs
Caltech.
All algorithms using the SVM with an RBF-Kernel, except GFK which suggest a k nearest
[Figure]
[Table] [Table]
neighbor classiﬁer.
The C parameter of the SVM is set to 10 for all experiments.
For SVM the
LibSVM is used. The TCA also has one parameter which gives the subspace dimensions and is set to µ = 50 for both datasets. The JDA has two model parameters. First, the number of subspace bases k , which is set to 100.
Second, the regularization parameter λ is set to 1 for both sets.
The TKL approach has the eigenvalue dumping factor ξ as a parameter, which was set to 2 for the text datasets and 1 . 1 for the image datasets.
For the GFK solution, the parameter, number of subspace dimensions is set to 20 for image and 40 for text data.
The SA subspace dimension parameter d is set to 5 for every dataset.
The CORAL has no free parameters and CGCA is not evaluated, because of numerical issues.
The BT approach has no free parameter and for NBT, the subspace dimension is set to 500 for text and 210 for images.
A. T IME C OMPARISON
The mean time results in seconds of the cross-validation study per data set group are shown in the table 3.
Note that SVM is the underlying classiﬁer for the compared approaches and is presented for the baseline and not listed for stand-alone comparison but included into the time measurement of domain adaptation approaches.
In the overall comparison, NBT is the fastest compared to the discussed solutions and in contrast to BT, the required time is a magnitude smaller, especially at
Newsgroup, which is a high dimensional dataset.
This is an expected effect due to Nystr¨om approximation.
Individually, the NBT is the fastest algorithm at the Newsgroup and Ofﬁce-Caltech data set.
At Reuters, JDA is the fastest.
In summary, the differences between the winning algorithm and other algorithms are very small.
However, GFK and CORAL are the slowest methods with a factor of at least 90 compared to the fastest approach for Newsgroup and are clear outliers in this time comparison.
A. D ETAILS OF P REDICTION P ERFORMANCE
The detailed version of the results of the experiments are shown in table 4.
It is an extended version of table 1 of the main paper.
[Table]
Sci vs Talk 21.
Newsgroup Mean 17.8 ∗ 14.7 ∗ 17.4 ∗ 6.
22.5 ∗ 2.
28.1 ∗ 2.
C vs A 50.
C vs W 58.
C vs D 54.
A vs C 58.
A vs W 66.
A vs D 59.
D vs C 76.
D vs A 72.
D vs W 24.
W vs C 74.
[Table]