F UNDAMENTAL L IMITS OF T RANSFER L EARNING IN B INARY C LASSIFICATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
A critical performance barrier in modern machine learning is scarcity of labeled data required for training state of the art massive models, especially in quickly emerging problems with lack of extensive data sets or scenarios where data collection and labeling is expensive/time consuming.
Transfer learning is gaining traction as a promising technique to alleviate this barrier by utilizing the data of a related but different source task to compensate for the lack of data in a target task where there are few labeled training data.
While there has been many recent algorithmic advances in this domain, a fundamental understanding of when and how much one can transfer knowledge from a related domain to reduce the amount of labeled training data is far from understood.
We provide a precise answer to this question for binary classiﬁcation problems by deriving a novel lower bound on the generalization error that can be achieved by any transfer learning algorithm (regardless of its computational complexity) as a function of the amount of source and target samples.
Our lower bound depends on a natural notion of distance that can be easily computed on real world data sets.
Other key features of our lower bound are that it applies to any arbitrary source/target data distributions and requires minimal assumptions that enables it application to a broad range of problems.
We also consider a more general setting where there are more than one source domains for knowledge transfer to the target task and develop new bounds on generalization error in this setting.
We also corroborate our theoretical ﬁndings on real image classiﬁcation and action recognition data sets.
These experiments demonstrate that our natural notion of distance is indicative of the difﬁculty of knowledge transfer between different pairs of source/target tasks, allowing us to investigate the effect of different sources on the target generalization error.
Furthermore, to evaluate the sharpness of our bounds we compare our developed lower bounds with upper-bounds achieved by transfer learning base-lines that utilize weighted empirical risk minimization on the combination of source(s) and target data sets.
I NTRODUCTION
Modern machine learning models such as deep neural networks have enjoyed wide success in many domains Krizhevsky et al.
The success of such deep models critically relies on an enormous amount of data required for training these massive models.
For instance, GPT3 which is the state of the art model for natural language process has 175 billion parameters and requires a data set of size 45 terabytes for training.
However, in new or emerging application domains it is often extremely difﬁcult or costly to gather such large labeled training data.
A promising approach to this problem has been via transfer learning which aims at leveraging abundant available labeled data from a related source task to reduce the amount of labeled data required for the target task Pan & Yang (2009);
Weiss et al.
From a practical perspective transfer learning has been rather successful empirically.
In particular, state of the art transfer learning approaches based on pretrained models and ﬁne tuning has led to signiﬁcant improvements on various benchmark datasets.
Despite this empirical success however there is a huge gap between theory and practice in transfer learning and the fundamental limits and beneﬁts of transfer learning are not well understood.
Key challenging questions include: What is an appropriate notion of similarity between different tasks and how can it be quantitatively deﬁned and computed on real data?
What is the best achievable accuracy of any transfer learning algorithm with only a limited number of source and target samples?
How does this accuracy depend on the number of samples and the similarity between the source and target tasks?
While the answer to these challenging questions are still not fully understood, they have indeed attracted a lot of interesting theoretical work in this area Galanti et al.
We will discuss this literature in thorough detail in Section 2.
In this paper, we take a step towards answering the aforementioned key questions enabling a better understanding of the fundamental limits of transfer learning.
We focus on binary classiﬁcations where the goal is to learn a classiﬁer from a hypothesis class with a ﬁnite VC-dimension.
This covers most contemporary classiﬁcation models including the training deep neural networks for binary classiﬁcation.
In this setting, we ﬁrst deﬁne a natural notion of similarity between source and target tasks via the performance of the best source hypothesis on the target task.
Then equipped with this notion of similarity, we derive a statistical minimax lower bound on the target generalization error in terms of the number of labeled data from source and target tasks as well as the VC dimension of the hypothesis class and the similarity between source and target tasks.
Furthermore, we extend this result to the case where there are multiple sources with different similarity to the target.
Our results demonstrate that sources with high similarity to the target are more effective at reducing the target generalization error.
Towards bridging the theory-practice gap in transfer learning we also demonstrate the utility of our theoretical result in concrete applications.
Indeed, a key feature of our result is that our lower bounds can be easily and efﬁciently computed on real data sets and apply to a broad class of practical settings.
In summary our key contributions are as follows:
• We develop a novel statistical minimax lower bound on the generalization error that can be achieved by any transfer learning algorithm as a function of the amount of source and target samples and a natural notion of similarity between source and target tasks.
• A key features of our lower bound (including our notion of similarity) is that it can be easily computed on real world data sets.
Furthermore, our lower bound holds for any source/target distribution and applies with minimal assumptions to a wide variety of contemporary learning models including deep neural networks.
• We investigate the sharpness of our lower bounds and demonstrate their utility via experiments on action recognition and image classiﬁcation.
P RIOR W ORKS
A closely related literature to transfer learning is domain adaptation where there is no or very few labeled target data and the goal is to adapt the hypothesis learned on the source domain to achieve a low target generalization error Chen et al.
; Blitzer et al.
Azizzadenesheli et al.
Long et al.
; Shen et al.
Most of this literature assume that source and target share a common labeling rule but there is a shift in the marginal distributions.
There are many upper bounds for the target generalization error in this setting this setting.
For instance, Ben-David et al.
(2007; 2010) gives an upper bound for the target generalization error in terms of source generalization error and a divergence measure between the domains that can be estimated by ﬁnitely many unlabeled data from the source and target.
In another work Mansour et al.
(2009) introduces a new discrepancy distance and generalizes the results of Ben-David et al.
for a wide family of loss functions using Rademacher complexity.
Similar to this setting, but for multiple source domain adaption scheme, Mansour et al.
(2021) proposes a family of algorithms based on the idea of model selection under the assumption that target distribution is close to some convex combination of sources.
A more recent work Lei et al.
(2021) studies linear regression under shift distribution including covariate shift (i.e. conditional distributions of source and target are the same) as well as model shift (i.e. only distributions of the features of the source and target are the same) and develops algorithms achieving near optimal minimax risk in this setting.
In addition to upper bounds, there are also a few results which provide lower bounds for target generalization error.
David et al.
(2010) provides impossibility results under the assumption of covariate shift and small discrepancy of unlabeled distributions.
Mousavi Kalan et al.
(2020) studies transfer learning with one hidden layer neural networks for regression problems.
This result deﬁnes a notion of similarity between the source and target tasks based on a distance between the ground truth parameters of the source and target networks.
Using this distance this paper develops a statistical minimax lower bound for the target generalization error in terms of the number of source and target samples as well as the deﬁned similarity of the source and target under the distribution shift with the assumption that the features are generated by Gaussian distributions.
Compared to Mousavi Kalan et al.
our result has quite a few unique advantages: (1) We do not assume that the source and target data are generated according to a planted (teacher) network and our results now even hold in the agnostic setting.
(2) Mousavi Kalan et al.
(2020) applies to regression problems but this result covers classiﬁcation (3) Mousavi Kalan et al.
(2020) only considered one-hidden layer neural networks for predicting the labels of extracted features.
In this result we can handle arbitrary deep neural networks.
Our notion of similarity between the source and target distributions can be much more easily estimated by using only a few target data without the need for estimating the ground truth target parameters which requires lots of labeled target data.
More closely related to this work Hanneke & Kpotufe (2019) derives a minimax lower bound for target generalization error in binary classiﬁcation under the assumption of a relaxed version of covariate shift and small transfer exponent parameter which is deﬁned to measure the discrepancy of the source and target distributions.
Our work differs from this previous work as except for assuming the VC dimension of the model is ﬁnite we do not make any further assumptions.
This makes our results applicable in a much broader set of classiﬁcations or decision making problems.
Furthermore, our lower bound can be evaluated on real data sets and serve as a guideline to practitioners helping them decide when utilizing additional knowledge from a source domain is useful for a given target task.
Most of the literature in transfer learning try to provide sufﬁciency and necessity results by deriving upper and lower bounds for target generalization error in a relatively general setting. However, these papers often require a variety of assumptions to ﬁnd the optimal classiﬁer in a target domain in closed form. For instance, Karbalayghareh et al. (2019; 2018) deﬁnes a joint prior distribution of source and target domains using a Wishart distribution which relate the source and target tasks and then makes it possible to study and understand the transferability between domains.
Furthermore, in this setting, the authors develop a closed form optimal Bayesian transfer learning and demonstrate its advantage over a classiﬁer obtained by only target data.
Related to this setting but for regressions,
Karbalayghareh et al.
(2018) obtains the optimal Bayesian transfer learning under setting of joint
Gaussian feature/label distribution.
In contrast with the above in our paper we do not make any assumptions about the distribution of the data.
P ROBLEM F ORMULATION
We consider a transfer learning problem where there are some labeled training data from a source task and a target task with the goal of inferring a hypothesis function with small generalization error in the target task.
More speciﬁcally, we assume have n S and n T source and target labeled data where each training data consists of an input/feature as well as an output/label.
We denote the source and training data by ( x S , y S ) ∼ P and ( x T , y T ) ∼ Q , respectively, where y S , y T ∈ { 0 , 1 } and P , Q are the joint feature-label distributions of source and target data.
Additionally, we assume that source and target features/inputs share a same domain, x S , x T ∈ χ , and H ⊂ 2 χ denotes a ﬁxed hypothesis class with d H VC-dimension.
In transfer learning the goal is to ﬁnd a hypothesis from H that minimizing the target excess risk deﬁned below based on a combination of source and target data.
Deﬁnition 1 (Excess risk)
For a hypothesis function h ∈ H and source and target label-feature data generated according to distributions P and Q ( ( x S , y S ) ∼ P and ( x T , y T ) ∼ Q ), we deﬁne the source and target excess risks as follows
[Equation]
and
[Equation]
S ] − P [ h ∗ S ( x
[Equation]
) (cid:54) = y S ] where h ∗ T = arg min h ∈H Q [ h ( x T ) (cid:54) =
y T ] and h ∗ S = arg min h ∈H P
[ h ( x S ) (cid:54) =
y S ]
Next, we need to deﬁne an appropriate notion of distance between the source and target.
In the literature of domain adaptation, where the conditional expectation remains unchanged and there is only a shift in input distributions, it is common to deﬁne the distance as the error of performance of the best source hypothesis in the target task.
We also deﬁne the distance between source and target as the target excess risk of the best source hypothesis.
Deﬁnition 2 (Transfer distance)
We deﬁne the transfer distance between a source and a target with distributions P and Q as follows
[Equation]
Since we aim to derive a minimax lower bound for transfer learning in binary classiﬁcations, we consider the class of pairs of distributions whose transfer distance is within a ﬁxed number ∆ .
As we will elaborate further in Remark 7 below this notion of distance can be easily estimated/computed in practice.
M AIN R ESULTS
In this section we characterize the fundamental limits of transfer learning in binary classiﬁcations by deriving a minimax lower bound via information-theoretic arguments.
Theorem 1 Consider a transfer learning problem where there are n S and n T number of source as well as target data and the hypothesis class H has
VC dimension
d H obeying d H ≥ 10 .
Furthermore, suppose that ˆ h
=
ˆ h
( S P , S Q ) is an estimated hypothesis for the target task using source and target data in which S P and S Q denote i.i.d.
feature-label data pairs { ( x ( i ) S , y ( i ) S ) } n S i =1 and { ( x ( i ) T , y ( i ) T ) } n T i = generated according to the source and target distributions P and Q .
Fix a transfer distance ∆ < 0 . 99 .
Then for any ˆ h there exists ( P , Q ) with ρ ( P , Q ) ≤ ∆ and a universal constant c such that
[Equation]
where
[Equation]
This also implies that
[Equation]
Remark
The bound above characterizes the fundamental limits of transfer learning by providing a lower bound on the excess risk of any algorithm (regardless of computational tractability) as a function of the number of source and target training data, the similarity/distance between the source and target tasks and the dimension of the hypothesis class used.
Remark
The assumption ∆ < 0 . 99 in the statement of Theorem 1 is just made for simplifying the analysis and the upper bound of 0 .
99 can be replaced by any constant in the interval (0 , 1) .
Remark 3 One can show that the numerical constant c in equation 4.1 obeys c > 3 − 2 √ 2 48 .
Remark 4 (Connection to PAC learning)
We note that the well-known agnostic PAC learning result for a single task gives a lower bound of c · (cid:113)
d H n where n is the number of samples of the task.
Theorem 1 recovers this result when there is not any source task, namely n S = 0 , and the transfer learning problem reduces to learning a task without any prior knowledge from the source.
Remark 5 (Identical source and target)
When the source and target tasks are identical, then the transfer learning problem reduces to learning a single task with n S + n T training data.
Theorem 1, also leads to the same conclusion in this special case as when the source and target data are identical ∆ = 0 and thus (cid:15) =
(cid:113)
d H n S + n T which states that the lower bound is proportional to reciprocal of combination of source and target samples as expected.
Remark 6 (Sharpness in a special case)
We note that the above lower bound is known to be tight in special cases.
For instance when there is a small amount of source data and ∆ is rather large, the lower bound reduces to (cid:113)
d H n T which is known to be tight based on known agnostic PAC learning bounds.
Remark 7 (How to apply Theorem 1 in practical settings.)
In this remark we explain how Theorem 1 can be applied when using contemporary machine learning models involving artiﬁcial neural networks.
In this case, the hypothesis class corresponds to all neural networks with a ﬁxed architecture but different parameters.
It is known that the class of neural networks with a ﬁxed architecture has ﬁnite VC dimension and Harvey et al.
(2017) gives upper and lower bounds for VC dimension of neural networks with ReLU activation functions.
Thus, to apply Theorem 1, one only needs to have an estimate of the transfer distance per Deﬁnition 2.
We note that the transfer distance 3.
consists of two terms: To estimate the ﬁrst term, we note that h ∗ S can be easily estimated due to the abundance of source data in most applications.
Also with an estimate of h ∗ S in hand one can estimate Q [ h ∗ S ( x T ) (cid:54) =
y T ] rather accurately using a simple empirical average with a few target test data as well-known concentration of bounded functions imply that this empirical average is well concentrated around Q [ h ∗ S ( x T ) (cid:54) =
y T ] .
Up on ﬁrst glance it seems that estimating the second term which corresponds to the lowest possible error in the target domain among the hypothesis class, requires a large amount of labeled target data which is not available in a practical problem.
However, in an overparametrized setting, it is typical to assume that there exists a network which achieves very small target generalization error so we can ignore the second term in most practical problems.
Finally we note that as stated earlier the lower bound on the target excess risk gives an estimate of what generalization performance we can expect with a certain number of source and target samples.
Furthermore, by comparing the estimated transfer distance of different pairs of tasks, we can ﬁnd the pairs that are more suitable for transfer learning.
This knowledge can in turn signiﬁcantly reduce the required number of target samples to achieve a certain accuracy.
Next, we extend our result to a multiple source transfer learning setup where instead of only one source task there are several source tasks available and the goal is to transfer knowledge from multiple sources to a given target task to achieve a small target generalization error.
Theorem 2 Suppose that there are n S 1 , n S 2 , ..., n S N number of samples from N source tasks as well as n T number of samples from a target task and the hypothesis class H has VC dimension
d H obeying d H ≥ max ( N + 9 , N/ 2) .
Furthermore, suppose that ˆ h
=
ˆ h
( S P 1 , S P 2 , ..., S P N , S Q ) is an estimated hypothesis for the target task using N sources and target data where S P j and S Q denote i.i.d.
data { ( x ( i ) S j , y ( i ) S j ) } n Sj i =1 and { ( x ( i ) T , y ( i ) T ) } n T
i
=1 generated according to souce and target distributions P j and
Q for j = 1 , ..., N .
Fix transfer distances { ∆ j } Nj =1 where 0 ≤ ∆ j ≤ 1 .
Then for any ˆ h there exists ( P 1 , ..., P M , Q ) with ρ ( P j , Q ) ≤ ∆ j and a universal constant c such that
[Equation]
where
[Equation]
n
[Equation]
This in turn implies that
[Equation]
Remark 8 Similar to the previous theorem, Theorem 2 provides a minimax lower bound for target excess risk with the key distinction that now it applies in the setting where there are multiple source with different transfer distances to the target.
This theorem characterizes the exces risk achievable by any algorithm as a function of these transfer distances as well as the number of samples from the different sources and the target data.
Theorem 2 indicates that the more sources we have, the better performance we can achieve in the target domain.
However, this performance gain maybe marginal for source tasks that have a large transfer distance to the target or where there are very few training data.
In these cases of course it may be more computationally efﬁcient to discard these sources given the marginal improvement in the generalization performance suggested by this theorem.
Remark 9 (Identical sources) if all the source tasks are identical, then there are effectively n S 1 + ... + n S N number of source samples and by Theorem 1 the lower bound would be (cid:115)
[Equation]
Theorem 2 also gives the same order wise lower bound as
[Equation]
Remark 10 (Inﬁnitely many source samples) When ∆ i > 0 and n S i → ∞ , the fraction [Equation] saturates at 1∆ i which shows that when the source and target have positive distance, the source can never compensate for the target samples.
Remark
In the lower bound, the product terms ∆ i n S i appear which indicate that a source with large transfer distance can sometimes be as useful as a source with small transfer distance when there is a large amount of training data available from that source.
E XPERIMENTAL R ESULTS
In this section we evaluate our theoretical results on real data sets for action recognition and image classiﬁcation tasks.
By estimating the parameters appearing in Theorem 1 for different pairs of tasks, we ﬁrst plot the lower bounds and then by running weighted empirical risk minimization investigate the sharpness of the bounds.
We also investigate the effectiveness of different source tasks with different transfer distances on the target generalization error.
5. A CTION R ECOGNITION
Experimental setup.
We ﬁrst perform experiments on the UCF101 action recognition data set.
We pick CricketBowling and TableTennis videos from UCF101 as the target task as well as three different pairs of classes as the source tasks:
1- CricketBowling and BaseballPitch,
2- Cricketshot and
Archery,
3- BasketballDunk and Basketball.
We pass the videos through an i3d network pretrained on kinetics400 Carreira & Zisserman (2017) with the fully connected top classiﬁer removed and extract the corresponding features of dimension 2048 from the raw videos.
We then work with the extracted features instead of the raw videos.
Training.
We train a one hidden layer neural network with 15 number of hidden units and ReLU activation functions for each pair of data sets.
Table 3 consists of test accuracy on CricketBowling vs. TableTennis, when using the network trained on each source task.
We use these accuracies for deriving the corresponding lower bounds.
Furthermore, we run weighted empirical risk minimization as a simple transfer learning approach to ﬁnd some upper bounds on the target generalization error.
Given n S and n T number of source and target samples, for estimating the corresponding one hidden layer neural network parameters we minimize the following weighted empirical risk
[Equation]
where the function Cost denotes the logistic regression cost and λ ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 }
We then pick the lambda which minimizes the target test error.
Results.
First we calculate the transfer distance by Deﬁnition 2 for each source/target pairs using
Table 1.
To this end, we assume that best target generalization error is zero and using the Table 1 we obtain the transfer distance for each pair which is demonstrated in Table 2.
As it can be observed by Table 2, the pair of Source1 and Target has the lowest transfer distance among other pairs since both of the source and target tasks share a same class which is CricketBowling.
Furthermore, Tabledetermines which pairs are more suitable for transferring the source knowledge to the target. [Table] [Table]
[Figure] Next, we draw the lower bound curves for each pair in Fig 1a.
To this end, we need to ﬁnd the
VC dimension of the hypothesis class which consists of neural networks with the architecture of 2048 ∗ 15 ∗ 1 with ReLU activation functions.
Theorem 1 in Harvey et al.
(2017) gives a lower bound for VC dimension of neural networks with ReLU activation functions by 1640 WL log 2 WL where W and L are the number of parameters and layers, respectively.
Then in Figure 1b we plot the lower bounds along with the upper bounds obtained via Formula 5.1 for three different pairs of source and target as well as using only target samples.
We obtained these upper bounds by running Formula 5.
ﬁve times and then averaging the results.
Fig 1b shows that when the distance of a source from the target is small it would be more effective in achieving small target generalization error.
We would like to mention that in all of these plots we choose the same number of source samples for each pair.
Figure 2 shows the average λ , the weight appearing in Formula 5.1, when the number of target samples is 100 to 150 .
It shows that in the pair Source
and Target the average λ is high which demonstrate the usefulness of the source in the target task.
Furthermore, the small value of λ in the pair Source3 and Target suggests that when the transfer distance is high, source samples are no longer usefull.
5. I MAGE C LASSIFICATION
Experimental setup.
In this section we focus on image classiﬁcation tasks and utilize Theorem 1 to recognize appropriate pairs of tasks that are suitable for transfer learning.
We choose some classes of the DomainNet data set Peng et al.
as source and target tasks.
We pick Clock and Ambulance from DomainNet Clipart for the target task and three different pairs of classes as the source tasks: 1- Clock and Ambulance,
2- Cricketshot and TableTennis,
3- TableTennis and FrontCraw. Here we A v e r a g e λ [Figure]
Pair
[Figure]
[Table]
[Figure]
[Table]
use ResNet50 network pretrained on Imagenet for extracting features of dimension 2048 and in the sequel we work with the extracted features rather than the raw image data.
Training.
We train a one hidden layer neural network with 15 number of hidden units and ReLU activation functions for each of pairs of the tasks.
Table 3 includes the test accuracy on the target task when using the networks trained on different sources, which is necessary for estimating/calculating the transfer distance as demonstrated in Table 4.
Similar to the subsection 5.1, we also run weighted empirical risk minimization for ﬁnding upper bounds for the pairs of the source and target tasks.
Results.
Similar to the previous section on action recognition, using Table 3 we can obtain the transfer distances and based on this distance we can identify suitable pairs of source and target tasks [Figure] for transfer learning.
In the pair1 Source and target tasks share the same objects which are Clock and
Ambulance which results in low transfer distance.
In pair2, still one of the objects which is Clock is the same in the source and target and we can see that the transfer distance for pair2 is lower than that for pair3.
Then we plot the lower bounds in Fig 3a and the corresponding upper bounds obtained by weighted empirical risk minimization in Fig 3b.
One can see that sources that are closer to the target according to our notion of distance are more effective in achieving small target generalization error.
CricketBowling is common both in the source and Target1.
This suggests that these tasks are similar to each other and the estimated transfer distance conforms with this intuition.
Furthermore,
CricketBowling and Cricketshot are intuitively similar to one another
and this is also reﬂected in the lower transfer distance between source and Target2.
In Fig 4 we plot the average λ , the weight appearing in Formula 5.1 when the number of target samples varies from 150 to 200.
4 demonstrates that when a source is close to the target the weight of source risk in weighted empirical risk becomes high which shows the effectiveness of source samples in achieving small target generalization error.
P ROOF OUTLINE
The main idea of proof is based on the following proposition proved in Tsybakov (2009)
Proposition 1 [Theorem 2.5 of Tsybakov (2009)] Assume that M ≥ 2 and the function d ( · , · ) is a semi-distance.
Also suppose that { P θ j } θ j ∈ Θ is a family of distributions indexed over a parameter space, Θ , and Θ contains elements θ 0 , θ 1 , ..., θ M such that:
(i)
d
( θ i , θ j ) ≥ 2 s > 0 , ∀ 0 ≤ j < k ≤ M (ii) P j (cid:28) P 0 , ∀ j = 1 , ..., M, , and M M (cid:88)
[Equation]
with 0 < α < 1 / 8 and P j = P θ j , j = 0 , 1 , ..., M and D kl denotes the KL-divergence.
Then
[Equation]
Based on Proposition 1 we construct a family of pairs of distributions, namely source and target distributions, whose transfer distances satisfy the ∆ -constraint.
To do so we pick some points from the domain χ shattered by the hypothesis class and deﬁne appropriate distributions on this set of points.
Furthermore, this family of distributions are indexed in the space of {− 1 , 1 } d which can be a metric space using Hamming distance.
In order to satisfy the condition (i) in Proposition 1, the indexes have to be well separated which can be achieved using the well-known Gilbert-Varshamov’s bound.
Finally we show that estimating a parameter with small hamming distance is equivalent to estimating an appropriate hypothesis with small excess risk error.
A PPENDIX 7. P ROOF OF T HEOREM
We also use the following famous result in information theory known as Gilbert-Varhsamov’s bound for packing argument.
Proposition 2 (Lemma 2.9 of Tsybakov (2009)) Let d ≥ 8 .
Then there exists a subset { w (0) , ..., w ( M ) } of Ω = {− 1 , 1 } d such that w (0) =
(1 , 1 , ..., 1) , dist ( w ( j ) , w ( k ) ) ≥ d 8 , ∀ 0 ≤ j < k ≤ M and M ≥ 2 d/ 8 , where dist ( w, w (cid:48) )
= (cid:80) dk
=
I ( w k (cid:54) = w (cid:48) k ) is the Hamming distance between binary sequences w = ( w 1 , ..., w d ) and w (cid:48)
=
( w (cid:48) 1 , ..., w (cid:48) d ) .
We will also use the following lemma proved in Hanneke & Kpotufe (2019).
We would like to mention that some ideas of the proof are similar to those in Hanneke & Kpotufe (2019).
However, as discussed in section 2, the problem setting of Hanneke & Kpotufe (2019) is different from that of this work which results in constructing a different set of distributions.
Lemma 1 Let 0 < (cid:15) < 1 / 2 and z ∈ {− 1 , 1 } .
Then
[Equation]
Now we are in place to provide the proof of Theorem 1.
Let d = d H − 2 and pick x − 1 , x 0 , ..., x d from χ shattered by H .
Next, we construct a family of pairs of distributions ( P w , Q w ) indexed by w ∈ {− 1 , 1 } d where {− 1 , 1 } d is the parameter space playing the role of Θ in Proposition 1.
For the following, ﬁx (cid:15) =
c 1 · (cid:15) ( n S , n T , d H , ∆) ≤ 12 for some constant c 1 to be determined later in proof and (cid:15) ( n S , n T , d H , ∆) is deﬁned in Theorem 1.
Distribution Q w : Q w is composed of a marginal and a conditional distribution, namely Q w = Q w x × Q wy | x .
We deﬁne the marginaldistributions as follows:
[Equation]
For the conditional distributions:
[Equation]
Distribution P w : P w is composed of a marginal and a conditional distribution, namely P w = P w x × P wy | x .
We deﬁne the marginal distributions as follows:
[Equation]
For the conditional distributions:
[Equation]
Verifying ρ ( P w , Q w ) ≤ ∆ : Bayes classiﬁer of the domain generated by P w is as follows: [Equation] , otherwise [Equation]
x
[Equation]
for i = 1 , ..
[Equation]
d Similarly for the domain generated by Q w , we have
[Equation]
So h ∗ S and h ∗ T disagree only on x − 1 which implies that
[Equation]
Since we want to derive a lower bound for the minimax risk stated in Theorem 1, among the hypotheses that they agree on x i for i = 1 , ..., d , the hypothesis that outputs x − 1 and x 0 asresults in a smaller target error.
Hence, we can restrict ourselves to ˜ H which is the projection of
H onto {− 1 , 1 } d with the constraint that h ( x − 1 )
=
h ( x 0 )
= 1 for all h ∈ ˜ H .
Furthermor, for any w, w (cid:48) ∈ {− 1 , 1 } d we have
[Equation]
when the target domain is generated by Q w
Reduction to a packing : By using Proposition 2, we can get a subset Σ of {− 1 , 1 } d whose cardinality is M ≥ 2 d/ 8 and for any w, w (cid:48) belonging to Σ we have dist ( w, w (cid:48) ) ≥ d/ 8 .
Furthermore, for any w, w (cid:48) ∈ Σ we have
[Equation]
On the other hand, there is a bijective map between {− 1 , 1 } d and elements of ˜ H and any classiﬁer ˆ h : { x i } → { 0 , 1 } with ˆ h ( x − 1 )
=
ˆ h ( x 0 )
= 1 can be reduced to a w ∈ {− 1 , 1 } d .
So we can choose Σ as the set of indices in Proposition 1 with Hamming distance as the semi-metric and the expression
P w ( dist ( ˆ w, w ) > d/ 8) translates into P w ( E T ( h ˆ w )
> c · (cid:15) ) .
KL divergence bound (part (ii) of Proposition 1) : Deﬁne P w = P n S w × Q n T w .
For any w, w (cid:48) ∈ Σ we have
[Equation]
if c 1 < 1 6 then c 0 c 2 1 < {+1 
+} and we can apply Proposition 1.
7. P ROOF OF T HEOREM
Proof of Theorem 2 is similar to that of Theorem 1.
However, we construct different target and source probability distributions.
Let d = d H − N − 1 and pick x − M , ..., x 0 , x 1 , ..., x d from χ shattered by H .
Then we construct a family of distributions ( P (1) w , ..., P ( N ) w , Q w ) indexed by w ∈ {− 1 , 1 } d .
Let (cid:15) =
c 1 · (cid:15) ( n S 1 , ..., n S N , n T , d H , ∆ 1 , ..., ∆ N ) for some constant c 1 < 1 to be determined later in proof.
Furthermore, without loss of generality assume that 1 ≥ ∆ 1 ≥ ∆ 2 ≥ ... ≥ ∆ N ≥ 0 .
Distribution Q w : Q w is composed of a marginal and a conditional distribution, namely Q w = Q w x × Q wy | x .
We deﬁne the marginal distributions as follows:
[Equation]
and
[Equation]
For the conditional distributions:
[Equation]
Distribution P ( i ) w : P ( i ) w is composed of a marginal and a conditional distribution, namely P ( i ) w =
P w x ( i ) × P wy | x ( i ) .
We deﬁne the marginal distributions as follows:
[Equation]
N +(cid:0) 1 − d d + n S i ∆ i (cid:1) for j = 1 , ..., N
P w x ( i ) ( x = x 0 ) =
N +(cid:0) 1 − d d + n S i ∆ i (cid:1)
P w x ( i ) ( x = x j ) =
d + n S i ∆ i for j = 1 , .., d
For the conditional distributions:
[-P wy | x ( y = 1 | x = x − j ) = 
-][Equation]
otherwise
[Equation]
P [Equation] x j [Equation]
[Equation]
for j [Equation] 1 , .., d Verifying ρ ( P ( i ) w , Q w ) ≤ ∆ i :
Bayes classiﬁer of the domain generated by P ( i ) w is as follows: h ∗
[Equation]
{+i ( x − j ) = 
+}
if j ≥ i, otherwise h ∗ S i ( x − j ) = 1 for j = 1 , ..., N h ∗ [Equation] x
[Equation] h ∗ S
i
[Equation]
{+x j ) = 
 if w j = 1 , otherwise h ∗ S i ( x j ) = 0 for j = 1 , .., d Similarly for the domain generated by Q w , we have h ∗ T ( x − j ) = 1 for j = 1 , ..., N h ∗ T ( x 0 ) =+}
h ∗ T ( x j ) = if w j = 1 , otherwise h ∗ T ( x j ) = 0 for j = 1 , .., d
So h ∗ S i and h ∗ T disagree on x − i , .., x − N which implies that
[Equation]
] = [Table] i With the same argument we used in the proof of Theorem 1 we can restrict ourselves to ˜ H which is the projection of H with the constraint that h ( x − N )
= ...
=
h ( x − 1 )
=
h ( x 0 )
= 1 for all h ∈ ˜ H .
The rest of the proof is exactly the same except the part regarding the KL divergence bound.
[Equation]
for small enough c 1 we can apply Proposition
[Equation]
7. A DDITIONAL E XPERIMENTAL R ESULTS
In section 5 we ﬁx number of source samples and vary the number of target samples.
Here in order to investigate the effect of source samples on the target generalization error, we ﬁx the number of target samples at n T = 3 and vary the number of source samples. Fig 5 depicts the theoretical lower bounds along with the upper bounds obtained by empirical risk minimization for image classiﬁcations. We use the same source/target pairs as used in section 5.2. Fig 5 demonstrates that Source1 is more helpful in reducing the target generalization error because it has a low distance from the target.
Furthermore, it shows that increasing the number of source samples is useful up to a point and beyond that point the error saturates and does not decrease further as discussed in Remark 10.