Evaluated CMI Bounds for Meta Learning: Tightness and Expressiveness
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent work has established that the conditional mutual information (CMI) framework of Steinke and Zakynthinou (2020) is expressive enough to capture generalization guarantees in terms of algorithmic stability, VC dimension, and related complexity measures for conventional learning (Harutyunyan et al., 2021, Haghifamet al., 2021).
Hence, it provides a unified method for establishing generalizationbounds.
In meta learning, there has so far been a divide between informationtheoretic results and results from classical learning theory.
In this work, we take afirst step toward bridging this divide.
Specifically, we present novel generalizationbounds for meta learning in terms of the evaluated CMI (e-CMI).
To demonstratethe expressiveness of the e-CMI framework, we apply our bounds to a representation learning setting, with n samples from ˆ n tasks parameterized by functions ofthe form f i ◦ h .
Here, each f i ( · ) is a task-specific function, and h ( · ) is the sharedrepresentation.
For this setup, we show that the e-CMI framework yields a boundthat scales as (cid:112) C ( H ) / ( n ˆ n ) + C ( F ) /n , where C ( · ) denotes a complexity measureof the hypothesis class.
This scaling behavior coincides with the one reported in
Tripuraneni et al.
(2021) using Gaussian complexity.
Introduction
Meta learning, sometimes referred to as learning to learn, is a process by which performance ona new machine learning task is increased by using knowledge acquired from separate, but related,tasks [1, 2].
Concretely, the meta learner ˆ A has access to training data from several different tasks,which are embedded in a common task environment and aims to extract information from this data.
The goal is to use this information to improve the performance of a base learner A on a new taskfrom the same task environment.
For instance, the task environment can consist of different imageclassification tasks, and the goal of the meta learner can be to learn a shared representation for thetasks or to find suitable hyperparameters for the learning algorithm.
As in conventional learning, a central goal in meta learning is to bound the gap between the loss onthe training data and the population risk on unseen data.
The two current approaches for achieving this goal are to: i) use techniques from classical learning theory to obtain minimax performance guarantees, or ii) to use information-theoretic methods to obtain algorithm-, data- and distribution- dependent guarantees.
So far, these two lines of work have evolved largely separately.
In this paper,we take some steps toward unifying them.
Specifically, we: i) derive new, tighter information-theoretic generalization bounds for meta learning, and ii) demonstrate that these bounds are expressive enoughto recover bounds for meta learning from classical learning theory.
To concretize the discussion in this introduction, we assume that the meta learner outputs a member h of a function class H based on n samples from ˆ n different tasks, and that a base learner selects a member f of a class F , basedon the output of the meta learner and n samples from a given task.
Classical learning theory for meta learning.
The theoretical analysis of the benefits of metalearning in terms of loss bounds dates back to [3], where the notion of task environment was formallyintroduced.
More recently, for the setting of representation learning, [4, Thm. 5] derived a risk boundthat scales as
1 (cid:112) C ( H ) / ˆ n + (cid:112) C ( F ) /n , where C ( · ) denotes a complexity measure of the functionclass.
This demonstrates the benefit of meta learning for tasks that share a common environment.
Indeed, in the conventional single-task learning scenario, the n samples from a given task need to beused for learning h and f simultaneously, leading to a (cid:112) C
( H × F ) /n bound.
The bound providedin [4, Thm. 5] was later improved by [5] to a scaling of (cid:112) C ( H ) /
( n ˆ n ) + C ( F ) /n .
This improvedscaling, where C ( H ) decays with the product n ˆ n , confirms the intuition that all of the n ˆ n samplesthat are observed are informative at the environment level.
Meta learning has also been extensivelystudied in several special cases.
For instance, [6, 7, 8] study a setting with linear features and taskmappings, while [9, 10] consider an online convex optimization setting.
In this paper, we will mainlyfocus on the representation learning setting.
Information-theoretic generalization bounds.
For conventional learning, the study of informationtheoretic bounds was initiated by [11, 12], where the average generalization gap of a learningalgorithm is bounded in terms of the information that the algorithm reveals about the training data.
At its heart, this line of work relies on a change of measure technique that relates the training loss tothe population loss.
While the first information-theoretic bounds were given in terms of the mutualinformation between the algorithms output and the full training data, recent work provides boundsin terms of the disintegrated mutual information between the loss that the algorithm incurs on asingle sample pair and a selection variable indicating which sample is used for training, given asupersample containing both the training and test data.
These developments are due to the samplewiseapproach of [13], the disintegration introduced by [14], the evaluated conditional mutual information(e-CMI) notion from [15], and combinations and extensions of these from [16, 17, 18, 19, 20].
Thisline of work is also intimately related to PAC-Bayesian generalization bounds [21, 22], where thegeneralization gap, averaged over the learning algorithm, is bounded with high probability over thedata in terms of a KL divergence.
This is explored further in [23, 24].
Information-theoretic analysis of meta learning.
Recently, information-theoretic generalizationbounds have also been applied to meta learning [25, 26, 27].
In parallel, a PAC-Bayesian analysisof meta learning has also been developed [28, 29, 30, 31, 32, 33].
Generalization bounds obtainedvia information-theoretic methods have also been used as training objectives in order to improveperformance [30, 34].
The quantity of interest in meta learning is the meta-population loss, which isthe population loss evaluated on a task that was not observed during the meta learning phase.
Whilethis quantity is unknown, the meta learner has indirect information about it through the observedmeta-training loss, which is the loss that the meta learner incurs on the training samples from each ofthe observed tasks during the meta learning phase.
The standard approach in the information-theoreticand PAC-Bayesian analysis of meta learning consists of two steps.
The first step involves boundingthe difference between the training loss and a suitably defined auxiliary loss.
The second step involvesbounding the difference between the meta-population loss and the auxiliary loss.
The two naturalcandidates for this auxiliary loss are the population loss of an observed task, and the training loss foran unobserved task.
One of these steps (the first or second, depending on the choice of the auxiliaryloss) is purely at the task level, while the other is purely at the environment level.
This makes itpossible to view each of these steps as a conventional learning problem, so that a standard informationtheoretic generalization bound can be applied.
By the use of the triangle inequality, these two resultsare then combined to obtain a bound on the meta-population loss in terms of the meta-training loss.
We will refer to this procedure as a two-step derivation.
An alternative approach was recently usedby [26], where a one-step procedure was employed.
Rather than relying on an auxiliary loss, [26]immediately bound the difference between the meta-population loss and the meta-training loss interms of a mutual information that captures both task level and environment level dependencies.
The environment and task level dependencies can then be obtained by decomposing this mutualinformation.
The resulting bound turns out to have a better scaling with ˆ n than the two-step bounds.
However, the information-theoretic analyses of meta learning reviewed so far do not provide anyrigorous characterization of the scaling behavior of the bounds.
In particular, the dependence of theinformation measures on the sample size is typically ignored.
This precludes a direct comparisonbetween these information-theoretic bounds and classical learning theory results.
Contributions.
Focusing on the meta learning setup, we present novel information-theoreticbounds based on the e-CMI framework, and demonstrate how to recover minimax results fromclassical learning theory via these bounds.
Our specific contributions are as follows.
In Section 3.1,we derive bounds for the average generalization error in terms of the disintegrated, samplewise e-CMIof the meta learner and base learner: in Theorem 1 and 2, we provide square-root bounds, whichare shown to be tighter than results in the literature; in Theorem 3, we derive novel bounds in termsof the binary KL divergence.
For low values of the training loss, the binary KL bounds display amore favorable dependence on the number of data samples than the square-root bounds.
Next, in
Section 3.2, we extend these average bounds to obtain high-probability generalization guarantees.
This is necessary to perform comparisons with high-probability bounds from classical learningtheory.
Finally, in Section 4, we demonstrate the expressiveness of our bounds by applying them to arepresentation learning setting.
Under certain assumptions about the hypothesis classes, we provideupper bounds on the information measures that appear in our bound in terms of complexity measures.
The results that we obtain via this procedure display a scaling behavior that coincides with the onereported in [5].
This demonstrates that the e-CMI framework is expressive enough to recover thescaling behavior of generalization guarantees for meta learning obtained via classical learning theory.
Problem Setup and Notation
We now introduce the CMI setup for meta learning and the necessary notation for stating our results.
Similar to [3], we consider a task environment formulation that includes the representation learningsetting of [4, 5] as a special case.
Our meta learning setup involves the following quantities.
We consider a task distribution (cid:98) D on thetask space (cid:98) T .
For a given task τ ∈ (cid:98) T , there is a corresponding in-task distribution D τ on the samplespace Z .
The objective of the meta learner is to output a meta hypothesis U ∈ U .
This is done on thebasis of n samples from ˆ n tasks.
Formally, the meta learner is a mapping ˆ A : Z n × ˆ n × (cid:98) R
→ U , wherea random variable
ˆ R ∈ (cid:98) R captures the potential stochasticity of the algorithm.
The goal of the baselearner is to output a hypothesis W ∈ W , given the output of the meta learner and n samples from aspecific task.
Formally, the base learner is a mapping A : Z n × R × U
→ W , where the vector ofrandomnesses for each task is R =
( R 1 , . . .
, R ˆ n ) ∈ R n .
These randomnesses are independent fromthe data, and are assumed to be identically distributed.
Here, the spaces U and W may be functionspaces or parameter spaces, depending on the learning algorithms.
We now describe how the training data is obtained from a meta-supersample, in accordance withthe meta-learning extension in [35] of the conditional mutual information (CMI) framework of [15].
In this setting, the meta-training set is formed as follows.
First, ˆ n tasks are randomly selected froma set of 2ˆ n tasks.
Then, for each task, n samples are randomly selected from a set of 2 n samples.
This setup, as well as our notation, is illustrated in Figure 1 for an example with ˆ n = 2 task pairsand n = 4 sample pairs for each task.
Formally, the meta-supersample ˆ Z can be viewed as a datastructure with 4 n ˆ n elements.
In Figure 1, we illustrate this as an n × 4ˆ n matrix.
We decompose ˆ Zas ( Z 1 , . . .
, Z ˆ n ) , where each element can be seen as a task pair.
Specifically, for i = 1 , . . .
, ˆ n ,the pair Z i can be decomposed as ( Z i, 0 , Z i, 1 ) , where each element is a task-specific supersample.
For k = 0 , 1 , the task-specific super sample is Z i,k =
( Z i,k 1 , . . .
, Z i,k n ) , where each element nowis a pair of data samples.
Specifically, for j = 1 , . .
. , n , each sample pair is Z i,kj =
( Z i,kj, 0 , Z i,kj, 1 ) ,where Z i,kj,l ∈ Z for l = 0 , 1 .
The elements of ˆ Z are generated as follows.
First, for i = 1 , . . .
, ˆ n
[Figure]
   
[Figure]
Sample
[Figure]
ˆ Z =  (cid:34) (cid:35) ˆ S =                   
 
S =
S 1 , ˆ S
Sample membership ˆ S
Meta membership
[Figure]

[Figure]

[Figure]
In this example, the meta-supersample contains the two task pairs Z 1 , which is marked in blue, and Z 2 .
In turn, Z 1 consists of the two in-task supersamples Z 1 , 0 and Z 1 , 1 , which is marked in brown.
Next, Z 1 , 1 consists of the four sample pairs Z 1 , 1 j , where j = 1 , . .
In the image, Z 1 , 1is marked in purple.
Finally, Z 1 , 1consists of a pair of samples, Z 1 , 1 1 , 0 and Z 1 , 1 1 , 1 .
When a binary vector is used as subscript or superscript, this indicates that we should enumerate the meta-supersample according to that vector.
To illustrate this, consider the construction of the meta-training set Z ˆ SS .
The meta-training set is formed on the basis of the meta-subset choice ˆ S and the observed-task subset choice S ˆ S .
For instance, from the first task-pair, ˆ S 1 indicates that we should select task 1 .
Then, from the first sample pair in this task, S 1 , ˆ S 1 indicates that we should select sample 0 , which is Z 1 , 1 1 , 0 , marked in green.
Repeating this for each sample pair in the in-task supersample Z 1 , 1 , we can identify the remaining elements of Z 1 , ˆ S 1 S 1 .
This procedure is performed for each task index until all samples of Z ˆ SS are identified.
The meta-test set Z − ˆ S − S is formed by an analogous procedure, but is now based on − ˆ S and − S − ˆ S .
The entries from the meta-supersample that are selected for Z − ˆ S − S are marked in orange.
For ˆ λ , which contains the losses induced on the meta-supersample, we use an analogous notation.
and k = 0 , 1 , we generate τ i,k ∼ (cid:98) D .
Then, for each j = 1 , . . . , n and k, l = 0 , 1 , we generate
Z i,kj,l ∼ D τ i,k independently of each other.
Next, we describe how the training data is selected from the meta-supersample ˆ Z .
This is doneon the basis of the meta-membership vector ˆ S and the in-task membership vector S .
Specifically,the meta-membership vector ˆ S =
( ˆ S 1 , . . .
, ˆ S ˆ n ) is an ˆ n -dimensional vector, with each elementdrawn independently from a Bern(1 / 2) distribution.
Furthermore, the sample membership vector S =
( S 1 , 0 , S 1 , 1 , . . .
, S ˆ n, 0 , S ˆ n, 1 ) is a collection of 2ˆ n vectors, where each S i,k =
( S i,k 1 , . . .
, S i,kn )is an n -dimensional vector with elements generated independently from a Bern(1 / 2) distribution.
For any Bernoulli matrix X , we let − X denote its elementwise complement modulo 2, i.e., 1 − X ,where 1 is the all-one matrix.
These membership vectors are used to form the meta-training set as follows.
We use the conventionthat, when a binary vector is used as a subscript or superscript of Z , this indicates that we should enumerate over this vector.
Thus, for i = 1 , . . .
, ˆ n and k = 0 , 1 , we let Z i,kS i,k
=
( Z i,k 1 ,S i,k 1 , . . .
, Z i,k n,S i,kn ) .
Here, Z i,kS i,k is the training set for the ( i, k ) th task.
To reduce clutter, we will often use the shorthands Z i,kS i
= Z i,kS i,k and Z i,k 1 ,S ij =
Z i,k 1 ,S i,kj .
In an analogous way, the test set for the ( i, k ) thtask Z i,k − S
i is constructed similarly, but on the basis of − S i,k .
The full meta-training set is givenby
Z ˆ SS =
( Z 1 , ˆ S 1 S 1 , ˆ Si , . . .
, Z ˆ n, ˆ S ˆ n S ˆ n, ˆ S ˆ n ) , where each in-task training set is constructed as indicated above.
Again, the meta-test set Z − ˆ S − S is formed by an analogous procedure.
We now describe how the performance of the meta learner and base learner is evaluated.
We denotethe output of the meta learner as U =
ˆ A ( Z ˆ SS , ˆ R ) , and the output of the base learner for task ( i, k )as W i,k
= A ( Z i,kS i , R i , U ) .
The performance of the learners is evaluated through the use of a lossfunction ℓ : F × Z → [0 , 1] .
We denote the losses that the meta learner and base learner induce onthe meta-supersample ˆ Z by ˆ λ , which inherits the subscript and superscript notation that we describedfor ˆ Z .
Thus, for i = 1 , . . .
, ˆ n , j = 1 , . . . , n , and k, l = 0 , 1 , we have λ i,kj,l = ℓ ( W i,k , z i,kj,l ) .
In otherwords, λ i,kj,l is the loss on the ( j, l ) th sample in the ( i, k ) th task.
On the basis of the loss matrix ˆ λ and the membership vectors ˆ S and S , we can compute four differentlosses.
The main quantity that we are interested in bounding is the average meta-population loss L (cid:98) D ,which is the loss on test data for unobserved tasks.
The quantity that the meta learner has access to isthe average meta-training loss, (cid:98) L , which is the training loss for observed tasks.
The other two lossesare the average auxiliary test loss, ¯ L , which is the loss on test data for observed tasks, and the averageauxiliary training loss (cid:101) L , which is the loss on training data for unobserved tasks.
In the two-stepderivations, one of these two quantities are used as the auxiliary loss.
These four losses are given by
[Equation]
Finally, we end this section by introducing some information-theoretic quantities that appear in ourbounds.
First, let P and Q be two probability measures such that P is absolutely continuous withrespect to Q .
The relative entropy between P and Q is denoted by D ( P || Q ) .
For the special casewhere P and Q are Bernoulli distributions with parameters p and q , we let
[Equation]
p 1 − q (cid:19) .
We refer to d ( p || q ) as the binary KL divergence.
The mutual information between the randomvariables X and Y is given by I ( X ; Y )
=
D ( P XY ||
P X P Y ) , where P XY is their joint distribution and P X and P Y are the corresponding marginals.
The disintegrated mutual informationbetween X and Y given a third random variable Z is given by I Z ( X ; Y ) =
D ( P XY | Z
||
P
X
| Z P Y | Z ) ,where P XY | Z is the joint distribution of X and Y given Z and P X | Z P Y | Z is the product distributionformed from the corresponding marginals.
The expectation over Z of the disintegrated mutualinformation is the conditional mutual information
I ( X ; Y | Z )
=
E Z (cid:2)
I Z ( X ; Y ) (cid:3) .
Generalization Bounds for Meta Learning with e-CMI
In this section, we present generalization bounds in terms of the e-CMI of the meta learner and baselearner.
In Section 3.1, we derive average square-root bounds that tighten results from [26, 35], as wellas novel binary KL bounds.
In Section 3.2, we extend these results to obtain novel, high-probabilityinformation-theoretic bounds for meta learning.
In Section 4, we will demonstrate the expressivenessof the e-CMI framework by using the bounds from this section to recover generalization guaranteesfrom classical learning theory for representation learning.
3.1 Average Bounds
In Theorem 1, we present a square-root bound for the average generalization error obtained through atwo-step derivation.
Specifically, one step consists of bounding the unobserved training loss (cid:101)
L interms of the observed training loss (cid:98) L , and the second step bounds the meta-population loss L (cid:98) D interms of (cid:101) L .
Chaining these two bounds, we obtain a bound on L (cid:98) D in terms of (cid:98) L .
The bound dependson the information captured by two random variables: the task-level variable λ
i, − ˆ S i j , which containsthe training loss and test loss for task ( i, − ˆ S i ) , as well as the environment-level variable λ ij,S ij , whichcontains the training losses for both the observed task ( i, ˆ S i ) and the unobserved task ( i, − ˆ S i ) .
Weprovide the proof of this result in Appendix A, along with the proofs of all other results in this paper.
Theorem 1 (Two-step square-root bound) .
Consider the setup described in Section 2.
Then,
[Equation]
The first term captures the environment-level generalization error while the second term captures thetask-level generalization error.
In order to clarify the relation between Theorem 1 and results fromthe literature, we relax it by upper-bounding the disintegrated individual-sample e-CMI terms bytheir integrated, full-sample, parametric CMI counterparts.
Corollary 1.
Theorem 1 implies that
[Equation]
This recovers the result of [35, Thm. 1], demonstrating that Theorem 4 is tighter.
Next, we present an alternative square-root bound that is obtained through a one-step derivation.
Thisbound depends on the information captured by the random variable λ ij , which contains the trainingand test loss for both the observed and unobserved tasks.
Theorem 2 (One-step square-root bound) .
Consider the setup described in Section 2.
Then,
[Equation]
[Equation]
Again, to compare this bound to results in the literature, we relax it by upper-bounding the disintegrated individual-sample e-CMI terms by their integrated, full-sample, parametric counterparts.
Corollary 2.
Let W i
=
( W i, 0 , W i, 1 ) and ˆ W = { W i } ˆ ni =1 .
Then,
[Equation]
Up to some constant factors, this recovers the result in [26, Thm. 5.1].
Note that, if ˆ A or A aredeterministic learning algorithms with continuous outputs, the mutual information terms in (8) areunbounded.
In contrast, the CMI terms in (7) are always finite.
This is discussed in more detail in [15].
Finally, in Theorem 3, we present two novel bounds in terms of the binary KL divergence.
Theadvantage of these bounds, as compared to the square-root bounds in Theorem 1 and 2, is that for lowtraining losses, they have a more favorable dependence on the number of samples.
We demonstratethis improved rate for representation learning in Section 4.
Theorem 3 (Binary KL bounds) .
For m > 1 , we let
[Equation]
Then,
[Equation]
Furthermore,
[Equation]
Interestingly, (11) provides a bound on the sum of the average meta-population loss L (cid:98) D , the testloss on observed tasks ¯ L , and the training loss on unobserved tasks (cid:101) L .
Due to the nonnegativity ofthe loss, we can obtain an explicit bound on L (cid:98) D by using the lower bound ¯ L, (cid:101) L ≥ 0 , which is asensible relaxation when L (cid:98) D is the dominant term.
By this relaxation, we weaken the bound at mostby a constant factor.
As previously mentioned, the bounds in Theorem 3 can have a more favorabledependence on the number of samples than the square-root bounds in Theorem 1 and 2 when thetraining loss is low.
In the following corollary, we present a bound on L (cid:98) D for the case where (cid:98) L = 0 .
Corollary 3.
Assume that (cid:98) L = 0 .
Then, Theorem 3 implies that
[Equation]
Compared to the bound in Theorem 2, there is no square root in Corollary 3.
As we show in Section 4,this can lead to a faster rate of decay with the number of samples.
3.2 High-probability Bounds
In the previous section, we provided bounds on the average generalization error.
However, metalearning bounds obtained via classical learning theory are typically high-probability bounds [4, 5].
Inorder to assess the expressiveness of the e-CMI framework in terms of its ability to recover theseresults, we now extend the bounds from Section 3.1 to the high-probability setting.
For this, we needsome additional notation.
We let L (cid:98) D ( ˆ Z, ˆ S, S ) and (cid:98) L ( ˆ Z, ˆ S, S ) denote the meta-population loss andtraining loss given that the meta-training set is constructed from ( ˆ Z, ˆ S, S ) .
Specifically,
[Equation]
We now present a high-probability version of the two-step square root bound in Theorem 1.
Tosimplify the presentation, we omit explicit constants and assume that ˆ A is indifferent to the order ofthe data samples.
The theorem statement is provided in more general form in Appendix A.
Theorem 4 (High-probability two-step square-root bound) .
Let ˆ Q 1 λ denote the conditional distributionof λ 1 ,S 1 given ( ˆ Z, ˆ S, S ) , and let ˆ P 1 λ denote E ˆ S (cid:104)
ˆ Q 1 λ (cid:105) .
Furthermore, let Q 1 λ denote the conditionaldistribution of λ 1 , − ˆ S 1 given ( ˆ Z, ˆ S 1 , S 1 , − S 1 ) , and let P 1 λ denote E S 1 , − S 1 (cid:2) Q 1 λ (cid:3) .
Then, there exist constants C 1 , C 2 such that, with probability at least 1 − δ under the draw of ( ˆ Z, ˆ S, S ) ,
[Equation]
The KL divergences in (15) can be interpreted as pointwise e-CMIs.
Indeed,
[Equation]
Finally, we present a high-probability version of the one-step square root bound in Theorem 2.
Theorem 5 (High-probability one-step square-root bound) .
Let ˆ Q λ denote the conditional distributionof
ˆ λ given ( ˆ Z, ˆ S, S ) , and let ˆ P λ denote E ˆ S,S (cid:104)
ˆ Q λ (cid:105) .
Then, with probability at least 1 − δ under thedraw of ( ˆ Z, ˆ S, S ) ,
[Equation]
Again, the KL divergence can be interpreted as a pointwise e-CMI, since
[Equation]
Expressiveness of the Bounds
In Section 3, we presented several new information-theoretic generalization bounds, and demonstratedthat they improve upon known bounds from the literature.
We now turn our focus to the expressivenessof the e-CMI framework.
In particular, we show that the bounds from Section 3 can be used torecover generalization guarantees for meta learning from classical learning theory.
Specifically, we consider the representation learning setting that is analyzed in [5].
We use thefollowing notation.
First, the sample space is split into an instance space and label space as Z = X ×Y .
The aim of the meta learner is to find a representation h U ∈ H , while the base learner outputs atask-specific function f W ∈ F .
Composing these functions leads to the mapping f W ◦ h
U : X → Y .
To obtain explicit minimax bounds, we will assume that H has finite Natarajan dimension
d Nand that F has finite VC dimension d VC .
This allows us to derive bounds on the entropy of therepresentations and predictions that the meta learner and base learner induce on the meta-supersample.
This, in turn, leads to bounds on the e-CMI terms that appear in the bounds in Section 3.
Inthe following corollary, we present the bounds that are obtained by bounding the e-CMI terms in
Theorem 1 and 2.
Corollary 4.
Assume that the Natarajan dimension of H is d N and the VC dimension of F is d VC .
Also, assume that 2 n ≥ d VC + 1 and 2ˆ n ≥ d N + 1 .
Then,
[Equation] [Equation]
Corollary 4 establishes that, for the average setting
[Equation]
we can use the bounds in Theorems 1 and 2 torecover known bounds for function classes with bounded Natarajan and VC dimensions.
Note that, in the upper-bound of (20), we have fully decoupled the complexity of the two function classes.
This ismade possible by the fact that we used (cid:101)
L as the auxiliary loss in the derivation of Theorem 1, ratherthan ¯ L .
We discuss this in more detail in Appendix A.
Next, we consider the interpolating setting, where (cid:98) L = 0 .
Under this assumption, we demonstrate thatwe can achieve a better rate of convergence with respect to the number of training samples.
The result,presented in the following corollary, relies on similarly bounding the e-CMI term in Corollary 3.
Corollary 5.
Assume that the Natarajan dimension of H is d N and the VC dimension of F is d VC . Also, assume that 2 n ≥ d VC + 1 and 2ˆ n ≥ d N + 1 .
Finally, assume that (cid:98) L = 0 .
Then,
[Equation]
The result in Corollary5 demonstrates that, for the interpolating setting, the e-CMI framework isexpressive enough to yield a bound that, ignoring logarithmic factors, decays as 1 / ( n ˆ n ) , oftenreferred to as a fast rate.
Finally, noting that the bounds in [4] and [5] are high-probability rather than average, we also derivehigh-probability generalization bounds.
In order to achieve this, we need probabilistic upper-boundson the KL divergences that appear in Theorem 4 and 5, similar to how the e-CMI terms were boundedfor Corollary 4 and 5.
The resulting bounds are presented in the following corollary.
Corollary 6.
Assume that the Natarajan dimension of H is d N and the VC dimension of F is d VC .
Also, assume that 2 n ≥ d VC + 1 and 2ˆ n ≥ d N + 1 . Then, there exist constants C 1 , C 2 , C 3 such that
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116) [Equation] [Equation]
[Equation] [Equation]
We now see that, suppressing logarithmic factors, the upper bound in (23) scales as (cid:112) C ( H ) / ˆ n + 285 (cid:112)
C ( F ) /n ,
whereas the upper bound in (24) scales as (cid:112) C ( H ) / ( n ˆ n ) + C ( F ) /n .
This matches therates obtained by [4] and [5], respectively, demonstrating that the e-CMI framework combined withthe one-step approach is expressive enough to recover the scaling of these results.
Note that there are some differences between our results and the ones in [4, 5].
First, while thecomplexity measures that we use are related to the Natarajan and VC dimension, [5] consider Gaussiancomplexity.
Furthermore, while we obtain generalization bounds for a generic new task, [5] considerexcess risk bounds for a fixed target task with m training samples.
To handle this, [5] introduces thenotion of task diversity, which is not needed in our analysis.
Furthermore, the bound in [5] containsan extra (cid:112) C ( F ) /m term, which is not present in our bounds.
Conclusions
In this paper, we derived new generalization bounds for meta learning using e-CMI, improving uponinformation-theoretic bounds found in the literature.
By considering a representation learning setting,we demonstrated that e-CMI bounds obtained via a conventional two-step approach lead to rates thatcoincide with those found in [4].
In contrast, we showed that by combining the e-CMI frameworkwith a one-step approach, we recover the more favourable scaling found in [5].
Note that, whilethe bounds in [5] are uniform over the hypothesis class, the information-theoretic bounds that wederive are inherently algorithm- and data-dependent.
As a consequence, they are nonvacuous whenapplied to settings such as classification with deep neural networks [20].
It should also be noted thatthe complexity measures that we consider differ from the Gaussian complexity in [5].
An intriguingtopic for further study is to clarify the connection between e-CMI and Gaussian complexity.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] The contributions section specifies where in the paper each contribution can be found.
(b) Did you describe the limitations of your work?
[Yes] We state the assumptions requiredfor our results in Section 2 and the statements of the results.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] Founda-
tional work, unrelated to a specific application.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We state the assumptions required for our results in Section 2 and the statements of the results.
(b) Did you include complete proofs of all theoretical results?
[Yes] Full proofs areincluded in the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
[N/ A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]