Original: T ASK R ELATEDNESS -B ASED G ENERALIZATION B OUNDS FOR M ETA L EARNING
Revised: T ASK R ELATEDNESS -B ASED G ENERALIZATION B OUNDS FOR M ETA L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Supposing the n training tasks and the new task are sampled from the same environment, traditional meta learning theory derives an error bound on the expected loss over the new task in terms of the empirical training loss, uniformly over the set of all hypothesis spaces.
Revised: Supposing the n training tasks and the new task are sampled from the same environment, traditional meta learning theory derives an error bound on the expected loss over the new task in terms of the empirical training loss, uniformly over the set of all hypothesis spaces.

Original: However, there is still little research on how the relatedness of these tasks can affect the full utilization of all mn training data (with m examples per task).
Revised: However, there is still little research on how the relatedness of these tasks can affect the full utilization of all mn training data (with m examples per task).

Original: In this paper, we propose to address this problem by deﬁning a new notion of task relatedness according to the existence of the bijective transformation between two tasks.
Revised: In this paper, we propose to address this problem by deﬁning a new notion of task relatedness according to the existence of the bijective transformation between two tasks.

Original: A novel generalization bound of O ( 1 √ mn ) for meta learning is thus derived by exploiting the proposed task relatedness.
Revised: A novel generalization bound of O ( 1 √ mn ) for meta learning is thus derived by exploiting the proposed task relatedness.

Original: Moreover, when investigating a special branch of meta learning that involves representation learning with deep neural networks, we establish spectrally-normalized bounds for both classiﬁcation and regression problems.
Revised: Moreover, when investigating a special branch of meta learning that involves representation learning with deep neural networks, we establish spectrally-normalized bounds for both classiﬁcation and regression problems.

Original: Finally, we demonstrate that the relatedness requirement between two tasks is satisﬁed when the sample space possesses the completeness and separability properties, validating the rationality and applicability of our proposed task-relatedness measure.
Revised: Finally, we demonstrate that the relatedness requirement between two tasks is satisﬁed when the sample space possesses the completeness and separability properties, validating the rationality and applicability of our proposed task-relatedness measure.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: By leveraging knowledge distilled from the training tasks 1 , meta learning (Thrun & Pratt, 1998;
Revised: By leveraging knowledge distilled from the training tasks 1 , meta learning (Thrun & Pratt, 1998;

Original: Baxter, 2000) learns to perform well on a new but related task.
Revised: Baxter, 2000) learns to perform well on a new but related task.

Original: One important branch of meta learning achieving great success in practical machine learning applications is representation learning (Krizhevsky et al., 2012; Bengio et al., 2013;
Revised: One important branch of meta learning achieving great success in practical machine learning applications is representation learning (Krizhevsky et al., 2012; Bengio et al., 2013;

Original: He et al., 2016), where one ﬁrst learns a shared feature extractor (e.g., deep neural networks) over the training tasks, and then learns a prediction function for the new task on the top of the features constructed from the extractor, within a few gradient steps (Finn et al., 2017; Li et al., 2017) or with a feedforward process (Snell et al., 2017; Sung et al., 2018) .
Revised: He et al., 2016), where one ﬁrst learns a shared feature extractor (e.g., deep neural networks) over the training tasks, and then learns a prediction function for the new task on the top of the features constructed from the extractor, within a few gradient steps (Finn et al., 2017; Li et al., 2017) or with a feedforward process (Snell et al., 2017; Sung et al., 2018) .

Original: If the feature extractor can capture the common information across tasks, it is possible that utilizing representation learning can help learners generalize well to a new task with much less data.
Revised: If the feature extractor can capture the common information across tasks, it is possible that utilizing representation learning can help learners generalize well to a new task with much less data.

Original: To build up a rigorous framework to support this intuition, the pioneering meta learning theory assumes that both the n training tasks and the new task are i.i.d.
Revised: To build up a rigorous framework to support this intuition, the pioneering meta learning theory assumes that both the n training tasks and the new task are i.i.d.

Original: generated from the same environment (Baxter, 2000).
Revised: generated from the same environment (Baxter, 2000).

Original: Then, analogous to the single task learning whose goal is to select from the hypothesis space H a hypothesis h to achieve the minimal expected loss on the task, meta learning expects to choose from the hypothesis space family H a hypothesis space H ( ∈ H ) that contains a good solution to any task sampled from the environment.
Revised: Then, analogous to the single task learning whose goal is to select from the hypothesis space H a hypothesis h to achieve the minimal expected loss on the task, meta learning expects to choose from the hypothesis space family H a hypothesis space H ( ∈ H ) that contains a good solution to any task sampled from the environment.

Original: Under the PAC learning framework (Valiant, 1984;
Revised: Under the PAC learning framework (Valiant, 1984;

Original: Vapnik, 1989), (Baxter, 2000) derives a uniform bound of O ( (cid:113) C
Revised: Vapnik, 1989), (Baxter, 2000) derives a uniform bound of O ( (cid:113) C

Original: 1 mn + (cid:113) C 2 n ) on the expected loss of a hypothesis space H over the new task in the environment, according to the empirical loss of H over the n training tasks, where C 1 , C 2 are different logarithms of the covering number (Anthony & Bartlett, 2002) about H .
Revised: 1 mn + (cid:113) C 2 n ) on the expected loss of a hypothesis space H over the new task in the environment, according to the empirical loss of H over the n training tasks, where C 1 , C 2 are different logarithms of the covering number (Anthony & Bartlett, 2002) about H .

Original: However, compared with the single task learning whose error bound
Revised: However, compared with the single task learning whose error bound

Original: O ( (cid:113) Cm ) of any h ∈ H can utilize all m training samples, where C represents certain complexity indicators such as the VC-dimension (Blumer et al., 1989) or the entropy (Pollard, 1984) of H , Baxter’s meta learning generalization bound of any H ∈ H cannot fully utilize all mn training samples (e.g., with a bound of O ( 1 √ nm ) , without extra terms of O ( 1 √ n ) or O ( 1 √ m ) ).
Revised: O ( (cid:113) Cm ) of any h ∈ H can utilize all m training samples, where C represents certain complexity indicators such as the VC-dimension (Blumer et al., 1989) or the entropy (Pollard, 1984) of H , Baxter’s meta learning generalization bound of any H ∈ H cannot fully utilize all mn training samples (e.g., with a bound of O ( 1 √ nm ) , without extra terms of O ( 1 √ n ) or O ( 1 √ m ) ).

Original: Nevertheless, there is still little theoretical research on how the relatedness of these tasks can affect the full utilization of all mn training data in meta learning under Baxter’s proposed i.i.d.
Revised: Nevertheless, there is still little theoretical research on how the relatedness of these tasks can affect the full utilization of all mn training data in meta learning under Baxter’s proposed i.i.d.

Original: task environmetn framework.
Revised: task environmetn framework.

Original: In this paper, we propose to address this problem by studying the task relatedness and provide a new generalization bound.
Revised: In this paper, we propose to address this problem by studying the task relatedness and provide a new generalization bound.

Original: To achieve full utilization of all training sample, our motivation is that different tasks need to be ‘related’ enough such that samples from various measures can be assumed to be generated from an ‘almost’ identical distribution.
Revised: To achieve full utilization of all training sample, our motivation is that different tasks need to be ‘related’ enough such that samples from various measures can be assumed to be generated from an ‘almost’ identical distribution.

Original: The equivalence relation of different data distributions actually corresponds to the measure-preserving isomorphism of their induced measure spaces(see Deﬁnition 7).
Revised: The equivalence relation of different data distributions actually corresponds to the measure-preserving isomorphism of their induced measure spaces(see Deﬁnition 7).

Original: Therefore in this paper, we deﬁne a new notion of task relatedness called almost Π -relatedness in Deﬁnition 3 according to the existence of a measure-preserving bijective transformation between two measure spaces associated with different tasks.
Revised: Therefore in this paper, we deﬁne a new notion of task relatedness called almost Π -relatedness in Deﬁnition 3 according to the existence of a measure-preserving bijective transformation between two measure spaces associated with different tasks.

Original: A PAC-style generalization error bound that fully utilize all training data is thus provided in Theorem 3, by exploiting the proposed task relatedness.
Revised: A PAC-style generalization error bound that fully utilize all training data is thus provided in Theorem 3, by exploiting the proposed task relatedness.

Original: We further employ this task relatedness notation to analyze the representation learning with deep neural networks, and establish non-parameter-count-based spectrally-normalized bounds for both classiﬁcation and regression problems under the meta learning framework.
Revised: We further employ this task relatedness notation to analyze the representation learning with deep neural networks, and establish non-parameter-count-based spectrally-normalized bounds for both classiﬁcation and regression problems under the meta learning framework.

Original: Finally, we demonstrate the rationality of our proposed task-relatedness notion from a theoretical standpoint.
Revised: Finally, we demonstrate the rationality of our proposed task-relatedness notion from a theoretical standpoint.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: (1) We propose a new notion of task relatedness, called almost Π -relatedness , by exploring the existence of a bijective transformation between two tasks in the environment.
Revised: (1) We propose a new notion of task relatedness, called almost Π -relatedness , by exploring the existence of a bijective transformation between two tasks in the environment.

Original: A novel PAC-style generalization error bound of O ( (cid:113)
Revised: A novel PAC-style generalization error bound of O ( (cid:113)

Original: Cmn + (cid:113) ln(1 /δ )
Revised: Cmn + (cid:113) ln(1 /δ )

Original: mn ) is thus derived for general meta learning by exploiting the proposed task relatedness, where C captures the logarithm of the covering number of a hypothesis space family, and δ ∈ (0 , 1) denotes the conﬁdence parameter in PAC framework.
Revised: mn ) is thus derived for general meta learning by exploiting the proposed task relatedness, where C captures the logarithm of the covering number of a hypothesis space family, and δ ∈ (0 , 1) denotes the conﬁdence parameter in PAC framework.

Original: (2) For meta learning that involves representation learning, we bound the covering number in Contribution (1) with two covering numbers that are both deﬁned over a single task,
Revised: (2) For meta learning that involves representation learning, we bound the covering number in Contribution (1) with two covering numbers that are both deﬁned over a single task,

Original: making our results suitable to be combined with recent works of deep neural network in the single task learning.
Revised: making our results suitable to be combined with recent works of deep neural network in the single task learning.

Original: In particular, we provide the spectrally-normalized bounds of O ( (cid:113) C 1 + nC 2 nm ) for classiﬁcation and regression problems in meta learning, where C 1 , C 2 are certain complexities that are dependent on the matrix norms, but independent of the total number of the parameters, in deep neural networks.
Revised: In particular, we provide the spectrally-normalized bounds of O ( (cid:113) C 1 + nC 2 nm ) for classiﬁcation and regression problems in meta learning, where C 1 , C 2 are certain complexities that are dependent on the matrix norms, but independent of the total number of the parameters, in deep neural networks.

Original: (3) We rigorously demonstrate that, any two tasks in the environment are almost Π
Revised: (3) We rigorously demonstrate that, any two tasks in the environment are almost Π

Original: -related if the focused sampled space is a complete separable metric space 3 , which validates the applicability of our proposed task-relatedness measure.
Revised: -related if the focused sampled space is a complete separable metric space 3 , which validates the applicability of our proposed task-relatedness measure.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Meta Learning Theory .
Revised: Meta Learning Theory .

Original: The ﬁrst theoretical analysis for meta learning is performed by (Baxter, 2000), which gives a bound on the expected loss on the unseen task of any hypothesis space in terms of the empirical training loss.
Revised: The ﬁrst theoretical analysis for meta learning is performed by (Baxter, 2000), which gives a bound on the expected loss on the unseen task of any hypothesis space in terms of the empirical training loss.

Original: Under this framework, (Pentina & Ben-David, 2015) studies the generalization error bound of the hypothesis space in the case of kernel learning.
Revised: Under this framework, (Pentina & Ben-David, 2015) studies the generalization error bound of the hypothesis space in the case of kernel learning.

Original: Although many follow-up works also assume that all tasks are sampled from the same environment, they bound different kinds of generalization error from various perspectives.
Revised: Although many follow-up works also assume that all tasks are sampled from the same environment, they bound different kinds of generalization error from various perspectives.

Original: One important branch is to bound the transfer risk over the new task of a deterministic procedure, such as the linear feature transformation algorithm
Revised: One important branch is to bound the transfer risk over the new task of a deterministic procedure, such as the linear feature transformation algorithm

Original: (Maurer, 2009), and the Gibbs algorithm in PAC-Bayes theory (Pentina & Lampert, 2014;
Revised: (Maurer, 2009), and the Gibbs algorithm in PAC-Bayes theory (Pentina & Lampert, 2014;

Original: Amit & Meir, 2018).
Revised: Amit & Meir, 2018).

Original: Among them, (Chen et al., 2020) derive a bound of O ( 1 √ n ) from the algorithm stability perspective, and (Pentina & Lampert, 2014) derives a bound of O ( 1 √ n + 1 √ m ) (we may suppress the complexity factor, if there exists, in the numerator for concision, similarly hereinafter).
Revised: Among them, (Chen et al., 2020) derive a bound of O ( 1 √ n ) from the algorithm stability perspective, and (Pentina & Lampert, 2014) derives a bound of O ( 1 √ n + 1 √ m ) (we may suppress the complexity factor, if there exists, in the numerator for concision, similarly hereinafter).

Original: Another branch aims to bound the excess risk of the task speciﬁc function returned by
Revised: Another branch aims to bound the excess risk of the task speciﬁc function returned by

Original: ERM algorithm over the unseen task.
Revised: ERM algorithm over the unseen task.

Original: They study meta learning theory from different views, such as multitask representation learning (Maurer et al., 2016) and online learning (Khodak et al., 2019).
Revised: They study meta learning theory from different views, such as multitask representation learning (Maurer et al., 2016) and online learning (Khodak et al., 2019).

Original: Among them, (Du et al., 2020) and (Tripuraneni et al., 2020) escape from Baxter’s proposed task environment setting, by deﬁning task similarity or task diversity notion, obtaining similar excess risk bounds of O ( 1 √ mn + 1 √ m ) in the case of high-dimensional transfer learning.
Revised: Among them, (Du et al., 2021) and (Tripuraneni et al., 2020) escape from Baxter’s proposed task environment setting, by deﬁning task similarity or task diversity notion, obtaining similar excess risk bounds of O ( 1 √ mn + 1 √ m ) in the case of high-dimensional transfer learning.

Original: Our work follows
Revised: Our work follows

Original: Baxter’s framework and gives a bound on the generalization error of the hypothesis space .
Revised: Baxter’s framework and gives a bound on the generalization error of the hypothesis space .

Original: Although it is hard to make a direct comparison among different bounds as they focus on different kinds of generalization error, we still present the form of these bounds to show their asymptotic properties.
Revised: Although it is hard to make a direct comparison among different bounds as they focus on different kinds of generalization error, we still present the form of these bounds to show their asymptotic properties.

Original: Task Relatedness .
Revised: Task Relatedness .

Original: Another related work (Ben-David & Schuller, 2003) also proposes a taskrelatedness concept, by deﬁning the bijective transformation π on the input space X .
Revised: Another related work (Ben-David & Schuller, 2003) also proposes a taskrelatedness concept, by deﬁning the bijective transformation π on the input space X .

Original: But in our work, the bijective transformation π is imposed on the sample space Z
Revised: But in our work, the bijective transformation π is imposed on the sample space Z

Original: = X
Revised: = X

Original: × Y (where Y is the output space).
Revised: × Y (where Y is the output space).

Original: It is not difﬁcult to see that the task relatedness considered in (Ben-David & Schuller, 2003) is actually a special case of our deﬁned task-relatedness measure.
Revised: It is not difﬁcult to see that the task relatedness considered in (Ben-David & Schuller, 2003) is actually a special case of our deﬁned task-relatedness measure.

Original: We further validate the rationality of our proposed notion by revealing the existence of the bijective transformation π on
Revised: We further validate the rationality of our proposed notion by revealing the existence of the bijective transformation π on

Original: Z when Z is a complete separable metric space, making our results more applicable than that in (Ben-David & Schuller, 2003).
Revised: Z when Z is a complete separable metric space, making our results more applicable than that in (Ben-David & Schuller, 2003).

Original: Besides, (Khodak et al., 2019) and (Du et al., 2020) also consider task-relatedness notions such as parameter-closeness or sharing a low-rank subspace for meta learning.
Revised: Besides, (Khodak et al., 2019) and (Du et al., 2021) also consider task-relatedness notions such as parameter-closeness or sharing a low-rank subspace for meta learning.

Original: To distinguish the notions proposed in this paper with others is truly one of our future directions.
Revised: To distinguish the notions proposed in this paper with others is truly one of our future directions.

Original:  P RELIMINARY
Revised:  P RELIMINARY

Original: In this paper, we use uppercase letters (e.g., Z ) to represent different spaces.
Revised: In this paper, we use uppercase letters (e.g., Z ) to represent different spaces.

Original: The boldface z represents a matrix, with z i as its i -th row and z ; j its j -th column.
Revised: The boldface z represents a matrix, with z i as its i -th row and z ; j its j -th column.

Original: (cid:126)z denotes a vector.
Revised: (cid:126)z denotes a vector.

Original: All proofs of theoretical results are deferred to the supplementary material for reader’s beneﬁts.
Revised: All proofs of theoretical results are deferred to the supplementary material for reader’s beneﬁts.

Original: 3. M ETA L EARNING
Revised: 3. M ETA L EARNING

Original: The formulation of meta learning problem can be summarized as follows.
Revised: The formulation of meta learning problem can be summarized as follows.

Original: We are given a sample space Z
Revised: We are given a sample space Z

Original: = X
Revised: = X

Original: × Y , where X is an input space and Y an output space.
Revised: × Y , where X is an input space and Y an output space.

Original: In this paper, we assume that
Revised: In this paper, we assume that

Original: Z is a complete separable metric space.
Revised: Z is a complete separable metric space.

Original: A loss function is deﬁned as l :
Revised: A loss function is deﬁned as l :

Original: Y × Y → R , and we assume that l has the range [0 , 1] , or equivalently, with rescaling technique, we assume that l is bounded.
Revised: Y × Y → R , and we assume that l has the range [0 , 1] , or equivalently, with rescaling technique, we assume that l is bounded.

Original: An environment ( P , Q ) is a two-tuple, where P is the set of all probability measures \ distributions \ tasks on X × Y and Q is a probability measure on P .
Revised: An environment ( P , Q ) is a two-tuple, where P is the set of all probability measures \ distributions \ tasks on X × Y and Q is a probability measure on P .

Original: A hypothesis space family is formulated as H =
Revised: A hypothesis space family is formulated as H =

Original: {H} , where each H ∈ H is a set of hypotheses \ functions h : X → Y .
Revised: {H} , where each H ∈ H is a set of hypotheses \ functions h : X → Y .

Original: In single task learning, the learner needs to choose an optimal hypothesis h ∗ ∈ H to minimize the expected loss of h over a probability measure P ( ∈ P ) on the product space
Revised: In single task learning, the learner needs to choose an optimal hypothesis h ∗ ∈ H to minimize the expected loss of h over a probability measure P ( ∈ P ) on the product space

Original: X × Y : er P ( h ) =
Revised: X × Y : er P ( h ) =

Original: (cid:82) X
Revised: (cid:82) X

Original: × Y l ( h ( x ) , y )
Revised: × Y l ( h ( x ) , y )

Original: d P ( x, y ) .
Revised: d P ( x, y ) .

Original: Similarly, the goal of meta learning is to ﬁnd a hypothesis space H ∗ ∈ H which contains a good solution to any task sampled from the environment, and minimize the following expected loss of H over a measure Q on P (Baxter, 2000, Eq.(6)): er Q ( H ) =
Revised: Similarly, the goal of meta learning is to ﬁnd a hypothesis space H ∗ ∈ H which contains a good solution to any task sampled from the environment, and minimize the following expected loss of H over a measure Q on P (Baxter, 2000, Eq.(6)): er Q ( H ) =

Original: (cid:82) P inf h ∈H er P ( h )d Q ( P ) .
Revised: (cid:82) P inf h ∈H er P ( h )d Q ( P ) .

Original: In practice, it is hard to minimize er Q ( H ) directly since we do not know the exact distribution of the environment measure Q .
Revised: In practice, it is hard to minimize er Q ( H ) directly since we do not know the exact distribution of the environment measure Q .

Original: We can only capture the information about the environment ( P , Q ) by observing the training data z generated from the n training tasks P j ( j = 1 , ..., n ) that are i.i.d.
Revised: We can only capture the information about the environment ( P , Q ) by observing the training data z generated from the n training tasks P j ( j = 1 , ..., n ) that are i.i.d.

Original: sampled from the environment measure Q .
Revised: sampled from the environment measure Q .

Original: Formally, this is achieved in the following manner: (1) Sample n times from P according to Q to generate (i.i.d.)
Revised: Formally, this is achieved in the following manner: (1) Sample n times from P according to Q to generate (i.i.d.)

Original: probability measures P 1 , ..., P n .
Revised: probability measures P 1 , ..., P n .

Original: (2) Sample m times from X × Y according to P j to generate { ( x 1 j , y 1 j ) , ..., ( x mj , y mj ) } ( 1 ≤ j ≤ n ).
Revised: (2) Sample m times from X × Y according to P j to generate { ( x 1 j , y 1 j ) , ..., ( x mj , y mj ) } ( 1 ≤ j ≤ n ).

Original: Denote z ij = ( x ij , y ij ) ∈ Z , and an ( m, n ) -sample will be generated, denoted by z and written as a matrix z
Revised: Denote z ij = ( x ij , y ij ) ∈ Z , and an ( m, n ) -sample will be generated, denoted by z and written as a matrix z

Original: =
Revised: =

Original: ( z ij ) m × n .
Revised: ( z ij ) m × n .

Original: We then choose to minimize the empirical loss ˆ er z ( H ) over the training data z for meta learning, which is deﬁned as ˆ er z ( H )
Revised: We then choose to minimize the empirical loss ˆ er z ( H ) over the training data z for meta learning, which is deﬁned as ˆ er z ( H )

Original: = 1 n (cid:80) nj =1 inf h ∈H ˆ er z ; j ( h ) , where z ; j is the j -th column of matrix z .
Revised: = 1 n (cid:80) nj =1 inf h ∈H ˆ er z ; j ( h ) , where z ; j is the j -th column of matrix z .

Original: Let P =
Revised: Let P =

Original: ( P 1 , ..., P n ) .
Revised: ( P 1 , ..., P n ) .

Original: We also consider the following loss ˆ er P ( H ) as the empirical estimate of the expected loss er Q ( H ) :
Revised: We also consider the following loss ˆ er P ( H ) as the empirical estimate of the expected loss er Q ( H ) :

Original: ˆ er P ( H ) =
Revised: ˆ er P ( H ) =

Original: 1 n (cid:80) nj =1 inf h ∈H er P j ( h ) .
Revised: 1 n (cid:80) nj =1 inf h ∈H er P j ( h ) .

Original: Deﬁnition 1 (Baxter, 2000)
Revised: Deﬁnition 1 (Baxter, 2000)

Original: For any hypothesis h : X → Y , deﬁne h l as the composition of loss function and hypothesis: h l : X × Y → [0 , 1] by h l ( x, y )
Revised: For any hypothesis h : X → Y , deﬁne h l as the composition of loss function and hypothesis: h l : X × Y → [0 , 1] by h l ( x, y )

Original: =
Revised: =

Original: l ( h ( x ) , y ) .
Revised: l ( h ( x ) , y ) .

Original: For any hypothesis space H ∈ H , deﬁne H l = { h l : h ∈ H} as the function space on Z .
Revised: For any hypothesis space H ∈ H , deﬁne H l = { h l : h ∈ H} as the function space on Z .

Original: For any sequence of n hypothesises ( h 1 , ..., h n ) , deﬁne ( h 1 , ..., h n ) l : ( X × Y ) n → [0 , 1] by ( h 1 , ..., h n ) l ( x 1 , y 1 , ..., x n , y n ) =
Revised: For any sequence of n hypothesises ( h 1 , ..., h n ) , deﬁne ( h 1 , ..., h n ) l : ( X × Y ) n → [0 , 1] by ( h 1 , ..., h n ) l ( x 1 , y 1 , ..., x n , y n ) =

Original: 1 /n (cid:80) ni
Revised: 1 /n (cid:80) ni

Original: =1 l ( h i ( x i ) , y i ) .
Revised: =1 l ( h i ( x i ) , y i ) .

Original: We will use h l to denote ( h 1 , ..., h n ) l .
Revised: We will use h l to denote ( h 1 , ..., h n ) l .

Original: ∀H ∈ H , deﬁne H nl
Revised: ∀H ∈ H , deﬁne H nl

Original: = { ( h 1 , ..., h n ) l : h 1 , ..., h n ∈ H} and H nl = (cid:83) H∈ H H nl .
Revised: = { ( h 1 , ..., h n ) l : h 1 , ..., h n ∈ H} and H nl = (cid:83) H∈ H H nl .

Original: Deﬁne
Revised: Deﬁne

Original: P ( h l ) =
Revised: P ( h l ) =

Original: (cid:82) X × Y h l ( x, y )
Revised: (cid:82) X × Y h l ( x, y )

Original: d P ( x, y ) and P ( H l ) = inf h l ∈H l P ( h l ) where P is a probability measure on Z , and we have P ( h l ) =
Revised: d P ( x, y ) and P ( H l ) = inf h l ∈H l P ( h l ) where P is a probability measure on Z , and we have P ( h l ) =

Original: er P ( h ) and er P ( H ) =
Revised: er P ( h ) and er P ( H ) =

Original: P ( H l ) .
Revised: P ( H l ) .

Original: Deﬁnition 2 Let ( M, d ) be a pseudo-metric 4 space.
Revised: Deﬁnition 2 Let ( M, d ) be a pseudo-metric 4 space.

Original: ∀ (cid:15) > 0 , a subset ˆ T is called an (cid:15) -cover of
Revised: ∀ (cid:15) > 0 , a subset ˆ T is called an (cid:15) -cover of

Original: T ⊆ M if ∀ t ∈ T , ∃ t (cid:48) ∈
Revised: T ⊆ M if ∀ t ∈ T , ∃ t (cid:48) ∈

Original: ˆ T such that d ( t, t (cid:48) ) ≤ (cid:15) .
Revised: ˆ T such that d ( t, t (cid:48) ) ≤ (cid:15) .

Original: Let P =
Revised: Let P =

Original: P 1 × · · · × P n be the product measure on Z n .
Revised: P 1 × · · · × P n be the product measure on Z n .

Original: For any ( H nl (cid:51) )
Revised: For any ( H nl (cid:51) )

Original: h l , h (cid:48)
Revised: h l , h (cid:48)

Original: l : Z n → [0 , 1] , deﬁne the pseudo-metric d P ( h l , h (cid:48) l )
Revised: l : Z n → [0 , 1] , deﬁne the pseudo-metric d P ( h l , h (cid:48) l )

Original: = (cid:82)
Revised: = (cid:82)

Original: Z n | h l ( (cid:126)z ) − h (cid:48) l ( (cid:126)z ) | d P ( (cid:126)z ) .
Revised: Z n | h l ( (cid:126)z ) − h (cid:48) l ( (cid:126)z ) | d P ( (cid:126)z ) .

Original: Then for any (cid:15) > 0 , the covering number N ( (cid:15), H nl , d P ) is deﬁned as min {| T | (cid:12)(cid:12) T is an (cid:15) -cover of H nl under the d P pseudo-metric } , where | T | is the cardinality of T .
Revised: Then for any (cid:15) > 0 , the covering number N ( (cid:15), H nl , d P ) is deﬁned as min {| T | (cid:12)(cid:12) T is an (cid:15) -cover of H nl under the d P pseudo-metric } , where | T | is the cardinality of T .

Original: We hope to use the empirical loss ˆ er z ( H ) or ˆ er P ( H ) to approximate the expected loss er Q ( H ) .
Revised: We hope to use the empirical loss ˆ er z ( H ) or ˆ er P ( H ) to approximate the expected loss er Q ( H ) .

Original: Instead of the traditional absolute value function d ( x, y ) =
Revised: Instead of the traditional absolute value function d ( x, y ) =

Original: | x − y | , we consider the following metric function ( ν > 0 , x, y ≥ 0)
Revised: | x − y | , we consider the following metric function ( ν > 0 , x, y ≥ 0)

Original: d ν [ x, y ] =
Revised: d ν [ x, y ] =

Original: | x − y | x + y + v to measure the distance between x and y , which is ﬁrst used by (Haussler, 1992).
Revised: | x − y | x + y + v to measure the distance between x and y , which is ﬁrst used by (Haussler, 1992).

Original: In Section 4, we will bound the deviation d ν [ ˆ er z ( H ) , er Q ( H )] by bounding d ν [ ˆ er z ( H ) , ˆ er P ( H )] and d ν [ ˆ er P ( H ) ,er Q ( H )] , respectively.
Revised: In Section 4, we will bound the deviation d ν [ ˆ er z ( H ) , er Q ( H )] by bounding d ν [ ˆ er z ( H ) , ˆ er P ( H )] and d ν [ ˆ er P ( H ) ,er Q ( H )] , respectively.

Original: 3. A LMOST Π -R ELATED T ASKS
Revised: 3. A LMOST Π -R ELATED T ASKS

Original: In this section, we ﬁrst propose a new concept of task relatedness, which is called almost Πrelatedness .
Revised: In this section, we ﬁrst propose a new concept of task relatedness, which is called almost Πrelatedness .

Original: This notion can be considered as the extension of the ‘ Π -relatedness’ notion proposed by (Ben-David & Schuller, 2003).
Revised: This notion can be considered as the extension of the ‘ Π -relatedness’ notion proposed by (Ben-David & Schuller, 2003).

Original: The distinctions between our proposed task relatedness notion and that of Ben-David & Schuller (2003) can be found in Section C.2 of Appendix C.
Revised: The distinctions between our proposed task relatedness notion and that of Ben-David & Schuller (2003) can be found in Section C.2 of Appendix C.

Original: Deﬁnition 3 (Almost Π -Related Tasks)
Revised: Deﬁnition 3 (Almost Π -Related Tasks)

Original: Let Π be a set of transformations π : Z → Z and let
Revised: Let Π be a set of transformations π : Z → Z and let

Original: P, P 1 be probability measures on Z = X
Revised: P, P 1 be probability measures on Z = X

Original: × Y .
Revised: × Y .

Original: We say that P, P 1 are almost Π -related probability measures \ tasks, if the following conditions are satisﬁed:
Revised: We say that P, P 1 are almost Π -related probability measures \ tasks, if the following conditions are satisﬁed:

Original: (1) ∃ N, N 1 ⊆
Revised: (1) ∃ N, N 1 ⊆

Original: Z such that P ( N ) =
Revised: Z such that P ( N ) =

Original: P 1 ( N 1 ) = 0 , and (2) ∃ π ∈ Π
Revised: P 1 ( N 1 ) = 0 , and (2) ∃ π ∈ Π

Original: , π is
Revised: , π is

Original: a one-
Revised: a one-

Original: to one
Revised: to one

Original: mapping
Revised: mapping

Original: from ( Z \ N, P ) onto ( Z \ N 1 , P
Revised: from ( Z \ N, P ) onto ( Z \ N 1 , P

Original: ∀
Revised: ∀

Original: A ⊆
Revised: A ⊆

Original: Z \ N , A is P measurable if and only if π ( A ) =
Revised: Z \ N , A is P measurable if and only if π ( A ) =

Original: { π ( x, y ) (cid:12)(cid:12) ( x, y
Revised: { π ( x, y ) (cid:12)(cid:12) ( x, y

Original: ) ∈ A } ⊆ ( Z \ N 1 ) is P 1 -measurable, and (3)
Revised: ) ∈ A } ⊆ ( Z \ N 1 ) is P 1 -measurable, and (3)

Original: (cid:82)
Revised: (cid:82)

Original: Z \ N
Revised: Z \ N

Original: A d P = (cid:82) Z \ N 1 1 π ( A d P 1 , where 1 A is the indicator function on the set A , and (
Revised: A d P = (cid:82) Z \ N 1 1 π ( A d P 1 , where 1 A is the indicator function on the set A , and (

Original: ) the image
Revised: ) the image

Original: π ( N )
Revised: π ( N )

Original: is
Revised: is

Original: a
Revised: a

Original: P measurable set, and the inverse image π
Revised: P measurable set, and the inverse image π

Original: −
Revised: −

Original: 1 ( N is a P -measurable set.
Revised: 1 ( N is a P -measurable set.

Original: Note that condition (4) is a weak requirement only to ensure the measurability as well as the integrability of the mapping π (or π − 1 ) on the measure zero set N (or N 1 ).
Revised: Note that condition (4) is a weak requirement only to ensure the measurability as well as the integrability of the mapping π (or π − 1 ) on the measure zero set N (or N 1 ).

Original: Actually, to validate whether two tasks (say P and P 1 ) are almost Π -related, the most important step is to ﬁnd the bijective transformation π which satisﬁes the listed conditions (1)-(3) in Deﬁnition 3.
Revised: Actually, to validate whether two tasks (say P and P 1 ) are almost Π -related, the most important step is to ﬁnd the bijective transformation π which satisﬁes the listed conditions (1)-(3) in Deﬁnition 3.

Original: Then, we can extend the transformation π to the measure zero set N by deﬁning π ( N ) as a P 1 -measurable set (e.g. a
Revised: Then, we can extend the transformation π to the measure zero set N by deﬁning π ( N ) as a P 1 -measurable set (e.g. a

Original: P 1 -measure zero set) and extend π − 1 to N 1 in a similar way, such that the extended transformation satisﬁes the condition (4).
Revised: P 1 -measure zero set) and extend π − 1 to N 1 in a similar way, such that the extended transformation satisﬁes the condition (4).

Original: The existence of such transformation between two tasks can be theoretically guaranteed by some general topological properties of the given sample space Z , which will be discussed in detail in Section 4.4.
Revised: The existence of such transformation between two tasks can be theoretically guaranteed by some general topological properties of the given sample space Z , which will be discussed in detail in Section 4.4.

Original: We next give a closure property assumption of the space H l ∈ H l when the transformation set Π is imposed on H l .
Revised: We next give a closure property assumption of the space H l ∈ H l when the transformation set Π is imposed on H l .

Original: Similar to (Ben-David & Schuller, 2003) which assumes the closure property of any hypothesis space under the transformation Π , we also assume that the closure condition is satisﬁed by any H l ∈ H l for deriving better generalization bound.
Revised: Similar to (Ben-David & Schuller, 2003) which assumes the closure property of any hypothesis space under the transformation Π , we also assume that the closure condition is satisﬁed by any H l ∈ H l for deriving better generalization bound.

Original: Deﬁnition 4 Let Π be the set of transformations on the complete separable metric space Z =
Revised: Deﬁnition 4 Let Π be the set of transformations on the complete separable metric space Z =

Original: X × Y .
Revised: X × Y .

Original: We say that the function space H l is closed under the transformations of Π , if for any h l ∈ H l , any π ∈ Π , we have the composition function h l ◦ π ∈ H l .
Revised: We say that the function space H l is closed under the transformations of Π , if for any h l ∈ H l , any π ∈ Π , we have the composition function h l ◦ π ∈ H l .

Original: We need to give more explanations to the rationality of the closure property of the function space
Revised: We need to give more explanations to the rationality of the closure property of the function space

Original: H l , as the closure property is a very important assumption to derive our generalization bounds.
Revised: H l , as the closure property is a very important assumption to derive our generalization bounds.

Original: Actually, the almost Π -relatedness deﬁned in Deﬁnition 3 can induce an equivalence relationship between two tasks, according to the existence of one bijective function π ∈ Π .
Revised: Actually, the almost Π -relatedness deﬁned in Deﬁnition 3 can induce an equivalence relationship between two tasks, according to the existence of one bijective function π ∈ Π .

Original: Then this function π can also induce an equivalence relationship between two functions in the space H l .
Revised: Then this function π can also induce an equivalence relationship between two functions in the space H l .

Original: That means, if a function space H l contains a good solution to one task P , then H l should also contain a good solution to the almost Π -related task P 1 of P .
Revised: That means, if a function space H l contains a good solution to one task P , then H l should also contain a good solution to the almost Π -related task P 1 of P .

Original: This is because the two tasks P and P 1 are equivalent to some extent, therefore the function space H l should simultaneously contain the good solutions to these two similar tasks.
Revised: This is because the two tasks P and P 1 are equivalent to some extent, therefore the function space H l should simultaneously contain the good solutions to these two similar tasks.

Original: With such closure property assumption, the complexity of the function space
Revised: With such closure property assumption, the complexity of the function space

Original: H l is related to the degree of relatedness between different tasks from the environment.
Revised: H l is related to the degree of relatedness between different tasks from the environment.

Original: If the tasks in the environment are all similar (e.g., almost Π -related), then it is sufﬁcient for H l to contain one good function as well as its Π -related variants to generalize well to all tasks.
Revised: If the tasks in the environment are all similar (e.g., almost Π -related), then it is sufﬁcient for H l to contain one good function as well as its Π -related variants to generalize well to all tasks.

Original: Such closure property assumption of the function space is also considered in Ben-David & Schuller (2003).
Revised: Such closure property assumption of the function space is also considered in Ben-David & Schuller (2003).

Original: Ben-David &
Revised: Ben-David &

Original: Schuller (2003) have demonstrated that as long as the function space is rich enough to contain some function as well as its equivalent class, then the closure property assumption can be fulﬁlled.
Revised: Schuller (2003) have demonstrated that as long as the function space is rich enough to contain some function as well as its equivalent class, then the closure property assumption can be fulﬁlled.

Original: Thus in this work we assume such basic closure property holds to derive new insights for meta-learning.
Revised: Thus in this work we assume such basic closure property holds to derive new insights for meta-learning.

Original: We introduce another concept induced from the π -related tasks, called almost Π -related environment , and its theoretical properties, which will be used to derive our theoretical analysis in Section 4.
Revised: We introduce another concept induced from the π -related tasks, called almost Π -related environment , and its theoretical properties, which will be used to derive our theoretical analysis in Section 4.

Original: Deﬁnition 5 (Almost Π -related Environment)
Revised: Deﬁnition 5 (Almost Π -related Environment)

Original: In meta learning set up, an environment ( P , Q ) on
Revised: In meta learning set up, an environment ( P , Q ) on

Original: X × Y is called an almost Π -related environment , if there exists a common probability measure
Revised: X × Y is called an almost Π -related environment , if there exists a common probability measure

Original: P on X × Y , such that for any measures P i ∈ P
Revised: P on X × Y , such that for any measures P i ∈ P

Original: ( i ∈ I , I is the index set), P and P i are almost Π -related in the sense of Deﬁnition 3.
Revised: ( i ∈ I , I is the index set), P and P i are almost Π -related in the sense of Deﬁnition 3.

Original: Lemma 1 Let ( P , Q ) be an almost Π -related environment on X × Y , H l a function space on
Revised: Lemma 1 Let ( P , Q ) be an almost Π -related environment on X × Y , H l a function space on

Original: X × Y .
Revised: X × Y .

Original: Let P be the underlying common distribution as deﬁned in Deﬁnition 5, and N be the
Revised: Let P be the underlying common distribution as deﬁned in Deﬁnition 5, and N be the

Original: P -measure zero set as deﬁned in Deﬁnition 3.
Revised: P -measure zero set as deﬁned in Deﬁnition 3.

Original: Then for any P -measurable set
Revised: Then for any P -measurable set

Original: A ⊆
Revised: A ⊆

Original: Z \ N , for any
Revised: Z \ N , for any

Original: P i ∈ P ( i ∈ I ) , h l ∈ H l , we have P ( A ) =
Revised: P i ∈ P ( i ∈ I ) , h l ∈ H l , we have P ( A ) =

Original: P i ( π i ( A )) , P i ( h l ◦ π − 1 i ) =
Revised: P i ( π i ( A )) , P i ( h l ◦ π − 1 i ) =

Original: P ( h l ) , where π i is the corresponding transformation between Π -related tasks P and P i (as deﬁned in Deﬁnition 3).
Revised: P ( h l ) , where π i is the corresponding transformation between Π -related tasks P and P i (as deﬁned in Deﬁnition 3).

Original: Lemma 2 Let ( P , Q ) be an almost Π -related environment on X × Y , P be the common distribution that is almost Π -related to any distribution from P .
Revised: Lemma 2 Let ( P , Q ) be an almost Π -related environment on X × Y , P be the common distribution that is almost Π -related to any distribution from P .

Original: If a function space H l (on X × Y ) is closed under the transformations of Π , then for any probability measure P i ∈ P , er P ( H )
Revised: If a function space H l (on X × Y ) is closed under the transformations of Π , then for any probability measure P i ∈ P , er P ( H )

Original: =
Revised: =

Original: er P i
Revised: er P i

Original: ( H ) .
Revised: ( H ) .

Original: We provide two insights into the theoretical result in Lemma 2 to further explain the motivation of our proposed task relatedness notion: (1) To fully utilize all training samples, two different tasks need to be similar enough so that the hypothesis space which performs well on one of these two tasks can also perform well on another.
Revised: We provide two insights into the theoretical result in Lemma 2 to further explain the motivation of our proposed task relatedness notion: (1) To fully utilize all training samples, two different tasks need to be similar enough so that the hypothesis space which performs well on one of these two tasks can also perform well on another.

Original: In other words, the ’best’ performance of the hypothesis space need to be similar in both tasks, which is the result in above lemma.
Revised: In other words, the ’best’ performance of the hypothesis space need to be similar in both tasks, which is the result in above lemma.

Original: (2) To achieve the goal in (1), the hypothesis need to be large enough to contain good solutions that are invariant to the transformation between different tasks.
Revised: (2) To achieve the goal in (1), the hypothesis need to be large enough to contain good solutions that are invariant to the transformation between different tasks.

Original: Hence, insight (1) motivates us to deﬁne the Π -relatedness notion to measure the similarity between two tasks in Deﬁnition 3, and insight (2) motivates us to assume the closeness property assumption of the hypothesis space in Deﬁnition 4.
Revised: Hence, insight (1) motivates us to deﬁne the Π -relatedness notion to measure the similarity between two tasks in Deﬁnition 3, and insight (2) motivates us to assume the closeness property assumption of the hypothesis space in Deﬁnition 4.

Original:  T HEORETICAL R ESULTS
Revised:  T HEORETICAL R ESULTS

Original: We ﬁrst give a novel bound for meta learning in the almost Π -related environment in Section 4.1.
Revised: We ﬁrst give a novel bound for meta learning in the almost Π -related environment in Section 4.1.

Original: This is particularly achieved by employing the theoretical properties of the almost Π -related environment in Lemma 2.
Revised: This is particularly achieved by employing the theoretical properties of the almost Π -related environment in Lemma 2.

Original: As shown in Theorem 6 in Section 4.4, assuming the environment to be almost Π -related is reasonable when the sample space Z possesses the completeness and separability (which are general topological properties satisﬁed by many metric spaces).
Revised: As shown in Theorem 6 in Section 4.4, assuming the environment to be almost Π -related is reasonable when the sample space Z possesses the completeness and separability (which are general topological properties satisﬁed by many metric spaces).

Original: Section 4.2 derives a covering number bound for representation learning, which is an active area of meta learning.
Revised: Section 4.2 derives a covering number bound for representation learning, which is an active area of meta learning.

Original: In
Revised: In

Original: Section 4.3, by bounding the covering number with the Lipschitz constants of the matrix and nonlinearity in each layer of the deep neural network, we further establish a spectrally-normalized bound that is independent on the number of the parameters in neural network in Theorem 5.
Revised: Section 4.3, by bounding the covering number with the Lipschitz constants of the matrix and nonlinearity in each layer of the deep neural network, we further establish a spectrally-normalized bound that is independent on the number of the parameters in neural network in Theorem 5.

Original: Moreover, we apply this theoretical result to analyze several practical scenarios like binary \ multiclass classiﬁcation and regression problems under the meta learning framework.
Revised: Moreover, we apply this theoretical result to analyze several practical scenarios like binary \ multiclass classiﬁcation and regression problems under the meta learning framework.

Original: 4. A C OVERING N UMBER B OUND FOR M ETA L EARNING IN A LMOST Π -R ELATED E NVIRONMENT
Revised: 4. A C OVERING N UMBER B OUND FOR M ETA L EARNING IN A LMOST Π -R ELATED E NVIRONMENT

Original: To bound | ˆ er z ( H ) − er Q ( H ) | , we choose to bound the deviation d ν [ ˆ er z ( H ) , ˆ er P ( H )] and d ν [ ˆ er P ( H ) ,er Q ( H )] , respectively.
Revised: To bound | ˆ er z ( H ) − er Q ( H ) | , we choose to bound the deviation d ν [ ˆ er z ( H ) , ˆ er P ( H )] and d ν [ ˆ er P ( H ) ,er Q ( H )] , respectively.

Original: We ﬁrst give an explicit PAC-style generalization bound on d ν [ ˆ er z ( H ) , ˆ er P ( H )] , which can be considered as an inference of Theorem 18 in (Baxter, 2000).
Revised: We ﬁrst give an explicit PAC-style generalization bound on d ν [ ˆ er z ( H ) , ˆ er P ( H )] , which can be considered as an inference of Theorem 18 in (Baxter, 2000).

Original: Theorem 1 Let H nl ⊆
Revised: Theorem 1 Let H nl ⊆

Original: H l ⊕ · · · ⊕ H l be the permissible 5 class of functions mapping ( X × Y ) n into [0,1], where H l = { h l : h ∈ H : H ∈ H } , ⊕ means direct product.
Revised: H l ⊕ · · · ⊕ H l be the permissible 5 class of functions mapping ( X × Y ) n into [0,1], where H l = { h l : h ∈ H : H ∈ H } , ⊕ means direct product.

Original: Let z be generated by m ≥ 2 / ( ν(cid:15) 2 ) independent trials from ( X × Y ) n according to the product measure P = P 1 × · · · × P n .
Revised: Let z be generated by m ≥ 2 / ( ν(cid:15) 2 ) independent trials from ( X × Y ) n according to the product measure P = P 1 × · · · × P n .

Original: For any ν > 0 , 0 < (cid:15) < 1 , for any H ∈ H , with probability at least 1 − δ over z , we have
Revised: For any ν > 0 , 0 < (cid:15) < 1 , for any H ∈ H , with probability at least 1 − δ over z , we have

Original: [Equation]
Revised: [Equation]

Original: where ¯ z ∈ Z (2 m × n ) , the top half of ¯ z is actually z itself and the bottom half of ¯ z is the copy of z .
Revised: where ¯ z ∈ Z (2 m × n ) , the top half of ¯ z is actually z itself and the bottom half of ¯ z is the copy of z .

Original: ∀ h l , h
Revised: ∀ h l , h

Original: [Equation]
Revised: [Equation]

Original: l ∈ H nl , d ¯
Revised: l ∈ H nl , d ¯

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: On the other hand, we can actually bound the deviation d ν [ ˆ er P ( H ) ,er Q ( H )] as follow, by using the theoretical properties of the almost Π -related environment in Lemma 2.
Revised: On the other hand, we can actually bound the deviation d ν [ ˆ er P ( H ) ,er Q ( H )] as follow, by using the theoretical properties of the almost Π -related environment in Lemma 2.

Original: Theorem 2 Let ( P , Q ) be an almost Π -related environment on the complete separable metric space
Revised: Theorem 2 Let ( P , Q ) be an almost Π -related environment on the complete separable metric space

Original: Z = X
Revised: Z = X

Original: × Y .
Revised: × Y .

Original: Let P ∈ P n be generated by n independent trials from P according to some product probability measure Q n .
Revised: Let P ∈ P n be generated by n independent trials from P according to some product probability measure Q n .

Original: Then for any ν > 0 , any H ∈ H , we have d ν [ ˆ er P ( H ) , er Q ( H )]
Revised: Then for any ν > 0 , any H ∈ H , we have d ν [ ˆ er P ( H ) , er Q ( H )]

Original: = 0 .
Revised: = 0 .

Original: Combining Theorem 1 and the Theorem 2, we can obtain a novel generalization bound of convergence rate O ( 1 √ nm ) on | ˆ er z ( H ) − er Q ( H ) | that fully utilizes all mn training data under the proposed meta learning framework in (Baxter, 2000), uniformly over the set of all hypothesis space H .
Revised: Combining Theorem 1 and the Theorem 2, we can obtain a novel generalization bound of convergence rate O ( 1 √ nm ) on | ˆ er z ( H ) − er Q ( H ) | that fully utilizes all mn training data under the proposed meta learning framework in (Baxter, 2000), uniformly over the set of all hypothesis space H .

Original: Theorem 3 Let ( P , Q ) be an almost Π -related environment on the complete separable metric space
Revised: Theorem 3 Let ( P , Q ) be an almost Π -related environment on the complete separable metric space

Original: Z = X
Revised: Z = X

Original: × Y .
Revised: × Y .

Original: Let z be an ( m, n ) -sample generated by the process described in Section 3.1.
Revised: Let z be an ( m, n ) -sample generated by the process described in Section 3.1.

Original: Let H =
Revised: Let H =

Original: {H} be any permissible hypothesis space family.
Revised: {H} be any permissible hypothesis space family.

Original: Then for any H ∈ H , (cid:15) ∈ (0 , 1) , with probability at least 1 − δ over z , we have
Revised: Then for any H ∈ H , (cid:15) ∈ (0 , 1) , with probability at least 1 − δ over z , we have

Original: [Equation]
Revised: [Equation]

Original: Combining the above theorem and Lemma 2, we subsequently give a corollary which reveals the relationship between task-related meta learning setting and i.i.d.
Revised: Combining the above theorem and Lemma 2, we subsequently give a corollary which reveals the relationship between task-related meta learning setting and i.i.d.

Original: single task learning setting.
Revised: single task learning setting.

Original: Corollary 

Revised: Corollary

Original: In the setting of Theorem 3, let P n +1 be the new task sampled from the environment.
Revised: In the setting of Theorem 3, let P n +1 be the new task sampled from the environment.

Original: Then for any H ∈ H , (cid:15) ∈ (0 , 1) , with probability at least 1 − δ over z , we have
Revised: Then for any H ∈ H , (cid:15) ∈ (0 , 1) , with probability at least 1 − δ over z , we have

Original: [Equation]
Revised: [Equation]

Original: There are two important points to note about Corollary 1: (1) If the hypothesis space H contains only one hypothesis and all probability measures in the environment are the same, then the we can choose the bijective transformation π between different distributions as the identity transformation, and the result in Corollary 1 degrades to the traditional covering number based generalization bound in single task learning (see Chapter 10 in (Anthony &
Revised: There are two important points to note about Corollary 1: (1) If the hypothesis space H contains only one hypothesis and all probability measures in the environment are the same, then the we can choose the bijective transformation π between different distributions as the identity transformation, and the result in Corollary 1 degrades to the traditional covering number based generalization bound in single task learning (see Chapter 10 in (Anthony &

Original: Bartlett, 2002)), which utilizes all mn i.i.d.
Revised: Bartlett, 2002)), which utilizes all mn i.i.d.

Original: training data.
Revised: training data.

Original: In this sense, meta learning theory can be considered as the extension of the conventional single task learning theory.
Revised: In this sense, meta learning theory can be considered as the extension of the conventional single task learning theory.

Original: (2) When bounding the generalization error of a hypothesis space H under Baxter’s proposed meta learning framework, even though different tasks may have different distributions, we can still properly estimate the expected loss on the new task according to the empirical loss on the training tasks, and fully leverage all mn (not necessarily i.i.d.) training data with the use of assumed task relatedness in the environment.
Revised: (2) When bounding the generalization error of a hypothesis space H under Baxter’s proposed meta learning framework, even though different tasks may have different distributions, we can still properly estimate the expected loss on the new task according to the empirical loss on the training tasks, and fully leverage all mn (not necessarily i.i.d.) training data with the use of assumed task relatedness in the environment.

Original: Actually, the relatedness condition can be satisﬁed by tasks that are focused on the complete separable sample space (see more explanations in Section 4.4).
Revised: Actually, the relatedness condition can be satisﬁed by tasks that are focused on the complete separable sample space (see more explanations in Section 4.4).

Original: 4. C OVERING N UMBER M ETA L EARNING B OUNDS WITH R EPRESENTATION L EARNING
Revised: 4. C OVERING N UMBER M ETA L EARNING B OUNDS WITH R EPRESENTATION L EARNING

Original: In single-task learning setup, representation learning aims to ﬁnd a good feature embedding f ∈ F (e.g., deep neural network) which maps the input space X into the feature space V .
Revised: In single-task learning setup, representation learning aims to ﬁnd a good feature embedding f ∈ F (e.g., deep neural network) which maps the input space X into the feature space V .

Original: A prediction function g ∈ G then projects the feature space V into the output space Y .
Revised: A prediction function g ∈ G then projects the feature space V into the output space Y .

Original: Hence, the hypothesis space can be written as H =
Revised: Hence, the hypothesis space can be written as H =

Original: G ◦ f , ( f ∈ F ) .
Revised: G ◦ f , ( f ∈ F ) .

Original: Further, let H l
Revised: Further, let H l

Original: =
Revised: =

Original: {H l } , each H l = G l ◦ f , f ∈ F , where G l = { g l } , g l = l ◦ g is the composition function of the prediction function g and the loss function l .
Revised: {H l } , each H l = G l ◦ f , f ∈ F , where G l = { g l } , g l = l ◦ g is the composition function of the prediction function g and the loss function l .

Original: We will omit the subscript l of g l ∈ G l for simplicity where the context is clear.
Revised: We will omit the subscript l of g l ∈ G l for simplicity where the context is clear.

Original: In meta learning setup, representation learning chooses to learn a common feature embedding across n training tasks and learn n task-speciﬁc functions respectively for n tasks, so we can denote the hypothesis space family H nl
Revised: In meta learning setup, representation learning chooses to learn a common feature embedding across n training tasks and learn n task-speciﬁc functions respectively for n tasks, so we can denote the hypothesis space family H nl

Original: = { g 1 ◦ f ⊕ · · · ⊕ g n ◦ f : g 1 , ..., g n ∈ G l , f ∈ F} = { g 1 ⊕ · · · ⊕ g n ◦ ¯ f : g 1 ⊕ · · · ⊕ g n ∈ G nl , ¯ f ∈ F} =
Revised: = { g 1 ◦ f ⊕ · · · ⊕ g n ◦ f : g 1 , ..., g n ∈ G l , f ∈ F} = { g 1 ⊕ · · · ⊕ g n ◦ ¯ f : g 1 ⊕ · · · ⊕ g n ∈ G nl , ¯ f ∈ F} =

Original: G nl ◦ F , by deﬁning F (cid:51) ¯ f : ( X × Y ) n
Revised: G nl ◦ F , by deﬁning F (cid:51) ¯ f : ( X × Y ) n

Original: → ( V × Y ) n with ¯ f ( x 1 , y 1 , ..., x n , y n ) =
Revised: → ( V × Y ) n with ¯ f ( x 1 , y 1 , ..., x n , y n ) =

Original: ( f ( x 1 ) , y 1 , · · · , f ( x n ) , y n ) .
Revised: ( f ( x 1 ) , y 1 , · · · , f ( x n ) , y n ) .

Original: We then deﬁne two pseudo-metrics over feature embedding space F and prediction function space G nl respectively.
Revised: We then deﬁne two pseudo-metrics over feature embedding space F and prediction function space G nl respectively.

Original: Deﬁnition 6 ∀ ¯ z
Revised: Deﬁnition 6 ∀ ¯ z

Original: =
Revised: =

Original: ( z ij ) 2 m × n ∈ Z
Revised: ( z ij ) 2 m × n ∈ Z

Original: (2 m × n ) , deﬁne the empirical measure P ¯ z on Z n which puts point mass 1 / 2 m on each row ¯ z i , i = 1 , ..., 2 m .
Revised: (2 m × n ) , deﬁne the empirical measure P ¯ z on Z n which puts point mass 1 / 2 m on each row ¯ z i , i = 1 , ..., 2 m .

Original: ∀ s = (cid:0) ( f ( x ij ) , y ij ) (cid:1) 2 m × n = (cid:0) ( v ij , y ij ) (cid:1) 2 m × n , where f ∈ F , deﬁne an empirical measure P s which puts mass 1 / 2 m on each row s i , i = 1 , ..., 2 m .
Revised: ∀ s = (cid:0) ( f ( x ij ) , y ij ) (cid:1) 2 m × n = (cid:0) ( v ij , y ij ) (cid:1) 2 m × n , where f ∈ F , deﬁne an empirical measure P s which puts mass 1 / 2 m on each row s i , i = 1 , ..., 2 m .

Original: ∀ ¯ f, ¯ f (cid:48) ∈ F , deﬁne the pseudo-metric d
Revised: ∀ ¯ f, ¯ f (cid:48) ∈ F , deﬁne the pseudo-metric d

Original: [ P ¯ z , G nl ] ( ¯ f, ¯ f (cid:48) )
Revised: [ P ¯ z , G nl ] ( ¯ f, ¯ f (cid:48) )

Original: = 1 / 2 m (cid:80) 2 mi =1 sup g
Revised: = 1 / 2 m (cid:80) 2 mi =1 sup g

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: f (¯ z i ) − g ◦ ¯ f (cid:48) (¯ z i ) | , and ∀ g, g (cid:48) ∈
Revised: f (¯ z i ) − g ◦ ¯ f (cid:48) (¯ z i ) | , and ∀ g, g (cid:48) ∈

Original: G nl , deﬁne the pseudo-metric d P s ( g, g (cid:48) )
Revised: G nl , deﬁne the pseudo-metric d P s ( g, g (cid:48) )

Original: = 1 / 2 m (cid:80) 2 mi =1 | g ( s i ) − g (cid:48)
Revised: = 1 / 2 m (cid:80) 2 mi =1 | g ( s i ) − g (cid:48)

Original: ( s i ) | .
Revised: ( s i ) | .

Original: Then the following two propositions bound the covering number of the hypothesis space family H nl with two covering numbers that are both deﬁned over the single task.
Revised: Then the following two propositions bound the covering number of the hypothesis space family H nl with two covering numbers that are both deﬁned over the single task.

Original: Proposition 1 Let ( P , Q ) be an environment on V × Y .
Revised: Proposition 1 Let ( P , Q ) be an environment on V × Y .

Original: Denote N ( (cid:15), G nl , 2 m ) =
Revised: Denote N ( (cid:15), G nl , 2 m ) =

Original: sup P ∼ ¯ Q n sup s ∼ P 2 m N ( (cid:15), G nl , d P s ) .
Revised: sup P ∼ ¯ Q n sup s ∼ P 2 m N ( (cid:15), G nl , d P s ) .

Original: Then for any (cid:15) = (cid:15) 1 + (cid:15) 2 ,
Revised: Then for any (cid:15) = (cid:15) 1 + (cid:15) 2 ,

Original: any ¯ z ∈ Z 2 m × n , we have
Revised: any ¯ z ∈ Z 2 m × n , we have

Original: N ( (cid:15) 1 + (cid:15) 2 , H nl , d ¯ z ) ≤ N ( (cid:15) 1 , F , d [ P ¯ z , G nl ] ) N ( (cid:15) 2 , G nl , 2 m ) .
Revised: N ( (cid:15) 1 + (cid:15) 2 , H nl , d ¯ z ) ≤ N ( (cid:15) 1 , F , d [ P ¯ z , G nl ] ) N ( (cid:15) 2 , G nl , 2 m ) .

Original: Proposition 2 Let ¯ z ; j be the j -th column of the data matrix ¯ z .
Revised: Proposition 2 Let ¯ z ; j be the j -th column of the data matrix ¯ z .

Original: Let N ( (cid:15), G l , 2 m ) =
Revised: Let N ( (cid:15), G l , 2 m ) =

Original: sup P ∼ ¯ Q sup (cid:126)s ∼ P 2 m N ( (cid:15), G l , d P (cid:126)s ) , where d
Revised: sup P ∼ ¯ Q sup (cid:126)s ∼ P 2 m N ( (cid:15), G l , d P (cid:126)s ) , where d

Original: [Equation]
Revised: [Equation]

Original: (cid:126)s = ( s 1 , ..., s 2 m ) .
Revised: (cid:126)s = ( s 1 , ..., s 2 m ) .

Original: Then for any (cid:15) > 0 , we have N ( (cid:15), F , d [ P ¯ z , G nl ] ) ≤ max 1 ≤ j ≤ n N ( (cid:15), F , d
Revised: Then for any (cid:15) > 0 , we have N ( (cid:15), F , d [ P ¯ z , G nl ] ) ≤ max 1 ≤ j ≤ n N ( (cid:15), F , d

Original: [ P ¯ z ; j , G l ] ) , and N ( (cid:15), G nl , 2 m ) ≤ N ( (cid:15), G l , 2 m ) n .
Revised: [ P ¯ z ; j , G l ] ) , and N ( (cid:15), G nl , 2 m ) ≤ N ( (cid:15), G l , 2 m ) n .

Original: Recalling Theorem 3 and Propositions 1-2, we can further establish the following covering number bound for meta learning with the representation learning.
Revised: Recalling Theorem 3 and Propositions 1-2, we can further establish the following covering number bound for meta learning with the representation learning.

Original: Theorem 4 Let ( P , Q ) be an almost Π -related environment on the complete separable metric space
Revised: Theorem 4 Let ( P , Q ) be an almost Π -related environment on the complete separable metric space

Original: Z = X
Revised: Z = X

Original: × Y .
Revised: × Y .

Original: Let H =
Revised: Let H =

Original: {H} be the set of hypothesis spaces of the form H =
Revised: {H} be the set of hypothesis spaces of the form H =

Original: G ◦ f , f ∈ F .
Revised: G ◦ f , f ∈ F .

Original: Then for any H ∈ H , any 0 < (cid:15) < 1 , with probability at least 1 − δ over z , we have
Revised: Then for any H ∈ H , any 0 < (cid:15) < 1 , with probability at least 1 − δ over z , we have

Original: [Equation]
Revised: [Equation]

Original: We need to highlight the important role of Theorem 4: the covering number of the hypothesis space family H nl = G nl ◦ F for meta learning (over n tasks) in Theorem 3 is converted into the multiplication of the covering number of the class F of the feature embeddings (over one task) and the n -th power of the covering number of the class G l of the prediction functions (over one task).
Revised: We need to highlight the important role of Theorem 4: the covering number of the hypothesis space family H nl = G nl ◦ F for meta learning (over n tasks) in Theorem 3 is converted into the multiplication of the covering number of the class F of the feature embeddings (over one task) and the n -th power of the covering number of the class G l of the prediction functions (over one task).

Original: This means that we can introduce recent covering number based theoretical results of deep neural network from single task learning into meta learning.
Revised: This means that we can introduce recent covering number based theoretical results of deep neural network from single task learning into meta learning.

Original: In particular, by bounding the covering number in Theorem 4 with the Lipschitz constants of the function in each layer of the deep neural network, we can achieve non-parameter-count-based bounds (or say norm-constraint-based bounds (Neyshabur et al., 2017)) for meta learning, which will be detailed in the next section.
Revised: In particular, by bounding the covering number in Theorem 4 with the Lipschitz constants of the function in each layer of the deep neural network, we can achieve non-parameter-count-based bounds (or say norm-constraint-based bounds (Neyshabur et al., 2017)) for meta learning, which will be detailed in the next section.

Original: 4. S PECTRALLY -N ORMALIZED B OUNDS FOR M ETA L EARNING WITH N EURAL N ETWORK
Revised: 4. S PECTRALLY -N ORMALIZED B OUNDS FOR M ETA L EARNING WITH N EURAL N ETWORK

Original: Based on the theoretical results obtained in Section 4.2, and the spectrally-normalized bounds for neural networks in the single task learning (Bartlett et al., 2017; Neyshabur et al., 2018), in this section we aim to derive spectrally-normalized bounds for meta learning with the deep neural network.
Revised: Based on the theoretical results obtained in Section 4.2, and the spectrally-normalized bounds for neural networks in the single task learning (Bartlett et al., 2017; Neyshabur et al., 2018), in this section we aim to derive spectrally-normalized bounds for meta learning with the deep neural network.

Original: We consider the L -layer depth fully-connected networks with nonlinearities (e.g., activation functions, pooling operators) for each layer, which computes an embedding function f : R d 0 → R d , where d 0 , d are the dimension of input data and embedded feature, respectively.
Revised: We consider the L -layer depth fully-connected networks with nonlinearities (e.g., activation functions, pooling operators) for each layer, which computes an embedding function f : R d 0 → R d , where d 0 , d are the dimension of input data and embedded feature, respectively.

Original: Each layer of the network has a weight matrix
Revised: Each layer of the network has a weight matrix

Original: A i and a ρ i -Lipschitz nonlinearity σ i , with σ i (0) = 0 .
Revised: A i and a ρ i -Lipschitz nonlinearity σ i , with σ i (0) = 0 .

Original: Then the composition function is given as σ i ◦
Revised: Then the composition function is given as σ i ◦

Original: A i : R d i − 1 → R d i , ∀ i ∈
Revised: A i : R d i − 1 → R d i , ∀ i ∈

Original: [ L ] .
Revised: [ L ] .

Original: For any A
Revised: For any A

Original: = ( A 1 , · · · , A L ) , any input data x ∈ X , deﬁne f A ( x )
Revised: = ( A 1 , · · · , A L ) , any input data x ∈ X , deﬁne f A ( x )

Original: =
Revised: =

Original: σ L ( A L σ L − 1 ( A L − 1 · · · σ 1 ( A 1 x ) · · · )) ∈ R d .
Revised: σ L ( A L σ L − 1 ( A L − 1 · · · σ 1 ( A 1 x ) · · · )) ∈ R d .

Original: Let F denote the class of functions computed by the corresponding networks, and D the maximum of { d 0 , d 1 , ..., d L − 1 , d } .
Revised: Let F denote the class of functions computed by the corresponding networks, and D the maximum of { d 0 , d 1 , ..., d L − 1 , d } .

Original: Further, deﬁne a sequence of reference matrix ( M 1 , ..., M L ) with the same dimensions as ( A 1 , ..., A L ) , where M i = 0 in AlexNet (Krizhevsky et al., 2012) and M i
Revised: Further, deﬁne a sequence of reference matrix ( M 1 , ..., M L ) with the same dimensions as ( A 1 , ..., A L ) , where M i = 0 in AlexNet (Krizhevsky et al., 2012) and M i

Original: =
Revised: =

Original: I in ResNet (He et al., 2016) to ensure good generalization performances.
Revised: I in ResNet (He et al., 2016) to ensure good generalization performances.

Original: Let (cid:107)·(cid:107) σ denote the spectral norm, and let (cid:107)·(cid:107) p,q denote the ( p, q ) -norm deﬁned by (cid:107) A (cid:107) p,q = (cid:13)(cid:13) ( (cid:107) A ;1 (cid:107) p , ..., (cid:107) A ; k (cid:107) p ) (cid:13)(cid:13) q for matrix A ∈ R d × k .
Revised: Let (cid:107)·(cid:107) σ denote the spectral norm, and let (cid:107)·(cid:107) p,q denote the ( p, q ) -norm deﬁned by (cid:107) A (cid:107) p,q = (cid:13)(cid:13) ( (cid:107) A ;1 (cid:107) p , ..., (cid:107) A ; k (cid:107) p ) (cid:13)(cid:13) q for matrix A ∈ R d × k .

Original: We next give a spectrally-normalized meta-learning bound with deep neural network.
Revised: We next give a spectrally-normalized meta-learning bound with deep neural network.

Original: For the ease of presentation, we bound O ( (cid:113) C 1 + nC 2 nm ) with O ( (cid:113) C 1 nm + (cid:113) C 2 m ) .
Revised: For the ease of presentation, we bound O ( (cid:113) C 1 + nC 2 nm ) with O ( (cid:113) C 1 nm + (cid:113) C 2 m ) .

Original: Theorem 5 Let ( P , Q ) be an almost Π -related environment on the complete separable metric space
Revised: Theorem 5 Let ( P , Q ) be an almost Π -related environment on the complete separable metric space

Original: Z = X
Revised: Z = X

Original: × Y .
Revised: × Y .

Original: Let H =
Revised: Let H =

Original: {H A } = {G ◦ f A : f A ∈ F , A
Revised: {H A } = {G ◦ f A : f A ∈ F , A

Original: =
Revised: =

Original: ( A 1 , ..., A L ) , (cid:107)
Revised: ( A 1 , ..., A L ) , (cid:107)

Original: A i (cid:107) σ ≤ s i , (cid:107) A (cid:62) i − M (cid:62) i (cid:107) 2 , 1 ≤ b i , i ∈
Revised: A i (cid:107) σ ≤ s i , (cid:107) A (cid:62) i − M (cid:62) i (cid:107) 2 , 1 ≤ b i , i ∈

Original: [ L ] } be a hypothesis space family where each H A is of the form
Revised: [ L ] } be a hypothesis space family where each H A is of the form

Original: H A =
Revised: H A =

Original: G ◦ f A =
Revised: G ◦ f A =

Original: { g ◦ f A ( · ) : g
Revised: { g ◦ f A ( · ) : g

Original: = σ ◦ W, W ∈ R k × d , (cid:107) W (cid:62) (cid:107) 2 , 1 ≤ θ } , where σ is an element-wise function with Lipschitz constant θ σ .
Revised: = σ ◦ W, W ∈ R k × d , (cid:107) W (cid:62) (cid:107) 2 , 1 ≤ θ } , where σ is an element-wise function with Lipschitz constant θ σ .

Original: Suppose that ∃ b > 0 , for any x ∈ X ⊆ R d 0 , (cid:107) x (cid:107) 2 ≤ b .
Revised: Suppose that ∃ b > 0 , for any x ∈ X ⊆ R d 0 , (cid:107) x (cid:107) 2 ≤ b .

Original: Suppose that the loss function l satisﬁes two conditions: (1) when composed with g , g l ( · , y ) is an α -Lipschitz function w.r.t.
Revised: Suppose that the loss function l satisﬁes two conditions: (1) when composed with g , g l ( · , y ) is an α -Lipschitz function w.r.t.

Original: the norm (cid:107)·(cid:107) 2 , ∀ ﬁxed y ∈ Y ; (2) ∀ ﬁxed v ∈ R d , ﬁxed y ∈ Y , ∀ g = σ ◦ W, g (cid:48)
Revised: the norm (cid:107)·(cid:107) 2 , ∀ ﬁxed y ∈ Y ; (2) ∀ ﬁxed v ∈ R d , ﬁxed y ∈ Y , ∀ g = σ ◦ W, g (cid:48)

Original: =
Revised: =

Original: σ ◦ W (cid:48)
Revised: σ ◦ W (cid:48)

Original: ∈ G , ∃ β > 0 , such that | g l ( v, y ) − g (cid:48)
Revised: ∈ G , ∃ β > 0 , such that | g l ( v, y ) − g (cid:48)

Original: l ( v, y ) | ≤ β (cid:107)
Revised: l ( v, y ) | ≤ β (cid:107)

Original: Wv − W (cid:48) v (cid:107) 2 .
Revised: Wv − W (cid:48) v (cid:107) 2 .

Original: Then for any H A ∈ H , for any 0 < (cid:15) < 1 / 8 , with probability at least 1 − δ over z , we have | ˆ
Revised: Then for any H A ∈ H , for any 0 < (cid:15) < 1 / 8 , with probability at least 1 − δ over z , we have | ˆ

Original: [Equation]
Revised: [Equation]

Original: When the loss function l ( · , y ) is a γ -Lipschitz function w.r.t.
Revised: When the loss function l ( · , y ) is a γ -Lipschitz function w.r.t.

Original: the norm (cid:107)·(cid:107) 2 , we can set α =
Revised: the norm (cid:107)·(cid:107) 2 , we can set α =

Original: γθ
Revised: γθ

Original: σ θ , β = γθ σ .
Revised: σ θ , β = γθ σ .

Original: This is a very useful corollary for a large number of general applications, such as the multiclass classiﬁcation and regression problems in the following subsections.
Revised: This is a very useful corollary for a large number of general applications, such as the multiclass classiﬁcation and regression problems in the following subsections.

Original: 4.3. A S PECTRALLY -N ORMALIZED B OUND FOR B INARY C LASSIFICATION
Revised: 4.3. A S PECTRALLY -N ORMALIZED B OUND FOR B INARY C LASSIFICATION

Original: We ﬁrst consider the binary classiﬁcation problem under the meta learning framework, where Y = { 0 , 1 } .
Revised: We ﬁrst consider the binary classiﬁcation problem under the meta learning framework, where Y = { 0 , 1 } .

Original: We choose the classical logistic regression model due to its simplicity and wide applicability in binary classiﬁcation scenarios.
Revised: We choose the classical logistic regression model due to its simplicity and wide applicability in binary classiﬁcation scenarios.

Original: Formally, let G = { σ ◦ w : w ∈ R d , (cid:107) w (cid:107) 1 ≤ θ } 6 be the class of prediction functions, where σ is the sigmoid activation function σ ( v ) = 1 1+ e − v ( ∈ [0 , 1]) .
Revised: Formally, let G = { σ ◦ w : w ∈ R d , (cid:107) w (cid:107) 1 ≤ θ } 6 be the class of prediction functions, where σ is the sigmoid activation function σ ( v ) = 1 1+ e − v ( ∈ [0 , 1]) .

Original: Let g l ◦ f
Revised: Let g l ◦ f

Original: ( x, y )
Revised: ( x, y )

Original: =
Revised: =

Original: g l ( v, y ) be the loss function on ( x, y ) where g l ( v, y ) =
Revised: g l ( v, y ) be the loss function on ( x, y ) where g l ( v, y ) =

Original: − y ln( σ ( w (cid:62) v )) − (1 − y )
Revised: − y ln( σ ( w (cid:62) v )) − (1 − y )

Original: ln(1 − σ ( w (cid:62) v )) is the cross-entropy loss.
Revised: ln(1 − σ ( w (cid:62) v )) is the cross-entropy loss.

Original: We then have the following claim.
Revised: We then have the following claim.

Original: Claim 

Revised: Claim

Original: In the binary classiﬁcation problem as described above, ∀ y ∈ Y , g l ( · , y ) is a 2 θ -Lipshcitz function w.r.t.
Revised: In the binary classiﬁcation problem as described above, ∀ y ∈ Y , g l ( · , y ) is a 2 θ -Lipshcitz function w.r.t.

Original: l 2 -norm.
Revised: l 2 -norm.

Original: Further, ﬁx v ∈ R d , y ∈ { 0 , 1 } , then we have ∀ w 1 , w 2 ∈ R d , | l ( w (cid:62) 1 v, y ) − l ( w (cid:62) 2 v, y ) | ≤ 2 | w (cid:62) 1 v − w (cid:62) 2 v | .
Revised: Further, ﬁx v ∈ R d , y ∈ { 0 , 1 } , then we have ∀ w 1 , w 2 ∈ R d , | l ( w (cid:62) 1 v, y ) − l ( w (cid:62) 2 v, y ) | ≤ 2 | w (cid:62) 1 v − w (cid:62) 2 v | .

Original: Thus, we can set α = 2 θ, β = 2 in Theorem 5 to obtain a spectrallynormalized bound for binary classiﬁcation with logistic regression model in meta learning.
Revised: Thus, we can set α = 2 θ, β = 2 in Theorem 5 to obtain a spectrallynormalized bound for binary classiﬁcation with logistic regression model in meta learning.

Original: 4.3. A S PECTRALLY -N ORMALIZED B OUND FOR M ULTICLASS C LASSIFICATION
Revised: 4.3. A S PECTRALLY -N ORMALIZED B OUND FOR M ULTICLASS C LASSIFICATION

Original: We further consider the multiclass classiﬁcation problem under the meta learning framework, where
Revised: We further consider the multiclass classiﬁcation problem under the meta learning framework, where

Original: Y = { 1 , ..., k } ( k ≥ 3) .
Revised: Y = { 1 , ..., k } ( k ≥ 3) .

Original: Let G =
Revised: Let G =

Original: { W ∈ R k × d : (cid:107) W (cid:62) (cid:107) 2 , 1 ≤ θ } be the class of prediction functions, and Φ ρ ◦ M ( g ◦ f ( x ) , y )( f ∈ F , g ∈ G ) be the loss on ( x, y ) .
Revised: { W ∈ R k × d : (cid:107) W (cid:62) (cid:107) 2 , 1 ≤ θ } be the class of prediction functions, and Φ ρ ◦ M ( g ◦ f ( x ) , y )( f ∈ F , g ∈ G ) be the loss on ( x, y ) .

Original: Φ ρ ( v ) =
Revised: Φ ρ ( v ) =

Original: min (cid:0) 1 , max(0 , 1 − vρ ) (cid:1) is called the margin loss and ρ is a positive margin parameter.
Revised: min (cid:0) 1 , max(0 , 1 − vρ ) (cid:1) is called the margin loss and ρ is a positive margin parameter.

Original: M ( g ◦ f ( x ) , y )
Revised: M ( g ◦ f ( x ) , y )

Original: =
Revised: =

Original: max j (cid:54) =
Revised: max j (cid:54) =

Original: y
Revised: y

Original: g ◦ f ( x )[ j ] − g ◦ f
Revised: g ◦ f ( x )[ j ] − g ◦ f

Original: ( x )[ y ] is deﬁned as the margin on ( x, y ) .
Revised: ( x )[ y ] is deﬁned as the margin on ( x, y ) .

Original: From Lemma A.3 in (Bartlett et al., 2017) about the
Revised: From Lemma A.3 in (Bartlett et al., 2017) about the

Original: Lipschitz property of the margin loss, we have another claim.
Revised: Lipschitz property of the margin loss, we have another claim.

Original: Claim 

Revised: Claim

Original: In the multiclass classiﬁcation problem as described above, for any y ∈ Y , the loss function l ( · , y ) =
Revised: In the multiclass classiﬁcation problem as described above, for any y ∈ Y , the loss function l ( · , y ) =

Original: Φ ρ ◦ M ( · , y ) is 2/ ρ -Lipschitz w.r.t.
Revised: Φ ρ ◦ M ( · , y ) is 2/ ρ -Lipschitz w.r.t.

Original: the norm (cid:107)·(cid:107) 2 .
Revised: the norm (cid:107)·(cid:107) 2 .

Original: Then we can set γ = 2/ ρ and θ σ =( σ is the identity map in this case) in Theorem 5 and derive a spectrally-normalized margin bound for multiclass classiﬁcation in meta learning.
Revised: Then we can set γ = 2/ ρ and θ σ =( σ is the identity map in this case) in Theorem 5 and derive a spectrally-normalized margin bound for multiclass classiﬁcation in meta learning.

Original: 4.3. A S PECTRALLY -N ORMALIZED B OUND FOR R EGRESSION
Revised: 4.3. A S PECTRALLY -N ORMALIZED B OUND FOR R EGRESSION

Original: For completeness, we ﬁnally consider the regression problem under the meta learning framework, where
Revised: For completeness, we ﬁnally consider the regression problem under the meta learning framework, where

Original: Y =
Revised: Y =

Original: Let G = { σ ◦ w : w ∈ R d , (cid:107) w (cid:107) 1 ≤ θ } be the class of prediction functions, where σ is the sigmoid activation function σ ( v ) = 1 1+ e − v ( ∈ [0 , 1]) with the Lipschitz constant 1 / 4 (note that the derivative σ (cid:48) ( v )
Revised: Let G = { σ ◦ w : w ∈ R d , (cid:107) w (cid:107) 1 ≤ θ } be the class of prediction functions, where σ is the sigmoid activation function σ ( v ) = 1 1+ e − v ( ∈ [0 , 1]) with the Lipschitz constant 1 / 4 (note that the derivative σ (cid:48) ( v )

Original: =
Revised: =

Original: σ ( v )(1 − σ ( v )) ≤ 1 / 4 ).
Revised: σ ( v )(1 − σ ( v )) ≤ 1 / 4 ).

Original: Let the loss function l be the squared loss function by deﬁning g l ◦ f ( x, y )
Revised: Let the loss function l be the squared loss function by deﬁning g l ◦ f ( x, y )

Original: = ( σ ( w (cid:62) f
Revised: = ( σ ( w (cid:62) f

Original: ( x )) − y ) 2 ( f ∈ F , g ∈ G ) .
Revised: ( x )) − y ) 2 ( f ∈ F , g ∈ G ) .

Original: Claim 

Revised: Claim

Original: In the regression problem as described above, for any y ∈ Y , the loss function l ( · , y ) is 2-Lipschitz w.r.t.
Revised: In the regression problem as described above, for any y ∈ Y , the loss function l ( · , y ) is 2-Lipschitz w.r.t.

Original: (cid:107)·(cid:107) 2 .
Revised: (cid:107)·(cid:107) 2 .

Original: Then we can set γ = 2 and θ σ = 1 / 4 ( σ is the sigmoid function in this case) in Theorem 5 and derive a spectrally-normalized bound for regression problem in meta learning.
Revised: Then we can set γ = 2 and θ σ = 1 / 4 ( σ is the sigmoid function in this case) in Theorem 5 and derive a spectrally-normalized bound for regression problem in meta learning.

Original: 4. W HEN THE E NVIRONMENT IS A LMOST Π -R ELATED ?
Revised: 4. W HEN THE E NVIRONMENT IS A LMOST Π -R ELATED ?

Original: In this section, we explore when the given environment ( P , Q ) is almost Π -related, i.e., whether there exists a common underlying distribution P that is almost Π -related with any measure P i sampled from the environment.
Revised: In this section, we explore when the given environment ( P , Q ) is almost Π -related, i.e., whether there exists a common underlying distribution P that is almost Π -related with any measure P i sampled from the environment.

Original: As the discussion below the Deﬁnition 3 shows, the most crucial step is to ﬁnd an (almost) bijective transformation π which satisﬁes the conditions (1)-(3) in Deﬁnition 3.
Revised: As the discussion below the Deﬁnition 3 shows, the most crucial step is to ﬁnd an (almost) bijective transformation π which satisﬁes the conditions (1)-(3) in Deﬁnition 3.

Original: We claim that this bijective transformation is actually equivalent to the almost isomorphism between two probability measure spaces (Chapter 9 in (Bogachev, 2007)).
Revised: We claim that this bijective transformation is actually equivalent to the almost isomorphism between two probability measure spaces (Chapter 9 in (Bogachev, 2007)).

Original: We ﬁrst give the deﬁnition of the almost isomorphism and the proof of Theorem 6 can be found in the Appendix.
Revised: We ﬁrst give the deﬁnition of the almost isomorphism and the proof of Theorem 6 can be found in the Appendix.

Original: Deﬁnition 7 (Almost Isomorphism) Let ( Z 1 , A , µ ) and ( Z 2 , B , ν ) be two measure spaces.
Revised: Deﬁnition 7 (Almost Isomorphism) Let ( Z 1 , A , µ ) and ( Z 2 , B , ν ) be two measure spaces.

Original: (1) A point isomorphism π of these spaces is a one-to-one mapping of Z 1 on to Z 2 such that µ ◦ π − 1 =
Revised: (1) A point isomorphism π of these spaces is a one-to-one mapping of Z 1 on to Z 2 such that µ ◦ π − 1 =

Original: ν
Revised: ν

Original: and π ( A )
Revised: and π ( A )

Original: =
Revised: =

Original: B .
Revised: B .

Original: That is, ∀ A ∈ A , π ( A ) ∈ B , and vice versa.
Revised: That is, ∀ A ∈ A , π ( A ) ∈ B , and vice versa.

Original: (2) ( Z 1 , A , µ ) and ( Z 2 , B , ν ) are called almost isomorphic if there exist sets N 1 ∈ A µ , N 2 ∈ B ν with µ ( N 1 )
Revised: (2) ( Z 1 , A , µ ) and ( Z 2 , B , ν ) are called almost isomorphic if there exist sets N 1 ∈ A µ , N 2 ∈ B ν with µ ( N 1 )

Original: =
Revised: =

Original: ν ( N 2 ) = 0 and a point isomorphism π of the spaces Z 1 \ N 1 and Z 2 \ N 2 that are equipped with the restriction of the measures µ
Revised: ν ( N 2 ) = 0 and a point isomorphism π of the spaces Z 1 \ N 1 and Z 2 \ N 2 that are equipped with the restriction of the measures µ

Original: and ν and the complete σ -algebra A µ and B ν .
Revised: and ν and the complete σ -algebra A µ and B ν .

Original: With elaborate treatment, we can reveal the existence of the almost isomorphism between any two complete separable metric spaces.
Revised: With elaborate treatment, we can reveal the existence of the almost isomorphism between any two complete separable metric spaces.

Original: That means, there exists a common distribution P that is almost Π -related to any distribution sampled from the same environment.
Revised: That means, there exists a common distribution P that is almost Π -related to any distribution sampled from the same environment.

Original: It is formally stated as below.
Revised: It is formally stated as below.

Original: Theorem 6 Let ( P , Q ) be an environment on the complete separable metric space Z .
Revised: Theorem 6 Let ( P , Q ) be an environment on the complete separable metric space Z .

Original: Then for any atomeless P i , P j ∈ P ( i (cid:54)
Revised: Then for any atomeless P i , P j ∈ P ( i (cid:54)

Original: =
Revised: =

Original: j, i, j ∈ I ) , the probability measure spaces ( Z, B i , P i ) and ( Z, B j , P j ) are almost isomorphic.
Revised: j, i, j ∈ I ) , the probability measure spaces ( Z, B i , P i ) and ( Z, B j , P j ) are almost isomorphic.

Original: In other words, the two measures P i and P j are almost Π -related.
Revised: In other words, the two measures P i and P j are almost Π -related.

Original: From Theorem 6 we have seen that, any two atomless probability measures P
Revised: From Theorem 6 we have seen that, any two atomless probability measures P

Original: i , P j ( i (cid:54) =
Revised: i , P j ( i (cid:54) =

Original: j ) on the complete separable metric space Z are almost Π -related.
Revised: j ) on the complete separable metric space Z are almost Π -related.

Original: Therefore, we can choose P 1 ∈ P as the common distribution that is almost Π -related to any probability measure P i ( i ∈ I ) sampled from the set P .
Revised: Therefore, we can choose P 1 ∈ P as the common distribution that is almost Π -related to any probability measure P i ( i ∈ I ) sampled from the set P .

Original: Hence, we demonstrate the existence of the common distribution P deﬁned in Deﬁnition 5.
Revised: Hence, we demonstrate the existence of the common distribution P deﬁned in Deﬁnition 5.

Original: Further, we can drop the condition of the Π -relatedness of all tasks sampled from the environment ( P , Q ) in Theorems 2-5, since this task relatedness requirement is fulﬁlled by the fact that Z is a complete separable metric space.
Revised: Further, we can drop the condition of the Π -relatedness of all tasks sampled from the environment ( P , Q ) in Theorems 2-5, since this task relatedness requirement is fulﬁlled by the fact that Z is a complete separable metric space.

Original: We also need to point out that, the completeness and separability are both very general topological properties that can be satisﬁed by a number of metric spaces such as the real space R d , the closed subspace in R d , and the product space of the complete separable metric spaces.
Revised: We also need to point out that, the completeness and separability are both very general topological properties that can be satisﬁed by a number of metric spaces such as the real space R d , the closed subspace in R d , and the product space of the complete separable metric spaces.

Original: For example, in d -dimensional regression problem, the sample space Z
Revised: For example, in d -dimensional regression problem, the sample space Z

Original: = R d × [ a, b ] ( a, b ∈ R ) is also a complete separable metric space.
Revised: = R d × [ a, b ] ( a, b ∈ R ) is also a complete separable metric space.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: This paper provides a covering number based generalization bound for meta learning by exploiting the task relatedness of the environment.
Revised: This paper provides a covering number based generalization bound for meta learning by exploiting the task relatedness of the environment.

Original: When analyzing the meta learning with deep neural network, we derive spectrally-normalized bounds for classiﬁcation and regression problems.
Revised: When analyzing the meta learning with deep neural network, we derive spectrally-normalized bounds for classiﬁcation and regression problems.

Original: Our bounds rely on two basic assumptions: the relatedness between different tasks and the closure property of the hypothesis space.
Revised: Our bounds rely on two basic assumptions: the relatedness between different tasks and the closure property of the hypothesis space.

Original: We demonstrate that, the ﬁrst task-relatedness assumption can be satisﬁed if the sample space is a complete separable metric space.
Revised: We demonstrate that, the ﬁrst task-relatedness assumption can be satisﬁed if the sample space is a complete separable metric space.

Original: We also show that the closure property assumption holds when the hypothesis space contains good solutions as well as their equivalent variants.
Revised: We also show that the closure property assumption holds when the hypothesis space contains good solutions as well as their equivalent variants.

Original: Our ongoing research includes analyzing the convolutional neural network as well as establishing sharper generalization bounds for meta learning via algorithmic analysis.
Revised: Our ongoing research includes analyzing the convolutional neural network as well as establishing sharper generalization bounds for meta learning via algorithmic analysis.

Original: APPENDIX A M OTIVATION OF T ASK R ELATEDNESS
Revised: APPENDIX A M OTIVATION OF T ASK R ELATEDNESS

Original: We simply explain our motivation of exploring ‘task relatedness’ by comparing the differences between single task learning and meta learning in Table 1, to fully utilize all training samples.
Revised: We simply explain our motivation of exploring ‘task relatedness’ by comparing the differences between single task learning and meta learning in Table 1, to fully utilize all training samples.

Original: H )
Revised: H )

Original: er z ( H
Revised: er z ( H

Original: = (cid:82) h
Revised: = (cid:82) h

Original: P ( h )d Q ( P ) ni = inf h
Revised: P ( h )d Q ( P ) ni = inf h

Original: ˆ er P i ( h )
Revised: ˆ er P i ( h )

Original: The proposal of our task relatedness notion comes from the motivation probabilities measures { P i } ni =1 sampled from the environment ( P , Q ) need to be equivalent , namely, almost Π -related .
Revised: The proposal of our task relatedness notion comes from the motivation probabilities measures { P i } ni =1 sampled from the environment ( P , Q ) need to be equivalent , namely, almost Π -related .

Original: [Table]
Revised: [Table]

Original: Requirement { z i } mi =1 are similar enough i.e., identical distributed samples.
Revised: Requirement { z i } mi =1 are similar enough i.e., identical distributed samples.

Original: { P i } ni =1 need to be similar enough
Revised: { P i } ni =1 need to be similar enough

Original: [Table]
Revised: [Table]

Original: ., equivalent measures
Revised: ., equivalent measures

Original: [Table]
Revised: [Table]

Original: B D ETAILED P ROOFS OF O UR THEORETICAL R ESULTS B. P ROOF OF T HEORETICAL P ROPERTIES OF A LMOST Π -R ELATED E NVIRONEMENT
Revised: B D ETAILED P ROOFS OF O UR THEORETICAL R ESULTS B. P ROOF OF T HEORETICAL P ROPERTIES OF A LMOST Π -R ELATED E NVIRONEMENT

Original: Proof of Lemma 1 in the main paper.
Revised: Proof of Lemma 1 in the main paper.

Original: From Deﬁnition 3, ∃ N i ⊆ Z , such that P i ( N i )
Revised: From Deﬁnition 3, ∃ N i ⊆ Z , such that P i ( N i )

Original: = 0 .
Revised: = 0 .

Original: Then for any P -measurable set
Revised: Then for any P -measurable set

Original: A ⊆
Revised: A ⊆

Original: Z \ N ,
Revised: Z \ N ,

Original: [Equation]
Revised: [Equation]

Original: where the third equality holds due to the condition (3) in Deﬁnition 3, and the sixth equality holds due to the equivalent integral transformation (cid:82)
Revised: where the third equality holds due to the condition (3) in Deﬁnition 3, and the sixth equality holds due to the equivalent integral transformation (cid:82)

Original: f ◦ g − 1 d P = (cid:82)
Revised: f ◦ g − 1 d P = (cid:82)

Original: f d P ◦ g .
Revised: f d P ◦ g .

Original: Similarly, recalling P ( h l ) from Deﬁnition 1 and noticing π − 1 i ( N i ) is P -measurable from condition (4) in Deﬁnition 3, we have
Revised: Similarly, recalling P ( h l ) from Deﬁnition 1 and noticing π − 1 i ( N i ) is P -measurable from condition (4) in Deﬁnition 3, we have

Original: [Equation]
Revised: [Equation]

Original: The above second equality holds due to the fact that (cid:82) N i
Revised: The above second equality holds due to the fact that (cid:82) N i

Original: h l ◦ π − 1 i d P i = 

Revised: h l ◦ π − 1 i d P i =

Original: (since P i ( N i ) = 0 and h l is a bounded function), and the condition (3) of π − 1 i in Deﬁnition 3.
Revised: (since P i ( N i ) = 0 and h l is a bounded function), and the condition (3) of π − 1 i in Deﬁnition 3.

Original: (cid:4)
Revised: (cid:4)

Original: Proof of Lemma 2 in the main paper.
Revised: Proof of Lemma 2 in the main paper.

Original: To prove inf h ∈H er P ( h ) = inf h ∈H er P i ( h ) , it is equivalent to prove inf h l ∈H l P ( h l ) =
Revised: To prove inf h ∈H er P ( h ) = inf h ∈H er P i ( h ) , it is equivalent to prove inf h l ∈H l P ( h l ) =

Original: inf h l ∈H l P
Revised: inf h l ∈H l P

Original: i ( h l ) , then it sufﬁces to show that ∀ h l ∈ H l , ∃ h (cid:48)
Revised: i ( h l ) , then it sufﬁces to show that ∀ h l ∈ H l , ∃ h (cid:48)

Original: l ∈
Revised: l ∈

Original: H
Revised: H

Original: l
Revised: l

Original: we have P i ( h (cid:48) l ) ≤ P ( h l ) .
Revised: we have P i ( h (cid:48) l ) ≤ P ( h l ) .

Original: Since the symmetricity of P and P i (i.e., P ◦ π − 1 i = P i , P i ◦ π i = P holds almost everywhere), we can ﬁnd another h (cid:48)(cid:48)
Revised: Since the symmetricity of P and P i (i.e., P ◦ π − 1 i = P i , P i ◦ π i = P holds almost everywhere), we can ﬁnd another h (cid:48)(cid:48)

Original: l such that P ( h (cid:48)(cid:48) l ) ≤ P
Revised: l such that P ( h (cid:48)(cid:48) l ) ≤ P

Original: i ( h l ) .
Revised: i ( h l ) .

Original: In fact, let h (cid:48) l
Revised: In fact, let h (cid:48) l

Original: = h l ◦ π − 1 i , from Lemma 1, we have
Revised: = h l ◦ π − 1 i , from Lemma 1, we have

Original: P i ( h (cid:48) l ) ≤ P ( h l ) .
Revised: P i ( h (cid:48) l ) ≤ P ( h l ) .

Original: (cid:4)
Revised: (cid:4)

Original: B. P ROOF OF THE C OVERING N UMBER B OUND FOR M ETA L EARNING IN A LMOST Π -R ELATED E NVIRONMENT
Revised: B. P ROOF OF THE C OVERING N UMBER B OUND FOR M ETA L EARNING IN A LMOST Π -R ELATED E NVIRONMENT

Original: Lemma 3 (Theorem 18 in (Baxter, 2000)) Let H ⊆
Revised: Lemma 3 (Theorem 18 in (Baxter, 2000)) Let H ⊆

Original: H 1 ⊕· · ·⊕H n be a permissible class of functions mapping ( X × Y ) n into [0 , 1] .
Revised: H 1 ⊕· · ·⊕H n be a permissible class of functions mapping ( X × Y ) n into [0 , 1] .

Original: Let z ∈
Revised: Let z ∈

Original: ( X × Y ) ( m,n ) be generated by m ≥ 2 / ( (cid:15) 2 ν ) independent trials from ( X × Y ) n according to some product probability measure P = P 1 × · · · × P n .
Revised: ( X × Y ) ( m,n ) be generated by m ≥ 2 / ( (cid:15) 2 ν ) independent trials from ( X × Y ) n according to some product probability measure P = P 1 × · · · × P n .

Original: For any ν > 0 , 0 < (cid:15) < 1 , for any h ∈ H , we have
Revised: For any ν > 0 , 0 < (cid:15) < 1 , for any h ∈ H , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1 in the main paper .
Revised: Proof of Theorem 1 in the main paper .

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality holds due to the Lemma 24 in (Baxter, 2000), and the second inequality holds due to the Lemma 3 in the supplementary material.
Revised: where the ﬁrst inequality holds due to the Lemma 24 in (Baxter, 2000), and the second inequality holds due to the Lemma 3 in the supplementary material.

Original: Let the r.h.s.
Revised: Let the r.h.s.

Original: of the above inequality be less than the conﬁdence parameter δ ∈ (0 , 1) , then we have (cid:15) 2 ≥
Revised: of the above inequality be less than the conﬁdence parameter δ ∈ (0 , 1) , then we have (cid:15) 2 ≥

Original: [Equation]
Revised: [Equation]

Original: , which gives the explicit PAC-style generalization bound on d ν [ ˆ er z ( H ) , ˆ er P ( H )] in Theorem 1.
Revised: , which gives the explicit PAC-style generalization bound on d ν [ ˆ er z ( H ) , ˆ er P ( H )] in Theorem 1.

Original: (cid:4)
Revised: (cid:4)

Original: Proof of Theorem 2 in the main paper.
Revised: Proof of Theorem 2 in the main paper.

Original: According to Lemma 2, there exists a common distribution P ∈ P such that er P ( H )
Revised: According to Lemma 2, there exists a common distribution P ∈ P such that er P ( H )

Original: =
Revised: =

Original: er P i
Revised: er P i

Original: ( H ) for any P i ∈ P .
Revised: ( H ) for any P i ∈ P .

Original: Then er Q (
Revised: Then er Q (

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: P i ) =
Revised: P i ) =

Original: er P ( H ) =
Revised: er P ( H ) =

Original: 1 /n (cid:80) nj =1 er P j
Revised: 1 /n (cid:80) nj =1 er P j

Original: ( H )
Revised: ( H )

Original: =
Revised: =

Original: ˆ er P ( H ) , which indicates | er Q ( H ) − ˆ er P ( H ) | = 0 .
Revised: ˆ er P ( H ) , which indicates | er Q ( H ) − ˆ er P ( H ) | = 0 .

Original: (cid:4)
Revised: (cid:4)

Original: Proof of Theorem 3 in the main paper.
Revised: Proof of Theorem 3 in the main paper.

Original: By the fact that the loss function l has range [0 , 1] , and the triangle inequality of metric d ν , | ˆ er z
Revised: By the fact that the loss function l has range [0 , 1] , and the triangle inequality of metric d ν , | ˆ er z

Original: [Equation]
Revised: [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: er Q
Revised: er Q

Original: [Equation]
Revised: [Equation]

Original: H )
Revised: H )

Original: [Equation]
Revised: [Equation]

Original: δ .
Revised: δ .

Original: Letting ν = 2 (to minimize √ ν + 2 √ ν ) gives the result.
Revised: Letting ν = 2 (to minimize √ ν + 2 √ ν ) gives the result.

Original: (cid:4)
Revised: (cid:4)

Original: B. P ROOF OF C OVERING N UMBER B OUNDS FOR M ETA L EARNING WITH R EPRESENTATION L EARNING
Revised: B. P ROOF OF C OVERING N UMBER B OUNDS FOR M ETA L EARNING WITH R EPRESENTATION L EARNING

Original: Proof of Proposition 1 in the main paper .
Revised: Proof of Proposition 1 in the main paper .

Original: Fix an empirical measure P ¯ z on ( X × Y ) n , and let (cid:98) F be a minimum size (cid:15) 1 -cover for ( F , d
Revised: Fix an empirical measure P ¯ z on ( X × Y ) n , and let (cid:98) F be a minimum size (cid:15) 1 -cover for ( F , d

Original: [ P ¯ z , G nl ] ) , then we have | (cid:98) F|
Revised: [ P ¯ z , G nl ] ) , then we have | (cid:98) F|

Original: =
Revised: =

Original: N ( (cid:15) 1 , F , d
Revised: N ( (cid:15) 1 , F , d

Original: [ P ¯ z , G nl ] ) , and ∀ ¯ f ∈ F , ∃ ˆ f ∈ (cid:98) F such that d [ P ¯ z , G nl ] ( ¯ f, ˆ f ) ≤ (cid:15) 1 .
Revised: [ P ¯ z , G nl ] ) , and ∀ ¯ f ∈ F , ∃ ˆ f ∈ (cid:98) F such that d [ P ¯ z , G nl ] ( ¯ f, ˆ f ) ≤ (cid:15) 1 .

Original: For any ˆ f ∈ (cid:98) F , let P ¯ z ◦ ˆ f − 1 be the induced probability measure on ( V × Y ) n by deﬁning
Revised: For any ˆ f ∈ (cid:98) F , let P ¯ z ◦ ˆ f − 1 be the induced probability measure on ( V × Y ) n by deﬁning

Original: P ¯ z ◦ ˆ f − 1 ( A ) =
Revised: P ¯ z ◦ ˆ f − 1 ( A ) =

Original: P ¯ z ( ˆ f − 1 ( A )) , ∀ A ∈ σ -algebra on ( V × Y ) n .
Revised: P ¯ z ( ˆ f − 1 ( A )) , ∀ A ∈ σ -algebra on ( V × Y ) n .

Original: Let G
Revised: Let G

Original: ˆ f be the minimum size (cid:15) 2 -cover for ( G nl , d
Revised: ˆ f be the minimum size (cid:15) 2 -cover for ( G nl , d

Original: P ¯ z ◦ ˆ f − 1 ) .
Revised: P ¯ z ◦ ˆ f − 1 ) .

Original: Hence |G ˆ f | = N ( (cid:15) 2 , G nl , d P ¯ z ◦ ˆ f − 1 ) ≤ N ( (cid:15) 2 , G nl , 2 m ) .
Revised: Hence |G ˆ f | = N ( (cid:15) 2 , G nl , d P ¯ z ◦ ˆ f − 1 ) ≤ N ( (cid:15) 2 , G nl , 2 m ) .

Original: Let N =
Revised: Let N =

Original: { g ◦ f : f ∈ (cid:98) F and g ∈ G f } , then we have | N | ≤ N ( (cid:15) 1 , F , d
Revised: { g ◦ f : f ∈ (cid:98) F and g ∈ G f } , then we have | N | ≤ N ( (cid:15) 1 , F , d

Original: [ P ¯ z , G nl ] ) N ( (cid:15) 2 , G nl , 2 m ) , which satisﬁes the required cardinality condition.
Revised: [ P ¯ z , G nl ] ) N ( (cid:15) 2 , G nl , 2 m ) , which satisﬁes the required cardinality condition.

Original: It remains to show that N is an (cid:15) 1 + (cid:15) 2 − cover of H nl .
Revised: It remains to show that N is an (cid:15) 1 + (cid:15) 2 − cover of H nl .

Original: Actually, ∀ g 1 ⊕ · · · ⊕ g n ◦ ¯ f ∈ H nl , choose ˆ f ∈ (cid:98) F such that d [ P ¯ z , G nl ] ( ¯ f, ˆ f ) ≤ (cid:15) 1 , and choose g (cid:48) 1 ⊕ · · · ⊕ g (cid:48) n ∈ G ˆ f such that d P ¯ z ◦ ˆ f − 1 ( g 1 ⊕ · · · ⊕ g n , g (cid:48) 1 ⊕ · · · ⊕ g (cid:48) n ) ≤ (cid:15) 2 .
Revised: Actually, ∀ g 1 ⊕ · · · ⊕ g n ◦ ¯ f ∈ H nl , choose ˆ f ∈ (cid:98) F such that d [ P ¯ z , G nl ] ( ¯ f, ˆ f ) ≤ (cid:15) 1 , and choose g (cid:48) 1 ⊕ · · · ⊕ g (cid:48) n ∈ G ˆ f such that d P ¯ z ◦ ˆ f − 1 ( g 1 ⊕ · · · ⊕ g n , g (cid:48) 1 ⊕ · · · ⊕ g (cid:48) n ) ≤ (cid:15) 2 .

Original: Then the triangle inequality of the distance metric d ¯ z implies that
Revised: Then the triangle inequality of the distance metric d ¯ z implies that

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: The second inequality holds due to the deﬁnition of
Revised: The second inequality holds due to the deﬁnition of

Original: [Equation]
Revised: [Equation]

Original: and the equivalent integral transformation
Revised: and the equivalent integral transformation

Original: [Equation]
Revised: [Equation]

Original: Proof of Proposition 2 in the main paper .
Revised: Proof of Proposition 2 in the main paper .

Original: For any ¯ f, ¯ f (cid:48) ∈ F , we have
Revised: For any ¯ f, ¯ f (cid:48) ∈ F , we have

Original: [Equation]
Revised: [Equation]

Original: which completes the proof of the ﬁrst inequality.
Revised: which completes the proof of the ﬁrst inequality.

Original: Similarly, for the second one, ∀ P ∼ Q n , s ∼
Revised: Similarly, for the second one, ∀ P ∼ Q n , s ∼

Original: P 2 m , ∀ g, g (cid:48)
Revised: P 2 m , ∀ g, g (cid:48)

Original: ∈
Revised: ∈

Original: G nl ,
Revised: G nl ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, N ( (cid:15), G nl , d P s ) ≤
Revised: Therefore, N ( (cid:15), G nl , d P s ) ≤

Original: [Equation]
Revised: [Equation]

Original: max 1 ≤ j ≤ n N ( (cid:15), G l , d P s ; j ) (cid:1) n and thus N ( (cid:15), G nl , 2 m ) ≤
Revised: max 1 ≤ j ≤ n N ( (cid:15), G l , d P s ; j ) (cid:1) n and thus N ( (cid:15), G nl , 2 m ) ≤

Original: N ( (cid:15), G l , 2 m ) n .
Revised: N ( (cid:15), G l , 2 m ) n .

Original: (cid:4)
Revised: (cid:4)

Original: B. P ROOF OF S PECTRALLY -N ORMALIZED B OUNDS FOR M ETA L EARNING WITH D EEP N EURAL N ETWORK
Revised: B. P ROOF OF S PECTRALLY -N ORMALIZED B OUNDS FOR M ETA L EARNING WITH D EEP N EURAL N ETWORK

Original: To demonstrate Theorem 5 in the main paper, we need to give the following two important lemmas, which gives the non-parameter-count-based spectrally-normalized bounds for deep neural network in the single task learning.
Revised: To demonstrate Theorem 5 in the main paper, we need to give the following two important lemmas, which gives the non-parameter-count-based spectrally-normalized bounds for deep neural network in the single task learning.

Original: Lemma 4 (Bartlett et al., 2017) Let positive reals ( α, β, (cid:15) ) and positive integer k be given.
Revised: Lemma 4 (Bartlett et al., 2017) Let positive reals ( α, β, (cid:15) ) and positive integer k be given.

Original: Let matrix V (cid:62)
Revised: Let matrix V (cid:62)

Original: ∈
Revised: ∈

Original: R 2 m × d be given with (cid:107) V (cid:62) (cid:107) 2 ≤ β .
Revised: R 2 m × d be given with (cid:107) V (cid:62) (cid:107) 2 ≤ β .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Lemma 5 (Bartlett et al., 2017) Let ﬁxed nonlinearities ( σ 1 , ..., σ L ) and reference matrices ( M 1 , ..., M L ) be given, where σ i is ρ i -Lipschitz and σ i (0)
Revised: Lemma 5 (Bartlett et al., 2017) Let ﬁxed nonlinearities ( σ 1 , ..., σ L ) and reference matrices ( M 1 , ..., M L ) be given, where σ i is ρ i -Lipschitz and σ i (0)

Original: = 0 .
Revised: = 0 .

Original: Let spectral norm bounds ( s 1 , ..., s L ) , and matrix (2 , 1) -norm bounds ( b 1 , ..., b L ) be given.
Revised: Let spectral norm bounds ( s 1 , ..., s L ) , and matrix (2 , 1) -norm bounds ( b 1 , ..., b L ) be given.

Original: Let the input data matrix
Revised: Let the input data matrix

Original: X ∈ R 2 m × d 0 be given, where the m rows correspond to data points.
Revised: X ∈ R 2 m × d 0 be given, where the m rows correspond to data points.

Original: Let H X denote the family of matrices obtained by evaluating X with all choices of network f A : H X
Revised: Let H X denote the family of matrices obtained by evaluating X with all choices of network f A : H X

Original: = { f A ( X (cid:62) ) |A
Revised: = { f A ( X (cid:62) ) |A

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: , ..., A L ) , (cid:107)
Revised: , ..., A L ) , (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: , (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: M

Revised: [Equation]

Revised: (cid:107) 2 ,

Original: [Equation]
Revised: [Equation]

Original: ≤ b i } , where each matrix has dimension at most D along each axis.
Revised: ≤ b i } , where each matrix has dimension at most D along each axis.

Original: Then for any (cid:15) > 0 ,
Revised: Then for any (cid:15) > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 5 in the main paper .
Revised: Proof of Theorem 5 in the main paper .

Original: First, we bound ln N ( (cid:15), F , d [ P (cid:126) z , G l ] ) .
Revised: First, we bound ln N ( (cid:15), F , d [ P (cid:126) z , G l ] ) .

Original: For any P ∈ P ,(cid:126)z ∼ P 2 m , for any f A , f A (cid:48) ∈ F , 
Revised: For any P ∈ P ,(cid:126)z ∼ P 2 m , for any f A , f A (cid:48) ∈ F , d [Equation] P

Original: [Equation]
Revised: [Equation]

Revised: Lipschitz)

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma
Revised: Applying Lemma

Original: [Equation]
Revised: [Equation]

Original: we then have
Revised: we then have

Original: [Equation]
Revised: [Equation]

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we bound ln N ( (cid:15), G l , d P (cid:126)s ) .
Revised: we bound ln N ( (cid:15), G l , d P (cid:126)s ) .

Original: Actually, for any P ∼ Q,(cid:126)s ∼ P 2 m , for any g l , g (cid:48) l ∈ G l , using condition (2) of the loss function l and Jensen inequality [Equation] we have
Revised: Actually, for any P ∼ Q,(cid:126)s ∼ P 2 m , for any g l , g (cid:48) l ∈ G l , using condition (2) of the loss function l and Jensen inequality, we have

Original: [Equation]
Revised: [Equation]

Original: where V = ( v 1 , ..., v 2 m ) ∈ R
Revised: where V = ( v 1 , ..., v 2 m ) ∈ R

Original: d × 2 m .
Revised: d × 2 m .

Original: Then combining the above results and Lemma 4, we have ln N ( (cid:15), G l , d P (cid:126)s )
Revised: Then combining the above results and Lemma 4, we have ln N ( (cid:15), G l , d P (cid:126)s )

Original: [Equation]
Revised: [Equation]

Original: The last inequality holds due to the Lipschitz property of the activation σ l and the matrix A l ( l = 1 , ..., L ) , ∀ i ∈ [2 m ]
Revised: The last inequality holds due to the Lipschitz property of the activation σ l and the matrix A l ( l = 1 , ..., L ) , ∀ i ∈ [2 m ]

Original: [Equation]
Revised: [Equation]

Original: Recalling Theorem 4 in the main paper and Eqs.(1)-(2) in the supplementary gives the result. 
Revised: Recalling Theorem 4 in the main paper and Eqs.(1)-(2) in the supplementary gives the result. [Equation]

Revised: Remark 1 ( A situation where our spectrally-normalized bound for meta-learning is informative )

Original: (cid:4)
Revised: We further provide a situation where the neural network is a two-layer network (i.e. composed of a hidden layer with D units and an output layer) for k -class classiﬁcation problem, to provide more information of our spectrally-normalized bound from two aspects: (i) Our bound in Theorem 5 is more informative than the traditional VC-dimension bound. Actually, our bound in Theorem 5 for two-layer network can be rewritten as follow: [Equation]

Revised: which is of order O ( b 1 √ ln D 2 √ nm

Revised: [Equation]

Revised: Under the same setting, the VC-dimension based meta-learning bound for neural networks (i.e. obtained with techniques from Theorem 8 in Baxter (2000)) is of order O

Revised: ( v √ nm + v √ m ) .

Revised: Further note that the VC-dimension of the neural networks is v = W ln W (where W is the total number of the parameters, see Bartlett et al.

Revised: thus the VC-dimension for two-layer neural networks is about v =

Revised: ( D 2 + Dk ) ln

Revised: ( D 2 + Dk ) and the meta-learning bound is about O

Revised: ( ( D 2 + Dk )ln

Revised: ( D 2 + Dk ) √ nm +

Revised: [Equation]

Revised: ) , which is looser than our spectrally-normalized meta-learning bound of O ( b 1 √ ln D 2 √ nm

Revised: [Equation]

Revised: Dk

Revised: [Equation]

Revised: (ii) Our bound in

Revised: Theorem 5 is informative under the implicit regularization framework of SGD.

Revised: Note that SGD can always ﬁnd a minimum trace/nuclear norm for neural networks (especially for two-layer networks, see Gunasekar et al.

Revised: , therefore the norm parameters s 1 and b 1 in our bound for two-layer neural networks can be small, and hence our spectrally-normalized meta-learning bound of order

Revised: O (

Revised: [Equation]

Revised: ) can be informative.

Original: Proof of the Corollary under Theorem 5 in the main paper . 
Revised: Proof of the Corollary under Theorem 5 in the main paper . ∀

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:107) 2 , where the second inequality holds since (cid:107) W (cid:62) (cid:107) 2 , 1 is a kind of matrix norm of W .
Revised: where the second inequality holds since (cid:107) W (cid:62) (cid:107) 2 , 1 is a kind of matrix norm of W .

Original: Hence we can set α = γθ σ θ .
Revised: Hence we can set α = γθ σ θ .

Original: Similarly, ∀ g l , g (cid:48) l ∈ G l , v ∈ R d , y ∈ Y , we have | g l ( v, y ) 
Revised: Similarly, ∀ g l , g (cid:48) l ∈ G l , v ∈ R d , y ∈ Y , we have | g l ( v, y ) − g

Original: − g (cid:48) l ( v, y) |
Revised: [Equation] l ( [Equation] , y

Original: [Equation]
Revised: [Equation]

Original: Then we can obtain β = γθ σ .
Revised: Then we can obtain β = γθ σ .

Original: Combining the above results with Theorem 5 gives the result.
Revised: Combining the above results with Theorem 5 gives the result.

Original: [Equation]
Revised: (cid:4)

Original: Proof of Claim 1 in the main paper .
Revised: Proof of Claim 1 in the main paper .

Original: ∀ v ∈ R d , g l ( v, y )
Revised: ∀ v ∈ R d , g l ( v, y )

Original: =
Revised: =

Original: − y ln( σ ( w (cid:62) v )) − (1 − y )
Revised: − y ln( σ ( w (cid:62) v )) − (1 − y )

Original: ln(1 − σ ( w (cid:62) v )) .
Revised: ln(1 − σ ( w (cid:62) v )) .

Original: Therefore, ∀ v 1 , v 2 ∈ R d , we have 
Revised: Therefore, ∀ v 1 , v 2 ∈ R d , we have | g l ( v 1 , y ) − g l ( v 2 , y ) | = [Equation] (cid:62) [Equation]

Revised: ≤| w (cid:62)

Original: [Equation]
Revised: [Equation]

Revised: (cid:62)

Revised: [Equation]

Revised: (cid:62)

Revised: [Equation]

Revised: [Equation]

Original: e
Revised: e

Original: [Equation]
Revised: [Equation]

Original: Lipschitz)
Revised: (cid:62)

Original: [Equation]
Revised: [Equation]

Original: (Schwarz and (cid:107) w (cid:107) 2 ≤ (cid:107) w (cid:107) 1 ) .
Revised: ≤ 2 | w (cid:62) v 2 − w (cid:62) v 1 | (ln(1 + e x ) is 1 − Lipschitz) ≤ 2 θ (cid:107) v 2 − v 1 (cid:107)(Schwarz and (cid:107) w (cid:107) 2 ≤ (cid:107) w (cid:107) 1 ) .

Original: Similarly, since l ( w (cid:62) v, y ) = − y ln
Revised: Similarly, since l ( w (cid:62) v, y ) = − y ln

Original: ( σ ( w (cid:62) v ))
Revised: ( σ ( w (cid:62) v ))

Original: − [Equation] 1 − y )
Revised: − (1 − y )

Original: ln(1 − σ ( w [Equation] v )) , for any y ∈ { 0 , 1 } we can obtain
Revised: ln(1 − σ ( w (cid:62) v )) , for any y ∈ { 0 , 1 } we can obtain

Original: [Equation]
Revised: [Equation]

Original: Proof of Claim 3 in the main paper .
Revised: Proof of Claim 3 in the main paper .

Original: For any ﬁxed y ∈ [0 , 1] , for any v 1 , v 2 ∈ R d , notice that both g ( v 1 ) and g ( v 2 ) also lie into the interval [0 , 1] , then 
Revised: For any ﬁxed y ∈ [0 , 1] , for any v 1 , v 2 ∈ R d , notice that both g ( v 1 ) and g ( v 2 ) also lie into the interval [0 , 1] , then | l ( g ( v 1 ) , y ) − l ( g ( v 2 ) , y ) [Equation] = [Equation]

Revised: g

Original: [Equation]
Revised: [Equation]

Revised: v 1 )

Revised: [Equation]

Revised: y )

Revised: [Equation]

Revised: g

Revised: [Equation]

Revised: v 2 ) − y

Revised: [Equation]

Revised: 2 | ≤

Revised: [Equation]

Revised: | g

Revised: [Equation]

Revised: v 1 ) − g

Revised: [Equation]

Revised: v 2 ) | .

Revised: (cid:4)

Original: B. P ROOF OF W HEN THE E NVIRONMENT IS A LMOST Π -R ELATED ?
Revised: B. P ROOF OF W HEN THE E NVIRONMENT IS A LMOST Π -R ELATED ?

Original: To obtain a complete proof of Theorem 6 in the main paper, we ﬁrst give an important lemma that states the existence of the almost isomorphism between a complete separable metric space and the unit interval [0 , 1] .
Revised: To obtain a complete proof of Theorem 6 in the main paper, we ﬁrst give an important lemma that states the existence of the almost isomorphism between a complete separable metric space and the unit interval [0 , 1] .

Original: We also introduce the formal deﬁnitions of metric Boolean algebra and metric
Revised: We also introduce the formal deﬁnitions of metric Boolean algebra and metric

Original: Boolean isomorphism (Bogachev, 2007).
Revised: Boolean isomorphism (Bogachev, 2007).

Original: Theorem 7 (Bogachev, 2007) Let ( Z, µ ) be a complete separable metric space with a Borel probability measure µ .
Revised: Theorem 7 (Bogachev, 2007) Let ( Z, µ ) be a complete separable metric space with a Borel probability measure µ .

Original: Then ( Z, µ ) is almost isomorphic to the space ([0 , 1] , ν ) , where ν is some Borel probability measure.
Revised: Then ( Z, µ ) is almost isomorphic to the space ([0 , 1] , ν ) , where ν is some Borel probability measure.

Original: If µ is an atomless measure, then one can take for ν Lebesgue measure.
Revised: If µ is an atomless measure, then one can take for ν Lebesgue measure.

Original: Deﬁnition 8 (Metric Boolean Algebra) Let ( Z, B , µ ) be a measure space with a ﬁnite nonnegative measure µ .
Revised: Deﬁnition 8 (Metric Boolean Algebra) Let ( Z, B , µ ) be a measure space with a ﬁnite nonnegative measure µ .

Original: Let d ( A, B ) =
Revised: Let d ( A, B ) =

Original: µ ( A (cid:52) B ) , A, B ∈ B .
Revised: µ ( A (cid:52) B ) , A, B ∈ B .

Original: The function d is called the Frechet-Nikodym metric and we can introduce the following equivalence relation on B : A ∼ B if d ( A, B ) = 0 .
Revised: The function d is called the Frechet-Nikodym metric and we can introduce the following equivalence relation on B : A ∼ B if d ( A, B ) = 0 .

Original: Then the metric space ( B /µ, d ) is called the metric Boolean algebra, or measure algebra, often denoted by E u .
Revised: Then the metric space ( B /µ, d ) is called the metric Boolean algebra, or measure algebra, often denoted by E u .

Original: Further, the metric space ( B /µ, d ) is separable,i.e.,contains a countable everywhere dense subset, if and only if the corresponding measure µ is separable.
Revised: Further, the metric space ( B /µ, d ) is separable,i.e.,contains a countable everywhere dense subset, if and only if the corresponding measure µ is separable.

Original: The separability of µ is equivalent to the existence of a countable collection of sets B n ∈ B such that ∀ B ∈ B and (cid:15) > 0 , there exists an integer n with µ ( B (cid:52) B n ) ≤ (cid:15) .
Revised: The separability of µ is equivalent to the existence of a countable collection of sets B n ∈ B such that ∀ B ∈ B and (cid:15) > 0 , there exists an integer n with µ ( B (cid:52) B n ) ≤ (cid:15) .

Original: In addition, a metric space ( B /µ, d ) is complete if B is a σ -algebra and µ is countably additive (e.g., µ is a measure).
Revised: In addition, a metric space ( B /µ, d ) is complete if B is a σ -algebra and µ is countably additive (e.g., µ is a measure).

Original: Deﬁnition 9 (Metric Boolean Isomorphism)
Revised: Deﬁnition 9 (Metric Boolean Isomorphism)

Original: Two measure algebras E u 1 and E u 2 generated by measure spaces ( Z 1 , B 1 , µ 1 ) and ( Z 2 , B 2 , µ 2 ) are called isomorphic if there exists a one-to-one mapping
Revised: Two measure algebras E u 1 and E u 2 generated by measure spaces ( Z 1 , B 1 , µ 1 ) and ( Z 2 , B 2 , µ 2 ) are called isomorphic if there exists a one-to-one mapping

Original: J from E µ 1 onto E µ 2 (called a metric Boolean isomorphism) such that J preserves the measure, i.e., µ 2 ( J ( A ))
Revised: J from E µ 1 onto E µ 2 (called a metric Boolean isomorphism) such that J preserves the measure, i.e., µ 2 ( J ( A ))

Original: = µ 1 ( A ) , ∀ A ∈ E µ 1 .
Revised: = µ 1 ( A ) , ∀ A ∈ E µ 1 .

Original: Theorem 8 (Bogachev, 2007)
Revised: Theorem 8 (Bogachev, 2007)

Original: Every separable atomless measure algebra is isomorphic to the measure algebra of some interval (e.g., [0,1]) with Lebesgue measure.
Revised: Every separable atomless measure algebra is isomorphic to the measure algebra of some interval (e.g., [0,1]) with Lebesgue measure.

Original: Deﬁnition 10 (Lebesgue-Rohlin Space)
Revised: Deﬁnition 10 (Lebesgue-Rohlin Space)

Original: A measure space ( Z, B , µ ) is called a Lebesgue-Rohlin space if it is almost isomorphic to some measure space ( Z (cid:48) , B (cid:48) , µ (cid:48) ) with a countable basis with respect to which Z (cid:48) is complete.
Revised: A measure space ( Z, B , µ ) is called a Lebesgue-Rohlin space if it is almost isomorphic to some measure space ( Z (cid:48) , B (cid:48) , µ (cid:48) ) with a countable basis with respect to which Z (cid:48) is complete.

Original: Example 1 The space ([0 , 1] , B ([0 , 1]) , λ ) , where λ is Lebesgue measure, has a countable basis with respect to which it is complete.
Revised: Example 1 The space ([0 , 1] , B ([0 , 1]) , λ ) , where λ is Lebesgue measure, has a countable basis with respect to which it is complete.

Original: Theorem 9 (von Neumann, 1932)
Revised: Theorem 9 (von Neumann, 1932)

Original: Let ( Z 1 , B 1 , µ 1 ) and ( Z 2 , B 2 , µ 2 ) be Lebesgue-Rohlin spaces with probability measures.
Revised: Let ( Z 1 , B 1 , µ 1 ) and ( Z 2 , B 2 , µ 2 ) be Lebesgue-Rohlin spaces with probability measures.

Original: If the corresponding measure algebra E µ 1 and E µ 2 are isomorphic in the sense of Deﬁnition 9, then there exists an almost isomorphism between these spaces.
Revised: If the corresponding measure algebra E µ 1 and E µ 2 are isomorphic in the sense of Deﬁnition 9, then there exists an almost isomorphism between these spaces.

Original: In particular, this is the case if both measures are atomless.
Revised: In particular, this is the case if both measures are atomless.

Original: Proof of Theorem 6 in the main paper .
Revised: Proof of Theorem 6 in the main paper .

Original: The proof contains 3 main steps: (1) Since Z is the complete separable metric space and the probability measures P
Revised: The proof contains 3 main steps: (1) Since Z is the complete separable metric space and the probability measures P

Original: i , P j are atomless, then from Theorem 7, ( Z, B i , P i ) and ( Z, B j , P j ) are both almost isomorphic to the measure space ([0 , 1] , B ([0 , 1]) , λ ) where λ is the Lebesgue measure.
Revised: i , P j are atomless, then from Theorem 7, ( Z, B i , P i ) and ( Z, B j , P j ) are both almost isomorphic to the measure space ([0 , 1] , B ([0 , 1]) , λ ) where λ is the Lebesgue measure.

Original: From
Revised: From

Original: Deﬁnition 10 and Example 1, ( Z, B i , P i ) and ( Z, B j , P j ) are Lebesgue-Rohlin spaces.
Revised: Deﬁnition 10 and Example 1, ( Z, B i , P i ) and ( Z, B j , P j ) are Lebesgue-Rohlin spaces.

Original: (2) From Example 1, the complete measure space ([0 , 1] \ M, B ([0 , 1]) λ , λ ) (w.r.t.
Revised: (2) From Example 1, the complete measure space ([0 , 1] \ M, B ([0 , 1]) λ , λ ) (w.r.t.

Original: the measure λ and λ ( M ) = 0 ) has a countable basis { B n } ∞ n =1 ⊂ B ([0 , 1]) λ .
Revised: the measure λ and λ ( M ) = 0 ) has a countable basis { B n } ∞ n =1 ⊂ B ([0 , 1]) λ .

Original: The almost isomorphism π i from ([0 , 1] , B ([0 , 1]) , λ ) into ( Z, B i , P i ) can induce a countable basis { π i ( B n ) } ∞ n =1 ⊂ B i , which guarantees the separability of the measure algebras E P i .
Revised: The almost isomorphism π i from ([0 , 1] , B ([0 , 1]) , λ ) into ( Z, B i , P i ) can induce a countable basis { π i ( B n ) } ∞ n =1 ⊂ B i , which guarantees the separability of the measure algebras E P i .

Original: The separability of the measure algebras E P j can be guaranteed in the same way.
Revised: The separability of the measure algebras E P j can be guaranteed in the same way.

Original: Then from Theorem 8, the measure algebras E P i and E P j , generated by measure spaces ( Z, B i , P i ) and ( Z, B j , P j ) , are both isomorphic to the measure algebra of the interval [0,1] with Lebesgue measure.
Revised: Then from Theorem 8, the measure algebras E P i and E P j , generated by measure spaces ( Z, B i , P i ) and ( Z, B j , P j ) , are both isomorphic to the measure algebra of the interval [0,1] with Lebesgue measure.

Original: Therefore, the two measure algebra E P i and E P j are isomorphic (since isomorphism is an equivalence relation).
Revised: Therefore, the two measure algebra E P i and E P j are isomorphic (since isomorphism is an equivalence relation).

Original: (3) Combining (1) and (2), and recalling Theorem 9, the two measure spaces ( Z, B i , P i ) and ( Z, B j , P j ) are almost isomorphic.
Revised: (3) Combining (1) and (2), and recalling Theorem 9, the two measure spaces ( Z, B i , P i ) and ( Z, B j , P j ) are almost isomorphic.

Original: So P i and P j are almost Π -related in the sense of Deﬁnition 3.
Revised: So P i and P j are almost Π -related in the sense of Deﬁnition 3.

Original: [Equation]
Revised: (cid:4)

Original: C M ORE DETAILED C OMPARISONS WITH R ELATED W ORKS C. D ETAILED C OMPARISON WITH G ENERALIZATION B OUNDS OF (B AXTER , 2000)
Revised: C M ORE DETAILED C OMPARISONS WITH R ELATED W ORKS C. D ETAILED C OMPARISON WITH G ENERALIZATION B OUNDS OF (B AXTER , 2000)

Original: This paper can be considered as the extension of the meta learning theoretical work in (Baxter, 2000), by further exploring the task relatedness for the environment.
Revised: This paper can be considered as the extension of the meta learning theoretical work in (Baxter, 2000), by further exploring the task relatedness for the environment.

Original: In this section, we detail our improvements over this pioneering work.
Revised: In this section, we detail our improvements over this pioneering work.

Original: First, we introduce a new notation called H ∗ .
Revised: First, we introduce a new notation called H ∗ .

Original: For any hypothesis space H ∈ H , deﬁne H ∗ : P → [0 , 1] by H ∗ ( P )
Revised: For any hypothesis space H ∈ H , deﬁne H ∗ : P → [0 , 1] by H ∗ ( P )

Original: =
Revised: =

Original: inf h ∈H er P ( h ) , and for any hypothesis space family H , deﬁne H ∗
Revised: inf h ∈H er P ( h ) , and for any hypothesis space family H , deﬁne H ∗

Original: = {H ∗ : H ∈ H } .
Revised: = {H ∗ : H ∈ H } .

Original: Although Baxter does not give the explicit
Revised: Although Baxter does not give the explicit

Original: PAC-style learning bound in its main results, its bound on | ˆ er z ( H ) − er Q ( H ) | can be expressed as follow:
Revised: PAC-style learning bound in his main paper, its bound on | ˆ er z ( H ) − er Q ( H ) | can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: where C ( (cid:15), H nl ) =
Revised: where C ( (cid:15), H nl ) =

Original: sup P N ( (cid:15), H nl , d P ) , C ( (cid:15), H ∗ ) =
Revised: sup P N ( (cid:15), H nl , d P ) , C ( (cid:15), H ∗ ) =

Original: sup Q N ( (cid:15), H ∗ , d Q ) (see its Theorem 4).
Revised: sup Q N ( (cid:15), H ∗ , d Q ) (see its Theorem 4).

Original: Our improvements can be summarized in the following three aspects:
Revised: Our improvements can be summarized in the following three aspects

Original:  (1) We exploit the proposed task relatedness to reduce the complexity C ( (cid:15), H ∗ ) in Eq.
Revised: : (1) We exploit the proposed task relatedness to reduce the complexity C ( (cid:15), H ∗ ) in Eq.

Original: (3) to zero in Theorems 3.
Revised: (3) to zero in Theorems 3.

Original: In Theorem 6, we further show the rationality of our task relatedness assumption when the sample space Z is a complete separable metric space.
Revised: In Theorem 6, we further show the rationality of our task relatedness assumption when the sample space Z is a complete separable metric space.

Original: Given that (Baxter, 2000) also assumes Z to be a separable metric space to obtain theoretical results, our derived meta learning bound in Theorem 3, albeit depending on a slightly stronger assumption, is a non-trivial enhancement.
Revised: Given that (Baxter, 2000) also assumes Z to be a separable metric space to obtain theoretical results, our derived meta learning bound in Theorem 3, albeit depending on a slightly stronger assumption, is a non-trivial enhancement.

Original: Nevertheless, we admit that our results also rely on the closure property of the function class H l .
Revised: Nevertheless, we admit that our results also rely on the closure property of the function class H l .

Original:  Our covering number results (e g., Theorem 4) are based on the metric deﬁned w.r.t.
Revised: (2) Our covering number results (e.g., Theorem 4) are based on the metric deﬁned w.r.t.

Original: the empirical measure, instead of the abstract measure in Baxter’s work (e g., Theorem 6 in (Baxter, 2000)).
Revised: the empirical measure, instead of the abstract measure in Baxter’s work (e.g., Theorem 6 in (Baxter, 2000)).

Original: Therefore it is more suitable to combine our results with the recent theoretical results of covering number bounds for deep neural networks in single task learning.
Revised: Therefore it is more suitable to combine our results with the recent theoretical results of covering number bounds for deep neural networks in single task learning.

Original: (3) When bounding the covering number (or say capacity) of the neural network C ( (cid:15) 2 , F ) (i e., Theorem 8 in (Baxter, 2000)), Baxter uses traditional Pseudo-dimension indicator which is developed by (Haussler, 1992), resulting in a parameter
Revised: (3) When bounding the covering number (or say capacity) of the neural network C ( (cid:15) 2 , F ) (i.e., Theorem 8 in (Baxter, 2000)), Baxter uses traditional Pseudo-dimension indicator which is developed by (Haussler, 1992), resulting in a parameter-count-based bounds for meta learning with neural network.

Original: count
Revised: However, this is not suitable for the analysis of modern overparameterized deep networks.

Original: based bounds for meta learning with neural network 
Revised: In this paper, we introduce a new complexity indicator, spectrally-normalized bound for deep neural network (Bartlett et al., 2017), into the meta learning framework.

Original: However
Revised: The obtained bounds for meta learning with deep network for classiﬁcation and regression problems in Claims 1-3 are all independent of the size of total parameters, outperforming the results in Theorem 8 of (Baxter, 2000).

Original: this is not suitable for 
Revised: Remark 2 ( The three main technical difﬁculties when applying spectrally-normalized margin bounds for meta-learning with deep neural networks )

Original:  the analysis of modern overparameterized deep  networks.
Revised: Now, we give more explanations about the three main technical difﬁculties when applying spectrallynormalized margin bounds (Bartlett et al., 2017) for meta-learning with deep neural networks.

Original: In
Revised: (i)

Original: this paper, we introduce a new complexity indicator, spectrally-normalized bound for deep neural network (Bartlett et al., 2017), into the meta learning framework. The obtained bounds for meta learning with deep network for classiﬁcation and regression problems in Claims 1-3 are all independent of  the size of total parameters, outperforming the results in Theorem 8 of (Baxter, 2000).
Revised: We need to remove the second covering number complexity (i.e. the covering number complexity of H ∗ ) in the meta-learning bound in Eq. (3) in Section C.1 (i.e. the original bound in Theorem 4 of (Baxter, 2000)).

Original: C
Revised: Since such covering number complexity is deﬁned (and can only be deﬁned) based on the distance with respect to the abstract measure (instead of the empirical measure, see Deﬁnitions 3-4 in (Baxter, 2000)), we cannot use any modern theoretical results in deep learning (e.g., the spectral norm of the neural networks in (Bartlett et al., 2017), the compression bound in (Arora et al., 2018) and the ALL-layer margin in (Wei & Ma, 2020)) but the traditional VC-dimension to bound such covering number complexity, hence leading to the vacuous parameter-count-based bounds for deep neural networks. The aforementioned challenge motivated us to propose the Π -relatedness notation to measure the similarity between different tasks and ﬁnally removed the second covering number complexity in Eq. (3) to obtain our main generalization bound in Theorem 3 that can fully utilize the whole n ∗ m training samples.

Revised: (ii)

Revised: For the ﬁrst covering number complexity in Eq.

Revised: (3) (which is still deﬁned based on the distance w.r.t.

Revised: the abstract measure in Baxter’s original paper), we still need to transform it into the complexity deﬁned on the distance w.r.t.

Revised: the empirical measure with our proposed techniques in Propositions 1-2 (also see our Deﬁnition 6), hence we can extend our generalization bound in Theorem 3 to the representation setting and make our meta-learning bound with representation learning (i.e. our Theorem 4) much easier to be combined with modern theoretical results (Bartlett et al., 2017; Arora et al., 2018; Wei & Ma, 2020) for deep neural network in single-task learning.

Revised: (iii)

Revised: The last difﬁculty is to connect the covering number complexity in meta-learning setting (e.g.

Revised: see our Deﬁnition 6, w.r.t.

Revised: the empirical measure) with the spectrally-normalized based covering number complexity from Bartlett et al.

Revised: (2017) (i.e. Lemmas 4-5 in our Appendix B.4).

Revised: Such difﬁculty is overcome by using the Lipschitness property of neural networks and the theoretical properties of covering number (see the proof of Theorem 5 in Appendix B.4).

Original:  D ETAILED C OMPARISON WITH T ASK R ELATEDNESS N OTION OF (B EN -D AVID & S CHULLER , 2003)
Revised: C. D ETAILED C OMPARISON WITH T ASK R ELATEDNESS N OTION OF (B EN -D AVID & S CHULLER , 2003)

Original: In this section, we detail more distinctions between our proposed almost Π -relatedness notation and the task relatedness notion deﬁned in (Ben-David & Schuller, 2003).
Revised: In this section, we detail more distinctions between our proposed almost Π -relatedness notation and the task relatedness notion deﬁned in (Ben-David & Schuller, 2003).

Original: We ﬁrst recall this previous concept in (Ben-David & Schuller, 2003, deﬁnition 2.1).
Revised: We ﬁrst recall this previous concept in (Ben-David & Schuller, 2003, deﬁnition 2.1).

Original: Deﬁnition 11 (Ben-David & Schuller, 2003)
Revised: Deﬁnition 11 (Ben-David & Schuller, 2003)

Original: Let F be a set of transformations f : X → X , and let P 1 , P 2 be probability distributions over X × { 0 , 1 } .
Revised: Let F be a set of transformations f : X → X , and let P 1 , P 2 be probability distributions over X × { 0 , 1 } .

Original: We say that P 1 , P 2 are F -related distributions if there exists some f ∈ F such that for any T ⊂ X × { 0 , 1 } , T is P 1 -measurable iff f
Revised: We say that P 1 , P 2 are F -related distributions if there exists some f ∈ F such that for any T ⊂ X × { 0 , 1 } , T is P 1 -measurable iff f

Original: [ T ] = { ( f ( x ) , b ) | ( x, b ) ∈ T } is P 2 -measurable and P 1 ( T )
Revised: [ T ] = { ( f ( x ) , b ) | ( x, b ) ∈ T } is P 2 -measurable and P 1 ( T )

Original: =
Revised: =

Original: P 2 ( f [ T ]) .
Revised: P 2 ( f [ T ]) .

Original: Discussion .
Revised: Discussion .

Original: Note that in Deﬁnition 11, the condition P 1 ( T ) =
Revised: Note that in Deﬁnition 11, the condition P 1 ( T ) =

Original: P 2 ( f [ T ]) means that P 1 ( T ) =
Revised: P 2 ( f [ T ]) means that P 1 ( T ) =

Original: P 2 ◦ ( f × I )
Revised: P 2 ◦ ( f × I )

Original: [ T ] for all T ⊂ X × { 0 , 1 } .
Revised: [ T ] for all T ⊂ X × { 0 , 1 } .

Original: Therefore P 1 = P 2 ◦ ( f × I ) and hence f
Revised: Therefore P 1 = P 2 ◦ ( f × I ) and hence f

Original: × I is a (bijective) measure-preserving transformation on the product space X ×{ 0 , 1 } .
Revised: × I is a (bijective) measure-preserving transformation on the product space X ×{ 0 , 1 } .

Original: Recall the deﬁnition of our proposed task relatedness concept in Deﬁnition 3 in the main paper.
Revised: Recall the deﬁnition of our proposed task relatedness concept in Deﬁnition 3 in the main paper.

Original: It’s not difﬁcult to see that there are two main differences between these two task-relatedness notions: (i)
Revised: It’s not difﬁcult to see that there are two main differences between these two task-relatedness notions: (i)

Original: The bijective transformation f in Deﬁnition 11 of David’s work is deﬁned between the input space X and the input space X .
Revised: The bijective transformation f in Deﬁnition 11 of David’s work is deﬁned between the input space X and the input space X .

Original: While in our work, the bijective transformation π in Deﬁnition 3 of the main paper is deﬁned between the sample space
Revised: While in our work, the bijective transformation π in Deﬁnition 3 of the main paper is deﬁned between the sample space

Original: Z (= X × Y ) and the sample space Z .
Revised: Z (= X × Y ) and the sample space Z .

Original: Note that the bijective transformation f in Deﬁnition 11 can be viewed as the bijective transformation f
Revised: Note that the bijective transformation f in Deﬁnition 11 can be viewed as the bijective transformation f

Original: ×
Revised: ×

Original: I on the product space X × Y , where I is the identity map on the output space Y .
Revised: I on the product space X × Y , where I is the identity map on the output space Y .

Original: Therefore the transformation given by (Ben-David & Schuller, 2003) can be considered as a special case of our proposed taskrelatedness concept.
Revised: Therefore the transformation given by (Ben-David & Schuller, 2003) can be considered as a special case of our proposed taskrelatedness concept.

Original: (ii) Furthermore, the bijective transformation f given in Deﬁnition 11 is a function map imposed on the whole input space X , whereas our proposed bijective transformation π is imposed on the almost whole sample space
Revised: (ii) Furthermore, the bijective transformation f given in Deﬁnition 11 is a function map imposed on the whole input space X , whereas our proposed bijective transformation π is imposed on the almost whole sample space

Original: Z excluding a measure-zero set.
Revised: Z excluding a measure-zero set.

Original: In other words, our proposed π is a bijective map between Z \ N and Z \ N 1 , where N, N 1 are P -measure zero and P 1 -measure zero sets, respectively.
Revised: In other words, our proposed π is a bijective map between Z \ N and Z \ N 1 , where N, N 1 are P -measure zero and P 1 -measure zero sets, respectively.

Original: Such design will allow more ﬂexibility of the choice of the bijective map π , and can help us theoretically guarantee the existence of the almost bijective function on the whole sample space Z (see more explanations in Section B.5).
Revised: Such design will allow more ﬂexibility of the choice of the bijective map π , and can help us theoretically guarantee the existence of the almost bijective function on the whole sample space Z (see more explanations in Section B.5).

Original: In contrast, (BenDavid & Schuller, 2003) does not provide a rigorous demonstration of the existence of its deﬁned bijective transformation f .
Revised: In contrast, (BenDavid & Schuller, 2003) does not provide a rigorous demonstration of the existence of its deﬁned bijective transformation f .

Original: Actually, it is not easy to ﬁnd such bijective transformation f on the input space X , which meanwhile satisﬁes the condition that the bijective transformation f
Revised: Actually, it is not easy to ﬁnd such bijective transformation f on the input space X , which meanwhile satisﬁes the condition that the bijective transformation f

Original: × I is a measure-preserving bijective on the product space X × Y .
Revised: × I is a measure-preserving bijective on the product space X × Y .

Original: In this sense, our work can be considered as the extension version of that in (Ben-David & Schuller, 2003).
Revised: In this sense, our work can be considered as the extension version of that in (Ben-David & Schuller, 2003).

Original: As for our proposed task-relatedness concept, the explicit form of the bijective transformation π in
Revised: As for our proposed task-relatedness concept, the explicit form of the bijective transformation π in

Original: Theorem 6 of main paper can be bound in (Bogachev, 2007, Theorem 6.5.7).
Revised: Theorem 6 of main paper can be found in (Bogachev, 2007, Theorem 6.5.7).

Original: In that result, Bogachev takes π as the composition of functions (cid:80) ∞ n =1 I E n and ( (cid:80) ∞ n =1 I B n ) − 1 , where E n belongs to the σalgebra B µ over the sample space Z associated with the measure µ , B n belongs to the σ -algebra B ν over the sample space Z associated with the measure
Revised: In that result, Bogachev takes π as the composition of functions (cid:80) ∞ n =1 I E n and ( (cid:80) ∞ n =1 I B n ) − 1 , where E n belongs to the σalgebra B µ over the sample space Z associated with the measure µ , B n belongs to the σ -algebra B ν over the sample space Z associated with the measure

Original: ν , I is the indicator function from the σ -algebra over Z to the interval [0 , 1] .
Revised: ν , I is the indicator function from the σ -algebra over Z to the interval [0 , 1] .

Original: A simpler (but not rigorous) example just for intuitive comprehension can be found in the regression problem, where the sample space is Z = R d +1 =
Revised: A simpler (but not rigorous) example just for intuitive comprehension can be found in the regression problem, where the sample space is Z = R d +1 =

Original: X
Revised: X

Original: × Y = R d × R , the focused measure is Lebesgue measure and the bijective function π : π ( (cid:126)z ) =
Revised: × Y = R d × R , the focused measure is Lebesgue measure and the bijective function π : π ( (cid:126)z ) =

Original: (cid:126)z + (cid:126)a,(cid:126)a ∈ Z is a translation over high-dimensional Euclidean space.
Revised: (cid:126)z + (cid:126)a,(cid:126)a ∈ Z is a translation over high-dimensional Euclidean space.