Original: Fine-Grained Analysis of Stability and Generalization for Modern Meta Learning Algorithms
Revised: Fine-Grained Analysis of Stability and Generalization for Modern Meta Learning Algorithms

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The support/query episodic training strategy has been widely applied in modernmeta learning algorithms.
Revised: The support/query episodic training strategy has been widely applied in modernmeta learning algorithms.

Original: Supposing the n training episodes and the test episodesare sampled independently from the same environment, previous work has deriveda generalization bound of O (1 / √ n ) for smooth non-convex functions via algorithmic stability analysis.
Revised: Supposing the n training episodes and the test episodesare sampled independently from the same environment, previous work has deriveda generalization bound of O (1 / √ n ) for smooth non-convex functions via algorithmic stability analysis.

Original: In this paper, we provide fine-grained analysis of stabilityand generalization for modern meta learning algorithms by considering more general situations.
Revised: In this paper, we provide fine-grained analysis of stabilityand generalization for modern meta learning algorithms by considering more general situations.

Original: Firstly, we develop matching lower and upper stability bounds formeta learning algorithms with two types of loss functions: (1) nonsmooth convexfunctions with α -Hölder continuous subgradients ( α ∈ [0 , 1)) ; (2) smooth (including convex and non-convex) functions.
Revised: Firstly, we develop matching lower and upper stability bounds formeta learning algorithms with two types of loss functions: (1) nonsmooth convexfunctions with α -Hölder continuous subgradients ( α ∈ [0 , 1)) ; (2) smooth (including convex and non-convex) functions.

Original: Our tight stability bounds show that, inthe nonsmooth convex case, meta learning algorithms can be inherently less stablethan in the smooth convex case.
Revised: Our tight stability bounds show that, inthe nonsmooth convex case, meta learning algorithms can be inherently less stablethan in the smooth convex case.

Original: For the smooth non-convex functions, our stabilitybound is sharper than the existing one, especially in the setting where the numberof iterations is larger than the number n of training episodes.
Revised: For the smooth non-convex functions, our stabilitybound is sharper than the existing one, especially in the setting where the numberof iterations is larger than the number n of training episodes.

Original: Secondly, we deriveimproved generalization bounds for meta learning algorithms that hold with highprobability.
Revised: Secondly, we deriveimproved generalization bounds for meta learning algorithms that hold with highprobability.

Original: Specifically, we first demonstrate that, under the independent episodeenvironment assumption, the generalization bound of O (1 / √ n ) via algorithmicstability analysis is near optimal.
Revised: Specifically, we first demonstrate that, under the independent episodeenvironment assumption, the generalization bound of O (1 / √ n ) via algorithmicstability analysis is near optimal.

Original: To attain faster convergence rate, we show howto yield a  generalization bound of O (ln n/n ) with additional curvature conditionofthe loss function.
Revised: To attain faster convergence rate, we show how toyield a deformed generalization bound of O (ln n/n ) with the curvature conditionof loss functions.

Original: Finally, we establish a generalization bound for meta learningwith dependent episodes whose dependency relation is characterized by a graph.
Revised: Finally, we obtain a generalization bound for meta learningwith dependent episodes whose dependency relation is characterized by a graph.

Original: Experiments on regression problems are conducted to verify our theoretical results.
Revised: Experiments on regression problems are conducted to verify our theoretical results.

Original:  Introduction
Revised:  Introduction

Original: The last decade has witnessed the success of deep learning techniques in machine learning community[28, 25, 11].
Revised: The last decade has witnessed the success of deep learning techniques in machine learning community[28, 25, 11].

Original: However, the need of large amount of annotated data hinders their application in real-lifescenarios.
Revised: However, the need of large amount of annotated data hinders their application in real-lifescenarios.

Original: To alleviate this issue, meta learning [4] which employs knowledge from past tasks tofacilitate adaptation to the new task, has emerged as a promising direction to reduce annotation cost.
Revised: To alleviate this issue, meta learning [4] which employs knowledge from past tasks tofacilitate adaptation to the new task, has emerged as a promising direction to reduce annotation cost.

Original: Traditional meta learning algorithms directly minimize the empirical error over all samples in thetraining tasks [3, 33, 34, 37, 35].
Revised: Traditional meta learning algorithms directly minimize the empirical error over all samples in thetraining tasks [3, 33, 34, 37, 35].

Original: To improve the generalization ability of meta learning algorithms,recent works propose the support/query (S/Q) episodic training strategy [44, 20, 40].
Revised: To improve the generalization ability of meta learning algorithms,recent works propose the support/query (S/Q) episodic training strategy [44, 20, 40].

Original: Specifically, inmodern meta learning algorithms, each episode/task is split into two non-overlapped parts: support setand query set.
Revised: Specifically, inmodern meta learning algorithms, each episode/task is split into two non-overlapped parts: support setand query set.

Original: The support set is used to learn a hypothesis, and the query set is used to measure theperformance of the learned hypothesis on that episode.
Revised: The support set is used to learn a hypothesis, and the query set is used to measure theperformance of the learned hypothesis on that episode.

Original: Therefore, the S/Q episodic strategy regardseach task as a training instance and updates the meta learning model by implementing episode-levelstochastic gradient descent (SGD).
Revised: Therefore, the S/Q episodic strategy regardseach task as a training instance and updates the meta learning model by implementing episode-levelstochastic gradient descent (SGD).

Original: Supposing the n training episodes and the test episodes aresampled independently from the same environment, previous work [8] has derived a high-probability generalization bound of O (1 / √ n ) for modern meta learning.
Revised: Supposing the n training episodes and the test episodes aresampled independently from the same environment, previous work [8] has derived a high-probability generalization bound of O (1 / √ n ) for modern meta learning.

Original: Such bound is obtained via algorithmicstability analysis [6] for smooth non-convex loss functions.
Revised: Such bound is obtained via algorithmicstability analysis [6] for smooth non-convex loss functions.

Original: However, it is still unknown whether suchgeneralization bound of O (1 / √ n ) is optimal, and whether we can obtain sharper bounds for modernmeta learning.
Revised: However, it is still unknown whether suchgeneralization bound of O (1 / √ n ) is optimal, and whether we can obtain sharper bounds for modernmeta learning.

Original: Further, there is still lack of comprehensive comparisons between the bounds obtainedvia S/Q episodic training and the bounds obtained via traditional empirical risk minimizing (ERM).
Revised: Further, there is still lack of comprehensive comparisons between the bounds obtainedvia S/Q episodic training and the bounds obtained via traditional empirical risk minimizing (ERM).

Original: In this work, we will address the above problems via algorithmic stability analysis.
Revised: In this work, we will address the above problems via algorithmic stability analysis.

Original: Algorithmicstability, roughly speaking, bounds the change in the model output by the algorithm when a single datain the dataset is replaced.
Revised: Algorithmicstability, roughly speaking, bounds the change in the model output by the algorithm when a single datain the dataset is replaced.

Original: Our goal is to provide fine-grained analysis of stability and generalizationfor modern meta learning algorithms by considering more general situations.
Revised: Our goal is to provide fine-grained analysis of stability and generalizationfor modern meta learning algorithms by considering more general situations.

Original: Firstly, we developmatching lower and upper stability bounds for meta learning algorithms with two types of lossfunctions: (1) nonsmooth convex functions with α -Hölder continuous subgradients where 0 ≤ α < 1 ;(2) smooth (including convex and non-convex) functions.
Revised: Firstly, we developmatching lower and upper stability bounds for meta learning algorithms with two types of lossfunctions: (1) nonsmooth convex functions with α -Hölder continuous subgradients where 0 ≤ α < 1 ;(2) smooth (including convex and non-convex) functions.

Original: Our tight stability bounds demonstrate that,in the nonsmooth convex case, modern meta learning algorithms can be less stable than in the smoothconvex case.
Revised: Our tight stability bounds demonstrate that,in the nonsmooth convex case, modern meta learning algorithms can be less stable than in the smoothconvex case.

Original: In particular, the lower stability bound for nonsmooth convex functions is vacuouseven if we train modern meta learning algorithms with a relatively small constant step size in SGD.
Revised: In particular, the lower stability bound for nonsmooth convex functions is vacuouseven if we train modern meta learning algorithms with a relatively small constant step size in SGD.

Original: In the smooth non-convex case, our derived bound is sharper than the existing one [8], especiallyin the setting where the number of SGD iterations is larger than the number n of training episodes.
Revised: In the smooth non-convex case, our derived bound is sharper than the existing one [8], especiallyin the setting where the number of SGD iterations is larger than the number n of training episodes.

Original: Secondly, we provide high-probability generalization bounds for modern meta learning algorithmswith the aforementioned two types of loss functions.
Revised: Secondly, we provide high-probability generalization bounds for modern meta learning algorithmswith the aforementioned two types of loss functions.

Original: Specifically, we first demonstrate that, underthe independent episode environment assumption, the bound of O (1 / √ n ) is near optimal and isindependent of the sample size m per episode.
Revised: Specifically, we first demonstrate that, underthe independent episode environment assumption, the bound of O (1 / √ n ) is near optimal and isindependent of the sample size m per episode.

Original: We thus show that, in terms of the sharpness of thegeneralization bounds, the S/Q episodic training strategy is superior to the traditional ERM strategyfor meta learning (see Remark 6).
Revised: We thus show that, in terms of the sharpness of thegeneralization bounds, the S/Q episodic training strategy is superior to the traditional ERM strategyfor meta learning (see Remark 6).

Original: To obtain faster convergence rate, we next show how to yield ageneralization bound of O (ln n/n ) with additional curvature assumption (i.e., the Polyak-Łojasiewiczcondition [45]) of the objective function.
Revised: To obtain faster convergence rate, we next show how to yielda deformed generalization bound of O (ln n/n ) with additional curvature assumption (i.e., PolyakŁojasiewicz condition [45]) of the loss function.

Original: Finally, we leverage the graph approximation technique[46] to obtain a bound for meta learning with dependent episodes whose dependency relation can becharacterized by a graph.
Revised: Finally, we use the graph approximation technique[46] to obtain a bound for meta learning with dependent episodes whose dependency relation can becharacterized by a graph.

Original: To the best of our knowledge, this is the first bound that captures how thedependency between episodes can affect the generalization performance of meta learning algorithms.
Revised: To the best of our knowledge, this is the first bound that captures how thedependency between episodes can affect the generalization behavior of meta learning algorithms.

Original: Overall, our contributions are four-fold: (1) We provide matching lower and upper stability boundsfor modern meta learning algorithms with general loss functions.
Revised: Overall, our contributions are four-fold: (1) We provide matching lower and upper stability boundsfor modern meta learning algorithms with general loss functions.

Original: The stability bound for nonsmoothconvex functions implies that modern meta learning algorithms are not stable enough; and the stabilitybound in the smooth non-convex case is sharper than the existing one.
Revised: The stability bound for nonsmoothconvex functions implies that modern meta learning algorithms are not stable enough; and the stabilitybound in the smooth non-convex case is sharper than the existing one.

Original: (2) We develop a near-optimalhigh-probability bound of O (1 / √ n ) on the transfer error for meta learning.
Revised: (2) We develop a near-optimalhigh-probability bound of O (1 / √ n ) on the transfer error in meta learning.

Original: Such bound is furtherused to reveal the advantage of the S/Q episodic strategy for meta learning over the traditional ERMstrategy.
Revised: Such bound is also used toreveal the advantage of the S/Q episodic strategy for meta learning over the traditional ERM strategy.(3)

Original: (3) We derive a  generalization bound of O (ln n/n ) with additional curvature condition ofloss functions.
Revised: We derive a deformed generalization bound of O (ln n/n ) with additional curvature condition ofloss functions.

Original: (4) We obtain the first bound for meta learning with dependent episodes.
Revised: (4) We obtain the first bound for meta learning with dependent episodes.

Original: Experimentson regression problems are conducted to validate the convergence of our generalization bounds.
Revised: Experimentson regression problems are conducted to validate the convergence of our generalization bounds.

Original:  Related Work
Revised:  Related Work

Original: Algorithmic Stability Theory .
Revised: Algorithmic Stability Theory .

Original: Algorithmic stability analysis is an important tool to providetheoretical guarantee for the learnability of machine learning models.
Revised: Algorithmic stability analysis is an important tool to providetheoretical guarantee for the learnability of machine learning models.

Original: [39] has shown that thereare non-trivial problems where traditional uniform convergence analysis (i.e., empirical processtheory [43]) fails to hold, but stability can be identified as the sufficient and necessary conditionfor learnability.
Revised: [39] has shown that thereare non-trivial problems where traditional uniform convergence analysis (i.e., empirical processtheory [43]) fails to hold, but stability can be identified as the sufficient and necessary conditionfor learnability.

Original: There are two main groups in this direction: (1) The first group develop differentnotations of stability and connect their relation to the generalization of specific algorithms.
Revised: There are two main groups in this direction: (1) The first group develop differentnotations of stability and connect their relation to the generalization of specific algorithms.

Original: Amongthem, uniform stability is the most widely used notation and has been utilized to analyze the stabilityand generalization of regularized ERM algorithms [6].
Revised: Amongthem, uniform stability is the most widely used notation and has been utilized to analyze the stabilityand generalization of regularized ERM algorithms [6].

Original: Hypothesis stability is a weaker notation andhas been used to show the stability of k -Nearest Neighbor model [12].
Revised: Hypothesis stability is a weaker notation andhas been used to show the stability of k -Nearest Neighbor model [12].

Original: Both of the above algorithmicstability notions have been extended to the randomized setting to demonstrate the stability of Baggingalgorithm [15].
Revised: Both of the above algorithmicstability notions have been extended to the randomized setting to demonstrate the stability of Baggingalgorithm [15].

Original: In recent years, different notations have been employed to analyze the stabilityand in-expectation generalization bounds of stochastic gradient descent method, which includeuniform stability [24], on-average stability [29], uniform argument stability [32, 2], on-averagemodel stability [30] and locally elastic stability [10].
Revised: In recent years, different notations have been employed to analyze the stabilityand in-expectation generalization bounds of stochastic gradient descent method, which includeuniform stability [24], on-average stability [29], uniform argument stability [32, 2], on-averagemodel stability [30] and locally elastic stability [10].

Original: The second group aims to derive tighthigh-probability generalization bounds for uniformly stable algorithm in single-task learning.
Revised: The second group aims to derive tighthigh-probability generalization bounds for uniformly stable algorithm in single-task learning.

Original: Thefirst high-probability bound has been derived by [6], and has been improved in [18].
Revised: Thefirst high-probability bound has been derived by [6], and has been improved in [18].

Original: Recently, nearlyoptimal generalization bounds of O (1 / √ n ) have been established in [19, 7], where n is the size oftraining dataset.
Revised: Recently, nearlyoptimal generalization bounds of O (1 / √ n ) have been established in [19, 7], where n is the size oftraining dataset.

Original: Further, with additional Bernstein condition, [27] derives a generalization boundof O (1 /n ) .
Revised: Further, with additional Bernstein condition, [27] derives a generalization boundof O (1 /n ) .

Original: In this work, we aim to provide tight stability bounds and improved high-probability generalization bounds for episodic meta learning algorithms.
Revised: In this work, we aim to provide tight stability bounds and improved high-probability generalization bounds for episodic meta learning algorithms.

Original: The key step to achieve our goal is toreveal the equivalence of notations between single-task learning and episodic meta learning, hencewe can extend the demonstration techniques from [7, 27, 46] to the episodic meta learning setting.
Revised: The key step to achieve our goal is toreveal the equivalence of notations between single-task learning and episodic meta learning, hencewe can extend the demonstration techniques from [7, 27, 46] to the episodic meta learning setting.

Original: Generalization Bounds for Meta Learning .
Revised: Generalization Bounds for Meta Learning .

Original: Supposing the n training tasks and the novel tasksare sampled independently from the same environment, [4] derives the first generalization bound onthe transfer error over the novel task for meta learning.
Revised: Supposing the n training tasks and the novel tasksare sampled independently from the same environment, [4] derives the first generalization bound onthe transfer error over the novel task for meta learning.

Original: Under the independent task environmentassumption, we can categorize existing transfer error bounds into three main groups: (1) transfererror bounds of hypothesis space.
Revised: Under the independent task environmentassumption, we can categorize existing transfer error bounds into three main groups: (1) transfererror bounds of hypothesis space.

Original: Such bounds are always achieved via covering number analysis [4]or VC theory [5], and hence are always dimension-dependent.
Revised: Such bounds are always achieved via covering number analysis [4]or VC theory [5], and hence are always dimension-dependent.

Original: The latest upper bound in this groupis of O (1 / √ nm + 1 / √ m ) in [22, Theorem 5], where m is the sample size per task.
Revised: The latest upper bound in this groupis of O (1 / √ nm + 1 / √ m ) in [22, Theorem 5], where m is the sample size per task.

Original: (2) transfererror bounds of the hyper-distribution of prior.
Revised: (2) transfererror bounds of the hyper-distribution of prior.

Original: Such bounds are obtained via PAC-Bayes analysis[37, 38, 13].
Revised: Such bounds are obtained via PAC-Bayes analysis[37, 38, 13].

Original: The tightest bound in this group is of order O (1 / √ n + 1 /m ) in [17, Theorem 3].
Revised: The tightest bound in this group is of order O (1 / √ n + 1 /m ) in [17, Theorem 3].

Original: (3)transfer error bounds of the algorithm.
Revised: (3)transfer error bounds of the algorithm.

Original: Such bounds are obtained via algorithmic stability analysis[33, 1].
Revised: Such bounds are obtained via algorithmic stability analysis[33, 1].

Original: The tightest bound in this group is of O (1 / √ n ) in [8, Theorem 4] for episodic meta learningalgorithms.
Revised: The tightest bound in this group is of O (1 / √ n ) in [8, Theorem 4] for episodic meta learningalgorithms.

Original: Detailed comparisons between different transfer error bounds can be found in Table
Revised: Detailed comparisons between different transfer error bounds can be found in Table

Original: A.2of Appendix A. There also exist other works without the task environment assumption.
Revised: A.2of Appendix A. There also exist other works without the task environment assumption.

Original: Instead, theychoose to bound the excess risk on the novel task by proposing task-similarity measurement [14, 42],or using the total variation distance as the diversity measurement between novel task and trainingtasks [16].
Revised: Instead, theychoose to bound the excess risk on the novel task by proposing task-similarity measurement [14, 42],or using the total variation distance as the diversity measurement between novel task and trainingtasks [16].

Original: In this work, we take the task environment assumption and follow the work of [8].
Revised: In this work, we take the task environment assumption and follow the work of [8].

Original: Ourfirst improvement is to demonstrate that the bound of O (1 / √ n ) is near optimal for modern metalearning.
Revised: Ourfirst improvement is to demonstrate that the bound of O (1 / √ n ) is near optimal.

Original: Besides, we show how to obtain a  bound of O (ln n/n ) with additional curvature assumptionof the loss function.
Revised: Besides, we showhow to obtain a deformed generalization bound of O (ln n/n ) with additional curvature assumptionof the loss function.

Original: Further, we derive a bound with dependent training episodes, revealing howdependency relation between episodes can affect the generalization of meta learning algorithms.
Revised: Further, we derive a bound with dependent training episodes, revealing howdependency relation between episodes can affect the generalization of meta learning algorithms.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: In supervised learning, a sample space Z
Revised: In supervised learning, a sample space Z

Original: =
Revised: =

Original: X ×Y is a product space of an input space X and an output 118 space
Revised: X ×Y is a product space of an input space X and an output 118 space

Original: Y .
Revised: Y .

Original: H =
Revised: H =

Original: { h w : w ∈ W} is the hypothesis space where the hypothesis h w ∈ H is parameterized 119 by parameter w in the parameter space
Revised: { h w : w ∈ W} is the hypothesis space where the hypothesis h w ∈ H is parameterized 119 by parameter w in the parameter space

Original: W .
Revised: W .

Original: A measurable function f : H × Z → [0 , M ]( M > 0) is 120 defined as a nonnegative and bounded loss function, whose loss of a hypothesis h w over a sample zis denoted by f ( h w , z ) or f ( w, z ) .
Revised: A measurable function f : H × Z → [0 , M ]( M > 0) is 120 defined as a nonnegative and bounded loss function, whose loss of a hypothesis h w over a sample zis denoted by f ( h w , z ) or f ( w, z ) .

Original: Let M 1 ( A ) denote the set of probability measures over the set A .
Revised: Let M 1 ( A ) denote the set of probability measures over the set A .

Original: Loss Functions .
Revised: Loss Functions .

Original: Throughout we assume that the parameter space W ⊂ R d .
Revised: Throughout we assume that the parameter space W ⊂ R d .

Original: Thus, we use unambiguously || · || = || · || 2 as the Euclidean norm.
Revised: Thus, we use unambiguously || · || = || · || 2 as the Euclidean norm.

Original: Let Proj W be the Euclidean projection onto W , which is 124 nonexpansive
Revised: Let Proj W be the Euclidean projection onto W , which is 124 nonexpansive

Original: ||
Revised: ||

Original: Proj W ( u ) − Proj W ( v ) ||
Revised: Proj W ( u ) − Proj W ( v ) ||

Original: ≤ || u − v || .
Revised: ≤ || u − v || .

Original: For any fixed z ∈ Z , a function f ( · , z ) : W → R 125 is convex if for all u, v ∈ W , f ( u, z ) ≥ f
Revised: For any fixed z ∈ Z , a function f ( · , z ) : W → R 125 is convex if for all u, v ∈ W , f ( u, z ) ≥ f

Original: ( v, z ) + ⟨ g, u − v ⟩ , where g ∈ ∂f ( v, z ) , and ∂f ( v, z ) 126 denotes the set of subgradients of f ( · , z ) at v .
Revised: ( v, z ) + ⟨ g, u − v ⟩ , where g ∈ ∂f ( v, z ) , and ∂f ( v, z ) 126 denotes the set of subgradients of f ( · , z ) at v .

Original: Let ∂ 

Revised: Let ∂

Original: f ( v, z ) denote the subgradient with the leastnorm.
Revised: f ( v, z ) denote the subgradient with the leastnorm.

Original: If f ( · , z ) is differentiable, ∂f ( · , z ) denotes the gradient of f ( · , z ) , i.e., ∂f ( · , z ) = {∇ f ( · , z ) } .
Revised: If f ( · , z ) is differentiable, ∂f ( · , z ) denotes the gradient of f ( · , z ) , i.e., ∂f ( · , z ) = {∇ f ( · , z ) } .

Original: For any z ∈ Z , f ( · , z ) is σ -Lipschitz if ∀ u, v ∈ W , | f ( u, z ) − f ( v, z ) | ≤ σ || u − v || .
Revised: For any z ∈ Z , f ( · , z ) is σ -Lipschitz if ∀ u, v ∈ W , | f ( u, z ) − f ( v, z ) | ≤ σ || u − v || .

Original: For any z ∈ Z , 129 f ( · , z ) is G -smooth
Revised: For any z ∈ Z , 129 f ( · , z ) is G -smooth

Original: if ∀ u, v ∈ W , || ∂f ( u, z ) − ∂f ( v, z ) || ≤ G || u − v || .
Revised: if ∀ u, v ∈ W , || ∂f ( u, z ) − ∂f ( v, z ) || ≤ G || u − v || .

Original: We also give the definition 130 of function with ( α, G ) -Hölder continuous subgradient as follows.
Revised: We also give the definition 130 of function with ( α, G ) -Hölder continuous subgradient as follows.

Original: We may refer to such functions as( α, G ) -Hölder smooth or α -Hölder smooth function for simplicity when the context is clear.
Revised: We may refer to such functions as( α, G ) -Hölder smooth or α -Hölder smooth function for simplicity when the context is clear.

Original: Definition 1 Let G > 0 , α ∈ [0 , 1] .
Revised: Definition 1 Let G > 0 , α ∈ [0 , 1] .

Original: For any z ∈ Z , a function f ( · , z ) is called ( α, G ) -Höldersmooth if its subgradient is ( α, G ) -Hölder continuous, i.e., ∂f ( · , z ) satisfies the following conditions:
Revised: For any z ∈ Z , a function f ( · , z ) is called ( α, G ) -Höldersmooth if its subgradient is ( α, G ) -Hölder continuous, i.e., ∂f ( · , z ) satisfies the following conditions:

Original: [Equation]
Revised: [Equation]

Original: If (1) holds with α = 1 , then f ( · , z ) is a G -smooth function; if (1) holds with α = 0 , this implies thesubgradient boundedness of f ( · , z ) .
Revised: If (1) holds with α = 1 , then f ( · , z ) is a G -smooth function; if (1) holds with α = 0 , this implies thesubgradient boundedness of f ( · , z ) .

Original: The examples of loss functions in machine learning satisfying(1) include the q -norm hinge-loss f ( w, z ) = (cid:0) max(0 , 1 − y ⟨ w, x ⟩ ) (cid:1) q for classification and the q -thpower absolute distance loss f ( w, z )
Revised: The examples of loss functions in machine learning satisfying(1) include the q -norm hinge-loss f ( w, z ) = (cid:0) max(0 , 1 − y ⟨ w, x ⟩ ) (cid:1) q for classification and the q -thpower absolute distance loss f ( w, z )

Original: =
Revised: =

Original: | y − ⟨ w, x ⟩| q for regression, whose subgradients are both 139 ( q − 1 , C ) -Hölder continuous for some C > 0 if q ∈ [1 , 2] (see [9]).
Revised: | y − ⟨ w, x ⟩| q for regression, whose subgradients are both 139 ( q − 1 , C ) -Hölder continuous for some C > 0 if q ∈ [1 , 2] (see [9]).

Original: For ( α, G ) -Hölder smoothfunction, define c α = (1 + 1 /α )
Revised: For ( α, G ) -Hölder smoothfunction, define c α = (1 + 1 /α )

Original: α 1+ α G 11+ α if α ∈ (0 , 1] ; and c α = sup z
Revised: α 1+ α G 11+ α if α ∈ (0 , 1] ; and c α = sup z

Original: ||
Revised: ||

Original: ∂f
Revised: ∂f

Original: (0 , z )
Revised: (0 , z )

Original: ||
Revised: ||

Original: + G , if α = 0 .
Revised: + G , if α = 0 .

Original: Single-Task Learning .
Revised: Single-Task Learning .

Original: The training dataset S =
Revised: The training dataset S =

Original: { z j = ( x j , y j ) } mj =1 is given by m independentdraws from an unknown distribution D on Z (i.e., D ∈ M 1 ( Z ) ).
Revised: { z j = ( x j , y j ) } mj =1 is given by m independentdraws from an unknown distribution D on Z (i.e., D ∈ M 1 ( Z ) ).

Original: An algorithm A takes S as input 143 and outputs a hypothesis A ( S ) in
Revised: An algorithm A takes S as input 143 and outputs a hypothesis A ( S ) in

Original: H .
Revised: H .

Original: The set of such algorithms depends only on H and Z and will 144 be denoted by
Revised: The set of such algorithms depends only on H and Z and will 144 be denoted by

Original: A ( H , Z ) .
Revised: A ( H , Z ) .

Original: In single-task learning, a hypothesis A ( S ) is always obtained by minimizingthe empirical error on S :
Revised: In single-task learning, a hypothesis A ( S ) is always obtained by minimizingthe empirical error on S :

Original: ˆ L ( A ( S ) , S ) ≜ 1 m (cid:80) mj
Revised: ˆ L ( A ( S ) , S ) ≜ 1 m (cid:80) mj

Original: =1 f
Revised: =1 f

Original: ( A ( S ) , z j ) .
Revised: ( A ( S ) , z j ) .

Original: The performance of the returned hypothesis A ( S ) is measured by the expected/generalization error with respect to (w.r.t.)
Revised: The performance of the returned hypothesis A ( S ) is measured by the expected/generalization error with respect to (w.r.t.)

Original: the datadistribution D : L ( A ( S ) , D ) ≜
Revised: the datadistribution D : L ( A ( S ) , D ) ≜

Original: E z ∼ D f
Revised: E z ∼ D f

Original: ( A ( S ) , z ) .
Revised: ( A ( S ) , z ) .

Original: The goal of learning theory is thus to give a (loweror upper) bound on the expected error based on the empirical error on the training dataset S .
Revised: The goal of learning theory is thus to give a (loweror upper) bound on the expected error based on the empirical error on the training dataset S .

Original: Meta Learning .
Revised: Meta Learning .

Original: Following existing theoretical works for meta learning[4, 33, 37, 8], we assumethat the distributions { D i } ni =1 associated with different training tasks are drawn from the same task 151 environment τ , which is a probability distribution over the set of all data distributions on
Revised: Following existing theoretical works for meta learning[4, 33, 37, 8], we assumethat the distributions { D i } ni =1 associated with different training tasks are drawn from the same task 151 environment τ , which is a probability distribution over the set of all data distributions on

Original: Z (i.e., 152 τ ∈ M 1 ( M 1 ( Z )) ).
Revised: Z (i.e., 152 τ ∈ M 1 ( M 1 ( Z )) ).

Original: During the training process, a meta-sample S =
Revised: During the training process, a meta-sample S =

Original: { S i =
Revised: { S i =

Original: S tri ∪ S tsi } ni =1 isavailable, where S tri i.i.d.
Revised: S tri ∪ S tsi } ni =1 isavailable, where S tri i.i.d.

Original: ∼ D Ki of size K is the training set, and S tsi i.i.d.
Revised: ∼ D Ki of size K is the training set, and S tsi i.i.d.

Original: ∼ D qi of size q is the test set ofthe i -th training task.
Revised: ∼ D qi of size q is the test set ofthe i -th training task.

Original: In this work, we assume that K + q = m for notation convenience.
Revised: In this work, we assume that K + q = m for notation convenience.

Original: The trainingset and the test set are also called support set and query set [8], respectively.
Revised: The trainingset and the test set are also called support set and query set [8], respectively.

Original: Let A ( A ( H , Z ) , Z m ) 156 be the set of meta learning algorithms.
Revised: Let A ( A ( H , Z ) , Z m ) 156 be the set of meta learning algorithms.

Original: For any A ∈ A ( A ( H , Z ) , Z m ) , it takes the meta-sample 157 S
Revised: For any A ∈ A ( A ( H , Z ) , Z m ) , it takes the meta-sample 157 S

Original: = { S i } ni =1 as input and outputs an algorithm (inner-task algorithm)
Revised: = { S i } ni =1 as input and outputs an algorithm (inner-task algorithm)

Original: A ( S ) :
Revised: A ( S ) :

Original: ∪ ∞ m =1 Z m → H .
Revised: ∪ ∞ m =1 Z m → H .

Original: The 158 performance of the learned inner-task algorithm is measured by the expectation of the generalizationerror w.r.t.
Revised: The 158 performance of the learned inner-task algorithm is measured by the expectation of the generalizationerror w.r.t.

Original: the task environment τ , which is defined as the transfer error by [33, 8] as follows:
Revised: the task environment τ , which is defined as the transfer error by [33, 8] as follows:

Original: [Equation]
Revised: [Equation]

Original: Actually, the environment τ can define an induced distribution D τ ∈ M 1 ( Z m ) , by setting
Revised: Actually, the environment τ can define an induced distribution D τ ∈ M 1 ( Z m ) , by setting

Original: [Equation]
Revised: [Equation]

Original: m ( F ) for any measurable set F ⊆ Z m .
Revised: m ( F ) for any measurable set F ⊆ Z m .

Original: Define the estimator l ( A ( S ) , S ) ≜ˆ L ( A ( S )( S tr ) , S ts ) , where S = S tr ∪ S ts , S i.i.d.
Revised: Define the estimator l ( A ( S ) , S ) ≜ˆ L ( A ( S )( S tr ) , S ts ) , where S = S tr ∪ S ts , S i.i.d.

Original: ∼ D m .
Revised: ∼ D m .

Original: Then we can rewrite the transfer error asa simple form: er ( A ( S ) , τ ) =
Revised: Then we can rewrite the transfer error asa simple form: er ( A ( S ) , τ ) =

Original: E S ∼ D τ l ( A ( S ) , S ) .
Revised: E S ∼ D τ l ( A ( S ) , S ) .

Original: This means that, the training error l ( A ( S ) , S )is the unbiased version of the transfer error
Revised: This means that, the training error l ( A ( S ) , S )is the unbiased version of the transfer error

Original: er ( A ( S ) , τ ) =
Revised: er ( A ( S ) , τ ) =

Original: E S ∼ D τ l ( A ( S ) , S ) .
Revised: E S ∼ D τ l ( A ( S ) , S ) .

Original: This is similar tothe fact that, in single-task learning, the empirical error f
Revised: This is similar tothe fact that, in single-task learning, the empirical error f

Original: ( A ( S ) , z ) is the unbiased version of thegeneralization error
Revised: ( A ( S ) , z ) is the unbiased version of thegeneralization error

Original: L ( A ( S ) , D ) =
Revised: L ( A ( S ) , D ) =

Original: E z ∼ D f
Revised: E z ∼ D f

Original: ( A ( S ) , z ) .
Revised: ( A ( S ) , z ) .

Original: Therefore, a transfer error bound is formallyequivalent to a single-task generalization error bound under the identifications Z ↔ Z m , f ↔ l , 168 A ↔ A .
Revised: Therefore, a transfer error bound is formallyequivalent to a single-task generalization error bound under the identifications Z ↔ Z m , f ↔ l , 168 A ↔ A .

Original: The relation of the notations between single-task learning and meta learning is listed in
Revised: The relation of the notations between single-task learning and meta learning is listed in

Original: Table B.1 in Appendix B.
Revised: Table B.1 in Appendix B.

Original: In practice, it is difficult to minimize er ( A ( S ) , τ ) directly as we haveno information of the environment distribution τ .
Revised: In practice, it is difficult to minimize er ( A ( S ) , τ ) directly as we haveno information of the environment distribution τ .

Original: Instead, we choose to minimize the followingempirical risk based on the S/Q episodic training strategy.
Revised: Instead, we choose to minimize the followingempirical risk based on the S/Q episodic training strategy.

Original: The goal of meta learning theory is thus togive a bound on the transfer error, based on the empirical multi-task error on the meta-sample S :
Revised: The goal of meta learning theory is thus togive a bound on the transfer error, based on the empirical multi-task error on the meta-sample S :

Original: [Equation]
Revised: [Equation]

Original: Uniform Stability of Meta Learning Algorithms .
Revised: Uniform Stability of Meta Learning Algorithms .

Original: We say two meta-samples S
Revised: We say two meta-samples S

Original: = { S i } ni =1 and 174 S ′ =
Revised: = { S i } ni =1 and 174 S ′ =

Original: { S ′ i } ni
Revised: { S ′ i } ni

Original: =1 are neighboring, denoted by S ≃ S ′ , if they only differ on a single entry, i.e., thereexists i ∈ [ n ]
Revised: =1 are neighboring, denoted by S ≃ S ′ , if they only differ on a single entry, i.e., thereexists i ∈ [ n ]

Original: s.t. ∀
Revised: s.t. ∀

Original: j ̸
Revised: j ̸

Original: = i, S j = S ′ j ; and S i ̸
Revised: = i, S j = S ′ j ; and S i ̸

Original: = S ′
Revised: = S ′

Original: i .
Revised: i .

Original: We also define S i
Revised: We also define S i

Original: = { S 1 , .., S
Revised: = { S 1 , .., S

Original: ′ i , ...S n } as theneighboring meta sample of S that differs only on the i -th entry.
Revised: ′ i , ...S n } as theneighboring meta sample of S that differs only on the i -th entry.

Original: We next define the uniform stabilityof meta algorithms with episodic training strategy, which is formulated explicitly in [8, Definition 3].
Revised: We next define the uniform stabilityof meta algorithms with episodic training strategy, which is formulated explicitly in [8, Definition 3].

Original: Definition 2 (Uniform stability of modern meta learning algorithms)
Revised: Definition 2 (Uniform stability of modern meta learning algorithms)

Original: A meta algorithm A hasuniform stability w.r.t.
Revised: A meta algorithm A hasuniform stability w.r.t.

Original: the loss function ˆ L if the following holds for any meta-sample S and for anyi ∈ [ n ] , any D ∼ τ , S tr ∼ D K , S ts ∼ D q : | ˆ L ( A ( S )( S tr ) , S ts ) − ˆ L ( A ( S i )( S tr ) , S ts ) | ≤ γ .
Revised: the loss function ˆ L if the following holds for any meta-sample S and for anyi ∈ [ n ] , any D ∼ τ , S tr ∼ D K , S ts ∼ D q : | ˆ L ( A ( S )( S tr ) , S ts ) − ˆ L ( A ( S i )( S tr ) , S ts ) | ≤ γ .

Original: Since l ( A ( S ) , S )
Revised: Since l ( A ( S ) , S )

Original: =
Revised: =

Original: ˆ L ( A ( S )( S tr ) , S ts ) , we can also define the uniform stability of A as: ∀ S ∼
Revised: ˆ L ( A ( S )( S tr ) , S ts ) , we can also define the uniform stability of A as: ∀ S ∼

Original: D τ , ∀ i ∈ [ n ] , | l ( A ( S ) , S ) − l
Revised: D τ , ∀ i ∈ [ n ] , | l ( A ( S ) , S ) − l

Original: ( A ( S i ) , S ) | ≤ γ .
Revised: ( A ( S i ) , S ) | ≤ γ .

Original: Such definition is analogous to the uniform stabilityof an inner-task algorithm A in single-task learning (see Definition E.1 in Appendix E) under theidentifications: l ↔ f, A ↔ A, S ↔ S .
Revised: Such definition is analogous to the uniform stabilityof an inner-task algorithm A in single-task learning (see Definition E.1 in Appendix E) under theidentifications: l ↔ f, A ↔ A, S ↔ S .

Original: Thus, we can directly apply the existing uniform stabilitybased generalization bound from single-task learning [6, Theorem 12] to obtain the uniform stabilitybased transfer bound for episodic meta learning [8, Theorem 2], without lengthy and somewhatduplicate proof in [8].
Revised: Thus, we can directly apply the existing uniform stabilitybased generalization bound from single-task learning [6, Theorem 12] to obtain the uniform stabilitybased transfer bound for episodic meta learning [8, Theorem 2], without lengthy and somewhatduplicate proof in [8].

Original: We list such fundamental uniform stability based transfer error bound in
Revised: We list such fundamental uniform stability based transfer error bound in

Original: Theorem 1 for later comparison.
Revised: Theorem 1 for later comparison.

Original: To derive sharper transfer error bounds, our key step is to utilize theequivalent relation between the notations of single-task learning and episodic meta learning, thusextending fast-rate bounds in single-task learning [7, 27, 46] to the episodic meta learning setting.
Revised: To derive sharper transfer error bounds, our key step is to utilize theequivalent relation between the notations of single-task learning and episodic meta learning, thusextending fast-rate bounds in single-task learning [7, 27, 46] to the episodic meta learning setting.

Original: Theorem 1 Suppose the S/Q episodic meta learning algorithm A has uniform stability γ w.r.t.
Revised: Theorem 1 Suppose the S/Q episodic meta learning algorithm A has uniform stability γ w.r.t.

Original: theestimator l ( · , S ) bounded by M .
Revised: theestimator l ( · , S ) bounded by M .

Original: Then, for any task distribution τ ∈ M 1 ( M 1 ( Z )) , any δ ∈ (0 , 1) , 193 the following inequality holds with probability at least− δ over the draw of meta sample S :
Revised: Then, for any task distribution τ ∈ M 1 ( M 1 ( Z )) , any δ ∈ (0 , 1) , 193 the following inequality holds with probability at least− δ over the draw of meta sample S :

Original: [Equation]
Revised: [Equation]

Original: Uniform Argument Stability Bounds of Meta Learning Algorithms
Revised: Uniform Argument Stability Bounds of Meta Learning Algorithms

Original: For a modern meta learning algorithm with deep neural networks [20, 40], we always employ stochastic gradient descent (SGD) method to minimize the empirical error 1 n (cid:80) ni
Revised: For a modern meta learning algorithm with deep neural networks [20, 40], we always employ stochastic gradient descent (SGD) method to minimize the empirical error 1 n (cid:80) ni

Original: =

Revised: =

Original: ˆ L ( A ( S )( S tri ) , S tsi ) tolearn a good feature embedding.
Revised: ˆ L ( A ( S )( S tri ) , S tsi ) tolearn a good feature embedding.

Original: Formally, we define the sampling-with-replacement gradient updaterule at ( t + 1) -th step as: w t
Revised: Formally, we define the sampling-with-replacement gradient updaterule at ( t + 1) -th step as: w t

Original: +

Revised: +

Original: =
Revised: =

Original: Proj W [ w t − η t ∂ w t
Revised: Proj W [ w t − η t ∂ w t

Original: ˆ L ( A ( S )( S tri t ) , S tsi t )] , where i t is independently and identically drawn (i.i.d.)
Revised: ˆ L ( A ( S )( S tri t ) , S tsi t )] , where i t is independently and identically drawn (i.i.d.)

Original: from the uniform distribution Unif ([ n ]) .
Revised: from the uniform distribution Unif ([ n ]) .

Original: Therefore, althoughˆ L ( A ( S )( S tri t ) , S tsi t ) is the loss only calculated over the query samples S tsi t , it is still related to thesupport samples S tri t , and
Revised: Therefore, althoughˆ L ( A ( S )( S tri t ) , S tsi t ) is the loss only calculated over the query samples S tsi t , it is still related to thesupport samples S tri t , and

Original: the updated parameter w t +

Revised: the updated parameter w t +

Original: is also related to S tri t .
Revised: is also related to S tri t .

Original: Therefore, we definean equivalent empirical loss ˆ R ( A ( S )( S ) , S ) ≜
Revised: Therefore, we definean equivalent empirical loss ˆ R ( A ( S )( S ) , S ) ≜

Original: ˆ L ( A ( S )( S tr ) , S ts ) to indicate that: the empiricalloss over the episode S = S tr ∪ S ts is related to the whole episode sample S , and so is the outputhypothesis A ( S )( S ) .
Revised: ˆ L ( A ( S )( S tr ) , S ts ) to indicate that: the empiricalloss over the episode S = S tr ∪ S ts is related to the whole episode sample S , and so is the outputhypothesis A ( S )( S ) .

Original: Therefore, for the empirical error 1 n (cid:80) ni
Revised: Therefore, for the empirical error 1 n (cid:80) ni

Original: =

Revised: =

Original: ˆ R ( A ( S )( S i ) , S i ) , the episode-level
Revised: ˆ R ( A ( S )( S i ) , S i ) , the episode-level

Original: SGD update rule is: w t +1 =
Revised: SGD update rule is: w t +1 =

Original: Proj W [ w t − η t ∂ w t R ( w t , S i t )] .
Revised: Proj W [ w t − η t ∂ w t R ( w t , S i t )] .

Original: The pseudo code as well as severalexamples of modern meta learning algorithms can be found in Appendix C.
Revised: The pseudo code as well as severalexamples of modern meta learning algorithms can be found in Appendix C.

Original: In this section, we providelower and upper stability bounds for meta learning with sampling-with-replacement SGD method.
Revised: In this section, we providelower and upper stability bounds for meta learning with sampling-with-replacement SGD method.

Original: We first give the definition of uniform argument stability of episodic meta learning algorithms.
Revised: We first give the definition of uniform argument stability of episodic meta learning algorithms.

Original: Definition 3 (Uniform argument stability of meta learning algorithms).
Revised: Definition 3 (Uniform argument stability of meta learning algorithms).

Original: Given a meta learningalgorithm A , any neighboring meta samples S , S ′ , and any training episode S ∈ Z m , we define theuniform argument stability random variable of A as δ A ( S , S ′ ; S ) =
Revised: Given a meta learningalgorithm A , any neighboring meta samples S , S ′ , and any training episode S ∈ Z m , we define theuniform argument stability random variable of A as δ A ( S , S ′ ; S ) =

Original: ||
Revised: ||

Original: A ( S )( S ) − A
Revised: A ( S )( S ) − A

Original: ( S ′ )
Revised: ( S ′ )

Original: ( S )
Revised: ( S )

Original: || .
Revised: || .

Original: A is defined as a uniform argument β
Revised: A is defined as a uniform argument β

Original: -stable meta learning algorithm if for some β > 0 , we havesup S ≃ S ′ ,
Revised: -stable meta learning algorithm if for some β > 0 , we havesup S ≃ S ′ ,

Original: S δ A
Revised: S δ A

Original: ( S , S ′ ; S ) ≤ β or sup S ≃ S ′ ,
Revised: ( S , S ′ ; S ) ≤ β or sup S ≃ S ′ ,

Original: S E
Revised: S E

Original: A δ A ( S , S ′ ; S ) ≤ β , where E A denote the expectation 215 w.r.t.
Revised: A δ A ( S , S ′ ; S ) ≤ β , where E A denote the expectation 215 w.r.t.

Original: the internal randomness of A .
Revised: the internal randomness of A .

Original: For a meta learning algorithm with SGD method, the internalrandomness of A comes from the randomness of sampling at each iteration.
Revised: For a meta learning algorithm with SGD method, the internalrandomness of A comes from the randomness of sampling at each iteration.

Original: Note that if ˆ R ( · , S ) is a
Revised: Note that if ˆ R ( · , S ) is a

Original: Lipschitz function for any S ∈ Z m , the uniform argument stability of A implies the uniform stabilityof A in Definition 2.
Revised: Lipschitz function for any S ∈ Z m , the uniform argument stability of A implies the uniform stabilityof A in Definition 2.

Original: In this work, we investigate the stability of modern meta learning algorithmswith deep neural network, and focus on the sampling-with-replacement SGD training strategy.
Revised: In this work, we investigate the stability of modern meta learning algorithmswith deep neural network, and focus on the sampling-with-replacement SGD training strategy.

Original: Thus,we will derive lower and upper bounds on E A
Revised: Thus,we will derive lower and upper bounds on E A

Original: || A ( S )( S )
Revised: || A ( S )( S )

Original: − A ( S ′ )
Revised: − A ( S ′ )

Original: ( S ) || across different settings.
Revised: ( S ) || across different settings.

Original: 4.1 Stability Bounds for Nonsmooth Functions with α -Hölder Continuous Subgradients
Revised: 4.1 Stability Bounds for Nonsmooth Functions with α -Hölder Continuous Subgradients

Original: In this subsection, we provide lower and upper stability bounds for episodic meta learning algorithmwhose loss function is nonsmooth convex and has α -Hölder continuous subgradient with 0 ≤ α < 1 .
Revised: In this subsection, we provide lower and upper stability bounds for episodic meta learning algorithmwhose loss function is nonsmooth convex and has α -Hölder continuous subgradient with 0 ≤ α < 1 .

Original: Theorem 2 ∀ fixed S ∈
Revised: Theorem 2 ∀ fixed S ∈

Original: Z m , let ˆ R ( · , S ) be a convex and ( α, G ) -Hölder smooth function, whereα ∈ [0 , 1) .
Revised: Z m , let ˆ R ( · , S ) be a convex and ( α, G ) -Hölder smooth function, whereα ∈ [0 , 1) .

Original: Let A be a meta learning algorithm with sampling-with-replacement SGD.
Revised: Let A be a meta learning algorithm with sampling-with-replacement SGD.

Original: Denote byw j and w ′ j the outputs after j ( j ∈ [ T ]) steps of SGD on S and S i , respectively.
Revised: Denote byw j and w ′ j the outputs after j ( j ∈ [ T ]) steps of SGD on S and S i , respectively.

Original: Define R S ( w ) =n − 1 (cid:80) ni
Revised: Define R S ( w ) =n − 1 (cid:80) ni

Original: =

Revised: =

Original: ˆ R ( w, S i ) , ∀ w ∈ W .
Revised: ˆ R ( w, S i ) , ∀ w ∈ W .

Original: Then ∀ S ∈
Revised: Then ∀ S ∈

Original: Z m , E A δ A ( S , S ′ ; S ) is upper bounded by√ 2 c
Revised: Z m , E A δ A ( S , S ′ ; S ) is upper bounded by√ 2 c

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In addition, if ˆ R ( · , S ) is bounded by M and the step size η j = η ∀ j ∈ [ T ] , we can obtain thelower and upper bounds of the uniform argument stability of A : c α M α 1+ α (min { 1 , Tn } η √ T
Revised: In addition, if ˆ R ( · , S ) is bounded by M and the step size η j = η ∀ j ∈ [ T ] , we can obtain thelower and upper bounds of the uniform argument stability of A : c α M α 1+ α (min { 1 , Tn } η √ T

Original: [Equation]
Revised: [Equation]

Original: ηTn ) ≤
Revised: ηTn ) ≤

Original: [Equation]
Revised: [Equation]

Original: Remark 1 Our upper stability bound in Eq.
Revised: Remark 1 Our upper stability bound in Eq.

Original: (4) depends on the empirical risk during the optimizationprocess.
Revised: (4) depends on the empirical risk during the optimizationprocess.

Original: Formally, Eq.
Revised: Formally, Eq.

Original: (4) shows that, the stability of modern meta algorithm increases if we findgood parameters w j with small empirical risk R S ( w j ) at the j -th optimization step.
Revised: (4) shows that, the stability of modern meta algorithm increases if we findgood parameters w j with small empirical risk R S ( w j ) at the j -th optimization step.

Original: This illustratesa key insight that optimization is beneficial to improve the generalization of algorithms.
Revised: This illustratesa key insight that optimization is beneficial to improve the generalization of algorithms.

Original: Besides, ourstability upper bound also implies the importance of a good embedding [41] (which may have a goodinitialization and low empirical risk during the first several optimization steps) to generalization.
Revised: Besides, ourstability upper bound also implies the importance of a good embedding [41] (which may have a goodinitialization and low empirical risk during the first several optimization steps) to generalization.

Original: Remark 

Revised: Remark

Original: We additionally suppose the function to be bounded by M such that the stability boundscan be used to analyze the generalization bounds in the next section where the loss function is alwaysassumed to be bounded.
Revised: We additionally suppose the function to be bounded by M such that the stability boundscan be used to analyze the generalization bounds in the next section where the loss function is alwaysassumed to be bounded.

Original: Note that when α = 0 , ˆ R ( · , S ) is a nonsmooth c α -Lipschitz convex function,and our lower and upper stability bounds recover the results in [2].
Revised: Note that when α = 0 , ˆ R ( · , S ) is a nonsmooth c α -Lipschitz convex function,and our lower and upper stability bounds recover the results in [2].

Original: For bounded convex α -Höldersmooth functions, the lower stability bound implies that modern meta learning algorithms are notstable enough even we train them relatively
Revised: For bounded convex α -Höldersmooth functions, the lower stability bound implies that modern meta learning algorithms are notstable enough even we train them relatively

Original: Remark 

Revised: Remark

Original: The above result shows that, for bounded convex α -Hölder smooth function ( α ∈ [0 , 1) ),the uniform argument stability parameter β
Revised: The above result shows that, for bounded convex α -Hölder smooth function ( α ∈ [0 , 1) ),the uniform argument stability parameter β

Original: = O (cid:0) c
Revised: = O (cid:0) c

Original: α M α 

Revised: α M α

Original: + α (min { 1 , T/n } η √ T + ηT/n ) (cid:1) .
Revised: + α (min { 1 , T/n } η √ T + ηT/n ) (cid:1) .

Original: Anotherwork [30] also focuses on convex α -Hölder smooth function.
Revised: Anotherwork [30] also focuses on convex α -Hölder smooth function.

Original: Using the technique from [30], we obtainthe upper stability bounds for bounded convex α -Hölder smooth function under the same conditions:β ≤ O (cid:0) c α M α 1+ α ( η 1 1 − α T + ηT/n ) (cid:1) or β ≤ O (cid:0) c α M α 1+ α ( η 1 1 − α √ T + η (cid:112) T/n ) (cid:1) (see Theorems D.1D.2 in Appendix D.2.2), both of which are larger than our tight stability bound in Theorem 2 underthe setting T ≤ n .
Revised: Using the technique from [30], we obtainthe upper stability bounds for bounded convex α -Hölder smooth function under the same conditions:β ≤ O (cid:0) c α M α 1+ α ( η 1 1 − α T + ηT/n ) (cid:1) or β ≤ O (cid:0) c α M α 1+ α ( η 1 1 − α √ T + η (cid:112) T/n ) (cid:1) (see Theorems D.1D.2 in Appendix D.2.2), both of which are larger than our tight stability bound in Theorem 2 underthe setting T ≤ n .

Original: When T > n , the upper bound β ≤ O (cid:0) c α M α 1+ α ( η 1 1 − α √ T + η (cid:112) T/n ) (cid:1) in
Revised: When T > n , the upper bound β ≤ O (cid:0) c α M α 1+ α ( η 1 1 − α √ T + η (cid:112) T/n ) (cid:1) in

Original: Theorem D.2 is slightly sharper than the upper bound O (cid:0) c α M α 

Revised: Theorem D.2 is slightly sharper than the upper bound O (cid:0) c α M α

Original: + α ( η √ T + ηT/n ) (cid:1) in Theorem 2.
Revised: + α ( η √ T + ηT/n ) (cid:1) in Theorem 2.

Original: 4.2 Stability Bounds for Smooth Functions
Revised: 4.2 Stability Bounds for Smooth Functions

Original: In this subsection, we provide lower and upper uniform argument stability bounds for modern metalearning algorithms with smooth functions.
Revised: In this subsection, we provide lower and upper uniform argument stability bounds for modern metalearning algorithms with smooth functions.

Original: First, we consider smooth convex functions.
Revised: First, we consider smooth convex functions.

Original: Theorem 3 ∀ fixed S ∈
Revised: Theorem 3 ∀ fixed S ∈

Original: Z m , let ˆ R ( · , S ) be a G -smooth convex function.
Revised: Z m , let ˆ R ( · , S ) be a G -smooth convex function.

Original: Let A be a meta learningalgorithm with sampling-with-replacement SGD.
Revised: Let A be a meta learningalgorithm with sampling-with-replacement SGD.

Original: Denote by w j and w ′ j the outputs after j ( j ∈ [ T ])steps of SGD on neighboring meta samples S and S i , respectively.
Revised: Denote by w j and w ′ j the outputs after j ( j ∈ [ T ])steps of SGD on neighboring meta samples S and S i , respectively.

Original: Then ∀ S ∈ Z m , η j ≤ 2 /G ,
Revised: Then ∀ S ∈ Z m , η j ≤ 2 /G ,

Original: [Equation]
Revised: [Equation]

Original: In addition, if ˆ R ( · , S ) is bounded by M , we can obtain the lower and upper bounds of the uniformargument stability of A : 1 n (cid:80)
Revised: In addition, if ˆ R ( · , S ) is bounded by M , we can obtain the lower and upper bounds of the uniformargument stability of A : 1 n (cid:80)

Original: Tj =1 η j ≤ sup S , S ′ ,S E
Revised: Tj =1 η j ≤ sup S , S ′ ,S E

Original: A δ A ( S , S ′ ; S ) ≤√ 2 MGn
Revised: A δ A ( S , S ′ ; S ) ≤√ 2 MGn

Original: [Equation]
Revised: [Equation]

Original: Tj
Revised: Tj

Original: [Equation]
Revised: [Equation]

Original: 1 η j .
Revised: 1 η j .

Original: If we set all η j = η , then for bounded convex functions, the tight stability bound of O ( ηTn ) under thesmooth case is sharper than the stability bound of O (min { 1 , Tn } η √ T + ηTn ) in Theorem 2 under thenonsmooth case.
Revised: If we set all η j = η , then for bounded convex functions, the tight stability bound of O ( ηTn ) under thesmooth case is sharper than the stability bound of O (min { 1 , Tn } η √ T + ηTn ) in Theorem 2 under thenonsmooth case.

Original: This indicates that in the smooth case, meta learning algorithms are more stablethan in the nonsmooth case.
Revised: This indicates that in the smooth case, meta learning algorithms are more stablethan in the nonsmooth case.

Original: Finally, we give stability bounds for smooth non-convex functions.
Revised: Finally, we give stability bounds for smooth non-convex functions.

Original: Theorem 4 ∀ fixed S ∈
Revised: Theorem 4 ∀ fixed S ∈

Original: Z m , let ˆ R ( · , S ) be a σ -Lipschitz and G -smooth function.
Revised: Z m , let ˆ R ( · , S ) be a σ -Lipschitz and G -smooth function.

Original: Let A be a metalearning algorithm.
Revised: Let A be a metalearning algorithm.

Original: Denote by w j and w ′ j the outputs after j ( j ∈ [ T ]) steps of SGD on S and S i ,respectively.
Revised: Denote by w j and w ′ j the outputs after j ( j ∈ [ T ]) steps of SGD on S and S i ,respectively.

Original: Define the learning rate η j = ajG , ∀ j ∈
Revised: Define the learning rate η j = ajG , ∀ j ∈

Original: [ T ] with a > 0 .
Revised: [ T ] with a > 0 .

Original: Then ∀ S ∈ Z m , the lower andupper stability bounds of A satisfy:
Revised: Then ∀ S ∈ Z m , the lower andupper stability bounds of A satisfy:

Original: [Equation]
Revised: [Equation]

Original: Under the same step size setting, existing upper uniform argument stability bound in [8, Theorem3]or in [32, Proposition 4] for non-convex, smooth and Lipschitz function is of O ( T a 1+ a /n ) .
Revised: Under the same step size setting, existing upper uniform argument stability bound in [8, Theorem3]or in [32, Proposition 4] for non-convex, smooth and Lipschitz function is of O ( T a 1+ a /n ) .

Original: Ourbound of order O
Revised: Ourbound of order O

Original: ( T a /n 1+ a ) is improved over the existing bound when T a 1+ a ≤ n .
Revised: ( T a /n 1+ a ) is improved over the existing bound when T a 1+ a ≤ n .

Original: Besides, ourstability bound can be non-vacuous for multi-pass SGD setting (i.e., when T = kn , k ∈ N ) wherethe number T of SGD iterations is larger than n , as long as k ≤ n 1 /a .
Revised: Besides, ourstability bound can be non-vacuous for multi-pass SGD setting (i.e., when T = kn , k ∈ N ) wherethe number T of SGD iterations is larger than n , as long as k ≤ n 1 /a .

Original:  High Probability Transfer Error Bounds for Meta Learning
Revised:  High Probability Transfer Error Bounds for Meta Learning

Original: In this section, we establish high probability bounds for transfer error
Revised: In this section, we establish high probability bounds for transfer error

Original: er ( A ( S ) , τ ) .
Revised: er ( A ( S ) , τ ) .

Original: Specifically, westill consider two kinds of loss function: (1) convex and ( α , G )-Hölder smooth function ( α ∈ [0 , 1] );(2) non-convex, σ -Lipschitz and G -smooth function.
Revised: Specifically, westill consider two kinds of loss function: (1) convex and ( α , G )-Hölder smooth function ( α ∈ [0 , 1] );(2) non-convex, σ -Lipschitz and G -smooth function.

Original: We always assume that the loss function ˆ R ( · , · )is bounded by M .
Revised: We always assume that the loss function ˆ R ( · , · )is bounded by M .

Original: Define σ α = c
Revised: Define σ α = c

Original: α M α 

Revised: α M α

Original: + α
Revised: + α

Original: if ˆ R ( w, S ) is a convex and ( α, G ) -Hölder smooth function;σ α
Revised: if ˆ R ( w, S ) is a convex and ( α, G ) -Hölder smooth function;σ α

Original: =
Revised: =

Original: σ if ˆ R ( w, S ) is a σ -Lipschitz and G -smooth function.
Revised: σ if ˆ R ( w, S ) is a σ -Lipschitz and G -smooth function.

Original: We just exhibit the generalizationbounds of randomized algorithm A by supposing P A [ δ A ( S , S ′ ; S ) > β ] ≤ δ 0 .
Revised: We just exhibit the generalizationbounds of randomized algorithm A by supposing P A [ δ A ( S , S ′ ; S ) > β ] ≤ δ 0 .

Original: We provide anexample to illustrate how to calculate δ 0 in Example E.1 in
Revised: We provide anexample to illustrate how to calculate δ 0 in Example E.1 in

Original: Appendix E. The generalization boundfor deterministic meta algorithm (e.g. with gradient descent) can be stated by setting δ 0 = 0 .
Revised: Appendix E. The generalization boundfor deterministic meta algorithm (e.g. with gradient descent) can be stated by setting δ 0 = 0 .

Original: 5.1 Near Optimal Transfer Error Bound for Meta Learning with Independent Episodes
Revised: 5.1 Near Optimal Transfer Error Bound for Meta Learning with Independent Episodes

Original: We denote by a ≲ b the existence of some universal constant c > 0 such that a ≤ cb .
Revised: We denote by a ≲ b the existence of some universal constant c > 0 such that a ≤ cb .

Original: Then we obtainthe following near optimal bound of O (1 / √ n ) under the independent task environment assumption.
Revised: Then we obtainthe following near optimal bound of O (1 / √ n ) under the independent task environment assumption.

Original: Theorem 5 Let A ∈ A ( A ( H , Z ) , Z m ) be a uniform argument β -stable meta algorithm, i.e.,sup S ≃ S ′ ,S E A
Revised: Theorem 5 Let A ∈ A ( A ( H , Z ) , Z m ) be a uniform argument β -stable meta algorithm, i.e.,sup S ≃ S ′ ,S E A

Original: ||
Revised: ||

Original: A ( S )( S ) − A ( S ′ )
Revised: A ( S )( S ) − A ( S ′ )

Original: ( S ) ||
Revised: ( S ) ||

Original: ≤ β .
Revised: ≤ β .

Original: For any S ∈ Z m , let ˆ R ( · , S ) be [0 , M ] -valued, and satisfy one of the two following conditions:
Revised: For any S ∈ Z m , let ˆ R ( · , S ) be [0 , M ] -valued, and satisfy one of the two following conditions:

Original: (1) ˆ R ( · , S ) is convex and ( α, G ) -Hölder smooth( α ∈ [0 , 1] ); (2) ˆ R ( · , S ) is σ -Lipschitz and G -smooth.
Revised: (1) ˆ R ( · , S ) is convex and ( α, G ) -Hölder smooth( α ∈ [0 , 1] ); (2) ˆ R ( · , S ) is σ -Lipschitz and G -smooth.

Original: Suppose P A [ δ A ( S , S ′ ; S ) > β ] ≤ δ 0 .
Revised: Suppose P A [ δ A ( S , S ′ ; S ) > β ] ≤ δ 0 .

Original: Thenfor any independent task environment τ ∈ M 1 ( M 1 ( Z )) , any δ ∈ (0 , 1) , the following holds with 289 probability at least− δ − δ 0 over the draw of S and the internal randomness of A :
Revised: Thenfor any independent task environment τ ∈ M 1 ( M 1 ( Z )) , any δ ∈ (0 , 1) , the following holds with 289 probability at least− δ − δ 0 over the draw of S and the internal randomness of A :

Original: [Equation]
Revised: [Equation]

Original: 291 Remark 4 Our transfer error bound in Theorem 5 has three advantages over the bound in Theorem1 from [8]: (1) Theorem 1 gives a high-probability upper bound of O ( √ nγ + M/ √ n ) for transfererror, where γ is the uniform stability parameter and always scales as O (1 /n ) ; in contrast, our upperbound of O ( β ln n + M/ √ n ) is improved by replacing the √ n factor before the stability parameterwith ln n .
Revised: 291 Remark 4 Our transfer error bound in Theorem 5 has three advantages over the bound in Theorem1 from [8]: (1) Theorem 1 gives a high-probability upper bound of O ( √ nγ + M/ √ n ) for transfererror, where γ is the uniform stability parameter and always scales as O (1 /n ) ; in contrast, our upperbound of O ( β ln n + M/ √ n ) is improved by replacing the √ n factor before the stability parameterwith ln n .

Original: In [8], the uniform stability γ =
Revised: In [8], the uniform stability γ =

Original: O ( T aa +1 /n ) , whereas our uniform argument stabilityβ =
Revised: O ( T aa +1 /n ) , whereas our uniform argument stabilityβ =

Original: O ( T a /n 1+ a ) is tighter when T aa +1 ≤ n , i.e., when the uniform stability bound γ
Revised: O ( T a /n 1+ a ) is tighter when T aa +1 ≤ n , i.e., when the uniform stability bound γ

Original: =
Revised: =

Original: O ( T aa +1 /n )is non-vacuous.
Revised: O ( T aa +1 /n )is non-vacuous.

Original: (3) Our high-probability transfer error bound of order O (1 / √ n ) is near optimal.
Revised: (3) Our high-probability transfer error bound of order O (1 / √ n ) is near optimal.

Original: Remark 

Revised: Remark

Original: We uncover two limitations of stability-based meta learning theory: (1) Recall the lowerstability bound for meta learning algorithms with convex α -Hölder smooth function ( α ∈ [0 , 1) ) in
Revised: We uncover two limitations of stability-based meta learning theory: (1) Recall the lowerstability bound for meta learning algorithms with convex α -Hölder smooth function ( α ∈ [0 , 1) ) in

Original: Theorem 2, we find that the lower transfer error bound in Theorem 5 is er ( A ( S ) , τ ) − ˆ er ( A ( S ) , S ) ≳σ α ln
Revised: Theorem 2, we find that the lower transfer error bound in Theorem 5 is er ( A ( S ) , τ ) − ˆ er ( A ( S ) , S ) ≳σ α ln

Original: (1 /δ )
Revised: (1 /δ )

Original: c α M α 

Revised: c α M α

Original: + α ( η √ T + ηT/n ) when T ≥ n .
Revised: + α ( η √ T + ηT/n ) when T ≥ n .

Original: This indicates that the lower transfer error boundis greater than a constant and will not converge to zero with the increase of n .
Revised: This indicates that the lower transfer error boundis greater than a constant and will not converge to zero with the increase of n .

Original: Thus, the stability-basedtransfer error bound is vacuous and cannot provide asymptotic guarantees for convex Hölder smoothfunctions.
Revised: Thus, the stability-basedtransfer error bound is vacuous and cannot provide asymptotic guarantees for convex Hölder smoothfunctions.

Original: The stability-based transfer error bound of O (1 / √ n ) in Theorem 5 is near optimal.
Revised: The stability-based transfer error bound of O (1 / √ n ) in Theorem 5 is near optimal.

Original: Such result is assistant with the observation in [35, Section 2] that under the (i.i.d.) task environmentassumption, the term O (1 / √ n ) in the generalization bound is unavoidable.
Revised: Such result is assistant with the observation in [35, Section 2] that under the (i.i.d.) task environmentassumption, the term O (1 / √ n ) in the generalization bound is unavoidable.

Original: Thus, to obtain sharpergeneralization bounds for meta learning (e.g. the bound of O (1 / √ mn ) or even O (1 /mn ) ), we needto consider other stability notions (e.g. [16]), or suppose stronger task relatedness in the environment(e.g.
Revised: Thus, to obtain sharpergeneralization bounds for meta learning (e.g. the bound of O (1 / √ mn ) or even O (1 /mn ) ), we needto consider other stability notions (e.g. [16]), or suppose stronger task relatedness in the environment(e.g.

Original: [5, 22]), or even drop the task environment assumption (e.g. [14, 42]).
Revised: [5, 22]), or even drop the task environment assumption (e.g. [14, 42]).

Original: Remark 6 Under the independent task environment assumption, we compare our bound of O (1 / √ n )via S/Q episodic training strategy with other transfer error bounds that are obtained via traditional
Revised: Remark 6 Under the independent task environment assumption, we compare our bound of O (1 / √ n )via S/Q episodic training strategy with other transfer error bounds that are obtained via traditional

Original: ERM strategy over all samples in training tasks.
Revised: ERM strategy over all samples in training tasks.

Original: In detail, the bound from [33, Theorems 2 and 6]via algorithmic stability analysis is of O (1 /m + 1 / √ n ) ; the bounds from [37, Theorem 1] and [38,
Revised: In detail, the bound from [33, Theorems 2 and 6]via algorithmic stability analysis is of O (1 /m + 1 / √ n ) ; the bounds from [37, Theorem 1] and [38,

Original: Theorem 2] via PAC-Bayes analysis are of O (1 / √ n + 1 / √ m ) ; the bound from [22, Theorem 5] viacovering number analysis is of O (1 / √ nm + 1 / √ m ) .
Revised: Theorem 2] via PAC-Bayes analysis are of O (1 / √ n + 1 / √ m ) ; the bound from [22, Theorem 5] viacovering number analysis is of O (1 / √ nm + 1 / √ m ) .

Original: All of these bounds via ERM strategy involve aterm O (1 / √ m ) , and such term can be large when m is relatively small (e.g. m = 5 or m = 10 in thefew-shot learning setting).
Revised: All of these bounds via ERM strategy involve aterm O (1 / √ m ) , and such term can be large when m is relatively small (e.g. m = 5 or m = 10 in thefew-shot learning setting).

Original: Thus, in terms of the tightness of transfer error bounds, the S/Q episodictraining strategy is superior to the ERM strategy for meta learning, when m << n .
Revised: Thus, in terms of the tightness of transfer error bounds, the S/Q episodictraining strategy is superior to the ERM strategy for meta learning, when m << n .

Original: Such result wasalso pointed out by [8] and is more rigorously demonstrated in this work .
Revised: Such result wasalso pointed out by [8] and is more rigorously demonstrated in this work .

Original: Detailed comparisonsbetween different transfer error bounds for meta learning are shown in Table A.2 in Appendix A.
Revised: Detailed comparisonsbetween different transfer error bounds for meta learning are shown in Table A.2 in Appendix A.

Original: 5.2 Fast Transfer Error Bound of O (ln n/n ) for Meta Learning with Independent Episodes
Revised: 5.2 Fast Transfer Error Bound of O (ln n/n ) for Meta Learning with Independent Episodes

Original: To obtain faster convergence rate, we need to take additional assumption.
Revised: To obtain faster convergence rate, we need to take additional assumption.

Original: The generalized Bernsteincondition is one of the most widely used condition to attain fast convergence rate of generalizationbound in single-task learning [32, 27].
Revised: The generalized Bernsteincondition is one of the most widely used condition to attain fast convergence rate of generalizationbound in single-task learning [32, 27].

Original: Next, we extend the generalized Bernstein condition to themeta learning setting, where we study the optimal algorithm A ∗ instead of the optimal hypothesis.
Revised: Next, we extend the generalized Bernstein condition to themeta learning setting, where we study the optimal algorithm A ∗ instead of the optimal hypothesis.

Original: Definition 4 (Generalized Bernstein Condition for Meta Learning) Assume that A ∗ ( H , Z ) =
Revised: Definition 4 (Generalized Bernstein Condition for Meta Learning) Assume that A ∗ ( H , Z ) =

Original: Argmin A ∈A ( H , Z ) er
Revised: Argmin A ∈A ( H , Z ) er

Original: ( A, τ ) is a set of risk minimizers in a closed set.
Revised: ( A, τ ) is a set of risk minimizers in a closed set.

Original: We say that an algorithm
Revised: We say that an algorithm

Original: A together with the environment τ and the empirical estimator l satisfies the generalized Bernsteincondition if for some B > 0 ,
Revised: A together with the environment τ and the empirical estimator l satisfies the generalized Bernsteincondition if for some B > 0 ,

Original: ∀
Revised: ∀

Original: A ∈
Revised: A ∈

Original: A ( H , Z ) , there is a A ∗ ∈
Revised: A ( H , Z ) , there is a A ∗ ∈

Original: A ∗ ( H , Z ) , such that
Revised: A ∗ ( H , Z ) , such that

Original: [Equation]
Revised: [Equation]

Original: 331 [27] has shown that in single-task learning, a strongly-convex and Lipschitz function satisfies thegeneralized Bernstein condition.
Revised: 331 [27] has shown that in single-task learning, a strongly-convex and Lipschitz function satisfies thegeneralized Bernstein condition.

Original: In this work, we relax the strong-convexity condition by consideringthe following Polyak-Łojasiewicz condition, one of the weakest curvature conditions of functions.
Revised: In this work, we relax the strong-convexity condition by consideringthe following Polyak-Łojasiewicz condition, one of the weakest curvature conditions of functions.

Original: Definition 5 (Polyak-Łojasiewicz [45])
Revised: Definition 5 (Polyak-Łojasiewicz [45])

Original: Any function f : W → R satisfies the Polyak-Łojasiewicz(PL) condition on W with parameter µ > 0 if for all w ∈ W , f ( w ) − f ( w ∗ ) ≤ 12 µ || ∂ 0 f ( w ) || 22 ,where w ∗ denotes the Euclidean projection of
Revised: Any function f : W → R satisfies the Polyak-Łojasiewicz(PL) condition on W with parameter µ > 0 if for all w ∈ W , f ( w ) − f ( w ∗ ) ≤ 12 µ || ∂ 0 f ( w ) || 22 ,where w ∗ denotes the Euclidean projection of

Original: A key insight into the PL condition is that it is the sufficient and necessary condition to guaranteethe linear convergence of gradient descent methods for smooth convex optimization problem [36].
Revised: A key insight into the PL condition is that it is the sufficient and necessary condition to guaranteethe linear convergence of gradient descent methods for smooth convex optimization problem [36].

Original: Such PL condition can also be satisfied by many non-convex neural network models, such as thetwo-layer neural networks with ReLU activation functions [31] and the deep linear residual networks[23].
Revised: Such PL condition can also be satisfied by many non-convex neural network models, such as thetwo-layer neural networks with ReLU activation functions [31] and the deep linear residual networks[23].

Original: We will show that if the functions in Theorem 5 additionally satisfy the PL condition, then theloss functions in meta learning also satisfy the generalized Bernstein condition in Definition 4.
Revised: We will show that if the functions in Theorem 5 additionally satisfy the PL condition, then theloss functions in meta learning also satisfy the generalized Bernstein condition in Definition 4.

Original: Thus,we can derive a "deformed" transfer error bound of O (ln n/n ) for modern meta learning algorithms.
Revised: Thus,we can derive a "deformed" transfer error bound of O (ln n/n ) for modern meta learning algorithms.

Original: Theorem 6 Under the same conditions of Theorem 5, for any fixed S ∈
Revised: Theorem 6 Under the same conditions of Theorem 5, for any fixed S ∈

Original: Z m , let ˆ R ( · , S ) additionallysatisfy Polyak-Łojasiewicz condition in Definition 5.
Revised: Z m , let ˆ R ( · , S ) additionallysatisfy Polyak-Łojasiewicz condition in Definition 5.

Original: Suppose P A [ δ A ( S , S ′ ; S ) > β ] ≤ δ 0 .
Revised: Suppose P A [ δ A ( S , S ′ ; S ) > β ] ≤ δ 0 .

Original: Then,there exist c > 0 , such that for any environment τ ∈ M 1 ( M 1 ( Z )) , and any δ ∈ (0 , 1) , the following 347 holds with probability at least
Revised: Then,there exist c > 0 , such that for any environment τ ∈ M 1 ( M 1 ( Z )) , and any δ ∈ (0 , 1) , the following 347 holds with probability at least

Original: [Equation]
Revised: [Equation]

Original: 0 over the draw of S and the internal randomness of A :
Revised: 0 over the draw of S and the internal randomness of A :

Original: [Equation]
Revised: [Equation]

Original: 349 Recall in Section 4, our stability parameter is always β
Revised: 349 Recall in Section 4, our stability parameter is always β

Original: = O (1 /n ) .
Revised: = O (1 /n ) .

Original: Hence, when the empirical errorin the RHS of the above bound is close to zero, the transfer error bound always scales as O (ln n/n ) .
Revised: Hence, when the empirical errorin the RHS of the above bound is close to zero, the transfer error bound always scales as O (ln n/n ) .

Original: 5.3 Transfer Error Bound for Meta Learning with Dependent Episodes
Revised: 5.3 Transfer Error Bound for Meta Learning with Dependent Episodes

Original: In this subsection, we investigate the generalization bound for meta learning algorithms with dependent episodes whose dependency relation can be characterized by a graph.
Revised: In this subsection, we investigate the generalization bound for meta learning algorithms with dependent episodes whose dependency relation can be characterized by a graph.

Original: The approach undertakento establish our results is based on the forest approximation of the dependency graph [26].
Revised: The approach undertakento establish our results is based on the forest approximation of the dependency graph [26].

Original: Formally,a dependency graph is an undirected graph Γ =
Revised: Formally,a dependency graph is an undirected graph Γ =

Original: ( V, E ) of a random vector S =
Revised: ( V, E ) of a random vector S =

Original: ( S 1 , ..., S n ) if thefollowing two conditions are satisfied: (1) V (Γ) =
Revised: ( S 1 , ..., S n ) if thefollowing two conditions are satisfied: (1) V (Γ) =

Original: [ n ] ; (2) if I, J ⊂ [ n ] are non-adjacent in Γ , then{ S i } i ∈
Revised: [ n ] ; (2) if I, J ⊂ [ n ] are non-adjacent in Γ , then{ S i } i ∈

Original: I and { S j } j ∈ J are independent.
Revised: I and { S j } j ∈ J are independent.

Original: We next give the concept of forest approximation.
Revised: We next give the concept of forest approximation.

Original: Definition 6 (Forest Approximation [46]) Given a graph Γ , a forest F , and a mapping ϕ
Revised: Definition 6 (Forest Approximation [46]) Given a graph Γ , a forest F , and a mapping ϕ

Original: V (Γ) →
Revised: V (Γ) →

Original: V ( F ) , if ϕ ( u ) =
Revised: V ( F ) , if ϕ ( u ) =

Original: ϕ ( v ) or ⟨ ϕ ( u ) , ϕ ( v ) ⟩ ∈ E ( F ) for any ⟨ u, v ⟩ ∈ E (Γ) , we say that ( ϕ, F ) is a forestapproximation of Γ .
Revised: ϕ ( v ) or ⟨ ϕ ( u ) , ϕ ( v ) ⟩ ∈ E ( F ) for any ⟨ u, v ⟩ ∈ E (Γ) , we say that ( ϕ, F ) is a forestapproximation of Γ .

Original: Let Φ(Γ) denote the set of forest approximations of Γ .
Revised: Let Φ(Γ) denote the set of forest approximations of Γ .

Original: Intuitively, a forest approximation transform a graph into a forest by merging vertices and removingself-loops.
Revised: Intuitively, a forest approximation transform a graph into a forest by merging vertices and removingself-loops.

Original: We then give the definition of forest complexity, which measures how a dependency graphlooks like a forest, and hence measures the strength of dependency among random variables in S .
Revised: We then give the definition of forest complexity, which measures how a dependency graphlooks like a forest, and hence measures the strength of dependency among random variables in S .

Original: Definition 7 (Forest Complexity [46]) Given a graph Γ and any forest approximation ( ϕ, F ) ∈
Revised: Definition 7 (Forest Complexity [46]) Given a graph Γ and any forest approximation ( ϕ, F ) ∈

Original: Φ( G )with F consisting of trees { T i } i ∈ [ k ] .
Revised: Φ( G )with F consisting of trees { T i } i ∈ [ k ] .

Original: Define λ ( ϕ,F ) =
Revised: Define λ ( ϕ,F ) =

Original: (cid:80) ⟨ u,v ⟩∈ E ( F ) (cid:0)
Revised: (cid:80) ⟨ u,v ⟩∈ E ( F ) (cid:0)

Original: | ϕ − 1 ( u ) | + | ϕ − 1 ( v ) | (cid:1) 

Revised: | ϕ − 1 ( u ) | + | ϕ − 1 ( v ) | (cid:1)

Original: +(cid:80) ki
Revised: +(cid:80) ki

Original: =1 min u ∈ V ( T i )
Revised: =1 min u ∈ V ( T i )

Original: | ϕ − 1 ( u ) | 2 .
Revised: | ϕ − 1 ( u ) | 2 .

Original: We call Λ(Γ)
Revised: We call Λ(Γ)

Original: =
Revised: =

Original: min ( ϕ,F ) ∈ Φ(Γ) λ
Revised: min ( ϕ,F ) ∈ Φ(Γ) λ

Original: ( ϕ,F ) the forest complexity of thegraph Γ =
Revised: ( ϕ,F ) the forest complexity of thegraph Γ =

Original: ( V, E ) .
Revised: ( V, E ) .

Original: Here, ϕ − 1 ( u ) is the set of pre-images of the element u .
Revised: Here, ϕ − 1 ( u ) is the set of pre-images of the element u .

Original: For sample S whose components are independent, we choose the identity map and its dependencygraph as the forest approximation.
Revised: For sample S whose components are independent, we choose the identity map and its dependencygraph as the forest approximation.

Original: Hence Λ(Γ)
Revised: Hence Λ(Γ)

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 1 2 = n .
Revised: =1 1 2 = n .

Original: For sample S whose dependencygraph Γ is a tree, the identity map between Γ and itself is a forest approximation of Γ .
Revised: For sample S whose dependencygraph Γ is a tree, the identity map between Γ and itself is a forest approximation of Γ .

Original: ThenΛ(Γ) ≤ | E (Γ)
Revised: ThenΛ(Γ) ≤ | E (Γ)

Original: | (1+1) 

Revised: | (1+1)

Original: +1 = 4 n − 3 =
Revised: +1 = 4 n − 3 =

Original: O ( n ) .
Revised: O ( n ) .

Original: More examples of forest approximation can be foundin [46, Section 3.3].
Revised: More examples of forest approximation can be foundin [46, Section 3.3].

Original: We next give a forest-complexity based transfer error bound for meta learning.
Revised: We next give a forest-complexity based transfer error bound for meta learning.

Original: Theorem 7 Under the same conditions of Theorem 5, except that S is a meta sample of size n withdependency graph Γ .
Revised: Theorem 7 Under the same conditions of Theorem 5, except that S is a meta sample of size n withdependency graph Γ .

Original: Let the maximum degree of the graph Γ is △ .
Revised: Let the maximum degree of the graph Γ is △ .

Original: Suppose P A [ δ A ( S , S ′ ; S ) >β ] ≤ δ 0 .
Revised: Suppose P A [ δ A ( S , S ′ ; S ) >β ] ≤ δ 0 .

Original: Then, for any environment τ ∈ M 1 ( M 1 ( Z )) , any δ ∈ (0 , 1) , the following holds with 376 probability at least− δ − δ 0 over the draw of S and the internal randomness of A :
Revised: Then, for any environment τ ∈ M 1 ( M 1 ( Z )) , any δ ∈ (0 , 1) , the following holds with 376 probability at least− δ − δ 0 over the draw of S and the internal randomness of A :

Original: [Equation]
Revised: [Equation]

Original: 378 When S is an independent sample, the forest complexity Λ(Γ) =
Revised: 378 When S is an independent sample, the forest complexity Λ(Γ) =

Original: n , the maximum degree △ = 0 , andthe above forest-complexity based generalization bound degenerates to the bound in Theorem 1 formeta learning with independent episodes.
Revised: n , the maximum degree △ = 0 , andthe above forest-complexity based generalization bound degenerates to the bound in Theorem 1 formeta learning with independent episodes.

Original: When S is a dependent sample, Λ(Γ) will be greater than n .
Revised: When S is a dependent sample, Λ(Γ) will be greater than n .

Original: Both the complexity Λ(Γ) and the maximum degree △ will increase with more dependency relationbetween samples in S (i.e., with more adjacent edges in its dependency graph Γ =
Revised: Both the complexity Λ(Γ) and the maximum degree △ will increase with more dependency relationbetween samples in S (i.e., with more adjacent edges in its dependency graph Γ =

Original: ( V, E ) ).
Revised: ( V, E ) ).

Original: In thenext section, we conduct experiments on regression problems to show the convergence performanceof the generalization bound for meta learning with dependent episodes.
Revised: In thenext section, we conduct experiments on regression problems to show the convergence performanceof the generalization bound for meta learning with dependent episodes.

Original: The corresponding forestcomplexity based generalization bound for such problem is provided in Example E.2 in Appendix E.3.
Revised: The corresponding forestcomplexity based generalization bound for such problem is provided in Example E.2 in Appendix E.3.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: E rr o r er ( A ( S ) ,τ ) ˆ er ( A ( S ) , S ) | er − ˆ er | (b) Bilevel, l 1 loss 1 . 0 1 .
Revised: E rr o r er ( A ( S ) ,τ ) ˆ er ( A ( S ) , S ) | er − ˆ er | (b) Bilevel, l 1 loss 1 . 0 1 .

Original: 5 4 . 0 log # of tasks 0 .0 .
Revised: 5 4 . 0 log # of tasks 0 .0 .

Original: E rr o r er ( A ( S ) ,τ ) ˆ er ( A ( S ) , S ) | er − ˆ er | (c) MAML, l 2 loss 1 . 0 1 .
Revised: E rr o r er ( A ( S ) ,τ ) ˆ er ( A ( S ) , S ) | er − ˆ er | (c) MAML, l 2 loss 1 . 0 1 .

Original: 5 4 . 0 log # of tasks 0 .0 .1 .1 .2 .2 .
Revised: 5 4 . 0 log # of tasks 0 .0 .1 .1 .2 .2 .

Original: E rr o r er ( A ( S ) ,τ ) ˆ er ( A ( S ) , S ) | er − ˆ er | (d) MAML, l 1 loss
Revised: E rr o r er ( A ( S ) ,τ ) ˆ er ( A ( S ) , S ) | er − ˆ er | (d) MAML, l 1 loss

Original: To verify our theoretical analysis, we conduct experiments on few-shot regression problems to showthe convergence performance of our generalization bounds with independent and dependent episodes.
Revised: To verify our theoretical analysis, we conduct experiments on few-shot regression problems to showthe convergence performance of our generalization bounds with independent and dependent episodes.

Original: Experimental Settings .
Revised: Experimental Settings .

Original: We follow the experimental setting in [20, 8].
Revised: We follow the experimental setting in [20, 8].

Original: The problem aims toapproximate the distribution of parameters of function f ( x )
Revised: The problem aims toapproximate the distribution of parameters of function f ( x )

Original: =
Revised: =

Original: α sin( βx ) .
Revised: α sin( βx ) .

Original: The task environment τ isthe joint distribution p ( α, β ) of the parameters α and β .
Revised: The task environment τ isthe joint distribution p ( α, β ) of the parameters α and β .

Original: We set p ( α ) =
Revised: We set p ( α ) =

Original: U [ − 5 , 5] , p ( β ) =
Revised: U [ − 5 , 5] , p ( β ) =

Original: U [0 , π ] .
Revised: U [0 , π ] .

Original: For independent setting, we construct training episodes by sampling pairs ( α, β ) from the taskdistribution τ = p ( α, β ) ; for dependent setting, we construct the first half training episodes bysampling pairs ( α, β ) from τ = p ( α, β ) independently, and construct the rest half training episodesby setting ( − α, π − β ) with ( α, β ) from the first half training episodes.
Revised: For independent setting, we construct training episodes by sampling pairs ( α, β ) from the taskdistribution τ = p ( α, β ) ; for dependent setting, we construct the first half training episodes bysampling pairs ( α, β ) from τ = p ( α, β ) independently, and construct the rest half training episodesby setting ( − α, π − β ) with ( α, β ) from the first half training episodes.

Original: Each training episode contains5 support samples and 1 query samples.
Revised: Each training episode contains5 support samples and 1 query samples.

Original: In both settings, the 600 test episodes are constructed bysampling ( α, β ) from τ independently, each containing 5 support samples and 15 query samples.
Revised: In both settings, the 600 test episodes are constructed bysampling ( α, β ) from τ independently, each containing 5 support samples and 15 query samples.

Original: Weimplement meta learning algorithms MAML [20] and Bilevel Programming [21] with square loss ( l 2 )and absolute loss ( l 1 ).
Revised: Weimplement meta learning algorithms MAML [20] and Bilevel Programming [21] with square loss ( l 2 )and absolute loss ( l 1 ).

Original: The neural network has two hidden layers of size 40 with ReLU activationfunctions.
Revised: The neural network has two hidden layers of size 40 with ReLU activationfunctions.

Original: The generalization gap is the absolute distance between the training error and test error.
Revised: The generalization gap is the absolute distance between the training error and test error.

Original: Experimental Results .
Revised: Experimental Results .

Original: From Figures 1-2, we can observe that: (1) The generalization gap in bothindependent and weakly dependent episode settings can converge to 0 with the increase of the trainingepisodes, demonstrating the asymptotic behaviour of our transfer error bounds in Theorems 5 and 7.(2)
Revised: From Figures 1-2, we can observe that: (1) The generalization gap in bothindependent and weakly dependent episode settings can converge to 0 with the increase of the trainingepisodes, demonstrating the asymptotic behaviour of our transfer error bounds in Theorems 5 and 7.(2)

Original: The generalization gap with independent episodes can converge to zero more quickly than the gapwith dependent episodes.
Revised: The generalization gap with independent episodes can converge to zero more quickly than the gapwith dependent episodes.

Original: The test error with independent episodes also always converge to the lowerlevel than the one with dependent episodes.
Revised: The test error with independent episodes also always converge to the lowerlevel than the one with dependent episodes.

Original: The better convergence performance with independentepisodes truly demonstrate how the dependency between episodes can affect the generalizationof meta learning algorithms.
Revised: The better convergence performance with independentepisodes truly demonstrate how the dependency between episodes can affect the generalizationof meta learning algorithms.

Original: (3) With non-convex neural network models, both square loss andnonsmooth absolute loss can lead to similar convergence performance of generalization bounds.
Revised: (3) With non-convex neural network models, both square loss andnonsmooth absolute loss can lead to similar convergence performance of generalization bounds.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: In this work, we provide fine-grained analysis of stability and generalization for modern meta learningalgorithms.
Revised: In this work, we provide fine-grained analysis of stability and generalization for modern metalearning algorithms.

Original: From the perspective of stability, our tight stability bounds implies that in the nonsmoothconvex case the meta learning algorithm is less stable than in the smooth convex case.
Revised: From the perspective of stability, our tight stability bounds implies that in thenonsmooth convex case the meta learning algorithm is less stable than in the smooth convex case.

Original: The stabilitybounds in the smooth non-convex case enjoys an order of O (1 /n ) even for the multi-pass SGD setting.
Revised: Thestability bounds in the smooth non-convex case enjoys an order of O (1 /n ) even for the multi-pass SGD setting.

Original: From the perspective of generalization, we demonstrate that the high-probability transfer error boundof O (1 / √ n ) is optimal.
Revised: From the perspective of generalization, we demonstrate that the high-probabilitytransfer error bound of O (1 / √ n ) is optimal.

Original: Based on this bound, we uncover the limitations of algorithmic stabilityanalysis for meta learning, and reveal the advantage of episodic training strategy for meta learningover tradition ERM training strategy.
Revised: Based on this bound, we uncover the limitations ofalgorithmic stability analysis for meta learning, and reveal the advantage of episodic training strategyfor meta learning over tradition ERM training strategy.

Original: Further, by extending the generalized Berstein condition to themeta learning setting, we obtain a fast-rate generalization bound of O (ln n/n ) with additional PolyakŁojasiewicz condition.
Revised: Further, by extending the generalized Bersteincondition to the meta learning setting, we obtain a deformed generalization bound of O (ln n/n ) withadditional Polyak-Łojasiewicz condition.

Original: Finally, we derive a generalization bound for meta learning with dependentepisodes.
Revised: Finally, we derive a generalization bound for meta learningwith dependent episodes.

Original: Experiments are also provided to show the convergence performance of generalizationerror with independent and dependent episodes.
Revised: Experiments are also provided to show the convergence performance of generalization error with independent and dependent episodes.

Original: In the future, we will explore new stability notionsto see whether we can develop sharper generalization bounds(e.g. of O ( 1 nm ) ) for meta learning.
Revised: In the future, we will explore new stability notions to see whether we can develop sharper generalization bounds for meta learning

Original: Checklist
Revised: Checklist

Original: authors...
Revised: authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Remark 5.
Revised: [Yes] See Remark 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Sections
Revised: [Yes] See Sections

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [Yes] See Supple-
Revised: [Yes] See Supple-

Original: mentary Material.
Revised: mentary Material.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See Section 6.
Revised: [Yes] See Section 6.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] See Figures 1-2.
Revised: [Yes] See Figures 1-2.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Supplementary Material.
Revised: [Yes] See Supplementary Material.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 6.
Revised: [Yes] See Section 6.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A] [Table]
Revised: [N/A] [Table]