Original: T RANSFERRING K NOWLEDGE ACROSS L EARNING P ROCESSES
Revised: T RANSFERRING K NOWLEDGE ACROSS L EARNING P ROCESSES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks.
Revised: In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks.

Original: Approaches that transfer information contained only in the ﬁnal parameters of a source model will therefore struggle.
Revised: Approaches that transfer information contained only in the ﬁnal parameters of a source model will therefore struggle.

Original: Instead, transfer learning at a higher level of abstraction is needed.
Revised: Instead, transfer learning at a higher level of abstraction is needed.

Original: We propose Leap, a framework that achieves this by transferring knowledge across learning processes.
Revised: We propose Leap, a framework that achieves this by transferring knowledge across learning processes.

Original: We associate each task with a manifold on which the training process travels from initialization to ﬁnal parameters and construct a meta learning objective that minimizes the expected length of this path.
Revised: We associate each task with a manifold on which the training process travels from initialization to ﬁnal parameters and construct a meta learning objective that minimizes the expected length of this path.

Original: Our framework leverages only information obtained during training and can be computed on the ﬂy at negligible cost.
Revised: Our framework leverages only information obtained during training and can be computed on the ﬂy at negligible cost.

Original: We demonstrate that our framework outperforms competing methods, both in meta learning and transfer learning, on a set of computer vision tasks.
Revised: We demonstrate that our framework outperforms competing methods, both in meta learning and transfer learning, on a set of computer vision tasks.

Original: Finally, we demonstrate that Leap can transfer knowledge across learning processes in demanding Reinforcement learning environments (Atari) that involve millions of gradient steps.
Revised: Finally, we demonstrate that Leap can transfer knowledge across learning processes in demanding Reinforcement learning environments (Atari) that involve millions of gradient steps.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transfer learning is the process of transferring knowledge encoded in one model trained on one set of tasks to another model that is applied to a new task.
Revised: Transfer learning is the process of transferring knowledge encoded in one model trained on one set of tasks to another model that is applied to a new task.

Original: Since a trained model encodes information in its learned parameters, transfer learning typically transfers knowledge by encouraging the target model’s parameters to resemble those of a previous (set of) model(s) (Pan & Yang, 2009).
Revised: Since a trained model encodes information in its learned parameters, transfer learning typically transfers knowledge by encouraging the target model’s parameters to resemble those of a previous (set of) model(s) (Pan & Yang, 2009).

Original: This approach limits transfer learning to settings where good parameters for a new task can be found in the neighborhood of parameters that were learned from a previous task.
Revised: This approach limits transfer learning to settings where good parameters for a new task can be found in the neighborhood of parameters that were learned from a previous task.

Original: For this to be a viable assumption, the two tasks must have a high degree of structural afﬁnity, such as when a new task can be learned by extracting features from a pre-trained model (Girshick et al., 2014;
Revised: For this to be a viable assumption, the two tasks must have a high degree of structural afﬁnity, such as when a new task can be learned by extracting features from a pretrained model (Girshick et al., 2014;

Original: He et al., 2017;
Revised: He et al., 2017;

Original: Mahajan et al., 2018).
Revised: Mahajan et al., 2018).

Original: If not, this approach has been observed to limit knowledge transfer since the training process on one task will discard information that was irrelevant for the task at hand, but that would be relevant for another task (Higgins et al., 2017; Achille et al., 2018).
Revised: If not, this approach has been observed to limit knowledge transfer since the training process on one task will discard information that was irrelevant for the task at hand, but that would be relevant for another task (Higgins et al., 2017; Achille et al., 2018).

Original: In this paper we argue that such information can be harnessed, even when the downstream task is unknown, by transferring knowledge of the learning process itself.
Revised: In this paper we argue that such information can be harnessed, even when the downstream task is unknown, by transferring knowledge of the learning process itself.

Original: In particular, we propose a meta learning framework for aggregating information across task geometries as they are observed during training.
Revised: In particular, we propose a meta learning framework for aggregating information across task geometries as they are observed during training.

Original: These geometries, formalized as the loss surface over the training data, encode all information seen during training and therefore avoid catastrophic loss of information.
Revised: These geometries, formalized as the loss surface over the training data, encode all information seen during training and therefore avoid catastrophic loss of information.

Original: Moreover, by transferring knowledge across learning processes, information from previous tasks is distilled to explicitly facilitate the learning of a new task.
Revised: Moreover, by transferring knowledge across learning processes, information from previous tasks is distilled to explicitly facilitate the learning of a new task.

Original: Meta learning frames the learning of a new task as a learning problem itself, typically in the few-shot learning paradigm (Lake et al., 2011; Santoro et al., 2016; Vinyals et al., 2016).
Revised: Meta learning frames the learning of a new task as a learning problem itself, typically in the few-shot learning paradigm (Lake et al., 2011; Santoro et al., 2016; Vinyals et al., 2016).

Original: In this environment, learning is a problem of rapid adaptation and can be solved by training a meta learner by backpropagating through the entire training process (Ravi & Larochelle, 2016; Andrychowicz et al., 2016;
Revised: In this environment, learning is a problem of rapid adaptation and can be solved by training a meta learner by backpropagating through the entire training process (Ravi & Larochelle, 2016; Andrychowicz et al., 2016;

Original: Finn et al., 2017).
Revised: Finn et al., 2017).

Original: For more demanding tasks, formulating the meta learning in this manner is challenging; backpropagating through thousands of gradient steps is both impractical and susceptible to instability.
Revised: For more demanding tasks, formulating the meta learning in this manner is challenging; backpropagating through thousands of gradient steps is both impractical and susceptible to instability.

Original: We argue that as the training process grows longer in terms of the distance traversed on the loss landscape, the geometry of this landscape grows increasingly important.
Revised: We argue that as the training process grows longer in terms of the distance traversed on the loss landscape, the geometry of this landscape grows increasingly important.

Original: When adapting to a new task through a single or a handful of gradient steps, the geometry can largely be ignored.
Revised: When adapting to a new task through a single or a handful of gradient steps, the geometry can largely be ignored.

Original: In contrast, with more gradient steps, it is the dominant feature of the training process.
Revised: In contrast, with more gradient steps, it is the dominant feature of the training process.

Original: [Figure]
Revised: [Figure]

Original: To scale meta learning beyond few-shot learning, we propose Leap , a light-weight framework for meta learning over task manifolds that does not need any forward- or backward-passes beyond those already performed by the underlying training process.
Revised: To scale meta learning beyond few-shot learning, we propose Leap , a light-weight framework for meta learning over task manifolds that does not need any forward- or backward-passes beyond those already performed by the underlying training process.

Original: We demonstrate empirically that Leap is a superior method to similar meta and transfer learning methods when learning a task requires more than a handful of training steps.
Revised: We demonstrate empirically that Leap is a superior method to similar meta and transfer learning methods when learning a task requires more than a handful of training steps.

Original: Finally, we evaluate Leap in a Reinforcement Learning environment (Atari 2600; Bellemare et al., 2013), demonstrating that it can transfer knowledge across learning processes that require millions of gradient steps to converge.
Revised: Finally, we evaluate Leap in a Reinforcement Learning environment (Atari 2600; Bellemare et al., 2013), demonstrating that it can transfer knowledge across learning processes that require millions of gradient steps to converge.

Original:  T RANSFERRING K NOWLEDGE OVER L EARNING P ROCESSES
Revised:  T RANSFERRING K NOWLEDGE OVER L EARNING P ROCESSES

Original: We start in section 2.1 by introducing the gradient descent algorithm from a geometric perspective.
Revised: We start in section 2.1 by introducing the gradient descent algorithm from a geometric perspective.

Original: Section 2.2 builds a framework for transfer learning and explains how we can leverage geometrical quantities to transfer knowledge across learning processes by guiding gradient descent.
Revised: Section 2.2 builds a framework for transfer learning and explains how we can leverage geometrical quantities to transfer knowledge across learning processes by guiding gradient descent.

Original: We focus on the point of initialization for simplicity, but our framework can readily be extended.
Revised: We focus on the point of initialization for simplicity, but our framework can readily be extended.

Original: Section 2.
Revised: Section 2.

Original: presents Leap, our lightweight algorithm for transfer learning across learning processes.
Revised: presents Leap, our lightweight algorithm for transfer learning across learning processes.

Original: 2. G RADIENT P ATHS ON T ASK M ANIFOLDS
Revised: 2. G RADIENT P ATHS ON T ASK M ANIFOLDS

Original: Central to our framework is the notion of a learning process; the harder a task is to learn, the harder it is for the learning process to navigate on the loss surface (ﬁg. 1).
Revised: Central to our framework is the notion of a learning process; the harder a task is to learn, the harder it is for the learning process to navigate on the loss surface (ﬁg. 1).

Original: Our framework is based on the idea that transfer learning can be achieved by leveraging information contained in similar learning processes.
Revised: Our framework is based on the idea that transfer learning can be achieved by leveraging information contained in similar learning processes.

Original: Exploiting that this information is encoded in the geometry of the loss surface, we leverage geometrical quantities to facilitate the learning process with respect to new tasks.
Revised: Exploiting that this information is encoded in the geometry of the loss surface, we leverage geometrical quantities to facilitate the learning process with respect to new tasks.

Original: We focus on the supervised learning setting for simplicity, though our framework applies more generally.
Revised: We focus on the supervised learning setting for simplicity, though our framework applies more generally.

Original: Given a learning objective f that consumes an input x ∈ R m and a target y ∈ R c and maps a parameterization θ ∈ R n to a scalar loss value, we have the gradient descent update as
Revised: Given a learning objective f that consumes an input x ∈ R m and a target y ∈ R c and maps a parameterization θ ∈ R n to a scalar loss value, we have the gradient descent update as

Original: [Equation]
Revised: [Equation]

Original: (1) where ∇ f ( θ i ) =
Revised: (1) where ∇ f ( θ i ) =

Original: E x,y ∼ p ( x,y ) (cid:2) ∇ f
Revised: E x,y ∼ p ( x,y ) (cid:2) ∇ f

Original: ( θ i , x, y ) (cid:3) .
Revised: ( θ i , x, y ) (cid:3) .

Original: We take the learning rate schedule { α i } i and preconditioning matrices { S i } i as given, but our framework can be extend to learn these jointly with the initialization.
Revised: We take the learning rate schedule { α i } i and preconditioning matrices { S i } i as given, but our framework can be extend to learn these jointly with the initialization.

Original: Differences schemes represent different optimizers; for instance, α i = α, S i
Revised: Differences schemes represent different optimizers; for instance, α i = α, S i

Original: = I n yields gradient descent, while deﬁning S i as the inverse Fisher matrix results in natural gradient descent (Amari, 1998).
Revised: = I n yields gradient descent, while deﬁning S i as the inverse Fisher matrix results in natural gradient descent (Amari, 1998).

Original: We assume this process converges to a stationary point after K gradient steps.
Revised: We assume this process converges to a stationary point after K gradient steps.

Original: To distinguish different learning processes originating from the same initialization, we need a notion of their length.
Revised: To distinguish different learning processes originating from the same initialization, we need a notion of their length.

Original: The longer the process, the worse the initialization is (conditional on reaching equivalent performance, discussed further below).
Revised: The longer the process, the worse the initialization is (conditional on reaching equivalent performance, discussed further below).

Original: Measuring the Euclidean distance between initialization and ﬁnal parameters is misleading as it ignores the actual path taken.
Revised: Measuring the Euclidean distance between initialization and ﬁnal parameters is misleading as it ignores the actual path taken.

Original: This becomes crucial when we compare paths from different tasks, as gradient paths from different tasks can originate from the same initialization and converge to similar ﬁnal parameters, but take very different paths.
Revised: This becomes crucial when we compare paths from different tasks, as gradient paths from different tasks can originate from the same initialization and converge to similar ﬁnal parameters, but take very different paths.

Original: Therefore, to capture the length of a learning process we must associate it with the loss surface it traversed.
Revised: Therefore, to capture the length of a learning process we must associate it with the loss surface it traversed.

Original: The process of learning a task can be seen as a curve on a speciﬁc task manifold M .
Revised: The process of learning a task can be seen as a curve on a speciﬁc task manifold M .

Original: While this manifold can be constructed in a variety of ways, here we exploit that, by deﬁnition, any learning process traverses the loss surface of f .
Revised: While this manifold can be constructed in a variety of ways, here we exploit that, by deﬁnition, any learning process traverses the loss surface of f .

Original: As such, to accurately describe the length of a gradient-based learning process, it is sufﬁcient to deﬁne the task manifold as the loss surface.
Revised: As such, to accurately describe the length of a gradient-based learning process, it is sufﬁcient to deﬁne the task manifold as the loss surface.

Original: In particular, because the learning process in eq.
Revised: In particular, because the learning process in eq.

Original: 1 follows the gradient trajectory, it constantly provides information about the geometry of the loss surface.
Revised: 1 follows the gradient trajectory, it constantly provides information about the geometry of the loss surface.

Original: Gradients that largely point in the same direction indicate a well-behaved loss surface, whereas gradients with frequently opposing directions indicate an ill-conditioned loss surface, something we would like to avoid.
Revised: Gradients that largely point in the same direction indicate a well-behaved loss surface, whereas gradients with frequently opposing directions indicate an ill-conditioned loss surface, something we would like to avoid.

Original: Leveraging this insight, we propose a framework for transfer learning that exploits the accumulation of geometric information by constructing a meta objective that minimizes the expected length of the gradient descent path across tasks .
Revised: Leveraging this insight, we propose a framework for transfer learning that exploits the accumulation of geometric information by constructing a meta objective that minimizes the expected length of the gradient descent path across tasks .

Original: In doing so, the meta objective intrinsically balances local geometries across tasks and encourages an initialization that makes the learning process as short as possible.
Revised: In doing so, the meta objective intrinsically balances local geometries across tasks and encourages an initialization that makes the learning process as short as possible.

Original: To formalize the notion of the distance of a learning process, we deﬁne a task manifold M as a submanifold of R n +1 given by the graph of f .
Revised: To formalize the notion of the distance of a learning process, we deﬁne a task manifold M as a submanifold of R n +1 given by the graph of f .

Original: Every point p =
Revised: Every point p =

Original: ( θ, f ( θ ))
Revised: ( θ, f ( θ ))

Original: ∈ M is locally homeomorphic to a Euclidean subspace, described by the tangent space T p M .
Revised: ∈ M is locally homeomorphic to a Euclidean subspace, described by the tangent space T p M .

Original: Taking R n +1 to be
Revised: Taking R n +1 to be

Original: Euclidean, it is a Riemann manifold.
Revised: Euclidean, it is a Riemann manifold.

Original: By virtue of being a submanifold of R n +1 , M is also a Riemann manifold.
Revised: By virtue of being a submanifold of R n +1 , M is also a Riemann manifold.

Original: This is important because a Riemann manifold comes equipped with an smoothly varying inner product g p : T p M × T p M (cid:55)→ R on tangent spaces, allowing us to measure the length of a path on M .
Revised: As such, M comes equipped with an smoothly varying inner product g p : T p M × T p M (cid:55)→ R on tangent spaces, allowing us to measure the length of a path on M .

Original: In particular, the length (or energy) of any curve γ : [0 , 1] (cid:55)→ M is deﬁned by accumulating inﬁnitesimal changes along the trajectory,
Revised: In particular, the length (or energy) of any curve γ : [0 , 1] (cid:55)→ M is deﬁned by accumulating inﬁnitesimal changes along the trajectory,

Original: [Equation]
Revised: [Equation]

Original: where ˙ γ ( t ) = ddt γ ( t ) ∈
Revised: where ˙ γ ( t ) = ddt γ ( t ) ∈

Original: T γ ( t ) M is a tangent vector of γ ( t ) =
Revised: T γ ( t ) M is a tangent vector of γ ( t ) =

Original: ( θ ( t ) , f ( θ ( t )))
Revised: ( θ ( t ) , f ( θ ( t )))

Original: ∈ M .
Revised: ∈ M .

Original: We use parenthesis (i.e. γ ( t ) ) to differentiate discrete and continuous domains.
Revised: We use parenthesis (i.e. γ ( t ) ) to differentiate discrete and continuous domains.

Original: With M being a submanifold of R n +1 , it inherits its metric via pullback, with induced metric g γ ( t ) (˙ γ ( t ) , ˙ γ ( t ))
Revised: With M being a submanifold of R n +1 , it inherits its metric via pullback, with induced metric g γ ( t ) (˙ γ ( t ) , ˙ γ ( t ))

Original: = (cid:104) ˙
Revised: = (cid:104) ˙

Original: γ ( t ) , ˙ γ ( t ) (cid:105) .
Revised: γ ( t ) , ˙ γ ( t ) (cid:105) .

Original: Different Riemann metrics yield different manifold structures.
Revised: Different Riemann metrics yield different manifold structures.

Original: In particular, if the model underlying f admits a predictive probability distribution P ( y | x ) , the task manifold can be given an information geometric interpretation by choosing the Fisher matrix as Riemann metric, in which case the task manifold is deﬁned over the space of probability distributions (Amari & Nagaoka, 2007).
Revised: In particular, if the model underlying f admits a predictive probability distribution P ( y | x ) , the task manifold can be given an information geometric interpretation by choosing the Fisher matrix as Riemann metric, in which case the task manifold is deﬁned over the space of probability distributions (Amari & Nagaoka, 2007).

Original: If eq.is deﬁned as natural gradient descent, the learning process corresponds to gradient descent on this manifold (Amari, 1998; Martens, 2010; Pascanu & Bengio, 2014; Luk & Grosse, 2018).
Revised: If eq.is deﬁned as natural gradient descent, the learning process corresponds to gradient descent on this manifold (Amari, 1998; Martens, 2010; Pascanu & Bengio, 2014; Luk & Grosse, 2018).

Original: Having a complete description of a task manifold, we can measure the length of a learning process by noting gradient descent can be seen as a discrete approximation to the scaled gradient ﬂow ˙ θ ( t ) = − S ( t ) ∇ f ( θ ( t )) .
Revised: Having a complete description of a task manifold, we can measure the length of a learning process by noting gradient descent can be seen as a discrete approximation to the scaled gradient ﬂow ˙ θ ( t ) = − S ( t ) ∇ f ( θ ( t )) .

Original: This ﬂow describes a curve that originates in γ (0) =
Revised: This ﬂow describes a curve that originates in γ (0) =

Original: ( θ 0 , f
Revised: ( θ 0 , f

Original: ( θ 0 ))
Revised: ( θ 0 ))

Original: and follows the  gradient at each point.
Revised: and follows the scaled gradient at each point.

Original: Going forward, we deﬁne γ to be this unique curve and refer to it as the gradient path from θ 0 on M .
Revised: Going forward, we deﬁne γ to be this unique curve and refer to it as the gradient path from θ 0 on M .

Original: The gradient path length or energy can be computed exactly, but in practice we observe a discrete learning process.
Revised: The gradient path length or energy can be computed exactly, but in practice we observe a discrete learning process.

Original: Analogously to how the gradient update rule approximates the gradient ﬂow, the gradient path length or energy can be approximated by the cumulative chordal distance (Ahlberg et al., 1967),
Revised: Analogously to how the gradient update rule approximates the gradient ﬂow, the gradient path length or energy can be approximated by the cumulative chordal distance (Ahlberg et al., 1967),

Original: [Equation]
Revised: [Equation]

Original: We write d when the distinction between the length or energy metric is immaterial.
Revised: We write d when the distinction between the length or energy metric is immaterial.

Original: Using the energy yields a slightly simpler objective, but the length normalizes each length segment and as such protects against differences in scale between task objectives.
Revised: Using the energy yields a slightly simpler objective, but the length normalizes each length segment and as such protects against differences in scale between task objectives.

Original: In appendix C, we carry out an ablation study and ﬁnd that they perform similarly, though using the length leads to faster convergence.
Revised: In appendix C, we conduct an ablation study and ﬁnd that they perform similarly, though using the length leads to faster convergence.

Original: Importantly, d involves only terms seen during task training.
Revised: Importantly, d involves only terms seen during task training.

Original: We exploit this later when we construct the meta gradient, enabling us to perform gradient descent on the meta objective at negligible cost (eq. 8).
Revised: We exploit this later when we construct the meta gradient, enabling us to perform gradient descent on the meta objective at negligible cost (eq. 8).

Original: We now turn to the transfer learning setting, where we face a set of tasks, each with a distinct task manifold.
Revised: We now turn to the transfer learning setting, where we face a set of tasks, each with a distinct task manifold.

Original: Our framework is built on the idea that we can transfer knowledge across learning processes via the local geometry by aggregating information obtained along observed gradient paths.
Revised: Our framework is built on the idea that we can transfer knowledge across learning processes via the local geometry by aggregating information obtained along observed gradient paths.

Original: As such,
Revised: As such,

Original: Leap ﬁnds an initialization from which learning converges as rapidly as possible in expectation.
Revised: Leap ﬁnds an initialization from which learning converges as rapidly as possible in expectation.

Original: [Figure]

Original: τ ψ

Original: [Figure]
Revised: [Figure]

Original: [-ψ 2 τ ψ 1 τ ψ 0 τ θ 3 τ θ 2 τ θ 1 τ θ 
-]
Revised: [-ψ 2 τ ψ 1 τ ψ 0 τ θ 3 τ θ 2 τ θ 1 τ θ 
-]

Original: τ 2. M ETA L EARNING ACROSS T ASK M ANIFOLDS
Revised: M ETA L EARNING ACROSS T ASK M ANIFOLDS

Original: Formally, we deﬁne a task τ
Revised: Formally, we deﬁne a task τ

Original: =
Revised: =

Original: ( f τ , p τ , u τ ) as the process of learning to approximate the relationship x (cid:55)→ y through samples from the data distribution p τ ( x, y ) .
Revised: ( f τ , p τ , u τ ) as the process of learning to approximate the relationship x (cid:55)→ y through samples from the data distribution p τ ( x, y ) .

Original: This process is deﬁned by the gradient update rule u τ (as deﬁned in eq. 1), applied K τ times to minimize the task objective f τ .
Revised: This process is deﬁned by the gradient update rule u τ (as deﬁned in eq. 1), applied K τ times to minimize the task objective f τ .

Original: Thus, a learning process starts at θ 0 τ = θ 0 and progress via θ i +

Revised: Thus, a learning process starts at θ 0 τ = θ 0 and progress via θ i +

Original: τ =
Revised: τ =

Original: u τ ( θ iτ ) until θ K τ τ is obtained.
Revised: u τ ( θ iτ ) until θ K τ τ is obtained.

Original: The sequence { θ iτ } K τ i =0 deﬁnes an approximate gradient path on the task manifold M τ with distance d ( θ 0 ; M τ ) .
Revised: The sequence { θ iτ } K τ i =0 deﬁnes an approximate gradient path on the task manifold M τ with distance d ( θ 0 ; M τ ) .

Original: To understand how d transfers knowledge across learning processes, consider two distinct tasks.
Revised: To understand how d transfers knowledge across learning processes, consider two distinct tasks.

Original: We can transfer knowledge across these tasks’ learning processes by measuring how good a shared initialization is.
Revised: We can transfer knowledge across these tasks’ learning processes by measuring how good a shared initialization is.

Original: Assuming two candidate initializations converge to limit points with equivalent performance on each task, the initialization with shortest  distance encodes more knowledge sharing.
Revised: Assuming two candidate initializations converge to limit points with equivalent performance on each task, the initialization with shortest expected gradient path distance encodes more knowledge sharing.

Original: In particular, if both tasks have convex loss surfaces there is a unique optimal initialization that achieves Pareto optimality in terms of total path distance.
Revised: In particular, if both tasks have convex loss surfaces there is a unique optimal initialization that achieves Pareto optimality in terms of total path distance.

Original: This can be crucial for tasks with limited data: rapid convergence may be the difference between learning a task and failing due to overﬁtting (Finn et al., 2017).
Revised: This can be crucial in data sparse regimes: rapid convergence may be the difference between learning a task and failing due to overﬁtting (Finn et al., 2017).

Original: Given a distribution of tasks p ( τ ) , each candidate initialization θ 0 is associated with a measure of its expected gradient path distance, E τ ∼ p ( τ ) (cid:2)
Revised: Given a distribution of tasks p ( τ ) , each candidate initialization θ 0 is associated with a measure of its expected gradient path distance, E τ ∼ p ( τ ) (cid:2)

Original: d ( θ 0 ; M τ ) (cid:3) , that summarizes the suitability of the initialization to the task distribution.
Revised: d ( θ 0 ; M τ ) (cid:3) , that summarizes the suitability of the initialization to the task distribution.

Original: The initialization (or a set thereof) with shortest expected gradient path distance maximally transfers knowledge across learning processes and is Pareto optimal in this regard.
Revised: The initialization (or a set thereof) with shortest expected gradient path distance maximally transfers knowledge across learning processes and is Pareto optimal in this regard.

Original: Above, we have assumed that all candidate initializations converge to limit points of equal performance.
Revised: Above, we have assumed that all candidate initializations converge to limit points of equal performance.

Original: If the task objective f τ is non-convex this is not a trivial assumption and the gradient path distance itself does not differentiate between different levels of ﬁnal performance.
Revised: If the task objective f τ is non-convex this is not a trivial assumption and the gradient path distance itself does not differentiate between different levels of ﬁnal performance.

Original: As such, it is necessary to introduce a feasibility constraint to ensure only initializations with some minimum level of performance are considered.
Revised: As such, it is necessary to introduce a feasibility constraint to ensure only initializations with some minimum level of performance are considered.

Original: We leverage that transfer learning never happens in a vacuum; we always have a second-best option, such as starting from a random initialization or a pre-trained model.
Revised: We leverage that transfer learning never happens in a vacuum; we always have a second-best option, such as starting from a random initialization or a pretrained model.

Original: Starting from this “second-best” initialization, ψ 0 , gives us the level of performance we would obtain on each task in the absence of transferring knowledge across learning processes.
Revised: Starting from this “second-best” initialization, ψ 0 , gives us the level of performance we would obtain on each task in the absence of transferring knowledge across learning processes.

Original: As such, performance obtained by initializing from ψ 0 provides us with an upper bound for each task: a candidate solution θ 0 has to yield at least as good performance on any task in p ( τ ) to be a viable solution.
Revised: As such, performance obtained by initializing from ψ 0 provides us with an upper bound for each task: a candidate solution θ 0 has to yield at least as good performance on any task in p ( τ ) to be a viable solution.

Original: Formally, this implies the task-speciﬁc requirement that a candidate θ 0 must satisfy f τ ( θ K τ τ ) ≤ f τ ( ψ K τ τ ) .
Revised: Formally, this implies the task-speciﬁc requirement that a candidate θ 0 must satisfy f τ ( θ K τ τ ) ≤ f τ ( ψ K τ τ ) .

Original: As this must hold for every task, we obtain the canonical meta objective
Revised: As this must hold for every task, we obtain the canonical meta objective

Original: [Equation]
Revised: [Equation]

Original: Require: p ( τ ) , τ = ( f τ , u τ , p τ ) : distribution over tasks
Revised: Require: p ( τ ) , τ = ( f τ , u τ , p τ ) : distribution over tasks

Original: Require: β : step size 1: randomly initialize θ2: while not done do 3: ∇
Revised: Require: β : step size 1: randomly initialize θ2: while not done do 3: ∇

Original: F
Revised: F

Original: B do 6: ψ
Revised: B do 6: ψ

Original: 7: for do 8:
Revised: 7: for do 8:

Original: 10: increment ∇
Revised: 10: increment ∇

Original: 11: end
Revised: 11: end

Original: 12: end for 13:
Revised: 12: end for 13:

Original: 14: end while
Revised: 14: end while

Original: This meta objective is robust to variations in the geometry of loss surfaces, as it balances complementary and competing learning processes (ﬁg. 2).
Revised: This meta objective is robust to variations in the geometry of loss surfaces, as it balances complementary and competing learning processes (ﬁg. 2).

Original: For instance, there may be an initialization that can solve a small subset of tasks in a handful of gradient steps, but would be catastrophic for other related tasks.
Revised: For instance, there may be an initialization that can solve a small subset of tasks in a handful of gradient steps, but would be catastrophic for other related tasks.

Original: When transferring knowledge via the initialization, we must trade off commonalities and differences between gradient paths.
Revised: When transferring knowledge via the initialization, we must trade off commonalities and differences between gradient paths.

Original: In eq.
Revised: In eq.

Original: 4 these trade-offs arise naturally.
Revised: 4 these trade-offs arise naturally.

Original: For instance, as the number of tasks whose gradient paths move in the same direction increases, so does their pull on the initialization.
Revised: For instance, as the number of tasks whose gradient paths move in the same direction increases, so does their pull on the initialization.

Original: Conversely, as the updates to the initialization renders some gradient paths longer, these act as springs that exert increasingly strong pressure on the initialization.
Revised: Conversely, as the updates to the initialization renders some gradient paths longer, these act as springs that exert increasingly strong pressure on the initialization.

Original: The solution to eq.thus achieves an equilibrium between these competing forces.
Revised: The solution to eq.thus achieves an equilibrium between these competing forces.

Original: Solving eq.
Revised: Solving eq.

Original: 4 naively requires training to convergence on each task to determine whether an initialization satisﬁes the feasibility constraint, which can be very costly.
Revised: 4 naively requires training to convergence on each task to determine whether an initialization satisﬁes the feasibility constraint, which can be very costly.

Original: Fortunately, because we have access to a second-best initialization, we can solve eq.
Revised: Fortunately, because we have access to a second-best initialization, we can solve eq.

Original: 4 more efﬁciently by obtaining gradient paths from ψ 0 and use these as baselines that we incrementally improve upon.
Revised: 4 more efﬁciently by obtaining gradient paths from ψ 0 and use these as baselines that we incrementally improve upon.

Original: This improved initialization converges to the same limit points, but with shorter expected gradient path distances (theorem 1).
Revised: This improved initialization converges to the same limit points, but with shorter expected gradient path distances (theorem 1).

Original: As such, it becomes the new second-best option; Leap (algorithm 1) repeats this process of improving upon increasingly demanding baselines, ultimately ﬁnding a solution to the canonical meta objective.
Revised: As such, it becomes the new second-best option; Leap (algorithm 1) repeats this process of improving upon increasingly demanding baselines, ultimately ﬁnding a solution to the canonical meta objective.

Original: 2. L EAP
Revised: 2. L EAP

Original: Leap starts from a given second-best initialization ψ 0 , shared across all tasks, and construct baseline gradient paths
Revised: Leap starts from a given second-best initialization ψ 0 , shared across all tasks, and construct baseline gradient paths

Original: Ψ τ = { ψ iτ } K τ i =0 for each task τ in a batch B .
Revised: Ψ τ = { ψ iτ } K τ i =0 for each task τ in a batch B .

Original: These provide a set of baselines Ψ
Revised: These provide a set of baselines Ψ

Original: = { Ψ τ } τ ∈B ; recall that all tasks share the same initialization, ψ 0 τ = ψ 0 ∈ Θ .
Revised: = { Ψ τ } τ ∈B ; recall that all tasks share the same initialization, ψ 0 τ = ψ 0 ∈ Θ .

Original: We use these baselines, corresponding to task-speciﬁc learning processes, to modify the gradient path distance metric in eq. 3 by freezing the forward point γ i +1 τ in all norms,
Revised: We use these baselines, corresponding to task-speciﬁc learning processes, to modify the gradient path distance metric in eq. 3 by freezing the forward point γ i +1 τ in all norms,

Original: [Equation]
Revised: [Equation]

Original: where ¯ γ iτ = ( ψ iτ , f ( ψ iτ )) represents the frozen forward point from the baseline and γ iτ =
Revised: where ¯ γ iτ = ( ψ iτ , f ( ψ iτ )) represents the frozen forward point from the baseline and γ iτ =

Original: ( θ iτ , f ( θ iτ ))
Revised: ( θ iτ , f ( θ iτ ))

Original: the point on the gradient path originating from θ 0 .
Revised: the point on the gradient path originating from θ 0 .

Original: This surrogate distance metric directly encodes the feasibility constraint; optimizing θ 0 with respect to Ψ therefore pulls the initialization forward along each task-speciﬁc gradient path in an unconstrained variant of eq. 4 that replaces Θ with Ψ ,
Revised: This surrogate distance metric directly encodes the feasibility constraint; optimizing θ 0 with respect to Ψ therefore pulls the initialization forward along each task-speciﬁc gradient path in an unconstrained variant of eq. 4 that replaces Θ with Ψ ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We refer to eq.
Revised: We refer to eq.

Original: 6 as the pull-forward objective.
Revised: 6 as the pull-forward objective.

Original: Incrementally improving θ 0 over ψ 0 leads to a new second-best option that Leap uses to generate a new set of more demanding baselines, to further improve the initialization.
Revised: Incrementally improving θ 0 over ψ 0 leads to a new second-best option that Leap uses to generate a new set of more demanding baselines, to further improve the initialization.

Original: Iterating this process, Leap produces a sequence of candidate solutions to eq.
Revised: Iterating this process, Leap produces a sequence of candidate solutions to eq.

Original: 4, all in Θ , with incrementally shorter gradient paths.
Revised: 4, all in Θ , with incrementally shorter gradient paths.

Original: In theorem 1, we show that the limit point of this sequence lies in Θ , and as such is equivalent to performing gradient descent directly on the canonical meta objective (eq. 4).
Revised: In theorem 1, we show that the limit point of this sequence lies in Θ , and as such is equivalent to performing gradient descent directly on the canonical meta objective (eq. 4).

Original: In principle, ¯ F can be evaluated for any θ 0 , but a more efﬁcient strategy is to evaluate θ 0 at ψ 0 .
Revised: In principle, ¯ F can be evaluated for any θ 0 , but a more efﬁcient strategy is to evaluate θ 0 at ψ 0 .

Original: In this case, ¯ d = d , so that ¯ F = F , and gradient descent on ¯ F is equivalent to gradient descent on F .
Revised: In this case, ¯ d = d , so that ¯ F = F , and minimizing ¯ F directly translated into minimizing F .

Original: Theorem 1 (Pull-forward) .
Revised: Theorem 1 (Pull-forward) .

Original: Deﬁne a sequence of initializations { ψ 0 s } s ∈ N by
Revised: Deﬁne a sequence of initializations { ψ 0 s } s ∈ N by

Original: [Equation]
Revised: [Equation]

Original: For β s > 0 sufﬁciently small, there exist learning rates schedules { α iτ } K τ i
Revised: For β s > 0 sufﬁciently small, there exist learning rates schedules { α iτ } K τ i

Original: =1 for all tasks such that ψ 0 k →∞ is a limit point in Θ .
Revised: =1 for all tasks such that ψ 0 k →∞ is a limit point in Θ .

Original: Proof; see appendix A. While the pull-forward objective can be solved with any optimization algorithm, we consider gradient-based methods.
Revised: Proof; see appendix A. While the pull-forward objective can be solved with any optimization algorithm, we consider gradient-based methods.

Original: Because the meta gradient requires differentiating the learning process, we must adopt an approximation.
Revised: Because the meta gradient requires differentiating the learning process, we must adopt an approximation.

Original: In doing so, we obtain a meta-gradient that can be computed analytically on the ﬂy during task training.
Revised: In doing so, we obtain a meta-gradient that can be computed analytically on the ﬂy during task training.

Original: Differentiating ¯ F , we have∇
Revised: Differentiating ¯ F , we have

Original: [Equation]
Revised: [Equation]

Original: where J iτ denotes the Jacobian of θ iτ with respect to the initialization, ∆ f iτ =
Revised: where J iτ denotes the Jacobian of θ iτ with respect to the initialization, ∆ f iτ =

Original: f τ ( ψ i +1 τ [Equation] and ∆ θ iτ = ψ
Revised: f τ ( ψ i +1 τ ) − f τ ( θ iτ ) and ∆ θ iτ = ψ

Original: i +1 τ − θ iτ .
Revised: i +1 τ − θ iτ .

Original: To render the meta gradient tractable, we need to approximate the Jacobians, as these are extremely costly to compute.
Revised: To render the meta gradient tractable, we need to approximate the Jacobians, as these are extremely costly to compute.

Original: Empirical evidence suggest that they are largely redundant (Finn et al., 2017; Nichol et al., 2018).
Revised: Empirical evidence suggest that they are largely redundant (Finn et al., 2017; Nichol et al., 2018).

Original: Nichol et al.
Revised: Nichol et al.

Original: (2018) further shows that the resulting meta-gradient remains faithful to the original meta objective.
Revised: (2018) further shows that the resulting meta-gradient remains faithful to the original meta objective.

Original: We provide some further support for this approximation (see appendix B).
Revised: We provide some further support for this approximation (see appendix B).

Original: First, we note that the learning rate directly controls the quality of the approximation; for any K τ , the identity approximation can be made arbitrarily accurate by choosing a sufﬁciently small learning rate scheme.
Revised: First, we note that the learning rate directly controls the quality of the approximation; for any K τ , the identity approximation can be made arbitrarily accurate by choosing a sufﬁciently small learning rate scheme.

Original: We conduct an ablation study to ascertain how severe this limitation is and ﬁnd that it is relatively loose.
Revised: We conduct an ablation study to ascertain how severe this limitation is and ﬁnd that it is relatively loose.

Original: For the best-performing learning rate, the identity approximation is accurate to four decimal places and shows no signs of signiﬁcant deterioration as the number of training steps increases.
Revised: For the best-performing learning rate, the identity approximation is accurate to four decimal places and shows no signs of signiﬁcant deterioration as the number of training steps increases.

Original: As such, we assume J i
Revised: As such, we assume J i

Original: ≈ I
Revised: ≈ I

Original: n throughout.
Revised: n throughout.

Original: Finally, by evaluating ∇ ¯ F at θ 0 = ψ 0 , the meta gradient contains only terms seen during standard training and can be computed on asynchronously on the ﬂy at negligible cost.
Revised: Finally, by evaluating ∇ ¯ F at θ 0 = ψ 0 , the meta gradient contains only terms seen during standard training and can be computed on asynchronously on the ﬂy at negligible cost.

Original: In practice we use stochastic gradient descent during task training.
Revised: In practice we use stochastic gradient descent during task training.

Original: This injects noise in f as well as in its gradient, resulting in a noisy gradient path.
Revised: This injects noise in f as well as in its gradient, resulting in a noisy gradient path.

Original: Noise in the gradient path does not prevent
Revised: Noise in the gradient path does not prevent

Original: Leap from converging, as long as the gradient path converges.
Revised: Leap from converging, as long as the gradient path converges.

Original: However, noise reduces the rate of convergence, in particular when a noisy gradient step results in f τ ( ψ s +1 τ ) − f τ ( θ iτ ) > 0 .
Revised: However, noise reduces the rate of convergence, in particular when a noisy gradient step results in f τ ( ψ s +1 τ ) − f τ ( θ iτ ) > 0 .

Original: This can either be the result of a poor estimate of the true gradient, or because of too large a step size.
Revised: This can either be the result of a poor estimate of the true gradient, or because of too large a step size.

Original: In either case, we found that de-prioritizing such line segments signiﬁcantly speeds up convergence, as well as allows the use of larger learning rates.
Revised: In either case, we found that de-prioritizing such line segments signiﬁcantly speeds up convergence, as well as allows the use of larger learning rates.

Original: In this paper, we augment ¯ F with a stabilizer of the form µ (cid:0) f τ 
Revised: In this paper, we augment ¯ F with a stabilizer of the form µ (cid:0) f τ ( θ iτ ); f τ

Original: [Equation]
Revised: [Equation]

Original: else
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  The stabilizer points in the opposite direction of the gradient, and as such reduces the weight placed on the gradient of f τ ( θ iτ ) .
Revised: else . The stabilizer points in the opposite direction of the gradient of f ( θ iτ ) , and as such reduces the weight placed on line segment (cid:107) ¯ γ i +1 τ − γ iτ (cid:107) p 2 .

Original: This counteracts both poor estimates and overly large step sizes.
Revised: This counteracts both poor estimates and overly large step sizes.

Original: In appendix C we perform an ablation study and ﬁnd that the stabilizer is not necessary for Leap to converge, but it does speed up convergence signiﬁcantly.
Revised: In appendix C we perform an ablation study and ﬁnd that the stabilizer is not necessary for Leap to converge, but it does speed up convergence signiﬁcantly.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Transfer learning has been explored in a variety of settings, the most typical approach attempting to infuse knowledge in a target model’s parameters by encouraging them to lie close to those of a pretrained source model (Pan & Yang, 2009).
Revised: Transfer learning has been explored in a variety of settings, the most typical approach attempting to infuse knowledge in a target model’s parameters by encouraging them to lie close to those of a pretrained source model (Pan & Yang, 2009).

Original: Because such approaches can limit knowledge transfer (Higgins et al., 2017; Achille et al., 2018), applying standard transfer learning techniques leads to catastrophic forgetting , by which the model is rendered unable to perform a previously mastered task (McCloskey & Cohen, 1989; Goodfellow et al., 2013).
Revised: Because such approaches can limit knowledge transfer (Higgins et al., 2017; Achille et al., 2018), applying standard transfer learning techniques leads to catastrophic forgetting , by which the model is rendered unable to perform a previously mastered task (McCloskey & Cohen, 1989; Goodfellow et al., 2013).

Original: These problems are further accentuated when there is a larger degree of diversity among tasks that push optimal parameterizations further apart.
Revised: These problems are further accentuated when there is a larger degree of diversity among tasks that push optimal parameterizations further apart.

Original: In these cases, transfer learning can in fact be worse than training from scratch.
Revised: In these cases, transfer learning can in fact be worse than training from scratch.

Original: Recent approaches extend standard ﬁnetuning by adding regularizing terms to the training objective that encourage the model to learn parameters that both solve a new task and retain high performance on previous tasks.
Revised: Recent approaches extend standard ﬁnetuning by adding regularizing terms to the training objective that encourage the model to learn parameters that both solve a new task and retain high performance on previous tasks.

Original: These regularizers operate by protecting the parameters that affect the loss function the most (Miconi et al., 2018; Zenke et al., 2017; Kirkpatrick et al., 2017; Lee et al., 2017; Serrà et al., 2018).
Revised: These regularizers operate by protecting the parameters that affect the loss function the most (Miconi et al., 2018; Zenke et al., 2017; Kirkpatrick et al., 2017; Lee et al., 2017; Serrà et al., 2018).

Original: Because these approaches use a single model to encode both global task-general information and local task-speciﬁc information, they can over-regularize, preventing the model from learning further tasks.
Revised: Because these approaches use a single model to encode both global task-general information and local task-speciﬁc information, they can over-regularize, preventing the model from learning further tasks.

Original: More importantly, Schwarz et al.
Revised: More importantly, Schwarz et al.

Original: (2018) found that while these approaches mitigate catastrophic forgetting, they are unable to facilitate knowledge transfer on the benchmark they considered.
Revised: (2018) found that while these approaches mitigate catastrophic forgetting, they are unable to facilitate knowledge transfer on the benchmark they considered.

Original: Ultimately, if a single model must encode both task-generic and task-speciﬁc information, it must either saturate or grow in size (Rusu et al., 2016).
Revised: Ultimately, if a single model must encode both task-generic and task-speciﬁc information, it must either saturate or grow in size (Rusu et al., 2016).

Original: In contrast, meta learning aims to learn the learning process itself (Schmidhuber, 1987; Bengio et al., 1991;
Revised: In contrast, meta learning aims to learn the learning process itself (Schmidhuber, 1987; Bengio et al., 1991;

Original: Santoro et al., 2016; Ravi & Larochelle, 2016; Andrychowicz et al., 2016; Vinyals et al., 2016;
Revised: Santoro et al., 2016; Ravi & Larochelle, 2016; Andrychowicz et al., 2016; Vinyals et al., 2016;

Original: Finn et al., 2017).
Revised: Finn et al., 2017).

Original: Most of the literature focuses on the few-shot learning setting, where a task is some variation on a common theme, such as subsets of classes from a larger pool of data (Lake et al., 2015; Vinyals et al., 2016).
Revised: Most of the literature focuses on the few-shot learning setting, where a task is some variation on a common theme, such as subsets of classes from a larger pool of data (Lake et al., 2015; Vinyals et al., 2016).

Original: The meta learning algorithm adapts a model to a new task given a handful of samples.
Revised: The meta learning algorithm adapts a model to a new task given a handful of samples.

Original: Recent attention has been devoted to three main approaches.
Revised: Recent attention has been devoted to three main approaches.

Original: One trains the meta learner to adapt to a new task by comparing an input to samples from previous tasks (Vinyals et al., 2016;
Revised: One trains the meta learner to adapt to a new task by comparing an input to samples from previous tasks (Vinyals et al., 2016;

Original: Mishra et al., 2018; Snell et al., 2017).
Revised: Mishra et al., 2018; Snell et al., 2017).

Original: More relevant to our framework are approaches that parameterize the training process through a recurrent neural network that takes the gradient as input and produces a new set of parameters (Ravi & Larochelle, 2016;
Revised: More relevant to our framework are approaches that parameterize the training process through a recurrent neural network that takes the gradient as input and produces a new set of parameters (Ravi & Larochelle, 2016;

Original: Santoro et al., 2016; Andrychowicz et al., 2016; Hochreiter et al., 2001).
Revised: Santoro et al., 2016; Andrychowicz et al., 2016; Hochreiter et al., 2001).

Original: The approach most closely related to us learns an initialization such that the model can adapt to a new task through one or a few gradient updates (Finn et al., 2017;
Revised: The approach most closely related to us learns an initialization such that the model can adapt to a new task through one or a few gradient updates (Finn et al., 2017;

Original: Nichol et al., 2018; Al-Shedivat et al., 2017; Lee & Choi, 2018).
Revised: Nichol et al., 2018; Al-Shedivat et al., 2017; Lee & Choi, 2018).

Original: In contrast to our work, these methods focus exclusively on few-shot learning, where the gradient path is trivial as only a single or a handful of training steps are allowed, limiting them to settings where the current task is closely related to previous ones.
Revised: In contrast to our work, these methods focus exclusively on few-shot learning, where the gradient path is trivial as only a single or a handful of training steps are allowed, limiting them to settings where the current task is closely related to previous ones.

Original: It is worth noting that the Model Agnostic Meta Learner (MAML: Finn et al., 2017) can be written as E τ ∼ p ( τ ) (cid:2)
Revised: It is worth noting that the Model Agnostic Meta Learner (MAML: Finn et al., 2017) can be written as E τ ∼ p ( τ ) (cid:2)

Original: f τ ( θ Kτ ) (cid:3) .
Revised: f τ ( θ Kτ ) (cid:3) .

Original: As such it arises as a special case of Leap where only the ﬁnal parameterization is evaluated in terms of its ﬁnal performance.
Revised: As such it arises as a special case of Leap where only the ﬁnal parameterization is evaluated in terms of its ﬁnal performance.

Original: Similarly, the Reptile algorithm (Nichol et al., 2018), which proposes to update rule θ 0 ← θ 0 + (cid:15) (cid:0) E τ ∼ p ( τ ) (cid:2)
Revised: Similarly, the Reptile algorithm (Nichol et al., 2018), which proposes to update rule θ 0 ← θ 0 + (cid:15) (cid:0) E τ ∼ p ( τ ) (cid:2)

Original: θ Kτ (cid:3) − θ 0 (cid:1) , can be seen as a naive version of
Revised: θ Kτ (cid:3) − θ 0 (cid:1) , can be seen as a naive version of

Original: Leap that assumes all task geometries are Euclidean.
Revised: Leap that assumes all task geometries are Euclidean.

Original: In particular, Leap reduces to Reptile if f τ is removed from the task manifold and the energy metric without stabilizer is used.
Revised: In particular, Leap reduces to Reptile if f τ is removed from the task manifold and the energy metric without stabilizer is used.

Original: We ﬁnd this conﬁguration to perform signiﬁcantly worse than any other (section 4.1, appendix C).
Revised: We ﬁnd this conﬁguration to perform signiﬁcantly worse than any other (section 4.1, appendix C).

Original: Related work studying models from a geometric perspective have explored how to interpolate in a generative model’s learned latent space
Revised: Related work studying models from a geometric perspective have explored how to interpolate in a generative model’s learned latent space

Original: Tosi et al.
Revised: Tosi et al.

Original: ; Shao et al.
Revised: ; Shao et al.

Original: ; Arvanitidis et al.
Revised: ; Arvanitidis et al.

Original: Chen et al.
Revised: Chen et al.

Original: ; Kumar et al.
Revised: ; Kumar et al.

Original: Riemann manifolds have also garnered attention in the context of optimization, as a preconditioning matrix can be understood as the instantiation of some
Revised: Riemann manifolds have also garnered attention in the context of optimization, as a preconditioning matrix can be understood as the instantiation of some

Original: Riemann metric (Amari & Nagaoka, 2007; Abbati et al., 2018; Luk & Grosse, 2018).
Revised: Riemann metric (Amari & Nagaoka, 2007; Abbati et al., 2018; Luk & Grosse, 2018).

Original:  E MPIRICAL R ESULTS
Revised:  E MPIRICAL R ESULTS

Original: We consider three experiments designed to provide increasingly complex transfer learning environments.
Revised: We consider three experiments designed to provide increasingly complex transfer learning environments.

Original: We measure transfer learning in terms of ﬁnal test error and area under the error curve , deﬁned as the error score on the training set during training.
Revised: We measure transfer learning in terms of ﬁnal test error and area under the error curve , deﬁned as the error score on the training set during training.

Original: We compare Leap to competing meta
Revised: We compare Leap to competing meta

Original: [Figure]
Revised: [Figure]

Original: learning methods on the Omniglot dataset by tasking the meta learning with transferring knowledge across alphabets (section 4.1).
Revised: learning methods on the Omniglot dataset by tasking the meta learning with transferring knowledge across alphabets (section 4.1).

Original: Further, we study Leap’s ability to transfer knowledge over more complex and diverse tasks in the Multi-CV experiment (section 4.2).
Revised: Further, we study Leap’s ability to transfer knowledge over more complex and diverse tasks in the Multi-CV experiment (section 4.2).

Original: Finally, we evaluate Leap on a more demanding reinforcement learning problem, demonstrating its ability to scale (section 4.3).
Revised: Finally, we evaluate Leap on a more demanding reinforcement learning problem, demonstrating its ability to scale (section 4.3).

Original: 4. O MNIGLOT
Revised: 4. O MNIGLOT

Original: The Omniglot (Lake et al., 2015) dataset consists of 50 alphabets, which we deﬁne to bea distinct task.
Revised: The Omniglot (Lake et al., 2015) dataset consists of 50 alphabets, which we deﬁne to be distinct tasks.

Original: We hold 10 out for validation during meta training, and 10 for ﬁnal evaluation.
Revised: We hold 10 out for validation during meta training, and 10 for ﬁnal evaluation.

Original: We use a subset of the remaining 30 for pretraining.
Revised: We use a subset of the remaining 30 for pretraining.

Original: We compare ﬁnal performance and the rate of convergence on held-out tasks as we vary the number of pretraining tasks from 1 to 30.
Revised: We compare ﬁnal performance and the rate of convergence on held-out tasks as we vary the number of pretraining tasks from 1 to 30.

Original: As baselines, we compare against no pretraining, ﬁnetuning in multi-headed mode, the ﬁrst-order approximation version of
Revised: As baselines, we compare against no pretraining, ﬁnetuning in multi-headed mode, the ﬁrst-order approximation version of

Original: MAML (FOMAML; Finn et al., 2017), and Reptile.
Revised: MAML (FOMAML; Finn et al., 2017), and Reptile.

Original: We ﬁnd that the length metric ( d 2 ) performs marginally better, and as such we report scores for Leap under the length metric.
Revised: We ﬁnd that the length metric ( d 2 ) performs marginally better, and as such we report scores for Leap under the length metric.

Original: We perform an ablation study across conﬁgurations in appendix C. For further details, appendix D.
Revised: We perform an ablation study across conﬁgurations in appendix C. For further details, appendix D.

Original: Leap consistently outperforms both multi-headed ﬁnetuning, FOMAML and Reptile (ﬁg. 3).
Revised: Leap consistently outperforms both multi-headed ﬁnetuning, FOMAML and Reptile (ﬁg. 3).

Original: The gap between Leap and other meta learning frameworks is large and Leap is in fact the only meta learner to outperform ﬁnetuning.
Revised: The gap between Leap and other meta learning frameworks is large and Leap is in fact the only meta learner to outperform ﬁnetuning.

Original: That Leap does better than Reptile, which in turn does better than FOMAML, underscores the importance of leveraging the task geometry in meta learning.
Revised: That Leap does better than Reptile, which in turn does better than FOMAML, underscores the importance of leveraging the task geometry in meta learning.

Original: Multi-task ﬁnetuning is a tough benchmark to beat as tasks are very similar.
Revised: Multi-task ﬁnetuning is a tough benchmark to beat as tasks are very similar.

Original: It is striking that Leap can outperform it even with a single pretraining task, and perhaps more so that the performance margin grows with the number of pretraining tasks.
Revised: It is striking that Leap can outperform it even with a single pretraining task, and perhaps more so that the performance margin grows with the number of pretraining tasks.

Original: 4. M ULTI -CV
Revised: 4. M ULTI -CV

Original: Inspired by Serrà et al.
Revised: Inspired by Serrà et al.

Original: (2018), we consider a set of computer vision datasets as distinct tasks.
Revised: (2018), we consider a set of computer vision datasets as distinct tasks.

Original: We pretrain on all but one task, which is held out for ﬁnal evaluation.
Revised: We pretrain on all but one task, which is held out for ﬁnal evaluation.

Original: For details, see appendix E.
Revised: For details, see appendix E.

Original: To reduce the computational burden during meta training, we pretrain on each task in the meta batch for ten epochs using the energy metric ( d 1 ).
Revised: To reduce the computational burden during meta training, we pretrain on each task in the meta batch for ten epochs using the energy metric ( d 1 ).

Original: We found this to reach equivalent performance to training on longer gradient paths or using the length metric.
Revised: We found this to reach equivalent performance to training on longer gradient paths or using the length metric.

Original: This indicates that it is sufﬁcient for Leap to see a partial trajectory to correctly infer shared structures across task geometries.
Revised: This indicates that it is sufﬁcient for Leap to see a partial trajectory to correctly infer shared structures across task geometries.

Original: We compare Leap against a random initialization, multi-headed ﬁnetuning, a non-sequential version of
Revised: We compare Leap against a random initialization, multi-headed ﬁnetuning, a non-sequential version of

Original: HAT (Serrà et al., 2018) (i.e. allowing revisits) and a non-sequential version of Progressive Nets (Rusu et al., 2016), where we allow lateral connection between every task.
Revised: HAT (Serrà et al., 2018) (i.e. allowing revisits) and a non-sequential version of Progressive Nets (Rusu et al., 2016), where we allow lateral connection between every task.

Original: Note that this makes Progressive
Revised: Note that this makes Progressive

Original: Nets over 8 times larger in terms of learnable parameters.
Revised: Nets over 8 times larger in terms of learnable parameters.

Original: The Multi-CV experiment is more challenging both due to greater task diversity and greater complexity among tasks.
Revised: The Multi-CV experiment is more challenging both due to greater task diversity and greater complexity among tasks.

Original: We report results on held-out tasks in table 1.
Revised: We report results on held-out tasks in table 1.

Original: Leap outperforms all baselines on all but one transfer learning tasks (Facescrub), where Progressive Nets does marginally better than a random initialization owing to its increased parameter count.
Revised: Leap outperforms all baselines on all but one transfer learning tasks (Facescrub), where Progressive Nets does marginally better than a random initialization owing to its increased parameter count.

Original: Notably, while Leap does marginally worse than a random initialization, ﬁnetuning and HAT leads to a substantial drop in performance.
Revised: Notably, while Leap does marginally worse than a random initialization, ﬁnetuning and HAT leads to a substantial drop in performance.

Original: On all other tasks, Leap converges faster to optimal performance and achieves superior ﬁnal performance.
Revised: On all other tasks, Leap converges faster to optimal performance and achieves superior ﬁnal performance.

Original: [Table]
Revised: [Table]

Original: A TARI
Revised: A TARI

Original: To demonstrate that Leap can scale to large problems, both in computational terms but also in the complexity of tasks, we apply it in a reinforcement learning domain, speciﬁcally Atarigames (Bellemare et al., 2013).
Revised: To demonstrate that Leap can scale to large problems, both in computational terms but also in the complexity of tasks, we apply it in a reinforcement learning domain, speciﬁcally Atarigames (Bellemare et al., 2013).

Original: We use a variant of the actor-critic architecture for all experiments (Sutton et al., 1998).
Revised: We use a variant of the actor-critic architecture for all experiments (Sutton et al., 1998).

Original: We learn both the policy π and the value function V from raw pixels, with π and V sharing a convolutional encoder (Schwarz et al., 2018).
Revised: We learn both the policy π and the value function V from raw pixels, with π and V sharing a convolutional encoder (Schwarz et al., 2018).

Original: We apply Leap with respect to this convolutional encoder, using the energy metric ( d 1 ).
Revised: We apply Leap with respect to this convolutional encoder using the energy metric ( d 1 ).

Original: During meta training, we sample mini-batches from 27 games that has an action space dimensionality of at most 10, holding out two games with similar action space dimensionality for evaluation, as well as games with larger action spaces (table 4).
Revised: During meta training, we sample mini-batches from 27 games that has an action space dimensionality of at most 10, holding out two games with similar action space dimensionality for evaluation, as well as games with larger action spaces (table 6).

Original: During meta training, we train on each task in the mini-batch for ﬁve million training steps, accumulating the meta gradient as in algorithm 1.
Revised: During meta training, we train on each task in the mini-batch for ﬁve million training steps, accumulating the meta gradient as in algorithm 1.

Original: See appendix F for details.
Revised: See appendix F for details.

Original: We train for 100 meta training steps, which is sufﬁcient to see a distinct improvement with respect to the initialization; we expect a longer meta-training phase to yield further gains.
Revised: We train for 100 meta training steps, which is sufﬁcient to see a distinct improvement with respect to the initialization; we expect a longer meta-training phase to yield further gains.

Original: We ﬁnd that
Revised: We ﬁnd that

Original: Leap generally outperforms a random initialization, and in all but one case at least performs on par with it.
Revised: Leap generally outperforms a random initialization, and in all but one case at least performs on par with it.

Original: This performance gain is primarily driven by less volatile exploration, as seen by the conﬁdence intervals in ﬁg. 4 (see also ﬁg. 8).
Revised: This performance gain is primarily driven by less volatile exploration, as seen by the conﬁdence intervals in ﬁg. 4 (see also ﬁg. 8).

Original: Leap ﬁnds a useful exploration space faster and more consistently, demonstrating that Leap can ﬁnd shared structures across learning processes on as diverse and complex tasks as training a deep actor-critic model on Atari games.
Revised: Leap ﬁnds a useful exploration space faster and more consistently, demonstrating that Leap can ﬁnd shared structures across learning processes on as diverse and complex tasks as training a deep actor-critic model on Atari games.

Original: However, we also note that this regularity may not cater equally to all tasks, as the result on WizardOfWor (part of the pre-training set) shows.
Revised: However, we also note that this regularity may not cater equally to all tasks, as the result on WizardOfWor (part of the pretraining set) shows.

Original: In this outlying case, Leap exhibits two modes: in one it performs on par 
Revised: In this outlying case, Leap exhibits two modes: in one it performs on par with

Original: AirRaid    4 0 .0 .0 .0 .0 .1 .
Revised: AirRaid    4 0 .0 .0 .0 .0 .1 .

Original: Alien    4 0 .0 .0 .0 .0 .1 .
Revised: Alien    4 0 .0 .0 .0 .0 .1 .

Original: UpNDown    4 0 .0 .0 .0 .0 .1 .
Revised: UpNDown    4 0 .0 .0 .0 .0 .1 .

Original: SpaceInvaders    4 0 .0 .0 .0 .0 .1 .
Revised: SpaceInvaders    4 0 .0 .0 .0 .0 .1 .

Original: RoadRunner    4 0 .0 .0 .0 .0 .1 .
Revised: RoadRunner    4 0 .0 .0 .0 .0 .1 .

Original: WizardOfWor
Revised: WizardOfWor

Original: Leap (orange) generally outperforms a random initialization (blue), even when the action space is twice aslarge asduring pretraining (tablewith a random initialization while exhibiting less variance across runs; it takes considerably longer to ﬁnd a useful exploration space (ﬁg. 8).
Revised: Leap (orange) generally outperforms a random initialization (blue), even when the action space is twice as as pretraining (table a random initialization while exhibiting less variance across runs; it takes considerably longer to ﬁnd a useful exploration space (ﬁg. 8).

Original: This outlier stems from the fact that in deep Reinforcement
Revised: This phenomenon stems from the fact that in deep Reinforcement

Original: Learning, starting from the same initialization can yield very different gradient paths under different random seeds.
Revised: Learning, starting from the same initialization can yield very different gradient paths under different random seeds.

Original: As such, it can be mitigated by increasing Leap’s exposure to the task; that is, by training for a greater number of steps.
Revised: As such, it can be mitigated by increasing Leap’s exposure to the task; that is, by training for a greater number of steps.

Original: That Leap can outperform a random initialization on the pretraining set (AirRaid, UpNDown) is perhaps not surprising.
Revised: That Leap can outperform a random initialization on the pretraining set (AirRaid, UpNDown) is perhaps not surprising.

Original: More striking is that it exhibits the same patters on out-of-distribution tasks.
Revised: More striking is that it exhibits the same patters on out-of-distribution tasks.

Original: In particular, Alien, Gravitar and RoadRunner all have at least 50% larger state space than anything encountered during pretraining, yet Leap outperforms a random initialization.
Revised: In particular, Alien, Gravitar and RoadRunner all have at least 50% larger state space than anything encountered during pretraining (appendix F, table 6), yet Leap outperforms a random initialization.

Original: This suggests that transferring knowledge at a higher level of abstraction, such as in the space of gradient paths, generalizes to unseen task variations provided the underlying learning dynamics are similar.
Revised: This suggests that transferring knowledge at a higher level of abstraction, such as in the space of gradient paths, generalizes to unseen task variations as long as underlying learning dynamics agree.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Transfer learning typically ignores the process of learning a task, restricting it to scenarios where target tasks are very similar to source tasks.
Revised: Transfer learning typically ignores the process of learning a task, restricting it to scenarios where target tasks are very similar to source tasks.

Original: In this paper we present Leap, a framework that facilitates transfer learning at a higher level of abstraction.
Revised: In this paper, we present Leap, a framework that facilitates transfer learning at a higher level of abstraction.

Original: By formalizing transfer learning as minimizing the expected gradient path over task-speciﬁc loss surfaces, we propose a method for meta learning that scales to highly demanding problems.
Revised: By formalizing transfer learning as minimizing the expected gradient path over task-speciﬁc loss surfaces, we propose a method for meta learning that scales to highly demanding problems.

Original: We ﬁnd empirically that Leap has superior generalizing properties to ﬁnetuning and competing methods.
Revised: We ﬁnd empirically that Leap has superior generalizing properties to ﬁnetuning and competing methods.

Original: Moreover, by virtue of transferring knowledge across learning processes, it is robust to non-trivial diversity in task distributions
Revised: Moreover, by virtue of transferring knowledge across learning processes, it is robust to non-trivial diversity in task distributions

Original: APPENDIX A P ROOF OF THEOREM
Revised: A

Revised: A P ROOF OF THEOREM

Original: Proof.
Revised: Proof.

Original: We ﬁrst establish that, for all s ,d ( ψ 0 s +1 )
Revised: We ﬁrst establish that, for all s ,

Original: = F ( ψ 0 s +1 ; Ψ s +1 ) ≤ F ( ψ 0 s ; Ψ s ) =
Revised: [Equation]

Original: d ( ψ 0 s ) , with strict inequality for at least some s .
Revised: with strict inequality for at least some s .

Original: Because { β s } ∞ s =1 satisﬁes the gradient descent criteria, it follows that the sequence { ψ 0 s } ∞ s =1 is convergent, and that, for all s , lim i →∞ ψ is +

Revised: Because { β s } ∞ s =1 satisﬁes the gradient descent criteria, it follows that the sequence { ψ 0 s } ∞ s =1 is convergent. To complete the proof we must show that this limit point is in Θ . To this end, we show that for β s sufﬁciently small, for all s , lim i →∞ ψ is +

Original: =
Revised: =

Original: lim
Revised: lim

Original: i →∞ ψ is .
Revised: i →∞ ψ is .

Original: That is, each  initialization incrementally reduces the expected gradient path length while converging to the same limit point as ψ 00 , and since ψ 00 ∈ Θ by assumption, we obtain ψ 0 s ∈ Θ for all s as an immediate consequence.
Revised: That is, each updated initialization incrementally reduces the expected gradient path length while converging to the same limit point as ψ 00 . Since ψ 00 ∈ Θ by assumption, we obtain ψ 0 s ∈ Θ for all s as an immediate consequence.

Original: To establish  d ( ψ 0 s +1  ) ≤  d ( ψ 0 s  ) , with strict inequality for some s , let
Revised: To establish E τ d ( ψ 0 s +1 , M τ ) ≤ E τ d ( ψ 0 s , M τ ) , with strict inequality for some s , let

Original: [Equation]
Revised: [Equation]

Original: and denote by E τ,i the expectation over gradient paths, E τ ∼ p [Equation] τ ) 
Revised: and denote by E τ,i the expectation over gradient paths, E τ ∼ p ( τ ) (cid:80)

Original: (cid:80) K τ i =1 .
Revised: K τ i =1 .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: with p = 2 deﬁning the meta objective in terms of the gradient path energy and p = 1 in terms of the gradient path length.
Revised: with p = 2 deﬁning the meta objective in terms of the gradient path energy and p = 1 in terms of the gradient path length.

Original: As we are exclusively concerned with the Euclidean norm, we drop the subscript going forward.
Revised: As we are exclusively concerned with the Euclidean norm, we omit the subscript.

Original: By assumption, every β s is sufﬁciently small to satisfy the gradient descent criteria  F ( ψ 0 s ; Ψ s ) ≥  F ( ψ 0 s +1 ; Ψ s ) .
Revised: By assumption, every β s is sufﬁciently small to satisfy the gradient descent criteria ¯ F ( ψ 0 s ; Ψ s ) ≥ ¯ F ( ψ 0 s +1 ; Ψ s ) .

Original: Adding and subtracting d ( ψ 0 s +1 ) = F ( ψ 0 s +1 , Ψ s +1 ) to the RHS, we have
Revised: Adding and subtracting ¯ F ( ψ 0 s +1 , Ψ s +1 ) to the RHS, we have

Original: [Equation]
Revised: [Equation]

Original: Consequently, it follows that  d ( ψ 0 s +1 ) ≤ d ( ψ 0 s  ) if E τ,i (cid:107)
Revised: It follows that E τ d ( ψ 0 s , M τ ) ≥ E τ d ( ψ 0 s +1 , M τ ) if E τ

Original: [Equation]
Revised: [Equation]

Original: p
Revised: i

Original: [Equation]
Revised: [Equation]

Revised: ≥ E

Revised: [Equation]

Original: E
Revised: i

Original: [Equation]
Revised: [Equation]

Original: y iτ − x iτ (cid:107) p .
Revised: y iτ − x iτ (cid:107) p .

Original: As our main concern is existence, we will show something stronger, namely that there exists α iτ such that
Revised: As our main concern is existence, we will show something stronger, namely that there exists α iτ such that

Original: [Equation]
Revised: [Equation]

Original: with at least one such inequality is strict, in which case d  ( ψ 0 s +1  ) <
Revised: with at least one such inequality strict for some i, τ, s , in which case d p ( ψ 0 s +1 , M τ ) <

Original: d  ( ψ 0 s ) .
Revised: d p ( ψ [Equation]

Revised: M τ ) for any p ∈ { 1 , 2 } .

Original: We proceed by establishing the inequality for p = 2 and obtain p = 1 as an immediate consequence of monotonicity of the square root.
Revised: We proceed by establishing the inequality for p = 2 and obtain p = 1 as an immediate consequence of monotonicity of the square root.

Original: Expandingthe left hand side, (cid:107) h iτ − x iτ (cid:107) 2, we have
Revised: Expanding (cid:107) h iτ − x iτ (cid:107) 2 we have

Original: [Equation]
Revised: [Equation]

Original: Every term except (cid:107) z iτ − x iτ (cid:107) 2 can be minimized by choosing α iτ small, whereas (cid:107) z
Revised: Every term except (cid:107) z iτ − x iτ (cid:107) 2 can be minimized by choosing α iτ small, whereas (cid:107) z

Original: iτ − x iτ (cid:107) 2 is controlled by β s .
Revised: iτ − x iτ (cid:107) 2 is controlled by β s .

Original: Thus, our strategy is to make all terms except (cid:107) z iτ − x iτ (cid:107) 2 small, for a given β s , by placing an upper bound on α iτ .
Revised: Thus, our strategy is to make all terms except (cid:107) z iτ − x iτ (cid:107) 2 small, for a given β s , by placing an upper bound on α iτ .

Original: We ﬁrst show that (cid:107) h iτ − z iτ (cid:107) 2 − (cid:107)
Revised: We ﬁrst show that (cid:107) h iτ − z iτ (cid:107) 2 − (cid:107)

Original: y
Revised: y

Original: iτ − x iτ (cid:107) 

Revised: iτ − x iτ (cid:107)

Original: = O (cid:16) α iτ 2 (cid:17) .
Revised: = O (cid:16) α iτ 2 (cid:17) .

Original: Some care is needed as the ( n + 1 )th dimension is the loss value associated with the other n dimensions.
Revised: Some care is needed as the ( n + 1 )th dimension is the loss value associated with the other n dimensions.

Original: To be consistent in our notation, deﬁne ˆ z iτ = ψ s,iτ , so that z iτ =
Revised: To be consistent in our notation, deﬁne ˆ z iτ = ψ [Equation] ,iτ , so that z iτ =

Original: (ˆ z iτ , f τ (ˆ z iτ )) .
Revised: [Equation] z iτ )) .

Original: Similarly deﬁne ˆ x iτ , ˆ h iτ , and ˆ y iτ to obtain
Revised: Similarly deﬁne ˆ x iτ , ˆ h iτ , and ˆ y iτ to obtain

Original: [Equation]
Revised: [Equation]

Revised: iτ

Original:  Consider (cid:107) ˆ h iτ − ˆ z iτ (cid:107) 2 − (cid:107) ˆ y iτ − ˆ x iτ (cid:107) 2 .
Revised: [Equation] ˆ [Equation] Consider (cid:107) ˆ h iτ − ˆ z iτ (cid:107) 2 − (cid:107) ˆ y iτ − ˆ x iτ (cid:107) 2 .

Original: Note that ˆ h iτ =
Revised: Note that ˆ h iτ =

Original: ˆ z
Revised: ˆ z

Original: iτ − α iτ g (ˆ z iτ ) , where g (ˆ z iτ )
Revised: iτ − α iτ g (ˆ z iτ ) , where g (ˆ z iτ )

Original: =
Revised: =

Original: S s,iτ ∇ f
Revised: S s,iτ ∇ f

Original: (ˆ z iτ ) , and similarly ˆ y iτ =
Revised: (ˆ z iτ ) , and similarly ˆ y iτ =

Original: ˆ x iτ − α iτ g
Revised: ˆ x iτ − α iτ g

Original: (ˆ x iτ ) with g (ˆ x iτ )
Revised: (ˆ x iτ ) with g (ˆ x iτ )

Original: =
Revised: =

Original: S s +1 ,i τ ∇ f
Revised: S s +1 ,i τ ∇ f

Original: (ˆ x iτ ) .
Revised: (ˆ [Equation] iτ ) .

Original: Thus, (cid:107) ˆ h iτ − ˆ z iτ (cid:107) 2 = α iτ 2 (cid:107) g
Revised: Thus, (cid:107) ˆ h iτ − ˆ z iτ (cid:107) 2 = α iτ 2 (cid:107) g

Original: (ˆ z iτ ) (cid:107) 2 and similarly for (cid:107) ˆ y iτ − ˆ x iτ (cid:107) 2 , so (cid:107)[-ˆ h iτ − ˆ z iτ (cid:107) 2 − (cid:107) ˆ y iτ − ˆ x iτ (cid:107) 
 = α iτ 2 (cid:16) (cid:107) g (ˆ z iτ ) (cid:107)-]
Revised: (ˆ z iτ ) (cid:107) 2 and similarly for (cid:107) ˆ y iτ − ˆ x iτ (cid:107) 2 , so (cid:107)[-ˆ h iτ − ˆ z iτ (cid:107) 2 − (cid:107) ˆ y iτ − ˆ x iτ (cid:107) 
 = α iτ 2 (cid:16) (cid:107) g (ˆ z iτ ) (cid:107)-]

Original: [Equation]
Revised: [Equation]

Original: Now consider ( f τ (ˆ h iτ ) − f τ (ˆ z iτ ))
Revised: Now consider ( f τ ( ˆ h iτ ) − f τ (ˆ z iτ )

Original:  2 − ( f τ (ˆ yiτ )
Revised: [Equation] 2 − ( f τ (ˆ y

Revised: [Equation]

Original:  − f τ (ˆ x iτ )) 2 .
Revised: ) − f τ (ˆ x iτ )) 2 .

Original: Using the above identities and ﬁrst-order
Revised: Using the above identities and ﬁrst-order

Original: Taylor series expansion, we have
Revised: Taylor series expansion, we have

Original: [Equation]
Revised: [Equation]

Original: and similarly for ( f τ (ˆ y iτ ) − f τ (ˆ x iτ )) 2 .
Revised: and similarly for ( f τ (ˆ y iτ ) − f τ (ˆ x iτ )) 2 .

Original: As such, (cid:107) h
Revised: As such, (cid:107) h

Original: [Equation]
Revised: [Equation]

Original: 2 −
Revised: z

Original: [Equation]
Revised: [Equation]

Original: Returning to (cid:107) h iτ − x iτ (cid:107) [Equation] ≥
Revised: Returning to (cid:107) h iτ − x iτ (cid:107) 2 − (cid:107) y

Original: [Equation] y iτ − x iτ [Equation] 2 , substituting in the above results, we have
Revised: iτ − x iτ (cid:107) 2 , substituting in the above results, we have

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is non-negative, and importantly, always non-zero whenever β s (cid:54) = 0 .
Revised: The ﬁrst term is non-negative, and importantly, always non-zero whenever β s (cid:54) = 0 .

Original: Furthermore, α iτ can always be made sufﬁciently small for (cid:107) z iτ − x iτ (cid:107) 2 to dominate the residual, so we can focus on the inner product (cid:104) g ( z iτ ) , z iτ − x iτ (cid:105) .
Revised: Furthermore, α iτ can always be made sufﬁciently small for (cid:107) z iτ − x iτ (cid:107) 2 to dominate the residual, so we can focus on the inner product (cid:104) g ( z iτ ) , z iτ − x iτ (cid:105) .

Original: If it is negative, all terms are positive and we have(cid:107) h iτ
Revised: If it is negative, all terms are positive and we have

Original: − x iτ (cid:107) 2 − (cid:107) y 
Revised: [Equation] y [Equation]

Original: iτ − x iτ (cid:107) 2> 0 as desired.
Revised: 2 as desired.

Original: If not, (cid:107) z iτ − x iτ (cid:107) 2 dominates if
Revised: If not, (cid:107) z iτ − x iτ (cid:107) 2 dominates if

Original: [Equation]
Revised: [Equation]

Original: Thus, for α iτ sufﬁciently small, we have (cid:107) h
Revised: Thus, for α iτ sufﬁciently small, we have (cid:107) h

Original: iτ − x iτ (cid:107) 2 ≥ (cid:107) y
Revised: iτ − x iτ (cid:107) 2 ≥ [Equation]

Original:  iτ − x iτ (cid:107) 2  , with strict inequality whenever (cid:104) g
Revised: y iτ − x iτ (cid:107) 2 ∀ i, τ, s , with strict inequality whenever (cid:104) g

Original: ( z iτ ) , z iτ − x iτ (cid:105) < 0 or the bound on α iτ holds strictly.
Revised: ( z iτ ) , z iτ − x iτ (cid:105) < 0 or the bound on α iτ holds strictly.

Original: This establishes d 2 ( ψ 0 s +1  ) ≤ d 2 ( ψ 0 s  )  , with strict inequality forsome at least some  s .
Revised: This establishes d 2 ( ψ 0 s +1 , M τ ) ≤ d 2 ( ψ 0 s , M τ ) for all τ, s , with strict inequality for at least some τ, s .

Original: To also establish it for the gradient path length, taking square roots on both sides of (cid:107) h iτ
Revised: To also establish it for the gradient path length ( p = 1 ), taking square roots on both sides of (cid:107) h iτ

Original: − x iτ (cid:107) 2 ≥ (cid:107) y
Revised: − x iτ (cid:107) 2 ≥ (cid:107) y

Original: iτ − x iτ (cid:107) 2 yields the desired results, and so E τ,i (cid:107) h iτ − x iτ (cid:107) p ≥ E τ,i (cid:107) y iτ − x iτ (cid:107) p for p ∈ { 1 , 2 } , establishing d
Revised: iτ − x iτ (cid:107) 2 yields the desired results, and so [Equation] p for p ∈ [Equation]

Revised: [Equation]

Revised: , establishing

Original: [Equation]
Revised: [Equation]

Original: with strict inequality  whenever β s (cid:54) = 

Revised: with strict inequality for at least some τ, s , in particular whenever β s (cid:54) =

Original: and α iτis sufﬁciently small.
Revised: and α iτ sufﬁciently small.

Original: Then, to see that the limit point of Ψ s +1 is the same as that of Ψ s for β s sufﬁciently small, note that x iτ =
Revised: Then, to see that the limit point of Ψ s +1 is the same as that of Ψ s for β s sufﬁciently small, note that x iτ =

Original: y
Revised: y

Original: i − 1 τ .
Revised: i − 1 τ .

Original: As before, by the gradient descent criteria, β s is such that
Revised: As before, by the gradient descent criteria, β s is such that

Original: [Equation]
Revised: [Equation]

Original: Deﬁne (cid:15) iτ as the noise residual from the expectation; each y i − 1 τ is boundedby (cid:107)
Revised: Deﬁne (cid:15) iτ as the noise residual from the expectation; each y i − 1 τ is bounded

Original: [Equation]
Revised: [Equation]

Original: ≤ α iτp (cid:107) g
Revised: ≤ α iτp (cid:107) g

Original: (ˆ z iτ ) (cid:107) p + (cid:15) iτ .
Revised: (ˆ z iτ ) (cid:107) p + (cid:15) iτ .

Original: For β s small this noise component vanishes, and since { α iτ } i is a converging sequence, the bound on y i − 1 τ grows increasingly tight.
Revised: For β s small this noise component vanishes, and since { α iτ } i is a converging sequence, the bound on y i − 1 τ grows increasingly tight.

Original: It follows then that { ψ is +1 } ∞ i =1 converges to the same limit point as { ψ is } ∞ i =1 , yielding ψ 0 s +1 ∈ Θ for all s , as desired.
Revised: It follows then that { ψ is +1 } ∞ i =1 converges to the same limit point as { ψ is } ∞ i =1 , yielding ψ 0 s +1 ∈ Θ for all s , as desired.

Original: (cid:4)
Revised: [Equation]

Original: B A BLATION S TUDY : A PPROXIMATING J ACOBIANS J i ( θ 0 )
Revised: B A BLATION S TUDY : A PPROXIMATING J ACOBIANS J i ( θ 0 )

Original: To understand the role of the Jacobians, note that (we drop task subscripts for simplicity)
Revised: To understand the role of the Jacobians, note that (we drop task subscripts for simplicity)

Original: [Equation]
Revised: [Equation]

Original: where H f ( θ j ) denotes the Hessian of f at θ j .
Revised: where H f ( θ j ) denotes the Hessian of f at θ j .

Original: Thus, changes to θ i +1 are translated into θ 0 via all intermediary Hessians.
Revised: Thus, changes to θ i +1 are translated into θ 0 via all intermediary Hessians.

Original: This makes the Jacobians memoryless up to second-order curvature.
Revised: This makes the Jacobians memoryless up to second-order curvature.

Original: Importantly, the effect of curvature can directly be controlled via α i , and by choosing α i small we can ensure J i ( θ 0 )
Revised: Importantly, the effect of curvature can directly be controlled via α i , and by choosing α i small we can ensure J i ( θ 0 )

Original: ≈ I n to be a reasonable assumption.
Revised: ≈ I n to be a reasonable assumption.

Original: In practice, this approximation works well (c.f.
Revised: In practice, this approximation works well (c.f.

Original: Finn et al., 2017; Nichol et al., 2018).
Revised: Finn et al., 2017; Nichol et al., 2018).

Original: Moreover, as a practical matter, if the alternative is some other approximation to the Hessians, the amount of noise injected grows exponentially with every iteration, making it an extremely challenging problem beyond the scope of this paper. 
Revised: Moreover, as a practical matter, if the alternative is some other approximation to the Hessians, the amount of noise injected grows exponentially with every iteration, making it an extremely challenging problem beyond the scope of this paper. [Figure]

Original: To understand how this approximation limits our choice of learning rates α i we conduct an ablation study in the Omniglot experiment setting.
Revised: To understand how this approximation limits our choice of learning rates α i we conduct an ablation study in the Omniglot experiment setting.

Original: We are interested in the relative precision of the identity approximation under different learning rates and across time steps, which we deﬁne as
Revised: We are interested in the relative precision of the identity approximation under different learning rates and across time steps, which we deﬁne as

Original: [Equation]
Revised: [Equation]

Original: We use the same four-layer convolutional neural network as in the Omniglot experiment (appendix D).
Revised: We use the same four-layer convolutional neural network as in the Omniglot experiment (appendix D).

Original: For each choice of learning rate, we train a model from a random initialization for 20 steps and compute ρ every 5 steps.
Revised: For each choice of learning rate, we train a model from a random initialization for 20 steps and compute ρ every 5 steps.

Original: Due to exponential growth of memory consumption, we were unable to compute ρ for more than 20 gradient steps.
Revised: Due to exponential growth of memory consumption, we were unable to compute ρ for more than 20 gradient steps.

Original: We report the relative precision of the ﬁrst convolutional  layer, as computing the Jacobian with respect to the other layers, all being considerably larger, was too costly.
Revised: We report the relative precision of the ﬁrst convolutional [Figure] A v e r ag e tr a i n i n g l o ss p =2, µ =0, f τ = p =2, µ =1, f τ = p =2, µ =0, f τ = p =1, µ =0, f τ = p =1, µ =1, f τ = p =1, µ =0, f τ = layer, as computing the Jacobian with respect to the other layers, all being considerably larger, was too costly.

Original: We computed ρ for all layers on the ﬁrst ﬁve gradient steps and found no signiﬁcant variation in precision across layers.
Revised: We computed ρ for all layers on the ﬁrst ﬁve gradient steps and found no signiﬁcant variation in precision across layers.

Original: Consequently, we prioritize reporting how precision varies with the number of gradient steps.
Revised: Consequently, we prioritize reporting how precision varies with the number of gradient steps.

Original: As in the main experiments, we use stochastic gradient descent.
Revised: As in the main experiments, we use stochastic gradient descent.

Original: We evaluate α i = α ∈ { 0 . 01 , 0 . 1 , 0 . 5 } across 5 different tasks.
Revised: We evaluate α i = α ∈ { 0 . 01 , 0 . 1 , 0 . 5 } across 5 different tasks.

Original: Figure 5 summarizes our results.
Revised: Figure 5 summarizes our results.

Original: Reassuringly, we ﬁnd the identity approximation to be accurate to at least the fourth decimal for learning rates we use in practice, and to the third decimal for the largest learning rate ( 0 . 5 ) we were able to converge with.
Revised: Reassuringly, we ﬁnd the identity approximation to be accurate to at least the fourth decimal for learning rates we use in practice, and to the third decimal for the largest learning rate ( 0 . 5 ) we were able to converge with.

Original: Importantly, except for the smallest learning rate, the quality of the approximation is constant in the number of gradient steps.
Revised: Importantly, except for the smallest learning rate, the quality of the approximation is constant in the number of gradient steps.

Original: The smallest learning exhibit some deterioration on the fourth decimal, however larger learning rates provide an upper bound that is constant on the third decimal, indicating that this is of minor concern.
Revised: The smallest learning exhibit some deterioration on the ﬁfth decimal, however larger learning rates provide an upper bound that is constant on the fourth decimal, indicating that this is of minor concern.

Original: Finally, we note that while these results suggest the identity approximation to be a reasonable approach on the class of problems we consider, other settings may put stricter limits on the effective size of learning rates.
Revised: Finally, we note that while these results suggest the identity approximation to be a reasonable approach on the class of problems we consider, other settings may put stricter limits on the effective size of learning rates.

Original: C A BLATION S TUDY : L EAP H YPER -P ARAMETERS
Revised: C A BLATION S TUDY : L EAP H YPER -P ARAMETERS

Original: As Leap is a general framework, we have several degrees of freedom to specify aparticular meta learner.
Revised: As Leap is a general framework, we have several degrees of freedom in specifying a meta learner.

Original: In particular,  thechoice of task manifold, gradient path distance  (i.e. d p ), and whether to incorporate stabilizers.
Revised: In particular, we hare free to choose the task manifold structure, the gradient path distance metric (i.e. d p ) and whether to incorporate stabilizers.

Original: These are non-trivial choices and to ascertain the importance of each, we conduct an ablation study.
Revised: These are non-trivial choices and to ascertain the importance of each, we conduct an ablation study.

Original: We vary  the task manifold, between  the full loss surface and only parameter space,  the use of path  energy or lengthas distance measure (i.e. d p ), and the importance of the [Figure] [Figure]
Revised: We vary (a) the task manifold between using the full loss surface and only parameter space, (b) the gradient path distance metric between using the energy or length (i.e. d p ), and (c) inclusion of the stabilizer µ in the meta objective.

Original: A v e r ag e tr a i n i n g l o ss p =2, µ =0, f τ = p =2, µ =1, f τ = p =2, µ =0, f τ = p =1, µ =0, f τ = p =1, µ =1, f τ = p =1, µ =0, f τ = stabilizer µ .

Original: We stay as close as possible to the set-up used in the Omniglot experiment (section 4.1), but ﬁx the number of pre-training tasks to 30 and use fewer training steps in the inner and outer loop during meta-training; we perform 500 meta-gradient updates, which is sufﬁcient to converge, and use a maximum of 100 gradient steps on each task during meta-training (i.e. K τ ≤ 100 ; c.f.ﬁg. 3).
Revised: We stay as close as possible to the set-up used in the Omniglot experiment (appendix D), but ﬁx the number of pretraining tasks to 30 and use fewer training steps in the inner and outer loop during meta-training; we performmeta gradient updates, which is sufﬁcient to converge, and use a maximum of 100 gradient steps on each task during meta-training (i.e. K τ ≤ 100 ; c.f. ﬁg. 3).

Original: All other hyper-parameters are the same.
Revised: All other hyper-parameters are the same.

Original: Our ablation study indicates that the richer the task manifold and the more accurate the gradient path length is approximated, the better Leap performs (ﬁg. 6).
Revised: Our ablation study indicates that the richer the task manifold and the more accurate the gradient path length is approximated, the better Leap performs (ﬁg. 6).

Original: Further, adding a stabilizer has the intended effect and leads to signiﬁcantly faster convergence.
Revised: Further, adding a stabilizer has the intended effect and leads to signiﬁcantly faster convergence.

Original: The simplest conﬁguration, deﬁned in terms of the gradient path energy and with the task manifold equalequal to parameter space, yields a meta-gradient equivalent to the update rule used in Reptile.
Revised: The simplest conﬁguration, deﬁned in terms of the gradient path energy and with the task manifold equal to parameter space, yields a meta gradient equivalent to the update rule used in Reptile.

Original: We ﬁnd this conﬁguration to be least efﬁcient in terms of convergence and observe a signiﬁcant deterioration in performance.
Revised: We ﬁnd this conﬁguration to be least efﬁcient in terms of convergence and observe a signiﬁcant deterioration in performance.

Original: Extending the task manifold to the loss surface does not improve meta-training convergence speed, but does cut prediction error in half.
Revised: Extending the task manifold to the loss surface does not improve meta-training convergence speed, but does cut prediction error in half.

Original: Adding the stabilizer signiﬁcantly speeds up convergence, as intended.
Revised: Adding the stabilizer signiﬁcantly speeds up convergence, as intended.

Original: These conclusions also hold under the gradient path length as distance measure, and in general using the gradient path length does better than using the gradient path energy as the distance measure.
Revised: These conclusions also hold under the gradient path length as distance measure, and in general using the gradient path length does better than using the gradient path energy as the distance measure.

Original: D E XPERIMENT DETAILS : O MNIGLOT
Revised: D E XPERIMENT DETAILS : O MNIGLOT

Original: Omniglot contains 50 alphabets, each consisting of a set of characters that in turn have 20 unique samples.
Revised: [Table] [Table] [Table] learning rate 0.

Revised: [Table]

Revised: [Table]

Revised: steps    —

Revised: Meta batch size    —

Revised: Task batch size    —

Revised: Omniglot contains 50 alphabets, each with a set of characters that in turn have 20 unique samples.

Original: We  pretrain on up to 30 tasks, holding  10tasks out for validation and 10tasks for ﬁnal evaluation.
Revised: We treat alphabet as a distinct task and pretrain on up to 30 alphabets, holding up to 10 out for validation and 10 for ﬁnal evaluation.

Original: We use the same convolutional neural network architecture as in previous works (Vinyals et al., 2016;
Revised: We use the same convolutional neural network architecture as in previous works (Vinyals et al., 2016;

Original: Finn et al., 2017; Schwarz et al., 2018) and train using stochastic gradient descent.
Revised: Finn et al., 2017; Schwarz et al., 2018). This model stacks a module, comprised of a 3 × 3 convolution with 64 ﬁlters, followed by batch-normalization, ReLU activation and 2 × 2 max-pooling, four times. All images are downsampled to 28 × 28 , resulting in a 1 × 1 × 64 feature map that is passed on to a ﬁnal linear layer.

Original: For each alphabet, we deﬁne a task as a 20-class classiﬁcation problem.
Revised: We deﬁne a task as a 20-class classiﬁcation problem with classes drawn from a distinct alphabet.

Original: For alphabets with more thancharacters, we pick 20 characters at random, while alphabets with fewer characters (4) are dropped from the validationset. The validation set is used to for early stopping ofthe meta training process.
Revised: For alphabets with more than 20 characters, we pick 20 characters at random, while alphabets with fewer characters (4) are dropped from the validation set (used for early stopping of meta training).

Original: On each task, we train the model using stochastic gradient descent with a batch size of 20, and a learning rate of 0.1, except for ﬁne-tuning where we ﬁnd 0.01 to perform better than 0.1 or 0.001.
Revised: On each task, we train a model using stochastic gradient descent. For each model, we evaluated learning rates in the range [0 . 001 , 0 . 01 , 0 . 1 , 0 . 5] ; see table 3 for hyper-parameters.

Original: The initialization is trained using stochastic gradient descent with a learning rate of 0.1, except for in the case of Reptile where a lower learning rate of 0.01 performed best. We use a task batch size of 20, sampled with replacement. We train on a given task until we observe no more improvement on a set of held-out images.
Revised: To avoid unnecessary computation while evaluating each model across number of pretraining tasks and seeds we use early stopping in both task training loop and meta training loop. For task training, we stop when we reach close to perfect validation performance: each task has 20 characters, each of which has 20 samples.

Original: Following Schwarz et al. (2018), we augment the dataset with 36 random rotations.
Revised: For each character, we hold 5 of samples out to create a task validation

Original: [Table]
Revised: [Table]

Revised: set.

Original: [Table]
Revised: We stop training when validation error is below 2% on 30 batches in a row. We ran preliminary experiments to ensure this stopping criteria had no effect on ﬁnal test performance. For the meta training loop, the validation set consists of held-out tasks. Meta training is stopped when the average validation error on these validation tasks has not improved for 10 consecutive meta gradient steps.

Revised: To increase task complexity, we augment the all datasets with 36 uniformly spaced rotations.

Original: [Table]
Revised: [Table]

Original: E E XPERIMENT DETAILS : M ULTI -CV
Revised: E E XPERIMENT DETAILS : M ULTI -CV

Original: In this experiment, we allow different architectures between tasks by usinga distinct ﬁnal linear layers for each task.
Revised: We allow different architectures between tasks by using distinct ﬁnal linear layers for each task.

Original: We use the same neural network architecture as in the Omniglot experiment and compare Leap against (a) a baseline with no pretraining, (b) multitask ﬁnetuning, (c) HAT (Serrà et al., 2018), and (d) Progressive Nets (Rusu et al., 2016).
Revised: We use the same convolutional encoder as in the Omniglot experiment (appendix D). Leap learns an initialization for the convolutional encoder; on each task, the ﬁnal linear layer is always randomly initialized. We compare Leap against (a) a baseline with no pretraining, (b) multitask ﬁnetuning, (c) HAT (Serrà et al., 2018), and (d) Progressive Nets (Rusu et al., 2016).

Original: For HAT, we use the original formulation, but allow multiple  revisits (until convergenve).
Revised: For HAT, we use the original formulation, but allow multiple task revisits (until convergence).

Original: For Progressive Nets, we allow lateral connections between all tasks andallow multiple  revisits (until convergence).
Revised: For Progressive Nets, we allow lateral connections between all tasks and multiple task revisits (until convergence).

Original: Note that  Progressive Nets is overtimes larger in terms of learnable parameters than the other models.
Revised: Note that this makes Progressive Nets is over 8 times larger in terms of learnable parameters than the other models.

Original: On a given task, we trainthe model using stochastic gradient descent with a learning rate of 0.1 and a batch size of 32.
Revised: We train using stochastic gradient descent with cosine annealing (Loshchilov & Hutter, 2017).

Original: Training is stopped once the validation accuracy does not improve or once a preset number of epochs is reached. For pretraining, we cap the number of training epochs to 10, and  for  the held-out task we cap the number of epochs to 100.
Revised: During meta training, we sample a batch of 10 tasks at random from the pretraining set and train for until the early stopping criterion is trigger or the maximum amount of epochs is reached (see table 5).

Original: We train Leap using Adam (Kingma & Ba, 2015) with a learning rate of 0.01 and a task batch size of 10. Once meta training is complete, we train on a held-out dataset until the validation error stops improving, or the training process reaches 100 epochs. We use no dataset augmentation; MNIST images are zero padded to have 32 × 32 images; we use the same normalizations as in Serrà et al.
Revised: We used the same interval for selecting learning rates as in the Omniglot experiment (appendix D).

Original: [Table] HAT ‡ 9. No pre-training 8. Cifar
Revised: Only Leap beneﬁted from using more than 1 epoch as the upper limit on task training steps during pretraining.

Original: Leap 21.

Original: Fine-tuning 27. Progressive Nets ‡ 24. HAT ‡ 27. No pre-training 26.
Revised: In the case of Leap, the initialization is updated after all tasks in the meta batch has been trained to convergence; for other models, there is no distinction between initialization and task parameters.

Original: SVHN Leap 8. Fine-tuning 10. Progressive Nets ‡ 10.
Revised: On a given task, training is stopped if the maximum number of epochs is reached (table 5) or if the validation error fails to improve over 10 consecutive gradient steps.

Original: HAT ‡ 10. No pre-training 10. Cifar Leap 52.
Revised: Similarly, meta training is stopped once the mean validation error fails to improve over 10 consecutive meta training batches.

Original: Fine-tuning 59. Progressive Nets ‡ 55. HAT ‡ 62. No pre-training 54.
Revised: We use Adam (Kingma & Ba, 2015) for the meta gradient update with a constant learning rate of 0.01.

Original: Trafﬁc Signs

Original: Leap 2.

Original: Fine-tuning 5. Progressive Nets ‡ 3. HAT ‡ 5. No pre-training 3.
Revised: We use no dataset augmentation; MNIST images are zero padded to have 32 × 32 images; we use the same normalizations as Serrà et al.

Original: F E XPERIMENT DETAILS : A TARI
Revised: F E XPERIMENT DETAILS : A TARI

Original: We use the same network as in Mnih et al.
Revised: We use the same network as in Mnih et al.

Original: (2013), adopting it to actor-critic algorithms by estimating both value function and policy through linear layers connected to the ﬁnal output of a shared convolutional network.
Revised: (2013), adopting it to actor-critic algorithms by estimating both value function and policy through linear layers connected to the ﬁnal output of a shared convolutional network.

Original: As is standard, we use downsampled 84 × 84 × 3 RGB images as input.
Revised: Following standard practice, we use downsampled 84 × 84 × 3 RGB images as input.

Revised: Leap is applied with respect to the convolutional encoder (as ﬁnal linear layers vary in size across environments).

Original: On each task, we use  a batchsize of 32, an unroll length of 5 and update the model parameters with 
Revised: We use all environments with an action space of at most 10 as our pretraining pool, holding out Breakout and SpaceInvaders. During meta training, we sample a batch of 16 games at random from a pretraining pool of 27 games. On each game in the batch, a network is initialized using the shared initialization and trained independently for 5 million steps, accumulating the meta gradient across games on the ﬂy. Consequently, the only the baseline and Leap differs only with respect to the initialization of the convolutional encoder.

Original:  RMSProp (using (cid:15) = 10 − 4 , α = 0 . 99 ) with a learning rate of 10 − 4 .
Revised: We trained Leap for 100 steps, equivalent to training 1600 agents for 5 million steps. The meta learned initialization was evaluated on the held-out games, a random selection of games seen during pretraining, and a random selection of games with action spaces larger than 10 (table 6). On each task, we use a batch size of 32, an unroll length ofand update the model parameters with RMSProp (using (cid:15) = 10 − 4 , α = 0 . 99 ) with a learning rate of 10 − 4 .

Original: We set the entropy cost to 0.
Revised: We set the entropy cost to 0 .

Original:  and clip the absolute value of the rewards to maximum 5.0.
Revised: 01 and clip the absolute value of the rewards to maximum 5.0.

Original: The discounting factor was set to 0.99. 
Revised: We use a discounting factor of 0.99. [Figure]

Original: [Table]
Revised: [Table]

Original: AirRaid4 0 .0 .0 .0 .0 .1 .
Revised: AirRaid

Original: Alien    4 0 .0 .0 .0 .0 .1 . UpNDown    4 0 .0 .0 .0 .0 .1 . SpaceInvaders    4 0 .0 .0 .0 .0 .1 . RoadRunner    4 0 .0 .0 .0 .0 .1 .
Revised: [Table]

Original: WizardOfWor    4 0 .0 .0 .0 .0 .1 . Asteroids    4 0 .0 .0 .0 .0 .1 . Gravitar    4 0 .0 .0 .0 .0 .1 .
Revised: N

Original: Breakout [Figure]
Revised: [Figure] [Figure]