Original: G LOBAL C ONVERGENCE OF T HREE - LAYER N EURAL N ETWORKS IN THE M EAN F IELD R EGIME
Revised: G LOBAL C ONVERGENCE OF T HREE - LAYER N EURAL N ETWORKS IN THE M EAN F IELD R EGIME

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In the mean ﬁeld regime, neural networks are appropriately scaled so that as the width tends to inﬁnity, the learning dynamics tends to a nonlinear and nontrivial dynamical limit, known as the mean ﬁeld limit.
Revised: In the mean ﬁeld regime, neural networks are appropriately scaled so that as the width tends to inﬁnity, the learning dynamics tends to a nonlinear and nontrivial dynamical limit, known as the mean ﬁeld limit.

Original: This lends a way to study large-width neural networks via analyzing the mean ﬁeld limit.
Revised: This lends a way to study large-width neural networks via analyzing the mean ﬁeld limit.

Original: Recent works have successfully applied such analysis to two-layer networks and provided global convergence guarantees.
Revised: Recent works have successfully applied such analysis to two-layer networks and provided global convergence guarantees.

Original: The extension to multilayer ones however has been a highly challenging puzzle, and little is known about the optimization efﬁciency in the mean ﬁeld regime when there are more than two layers.
Revised: The extension to multilayer ones however has been a highly challenging puzzle, and little is known about the optimization efﬁciency in the mean ﬁeld regime when there are more than two layers.

Original: In this work, we prove a global convergence result for unregularized feedforward three-layer networks in the mean ﬁeld regime.
Revised: In this work, we prove a global convergence result for unregularized feedforward three-layer networks in the mean ﬁeld regime.

Original: We ﬁrst develop a rigorous framework to establish the mean ﬁeld limit of three-layer networks under stochastic gradient descent training.
Revised: We ﬁrst develop a rigorous framework to establish the mean ﬁeld limit of three-layer networks under stochastic gradient descent training.

Original: To that end, we propose the idea of a neuronal embedding , which comprises of a ﬁxed probability space that encapsulates neural networks of arbitrary sizes.
Revised: To that end, we propose the idea of a neuronal embedding , which comprises of a ﬁxed probability space that encapsulates neural networks of arbitrary sizes.

Original: The identiﬁed mean ﬁeld limit is then used to prove a global convergence guarantee under suitable regularity and convergence mode assumptions, which – unlike previous works on two-layer networks – does not rely critically on convexity.
Revised: The identiﬁed mean ﬁeld limit is then used to prove a global convergence guarantee under suitable regularity and convergence mode assumptions, which – unlike previous works on two-layer networks – does not rely critically on convexity.

Original: Underlying the result is a universal approximation property, natural of neural networks, which importantly is shown to hold at any ﬁnite training time (not necessarily at convergence) via an algebraic topology argument.
Revised: Underlying the result is a universal approximation property, natural of neural networks, which importantly is shown to hold at any ﬁnite training time (not necessarily at convergence) via an algebraic topology argument.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Interests in the theoretical understanding of the training of neural networks have led to the recent discovery of a new operating regime: the neural network and its learning rates are scaled appropriately, such that as the width tends to inﬁnity, the network admits a limiting learning dynamics in which all parameters evolve nonlinearly with time 1 .
Revised: Interests in the theoretical understanding of the training of neural networks have led to the recent discovery of a new operating regime: the neural network and its learning rates are scaled appropriately, such that as the width tends to inﬁnity, the network admits a limiting learning dynamics in which all parameters evolve nonlinearly with time 1 .

Original: This is known as the mean ﬁeld (MF) limit (Mei et al.
Revised: This is known as the mean ﬁeld (MF) limit (Mei et al.

Original: ; Chizat & Bach (2018); Rotskoff & Vanden-Eijnden (2018); Sirignano & Spiliopoulos (2018); Nguyen (2019); Araújo et al.
Revised: ; Chizat & Bach (2018); Rotskoff & Vanden-Eijnden (2018); Sirignano & Spiliopoulos (2018); Nguyen (2019); Araújo et al.

Original: ; Sirignano & Spiliopoulos (2019)).
Revised: ; Sirignano & Spiliopoulos (2019)).

Original: The four works Mei et al.
Revised: The four works Mei et al.

Original: ; Chizat & Bach (2018); Rotskoff & Vanden-Eijnden (2018); Sirignano & Spiliopoulos (2018) led the ﬁrst wave of efforts in 2018 and analyzed two-layer neural networks.
Revised: ; Chizat & Bach (2018); Rotskoff & Vanden-Eijnden (2018); Sirignano & Spiliopoulos (2018) led the ﬁrst wave of efforts in 2018 and analyzed two-layer neural networks.

Original: They established a connection between the network under training and its MF limit.
Revised: They established a connection between the network under training and its MF limit.

Original: They then used the MF limit to prove that two-layer networks could be trained to ﬁnd (near) global optima using variants of gradient descent, despite non-convexity (Mei et al.
Revised: They then used the MF limit to prove that two-layer networks could be trained to ﬁnd (near) global optima using variants of gradient descent, despite non-convexity (Mei et al.

Original: ; Chizat & Bach (2018)).
Revised: ; Chizat & Bach (2018)).

Original: The MF limit identiﬁed by these works assumes the form of gradient ﬂows in the measure space, which factors out the invariance from the action of a symmetry group on the model.
Revised: The MF limit identiﬁed by these works assumes the form of gradient ﬂows in the measure space, which factors out the invariance from the action of a symmetry group on the model.

Original: Interestingly, by lifting to the measure space, with a convex loss function (e.g. squared loss), one obtains a limiting optimization problem that is convex (Bengio et al.
Revised: Interestingly, by lifting to the measure space, with a convex loss function (e.g. squared loss), one obtains a limiting optimization problem that is convex (Bengio et al.

Original: ; Bach (2017)).
Revised: ; Bach (2017)).

Original: The analyses of Mei et al.
Revised: The analyses of Mei et al.

Original: Chizat & Bach (2018) utilize convexity, although the mechanisms to attain global convergence in these works are more sophisticated than the usual convex optimization setup in Euclidean spaces.
Revised: Chizat & Bach (2018) utilize convexity, although the mechanisms to attain global convergence in these works are more sophisticated than the usual convex optimization setup in Euclidean spaces.

Original: The extension to multilayer networks has enjoyed much less progresses.
Revised: The extension to multilayer networks has enjoyed much less progresses.

Original: The works Nguyen (2019);
Revised: The works Nguyen (2019);

Original: Araújo et al.
Revised: Araújo et al.

Original: ; Sirignano & Spiliopoulos (2019) argued, heuristically or rigorously, for the existence of a MF limiting behavior under gradient descent training with different assumptions.
Revised: ; Sirignano & Spiliopoulos (2019) argued, heuristically or rigorously, for the existence of a MF limiting behavior under gradient descent training with different assumptions.

Original: In fact, it has been argued that the difﬁculty is not simply technical, but rather conceptual (Nguyen (2019)): for instance, the presence of intermediate layers exhibits multiple symmetry groups with intertwined actions on the model.
Revised: In fact, it has been argued that the difﬁculty is not simply technical, but rather conceptual (Nguyen (2019)): for instance, the presence of intermediate layers exhibits multiple symmetry groups with intertwined actions on the model.

Original: Convergence to the global optimum of the model under gradientbased optimization has not been established when there are more than two layers.
Revised: Convergence to the global optimum of the model under gradientbased optimization has not been established when there are more than two layers.

Original: In this work, we prove a global convergence guarantee for feedforward three-layer networks trained with unregularized stochastic gradient descent (SGD) in the MF regime.
Revised: In this work, we prove a global convergence guarantee for feedforward three-layer networks trained with unregularized stochastic gradient descent (SGD) in the MF regime.

Original: After an introduction of the three-layer setup and its MF limit in Section 2, our development proceeds in two main steps:
Revised: After an introduction of the three-layer setup and its MF limit in Section 2, our development proceeds in two main steps:

Original: Step 1 (Theorem 3 in Section 3): We ﬁrst develop a rigorous framework that describes the MF limit and establishes its connection with a large-width SGD-trained three-layer network.
Revised: Step 1 (Theorem 3 in Section 3): We ﬁrst develop a rigorous framework that describes the MF limit and establishes its connection with a large-width SGD-trained three-layer network.

Original: Here we propose the new idea of a neuronal embedding , which comprises of an appropriate non-evolving probability space that encapsulates neural networks of arbitrary sizes.
Revised: Here we propose the new idea of a neuronal embedding , which comprises of an appropriate non-evolving probability space that encapsulates neural networks of arbitrary sizes.

Original: This probability space is in general abstract and is constructed according to the (not necessarily i.i.d.) initialization scheme of the neural network.
Revised: This probability space is in general abstract and is constructed according to the (not necessarily i.i.d.) initialization scheme of the neural network.

Original: This idea addresses directly the intertwined action of multiple symmetry groups, which is the aforementioned conceptual obstacle (Nguyen (2019)), thereby covering setups that cannot be handled by formulations in Araújo et al.
Revised: This idea addresses directly the intertwined action of multiple symmetry groups, which is the aforementioned conceptual obstacle (Nguyen (2019)), thereby covering setups that cannot be handled by formulations in Araújo et al.

Original: (2019); Sirignano & Spiliopoulos (2019) (see also
Revised: (2019); Sirignano & Spiliopoulos (2019) (see also

Original: Section 5 for a comparison).
Revised: Section 5 for a comparison).

Original: Our analysis follows the technique from Sznitman (1991); Mei et al.
Revised: Our analysis follows the technique from Sznitman (1991); Mei et al.

Original: and gives a quantitative statement: in particular, the MF limit yields a good approximation of the neural network as long as n − 1min log n max (cid:28) 1 independent of the data dimension, where n min and n max are the minimum and maximum of the widths.
Revised: and gives a quantitative statement: in particular, the MF limit yields a good approximation of the neural network as long as n − 1min log n max (cid:28) 1 independent of the data dimension, where n min and n max are the minimum and maximum of the widths.

Original: Step 2 (Theorem 8 in Section 4):
Revised: Step 2 (Theorem 8 in Section 4):

Original: We prove that the MF limit, given by our framework, converges to the global optimum under suitable regularity and convergence mode assumptions.
Revised: We prove that the MF limit, given by our framework, converges to the global optimum under suitable regularity and convergence mode assumptions.

Original: Several elements of our proof are inspired by Chizat & Bach (2018); the technique in their work however does not generalize to our three-layer setup.
Revised: Several elements of our proof are inspired by Chizat & Bach (2018); the technique in their work however does not generalize to our three-layer setup.

Original: Unlike previous two-layer analyses, we do not exploit convexity; instead we make use of a new element: a universal approximation property.
Revised: Unlike previous two-layer analyses, we do not exploit convexity; instead we make use of a new element: a universal approximation property.

Original: The result turns out to be conceptually new: global convergence can be achieved even when the loss function is non-convex.
Revised: The result turns out to be conceptually new: global convergence can be achieved even when the loss function is non-convex.

Original: An important crux of the proof is to show that the universal approximation property holds at any ﬁnite training time (but not necessarily at convergence, i.e. at inﬁnite time, since the property may not realistically hold at convergence).
Revised: An important crux of the proof is to show that the universal approximation property holds at any ﬁnite training time (but not necessarily at convergence, i.e. at inﬁnite time, since the property may not realistically hold at convergence).

Original: Together these two results imply a positive statement on the optimization efﬁciency of SGD-trained unregularized feedforward three-layer networks (Corollary 10).
Revised: Together these two results imply a positive statement on the optimization efﬁciency of SGD-trained unregularized feedforward three-layer networks (Corollary 10).

Original: Certain parts of our work can be readily extended to the general multilayer case or used to obtain new global convergence guarantees in the two-layer case.
Revised: Certain parts of our work can be readily extended to the general multilayer case or used to obtain new global convergence guarantees in the two-layer case.

Original: We choose to keep the current paper concise with the three-layer case being a prototypical setup that conveys the most of the ideas.
Revised: We choose to keep the current paper concise with the three-layer case being a prototypical setup that conveys the most of the ideas.

Original: Complete proofs are presented in appendices.
Revised: Complete proofs are presented in appendices.

Original: Notations.
Revised: Notations.

Original: K denotes a generic constant that may change from line to line.
Revised: K denotes a generic constant that may change from line to line.

Original: |·| denotes the absolute value for a scalar and the Euclidean norm for a vector.
Revised: |·| denotes the absolute value for a scalar and the Euclidean norm for a vector.

Original: For an integer n , we let [ n ] = { 1 , ..., n } .
Revised: For an integer n , we let [ n ] = { 1 , ..., n } .

Original:  T HREE - LAYER NEURAL NETWORKS AND THE MEAN FIELD LIMIT 2. T HREE - LAYER NEURAL NETWORK
Revised:  T HREE - LAYER NEURAL NETWORKS AND THE MEAN FIELD LIMIT 2. T HREE - LAYER NEURAL NETWORK

Original: We consider the following three-layer network  that takes as input x ∈ R d :
Revised: We consider the following three-layer network at time k ∈ N ≥ 0 that takes as input x ∈ R d :

Original: [Equation]
Revised: [Equation]

Original: in which W ( k ) = ( w 
Revised: in which W ( k ) = ( w 1 ( k, · ) , w

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) is the weight 2with w 1{+
+} ( k, j 1 ) ∈ R d , w 2 ( k, j 1 , j 2 ) ∈ R and w 3 ( k, j 2 ) ∈ R , ϕ 1 : R → R , ϕ 2 : R → R and ϕ 3 : R → R are the activations. Here the network has widths  n 1 , n 2 } , and k ∈ N ≥ the
Revised: , w 3 ( k, · )) consists of the weights 2 w {+
+} ( k, j 1 ) ∈ R d , w 2 ( k, j 1 , j 2 ) ∈ R and w 3 ( k, j 2 ) ∈ R . Here ϕ 1 : R → R , ϕ 2 : R → R and ϕ 3 : R → R are the activation functions, and the network has widths { n 1 , n 2 } .

Original: We train the network with SGD w.r.t.
Revised: We train the network with SGD w.r.t.

Original: the loss L : R × R → R ≥ 0 .
Revised: the loss L : R × R → R ≥ 0 .

Original: We assume that at each time k , we draw independently a fresh sample z ( k ) =
Revised: We assume that at each time k , we draw independently a fresh sample z ( k ) =

Original: ( x ( k ) , y ( k )) ∈
Revised: ( x ( k ) , y ( k )) ∈

Original: R
Revised: R

Original: d × R from a training distribution
Revised: d × R from a training distribution

Original: P .
Revised: P .

Original: Given an initialization W (0) , we update W ( k ) according to
Revised: Given an initialization W (0) , we update W ( k ) according to

Original: [Equation]
Revised: [Equation]

Original: ( k )) ,
Revised: ( k )) ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:15)ξ
Revised: (cid:15)ξ

Original: [Equation]
Revised: [Equation]

Original: ( k(cid:15) )
Revised: ( k(cid:15) )

Original: Grad
Revised: Grad

Original: [Equation]
Revised: [Equation]

Original: ( z ( k ) , j 1 , j
Revised: ( z ( k ) , j 1 , j

Original: [Equation]
Revised: [Equation]

Original: W ( k )) , w
Revised: W ( k )) , w

Original: [Equation]
Revised: [Equation]

Original: ( k
Revised: ( k

Original: [Equation]
Revised: [Equation]

Original: 1 , j 1 ) =
Revised: 1 , j 1 ) =

Original: w 1 ( k, j 1 ) − (cid:15)ξ 1 ( k(cid:15) )
Revised: w 1 ( k, j 1 ) − (cid:15)ξ 1 ( k(cid:15) )

Original: Grad 1 ( z ( k ) , j 1 ; W ( k )) , in which j 1 = 1 , ..., n 1 , j 2 = 1 , ..., n 2 , (cid:15) ∈ R > 0 is the learning rate, ξ i : R ≥ 

Revised: Grad 1 ( z ( k ) , j 1 ; W ( k )) , in which j 1 = 1 , ..., n 1 , j 2 = 1 , ..., n 2 , (cid:15) ∈ R > 0 is the learning rate, ξ i : R ≥

Original: (cid:55)→ R ≥ 0 is the learning rate schedule for w i , and for z = ( x, y ) , we deﬁne
Revised: (cid:55)→ R ≥ 0 is the learning rate schedule for w i , and for z = ( x, y ) , we deﬁne

Original: Grad 3 ( z, j 2 ; W ( k )) =
Revised: Grad 3 ( z, j 2 ; W ( k )) =

Original: ∂ 2 L ( y, ˆ y ( x ; W ( k )))
Revised: ∂ 2 L ( y, ˆ y ( x ; W ( k )))

Original: ϕ (cid:48)
Revised: ϕ (cid:48)

Original: 3 ( H 3 ( x ; W ( k )))
Revised: 3 ( H 3 ( x ; W ( k )))

Original: ϕ
Revised: ϕ

Original: 2 ( H 2 ( x, j 2 ; W ( k ))) ,
Revised: 2 ( H 2 ( x, j 2 ; W ( k ))) ,

Original: Grad 2 ( z, j 1 , j 2 ; W ( k ))
Revised: Grad 2 ( z, j 1 , j 2 ; W ( k ))

Original: =
Revised: =

Original: ∆ H 2 ( z, j
Revised: ∆ H 2 ( z, j

Original: [Equation]
Revised: [Equation]

Original: W ( k )) ϕ 1 ( (cid:104) w 1 ( k, j 1 ) , x (cid:105) ) ,
Revised: W ( k )) ϕ 1 ( (cid:104) w 1 ( k, j 1 ) , x (cid:105) ) ,

Original: Grad 1 ( z, j 1 ; W ( k ))
Revised: Grad 1 ( z, j 1 ; W ( k ))

Original: = (cid:18)nn 2 (cid:88)
Revised: = (cid:18)nn 2 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: We note that this setup follows the same scaling w.r.t.
Revised: We note that this setup follows the same scaling w.r.t.

Original: n 1 and n 2 , which is applied to both the forward pass and the learning rates in the backward pass, as Nguyen (2019).
Revised: n 1 and n 2 , which is applied to both the forward pass and the learning rates in the backward pass, as Nguyen (2019).

Original: 2. M EAN FIELD LIMIT
Revised: 2. M EAN FIELD LIMIT

Original: The MF limit is a continuous-time inﬁnite-width analog of the neural network under training.
Revised: The MF limit is a continuous-time inﬁnite-width analog of the neural network under training.

Original: To describe it, we ﬁrst introduce the concept of a neuronal ensemble .
Revised: To describe it, we ﬁrst introduce the concept of a neuronal ensemble .

Original: Given a product probability space (Ω , F , P ) =
Revised: Given a product probability space (Ω , F , P ) =

Original: (Ω 1 × Ω 2 , F 1 × F 1 , P 1 × P 2 ) , we independently sample C i ∼ P
Revised: (Ω 1 × Ω 2 , F 1 × F 1 , P 1 × P 2 ) , we independently sample C i ∼ P

Original: i , i = 1 , 2 .
Revised: i , i = 1 , 2 .

Original: In the following, we use E C i to denote the expectation w.r.t.
Revised: In the following, we use E C i to denote the expectation w.r.t.

Original: the random variable C i ∼ P
Revised: the random variable C i ∼ P

Original: i
Revised: i

Original: and
Revised: and

Original: c i to denote an arbitrary point c i ∈ Ω i .
Revised: c i to denote an arbitrary point c i ∈ Ω i .

Original: The space (Ω , F , P ) is referred to as a neuronal ensemble .
Revised: The space (Ω , F , P ) is referred to as a neuronal ensemble .

Original: Given a neuronal ensemble (Ω , F , P ) , the MF limit is described by a time-evolving system with state/parameter W ( t ) , where the time t ∈ R ≥ 0 and W ( t ) =
Revised: Given a neuronal ensemble (Ω , F , P ) , the MF limit is described by a time-evolving system with state/parameter W ( t ) , where the time t ∈ R ≥ 0 and W ( t ) =

Original: ( w 1 ( t, · ) , w 2 ( t, · , · ) , w 3 ( t, · )) with w 1 : R ≥ 0 × Ω 1 → R d , w 2 : R ≥ 

Revised: ( w 1 ( t, · ) , w 2 ( t, · , · ) , w 3 ( t, · )) with w 1 : R ≥ 0 × Ω 1 → R d , w 2 : R ≥

Original: × Ω 1 × Ω 2 → R and w 3 : R ≥ 

Revised: × Ω 1 × Ω 2 → R and w 3 : R ≥

Original: × Ω 2 → R .
Revised: × Ω 2 → R .

Original: It entails the quantities:
Revised: It entails the quantities:

Original: [Equation]
Revised: [Equation]

Original: Here for each t ∈ R ≥ 0 , w 1 ( t, · ) is (Ω 1 , F 1 ) -measurable, and similar for w 2 ( t, · , · ) , w 3 ( t, · ) .
Revised: Here for each t ∈ R ≥ 0 , w 1 ( t, · ) is (Ω 1 , F 1 ) -measurable, and similar for w 2 ( t, · , · ) , w 3 ( t, · ) .

Original: The MF limit evolves according to a continuous-time dynamics, described by a system of
Revised: The MF limit evolves according to a continuous-time dynamics, described by a system of

Original: ODEs, which we refer to as the MF ODEs .
Revised: ODEs, which we refer to as the MF ODEs .

Original: Speciﬁcally, given an initialization W (0) =
Revised: Speciﬁcally, given an initialization W (0) =

Original: ( w 1 (0 , · ) , w 2 (0 , · , · ) , w 3 (0 , · )) , the dynamics solves:
Revised: ( w 1 (0 , · ) , w 2 (0 , · , · ) , w 3 (0 , · )) , the dynamics solves:

Original: [Equation]
Revised: [Equation]

Original: Here c 1 ∈ Ω 1 , c 2 ∈ Ω 2 , E Z denotes the expectation w.r.t.
Revised: Here c 1 ∈ Ω 1 , c 2 ∈ Ω 2 , E Z denotes the expectation w.r.t.

Original: the data Z =
Revised: the data Z =

Original: ( X, Y ) ∼ P , and for z = ( x, y ) , we deﬁne
Revised: ( X, Y ) ∼ P , and for z = ( x, y ) , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: In Appendix B, we show well-posedness of MF ODEs under the following regularity conditions.
Revised: In Appendix B, we show well-posedness of MF ODEs under the following regularity conditions.

Original: Assumption 1 (Regularity) .
Revised: Assumption 1 (Regularity) .

Original: We assume that ϕ 1 and ϕ 2 are K -bounded, ϕ (cid:48) 1 , ϕ (cid:48) 2 and ϕ (cid:48) 3 are Kbounded and K -Lipschitz, ϕ (cid:48) 

Revised: We assume that ϕ 1 and ϕ 2 are K -bounded, ϕ (cid:48) 1 , ϕ (cid:48) 2 and ϕ (cid:48) 3 are Kbounded and K -Lipschitz, ϕ (cid:48)

Original: and ϕ (cid:48) 3 are non-zero everywhere, ∂ 2 L ( · , · ) is K -Lipschitz in the second variable and K -bounded, and | X | ≤ K with probability 1 .
Revised: and ϕ (cid:48) 3 are non-zero everywhere, ∂ 2 L ( · , · ) is K -Lipschitz in the second variable and K -bounded, and | X | ≤ K with probability 1 .

Original: Furthermore ξ 1 , ξ 2 and ξ 3 are
Revised: Furthermore ξ 1 , ξ 2 and ξ 3 are

Original: K -bounded and K -Lipschitz.
Revised: K -bounded and K -Lipschitz.

Original: Theorem 1.
Revised: Theorem 1.

Original: Under Assumption 1, given any neuronal ensemble and an initialization W (0) such that 3 ess - sup | w 2 (0 , C 1 , C 2 ) | , ess - sup | w 3 (0 , C 2 ) | ≤ K , there exists a unique solution W to the
Revised: Under Assumption 1, given any neuronal ensemble and an initialization W (0) such that 3 ess - sup | w 2 (0 , C 1 , C 2 ) | , ess - sup | w 3 (0 , C 2 ) | ≤ K , there exists a unique solution W to the

Original: MF ODEs on t ∈ [0 , ∞ ) .
Revised: MF ODEs on t ∈ [0 , ∞ ) .

Original: An example of a suitable setup is ϕ 1 =
Revised: An example of a suitable setup is ϕ 1 =

Original: ϕ 2 = tanh , ϕ 3 is the identity, L is the Huber loss, although a non-convex sufﬁciently smooth loss function sufﬁces.
Revised: ϕ 2 = tanh , ϕ 3 is the identity, L is the Huber loss, although a non-convex sufﬁciently smooth loss function sufﬁces.

Original: In fact, all of our developments can be easily modiﬁed to treat the squared loss with an additional assumption | Y | ≤ K with probability 1 .
Revised: In fact, all of our developments can be easily modiﬁed to treat the squared loss with an additional assumption | Y | ≤ K with probability 1 .

Original: So far, given an arbitrary neuronal ensemble (Ω , F , P ) , for each initialization W (0) , we have deﬁned a MF limit W ( t ) .
Revised: So far, given an arbitrary neuronal ensemble (Ω , F , P ) , for each initialization W (0) , we have deﬁned a MF limit W ( t ) .

Original: The connection with the neural network’s dynamics W ( k ) is established in the next section.
Revised: The connection with the neural network’s dynamics W ( k ) is established in the next section.

Original:  C ONNECTION BETWEEN NEURAL NETWORK AND ITS MEAN FIELD LIMIT 3. N EURONAL EMBEDDING AND THE COUPLING PROCEDURE
Revised:  C ONNECTION BETWEEN NEURAL NETWORK AND ITS MEAN FIELD LIMIT 3. N EURONAL EMBEDDING AND THE COUPLING PROCEDURE

Original: To formalize a connection between the neural network and its MF limit, we consider their initializations.
Revised: To formalize a connection between the neural network and its MF limit, we consider their initializations.

Original: In practical scenarios, to set the initial parameters W (0) of the neural network, one typically randomizes W (0) according to some distributional law ρ .
Revised: In practical scenarios, to set the initial parameters W (0) of the neural network, one typically randomizes W (0) according to some distributional law ρ .

Original: We note that since the neural network is deﬁned w.r.t.
Revised: We note that since the neural network is deﬁned w.r.t.

Original: a set of ﬁnite integers { n 1 , n 2 } , so is ρ .
Revised: a set of ﬁnite integers { n 1 , n 2 } , so is ρ .

Original: We consider a family Init of initialization laws, each of which is indexed by the set { n 1 , n 2 } :
Revised: We consider a family Init of initialization laws, each of which is indexed by the set { n 1 , n 2 } :

Original: Init = { ρ : ρ is the initialization law of a neural network of size { n 1 , n 2 } , n 1 , n 2 ∈ N > 0 } .
Revised: Init = { ρ : ρ is the initialization law of a neural network of size { n 1 , n 2 } , n 1 , n 2 ∈ N > 0 } .

Original: This is helpful when one is to take a limit that sends n 1 , n 2 → ∞ , in which case the size of this family | Init | is inﬁnite.
Revised: This is helpful when one is to take a limit that sends n 1 , n 2 → ∞ , in which case the size of this family | Init | is inﬁnite.

Original: More generally we allow | Init | < ∞ (for example, Init contains a single law ρ of a network of size { n 1 , n 2 } and hence | Init | = 1 ).
Revised: More generally we allow | Init | < ∞ (for example, Init contains a single law ρ of a network of size { n 1 , n 2 } and hence | Init | = 1 ).

Original: We make the following crucial deﬁnition.
Revised: We make the following crucial deﬁnition.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Given a family of initialization laws Init , we call (Ω , F , P, (cid:8) w 0 i (cid:9) i =1 , 2 , 3 ) a neuronal embedding of Init if the following hold:
Revised: Given a family of initialization laws Init , we call (Ω , F , P, (cid:8) w 0 i (cid:9) i =1 , 2 , 3 ) a neuronal embedding of Init if the following holds:

Original: (Ω , F , P ) =
Revised: (Ω , F , P ) =

Original: (Ω 1 × Ω 2 , F 1 × F 2 , P 1 × P 2 ) a product measurable space.
Revised: (Ω 1 × Ω 2 , F 1 × F 2 , P 1 × P 2 ) a product measurable space.

Original: As a reminder, we call it a neuronal ensemble.
Revised: As a reminder, we call it a neuronal ensemble.

Original: The deterministic functions w 01 : Ω 1 → R d , w 02 :
Revised: The deterministic functions w 01 : Ω 1 → R d , w 02 :

Original: Ω 1 × Ω 2 → R and w 03 : Ω 2 → R are such that, for each index { n 1 , n 2 } of Init and the law ρ of this index, if — with an abuse of notations — we independently sample { C i ( j i ) } j i ∈ [ n i ] ∼ P i i.i.d.
Revised: Ω 1 × Ω 2 → R and w 03 : Ω 2 → R are such that, for each index { n 1 , n 2 } of Init and the law ρ of this index, if — with an abuse of notations — we independently sample { C i ( j i ) } j i ∈ [ n i ] ∼ P i i.i.d.

Original: for each i = 1 , 2 , then Law (cid:0) w 01 ( C 1 ( j 1 )) , w 02 ( C 1 ( j 1 ) , C 2 ( j 2 )) , w 03 ( C 2 ( j 2 )) , j i ∈ [ n i ] , i = 1 , 2 (cid:1) = ρ.
Revised: for each i = 1 , 2 , then Law (cid:0) w 01 ( C 1 ( j 1 )) , w 02 ( C 1 ( j 1 ) , C 2 ( j 2 )) , w 03 ( C 2 ( j 2 )) , j i ∈ [ n i ] , i = 1 , 2 (cid:1) = ρ.

Original: To proceed, given Init and { n 1 , n 2 } in its index set, we perform the following coupling procedure :
Revised: To proceed, given Init and { n 1 , n 2 } in its index set, we perform the following coupling procedure :

Original: Let (Ω , F , P, (cid:8) w 0 i (cid:9) i =1 , 2 , 3 ) be a neuronal embedding of Init .
Revised: Let (Ω , F , P, (cid:8) w 0 i (cid:9) i =1 , 2 , 3 ) be a neuronal embedding of Init .

Original: We form the MF limit W ( t ) (for t ∈ R ≥ 0 ) associated with the neuronal ensemble (Ω , F , P ) by setting the initialization W (0) to w 1 (0 , · ) =
Revised: We form the MF limit W ( t ) (for t ∈ R ≥ 0 ) associated with the neuronal ensemble (Ω , F , P ) by setting the initialization W (0) to w 1 (0 , · ) =

Original: w 01 ( · ) , w 2 (0 , · , · ) =
Revised: w 01 ( · ) , w 2 (0 , · , · ) =

Original: w 02 ( · , · ) and w 3 (0 , · ) =
Revised: w 02 ( · , · ) and w 3 (0 , · ) =

Original: w 03 ( · ) and running the MF ODEs described in Section 2.2. 3.
Revised: w 03 ( · ) and running the MF ODEs described in Section 2.2. 3.

Original: We independently sample C i ( j i ) ∼ P i for i = 1 , 2 and j i =
Revised: We independently sample C i ( j i ) ∼ P i for i = 1 , 2 and j i =

Original: 1 , ..., n i .
Revised: 1 , ..., n i .

Original: We then form the neural network initialization W (0) with w 1 (0 , j 1 ) =
Revised: We then form the neural network initialization W (0) with w 1 (0 , j 1 ) =

Original: w 01 ( C 1 ( j 1 )) , w 2 (0 , j 1 , j 2 )
Revised: w 01 ( C 1 ( j 1 )) , w 2 (0 , j 1 , j 2 )

Original: =
Revised: =

Original: w 02 ( C 1 ( j 1 ) , C 2 ( j 2 )) and w 3 (0 , j 2 ) =
Revised: w 02 ( C 1 ( j 1 ) , C 2 ( j 2 )) and w 3 (0 , j 2 ) =

Original: w 03 ( C 2 ( j 2 )) for j 1 ∈ [ n 1 ] , j 2 ∈ [ n 2 ] .
Revised: w 03 ( C 2 ( j 2 )) for j 1 ∈ [ n 1 ] , j 2 ∈ [ n 2 ] .

Original: We obtain the network’s trajectory W ( k ) for k ∈ N ≥ 0 as in Section 2.1, with the data z ( k ) generated independently of { C i ( j i ) } i =1 , 2 and hence W (0) .
Revised: We obtain the network’s trajectory W ( k ) for k ∈ N ≥ 0 as in Section 2.1, with the data z ( k ) generated independently of { C i ( j i ) } i =1 , 2 and hence W (0) .

Original: We can then deﬁne a measure of closeness between W ( (cid:98) t/(cid:15) (cid:99) ) and W ( t ) for t ∈ [0 , T ] :
Revised: We can then deﬁne a measure of closeness between W ( (cid:98) t/(cid:15) (cid:99) ) and W ( t ) for t ∈ [0 , T ] :

Original: D T ( W, W ) = sup (cid:8) | w 1 ( (cid:98) t/(cid:15) (cid:99) , j 1 ) − w 1 ( t, C 1 ( j 1 ))
Revised: D T ( W, W ) = sup (cid:8) | w 1 ( (cid:98) t/(cid:15) (cid:99) , j 1 ) − w 1 ( t, C 1 ( j 1 ))

Original: | , | w 2 ( (cid:98) t/(cid:15) (cid:99) , j 1 , j 2 ) − w 2 ( t, C 1 ( j 1 ) , C 2 ( j 2 )) | , | w 3 ( (cid:98) t/(cid:15) (cid:99) , j 2 ) − w 3 ( t, C 2 ( j 2 ))
Revised: | , | w 2 ( (cid:98) t/(cid:15) (cid:99) , j 1 , j 2 ) − w 2 ( t, C 1 ( j 1 ) , C 2 ( j 2 )) | , | w 3 ( (cid:98) t/(cid:15) (cid:99) , j 2 ) − w 3 ( t, C 2 ( j 2 ))

Original: | : t ≤ T, j 1 ≤ n 1 , j 2 ≤ n 2 (cid:9) .
Revised: | : t ≤ T, j 1 ≤ n 1 , j 2 ≤ n 2 (cid:9) .

Original: Note that W ( t ) is a deterministic trajectory independent of { n 1 , n 2 } , whereas W ( k ) is random for all k ∈ N ≥ 

Revised: Note that W ( t ) is a deterministic trajectory independent of { n 1 , n 2 } , whereas W ( k ) is random for all k ∈ N ≥

Original: due to the randomness of { C i ( j i ) } i =1 , 2 and the generation of the training data z ( k ) .
Revised: due to the randomness of { C i ( j i ) } i =1 , 2 and the generation of the training data z ( k ) .

Original: Similarly D T ( W, W ) is a random quantity.
Revised: Similarly D T ( W, W ) is a random quantity.

Original: The idea of the coupling procedure is closely related to the coupling argument in Sznitman (1991);
Revised: The idea of the coupling procedure is closely related to the coupling argument in Sznitman (1991);

Original: Mei et al.
Revised: Mei et al.

Original: Here, instead of playing the role of a proof technique, the coupling serves as a vehicle to establish the connection between W and W on the basis of the neuronal embedding.
Revised: Here, instead of playing the role of a proof technique, the coupling serves as a vehicle to establish the connection between W and W on the basis of the neuronal embedding.

Original: This connection is shown in Theorem 3 below, which gives an upper bound on D T ( W, W ) .
Revised: This connection is shown in Theorem 3 below, which gives an upper bound on D T ( W, W ) .

Original: We note that the coupling procedure can be carried out to provide a connection between W and
Revised: We note that the coupling procedure can be carried out to provide a connection between W and

Original: W as long as there exists a neuronal embedding for Init .
Revised: W as long as there exists a neuronal embedding for Init .

Original: Later in Section 4.1, we show that for a common initialization scheme (in particular, i.i.d. initialization) for Init , there exists a neuronal embedding.
Revised: Later in Section 4.1, we show that for a common initialization scheme (in particular, i.i.d. initialization) for Init , there exists a neuronal embedding.

Original: Theorem 3 applies to, but is not restricted to, this initialization scheme.
Revised: Theorem 3 applies to, but is not restricted to, this initialization scheme.

Original: 3. M AIN RESULT : APPROXIMATION BY THE MF LIMIT
Revised: 3. M AIN RESULT : APPROXIMATION BY THE MF LIMIT

Original: Assumption 2 (Initialization of second and third layers) .
Revised: Assumption 2 (Initialization of second and third layers) .

Original: We assume that ess - sup (cid:12)(cid:12) w 02 ( C 1 , C 2 ) (cid:12)(cid:12) , ess - sup (cid:12)(cid:12)
Revised: We assume that ess - sup (cid:12)(cid:12) w 02 ( C 1 , C 2 ) (cid:12)(cid:12) , ess - sup (cid:12)(cid:12)

Original: w 03 ( C 2 ) (cid:12)(cid:12) ≤ K , where w 02 and w 03 are as described in Deﬁnition 2.
Revised: w 03 ( C 2 ) (cid:12)(cid:12) ≤ K , where w 02 and w 03 are as described in Deﬁnition 2.

Original: Theorem 3.
Revised: Theorem 3.

Original: Given a family Init of initialization laws and a tuple { n 1 , n 2 } that is in the index set of Init , perform the coupling procedure as described in Section 3.1.
Revised: Given a family Init of initialization laws and a tuple { n 1 , n 2 } that is in the index set of Init , perform the coupling procedure as described in Section 3.1.

Original: Fix a terminal time T ∈ (cid:15) N ≥ 0 .
Revised: Fix a terminal time T ∈ (cid:15) N ≥ 0 .

Original: Under Assumptions 1 and 2, for (cid:15) ≤ 1 , we have with probability at least 1 − 2 δ ,
Revised: Under Assumptions 1 and 2, for (cid:15) ≤ 1 , we have with probability at least 1 − 2 δ ,

Original: D T ( W, W ) ≤ e K T (cid:18)
Revised: D T ( W, W ) ≤ e K T (cid:18)

Original: [Equation]
Revised: [Equation]

Original: in which n min = min { n 1 , n 2 } , n max = max { n 1 , n 2 } , and K T = K (cid:0) 

Revised: in which n min = min { n 1 , n 2 } , n max = max { n 1 , n 2 } , and K T = K (cid:0)

Original: + T K (cid:1) .
Revised: + T K (cid:1) .

Original: The theorem gives a connection between W ( (cid:98) t/(cid:15) (cid:99) ) , which is deﬁned upon ﬁnite widths n 1 and n 2 , and the MF limit W ( t ) , whose description is independent of n 1 and n 2 .
Revised: The theorem gives a connection between W ( (cid:98) t/(cid:15) (cid:99) ) , which is deﬁned upon ﬁnite widths n 1 and n 2 , and the MF limit W ( t ) , whose description is independent of n 1 and n 2 .

Original: It lends a way to extract properties of the neural network in the large-width regime.
Revised: It lends a way to extract properties of the neural network in the large-width regime.

Original: Corollary 4.
Revised: Corollary 4.

Original: Under the same setting as Theorem 3, consider any test function ψ : R × R → R which is K -Lipschitz in the second variable uniformly in the ﬁrst variable (an example of ψ is the loss L ).
Revised: Under the same setting as Theorem 3, consider any test function ψ : R × R → R which is K -Lipschitz in the second variable uniformly in the ﬁrst variable (an example of ψ is the loss L ).

Original: For any δ > 0 , with probability at least 1 − 3 δ , sup t ≤ T | E
Revised: For any δ > 0 , with probability at least 1 − 3 δ , sup t ≤ T | E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Y,
Revised: Y,

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: These bounds hold for any n 1 and n 2 , similar to Mei et al.
Revised: These bounds hold for any n 1 and n 2 , similar to Mei et al.

Original: ; Araújo et al.
Revised: ; Araújo et al.

Original: (2019), in contrast with non-quantitative results in Chizat & Bach (2018); Sirignano & Spiliopoulos (2019).
Revised: (2019), in contrast with non-quantitative results in Chizat & Bach (2018); Sirignano & Spiliopoulos (2019).

Original: These bounds suggest that n 1 and n 2 can be chosen independent of the data dimension d .
Revised: These bounds suggest that n 1 and n 2 can be chosen independent of the data dimension d .

Original: This agrees with the experiments in Nguyen (2019), which found width ≈ 1000 to be typically sufﬁcient to observe
Revised: This agrees with the experiments in Nguyen (2019), which found width ≈ 1000 to be typically sufﬁcient to observe

Original: MF behaviors in networks trained with real-life high-dimensional data.
Revised: MF behaviors in networks trained with real-life high-dimensional data.

Original: We observe that the MF trajectory W ( t ) is deﬁned as per the choice of the neuronal embedding (Ω , F , P, (cid:8) w 

Revised: We observe that the MF trajectory W ( t ) is deﬁned as per the choice of the neuronal embedding (Ω , F , P, (cid:8) w

Original: i (cid:9) i =1 , 2 , 3 ) , which may not be unique.
Revised: i (cid:9) i =1 , 2 , 3 ) , which may not be unique.

Original: On the other hand, the neural network’s trajectory
Revised: On the other hand, the neural network’s trajectory

Original: W ( k ) depends on the randomization of the initial parameters W (0) according to an initialization law from the family Init (as well as the data z ( k ) ) and hence is independent of this choice.
Revised: W ( k ) depends on the randomization of the initial parameters W (0) according to an initialization law from the family Init (as well as the data z ( k ) ) and hence is independent of this choice.

Original: Another corollary of Theorem 3 is that given the same family Init , the law of the MF trajectory is insensitive to the choice of the neuronal embedding of Init .
Revised: Another corollary of Theorem 3 is that given the same family Init , the law of the MF trajectory is insensitive to the choice of the neuronal embedding of Init .

Original: Corollary 5.
Revised: Corollary 5.

Original: Consider a family Init of initialization laws, indexed by a set of tuples { m 1 , m 2 } that contains a sequence of indices { m 1 ( m ) , m 2 ( m ) : m ∈ N } in which as m → ∞ , min { m 1 ( m ) , m 2 ( m ) } − 1 log (max { m 1 ( m ) , m 2 ( m ) } )
Revised: Consider a family Init of initialization laws, indexed by a set of tuples { m 1 , m 2 } that contains a sequence of indices { m 1 ( m ) , m 2 ( m ) : m ∈ N } in which as m → ∞ , min { m 1 ( m ) , m 2 ( m ) } − 1 log (max { m 1 ( m ) , m 2 ( m ) } )

Original: → 0 .
Revised: → 0 .

Original: Let W ( t ) and ˆ W ( t ) be two MF trajectories associated with two choices of neuronal embeddings of Init , (Ω , F , P, (cid:8) w 

Revised: Let W ( t ) and ˆ W ( t ) be two MF trajectories associated with two choices of neuronal embeddings of Init , (Ω , F , P, (cid:8) w

Original: i (cid:9) i =1 , 2 , 3 ) and (ˆΩ , ˆ F , ˆ P, (cid:8) ˆ w 

Revised: i (cid:9) i =1 , 2 , 3 ) and (ˆΩ , ˆ F , ˆ P, (cid:8) ˆ w

Original: i (cid:9) i =1 , 2 , 3 ) .
Revised: i (cid:9) i =1 , 2 , 3 ) .

Original: The following statement holds for any T ≥ 0 and any two positive integers n 1 and n 2 : if we independently sample C i ( j i ) ∼ P i and
Revised: The following statement holds for any T ≥ 0 and any two positive integers n 1 and n 2 : if we independently sample C i ( j i ) ∼ P i and

Original: ˆ C i ( j i ) ∼
Revised: ˆ C i ( j i ) ∼

Original: ˆ P
Revised: ˆ P

Original: i for j i ∈ [ n i ] , i = 1 , 2 , then Law ( W ( n 1 , n 2 , T ))
Revised: i for j i ∈ [ n i ] , i = 1 , 2 , then Law ( W ( n 1 , n 2 , T ))

Original: =
Revised: =

Original: Law( ˆ W ( n 1 , n 2 , T )) , where we deﬁne W ( n 1 , n 2 , T ) as the below collection w.r.t.
Revised: Law( ˆ W ( n 1 , n 2 , T )) , where we deﬁne W ( n 1 , n 2 , T ) as the below collection w.r.t.

Original: W ( t ) , and similarly deﬁne ˆ W ( n 1 , n 2 , T ) w.r.t.
Revised: W ( t ) , and similarly deﬁne ˆ W ( n 1 , n 2 , T ) w.r.t.

Original: ˆ W ( t ) :
Revised: ˆ W ( t ) :

Original: [Equation]
Revised: [Equation]

Original: The proofs are deferred to Appendix C.
Revised: The proofs are deferred to Appendix C.

Original:  C ONVERGENCE TO GLOBAL OPTIMA
Revised:  C ONVERGENCE TO GLOBAL OPTIMA

Original: In this section, we prove a global convergence guarantee for three-layer neural networks via the MF limit.
Revised: In this section, we prove a global convergence guarantee for three-layer neural networks via the MF limit.

Original: We consider a common class of initialization: i.i.d. initialization.
Revised: We consider a common class of initialization: i.i.d. initialization.

Original: 4. I. I . D . INITIALIZATION
Revised: 4. I. I . D . INITIALIZATION

Original: Deﬁnition 6.
Revised: Deﬁnition 6.

Original: An initialization law ρ for a neural network of size { n 1 , n 2 } is called (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) - i.i.d. initialization (or i.i.d.
Revised: An initialization law ρ for a neural network of size { n 1 , n 2 } is called (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) - i.i.d. initialization (or i.i.d.

Original: initialization, for brevity), where ρ 1 , ρ 2 and ρ 3 are probability measures over R d , R and R respectively, if { w 1 (0 , j 1 ) } j 1 ∈ [ n 1 ] are generated i.i.d.
Revised: initialization, for brevity), where ρ 1 , ρ 2 and ρ 3 are probability measures over R d , R and R respectively, if { w 1 (0 , j 1 ) } j 1 ∈ [ n 1 ] are generated i.i.d.

Original: according to ρ 1 , { w 2 (0 , j 1 , j 2 ) } j 1 ∈ [ n 1 ] , j 2 ∈ [ n 2 ] are generated i.i.d.
Revised: according to ρ 1 , { w 2 (0 , j 1 , j 2 ) } j 1 ∈ [ n 1 ] , j 2 ∈ [ n 2 ] are generated i.i.d.

Original: according to ρ 2 and { w 3 (0 , j 2 ) } j 2 ∈ [ n 2 ] are generated i.i.d.
Revised: according to ρ 2 and { w 3 (0 , j 2 ) } j 2 ∈ [ n 2 ] are generated i.i.d.

Original: according to ρ 3 , and w 1 , w 2 and w 3 are independent.
Revised: according to ρ 3 , and w 1 , w 2 and w 3 are independent.

Original: Observe that given (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) , one can build a family Init of i.i.d.
Revised: Observe that given (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) , one can build a family Init of i.i.d.

Original: initialization laws that contains any index set { n 1 , n 2 } .
Revised: initialization laws that contains any index set { n 1 , n 2 } .

Original: Furthermore i.i.d.
Revised: Furthermore i.i.d.

Original: initializations are supported by our framework, as stated in the following proposition and proven in Appendix D.
Revised: initializations are supported by our framework, as stated in the following proposition and proven in Appendix D.

Original: Proposition 7.
Revised: Proposition 7.

Original: There exists a neuronal embedding (cid:16) Ω , F , P, (cid:8) w 0 i (cid:9) i =1 , 2 , 3 (cid:17) for any family Init of initialization laws, which are (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d.
Revised: There exists a neuronal embedding (cid:16) Ω , F , P, (cid:8) w 0 i (cid:9) i =1 , 2 , 3 (cid:17) for any family Init of initialization laws, which are (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d.

Original: 4. M AIN RESULT : GLOBAL CONVERGENCE
Revised: 4. M AIN RESULT : GLOBAL CONVERGENCE

Original: To measure the learning quality, we consider the loss averaged over the data Z ∼ P :
Revised: To measure the learning quality, we consider the loss averaged over the data Z ∼ P :

Original: [Equation]
Revised: [Equation]

Original: E Z [ L ( Y, ˆ y ( X ; V ))] , where V = ( v 1 , v 2 , v 3 )  a set of three measurable functions v 1 : Ω 1 → R d , v 2 : Ω 1 × Ω 2 → R , v 3 : Ω 2 → R .
Revised: E Z [ L ( Y, ˆ y ( X ; V ))] , where V = ( v 1 , v 2 , v 3 ) is a set of three measurable functions v 1 : Ω 1 → R d , v 2 : Ω 1 × Ω 2 → R , v 3 : Ω 2 → R .

Original: Assumption 3.
Revised: Assumption 3.

Original: Consider a neuronal embedding (cid:16) Ω , F , P, (cid:8) w 

Revised: Consider a neuronal embedding (cid:16) Ω , F , P, (cid:8) w

Original: i (cid:9) i =1 , 2 , 3 (cid:17) of the (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d. initialization, and the associated MF limit with initialization W (0) such that w 1 (0 , · ) =
Revised: i (cid:9) i =1 , 2 , 3 (cid:17) of the (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d. initialization, and the associated MF limit with initialization W (0) such that w 1 (0 , · ) =

Original: w 01 ( · ) , w 2 (0 , · , · ) =
Revised: w 01 ( · ) , w 2 (0 , · , · ) =

Original: w 02 ( · , · ) and w 3 (0 , · ) =
Revised: w 02 ( · , · ) and w 3 (0 , · ) =

Original: w 03 ( · ) .
Revised: w 03 ( · ) .

Original: Assume:
Revised: Assume:

Original: Support: The support of ρ 1 is R d . 2.
Revised: Support: The support of ρ 1 is R d . 2.

Original: Convergence mode : There exist limits ¯ w 1 , ¯ w 2 and ¯ w 3 such that as t → ∞ ,
Revised: Convergence mode: There exist limits ¯ w 1 , ¯ w 2 and ¯ w 3 such that as t → ∞ ,

Original: [Equation]
Revised: [Equation]

Original: Universal approximation: (cid:8) ϕ 

Revised: Universal approximation: (cid:8) ϕ

Original: (cid:104) u, ·(cid:105) ) :
Revised: (cid:104) u, ·(cid:105) ) :

Original: u ∈ R d (cid:9) has dense span in L 2 ( P X ) (the space of square integrable functions w.r.t.
Revised: u ∈ R d (cid:9) has dense span in L 2 ( P X ) (the space of square integrable functions w.r.t.

Original: P X the distribution of the input X ).
Revised: P X the distribution of the input X ).

Original: Assumption 3 is inspired by the work Chizat & Bach (2018) on two-layer networks, with certain differences.
Revised: Assumption 3 is inspired by the work Chizat & Bach (2018) on two-layer networks, with certain differences.

Original: Assumptions 3.1 and 3.3 are natural in neural network learning (Cybenko (1989); Chen & Chen (1995)), while we note Chizat & Bach (2018) does not utilize universal approximation.
Revised: Assumptions 3.1 and 3.3 are natural in neural network learning (Cybenko (1989); Chen & Chen (1995)), while we note Chizat & Bach (2018) does not utilize universal approximation.

Original: Similar to Chizat & Bach (2018), Assumption 3.2 is technical and does not seem removable.
Revised: Similar to Chizat & Bach (2018), Assumption 3.2 is technical and does not seem removable.

Original: Note that this assumption speciﬁes the mode of convergence and is not an assumption on the limits ¯ w 1 , ¯ wand ¯ w 3 .
Revised: Note that this assumption speciﬁes the mode of convergence and is not an assumption on the limits ¯ w 1 , ¯ wand ¯ w 3 .

Original: Speciﬁcally conditions (3)-(5) are similar to the convergence assumption in Chizat & Bach (2018).
Revised: Speciﬁcally conditions (3)-(5) are similar to the convergence assumption in Chizat & Bach (2018).

Original: We differ from Chizat & Bach (2018) fundamentally in the essential supremum condition (6).
Revised: We differ from Chizat & Bach (2018) fundamentally in the essential supremum condition (6).

Original: On one hand, this condition helps avoiding the Morse-Sard type condition in Chizat & Bach (2018), which is difﬁcult to verify in general and not simple to generalize to the three-layer case.
Revised: On one hand, this condition helps avoid the Morse-Sard type condition in Chizat & Bach (2018), which is difﬁcult to verify in general and not simple to generalize to the three-layer case.

Original: On the other hand, it turns out to be a natural assumption to make, in light of Remark 9 below.
Revised: On the other hand, it turns out to be a natural assumption to make, in light of Remark 9 below.

Original: We now state the main result of the section.
Revised: We now state the main result of the section.

Original: The proof is in Appendix D.
Revised: The proof is in Appendix D.

Original: Theorem 8.
Revised: Theorem 8.

Original: Consider a neuronal embedding (cid:16) Ω , F , P, (cid:8) w 

Revised: Consider a neuronal embedding (cid:16) Ω , F , P, (cid:8) w

Original: i (cid:9) i =1 , 2 , 3 (cid:17) of (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d. initialization.
Revised: i (cid:9) i =1 , 2 , 3 (cid:17) of (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d. initialization.

Original: Consider the MF limit corresponding to the network (1), such that they are coupled together by the coupling procedure in Section 3.1, under Assumptions 1, 2 and 3.
Revised: Consider the MF limit corresponding to the network (1), such that they are coupled together by the coupling procedure in Section 3.1, under Assumptions 1, 2 and 3.

Original: For simplicity, assume ξ 1 ( · ) =
Revised: For simplicity, assume ξ 1 ( · ) =

Original: ξ 2 ( · ) = 1 .
Revised: ξ 2 ( · ) = 1 .

Original: Further assume either:
Revised: Further assume either:

Original: • (untrained third layer) ξ 3 ( · ) = 0 and w 03 ( C 2 ) (cid:54) = 0 with a positive probability, or • (trained third layer)
Revised: • (untrained third layer) ξ 3 ( · ) = 0 and w 03 ( C 2 ) (cid:54) = 0 with a positive probability, or • (trained third layer)

Original: ξ 3 ( · ) = 1 and L (cid:0) w 01 , w 02 , w 03 (cid:1) <
Revised: ξ 3 ( · ) = 1 and L (cid:0) w 01 , w 02 , w 03 (cid:1) <

Original: E Z [ L ( Y, ϕ 3 (0))] .
Revised: E Z [ L ( Y, ϕ 3 (0))] .

Original: Then the following hold:
Revised: Then the following hold:

Original: • Case 1 (convex loss): If L is convex in the second variable, then
Revised: • Case 1 (convex loss): If L is convex in the second variable, then

Original: [Equation]
Revised: [Equation]

Original: • Case 2 (generic non-negative loss): Suppose that ∂ 2 L ( y, ˆ y ) = 

Revised: • Case 2 (generic non-negative loss): Suppose that ∂ 2 L ( y, ˆ y ) =

Original: implies L ( y, ˆ y )
Revised: implies L ( y, ˆ y )

Original: = 0 .
Revised: = 0 .

Original: If y = y ( x ) is a function of x , then L ( W ( t ))
Revised: If y = y ( x ) is a function of x , then L ( W ( t ))

Original: → 0 as t → ∞ .
Revised: → 0 as t → ∞ .

Original: Remarkably here the theorem allows for non-convex losses.
Revised: Remarkably here the theorem allows for non-convex losses.

Original: A further inspection of the proof shows that we do not rely critically on any convexity property.
Revised: A further inspection of the proof shows that no convexity-based property is used in Case 2 (see, for instance, the high-level proof sketch in Section 4.3); in Case 1, the key steps in the proof are the same, and the convexity of the loss function serves as a convenient technical assumption to handle the arbitrary extra randomness of Y conditional on X .

Original: We also remark that the same proof of global convergence should extend beyond the speciﬁc fully-connected architecture considered here.
Revised: We also remark that the same proof of global convergence should extend beyond the speciﬁc fully-connected architecture considered here.

Original: Similar to previous results on SGD-trained two-layer networks Mei et al.
Revised: Similar to previous results on SGD-trained two-layer networks Mei et al.

Original: ; Chizat & Bach (2018), our current result in the three-layer case is non-quantitative.
Revised: ; Chizat & Bach (2018), our current result in the three-layer case is non-quantitative.

Original: Remark 9 .
Revised: Remark 9 .

Original: Interestingly there is a converse relation between global convergence and the essential supremum condition (6): under the same setting, global convergence is unattainable if condition (6) does not hold.
Revised: Interestingly there is a converse relation between global convergence and the essential supremum condition (6): under the same setting, global convergence is unattainable if condition (6) does not hold.

Original: A similar observation was made in Wojtowytsch (2020) for two-layer ReLU networks.
Revised: A similar observation was made in Wojtowytsch (2020) for two-layer ReLU networks.

Original: A precise statement and its proof can be found in Appendix E.
Revised: A precise statement and its proof can be found in Appendix E.

Original: The following result is straightforward from Theorem 8 and Corollary 4, establishing the optimization efﬁciency of the neural network with SGD.
Revised: The following result is straightforward from Theorem 8 and Corollary 4, establishing the optimization efﬁciency of the neural network with SGD.

Original: Corollary 10.
Revised: Corollary 10.

Original: Consider the neural network (1).
Revised: Consider the neural network (1).

Original: Under the same setting as Theorem 8, in Case 1, 
Revised: Under the same setting as Theorem 8, in Case 1, lim

Original: [Equation]
Revised: [Equation]

Original: in probability, where the limit of the widths is such that min { n 1 , n 2 } − 1 log (max { n 1 , n 2 } ) → 0 .
Revised: in probability, where the limit of the widths is such that min { n 1 , n 2 } − 1 log (max { n 1 , n 2 } ) → 0 .

Original: In Case 2, the same holds with the right-hand side being 0 .
Revised: In Case 2, the same holds with the right-hand side being 0 .

Original: 4. H IGH - LEVEL IDEA OF THE PROOF
Revised: 4. H IGH - LEVEL IDEA OF THE PROOF

Original: We give a high-level discussion of the proof.
Revised: We give a high-level discussion of the proof.

Original: This is meant to provide intuitions and explain the technical crux, so our discussion may simplify and deviate from the actual proof.
Revised: This is meant to provide intuitions and explain the technical crux, so our discussion may simplify and deviate from the actual proof.

Original: Our ﬁrst insight is to look at the second layer’s weight w 2 .
Revised: Our ﬁrst insight is to look at the second layer’s weight w 2 .

Original: At convergence time t = ∞ , we expect to have zero movement and hence, denoting W ( ∞ ) =
Revised: At convergence time t = ∞ , we expect to have zero movement and hence, denoting W ( ∞ ) =

Original: ( ¯ w 1 , ¯ w 2 , ¯ w 3 ) :
Revised: ( ¯ w 1 , ¯ w 2 , ¯ w 3 ) :

Original: [Equation]
Revised: [Equation]

Original: for P -almost every c 1 , c 2 .
Revised: for P -almost every c 1 , c 2 .

Original: Suppose for the moment that we are allowed to make an additional (strong) assumption on the limit ¯ w 1 : supp ( ¯ w 1 ( C 1 ))
Revised: Suppose for the moment that we are allowed to make an additional (strong) assumption on the limit ¯ w 1 : supp ( ¯ w 1 ( C 1 ))

Original: =
Revised: =

Original: R d .
Revised: R d .

Original: It implies that the universal approximation property, described in Assumption 3, holds at t = ∞ ; more speciﬁcally, it implies { ϕ 1 ( (cid:104) ¯ w 1 ( c 1 ) , ·(cid:105) ) :
Revised: It implies that the universal approximation property, described in Assumption 3, holds at t = ∞ ; more speciﬁcally, it implies { ϕ 1 ( (cid:104) ¯ w 1 ( c 1 ) , ·(cid:105) ) :

Original: c 1 ∈ Ω 1 } has dense span in L 2 ( P X ) .
Revised: c 1 ∈ Ω 1 } has dense span in L 2 ( P X ) .

Original: This thus yields
Revised: This thus yields

Revised: E

Original: [Equation]
Revised: [Equation]

Original: for P -almost every x .
Revised: for P -almost every x .

Original: Recalling the deﬁnition of ∆ H 2 , one can then easily show that
Revised: Recalling the deﬁnition of ∆ H 2 , one can then easily show that

Original: [Equation]
Revised: [Equation]

Original: Global convergence follows immediately; for example, in Case 2 of Theorem 8, this is equivalent to that ∂ 2 L ( y ( x ) , ˆ y ( x ; W ( ∞ )))
Revised: Global convergence follows immediately; for example, in Case 2 of Theorem 8, this is equivalent to that ∂ 2 L ( y ( x ) , ˆ y ( x ; W ( ∞ )))

Original: = 

Revised: =

Original: and hence L ( y ( x ) , ˆ y ( x ; W ( ∞ )))
Revised: and hence L ( y ( x ) , ˆ y ( x ; W ( ∞ )))

Original: = 

Revised: =

Original: for P -almost every x .
Revised: for P -almost every x .

Original: In short, the gradient ﬂow structure of the dynamics of w 2 provides a seamless way to obtain global convergence.
Revised: In short, the gradient ﬂow structure of the dynamics of w 2 provides a seamless way to obtain global convergence.

Original: Furthermore there is no critical reliance on convexity.
Revised: Furthermore there is no critical reliance on convexity.

Original: However this plan of attack has a potential ﬂaw in the strong assumption that supp ( ¯ w 1 ( C 1 ))
Revised: However this plan of attack has a potential ﬂaw in the strong assumption that supp ( ¯ w 1 ( C 1 ))

Original: =
Revised: =

Original: R d , i.e. the universal approximation property holds at convergence time.
Revised: R d , i.e. the universal approximation property holds at convergence time.

Original: Indeed there are setups where it is desirable that supp ( ¯ w 1 ( C 1 )) (cid:54) =
Revised: Indeed there are setups where it is desirable that supp ( ¯ w 1 ( C 1 )) (cid:54) =

Original: R d (Mei et al.
Revised: R d (Mei et al.

Original: ; Chizat (2019)); for instance, it is the case where the neural network is to learn some “sparse and spiky” solution, and hence the weight distribution at convergence time, if successfully trained, cannot have full support.
Revised: ; Chizat (2019)); for instance, it is the case where the neural network is to learn some “sparse and spiky” solution, and hence the weight distribution at convergence time, if successfully trained, cannot have full support.

Original: On the other hand, one can entirely expect that if supp ( w 1 (0 , C 1 )) =
Revised: On the other hand, one can entirely expect that if supp ( w 1 (0 , C 1 )) =

Original: R d initially at t = 0 , then supp ( w 1 ( t, C 1 ))
Revised: R d initially at t = 0 , then supp ( w 1 ( t, C 1 ))

Original: =
Revised: =

Original: R d at any ﬁnite t ≥ 0 .
Revised: R d at any ﬁnite t ≥ 0 .

Original: The crux of our proof is to show the latter without assuming supp ( ¯ w 1 ( C 1 ))
Revised: The crux of our proof is to show the latter without assuming supp ( ¯ w 1 ( C 1 ))

Original: =
Revised: =

Original: R d .
Revised: R d .

Original: This task is the more major technical step of the proof.
Revised: This task is the more major technical step of the proof.

Original: To that end, we ﬁrst show that there exists a mapping ( t, u ) (cid:55)→ M ( t, u ) that maps from ( t, w 1 (0 , c 1 ))
Revised: To that end, we ﬁrst show that there exists a mapping ( t, u ) (cid:55)→ M ( t, u ) that maps from ( t, w 1 (0 , c 1 ))

Original: =
Revised: =

Original: ( t, u ) to w 1 ( t, c 1 ) via a careful measurability argument.
Revised: ( t, u ) to w 1 ( t, c 1 ) via a careful measurability argument.

Original: This argument rests on a scheme that exploits the symmetry in the network evolution.
Revised: This argument rests on a scheme that exploits the symmetry in the network evolution.

Original: Furthermore the map M is shown to be continuous.
Revised: Furthermore the map M is shown to be continuous.

Original: The desired conclusion then follows from an algebraic topology argument that the map M preserves a homotopic structure through time.
Revised: The desired conclusion then follows from an algebraic topology argument that the map M preserves a homotopic structure through time.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: The MF literature is fairly recent.
Revised: The MF literature is fairly recent.

Original: A long line of works (Nitanda & Suzuki (2017); Mei et al.
Revised: A long line of works (Nitanda & Suzuki (2017); Mei et al.

Original: Chizat & Bach (2018); Rotskoff & Vanden-Eijnden (2018); Sirignano & Spiliopoulos (2018);
Revised: Chizat & Bach (2018); Rotskoff & Vanden-Eijnden (2018); Sirignano & Spiliopoulos (2018);

Original: Wei et al.
Revised: Wei et al.

Original: ; Javanmard et al.
Revised: ; Javanmard et al.

Original: Mei et al.
Revised: Mei et al.

Original: ; Shevchenko & Mondelli (2019); Wojtowytsch (2020)) have focused mainly on two-layer neural networks, taking an interacting particle system approach to describe the MF limiting dynamics as Wasserstein gradient ﬂows.
Revised: ; Shevchenko & Mondelli (2019); Wojtowytsch (2020)) have focused mainly on two-layer neural networks, taking an interacting particle system approach to describe the MF limiting dynamics as Wasserstein gradient ﬂows.

Original: The three works Nguyen (2019); Araújo et al.
Revised: The three works Nguyen (2019); Araújo et al.

Original: ; Sirignano & Spiliopoulos (2019) independently develop different formulations for the MF limit in multilayer neural networks, under different assumptions.
Revised: ; Sirignano & Spiliopoulos (2019) independently develop different formulations for the MF limit in multilayer neural networks, under different assumptions.

Revised: These works take perspectives that are different from ours.

Revised: In particular, while the central object in Nguyen (2019) is a new abstract representation of each individual neuron, our neuronal embedding idea instead takes a keen view on a whole ensemble of neurons.

Original: The latter two works Araújo et al. 
Revised: Likewise our idea is also distant from Araújo et al. ; Sirignano & Spiliopoulos (2019): the central objects in Araújo et al. (2019) are paths over the weights across layers; those in Sirignano & Spiliopoulos (2019) are time-dependent functions of the initialization, which are simpliﬁed upon i.i.d. initializations.

Revised: The result of our perspective is a neuronal embedding framework that allows one to describe the MF limit in a clean and rigorous manner.

Original:  ; Sirignano & Spiliopoulos  (2019) develop MF formulations that hold exclusively for i.i.d.
Revised: In particular, it avoids extra assumptions made in Araújo et al. ; Sirignano & Spiliopoulos (2019): unlike our work, Araújo et al. (2019) assumes untrained ﬁrst and last layers and requires non-trivial technical tools; Sirignano & Spiliopoulos (2019) takes an unnatural sequential limit n 1 → ∞ before n 2 → ∞ and proves a non-quantitative result, unlike Theorem 3 which only requires sufﬁciently large min { n 1 , n 2 } .

Original: initializations.
Revised: We note that Theorem 3 can be extended to general multilayer networks using the neuronal embedding idea. The advantages of our framework come from the fact that while MF formulations in Araújo et al. ; Sirignano & Spiliopoulos (2019) are speciﬁc to and exploit i.i.d. initializations, our formulation does not.

Original: As shown in Araújo et al.
Revised: Remarkably as shown in Araújo et al.

Original: (2019), when there are more than three layers and no biases, i.i.d.
Revised: (2019), when there are more than three layers and no biases, i.i.d.

Original: initializations cause a strong simplifying effect such that the evolution of weights in one layer becomes independent of all other layers.
Revised: initializations lead to a certain simplifying effect on the MF limit.

Original: On the other hand, our frameworkis closer to the spirit of the heuristic work Nguyen (2019): the framework supports non-i.i.d.
Revised: On the other hand, our framework supports non-i.i.d.

Original: initializations which avoid the simplifying effect, as long as there exist suitable neuronal embeddings.
Revised: initializations which avoid the simplifying effect, as long as there exist suitable neuronal embeddings.

Original: Although our global convergence result  is proven in the context of i.i.d.
Revised: Although our global convergence result in Theorem 8 is proven in the context of i.i.d.

Original: initializations for three-layer networks, we believe non-i.i.d.
Revised: initializations for three-layer networks, we believe non-i.i.d.

Original: initializations could provide useful insights in the general case and leave that to another report.
Revised: initializations could provide useful insights in the general case and leave that to another report.

Original: Our Theorem 3 provides a quantitative bound and only requires sufﬁciently large min { n 1 , n 2 } , whereas Sirignano & Spiliopoulos (2019) is nonquantitative and takes an unnatural sequential limit n 1 → ∞ before n 2 → ∞ .
Revised: In this aspect, our framework follows closely the spirit of the work Nguyen (2019), whose MF formulation is also not speciﬁc to i.i.d.

Original: The theorem does not assume untrained input and output weights, compared to Araújo et al.
Revised: initializations.

Original: We note that Theorem 3 can be extended to general multilayer networks using the neuronal embedding idea.
Revised: Yet though similar in the spirit, Nguyen (2019) develops a heuristic formalism and does not prove global convergence.

Original: Global convergence in the two-layer case with convex losses has enjoyed multiple efforts with a lot of new and interesting results (Mei et al.
Revised: Global convergence in the two-layer case with convex losses has enjoyed multiple efforts with a lot of new and interesting results (Mei et al.

Original: (2018); Chizat & Bach (2018); Javanmard et al.
Revised: (2018); Chizat & Bach (2018); Javanmard et al.

Original: Rotskoff et al.
Revised: Rotskoff et al.

Original: ; Wei et al.
Revised: ; Wei et al.

Original: Our work is the ﬁrst to establish a global convergence guarantee for SGD-trained three-layer networks in the MF regime.
Revised: Our work is the ﬁrst to establish a global convergence guarantee for SGD-trained three-layer networks in the MF regime.

Original: Our proof sends a new message that the crucial factor is not necessarily convexity, but rather that the whole learning trajectory maintains the universal approximation property of the function class represented by the ﬁrst layer’s neurons, together with the gradient ﬂow structure of the second layer’s weights.
Revised: Our proof sends a new mes- sage that the crucial factor is not necessarily convexity, but rather that the whole learning trajectory maintains the universal approximation property of the function class represented by the ﬁrst layer’s neurons, together with the gradient ﬂow structure of the second layer’s weights.

Original: As a remark, our approach can also be applied to prove a similar global convergence guarantee for two-layer networks, removing the convex loss assumption in previous works.
Revised: As a remark, our approach can also be applied to prove a similar global convergence guarantee for two-layer networks, removing the convex loss assumption in previous works.

Original: The recent work Lu et al.
Revised: The recent work Lu et al.

Original: (2020) on a MF resnet model (a composition of many two-layer MF networks) and a recent update of Sirignano &
Revised: (2020) on a MF resnet model (a composition of many two-layer MF networks) and a recent update of Sirignano &

Original: Spiliopoulos (2019) essentially establish conditions of stationary points to be global optima.
Revised: Spiliopoulos (2019) essentially establish conditions of stationary points to be global optima.

Original: They however require the strong assumption of full support of the limit point.
Revised: They however require the strong assumption of full support of the limit point.

Original: As explained in Section 4.3, we analyze the training dynamics without such assumption and in fact allow it to be violated.
Revised: As explained in Section 4.3, we analyze the training dynamics without such assumption and in fact allow it to be violated.

Original: Our global convergence result is non-quantitative.
Revised: Our global convergence result is non-quantitative.

Original: An important, highly challenging future direction is to develop a quantitative version of global convergence; previous works on two-layer networks
Revised: An important, highly challenging future direction is to develop a quantitative version of global convergence; previous works on two-layer networks

Original: Javanmard et al.
Revised: Javanmard et al.

Original: ; Wei et al.
Revised: ; Wei et al.

Original: ; Rotskoff et al.
Revised: ; Rotskoff et al.

Original: ; Chizat (2019) have done so under sophisticated modiﬁcations of the architecture and training algorithms.
Revised: ; Chizat (2019) have done so under sophisticated modiﬁcations of the architecture and training algorithms.

Original: Finally we remark that our insights on global convergence here can be applied to prove similar results for more general multilayer networks.
Revised: Finally we remark that our insights on global convergence here can be applied to prove similar results for more general multilayer networks.

Original: The details are deferred to another report.
Revised: The details are deferred to another report.

Original: A N OTATIONAL PRELIMINARIES
Revised: A N OTATIONAL PRELIMINARIES

Original: For a real-valued random variable Z deﬁned on a probability space (Ω , F , P ) , we recall
Revised: For a real-valued random variable Z deﬁned on a probability space (Ω , F , P ) , we recall

Original: [Equation]
Revised: [Equation]

Original: We also introduce some convenient deﬁnitions which we use throughout the appendices.
Revised: We also introduce some convenient deﬁnitions which we use throughout the appendices.

Original: For a set of neural network’s parameter W , we deﬁne
Revised: For a set of neural network’s parameter W , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Similarly for a set of MF parameters W
Revised: Similarly for a set of MF parameters W

Original: [Equation]
Revised: [Equation]

Original: we deﬁne:
Revised: we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: For two sets of neural network’s parameters W (cid:48) , W (cid:48)(cid:48)
Revised: For two sets of neural network’s parameters W (cid:48) , W (cid:48)(cid:48)

Original: [Equation]
Revised: [Equation]

Original: we deﬁne their distance:
Revised: we deﬁne their distance:

Original: [Equation]
Revised: [Equation]

Original: Similarly for two sets of MF parameters W (cid:48) , W (cid:48)(cid:48)
Revised: Similarly for two sets of MF parameters W (cid:48) , W (cid:48)(cid:48)

Original: [Equation]
Revised: [Equation]

Original: we deﬁne their distance:
Revised: we deﬁne their distance:

Original: [Equation]
Revised: [Equation]

Original: B E XISTENCE AND UNIQUENESS OF THE SOLUTION TO MF ODE S
Revised: B E XISTENCE AND UNIQUENESS OF THE SOLUTION TO MF ODE S

Original: We ﬁrst collect some a priori estimates
Revised: We ﬁrst collect some a priori estimates

Original: [Equation]
Revised: [Equation]

Original: Lemma 11.
Revised: Lemma 11.

Original: Under Assumption
Revised: Under Assumption

Original: [Equation]
Revised: [Equation]

Original: consider a solution W to the MF ODEs with initialization W (0) such that (cid:57) W (cid:57) 0 < ∞ .
Revised: consider a solution W to the MF ODEs with initialization W (0) such that (cid:57) W (cid:57) 0 < ∞ .

Original: If this solution exists
Revised: If this solution exists

Original: [Equation]
Revised: [Equation]

Original: it satisﬁes the following a priori bounds, for any
Revised: it satisﬁes the following a priori bounds, for any

Original: T ≥ 0 :
Revised: T ≥ 0 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: T
Revised: T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , and consequently, (cid:57) W (cid:57) T ≤ 1 + max { K 0 , 2 ( T ) , K 0 , 3 ( T ) } .
Revised: , and consequently, (cid:57) W (cid:57) T ≤ 1 + max { K 0 , 2 ( T ) , K 0 , 3 ( T ) } .

Original: Proof.
Revised: Proof.

Original: The bounds can be obtained easily by bounding the respective initializations and update quantities separately.
Revised: The bounds can be obtained easily by bounding the respective initializations and update quantities separately.

Original: In particular,
Revised: In particular,

Original: [Equation]
Revised: [Equation]

Original: Inspired by the a priori bounds in Lemma
Revised: Inspired by the a priori bounds in Lemma

Original: [Equation]
Revised: [Equation]

Original: given an arbitrary terminal time T and the initialization
Revised: given an arbitrary terminal time T and the initialization

Original: W (0)
Revised: W (0)

Original: [Equation]
Revised: [Equation]

Original: let us consider:
Revised: let us consider:

Original: • for a tuple ( a, b ) ∈ R 2 ≥ 0 , a space W T ( a, b ) of W (cid:48)
Revised: • for a tuple ( a, b ) ∈ R 2 ≥ 0 , a space W T ( a, b ) of W (cid:48)

Original: [Equation]
Revised: [Equation]

Original: (cid:48) ( t ))
Revised: (cid:48) ( t ))

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: ( w (cid:48) 1 ( t, · ) , w (cid:48) 2 ( t, · , · ) , w (cid:48) 

Revised: ( w (cid:48) 1 ( t, · ) , w (cid:48) 2 ( t, · , · ) , w (cid:48)

Original: ( t
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: · )) t ≤ T such that
Revised: · )) t ≤ T such that

Original: [Equation]
Revised: [Equation]

Original: where w (cid:48) 1 : R ≥ 0 × Ω 1 → R d , w (cid:48) 2 : R ≥ 

Revised: where w (cid:48) 1 : R ≥ 0 × Ω 1 → R d , w (cid:48) 2 : R ≥

Original: × Ω 1 × Ω 2 (cid:55)→ R , w (cid:48) 3 : R ≥ 

Revised: × Ω 1 × Ω 2 (cid:55)→ R , w (cid:48) 3 : R ≥

Original: × Ω 3 (cid:55)→ R , • for a tuple ( a, b ) ∈ R 2 ≥ 0 and W (0) , a space W + T ( a, b, W (0)) of W (cid:48) ∈ W T ( a, b ) such that W (cid:48) (0) =
Revised: × Ω 3 (cid:55)→ R , • for a tuple ( a, b ) ∈ R 2 ≥ 0 and W (0) , a space W + T ( a, b, W (0)) of W (cid:48) ∈ W T ( a, b ) such that W (cid:48) (0) =

Original: W (0) additionally (and hence every W (cid:48) in this space shares the same initialization W (0) ).
Revised: W (0) additionally (and hence every W (cid:48) in this space shares the same initialization W (0) ).

Original: We equip the spaces with the metric (cid:107) W (cid:48) − W (cid:48)(cid:48) (cid:107) T .
Revised: We equip the spaces with the metric (cid:107) W (cid:48) − W (cid:48)(cid:48) (cid:107) T .

Original: It is easy to see that both spaces are complete.
Revised: It is easy to see that both spaces are complete.

Original: Note that Lemma 11 implies, under Assumption 1 and (cid:57) W (cid:57) 0 < ∞ , we have any MF solution W , if exists, is in W T ( (cid:57) W (cid:57) 

Revised: Note that Lemma 11 implies, under Assumption 1 and (cid:57) W (cid:57) 0 < ∞ , we have any MF solution W , if exists, is in W T ( (cid:57) W (cid:57)

Original: + K 0 , 2 ( T ) , (cid:57) W (cid:57) 

Revised: + K 0 , 2 ( T ) , (cid:57) W (cid:57)

Original: + K 0 , 3 ( T )) .
Revised: + K 0 , 3 ( T )) .

Original: For the proof of Theorem 1, we work mainly with W + T ( (cid:57) W (cid:57) 

Revised: For the proof of Theorem 1, we work mainly with W + T ( (cid:57) W (cid:57)

Original: + K 0 , 2 ( T ) ,
Revised: + K 0 , 2 ( T ) ,

Original: [Equation]
Revised: [Equation]

Original: T )
Revised: T )

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: 0)) , although several intermediate lemmas are proven in more generality for other uses.
Revised: 0)) , although several intermediate lemmas are proven in more generality for other uses.

Original: Lemma 12.
Revised: Lemma 12.

Original: Under Assumption
Revised: Under Assumption

Original: [Equation]
Revised: [Equation]

Original: for T ≥
Revised: for T ≥

Original: [Equation]
Revised: [Equation]

Original: any W (cid:48)
Revised: any W (cid:48)

Original: [Equation]
Revised: [Equation]

Original: W (cid:48)(cid:48) ∈ W T ( a, b ) and almost every z ∼ P :
Revised: W (cid:48)(cid:48) ∈ W T ( a, b ) and almost every z ∼ P :

Original: [Equation]
Revised: [Equation]

Original: where K a,b ≥ 1 is a generic constant that grows polynomially with a and b .
Revised: where K a,b ≥ 1 is a generic constant that grows polynomially with a and b .

Original: Proof.
Revised: Proof.

Original: The ﬁrst bound is easy to see:
Revised: The ﬁrst bound is easy to see:

Original: [Equation]
Revised: [Equation]

Original: We prove the second bound
Revised: We prove the second bound

Original: [Equation]
Revised: [Equation]

Original: invoking Assumption 1:
Revised: invoking Assumption 1:

Original: [Equation]
Revised: [Equation]

Original: which yields by the fact W (cid:48) ∈ W T ( a, b ) :
Revised: which yields by the fact W (cid:48) ∈ W T ( a, b ) :

Original: [Equation]
Revised: [Equation]

Original: Consequently, we have:
Revised: Consequently, we have:

Original: [Equation]
Revised: [Equation]

Original: ( t ))
Revised: ( t ))

Original: | ≤ K | w (cid:48) 3 ( t, C 2 ) | | ϕ 2 ( H 2 ( x, C 2 ; W (cid:48) ( t )))
Revised: | ≤ K | w (cid:48) 3 ( t, C 2 ) | | ϕ 2 ( H 2 ( x, C 2 ; W (cid:48) ( t )))

Original: − ϕ 2 ( H 2 ( x, C 2 ; W (cid:48)(cid:48) ( t )))
Revised: − ϕ 2 ( H 2 ( x, C 2 ; W (cid:48)(cid:48) ( t )))

Original: | + K | w (cid:48) 

Revised: | + K | w (cid:48)

Original: ( t, C 2 ) − w (cid:48)(cid:48) 3 ( t, C 2 ) |
Revised: ( t, C 2 ) − w (cid:48)(cid:48) 3 ( t, C 2 ) |

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: (cid:48) ( t )
Revised: (cid:48) ( t )

Original: [Equation]
Revised: [Equation]

Original: t ))
Revised: t ))

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: (cid:48)(cid:48)
Revised: (cid:48)(cid:48)

Original: [Equation]
Revised: [Equation]

Original: T , | ∂ 2 L ( y, ˆ y ( x
Revised: T , | ∂ 2 L ( y, ˆ y ( x

Original: [Equation]
Revised: [Equation]

Original: W (cid:48) ( t )))
Revised: W (cid:48) ( t )))

Original: − ∂ 2 L ( y
Revised: − ∂ 2 L ( y

Original: [Equation]
Revised: [Equation]

Original: ˆ y ( x
Revised: ˆ y ( x

Original: [Equation]
Revised: [Equation]

Original: W (cid:48)(cid:48) ( t )))
Revised: W (cid:48)(cid:48) ( t )))

Original: | ≤ K | ˆ y ( x
Revised: | ≤ K | ˆ y ( x

Original: [Equation]
Revised: [Equation]

Original: W (cid:48) ( t ))
Revised: W (cid:48) ( t ))

Original: − ˆ y
Revised: − ˆ y

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: W (cid:48)(cid:48) ( t ))
Revised: W (cid:48)(cid:48) ( t ))

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: t ))
Revised: t ))

Original: [Equation]
Revised: [Equation]

Original: which then yield the third and fourth bounds by the fact W (cid:48)
Revised: which then yield the third and fourth bounds by the fact W (cid:48)

Original: , W (cid:48)(cid:48)
Revised: , W (cid:48)(cid:48)

Original: ∈ W T ( a, b ) .
Revised: ∈ W T ( a, b ) .

Original: Using these bounds, we obtain the last bound:
Revised: Using these bounds, we obtain the last bound:

Original: [Equation]
Revised: [Equation]

Original: from which the last bound follows.
Revised: from which the last bound follows.

Original: To prove Theorem 1, for a given W (0) , we deﬁne a mapping F W (0) that maps from W (cid:48) =
Revised: To prove Theorem 1, for a given W (0) , we deﬁne a mapping F W (0) that maps from W (cid:48) =

Original: ( w (cid:48) 1 , w (cid:48) 2 , w (cid:48) 3 ) ∈ W T ( a, b ) to F W (0) ( W (cid:48) ) =
Revised: ( w (cid:48) 1 , w (cid:48) 2 , w (cid:48) 3 ) ∈ W T ( a, b ) to F W (0) ( W (cid:48) ) =

Original: ¯
Revised: ¯

Original: [Equation]
Revised: [Equation]

Original: ¯
Revised: ¯

Original: [Equation]
Revised: [Equation]

Original: by ¯
Revised: by ¯

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Notice that the right-hand sides do not involve ¯ W (cid:48) .
Revised: Notice that the right-hand sides do not involve ¯ W (cid:48) .

Original: Note that the MF ODEs’ solution, initialized at
Revised: Note that the MF ODEs’ solution, initialized at

Original: W (0) , is a ﬁxed point of this mapping.
Revised: W (0) , is a ﬁxed point of this mapping.

Original: We establish the following estimates for this mapping.
Revised: We establish the following estimates for this mapping.

Original: Lemma 13.
Revised: Lemma 13.

Original: Under Assumption 1, for T ≥ 0 , any initialization W (0) and any W (cid:48) , W (cid:48)(cid:48) ∈
Revised: Under Assumption 1, for T ≥ 0 , any initialization W (0) and any W (cid:48) , W (cid:48)(cid:48) ∈

Original: W T ( a, b ) ,
Revised: W T ( a, b ) ,

Original: [Equation]
Revised: [Equation]

Original: and consequently, if in addition W (cid:48) (0) =
Revised: and consequently, if in addition W (cid:48) (0) =

Original: W (cid:48)(cid:48) (0) (not necessarily equal W (0) ), then ess - sup sup t ≤ T |
Revised: W (cid:48)(cid:48) (0) (not necessarily equal W (0) ), then ess - sup sup t ≤ T |

Original: [Equation]
Revised: [Equation]

Original: w (cid:48)
Revised: w (cid:48)

Original: [Equation]
Revised: [Equation]

Original: ( t, C
Revised: ( t, C

Original: [Equation]
Revised: [Equation]

Original: ess - sup sup t
Revised: ess - sup sup t

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: w (cid:48)
Revised: w (cid:48)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ds
Revised: ds

Original: [Equation]
Revised: [Equation]

Original: ess
Revised: ess

Original: [Equation]
Revised: [Equation]

Original: in which ¯ W (cid:48)
Revised: in which ¯ W (cid:48)

Original: = ( ¯ w (cid:48) 1 , ¯ w (cid:48) 2 , ¯ w (cid:48) 3 ) =
Revised: = ( ¯ w (cid:48) 1 , ¯ w (cid:48) 2 , ¯ w (cid:48) 3 ) =

Original: F W (0) ( W (cid:48) ) , ¯ W (cid:48)(cid:48) =
Revised: F W (0) ( W (cid:48) ) , ¯ W (cid:48)(cid:48) =

Original: ( ¯ w (cid:48)(cid:48) 1 , ¯ w (cid:48)(cid:48) 2 , ¯ w (cid:48)(cid:48) 3 ) =
Revised: ( ¯ w (cid:48)(cid:48) 1 , ¯ w (cid:48)(cid:48) 2 , ¯ w (cid:48)(cid:48) 3 ) =

Original: F W (0) ( W (cid:48)(cid:48) ) and K a,b ≥ 1 is a generic constant that grows polynomially with a and b .
Revised: F W (0) ( W (cid:48)(cid:48) ) and K a,b ≥ 1 is a generic constant that grows polynomially with a and b .

Original: Proof.
Revised: Proof.

Original: From Assumption 1 and the fact W (cid:48) , W (cid:48)(cid:48)
Revised: From Assumption 1 and the fact W (cid:48) , W (cid:48)(cid:48)

Original: ∈ W T ( a, b ) , we get:
Revised: ∈ W T ( a, b ) , we get:

Original: [Equation]
Revised: [Equation]

Original: ( s ))
Revised: ( s ))

Original: | ≤ K E Z [ | ∂ 2 L ( Y, ˆ y ( X ; W (cid:48) ( s )))
Revised: | ≤ K E Z [ | ∂ 2 L ( Y, ˆ y ( X ; W (cid:48) ( s )))

Original: − ∂ 2 L ( Y, ˆ y ( X ; W (cid:48)(cid:48) ( s )))
Revised: − ∂ 2 L ( Y, ˆ y ( X ; W (cid:48)(cid:48) ( s )))

Original: | ] +
Revised: | ] +

Original: K E Z
Revised: K E Z

Original: [ | H 3 ( X ; W (cid:48) ( s ))
Revised: [ | H 3 ( X ; W (cid:48) ( s ))

Original: − H 3 ( X ; W (cid:48)(cid:48) ( s )) | ] + K E Z [ | H 2 ( X, C 2 ; W (cid:48) ( s ))
Revised: − H 3 ( X ; W (cid:48)(cid:48) ( s )) | ] + K E Z [ | H 2 ( X, C 2 ; W (cid:48) ( s ))

Original: − H 2 ( X, C 2 ; W (cid:48)(cid:48) ( s )) | ] , | ∆ 2 ( C 1 , C 2 ; W (cid:48) ( s ))
Revised: − H 2 ( X, C 2 ; W (cid:48)(cid:48) ( s )) | ] , | ∆ 2 ( C 1 , C 2 ; W (cid:48) ( s ))

Original: − ∆ 2 ( C 1 , C 2 ; W (cid:48)(cid:48) ( s ))
Revised: − ∆ 2 ( C 1 , C 2 ; W (cid:48)(cid:48) ( s ))

Original: | ≤ K a,b | w (cid:48) 1 ( s, C 1 ) − w (cid:48)(cid:48) 

Revised: | ≤ K a,b | w (cid:48) 1 ( s, C 1 ) − w (cid:48)(cid:48)

Original: ( s, C 1 )
Revised: ( s, C 1 )

Original: | + K (cid:12)(cid:12)
Revised: | + K (cid:12)(cid:12)

Original: E Z (cid:2)
Revised: E Z (cid:2)

Original: ∆ H 2 ( Z, C 2 ; W (cid:48) ( s )) − ∆ H 2 ( Z, C 2 ; W (cid:48)(cid:48) ( s ))
Revised: ∆ H 2 ( Z, C 2 ; W (cid:48) ( s )) − ∆ H 2 ( Z, C 2 ; W (cid:48)(cid:48) ( s ))

Original: (cid:3)(cid:12)(cid:12) , | ∆ 1 ( C 1 ; W (cid:48) ( s )) − ∆ 1 ( C 1 ; W (cid:48)(cid:48) ( s ))
Revised: (cid:3)(cid:12)(cid:12) , | ∆ 1 ( C 1 ; W (cid:48) ( s )) − ∆ 1 ( C 1 ; W (cid:48)(cid:48) ( s ))

Original: | ≤ K a,b E Z (cid:2)(cid:12)(cid:12)
Revised: | ≤ K a,b E Z (cid:2)(cid:12)(cid:12)

Original: ∆ H 2 ( Z, C 2 ; W (cid:48) ( s )) − ∆ H 2 ( Z, C 2 ; W (cid:48)(cid:48) ( s )) (cid:12)(cid:12)(cid:3)
Revised: ∆ H 2 ( Z, C 2 ; W (cid:48) ( s )) − ∆ H 2 ( Z, C 2 ; W (cid:48)(cid:48) ( s )) (cid:12)(cid:12)(cid:3)

Original: [Equation]
Revised: [Equation]

Original: from which the ﬁrst three estimates then follow, in light of Lemma 12.
Revised: from which the ﬁrst three estimates then follow, in light of Lemma 12.

Original: The last three estimates then follow from the fact that ¯ W (cid:48) (0) =
Revised: The last three estimates then follow from the fact that ¯ W (cid:48) (0) =

Original: ¯ W (cid:48)(cid:48) (0) and Assumption 1; for instance,
Revised: ¯ W (cid:48)(cid:48) (0) and Assumption 1; for instance,

Original: [Equation]
Revised: [Equation]

Original: We are now ready to prove Theorem
Revised: We are now ready to prove Theorem

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: We will use a Picard-type iteration.
Revised: We will use a Picard-type iteration.

Original: To lighten notations:
Revised: To lighten notations:

Original: [Equation]
Revised: [Equation]

Original: Since (cid:57) W (cid:57) 0 ≤ K by assumption, we have (cid:57) W (cid:57) 0 + K 0 , 2 ( T ) + K
Revised: Since (cid:57) W (cid:57) 0 ≤ K by assumption, we have (cid:57) W (cid:57) 0 + K 0 , 2 ( T ) + K

Original: [Equation]
Revised: [Equation]

Original: 3 ( T ) ≤ K
Revised: 3 ( T ) ≤ K

Original: [Equation]
Revised: [Equation]

Original: Recall that W + T is complete.
Revised: Recall that W + T is complete.

Original: For an arbitrary T > 0 , consider W (cid:48) , W (cid:48)(cid:48)
Revised: For an arbitrary T > 0 , consider W (cid:48) , W (cid:48)(cid:48)

Original: ∈ W + T .
Revised: ∈ W + T .

Original: Lemma 13 yields:
Revised: Lemma 13 yields:

Original: [Equation]
Revised: [Equation]

Original: Note that F maps to W + T under Assumption 1 by the same argument as Lemma 11.
Revised: Note that F maps to W + T under Assumption 1 by the same argument as Lemma 11.

Original: Hence we are allowed to iterating this inequality and get, for an arbitrary T > 0 ,
Revised: Hence we are allowed to iterating this inequality and get, for an arbitrary T > 0 ,

Original: [Equation]
Revised: [Equation]

Original: By substituting W (cid:48)(cid:48)
Revised: By substituting W (cid:48)(cid:48)

Original: =
Revised: =

Original: F ( W (cid:48) ) , we have:
Revised: F ( W (cid:48) ) , we have:

Original: [Equation]
Revised: [Equation]

Original: Hence as k → ∞ , F ( k ) ( W (cid:48) ) converges to a limit in W + T , which is a ﬁxed point of F .
Revised: Hence as k → ∞ , F ( k ) ( W (cid:48) ) converges to a limit in W + T , which is a ﬁxed point of F .

Original: The uniqueness of a ﬁxed point follows from the above estimate, since if W (cid:48) and W (cid:48)(cid:48) are ﬁxed points then
Revised: The uniqueness of a ﬁxed point follows from the above estimate, since if W (cid:48) and W (cid:48)(cid:48) are ﬁxed points then

Original: [Equation]
Revised: [Equation]

Original: while one can take k arbitrarily large.
Revised: while one can take k arbitrarily large.

Original: This proves that the solution exists and is unique on t ∈ [0 , T ] .
Revised: This proves that the solution exists and is unique on t ∈ [0 , T ] .

Original: Since T is arbitrary, we have existence and uniqueness of the solution on the time interval [0 , ∞ ) .
Revised: Since T is arbitrary, we have existence and uniqueness of the solution on the time interval [0 , ∞ ) .

Original: C C ONNECTION BETWEEN THE NEURAL NET AND ITS MF LIMIT : PROOFS FOR S ECTIONC. P ROOF OF T HEOREM
Revised: C C ONNECTION BETWEEN THE NEURAL NET AND ITS MF LIMIT : PROOFS FOR S ECTIONC. P ROOF OF T HEOREM

Original: We construct an auxiliary trajectory, which we call the particle ODEs :
Revised: We construct an auxiliary trajectory, which we call the particle ODEs :

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: X 
Revised: X 

Original: [Equation]
Revised: [Equation]

Original: in which j 1 = 1 , ..., n 1 , j 2 = 1 , ..., n 2 , ˜ W ( t )
Revised: in which j 1 = 1 , ..., n 1 , j 2 = 1 , ..., n 2 , ˜ W ( t )

Original: =
Revised: =

Original: ( ˜ w 1 ( t, · ) , ˜ w 2 ( t, · , · ) , ˜ w 3 ( t, · ))
Revised: ( ˜ w 1 ( t, · ) , ˜ w 2 ( t, · , · ) , ˜ w 3 ( t, · ))

Original: [Equation]
Revised: [Equation]

Original: and t ∈ R ≥ 0 .
Revised: and t ∈ R ≥ 0 .

Original: We specify the initialization ˜ W (0) :
Revised: We specify the initialization ˜ W (0) :

Original: ˜
Revised: ˜

Original: w
Revised: w

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: , ˜
Revised: , ˜

Original: [Equation]
Revised: [Equation]

Original: 0 , j 1 , j 2 ) =
Revised: 0 , j 1 , j 2 ) =

Original: w 0 2 ( C 1 ( j 1 ) , C 2 ( j 2 )) and ˜ w 3 (0 , j 3 ) =
Revised: w 0 2 ( C 1 ( j 1 ) , C 2 ( j 2 )) and ˜ w 3 (0 , j 3 ) =

Original: w 0 3 ( C 2 ( j 2 )) .
Revised: w 0 3 ( C 2 ( j 2 )) .

Original: That is, it shares the same initialization with the neural network one W (0) , and hence is coupled with the neural network and the MF ODEs.
Revised: That is, it shares the same initialization with the neural network one W (0) , and hence is coupled with the neural network and the MF ODEs.

Original: Roughly speaking, the particle ODEs are continuous-time trajectories of ﬁnitely many neurons, averaged over the data distribution.
Revised: Roughly speaking, the particle ODEs are continuous-time trajectories of ﬁnitely many neurons, averaged over the data distribution.

Original: We note that ˜ W ( t ) is random for all t ∈ R ≥ 

Revised: We note that ˜ W ( t ) is random for all t ∈ R ≥

Original: due to the randomness of { C i ( j i ) } i =1 , 2 .
Revised: due to the randomness of { C i ( j i ) } i =1 , 2 .

Original: The existence and uniqueness of the solution to the particle ODEs follows from the same proof as in Theorem 1, which we shall not repeat here.
Revised: The existence and uniqueness of the solution to the particle ODEs follows from the same proof as in Theorem 1, which we shall not repeat here.

Original: We equip ˜ W ( t ) with the norm
Revised: We equip ˜ W ( t ) with the norm

Original: [Equation]
Revised: [Equation]

Original: One can also deﬁne the measures D T (cid:16) W, ˜ W (cid:17) and D T (cid:16) ˜ W, W (cid:17) similar to Eq.
Revised: One can also deﬁne the measures D T (cid:16) W, ˜ W (cid:17) and D T (cid:16) ˜ W, W (cid:17) similar to Eq.

Original: [Equation]
Revised: [Equation]

Original: w 1 ( (cid:98) t/(cid:15) (cid:99) , j 1 ) − ˜ w 1 ( t, C 1 ( j 1 )) | , | w 2 ( (cid:98) t/(cid:15) (cid:99) , j 1 , j 2 ) − ˜ w 2 ( t, C 1 ( j 1 ) , C 2 ( j 2 ))
Revised: w 1 ( (cid:98) t/(cid:15) (cid:99) , j 1 ) − ˜ w 1 ( t, C 1 ( j 1 )) | , | w 2 ( (cid:98) t/(cid:15) (cid:99) , j 1 , j 2 ) − ˜ w 2 ( t, C 1 ( j 1 ) , C 2 ( j 2 ))

Original: | , | w 3 ( (cid:98) t/(cid:15) (cid:99) , j 2 ) − ˜ w 3 ( t, C 2 ( j 2 ))
Revised: | , | w 3 ( (cid:98) t/(cid:15) (cid:99) , j 2 ) − ˜ w 3 ( t, C 2 ( j 2 ))

Original: | : t ≤ T, j 1 ≤ n 1 , j 2 ≤ n 2 (cid:9) .
Revised: | : t ≤ T, j 1 ≤ n 1 , j 2 ≤ n 2 (cid:9) .

Original: We have the following results:
Revised: We have the following results:

Original: Theorem 14.
Revised: Theorem 14.

Original: Under the same setting as Theorem 3, for any δ > 0 , with probability at least 1 − δ ,
Revised: Under the same setting as Theorem 3, for any δ > 0 , with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: in which n min = min { n 1 , n 2 } , n max = max { n 1 , n 2 } , and K T = K (cid:0) 

Revised: in which n min = min { n 1 , n 2 } , n max = max { n 1 , n 2 } , and K T = K (cid:0)

Original: + T K (cid:1) .
Revised: + T K (cid:1) .

Original: Theorem 15.
Revised: Theorem 15.

Original: Under the same setting as Theorem 3, for any δ > 0 and (cid:15) ≤ 1 , with probability at least 1 − δ ,
Revised: Under the same setting as Theorem 3, for any δ > 0 and (cid:15) ≤ 1 , with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: in which K T = K (cid:0) 1 + T K (cid:1) .
Revised: in which K T = K (cid:0) 1 + T K (cid:1) .

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: Using the fact
Revised: Using the fact

Original: [Equation]
Revised: [Equation]

Original: the thesis is immediate from Theorems 14 and 15.
Revised: the thesis is immediate from Theorems 14 and 15.

Original: C. P ROOF OF T HEOREMS 14 AND
Revised: C. P ROOF OF T HEOREMS 14 AND

Original: Proof of Theorem 14.
Revised: Proof of Theorem 14.

Original: In the following, let K t denote an generic positive constant that may change from line to line and takes the form
Revised: In the following, let K t denote an generic positive constant that may change from line to line and takes the form

Original: [Equation]
Revised: [Equation]

Original: such that K t ≥ 1 and K t ≤ K T for all t ≤ T .
Revised: such that K t ≥ 1 and K t ≤ K T for all t ≤ T .

Original: We ﬁrst note that at initialization, D 0 (cid:16) W, ˜ W (cid:17) = 0 .
Revised: We ﬁrst note that at initialization, D 0 (cid:16) W, ˜ W (cid:17) = 0 .

Original: Since (cid:57) W (cid:57) 0 ≤ K , (cid:57) W (cid:57) T ≤ K T by Lemma 11.
Revised: Since (cid:57) W (cid:57) 0 ≤ K , (cid:57) W (cid:57) T ≤ K T by Lemma 11.

Original: Furthermore it is easy to see that (cid:57) ˜ W (cid:57) 0 ≤ (cid:57) W (cid:57) 0 ≤ K almost surely.
Revised: Furthermore it is easy to see that (cid:57) ˜ W (cid:57) 0 ≤ (cid:57) W (cid:57) 0 ≤ K almost surely.

Original: By the same argument as in Lemma 11, (cid:57) ˜ W (cid:57) T ≤ K T almost surely.
Revised: By the same argument as in Lemma 11, (cid:57) ˜ W (cid:57) T ≤ K T almost surely.

Original: We shall use all above bounds repeatedly in the proof.
Revised: We shall use all above bounds repeatedly in the proof.

Original: We decompose the proof into several steps.
Revised: We decompose the proof into several steps.

Original: Step 1 - Main proof.
Revised: Step 1 - Main proof.

Original: Let us deﬁne, for brevity
Revised: Let us deﬁne, for brevity

Original: [Equation]
Revised: [Equation]

Original: Consider t ≥ 0 .
Revised: Consider t ≥ 0 .

Original: We ﬁrst bound the difference in the updates between W and ˜ W .
Revised: We ﬁrst bound the difference in the updates between W and ˜ W .

Original: Let us start with w 3 and ˜ w 3 .
Revised: Let us start with w 3 and ˜ w 3 .

Original: By Assumption 1, we have:
Revised: By Assumption 1, we have:

Original: [Equation]
Revised: [Equation]

Original: Similarly, for w 2 and ˜ w 2 ,
Revised: Similarly, for w 2 and ˜ w 2 ,

Original: [Equation]
Revised: [Equation]

Original: and for w 1 and ˜ w 1 , by Lemma 12,
Revised: and for w 1 and ˜ w 1 , by Lemma 12,

Original: [Equation]
Revised: [Equation]

Original: )) (cid:12)(cid:12)
Revised: )) (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: To further the bounding, we now make the following two claims:
Revised: To further the bounding, we now make the following two claims:

Original: • Claim 1: For any ξ > 0 ,
Revised: • Claim 1: For any ξ > 0 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and similarly
Revised: and similarly

Original: [Equation]
Revised: [Equation]

Original: • Claim 2:
Revised: • Claim 2:

Original: For any γ 1 , γ 2 , γ 3 > 0 and t ≥ 0 ,
Revised: For any γ 1 , γ 2 , γ 3 > 0 and t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: with probability at most
Revised: with probability at most

Original: [Equation]
Revised: [Equation]

Original: Combining these claims with the previous bounds
Revised: Combining these claims with the previous bounds

Original: [Equation]
Revised: [Equation]

Original: taking a union bound over t ∈ { 0 , ξ, 2 ξ, ..., (cid:98) T/ξ (cid:99) ξ } for some ξ ∈ (0 , 1)
Revised: taking a union bound over t ∈ { 0 , ξ, 2 ξ, ..., (cid:98) T/ξ (cid:99) ξ } for some ξ ∈ (0 , 1)

Original: [Equation]
Revised: [Equation]

Original: we obtain that
Revised: we obtain that

Original: [Equation]
Revised: [Equation]

Original: ≤ K T
Revised: ≤ K T

Original: [Equation]
Revised: [Equation]

Original: D t
Revised: D t

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: + γ
Revised: + γ

Original: [Equation]
Revised: [Equation]

Original: + γ 3 + ξ (cid:17) , ∀ t ∈ [0 , T ] , with probability at least
Revised: + γ 3 + ξ (cid:17) , ∀ t ∈ [0 , T ] , with probability at least

Original: [Equation]
Revised: [Equation]

Original: The above event in turn implies
Revised: The above event in turn implies

Original: [Equation]
Revised: [Equation]

Original: and hence by Gronwall’s lemma and the fact D 0 (cid:16) W, ˜ W (cid:17) = 0 , we get
Revised: and hence by Gronwall’s lemma and the fact D 0 (cid:16) W, ˜ W (cid:17) = 0 , we get

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: The theorem then follows from the choice ξ =
Revised: The theorem then follows from the choice ξ =

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .
Revised: (cid:19) .

Original: We are left with proving the claims.
Revised: We are left with proving the claims.

Original: [Table]
Revised: [Table]

Original: By Lemma
Revised: By Lemma

Original: [Table]
Revised: [Table]

Original: we then obtain that
Revised: we then obtain that

Original: [Equation]
Revised: [Equation]

Original: Using these estimates
Revised: Using these estimates

Original: [Equation]
Revised: [Equation]

Original: we thus have
Revised: we thus have

Original: [Equation]
Revised: [Equation]

Original: by Assumption
Revised: by Assumption

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: The proof of the rest of the claim is similar
Revised: The proof of the rest of the claim is similar

Original: [Equation]
Revised: [Equation]

Original: Step 3 - Proof of Claim 2.
Revised: Step 3 - Proof of Claim 2.

Original: We recall the deﬁnitions of q ∆ , q 2 and q 3 .
Revised: We recall the deﬁnitions of q ∆ , q 2 and q 3 .

Original: Let us decompose them as follows.
Revised: Let us decompose them as follows.

Original: We start with q 2 :
Revised: We start with q 2 :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have for q 3 :
Revised: Similarly, we have for q 3 :

Original: [Equation]
Revised: [Equation]

Original: ≡ Q 3 , 1 ( x ) + Q 3 , 2 ( x ) .
Revised: ≡ Q 3 , 1 ( x ) + Q 3 , 2 ( x ) .

Original: Finally we have for q ∆ :
Revised: Finally we have for q ∆ :

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: ≡ Q 1 , 1 ( z, j 1 ) + Q 1 , 2 ( z, j 1 ) .
Revised: ≡ Q 1 , 1 ( z, j 1 ) + Q 1 , 2 ( z, j 1 ) .

Original: Now let us analyze each of the terms.
Revised: Now let us analyze each of the terms.

Original: • We start with Q 2 , 1 .
Revised: • We start with Q 2 , 1 .

Original: We have from Assumption 1,
Revised: We have from Assumption 1,

Original: [Equation]
Revised: [Equation]

Original: • To bound Q 2 , 2 , let us write:
Revised: • To bound Q 2 , 2 , let us write:

Original: [Equation]
Revised: [Equation]

Original: Recall that C 1 ( j 1 ) and C 2 ( j 2 ) are independent.
Revised: Recall that C 1 ( j 1 ) and C 2 ( j 2 ) are independent.

Original: We thus have:
Revised: We thus have:

Original: [Equation]
Revised: [Equation]

Original: j 2 ))] .
Revised: j 2 ))] .

Original: Furthermore { Z 2 ( C 1 ( j 1 ) , C 2 ( j 2 )) } j 1 ∈ [ n 1 ] are independent, conditional on C 2 ( j 2 ) .
Revised: Furthermore { Z 2 ( C 1 ( j 1 ) , C 2 ( j 2 )) } j 1 ∈ [ n 1 ] are independent, conditional on C 2 ( j 2 ) .

Original: We also have, almost surely |
Revised: We also have, almost surely |

Original: [Equation]
Revised: [Equation]

Original: ) , C
Revised: ) , C

Original: [Equation]
Revised: [Equation]

Original: by Assumption 1.
Revised: by Assumption 1.

Original: Then by Lemma 19,
Revised: Then by Lemma 19,

Original: [Equation]
Revised: [Equation]

Original: • To bound Q 3 , 1 , we have from Assumption 1,
Revised: • To bound Q 3 , 1 , we have from Assumption 1,

Original: [Equation]
Revised: [Equation]

Original: • To bound Q 3 , 2 , noticing that almost surely
Revised: • To bound Q 3 , 2 , noticing that almost surely

Original: [Equation]
Revised: [Equation]

Original: by Assumption 1, we obtain
Revised: by Assumption 1, we obtain

Original: [Equation]
Revised: [Equation]

Original: similar to the treatment of Q 2 , 2 .
Revised: similar to the treatment of Q 2 , 2 .

Original: • To bound Q 1 , 1 , using Assumption 1,
Revised: • To bound Q 1 , 1 , using Assumption 1,

Original: [Equation]
Revised: [Equation]

Original: • To bound Q 1 , 2 , we note that almost surely
Revised: • To bound Q 1 , 2 , we note that almost surely

Original: [Equation]
Revised: [Equation]

Original: Then similar to the bounding of Q 2 , 2 , we get
Revised: Then similar to the bounding of Q 2 , 2 , we get

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: combining all of these bounds together, applying suitably the union bound over j 1 ∈ [ n 1 ] and j 2 ∈ [ n 2 ] , we obtain the claim.
Revised: combining all of these bounds together, applying suitably the union bound over j 1 ∈ [ n 1 ] and j 2 ∈ [ n 2 ] , we obtain the claim.

Original: Proof of Theorem 15.
Revised: Proof of Theorem 15.

Original: We consider t ≤ T , for a given terminal time T ∈ (cid:15) N ≥ 0 .
Revised: We consider t ≤ T , for a given terminal time T ∈ (cid:15) N ≥ 0 .

Original: We again reuse the notation K t from the proof of Theorem 3.
Revised: We again reuse the notation K t from the proof of Theorem 3.

Original: Note that K t ≤ K T for all t ≤ T .
Revised: Note that K t ≤ K T for all t ≤ T .

Original: We also note that at initialization, D 0 (cid:16) W , ˜ W (cid:17) = 0 .
Revised: We also note that at initialization, D 0 (cid:16) W , ˜ W (cid:17) = 0 .

Original: We also recall from the proof of Theorem 3 that
Revised: We also recall from the proof of Theorem 3 that

Original: [Equation]
Revised: [Equation]

Original: almost surely.
Revised: almost surely.

Original: For brevity
Revised: For brevity

Original: [Equation]
Revised: [Equation]

Original: let us deﬁne several quantities that relate to the difference in the gradient updates between W and ˜ W :
Revised: let us deﬁne several quantities that relate to the difference in the gradient updates between W and ˜ W :

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: Let us also deﬁne:
Revised: Let us also deﬁne:

Original: [Equation]
Revised: [Equation]

Original: We proceed in several steps.
Revised: We proceed in several steps.

Original: Step 1: Decomposition.
Revised: Step 1: Decomposition.

Original: As shown in the proof of Theorem 3:
Revised: As shown in the proof of Theorem 3:

Original: [Equation]
Revised: [Equation]

Original: for any t ≥ 0 and ξ ≥ 0 .
Revised: for any t ≥ 0 and ξ ≥ 0 .

Original: These time-interpolation estimates, along with Assumption 1, allow to derive the following.
Revised: These time-interpolation estimates, along with Assumption 1, allow to derive the following.

Original: We ﬁrst have:
Revised: We ﬁrst have:

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne
Revised: where we deﬁne

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: similarly:
Revised: similarly:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: in which
Revised: in which

Original: [Equation]
Revised: [Equation]

Original: The task is now to bound Q 1 , 1 , Q 1 , 2 , Q 2 , 1 , Q 2 , 2 , Q 3 , 1 and Q 3 , 2 .
Revised: The task is now to bound Q 1 , 1 , Q 1 , 2 , Q 2 , 1 , Q 2 , 2 , Q 3 , 1 and Q 3 , 2 .

Original: Step 2:
Revised: Step 2:

Original: Bounding the terms.
Revised: Bounding the terms.

Original: Before we proceed, let us give some bounds for q H 3 and q H 2 , which hold for any x ∈ R d :
Revised: Before we proceed, let us give some bounds for q H 3 and q H 2 , which hold for any x ∈ R d :

Original: [Equation]
Revised: [Equation]

Original: With these, we have the following:
Revised: With these, we have the following:

Original: • Let us bound Q 3 , 1 .
Revised: • Let us bound Q 3 , 1 .

Original: By Assumption 1,
Revised: By Assumption 1,

Original: [Equation]
Revised: [Equation]

Original: We then get:
Revised: We then get:

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: to Q 3 , 1 , we
Revised: to Q 3 , 1 , we

Original: consider
Revised: consider

Original: Q 2 ,
Revised: Q 2 ,

Original: [Equation]
Revised: [Equation]

Original: , j 1 ) |
Revised: , j 1 ) |

Original: [Equation]
Revised: [Equation]

Original: which yields
Revised: which yields

Original: [Equation]
Revised: [Equation]

Original: • Again we get a similar bound for Q 1 , 1 :
Revised: • Again we get a similar bound for Q 1 , 1 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which yields
Revised: which yields

Original: [Equation]
Revised: [Equation]

Original: • Let us bound Q 3 , 2 .
Revised: • Let us bound Q 3 , 2 .

Original: Let us deﬁne:
Revised: Let us deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Let F k be the sigma-algebra generated by { z ( (cid:96) ) : (cid:96) ∈ { 0 , ..., k − 1 }} .
Revised: Let F k be the sigma-algebra generated by { z ( (cid:96) ) : (cid:96) ∈ { 0 , ..., k − 1 }} .

Original: Note that { r 3 ( k, j 2 ) } k ∈ N is a martingale adapted to {F k } k ∈ N .
Revised: Note that { r 3 ( k, j 2 ) } k ∈ N is a martingale adapted to {F k } k ∈ N .

Original: Furthermore, for k ≤ T/(cid:15) , the martingale difference is bounded: | r 3 ( k, z ( k ) , j 2 ) | ≤ K by Assumption 1.
Revised: Furthermore, for k ≤ T/(cid:15) , the martingale difference is bounded: | r 3 ( k, z ( k ) , j 2 ) | ≤ K by Assumption 1.

Original: Therefore, by
Revised: Therefore, by

Original: Theorem 20 and the union bound, we have:
Revised: Theorem 20 and the union bound, we have:

Original: [Equation]
Revised: [Equation]

Original: • The bounding of Q 2 , 2 is similar: | r 2 ( k, z ( k ) , j 1 , j 2 ) | ≤ K k(cid:15) almost surely by Assumption 1, and thus
Revised: • The bounding of Q 2 , 2 is similar: | r 2 ( k, z ( k ) , j 1 , j 2 ) | ≤ K k(cid:15) almost surely by Assumption 1, and thus

Original: [Equation]
Revised: [Equation]

Original: • Again the bounding of Q 1 , 2 is also similar: | r 1 ( k, z ( k ) , j 1 ) | ≤ K k(cid:15) almost surely by Assumption 1, and thus
Revised: • Again the bounding of Q 1 , 2 is also similar: | r 1 ( k, z ( k ) , j 1 ) | ≤ K k(cid:15) almost surely by Assumption 1, and thus

Original: [Equation]
Revised: [Equation]

Original: Step 3: Putting everything together.
Revised: Step 3: Putting everything together.

Original: All the above results give us
Revised: All the above results give us

Original: [Equation]
Revised: [Equation]

Original: which hold with probability at least
Revised: which hold with probability at least

Original: [Equation]
Revised: [Equation]

Original: The above event implies, by Gronwall’s lemma,
Revised: The above event implies, by Gronwall’s lemma,

Original: [Equation]
Revised: [Equation]

Original: Choosing ξ = K T (cid:112)
Revised: Choosing ξ = K T (cid:112)

Original: ( T + 1) (cid:15) log (2 n 1 n 2 /δ ) completes the proof.
Revised: ( T + 1) (cid:15) log (2 n 1 n 2 /δ ) completes the proof.

Original: C. P ROOFS OF C OROLLARIES 4 AND
Revised: C. P ROOFS OF C OROLLARIES 4 AND

Original: Proof of Corollary 4.
Revised: Proof of Corollary 4.

Original: By the assumption on ψ and Assumption 1, we have: | E Z [ ψ ( Y, ˆ y ( X ; W ( (cid:98) t/(cid:15) (cid:99) )))
Revised: By the assumption on ψ and Assumption 1, we have: | E Z [ ψ ( Y, ˆ y ( X ; W ( (cid:98) t/(cid:15) (cid:99) )))

Original: − ψ ( Y, ˆ y ( X ; W ( t )))]
Revised: − ψ ( Y, ˆ y ( X ; W ( t )))]

Original: |
Revised: |

Original: ≤ K E
Revised: ≤ K E

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: | ] ≤ K E Z
Revised: | ] ≤ K E Z

Original: [Equation]
Revised: [Equation]

Original: | ] .
Revised: | ] .

Original: An inspection of the proof of Theorem 3 (in particular, the proofs of Theorems 14 and 15) reveals that ﬁrstly,
Revised: An inspection of the proof of Theorem 3 (in particular, the proofs of Theorems 14 and 15) reveals that ﬁrstly,

Original: [Equation]
Revised: [Equation]

Original: and secondly, sup
Revised: and secondly, sup

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: Together with Theorem 3, we obtain the claim.
Revised: Together with Theorem 3, we obtain the claim.

Original: Proof of Corollary 5.
Revised: Proof of Corollary 5.

Original: Observe that for each index { N 1 , N 2 } of Init , one obtains a neural network initialization W (0) with law ρ by setting
Revised: Observe that for each index { N 1 , N 2 } of Init , one obtains a neural network initialization W (0) with law ρ by setting

Original: [Equation]
Revised: [Equation]

Original: We consider the evolution W ( k ) starting from W (0) , which is independent of W .
Revised: We consider the evolution W ( k ) starting from W (0) , which is independent of W .

Original: Note that
Revised: Note that

Original: W ( k ) is a deterministic function of its initialization W (0) and the data { z ( j ) } j ≤ k .
Revised: W ( k ) is a deterministic function of its initialization W (0) and the data { z ( j ) } j ≤ k .

Original: Similarly, we consider the counterpart for ˆ W : the evolution ˆ W ( k ) as a function of the initialization ˆ W (0) and the data { ˆ z ( j ) } j ≤ k .
Revised: Similarly, we consider the counterpart for ˆ W : the evolution ˆ W ( k ) as a function of the initialization ˆ W (0) and the data { ˆ z ( j ) } j ≤ k .

Original: Due to sharing the same distribution for both the initialization and the data, these evolutions have the same law; to be speciﬁc, W ( n 1 , n 2 , T ) and ˆ W ( n 1 , n 2 , T ) has the same distribution for any n 1 , n 2 and T , where we deﬁne
Revised: Due to sharing the same distribution for both the initialization and the data, these evolutions have the same law; to be speciﬁc, W ( n 1 , n 2 , T ) and ˆ W ( n 1 , n 2 , T ) has the same distribution for any n 1 , n 2 and T , where we deﬁne

Original: [Equation]
Revised: [Equation]

Original: and a similar deﬁnition for ˆ W ( n 1 , n 2 , T ) .
Revised: and a similar deﬁnition for ˆ W ( n 1 , n 2 , T ) .

Original: In other words,
Revised: In other words,

Original: [Equation]
Revised: [Equation]

Original: Theorem 3 implies that for any tuple { n 1 , n 2 } such that n 1 ≤ N 1 and n 2 ≤ N 2 , with probability at least 1 − 2 δ ,
Revised: Theorem 3 implies that for any tuple { n 1 , n 2 } such that n 1 ≤ N 1 and n 2 ≤ N 2 , with probability at least 1 − 2 δ ,

Original: [Equation]
Revised: [Equation]

Original: where ˜ O δ,T ( (cid:15), N 1 , N 2 ) → 0 as (cid:15) → 0 and N − 1min log N max → 0 with N min = min { N 1 , N 2 } and
Revised: where ˜ O δ,T ( (cid:15), N 1 , N 2 ) → 0 as (cid:15) → 0 and N − 1min log N max → 0 with N min = min { N 1 , N 2 } and

Original: N max = max { N 1 , N 2 } .
Revised: N max = max { N 1 , N 2 } .

Original: We also have a similar result for ˆ W and ˆ W .
Revised: We also have a similar result for ˆ W and ˆ W .

Original: As such, with probability at least 1 − 4 δ ,
Revised: As such, with probability at least 1 − 4 δ ,

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: ≤ 2 ˜
Revised: ≤ 2 ˜

Original: O
Revised: O

Original: δ,T
Revised: δ,T

Original: [Equation]
Revised: [Equation]

Original: , N 1 , N 2 )
Revised: , N 1 , N 2 )

Original: [Equation]
Revised: [Equation]

Original: By ﬁxing the tuple { n 1 , n 2 } while letting (cid:15) → 0 , N − 1min log N max → 0 and δ → 0 , we obtain the claim.
Revised: By ﬁxing the tuple { n 1 , n 2 } while letting (cid:15) → 0 , N − 1min log N max → 0 and δ → 0 , we obtain the claim.

Original: D G LOBAL CONVERGENCE : PROOFS FOR S ECTIOND. P ROOF OF P ROPOSITION
Revised: D G LOBAL CONVERGENCE : PROOFS FOR S ECTIOND. P ROOF OF P ROPOSITION

Original: Proof of Proposition 7.
Revised: Proof of Proposition 7.

Original: Consider a probability space (Λ , G , P 0 ) with random processes R d -valued p 1 ( θ 1 ) , R -valued p 2 ( θ 1 , θ 2 ) and R -valued p 3 ( θ 2 ) , which are indexed by ( θ 1 , θ 2 ) ∈ [0 , 1] ×
Revised: Consider a probability space (Λ , G , P 0 ) with random processes R d -valued p 1 ( θ 1 ) , R -valued p 2 ( θ 1 , θ 2 ) and R -valued p 3 ( θ 2 ) , which are indexed by ( θ 1 , θ 2 ) ∈ [0 , 1] ×

Original: [0 , 1] , such that the following holds.
Revised: [0 , 1] , such that the following holds.

Original: Let m 1 and m 2 be two arbitrary ﬁnite positive integers and, with these integers, let (cid:110)
Revised: Let m 1 and m 2 be two arbitrary ﬁnite positive integers and, with these integers, let (cid:110)

Original: θ ( k i ) i ∈
Revised: θ ( k i ) i ∈

Original: [0 , 1] : k i ∈ [ m i ] , i = 1 , 2 (cid:111) be an arbitrary collection.
Revised: [0 , 1] : k i ∈ [ m i ] , i = 1 , 2 (cid:111) be an arbitrary collection.

Original: For each i = 1 , 2 , let S i be the set of unique elements in (cid:110)
Revised: For each i = 1 , 2 , let S i be the set of unique elements in (cid:110)

Original: θ ( k i )
Revised: θ ( k i )

Original: i : k i ∈
Revised: i : k i ∈

Original: [ m i ] (cid:111) .
Revised: [ m i ] (cid:111) .

Original: Similarly, let R 2 be the set of unique pairs in (cid:110)(cid:16) θ ( k 1 ), θ ( k 2 )(cid:17) : k 1 ∈ [ m 1 ] , k 2 ∈ [ m 2 ] (cid:111) .
Revised: Similarly, let R 2 be the set of unique pairs in (cid:110)(cid:16) θ ( k 1 ), θ ( k 2 )(cid:17) : k 1 ∈ [ m 1 ] , k 2 ∈ [ m 2 ] (cid:111) .

Original: We have that { p 1 ( θ 1 ) : θ 1 ∈ S 1 } , { p 3 ( θ 2 ) : θ 2 ∈ S 2 } and { p 2 ( θ 1 , θ 2 ) : ( θ 1 , θ 2 ) ∈ R 2 } are all mutually independent.
Revised: We have that { p 1 ( θ 1 ) : θ 1 ∈ S 1 } , { p 3 ( θ 2 ) : θ 2 ∈ S 2 } and { p 2 ( θ 1 , θ 2 ) : ( θ 1 , θ 2 ) ∈ R 2 } are all mutually independent.

Original: In addition, we also have Law ( p 1 ( θ 1 ))
Revised: In addition, we also have Law ( p 1 ( θ 1 ))

Original: =
Revised: =

Original: ρ 1 , Law ( p 3 ( θ 2 ))
Revised: ρ 1 , Law ( p 3 ( θ 2 ))

Original: =
Revised: =

Original: ρ 3 and Law ( p 2 ( θ (cid:48) 1 , θ (cid:48) 2 ))
Revised: ρ 3 and Law ( p 2 ( θ (cid:48) 1 , θ (cid:48) 2 ))

Original: =
Revised: =

Original: ρ 2 for any θ 1 ∈ S 1 , θ 2 ∈ S 2 and ( θ (cid:48) 1 , θ (cid:48) 2 ) ∈ R 2 .
Revised: ρ 2 for any θ 1 ∈ S 1 , θ 2 ∈ S 2 and ( θ (cid:48) 1 , θ (cid:48) 2 ) ∈ R 2 .

Original: Such a space exists by Kolmogorov’s extension theorem.
Revised: Such a space exists by Kolmogorov’s extension theorem.

Original: We now construct the desired neuronal embedding.
Revised: We now construct the desired neuronal embedding.

Original: For i = 1 , 2 , consider Ω i =
Revised: For i = 1 , 2 , consider Ω i =

Original: Λ × [0 , 1] and
Revised: Λ × [0 , 1] and

Original: F i =
Revised: F i =

Original: G × B ([0 , 1]) , equipped with the product measure P 0 × Unif ([0 , 1]) in which Unif ([0 , 1]) is the uniform measure over [0 , 1] equipped with the Borel sigma-algebra B ([0 , 1]) .
Revised: G × B ([0 , 1]) , equipped with the product measure P 0 × Unif ([0 , 1]) in which Unif ([0 , 1]) is the uniform measure over [0 , 1] equipped with the Borel sigma-algebra B ([0 , 1]) .

Original: We construct Ω = Ω 1 × Ω 2 and F = F 1 × F 2 , equipped with the product measure P =
Revised: We construct Ω = Ω 1 × Ω 2 and F = F 1 × F 2 , equipped with the product measure P =

Original: ( P 0 × Unif ([0 , 1])) 2 .
Revised: ( P 0 × Unif ([0 , 1])) 2 .

Original: Deﬁne the deterministic functions w 01 :
Revised: Deﬁne the deterministic functions w 01 :

Original: Ω 1 → R d , w 02 :
Revised: Ω 1 → R d , w 02 :

Original: Ω 1 × Ω 2 → R and w 03 : Ω 2 → R :
Revised: Ω 1 × Ω 2 → R and w 03 : Ω 2 → R :

Original: [Equation]
Revised: [Equation]

Original: It is easy to check that this construction yields the desired neuronal embedding.
Revised: It is easy to check that this construction yields the desired neuronal embedding.

Original: D. P ROOF OF T HEOREM
Revised: D. P ROOF OF T HEOREM

Original: We ﬁrst present a measurability argument, which is crucial to showing that a certain universal approximation property holds throughout the course of training.
Revised: We ﬁrst present a measurability argument, which is crucial to showing that a certain universal approximation property holds throughout the course of training.

Original: Lemma 16 (Measurability argument) .
Revised: Lemma 16 (Measurability argument) .

Original: Consider a family Init of initialization laws, which are (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d.
Revised: Consider a family Init of initialization laws, which are (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d.

Original: , such that ρ 2 -almost surely | w 2 | ≤ K and ρ 3 -almost surely | w 3 | ≤ K .
Revised: , such that ρ 2 -almost surely | w 2 | ≤ K and ρ 3 -almost surely | w 3 | ≤ K .

Original: There exists a neuronal embedding (cid:16) Ω , F , P, (cid:8) w 0 i (cid:9) i =1 , 2 , 3 (cid:17) of Init such that there exist Borel functions w ∗and ∆ H ∗for which P -almost surely, for all t ≥ 0 ,
Revised: There exists a neuronal embedding (cid:16) Ω , F , P, (cid:8) w 0 i (cid:9) i =1 , 2 , 3 (cid:17) of Init such that there exist Borel functions w ∗and ∆ H ∗for which P -almost surely, for all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where W ( t ) is the MF dynamics formed under the coupling procedure with this neuronal embedding as described in Section 3.1.
Revised: where W ( t ) is the MF dynamics formed under the coupling procedure with this neuronal embedding as described in Section 3.1.

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: with initialization w ∗ 1 (0 , u 1 ) =
Revised: with initialization w ∗ 1 (0 , u 1 ) =

Original: u 1 for all u 1 ∈ supp (cid:0) ρ 1 (cid:1) and t ≥ 0 , where Z (cid:48) is an independent copy of Z .
Revised: u 1 for all u 1 ∈ supp (cid:0) ρ 1 (cid:1) and t ≥ 0 , where Z (cid:48) is an independent copy of Z .

Original: Proof.
Revised: Proof.

Original: We denote by K t a constant that may depend on t and is ﬁnite with ﬁnite t .
Revised: We denote by K t a constant that may depend on t and is ﬁnite with ﬁnite t .

Original: By Proposition 7, there exists a neuronal embedding that accommodates Init .
Revised: By Proposition 7, there exists a neuronal embedding that accommodates Init .

Original: We recall its construction and reuse the notations from the proof of Proposition 7; in particular:
Revised: We recall its construction and reuse the notations from the proof of Proposition 7; in particular:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let S 1 , S 3 and S 2 denote the sigma-algebras generated by w 01 ( C 1 ) , w 03 ( C 2 ) and (cid:0) w 01 ( C 1 ) , w 02 ( C 1 , C 2 ) , w 03 ( C 2 ) (cid:1) respectively.
Revised: Let S 1 , S 3 and S 2 denote the sigma-algebras generated by w 01 ( C 1 ) , w 03 ( C 2 ) and (cid:0) w 01 ( C 1 ) , w 02 ( C 1 , C 2 ) , w 03 ( C 2 ) (cid:1) respectively.

Original: Let S 13 denote the sigma-algebra generated by Sand S 3 .
Revised: Let S 13 denote the sigma-algebra generated by Sand S 3 .

Original: We also let S Z 1 to denote the sigma-algebra generated by S 1 and the sigma-algebra of the data Z .
Revised: We also let S Z 1 to denote the sigma-algebra generated by S 1 and the sigma-algebra of the data Z .

Original: We deﬁne similarly for S Z 2 and S Z 3 .
Revised: We deﬁne similarly for S Z 2 and S Z 3 .

Original: Step 1: Reduced dynamics.
Revised: Step 1: Reduced dynamics.

Original: Given the MF dynamics W ( t ) , let us deﬁne
Revised: Given the MF dynamics W ( t ) , let us deﬁne

Original: [Equation]
Revised: [Equation]

Original: We recall from the proof of Theorem 3 that for any t, s ≥ 0 ,
Revised: We recall from the proof of Theorem 3 that for any t, s ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Then by Lemma 13,
Revised: Then by Lemma 13,

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Kolmogorov continuity theorem, there exist continuous modiﬁcations of the (timeindexed) processes ¯∆ 1 , ¯∆ 2 and ¯∆ 3 .
Revised: Therefore, by Kolmogorov continuity theorem, there exist continuous modiﬁcations of the (timeindexed) processes ¯∆ 1 , ¯∆ 2 and ¯∆ 3 .

Original: We thus replace them with their continuous modiﬁcations, written by the same notations.
Revised: We thus replace them with their continuous modiﬁcations, written by the same notations.

Original: Given these continuous modiﬁcations, we consider the following reduced dynamics :
Revised: Given these continuous modiﬁcations, we consider the following reduced dynamics :

Original: [Equation]
Revised: [Equation]

Original: in which:
Revised: in which:

Original: • ¯ w 1 : R ≥ 0 × Ω 1 → R d , ¯ w 2 : R ≥ 0 × Ω 1 × Ω 2 (cid:55)→ R , ¯ w 3 : R ≥ 0 × Ω 3 (cid:55)→ R .
Revised: • ¯ w 1 : R ≥ 0 × Ω 1 → R d , ¯ w 2 : R ≥ 0 × Ω 1 × Ω 2 (cid:55)→ R , ¯ w 3 : R ≥ 0 × Ω 3 (cid:55)→ R .

Original: • ¯ W ( t ) =
Revised: • ¯ W ( t ) =

Original: { ¯ w 1 ( t, · ) , ¯ w 2 ( t, · , · ) , ¯ w 3 ( t, · ) } is the collection of reduced parameters at time t , • the initialization is ¯ w 1 (0 , · )
Revised: { ¯ w 1 ( t, · ) , ¯ w 2 ( t, · , · ) , ¯ w 3 ( t, · ) } is the collection of reduced parameters at time t , • the initialization is ¯ w 1 (0 , · )

Original: =
Revised: =

Original: w 01 ( · ) , ¯ w 2 (0 , · , · ) =
Revised: w 01 ( · ) , ¯ w 2 (0 , · , · ) =

Original: w 02 ( · , · ) and ¯ w 3 (0 , · ) =
Revised: w 02 ( · , · ) and ¯ w 3 (0 , · ) =

Original: w 03 ( · ) , i.e. ¯
Revised: w 03 ( · ) , i.e. ¯

Original: [Equation]
Revised: [Equation]

Original: Step 2:
Revised: Step 2:

Original: Measurability of the reduced dynamics.
Revised: Measurability of the reduced dynamics.

Original: It is easy to see that ¯ w 3 ( t, C 2 ) is S 3 -measurable by its construction and the fact ¯ w 3 (0 , C 2 ) =
Revised: It is easy to see that ¯ w 3 ( t, C 2 ) is S 3 -measurable by its construction and the fact ¯ w 3 (0 , C 2 ) =

Original: w 03 ( C 2 ) is S 3 -measurable.
Revised: w 03 ( C 2 ) is S 3 -measurable.

Original: Similarly, ¯ w 2 ( t, C 1 , C 2 ) is
Revised: Similarly, ¯ w 2 ( t, C 1 , C 2 ) is

Original: S 2 -measurable and ¯ w 1 ( t, C 1 ) is S 1 -measurable.
Revised: S 2 -measurable and ¯ w 1 ( t, C 1 ) is S 1 -measurable.

Original: Notice that there exist Borel functions ¯ w ∗ 1 , ¯ w ∗ 2 and ¯ w ∗ 3 for which P -almost surely,
Revised: Notice that there exist Borel functions ¯ w ∗ 1 , ¯ w ∗ 2 and ¯ w ∗ 3 for which P -almost surely,

Original: [Equation]
Revised: [Equation]

Original: Indeed, since ¯ w 2 ( t, C 1 , C 2 ) is S 2 -measurable, there exists a function ¯ w ∗ 2 ( t, · ) for each rational t such that the desired identity holds for P -almost
Revised: Indeed, since ¯ w 2 ( t, C 1 , C 2 ) is S 2 -measurable, there exists a function ¯ w ∗ 2 ( t, · ) for each rational t such that the desired identity holds for P -almost

Original: every ( C 1 , C 2 ) and for all rational t ≥ 0 .
Revised: every ( C 1 , C 2 ) and for all rational t ≥ 0 .

Original: Since ¯ wis continuous in time, there is a unique continuous (in time) function ¯ w ∗ 2 ( t, · ) such that the identity holds for all t ≥ 0 and for P -almost every ( C 1 , C 2 ) .
Revised: Since ¯ wis continuous in time, there is a unique continuous (in time) function ¯ w ∗ 2 ( t, · ) such that the identity holds for all t ≥ 0 and for P -almost every ( C 1 , C 2 ) .

Original: The same argument yields the construction of ¯ w ∗ 1 and ¯ w ∗ 3 .
Revised: The same argument yields the construction of ¯ w ∗ 1 and ¯ w ∗ 3 .

Original: Step 3:
Revised: Step 3:

Original: Measurability of constituent quantities.
Revised: Measurability of constituent quantities.

Original: We show that H 2 (cid:0) X, C 2 ; ¯ W ( t ) (cid:1) is S Z 3measurable.
Revised: We show that H 2 (cid:0) X, C 2 ; ¯ W ( t ) (cid:1) is S Z 3measurable.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: By the existence of ¯ w ∗ 1 and ¯ w ∗ 2 , for each t ≥ 0 , there exists a Borel function f t such that almost surely
Revised: By the existence of ¯ w ∗ 1 and ¯ w ∗ 2 , for each t ≥ 0 , there exists a Borel function f t such that almost surely

Original: [Equation]
Revised: [Equation]

Original: We recall that ρ 1 and ρ 2 are the laws of w 01 ( C 1 ) and w 02 ( C 1 , C 2 ) .
Revised: We recall that ρ 1 and ρ 2 are the laws of w 01 ( C 1 ) and w 02 ( C 1 , C 2 ) .

Original: We analyze the following:
Revised: We analyze the following:

Original: [Equation]
Revised: [Equation]

Original: Let us evaluate the ﬁrst term:
Revised: Let us evaluate the ﬁrst term:

Original: [Equation]
Revised: [Equation]

Original: where in step ( a )
Revised: where in step ( a )

Original: [Equation]
Revised: [Equation]

Original: we deﬁne C (cid:48) 1 to be an independent copy of C 1 ; in step ( b )
Revised: we deﬁne C (cid:48) 1 to be an independent copy of C 1 ; in step ( b )

Original: [Equation]
Revised: [Equation]

Original: we recall C 1 = ( λ 1 , θ 1 ) ; in step ( c )
Revised: we recall C 1 = ( λ 1 , θ 1 ) ; in step ( c )

Original: [Equation]
Revised: [Equation]

Original: we recall θ 1 , θ (cid:48) 1 ∼ Unif ([0 , 1]) and since C 1 is independent of C (cid:48)
Revised: we recall θ 1 , θ (cid:48) 1 ∼ Unif ([0 , 1]) and since C 1 is independent of C (cid:48)

Original: [Equation]
Revised: [Equation]

Original: we have θ 1 (cid:54)
Revised: we have θ 1 (cid:54)

Original: = θ (cid:48)
Revised: = θ (cid:48)

Original: 1 almost surely; step ( d ) is owing to the independence property of the construction of the functions p 1 , p 2 and p 3 .
Revised: 1 almost surely; step ( d ) is owing to the independence property of the construction of the functions p 1 , p 2 and p 3 .

Original: We calculate the second term:
Revised: We calculate the second term:

Original: [Equation]
Revised: [Equation]

Original: as well as the last term:
Revised: as well as the last term:

Original: [Equation]
Revised: [Equation]

Original: It is then easy to see that
Revised: It is then easy to see that

Original: [Equation]
Revised: [Equation]

Original: That is, we have almost surely
Revised: That is, we have almost surely

Original: [Equation]
Revised: [Equation]

Original: Note that the right-hand side is S Z 3 -measurable, and hence so is H 2 (cid:0) X, C 2 ; ¯ W ( t ) (cid:1) .
Revised: Note that the right-hand side is S Z 3 -measurable, and hence so is H 2 (cid:0) X, C 2 ; ¯ W ( t ) (cid:1) .

Original: Next we consider ∆ H 2 (cid:0) Z, C 2 ; ¯ W ( t ) (cid:1) .
Revised: Next we consider ∆ H 2 (cid:0) Z, C 2 ; ¯ W ( t ) (cid:1) .

Original: Recall that ∆
Revised: Recall that ∆

Original: [Equation]
Revised: [Equation]

Original: Then together with the existence of ¯ w ∗ 3 , we have ∆ H 2 (cid:0) Z, C 2 ; ¯ W ( t ) (cid:1) is S Z 3 -measurable.
Revised: Then together with the existence of ¯ w ∗ 3 , we have ∆ H 2 (cid:0) Z, C 2 ; ¯ W ( t ) (cid:1) is S Z 3 -measurable.

Original: Now we consider E C 2 (cid:2)
Revised: Now we consider E C 2 (cid:2)

Original: ∆ H 2 (cid:0) Z, C 2 ; ¯ W ( t ) (cid:1) ¯ w 2 ( t, C 1 , C 2 ) (cid:3) .
Revised: ∆ H 2 (cid:0) Z, C 2 ; ¯ W ( t ) (cid:1) ¯ w 2 ( t, C 1 , C 2 ) (cid:3) .

Original: With the existence of ¯ w ∗ 2 , there exists a
Revised: With the existence of ¯ w ∗ 2 , there exists a

Original: Borel function g t such that
Revised: Borel function g t such that

Original: [Equation]
Revised: [Equation]

Original: Then with the same argument as the treatment of H 2 (cid:0) X, C 2 ; ¯ W ( t ) (cid:1) , one can show that
Revised: Then with the same argument as the treatment of H 2 (cid:0) X, C 2 ; ¯ W ( t ) (cid:1) , one can show that

Original: [Equation]
Revised: [Equation]

Original: which is S Z 1 -measurable.
Revised: which is S Z 1 -measurable.

Original: Using these facts together with the existence of ¯ w ∗ 1 , ¯ w ∗ 2 and ¯ w ∗ 3 , we have ∆ 3 (cid:0) C 2 ; ¯ W ( t ) (cid:1) is S 3measurable, ∆ 2 (cid:0) C 1 , C 2 ; ¯ W ( t ) (cid:1) is S 13 -measurable and ∆ 1 (cid:0) C 1 ; ¯ W ( t ) (cid:1) is S 1 -measurable.
Revised: Using these facts together with the existence of ¯ w ∗ 1 , ¯ w ∗ 2 and ¯ w ∗ 3 , we have ∆ 3 (cid:0) C 2 ; ¯ W ( t ) (cid:1) is S 3measurable, ∆ 2 (cid:0) C 1 , C 2 ; ¯ W ( t ) (cid:1) is S 13 -measurable and ∆ 1 (cid:0) C 1 ; ¯ W ( t ) (cid:1) is S 1 -measurable.

Original: Step 4:
Revised: Step 4:

Original: Closeness between the MF dynamics and the reduced dynamics.
Revised: Closeness between the MF dynamics and the reduced dynamics.

Original: We shall use (cid:13)(cid:13)
Revised: We shall use (cid:13)(cid:13)

Original: W − ¯ W (cid:13)(cid:13) t with the same meaning as the distance between two sets of MF parameters.
Revised: W − ¯ W (cid:13)(cid:13) t with the same meaning as the distance between two sets of MF parameters.

Original: Recall by Lemma 11 that (cid:57) W (cid:57) T ≤ K T since (cid:57) W (cid:57) 0 ≤ K .
Revised: Recall by Lemma 11 that (cid:57) W (cid:57) T ≤ K T since (cid:57) W (cid:57) 0 ≤ K .

Original: By the same argument, (cid:13)(cid:13) ¯ W (cid:13)(cid:13)
Revised: By the same argument, (cid:13)(cid:13) ¯ W (cid:13)(cid:13)

Original: T ≤ K T .
Revised: T ≤ K T .

Original: Then by Lemma 13, we have for any t
Revised: Then by Lemma 13, we have for any t

Original: [Table]
Revised: [Table]

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: where step ( a ) is because ∆ 2 (cid:0) C
Revised: where step ( a ) is because ∆ 2 (cid:0) C

Original: [Equation]
Revised: [Equation]

Original: C 2 ; ¯ W ( t ) (cid:1) is S 13 -measurable from Step 3 and S 13 ⊆
Revised: C 2 ; ¯ W ( t ) (cid:1) is S 13 -measurable from Step 3 and S 13 ⊆

Original: S 2 .
Revised: S 2 .

Original: As such,
Revised: As such,

Original: [Equation]
Revised: [Equation]

Original: almost surely for all rational t ≤ T .
Revised: almost surely for all rational t ≤ T .

Original: By continuity in t of both sides, the same holds for all t ≤ T .
Revised: By continuity in t of both sides, the same holds for all t ≤ T .

Original: Hence by Assumption 1,
Revised: Hence by Assumption 1,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: for all t ≤ T almost surely, which leads to
Revised: for all t ≤ T almost surely, which leads to

Original: [Equation]
Revised: [Equation]

Original: almost surely.
Revised: almost surely.

Original: One can obtain similar results for ¯ w 1 versus w 1 and ¯ w 3 versus w 3 .
Revised: One can obtain similar results for ¯ w 1 versus w 1 and ¯ w 3 versus w 3 .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Since W (0) =
Revised: Since W (0) =

Original: ¯ W (0) , by Gronwall’s inequality, (cid:13)(cid:13)
Revised: ¯ W (0) , by Gronwall’s inequality, (cid:13)(cid:13)

Original: W − ¯ W (cid:13)(cid:13) t = 0 for all t ≤ T .
Revised: W − ¯ W (cid:13)(cid:13) t = 0 for all t ≤ T .

Original: In other words, since T is arbitrary, ¯
Revised: In other words, since T is arbitrary, ¯

Original: [Equation]
Revised: [Equation]

Original: for all t ≥ 0 almost surely.
Revised: for all t ≥ 0 almost surely.

Original: Step 5: Concluding.
Revised: Step 5: Concluding.

Original: The ﬁrst claim of the lemma is proven by the conclusion of Step 4 and by choosing w ∗ 

Revised: The ﬁrst claim of the lemma is proven by the conclusion of Step 4 and by choosing w ∗

Original: = ¯ w ∗ 1 , w ∗ 2 = ¯ w ∗ 2 and w ∗ 3 =
Revised: = ¯ w ∗ 1 , w ∗ 2 = ¯ w ∗ 2 and w ∗ 3 =

Original: ¯ w ∗ 3 , as well as the measurability facts from Step 3.
Revised: ¯ w ∗ 3 , as well as the measurability facts from Step 3.

Original: To prove the second claim, since ∆ H 2 (cid:0) Z, C 2 ; ¯ W ( t ) (cid:1) is S Z 3 -measurable
Revised: To prove the second claim, since ∆ H 2 (cid:0) Z, C 2 ; ¯ W ( t ) (cid:1) is S Z 3 -measurable

Original: and (cid:13)(cid:13)
Revised: and (cid:13)(cid:13)

Original: W − ¯ W (cid:13)(cid:13) t = 0 for all t ≥ 0 , there exists a Borel function ∆ H ∗such that
Revised: W − ¯ W (cid:13)(cid:13) t = 0 for all t ≥ 0 , there exists a Borel function ∆ H ∗such that

Original: [Equation]
Revised: [Equation]

Original: for all t ≥ 0 almost surely, by the same argument in Step 2.
Revised: for all t ≥ 0 almost surely, by the same argument in Step 2.

Original: These facts, together with the dynamics of w 1 and w 2 , imply that almost surely, for all t ≥ 0 ,
Revised: These facts, together with the dynamics of w 1 and w 2 , imply that almost surely, for all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: with initialization w ∗ 1 (cid:0) 0 , w 01 ( C 1 ) (cid:1) =
Revised: with initialization w ∗ 1 (cid:0) 0 , w 01 ( C 1 ) (cid:1) =

Original: w 01 ( C 1 ) .
Revised: w 01 ( C 1 ) .

Original: Substituting the ﬁrst equation into the second one
Revised: Substituting the ﬁrst equation into the second one

Original: [Equation]
Revised: [Equation]

Original: we get:
Revised: we get:

Original: [Equation]
Revised: [Equation]

Original: Note that by an argument similar to Step 3,
Revised: Note that by an argument similar to Step 3,

Original: [Equation]
Revised: [Equation]

Original: which holds for all t ≥ 0 almost surely by the same argument in Step 

Revised: which holds for all t ≥ 0 almost surely by the same argument in Step

Original: We thus obtain:
Revised: We thus obtain:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: An important ingredient of the proof is that the distribution of w 1 ( t, C 1 ) has full support at all time t ≥ 0 , even though we only need to assume this property at initialization t = 0 .
Revised: An important ingredient of the proof is that the distribution of w 1 ( t, C 1 ) has full support at all time t ≥ 0 , even though we only need to assume this property at initialization t = 0 .

Original: This key property is proven by a topology argument, supported by the measurability result of Lemma 16.
Revised: This key property is proven by a topology argument, supported by the measurability result of Lemma 16.

Original: We remark that a similar property for two-layer networks is established in Chizat & Bach (2018) using a different topology argument.
Revised: We remark that a similar property for two-layer networks is established in Chizat & Bach (2018) using a different topology argument.

Original: Lemma 17.
Revised: Lemma 17.

Original: Consider the same setting as Theorem 8.
Revised: Consider the same setting as Theorem 8.

Original: For all ﬁnite time t ≥ 0 , the support of
Revised: For all ﬁnite time t ≥ 0 , the support of

Original: Law ( w 1 ( t, C 1 )) is R d .
Revised: Law ( w 1 ( t, C 1 )) is R d .

Original: Proof.
Revised: Proof.

Original: By Lemma 16, one can choose a neural embedding such that there exists Borel functions w ∗and ∆ H ∗for which almost surely, for all t ≥ 0 ,
Revised: By Lemma 16, one can choose a neural embedding such that there exists Borel functions w ∗and ∆ H ∗for which almost surely, for all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where W ( t ) is the MF dynamics formed under the coupling procedure with this neuronal embedding as described in Section 3.1.
Revised: where W ( t ) is the MF dynamics formed under the coupling procedure with this neuronal embedding as described in Section 3.1.

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: with initialization w ∗ 1 (0 , u 1 ) =
Revised: with initialization w ∗ 1 (0 , u 1 ) =

Original: u 1 for all u 1 ∈ supp (cid:0) ρ 1 (cid:1) and t ≥ 0 , where Z (cid:48) is an independent copy of Z .
Revised: u 1 for all u 1 ∈ supp (cid:0) ρ 1 (cid:1) and t ≥ 0 , where Z (cid:48) is an independent copy of Z .

Original: We recall from Lemma 12 that
Revised: We recall from Lemma 12 that

Original: [Equation]
Revised: [Equation]

Original: where K t denotes a generic constant that depends on t and is ﬁnite with ﬁnite t .
Revised: where K t denotes a generic constant that depends on t and is ﬁnite with ﬁnite t .

Original: Therefore, by
Revised: Therefore, by

Original: Assumption
Revised: Assumption

Original: [Equation]
Revised: [Equation]

Original: for t ≤ T and u 1 , u (cid:48) 1 ∈ supp (cid:0) ρ 1 (cid:1) ,
Revised: for t ≤ T and u 1 , u (cid:48) 1 ∈ supp (cid:0) ρ 1 (cid:1) ,

Original: [Equation]
Revised: [Equation]

Original: Applying Gronwall’s lemma to the ﬁrst bound:
Revised: Applying Gronwall’s lemma to the ﬁrst bound:

Original: [Equation]
Revised: [Equation]

Original: Furthermore the second bound implies
Revised: Furthermore the second bound implies

Original: [Equation]
Revised: [Equation]

Original: Therefore ( t, u 1 ) (cid:55)→ w ∗ 1 ( t, u 1 ) is a continuous mapping on [0 , T ] × R d for an arbitrary T ≥ 0 .
Revised: Therefore ( t, u 1 ) (cid:55)→ w ∗ 1 ( t, u 1 ) is a continuous mapping on [0 , T ] × R d for an arbitrary T ≥ 0 .

Original: Given this continuity, we show the thesis by a topology argument.
Revised: Given this continuity, we show the thesis by a topology argument.

Original: Consider the sphere S d which is a compactiﬁcation of R d .
Revised: Consider the sphere S d which is a compactiﬁcation of R d .

Original: We can extend w ∗ 1 to a function M : [0 , T ] × S d → S d ﬁxing the point at inﬁnity, which remains a continuous map since | M ( t, u 1 ) − u 1 | = | M ( t, u 1 ) − M (0 , u 1 ) | ≤ K T t .
Revised: We can extend w ∗ 1 to a function M : [0 , T ] × S d → S d ﬁxing the point at inﬁnity, which remains a continuous map since | M ( t, u 1 ) − u 1 | = | M ( t, u 1 ) − M (0 , u 1 ) | ≤ K T t .

Original: Let M t : R d → R d be deﬁned by M t ( u 1 ) =
Revised: Let M t : R d → R d be deﬁned by M t ( u 1 ) =

Original: M ( t, u 1 ) .
Revised: M ( t, u 1 ) .

Original: We claim that M t is surjective for all ﬁnite t .
Revised: We claim that M t is surjective for all ﬁnite t .

Original: Indeed, if M t fails to be surjective for some t , then for some p ∈ S d , M t : S d → S d \ { p } → S d is homotopic to the constant map, but M then gives a homotopy from the identity map M 0 on the sphere to a constant map, which is a contradiction as the sphere S d is not contractible.
Revised: Indeed, if M t fails to be surjective for some t , then for some p ∈ S d , M t : S d → S d \ { p } → S d is homotopic to the constant map, but M then gives a homotopy from the identity map M 0 on the sphere to a constant map, which is a contradiction as the sphere S d is not contractible.

Original: Hence w ∗ 1 ( t, · ) is surjective for all ﬁnite t .
Revised: Hence w ∗ 1 ( t, · ) is surjective for all ﬁnite t .

Original: Recall that w 1 ( t, C 1 ) =
Revised: Recall that w 1 ( t, C 1 ) =

Original: w ∗ 1 (cid:0) t, w 01 ( C 1 ) (cid:1) almost surely and w 01 ( C 1 ) has full support.
Revised: w ∗ 1 (cid:0) t, w 01 ( C 1 ) (cid:1) almost surely and w 01 ( C 1 ) has full support.

Original: Now let us assume that w 1 ( t, C 1 ) does not have full support at some time t , which implies there is an open ball B in R d for which P ( w 1 ( t, C 1 ) ∈ B )
Revised: Now let us assume that w 1 ( t, C 1 ) does not have full support at some time t , which implies there is an open ball B in R d for which P ( w 1 ( t, C 1 ) ∈ B )

Original: = 0 .
Revised: = 0 .

Original: Then P (cid:0) w ∗ 1 (cid:0) t, w 01 ( C 1 ) (cid:1) ∈ B (cid:1) = 0 .
Revised: Then P (cid:0) w ∗ 1 (cid:0) t, w 01 ( C 1 ) (cid:1) ∈ B (cid:1) = 0 .

Original: Since w ∗ 1 ( t, · ) has full support, there is an open set U such that w ∗ 1 ( t, u 1 ) ∈ B for all u 1 ∈ U .
Revised: Since w ∗ 1 ( t, · ) has full support, there is an open set U such that w ∗ 1 ( t, u 1 ) ∈ B for all u 1 ∈ U .

Original: Then P (cid:0) w 01 ( C 1 ) ∈ U (cid:1) = 0 , contradicting the assumption that w 01 ( C 1 ) has full support.
Revised: Then P (cid:0) w 01 ( C 1 ) ∈ U (cid:1) = 0 , contradicting the assumption that w 01 ( C 1 ) has full support.

Original: Therefore w 1 ( t, C 1 ) must have full support at all t ≥ 0 .
Revised: Therefore w 1 ( t, C 1 ) must have full support at all t ≥ 0 .

Original: With this, we are ready to prove Theorem 8.
Revised: With this, we are ready to prove Theorem 8.

Original: Proof of Theorem 8.
Revised: Proof of Theorem 8.

Original: Recall, by Theorem 1, the solution to the MF ODEs exists uniquely, and by
Revised: Recall, by Theorem 1, the solution to the MF ODEs exists uniquely, and by

Original: Lemma 17, the support of Law ( w 1 ( t, C 1 )) is R d at all t .
Revised: Lemma 17, the support of Law ( w 1 ( t, C 1 )) is R d at all t .

Original: By the convergence assumption, we have that for any (cid:15) > 0 , there exists T ( (cid:15) )
Revised: By the convergence assumption, we have that for any (cid:15) > 0 , there exists T ( (cid:15) )

Original: such that for all t ≥ T ( (cid:15) ) and P -almost every c 1 :
Revised: such that for all t ≥ T ( (cid:15) ) and P -almost every c 1 :

Original: [Equation]
Revised: [Equation]

Original: Since Law ( w 1 ( t, C 1 )) has full support, we obtain that for u in a dense subset of R d ,
Revised: Since Law ( w 1 ( t, C 1 )) has full support, we obtain that for u in a dense subset of R d ,

Original: [Equation]
Revised: [Equation]

Original: By continuity of u (cid:55)→ ϕ 1 ( (cid:104) u, x (cid:105) ) , we extend the above to all u ∈ R d .
Revised: By continuity of u (cid:55)→ ϕ 1 ( (cid:104) u, x (cid:105) ) , we extend the above to all u ∈ R d .

Original: Since ϕ 1 is bounded,
Revised: Since ϕ 1 is bounded,

Original: [Equation]
Revised: [Equation]

Original: where the last step is by Assumption 1.
Revised: where the last step is by Assumption 1.

Original: Recall that the right-hand side converges to 0 as t → ∞ .
Revised: Recall that the right-hand side converges to 0 as t → ∞ .

Original: We thus obtain that for all u ∈ R d ,
Revised: We thus obtain that for all u ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: = 0 , which yields that for all u ∈ R d and P -almost every c 2 ,
Revised: = 0 , which yields that for all u ∈ R d and P -almost every c 2 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here we note that by Assumption 1, (cid:12)(cid:12) E
Revised: Here we note that by Assumption 1, (cid:12)(cid:12) E

Original: [Equation]
Revised: [Equation]

Original: ∆ H
Revised: ∆ H

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: ¯ w
Revised: ¯ w

Original: [Equation]
Revised: [Equation]

Original: ¯ w
Revised: ¯ w

Original: [Equation]
Revised: [Equation]

Original: , ¯
Revised: , ¯

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ≤ K | ¯ w
Revised: ≤ K | ¯ w

Original: [Equation]
Revised: [Equation]

Original: c 2 ) | , and so
Revised: c 2 ) | , and so

Original: E Z (cid:2) ∆ H 2 ( Z, c 2 ; ¯ w 1 , ¯ w 2 , ¯ w 3 )
Revised: E Z (cid:2) ∆ H 2 ( Z, c 2 ; ¯ w 1 , ¯ w 2 , ¯ w 3 )

Original: | X =
Revised: | X =

Original: x (cid:3) is in
Revised: x (cid:3) is in

Original: L 2 ( P X ) for
Revised: L 2 ( P X ) for

Original: P -almost every c 2 .
Revised: P -almost every c 2 .

Original: Since (cid:8) ϕ
Revised: Since (cid:8) ϕ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: u, ·(cid:105) ) : u ∈ R d (cid:9) has dense span in
Revised: u, ·(cid:105) ) : u ∈ R d (cid:9) has dense span in

Original: L 2 ( P X ) , we have
Revised: L 2 ( P X ) , we have

Original: E Z (cid:2) ∆ H 2 ( Z, c 2 ; ¯ w 1 , ¯ w 2 , ¯ w 3 )
Revised: E Z (cid:2) ∆ H 2 ( Z, c 2 ; ¯ w 1 , ¯ w 2 , ¯ w 3 )

Original: | X =
Revised: | X =

Original: x (cid:3) = 0 for P X -almost every x and P -almost every c 2 , and hence
Revised: x (cid:3) = 0 for P X -almost every x and P -almost every c 2 , and hence

Original: E Z [
Revised: E Z [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: We note that our assumptions guarantee that P ( ¯ w 3 ( C 2 ) (cid:54) = 0) is positive.
Revised: We note that our assumptions guarantee that P ( ¯ w 3 ( C 2 ) (cid:54) = 0) is positive.

Original: Indeed:
Revised: Indeed:

Original: • In the case w 03 ( C 2 ) (cid:54) = 0 with positive probability and ξ 3 ( · ) = 0 , the conclusion is obvious.
Revised: • In the case w 03 ( C 2 ) (cid:54) = 0 with positive probability and ξ 3 ( · ) = 0 , the conclusion is obvious.

Original: • In the case L (cid:0) w 01 , w 02 , w 03 (cid:1) <
Revised: • In the case L (cid:0) w 01 , w 02 , w 03 (cid:1) <

Original: E Z [ L ( Y, ϕ 3 (0))] , we recall the following standard property of gradient ﬂows:
Revised: E Z [ L ( Y, ϕ 3 (0))] , we recall the following standard property of gradient ﬂows:

Original: [Equation]
Revised: [Equation]

Original: for t ≥ t (cid:48) .
Revised: for t ≥ t (cid:48) .

Original: In particular, setting t (cid:48) = 

Revised: In particular, setting t (cid:48) =

Original: and taking t → ∞
Revised: and taking t → ∞

Original: it is easy to see that
Revised: it is easy to see that

Original: [Equation]
Revised: [Equation]

Original: If P ( ¯ w 3 ( C 2 ) = 0) = 1 then L ( ¯ w
Revised: If P ( ¯ w 3 ( C 2 ) = 0) = 1 then L ( ¯ w

Original: ¯ w 2 , ¯ w 3 )
Revised: ¯ w 2 , ¯ w 3 )

Original: =
Revised: =

Original: E Z [ L ( Y, ϕ 3 (0))] , a contradiction.
Revised: E Z [ L ( Y, ϕ 3 (0))] , a contradiction.

Original: Then since ϕ (cid:48) 2 and ϕ (cid:48) 3 are strictly non-zero, we have E Z [ ∂ 2 L ( Y, ˆ y ( X ; ¯ w 1 , ¯ w 2 , ¯ w 3 ))
Revised: Then since ϕ (cid:48) 2 and ϕ (cid:48) 3 are strictly non-zero, we have E Z [ ∂ 2 L ( Y, ˆ y ( X ; ¯ w 1 , ¯ w 2 , ¯ w 3 ))

Original: | X = x ] =for P X -almost every x .
Revised: | X = x ] =for P X -almost every x .

Original: In Case 1, since L convex in the second variable, for any measurable function ˜ y ( x ) ,
Revised: In Case 1, since L convex in the second variable, for any measurable function ˜ y ( x ) ,

Original: L ( y,
Revised: L ( y,

Original: [Equation]
Revised: [Equation]

Original: y (
Revised: y (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking expectation, we get E Z [ L ( Y, ˜ y
Revised: Taking expectation, we get E Z [ L ( Y, ˜ y

Original: ( X ))] ≥ L ( ¯ w 1 , ¯ w 2 , ¯ w 3 ) , i.e. ( ¯ w 1 , ¯ w 2 , ¯ w 3 ) is a global minimizer of L .
Revised: ( X ))] ≥ L ( ¯ w 1 , ¯ w 2 , ¯ w 3 ) , i.e. ( ¯ w 1 , ¯ w 2 , ¯ w 3 ) is a global minimizer of L .

Original: In Case 2, since y is a function of x , we obtain ∂ 2 L ( y, ˆ y ( x ; ¯ w 1 , ¯ w 2 , ¯ w 3 ))
Revised: In Case 2, since y is a function of x , we obtain ∂ 2 L ( y, ˆ y ( x ; ¯ w 1 , ¯ w 2 , ¯ w 3 ))

Original: = 0 and hence
Revised: = 0 and hence

Original: L ( y, ˆ y ( x ; ¯ w 1 , ¯ w 2 , ¯ w 3 ))
Revised: L ( y, ˆ y ( x ; ¯ w 1 , ¯ w 2 , ¯ w 3 ))

Original: = 

Revised: =

Original: for P X -almost every x .
Revised: for P X -almost every x .

Original: Finally we have from Assumptions 1, 3:
Revised: Finally we have from Assumptions 1, 3:

Original: [Equation]
Revised: [Equation]

Original: which tends to 0 as t → ∞ .
Revised: which tends to 0 as t → ∞ .

Original: This completes the proof.
Revised: This completes the proof.

Original: E C ONVERSE FOR GLOBAL CONVERGENCE : R EMARK
Revised: E C ONVERSE FOR GLOBAL CONVERGENCE : R EMARK

Original: We prove a converse statement for global convergence in relation with the essential supremum condition (6).
Revised: We prove a converse statement for global convergence in relation with the essential supremum condition (6).

Original: Proposition 18.
Revised: Proposition 18.

Original: Consider a neuronal embedding (cid:16) Ω , F , P, (cid:8) w 

Revised: Consider a neuronal embedding (cid:16) Ω , F , P, (cid:8) w

Original: i (cid:9) i =1 , 2 , 3 (cid:17) of (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d. initialization.
Revised: i (cid:9) i =1 , 2 , 3 (cid:17) of (cid:0) ρ 1 , ρ 2 , ρ 3 (cid:1) -i.i.d. initialization.

Original: Consider the MF limit corresponding to the network (1), such that they are coupled together by the coupling procedure in Section 3.1, under Assumptions 1, 2, ξ 1 ( · ) = ξ 2 ( · ) = 1 .
Revised: Consider the MF limit corresponding to the network (1), such that they are coupled together by the coupling procedure in Section 3.1, under Assumptions 1, 2, ξ 1 ( · ) = ξ 2 ( · ) = 1 .

Original: Assume that L ( y, ˆ y ) → ∞ as | ˆ y | → ∞ for each y .
Revised: Assume that L ( y, ˆ y ) → ∞ as | ˆ y | → ∞ for each y .

Original: Further assume that there exists ¯ w 3 such that as t → ∞ ,
Revised: Further assume that there exists ¯ w 3 such that as t → ∞ ,

Original: [Equation]
Revised: [Equation]

Original: Then the following hold:
Revised: Then the following hold:

Original: • Case 1 (convex loss): If L is convex in the second variable and lim t →∞ L ( W ( t )) =
Revised: • Case 1 (convex loss): If L is convex in the second variable and lim t →∞ L ( W ( t )) =

Original: inf V L ( V ) , then
Revised: inf V L ( V ) , then

Original: it must be that
Revised: it must be that

Original: [Equation]
Revised: [Equation]

Original: 0 as
Revised: 0 as

Original: [Equation]
Revised: [Equation]

Original: → ∞ .
Revised: → ∞ .

Original: • Case 2 (generic non-negative loss): Suppose that ∂ 2 L ( y, ˆ y ) = 

Revised: • Case 2 (generic non-negative loss): Suppose that ∂ 2 L ( y, ˆ y ) =

Original: implies L ( y, ˆ y )
Revised: implies L ( y, ˆ y )

Original: = 0 , and y
Revised: = 0 , and y

Original: = y ( x ) is a function of x .
Revised: = y ( x ) is a function of x .

Original: If L ( W ( t ))
Revised: If L ( W ( t ))

Original: → 0 as t → ∞ , then the same conclusion also holds.
Revised: → 0 as t → ∞ , then the same conclusion also holds.

Original: Proof.
Revised: Proof.

Original: We recall
Revised: We recall

Original: ∂
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: for c 1 ∈ Ω 1 , c 2 ∈ Ω 2 .
Revised: for c 1 ∈ Ω 1 , c 2 ∈ Ω 2 .

Original: By Assumption 1,
Revised: By Assumption 1,

Original: [Equation]
Revised: [Equation]

Original: Note that the right-hand side is independent of c 1 .
Revised: Note that the right-hand side is independent of c 1 .

Original: Since E C 2 [ | w 3 ( t, C 2 ) − ¯ w 3 ( C 2 ) | ] → 0 as t → ∞ , we have for some ﬁnite t 0 ≤ K ,
Revised: Since E C 2 [ | w 3 ( t, C 2 ) − ¯ w 3 ( C 2 ) | ] → 0 as t → ∞ , we have for some ﬁnite t 0 ≤ K ,

Original: [Equation]
Revised: [Equation]

Original: where the last step is by Lemma 11 and Assumption 2.
Revised: where the last step is by Lemma 11 and Assumption 2.

Original: As such, for all t sufﬁciently large, we have:
Revised: As such, for all t sufﬁciently large, we have:

Original: [Equation]
Revised: [Equation]

Original: The proof concludes once we show that E Z [ | ∂ 2 L ( Y, ˆ y ( X ; W ( t ))) | ] → 0 as t → ∞ .
Revised: The proof concludes once we show that E Z [ | ∂ 2 L ( Y, ˆ y ( X ; W ( t ))) | ] → 0 as t → ∞ .

Original: For a ﬁxed z = ( x, y ) , let us write L ( t, z )
Revised: For a ﬁxed z = ( x, y ) , let us write L ( t, z )

Original: =
Revised: =

Original: L ( y, ˆ y ( x ; W ( t ))) and ∂ 2 L ( t, z ) =
Revised: L ( y, ˆ y ( x ; W ( t ))) and ∂ 2 L ( t, z ) =

Original: ∂ 2 L ( y, ˆ y ( x ; W ( t ))) for brevity.
Revised: ∂ 2 L ( y, ˆ y ( x ; W ( t ))) for brevity.

Original: Consider Case 1.
Revised: Consider Case 1.

Original: We claim that if there is an increasing sequence of time t i so that lim
Revised: We claim that if there is an increasing sequence of time t i so that lim

Original: i →∞
Revised: i →∞

Original: [ L ( t i , z ) − inf ˆ y L ( y, ˆ y )]
Revised: [ L ( t i , z ) − inf ˆ y L ( y, ˆ y )]

Original: = 0 , then lim i →∞ | ∂ 2 L ( t i , z ) | = 0 .
Revised: = 0 , then lim i →∞ | ∂ 2 L ( t i , z ) | = 0 .

Original: Indeed, it sufﬁces to show that for any subsequence t i j of t i , there exists a further subsequence t i jk such that lim k →∞ (cid:12)(cid:12) (cid:12) ∂
Revised: Indeed, it sufﬁces to show that for any subsequence t i j of t i , there exists a further subsequence t i jk such that lim k →∞ (cid:12)(cid:12) (cid:12) ∂

Original: 2 L ( t i jk , z ) (cid:12)(cid:12) (cid:12) = 0 .
Revised: 2 L ( t i jk , z ) (cid:12)(cid:12) (cid:12) = 0 .

Original: In any subsequence t i j of t i , using that L ( t i j , z ) is convergent and the fact L ( y, ˆ y ) → ∞ as | ˆ y | → ∞ , we have ˆ y ( x ; W ( t i j )) is bounded.
Revised: In any subsequence t i j of t i , using that L ( t i j , z ) is convergent and the fact L ( y, ˆ y ) → ∞ as | ˆ y | → ∞ , we have ˆ y ( x ; W ( t i j )) is bounded.

Original: Hence, we obtain a subsequence t i jk for which ˆ y ( x ; W ( t i jk )) converges to some limit ˆ y ∗ .
Revised: Hence, we obtain a subsequence t i jk for which ˆ y ( x ; W ( t i jk )) converges to some limit ˆ y ∗ .

Original: By continuity, we have
Revised: By continuity, we have

Original: L ( y, ˆ y ∗ ) =
Revised: L ( y, ˆ y ∗ ) =

Original: lim k →∞ L ( t i jk , z ) =
Revised: lim k →∞ L ( t i jk , z ) =

Original: inf ˆ y L ( y, ˆ y ) .
Revised: inf ˆ y L ( y, ˆ y ) .

Original: Thus, since L is convex in the second variable, we have ∂ 2 L ( y, ˆ y ∗ ) = 0 .
Revised: Thus, since L is convex in the second variable, we have ∂ 2 L ( y, ˆ y ∗ ) = 0 .

Original: Thus, lim k →∞ (cid:12)(cid:12)(cid:12) ∂ 2 L ( t i jk , z ) (cid:12)(cid:12)(cid:12) =
Revised: Thus, lim k →∞ (cid:12)(cid:12)(cid:12) ∂ 2 L ( t i jk , z ) (cid:12)(cid:12)(cid:12) =

Original: | ∂ 2 L ( y, ˆ y ∗ ) | = 0 , as claimed.
Revised: | ∂ 2 L ( y, ˆ y ∗ ) | = 0 , as claimed.

Original: Similarly, we obtain in Case 2 that if there is an increasing sequence of time t i so that lim
Revised: Similarly, we obtain in Case 2 that if there is an increasing sequence of time t i so that lim

Original: i →∞
Revised: i →∞

Original: [ L ( t i , z )]
Revised: [ L ( t i , z )]

Original: = 0 , then lim
Revised: = 0 , then lim

Original: i →∞ | ∂ 2 L ( t i , z ) | = 0 .
Revised: i →∞ | ∂ 2 L ( t i , z ) | = 0 .

Original: To show that E Z [ | ∂ 2 L ( t, Z ) | ] → 0 as t → ∞ , it sufﬁces to show that for any increasing sequence of times t i tending to inﬁnity, there exists a subsequence t i j of t
Revised: To show that E Z [ | ∂ 2 L ( t, Z ) | ] → 0 as t → ∞ , it sufﬁces to show that for any increasing sequence of times t i tending to inﬁnity, there exists a subsequence t i j of t

Original: i such that E Z (cid:2)(cid:12)(cid:12) ∂ 2 L (cid:0) t i j , Z (cid:1)(cid:12)(cid:12)(cid:3) → 0 .
Revised: i such that E Z (cid:2)(cid:12)(cid:12) ∂ 2 L (cid:0) t i j , Z (cid:1)(cid:12)(cid:12)(cid:3) → 0 .

Original: In
Revised: In

Original: Case 1, we have lim i →∞
Revised: Case 1, we have lim i →∞

Original: [Equation]
Revised: [Equation]

Original: ) , so lim i →∞ E Z (cid:104) L
Revised: ) , so lim i →∞ E Z (cid:104) L

Original: ( t i , Z ) − inf ˆ Y L ( Y, ˆ Y ) (cid:105) = 0 .
Revised: ( t i , Z ) − inf ˆ Y L ( Y, ˆ Y ) (cid:105) = 0 .

Original: Since L ( t i , Z ) − inf ˆ Y L ( Y, ˆ Y ) is nonnegative, this implies that L ( t i , Z ) − inf ˆ Y L ( Y, ˆ Y ) converges to 0 in probability.
Revised: Since L ( t i , Z ) − inf ˆ Y L ( Y, ˆ Y ) is nonnegative, this implies that L ( t i , Z ) − inf ˆ Y L ( Y, ˆ Y ) converges to 0 in probability.

Original: Thus, there is a further subsequence t i j for which L (cid:0) t i j , Z (cid:1) − inf ˆ Y L ( Y, ˆ Y ) converges to 0 P -almost surely.
Revised: Thus, there is a further subsequence t i j for which L (cid:0) t i j , Z (cid:1) − inf ˆ Y L ( Y, ˆ Y ) converges to 0 P -almost surely.

Original: By the previous claim, (cid:12)(cid:12) ∂
Revised: By the previous claim, (cid:12)(cid:12) ∂

Original: 2 L (cid:0) t i j , Z (cid:1)(cid:12)(cid:12) converges to 0 P -almost surely.
Revised: 2 L (cid:0) t i j , Z (cid:1)(cid:12)(cid:12) converges to 0 P -almost surely.

Original: Since (cid:12)(cid:12) ∂ 2 L (cid:0) t i j , Z (cid:1)(cid:12)(cid:12) is bounded P -almost surely, we obtain that
Revised: Since (cid:12)(cid:12) ∂ 2 L (cid:0) t i j , Z (cid:1)(cid:12)(cid:12) is bounded P -almost surely, we obtain that

Original: E Z (cid:2)(cid:12)(cid:12) ∂
Revised: E Z (cid:2)(cid:12)(cid:12) ∂

Original: 2 L (cid:0) t i j , Z (cid:1)(cid:12)(cid:12)(cid:3) →
Revised: 2 L (cid:0) t i j , Z (cid:1)(cid:12)(cid:12)(cid:3) →

Original: 0 from the bounded convergence theorem.
Revised: 0 from the bounded convergence theorem.

Original: The result in Case 2 can be established similarly.
Revised: The result in Case 2 can be established similarly.

Original: F U SEFUL TOOLS
Revised: F U SEFUL TOOLS

Original: We ﬁrst present a useful concentration result.
Revised: We ﬁrst present a useful concentration result.

Original: In fact, the tail bound can be improved using the argument in Feldman & Vondrak (2018), but the following simpler version is sufﬁcient for our purposes.
Revised: In fact, the tail bound can be improved using the argument in Feldman & Vondrak (2018), but the following simpler version is sufﬁcient for our purposes.

Original: Lemma 19.
Revised: Lemma 19.

Original: Consider an integer n ≥ 1 and let x , c 1 , ..., c n be mutually independent random variables.
Revised: Consider an integer n ≥ 1 and let x , c 1 , ..., c n be mutually independent random variables.

Original: Let E x and E c denote the expectations w.r.t.
Revised: Let E x and E c denote the expectations w.r.t.

Original: x only and { c i } i ∈ [ n ] only, respectively.
Revised: x only and { c i } i ∈ [ n ] only, respectively.

Original: Consider a collection of mappings { f i } i ∈ [ n ] , which map to a Hilbert space F .
Revised: Consider a collection of mappings { f i } i ∈ [ n ] , which map to a Hilbert space F .

Original: Let f i ( x ) =
Revised: Let f i ( x ) =

Original: E c
Revised: E c

Original: [ f i ( c i , x )] .
Revised: [ f i ( c i , x )] .

Original: Assume that for some R > 0 and any m ≥ 1 ,
Revised: Assume that for some R > 0 and any m ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: x -almost surely , then there exist an absolute constant r > 0 such that for any δ > 0 ,
Revised: x -almost surely , then there exist an absolute constant r > 0 such that for any δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For brevity
Revised: For brevity

Original: [Equation]
Revised: [Equation]

Original: let us deﬁne
Revised: let us deﬁne

Original: Z n ( x ) = n (cid:88)
Revised: Z n ( x ) = n (cid:88)

Original: i =
Revised: i =

Original: ( f i ( c i , x ) − f i ( x
Revised: ( f i ( c i , x ) − f i ( x

Original: [Equation]
Revised: [Equation]

Original: By Theorem 21, for some universal constant r > 0 ,
Revised: By Theorem 21, for some universal constant r > 0 ,

Original: [Equation]
Revised: [Equation]

Original: and therefore,
Revised: and therefore,

Original: [Equation]
Revised: [Equation]

Original: since the right-hand side is uniform in x .
Revised: since the right-hand side is uniform in x .

Original: Next note that, w.r.t.
Revised: Next note that, w.r.t.

Original: the randomness of x only,
Revised: the randomness of x only,

Original: [Equation]
Revised: [Equation]

Original: As such, by Markov’s inequality and Cauchy-Schwarz’s inequality,
Revised: As such, by Markov’s inequality and Cauchy-Schwarz’s inequality,

Original: [Equation]
Revised: [Equation]

Original: Notice that since c 1 , ..., c n are independent and f i ( x ) =
Revised: Notice that since c 1 , ..., c n are independent and f i ( x ) =

Original: E c
Revised: E c

Original: [ f i ( c i , x )] ,
Revised: [ f i ( c i , x )] ,

Original: [Equation]
Revised: [Equation]

Original: We thus get:
Revised: We thus get:

Original: [Equation]
Revised: [Equation]

Original: This proves the claim.
Revised: This proves the claim.

Original: We state a martingale concentration result, which is a special case of (Pinelis, 1994, Theorem 3.5) which applies to a more general Banach space.
Revised: We state a martingale concentration result, which is a special case of (Pinelis, 1994, Theorem 3.5) which applies to a more general Banach space.

Original: Theorem 20 (Concentration of martingales in Hilbert spaces.) .
Revised: Theorem 20 (Concentration of martingales in Hilbert spaces.) .

Original: Consider a martingale Z n ∈ Z a separable Hilbert space such that | Z n − Z n − 1 | ≤ R and Z 0 = 0 .
Revised: Consider a martingale Z n ∈ Z a separable Hilbert space such that | Z n − Z n − 1 | ≤ R and Z 0 = 0 .

Original: Then for any δ > 0 ,
Revised: Then for any δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Next we state a concentration result for i
Revised: Next we state a concentration result for i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: d. random variables in Hilbert spaces
Revised: d. random variables in Hilbert spaces

Original: [Equation]
Revised: [Equation]

Original: which is a corollary of (Pinelis & Sakhanenko, 1986, Theorem 3)
Revised: which is a corollary of (Pinelis & Sakhanenko, 1986, Theorem 3)

Original: [Equation]
Revised: [Equation]

Original: Theorem 21 (Concentration of i
Revised: Theorem 21 (Concentration of i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: d. sum in Hilbert spaces.) .
Revised: d. sum in Hilbert spaces.) .

Original: Consider n i
Revised: Consider n i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: d. random variables
Revised: d. random variables

Original: X 1 , ..., X n in a Hilbert space.
Revised: X 1 , ..., X n in a Hilbert space.

Original: Suppose that there exists a constant R > 0 such that
Revised: Suppose that there exists a constant R > 0 such that

Original: [Equation]
Revised: [Equation]

Original: for any k ≥ 1 .
Revised: for any k ≥ 1 .

Original: Then for some absolute constant c > 0 and any δ > 0 ,
Revised: Then for some absolute constant c > 0 and any δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Without loss of generality, we can assume E [ X i ] = 0 .
Revised: Without loss of generality, we can assume E [ X i ] = 0 .

Original: By (Vershynin, 2010, Lemma 5.5), there is an absolute constant c > 

Revised: By (Vershynin, 2010, Lemma 5.5), there is an absolute constant c >

Original: such that for any λ ∈ R , E [exp ( λ | X i | )] ≤ exp (cid:0) cλ 2 R 2 (cid:1) .
Revised: such that for any λ ∈ R , E [exp ( λ | X i | )] ≤ exp (cid:0) cλ 2 R 2 (cid:1) .

Original: Then by (Pinelis & Sakhanenko, 1986, Theorem 3), for any x > 0 ,
Revised: Then by (Pinelis & Sakhanenko, 1986, Theorem 3), for any x > 0 ,

Original: [Equation]
Revised: [Equation]

Original: This yields the claim.
Revised: This yields the claim.