Original: P ARALLEL D EEP N EURAL N ETWORKS H AVE Z ERO D UALITY G AP
Revised: P ARALLEL D EEP N EURAL N ETWORKS H AVE Z ERO D UALITY G AP

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training deep neural networks is a well-known highly non-convex problem.
Revised: Training deep neural networks is a well-known highly non-convex problem.

Original: In recent work (Pilanci & Ergen, 2020), it is shown that there is no duality gap for regularized two-layer neural networks with ReLU activation, which enables global optimization via convex programs.
Revised: In recent work (Pilanci & Ergen, 2020), it is shown that there is no duality gap for regularized two-layer neural networks with ReLU activation, which enables global optimization via convex programs.

Original: For multi-layer linear networks with vector outputs, we formulate convex dual problems and demonstrate that the duality gap is non-zero for depth three and deeper networks.
Revised: For multi-layer linear networks with vector outputs, we formulate convex dual problems and demonstrate that the duality gap is non-zero for depth three and deeper networks.

Original: However, by modifying the deep networks to more powerful parallel architectures, we show that the duality gap is exactly zero.
Revised: However, by modifying the deep networks to more powerful parallel architectures, we show that the duality gap is exactly zero.

Original: Therefore, strong convex duality holds, and hence there exist equivalent convex programs that enable training deep networks to global optimality.
Revised: Therefore, strong convex duality holds, and hence there exist equivalent convex programs that enable training deep networks to global optimality.

Original: We also demonstrate that the weight decay regularization in the parameters explicitly encourages low-rank solutions via closed-form expressions.
Revised: We also demonstrate that the weight decay regularization in the parameters explicitly encourages low-rank solutions via closed-form expressions.

Original: For threelayer non-parallel ReLU networks, we show that strong duality holds for rankdata matrices, however, the duality gap is non-zero for whitened data matrices.
Revised: For threelayer non-parallel ReLU networks, we show that strong duality holds for rankdata matrices, however, the duality gap is non-zero for whitened data matrices.

Original: Similarly, by transforming the neural network architecture into a corresponding parallel version, the duality gap vanishes.
Revised: Similarly, by transforming the neural network architecture into a corresponding parallel version, the duality gap vanishes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks demonstrate outstanding representation and generalization abilities in popular learning problems ranging from computer vision, natural language processing to recommendation system.
Revised: Deep neural networks demonstrate outstanding representation and generalization abilities in popular learning problems ranging from computer vision, natural language processing to recommendation system.

Original: Although the training problem of deep neural networks is a highly non-convex optimization problem, simple algorithms, such as stochastic gradient descent, can ﬁnd a solution with good generalization properties.
Revised: Although the training problem of deep neural networks is a highly non-convex optimization problem, simple algorithms, such as stochastic gradient descent, can ﬁnd a solution with good generalization properties.

Original: The non-convex and non-linear nature of neural networks render the theoretical understanding of neural networks extremely challenging.
Revised: The non-convex and non-linear nature of neural networks render the theoretical understanding of neural networks extremely challenging.

Original: The Lagrangian dual problem (Boyd et al., 2004) plays an important role in the theory of convex and non-convex optimization.
Revised: The Lagrangian dual problem (Boyd et al., 2004) plays an important role in the theory of convex and non-convex optimization.

Original: For convex optimization problems, the convex duality is an important tool to determine its optimal value and to characterize the optimal solutions.
Revised: For convex optimization problems, the convex duality is an important tool to determine its optimal value and to characterize the optimal solutions.

Original: Even for a non-convex primal problem, the dual problem is a convex optimization problem the can be solved efﬁciently.
Revised: Even for a non-convex primal problem, the dual problem is a convex optimization problem the can be solved efﬁciently.

Original: As a result of weak duality, the optimal value of the dual problem serves as a non-trivial lower bound for the optimal primal objective value.
Revised: As a result of weak duality, the optimal value of the dual problem serves as a non-trivial lower bound for the optimal primal objective value.

Original: Although the duality gap is non-zero for non-convex problems, the dual problem provides a convex relaxation of the non-convex primal problem.
Revised: Although the duality gap is non-zero for non-convex problems, the dual problem provides a convex relaxation of the non-convex primal problem.

Original: For example, the semi-deﬁnite programming relaxation of the two-way partitioning problem can be derived from its dual problem Boyd et al.
Revised: For example, the semi-deﬁnite programming relaxation of the two-way partitioning problem can be derived from its dual problem Boyd et al.

Original: The convex duality also has important applications in machine learning.
Revised: The convex duality also has important applications in machine learning.

Original: In (Paternain et al., 2019), the design problem of an all-encompassing reward can be formulated as a constrained reinforcement learning problem, which is shown to have zero duality.
Revised: In (Paternain et al., 2019), the design problem of an all-encompassing reward can be formulated as a constrained reinforcement learning problem, which is shown to have zero duality.

Original: This property gives a theoretical convergence guarantee of the primal-dual algorithm for solving this problem.
Revised: This property gives a theoretical convergence guarantee of the primal-dual algorithm for solving this problem.

Original: Meanwhile, the minimax generative adversarial net (GAN) training problem can be tackled using duality (Farnia & Tse, 2018).
Revised: Meanwhile, the minimax generative adversarial net (GAN) training problem can be tackled using duality (Farnia & Tse, 2018).

Original: In lines of recent works, the convex duality can also be applied for analyzing the optimal layer weights of two-layer neural networks with linear or ReLU activations (Pilanci & Ergen, 2020; Ergen & Pilanci, 2020a).
Revised: In lines of recent works, the convex duality can also be applied for analyzing the optimal layer weights of two-layer neural networks with linear or ReLU activations (Pilanci & Ergen, 2020; Ergen & Pilanci, 2020a).

Original: Based on the convex duality framework, the training problem of two-layer neural networks with ReLU activation can be represented in terms of a single convex program in (Pilanci & Ergen, 2020).
Revised: Based on the convex duality framework, the training problem of two-layer neural networks with ReLU activation can be represented in terms of a single convex program in (Pilanci & Ergen, 2020).

Original: Such convex optimization formulations are extended to two-layer and three-layer convolutional neural network training problems in (Ergen & Pilanci, 2020c).
Revised: Such convex optimization formulations are extended to two-layer and three-layer convolutional neural network training problems in (Ergen & Pilanci, 2020c).

Original: Strong duality also holds for deep linear neural networks with scalar output (Ergen & Pilanci, 2020b).
Revised: Strong duality also holds for deep linear neural networks with scalar output (Ergen & Pilanci, 2020b).

Original: The convex optimization formulation essentially gives a detailed characterization of the global optimum of the training problem.
Revised: The convex optimization formulation essentially gives a detailed characterization of the global optimum of the training problem.

Original: This enables us to examine in numerical experiments whether popular optimizers for neural networks, such as gradient descent or stochastic gradient descent, converge to the global optimum of the training loss.
Revised: This enables us to examine in numerical experiments whether popular optimizers for neural networks, such as gradient descent or stochastic gradient descent, converge to the global optimum of the training loss.

Original: Admittedly, the zero duality gap is hard to achieve for deep neural networks, especially for those with vector outputs.
Revised: Admittedly, the zero duality gap is hard to achieve for deep neural networks, especially for those with vector outputs.

Original: This imposes more difﬁculty to train deep neural networks.
Revised: This imposes more difﬁculty to train deep neural networks.

Original: Fortunately, neural networks with parallel structures (also known as multi-branch architecture) appear to be easier to train.
Revised: Fortunately, neural networks with parallel structures (also known as multi-branch architecture) appear to be easier to train.

Original: Practically, the usage of parallel neural networks dates back to AlexNet (Krizhevsky et al., 2012).
Revised: Practically, the usage of parallel neural networks dates back to AlexNet (Krizhevsky et al., 2012).

Original: Modern neural network architecture including Inception (Szegedy et al., 2017), Xception (Chollet, 2017) and SqueezeNet (Iandola et al., 2016) utilize the parallel structure.
Revised: Modern neural network architecture including Inception (Szegedy et al., 2017), Xception (Chollet, 2017) and SqueezeNet (Iandola et al., 2016) utilize the parallel structure.

Original: As the “parallel” version of ResNet (He et al., 2016a;b), ResNeXt (Xie et al., 2017) and Wide ResNet (Zagoruyko &
Revised: As the “parallel” version of ResNet (He et al., 2016a;b), ResNeXt (Xie et al., 2017) and Wide ResNet (Zagoruyko &

Original: Komodakis, 2016) exhibit improved performance on many applications.
Revised: Komodakis, 2016) exhibit improved performance on many applications.

Original: Recently, it was shown that neural networks with the parallel architecture have smaller duality gaps (Zhang et al., 2019) compared to standard neural networks.
Revised: Recently, it was shown that neural networks with the parallel architecture have smaller duality gaps (Zhang et al., 2019) compared to standard neural networks.

Original: On the other hand, it is known that overparameterized parallel neural networks have benign training landscapes (Haeffele & Vidal, 2017).
Revised: On the other hand, it is known that overparameterized parallel neural networks have benign training landscapes (Haeffele & Vidal, 2017).

Original: For training (cid:96) 2 loss with deep linear networks using Schatten norm regularization, Zhang et al.
Revised: For training (cid:96) 2 loss with deep linear networks using Schatten norm regularization, Zhang et al.

Original: (2019) show that there is no duality gap.
Revised: (2019) show that there is no duality gap.

Original: From another perspective, the standard two-layer network is equivalent to the parallel two-layer network.
Revised: From another perspective, the standard two-layer network is equivalent to the parallel two-layer network.

Original: This may also explain why there is no duality gap for two-layer neural networks.
Revised: This may also explain why there is no duality gap for two-layer neural networks.

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: Following the convex duality framework introduced in (Ergen & Pilanci, 2020b;a), which showed the duality gap is zero for two-layer networks, we go beyond two-layer and study the convex duality for vector-output deep neural networks with linear activation and ReLU activation.
Revised: Following the convex duality framework introduced in (Ergen & Pilanci, 2020b;a), which showed the duality gap is zero for two-layer networks, we go beyond two-layer and study the convex duality for vector-output deep neural networks with linear activation and ReLU activation.

Original: Surprisingly, we prove that three-layer networks may have duality gaps depending on their architecture, unlike two-layer neural networks which always have zero duality gap .
Revised: Surprisingly, we prove that three-layer networks may have duality gaps depending on their architecture, unlike two-layer neural networks which always have zero duality gap .

Original: We summarize our contributions as follows.
Revised: We summarize our contributions as follows.

Original: • For training standard vector-output deep linear networks using (cid:96) 2 regularization, we precisely calculate the optimal value of the primal and dual problems and show that the duality gap is non-zero , i.e., Lagrangian relaxation is inexact.
Revised: • For training standard vector-output deep linear networks using (cid:96) 2 regularization, we precisely calculate the optimal value of the primal and dual problems and show that the duality gap is non-zero , i.e., Lagrangian relaxation is inexact.

Original: We also demonstrate that the (cid:96) 2regularization on the parameter explicitly forces a tendency toward a low-rank solution, which is boosted with the depth.
Revised: We also demonstrate that the (cid:96) 2regularization on the parameter explicitly forces a tendency toward a low-rank solution, which is boosted with the depth.

Original: However, we show that the optimal solution is available in closed-form .
Revised: However, we show that the optimal solution is available in closed-form .

Original: • For parallel deep linear networks, with certain convex regularization, we show that the duality gap is zero , i.e, Lagrangian relaxation is exact.
Revised: • For parallel deep linear networks, with certain convex regularization, we show that the duality gap is zero , i.e, Lagrangian relaxation is exact.

Original: • For training vector-output three-layer ReLU networks with standard architecture using (cid:96)regularization, even when the data is whitened, we show that the duality gap is non-zero .
Revised: • For training vector-output three-layer ReLU networks with standard architecture using (cid:96)regularization, even when the data is whitened, we show that the duality gap is non-zero .

Original: The gap can be closed by replacing the neural network to one with parallel architecture.
Revised: The gap can be closed by replacing the neural network to one with parallel architecture.

Original: • For parallel deep ReLU networks of arbitrary depth, with certain convex regularization, we prove strong duality,
Revised: • For parallel deep ReLU networks of arbitrary depth, with certain convex regularization, we prove strong duality,

Original: i
Revised: i

Original: e., the duality gap is zero .
Revised: e., the duality gap is zero .

Original: Remarkably, this guarantees that there is a convex program equivalent to the original deep ReLU neural network problem.
Revised: Remarkably, this guarantees that there is a convex program equivalent to the original deep ReLU neural network problem.

Original: We summarize the duality gaps for parallel/standard neural network in Table 1.
Revised: We summarize the duality gaps for parallel/standard neural network in Table 1.

Original: 1. N OTATIONS
Revised: 1. N OTATIONS

Original: We use bold capital letters to represent matrices and bold lowercase letters to represent vectors.
Revised: We use bold capital letters to represent matrices and bold lowercase letters to represent vectors.

Original: Denote [ n ] = { 1 , . . . , n } .
Revised: Denote [ n ] = { 1 , . . . , n } .

Original: For a matrix W l ∈ R m 1 × m 2 , for i ∈ [ m 1 ] and j ∈ [ m 2 ] , we denote w col l,i as its i -th column and w row l,j as its j -th row.
Revised: For a matrix W l ∈ R m 1 × m 2 , for i ∈ [ m 1 ] and j ∈ [ m 2 ] , we denote w col l,i as its i -th column and w row l,j as its j -th row.

Original: Throughout the paper, X ∈ R N × d is the data matrix consisting of d dimensional N samples and Y ∈
Revised: Throughout the paper, X ∈ R N × d is the data matrix consisting of d dimensional N samples and Y ∈

Original: R N × K is the label matrix for a regression/classiﬁcation task with K outputs.
Revised: R N × K is the label matrix for a regression/classiﬁcation task with K outputs.

Original: 1. M OTIVATIONS AND B ACKGROUND
Revised: 1. M OTIVATIONS AND B ACKGROUND

Original: Recently a series of papers (Pilanci & Ergen, 2020; Ergen & Pilanci, 2020b;a) studied two-layer neural networks via convex duality and proved that strong duality holds for these architectures.
Revised: Recently a series of papers (Pilanci & Ergen, 2020; Ergen & Pilanci, 2020b;a) studied two-layer neural networks via convex duality and proved that strong duality holds for these architectures.

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: Ours
Revised: This paper

Original: [Equation]
Revised: [Table]

Revised: (cid:55) (cid:55)

Revised: This paper (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 )

Original: Particularly, these prior works consider the following weight decay regularized training framework for classiﬁcation/regression tasks.
Revised: Particularly, these prior works consider the following weight decay regularized training framework for classiﬁcation/regression tasks.

Original: Given a data matrix X ∈ R N × d consisting of d dimensional N samples and the corresponding label matrix y ∈ R N , the weight-decay regularized training problem for a  neural network with m hidden neurons can be written as follows
Revised: Given a data matrix X ∈ R N × d consisting of d dimensional N samples and the corresponding label matrix y ∈ R N , the weight-decay regularized training problem for a scalar-output neural network with m hidden neurons can be written as follows

Original: [Equation]
Revised: [Equation]

Original: where W 1 ∈ R d × m and w 2 ∈ R m are the layer weights, β > 0 is a regularization parameter, and φ is the activation function, which can be linear φ ( z )
Revised: where W 1 ∈ R d × m and w 2 ∈ R m are the layer weights, β > 0 is a regularization parameter, and φ is the activation function, which can be linear φ ( z )

Original: =
Revised: =

Original: z or ReLU φ ( z ) =
Revised: z or ReLU φ ( z ) =

Original: max { z, 0 } .
Revised: max { z, 0 } .

Original: Then, one can take the dual of (1) with respect to W 1 and w 2 obtain the following dual optimization problem
Revised: Then, one can take the dual of (1) with respect to W 1 and w 2 obtain the following dual optimization problem

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst note that since the training problem (1) is non-convex, strong duality may not hold, i.e.,
Revised: We ﬁrst note that since the training problem (1) is non-convex, strong duality may not hold, i.e.,

Original: P ≥ D .
Revised: P ≥ D .

Original: Surprisingly, as shown in Pilanci & Ergen (2020); Ergen & Pilanci (2020b;a), strong duality in fact holds, i.e., P = D , for two-layer networks and therefore one can derive exact convex representations for the non-convex training problem in (1).
Revised: Surprisingly, as shown in Pilanci & Ergen (2020); Ergen & Pilanci (2020b;a), strong duality in fact holds, i.e., P = D , for two-layer networks and therefore one can derive exact convex representations for the non-convex training problem in (1).

Original: However, extensions of this approach to deeper and state-of-the-art architectures are not available in the literature.
Revised: However, extensions of this approach to deeper and state-of-the-art architectures are not available in the literature.

Original: Based on this observation, the central question we address in this paper is:
Revised: Based on this observation, the central question we address in this paper is:

Original: Does strong duality hold for deep neural networks
Revised: Does strong duality hold for deep neural networks?

Original: Depending on the answer to the question above, an immediate next questions we address is
Revised: Depending on the answer to the question above, an immediate next questions we address is

Original: Can we characterize the duality gap (P-D)?
Revised: Can we characterize the duality gap ( P-D)?

Original: Is there an architecture for which strong duality holds independent of the depth?
Revised: Is there an architecture for which strong duality holds independent of the depth?

Original: Consequently, throughout the paper, we provide a full characterization of convex duality for deeper neural networks.
Revised: Consequently, throughout the paper, we provide a full characterization of convex duality for deeper neural networks.

Original: Then, based on this characterization, we propose a modiﬁed architecture for which strong duality holds regardless of depth.
Revised: Then, based on this characterization, we propose a modiﬁed architecture for which strong duality holds regardless of depth.

Original: 1. C ONVEX DUALITY FOR TWO - LAYER NEURAL NETWORKS
Revised: 1. C ONVEX DUALITY FOR TWO - LAYER NEURAL NETWORKS

Original: We brieﬂy review the convex duality for two-layer neural networks introduced in (Ergen & Pilanci, 2020b;a).
Revised: We brieﬂy review the convex duality for two-layer neural networks introduced in (Ergen & Pilanci, 2020b;a).

Original: Consider the following weight-decay regularized training problem for a  neural network architecture with m hidden neurons
Revised: Consider the following weight-decay regularized training problem for a vector-output neural network architecture with m hidden neurons

Original: [Equation]
Revised: [Equation]

Original: where W 1 ∈ R d × m and W 2 ∈ R m × K are the variables, and β > 0 is a regularization parameter.
Revised: where W 1 ∈ R d × m and W 2 ∈ R m × K are the variables, and β > 0 is a regularization parameter.

Original: Here φ is the activation function, which can be linear φ ( z ) =
Revised: Here φ is the activation function, which can be linear φ ( z ) =

Original: z or ReLU φ ( z ) =
Revised: z or ReLU φ ( z ) =

Original: max { z, 0 } .
Revised: max { z, 0 } .

Original: As long as the network is sufﬁciently overparameterized, there exists a feasible solution for such that φ ( XW 1 ) w 2 = Y .
Revised: As long as the network is sufﬁciently overparameterized, there exists a feasible solution for such that φ ( XW 1 ) W 2 = Y .

Original: Then, a minimal norm variant 1 of the training problem in (3) is given by
Revised: Then, a minimal norm variant 1 of the training problem in (3) is given by

Original: [Equation]
Revised: [Equation]

Original: As shown in (Pilanci & Ergen, 2020), after a suitable rescaling, this problem can be reformulated as
Revised: As shown in (Pilanci & Ergen, 2020), after a suitable rescaling, this problem can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: where [ m ] = { 1 , . .
Revised: where [ m ] = { 1 , . .

Original: , m } .
Revised: , m } .

Original: Here w row2 ,j represents the j -th row of W 2 and w col1 ,j denotes the j -th column of W 1 .
Revised: Here w row2 ,j represents the j -th row of W 2 and w col1 ,j denotes the j -th column of W 1 .

Original: The rescaling does not change the solution to (4).
Revised: The rescaling does not change the solution to (4).

Original: By taking the dual with respect to W 1 and W 2 , the dual problem of (5) with respect to variables is a convex optimization problem given by
Revised: By taking the dual with respect to W 1 and W 2 , the dual problem of (5) with respect to variables is a convex optimization problem given by

Original: [Equation]
Revised: [Equation]

Original: where Λ ∈ R N × K is the dual variable.
Revised: where Λ ∈ R N × K is the dual variable.

Original: Provided that m ≥ m ∗ , where m ∗ ≤ N + 1 , the strong duality holds, i.e., the optimal value of the primal problem (5) equals to the optimal value of the dual problem (6).
Revised: Provided that m ≥ m ∗ , where m ∗ ≤ N + 1 , the strong duality holds, i.e., the optimal value of the primal problem (5) equals to the optimal value of the dual problem (6).

Original: 1. O RGANIZATION
Revised: 1. O RGANIZATION

Original: This paper is organized as follows.
Revised: This paper is organized as follows.

Original: In Section 2, we review standard neural networks and introduce parallel architectures.
Revised: In Section 2, we review standard neural networks and introduce parallel architectures.

Original: For deep linear networks, we derive primal and dual problems for both standard and parallel architectures and provide calculations of optimal values of these problems in
Revised: For deep linear networks, we derive primal and dual problems for both standard and parallel architectures and provide calculations of optimal values of these problems in

Original: Section 3.
Revised: Section 3.

Original: We derive primal and dual problems for three-layer ReLU networks with standard architecture and precisely calculate the optimal values for whitened data in Section 4.
Revised: We derive primal and dual problems for three-layer ReLU networks with standard architecture and precisely calculate the optimal values for whitened data in Section 4.

Original: We also show that deep ReLU networks with parallel structures have no duality gap.
Revised: We also show that deep ReLU networks with parallel structures have no duality gap.

Original:  S TANDARD NEURAL NETWORKS VS PARALLEL ARCHITECTURES
Revised:  S TANDARD NEURAL NETWORKS VS PARALLEL ARCHITECTURES

Original: We start with the L -layer neural network with the standard architecture:
Revised: We start with the L -layer neural network with the standard architecture:

Original: [Equation]
Revised: [Equation]

Original: where φ is the activation function, W l ∈ R m l − 1 × m l is the weight matrix in the l -th layer and θ =
Revised: where φ is the activation function, W l ∈ R m l − 1 × m l is the weight matrix in the l -th layer and θ =

Original: ( W 1 , . . .
Revised: ( W 1 , . . .

Original: , W L ) represents the parameter of the neural network.
Revised: , W L ) represents the parameter of the neural network.

Original: We also denote the input and output dimensions as m 0 =
Revised: We also denote the input and output dimensions as m 0 =

Original: d and m L = K for simplicity and assume that m l ≥ max { d, K } , ∀ l ∈
Revised: d and m L = K for simplicity and assume that m l ≥ max { d, K } , ∀ l ∈

Original: [ L − 1] .
Revised: [ L − 2] .

Original: We then introduce the neural network with the parallel architecture:
Revised: We then introduce the neural network with the parallel architecture:

Original: [Equation]
Revised: [Equation]

Original: Here for l ∈
Revised: Here for l ∈

Original: [ L − 1] , the l -th layer has m weight matrices W l,j ∈ R m l − 1 × m l where j ∈ [ m ] .
Revised: [ L − 1] , the l -th layer has m weight matrices W l,j ∈ R m l − 1 × m l where j ∈ [ m ] .

Original: Speciﬁcally, we let m L − 1 = 1 .
Revised: Speciﬁcally, we let m L − 1 = 1 .

Original: In short, we can view the output A L − 1 from a parallel neural network as a concatenation of m scalar-output standard neural work.
Revised: In short, we can view the output A L − 1 from a parallel neural network as a concatenation of m scalar-output standard neural work.

Original: In Figure 1 and 2, we provide examples of neural networks with standard and parallel architectures.
Revised: In Figure 1 and 2, we provide examples of neural networks with standard and parallel architectures.

Original: We shall emphasize that for
Revised: We shall emphasize that for

Original: L = 2 , the standard neural network is identical to the parallel neural network.
Revised: L = 2 , the standard neural network is identical to the parallel neural network.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  D EEP LINEAR NETWORKS
Revised:  D EEP LINEAR NETWORKS

Original: In this section [Equation] we discuss the convex duality of deep linear network with vector output.
Revised: In this section, we discuss the convex duality of deep linear network with vector output.

Original: 3. S TANDARD DEEP LINEAR NETWORKS
Revised: 3. S TANDARD DEEP LINEAR NETWORKS

Original: We ﬁrst consider the neural network with standard architecture, i.e., f θ ( X ) =
Revised: We ﬁrst consider the neural network with standard architecture, i.e., f θ ( X ) =

Original: XW 1 . . .
Revised: XW 1 . . .

Original: W L .
Revised: W L .

Original: Consider the following minimal norm optimization problem:
Revised: Consider the following minimal norm optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where the variables are W 1 , .
Revised: where the variables are W 1 , .

Original: , W L .
Revised: , W L .

Original: As shown in the Proposition 3.1 in (Ergen & Pilanci, 2020b), by introducing a scale parameter t , the problem (7) can be reformulated as
Revised: As shown in the Proposition 3.1 in (Ergen & Pilanci, 2020b), by introducing a scale parameter t , the problem (7) can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: where P lin ( t ) is deﬁned as
Revised: where P lin ( t ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The following proposition characterize the dual problem of P lin ( t ) and its bi-dual, i
Revised: The following proposition characterize the dual problem of P lin ( t ) and its bi-dual, i

Original: [Equation]
Revised: [Equation]

Original: e., dual of the dual problem.
Revised: e., dual of the dual problem.

Original: Proposition 

Revised: Proposition

Original: The dual problem of P lin ( t ) deﬁned in (8) is a convex optimization problem given by
Revised: The dual problem of P lin ( t ) deﬁned in (8) is a convex optimization problem given by

Original: [Equation]
Revised: [Equation]

Original: There exists m ∗ < ∞ such that the dual problem D lin ( t ) can be reformulated as the bi-dual, i.e., D lin ( t ) =
Revised: There exists m ∗ ≤ KN + 1 such that the dual problem D lin ( t ) can be reformulated as the bi-dual, i [Equation] e.,

Original: [Equation]
Revised: [Equation]

Original: Detailed derivation of the dual and the bi-dual are provided in Appendix B.1.
Revised: Detailed derivation of the dual and the bi-dual are provided in Appendix B.1.

Original: The reason why we do not directly take the dual of P lin is that the objective function in P lin involves the weights of ﬁrst
Revised: The reason why we do not directly take the dual of P lin is that the objective function in P lin involves the weights of ﬁrst

Original: L − 1 layer, which prevents obtaining a meaningful dual problem.
Revised: L − 1 layer, which prevents obtaining a meaningful dual problem.

Original: We note that bi-dual problem is related to the minimal norm problem of a parallel neural network with aligned weights.
Revised: We note that bi-dual problem is related to the minimal norm problem of a parallel neural network with aligned weights.

Original: Namely, the
Revised: Namely, the

Original: Frobenius norm of the weight matrices in each branch has the same upper bound t .
Revised: Frobenius norm of the weight matrices in each branch has the same upper bound t .

Original: For a matrix A ∈ R m × n and p > 0 , the Schatten- p quasi-norm of A is deﬁned as
Revised: For a matrix A ∈ R m × n and p > 0 , the Schatten- p quasi-norm of A is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where σ i ( A ) is the i -th largest singular value of A .
Revised: where σ i ( A ) is the i -th largest singular value of A .

Original: The optimal value of P lin ( t ) and D lin ( t ) can be precisely calculated.
Revised: The optimal value of P lin ( t ) and D lin ( t ) can be precisely calculated.

Original: Theorem 1 For ﬁxed t > 0 , the optimal value of P lin ( t ) and D lin ( t ) are given by
Revised: Theorem 1 For ﬁxed t > 0 , the optimal value of P lin ( t ) and D lin ( t ) are given by

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where X † is the pseudo inverse of X and (cid:107) · (cid:107) ∗ represents the nuclear norm.
Revised: where X † is the pseudo inverse of X and (cid:107) · (cid:107) ∗ represents the nuclear norm.

Original: As a result, the duality gap exists, i.e., P > D , for standard deep linear networks with L ≥ 3 .
Revised: As a result, the duality gap exists, i.e., P > D , for standard deep linear networks with L ≥ 3 , if the singular values of X † Y are not equal to the same value.

Original: We note that the optimal scale parameter t for the primal problem P lin is given by t ∗ =
Revised: We note that the optimal scale parameter t for the primal problem P lin is given by t ∗ =

Original: (cid:107) W ∗ (cid:107) 1 /LS 2 /L .
Revised: (cid:107) W ∗ (cid:107) 1 /LS 2 /L .

Original: To the best of our knowledge, this result wasn’t shown previously.
Revised: To the best of our knowledge, this result wasn’t shown previously.

Original: We leave the proof in Appendix B. and provide conditions on W 1 , . ..
Revised: We leave the proof in Appendix B.2 and provide conditions on W 1 , . .

Original: , W L to achieve the optimal value.
Revised: , W L to achieve the optimal value.

Original: In the calculation of P lin ( t ) , we utilize the following proposition.
Revised: In the calculation of P lin ( t ) , we utilize the following proposition.

Original: Proposition 2 Suppose that W ∈ R d × K with rank r is given.
Revised: Proposition 2 Suppose that W ∈ R d × K with rank r is given.

Original: Assume that m i ≥ r for i = 0 , . . .
Revised: Assume that m i ≥ r for i = 0 , . . .

Original: , L .
Revised: , L .

Original: Consider the following optimization problem:
Revised: Consider the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Then, the optimal value of the problem (13) is given by L 2 (cid:107) W (cid:107) 2 /LS 2 /L .
Revised: Then, the optimal value of the problem (13) is given by L 2 (cid:107) W (cid:107) 2 /LS 2 /L .

Original: 3. P ARALLEL DEEP LINEAR NEURAL NETWORKS
Revised: 3. P ARALLEL DEEP LINEAR NEURAL NETWORKS

Original: For the neural network with parallel structure
Revised: For the neural network with parallel structure

Original: [Equation]
Revised: [Equation]

Original: we consider the minimal norm optimization problem:
Revised: we consider the minimal norm optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Due to a rescaling to achieve the lower bound of the inequality of arithmetic and geometric means
Revised: Due to a rescaling to achieve the lower bound of the inequality of arithmetic and geometric means

Original: [Equation]
Revised: [Equation]

Original: we have the following result
Revised: we have the following result.

Original: [Equation]

Original: Proposition 

Revised: Proposition

Original: The problem (14) can be formulated as
Revised: The problem (14) can be formulated as

Original: [Equation]
Revised: [Equation]

Original: We note that z 2 /L is a non-convex of z and we cannot take a meaningful dual.
Revised: We note that z 2 /L is a non-convex of z and we cannot take a meaningful dual.

Original: The bi-dual problem of the standard neural network does NOT correspond to the primal problem of the parallel neural network because the Frobenius norm of each parallel part of the parallel architecture can be different in formulating (15).
Revised: The bi-dual problem of the standard neural network does NOT correspond to the primal problem of the parallel neural network because the Frobenius norm of each parallel part of the parallel architecture can be different in formulating (15).

Original: We can consider another regularization, i.e.,
Revised: We can consider another regularization, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Proposition 4 The problem (16) can be formulated as
Revised: Proposition 4 The problem (16) can be formulated as

Original: [Equation]
Revised: [Equation]

Original: The dual problem of P prllin is a convex problem
Revised: The dual problem of P prllin is a convex problem

Original: [Equation]
Revised: [Equation]

Original: For the parallel linear network, the strong duality holds.
Revised: For the parallel linear network, the strong duality holds.

Original: Theorem 2 There exists m ∗ < ∞ such that as long as the number of branches m ≥ m ∗ , the strong duality holds for the problem (16) .
Revised: Theorem 2 There exists m ∗ ≤ KN + 1 such that as long as the number of branches m ≥ m ∗ , the strong duality holds for the problem (16) .

Original: Namely, P prllin =
Revised: Namely, P prllin =

Original: D p rl lin .
Revised: D p rl lin .

Original: The optimal values are both L 2 (cid:107) X † Y (cid:107) ∗ .
Revised: The optimal values are both L [Equation]

Original: This implies that there exist equivalent convex problems which achieve the global optimum of the deep parallel linear network.
Revised: This implies that there exist equivalent convex problems which achieve the global optimum of the deep parallel linear network.

Original: Comparatively, optimizing deep parallel linear neural networks can be much easier than optimizing deep standard linear networks.
Revised: Comparatively, optimizing deep parallel linear neural networks can be much easier than optimizing deep standard linear networks.

Original: 3. G ENERAL LOSS FUNCTIONS
Revised: 3. G ENERAL LOSS FUNCTIONS

Original: Now we consider general loss functions, i.e.,
Revised: Now we consider general loss functions, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) ( Z , Y ) is a general loss function and β > 0 is a regularization parameter.
Revised: where (cid:96) ( Z , Y ) is a general loss function and β > 0 is a regularization parameter.

Original: According to
Revised: According to

Original: Proposition 2, the above problem is equivalent to
Revised: Proposition 2, the above problem is equivalent to

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The (cid:96) 2 regularization term becomes the Schatten- 2 /L quasi-norm on W to the power 2 /L .
Revised: The (cid:96) 2 regularization term becomes the Schatten- 2 /L quasi-norm on W to the power 2 /L .

Original: Suppose that there exists W such that l ( XW , Y ) = 0 .
Revised: Suppose that there exists W such that l ( XW , Y ) = 0 .

Original: With β → 0 , asymptotically, the optimal solution to the problem (19) converges to the optimal solution of
Revised: With β → 0 , asymptotically, the optimal solution to the problem (19) converges to the optimal solution of

Original: [Equation]
Revised: [Equation]

Original: In other words, the (cid:96) 2 regularization explicitly regularizes the training problem to ﬁnd a low-rank solution W .
Revised: In other words, the (cid:96) 2 regularization explicitly regularizes the training problem to ﬁnd a low-rank solution W .

Original:  N EURAL NETWORKS WITH R E LU ACTIVATION
Revised:  N EURAL NETWORKS WITH R E LU ACTIVATION

Original: Now, we focus on the three-layer neural network with ReLU activation, i.e., φ ( z ) = max { z, 0 } .
Revised: Now, we focus on the three-layer neural network with ReLU activation, i.e., φ ( z ) = max { z, 0 } .

Original: 4. S TANDARD THREE - LAYER R E LU NETWORKS
Revised: 4. S TANDARD THREE - LAYER R E LU NETWORKS

Original: We ﬁrst focus on the three-layer ReLU network with standard architecture.
Revised: We ﬁrst focus on the three-layer ReLU network with standard architecture.

Original: Consider the minimal norm problem
Revised: Consider the minimal norm problem

Original: [Equation]
Revised: [Equation]

Original: Here we denote ( z ) + = max { z, 0 } .
Revised: Here we denote ( z ) + = max { z, 0 } .

Original: Similarly, by introducing a scale parameter t , this problem can be formulated as
Revised: Similarly, by introducing a scale parameter t , this problem can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where P ReLU ( t ) is deﬁned as
Revised: where P ReLU ( t ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: For W 1 ∈ R d
Revised: The proof is analagous to the proof of Proposition 3.1 in (Ergen & Pilanci, 2020b).

Original:  × m [Equation] we deﬁne the set
Revised: For W 1 ∈ R d × m , we deﬁne the set

Original: [Equation]
Revised: [Equation]

Original: Proposition 

Revised: Proposition

Original: The dual problem of P ReLU ( t ) deﬁned in (22) is a convex problem deﬁned as
Revised: The dual problem of P ReLU ( t ) deﬁned in (22) is a convex problem deﬁned as

Original: [Equation]
Revised: [Equation]

Original: There exists m ∗ < ∞ such that the dual problem can be reformulated as the bi-dual problem, i.e.,
Revised: There exists m ∗ ≤ KN + 1 such that the dual problem can be reformulated as the bi-dual problem, i.e.,

Original: [Equation]
Revised: [Equation]

Original: For the case where the data matrix is with rank 1 and the neural network is with scalar output, there is no duality gap.
Revised: For the case where the data matrix is with rank 1 and the neural network is with scalar output, there is no duality gap.

Original: Theorem 3 For a three-layer scalar-output ReLU network, let X be a rank-one data matrix.
Revised: Theorem 3 For a three-layer scalar-output ReLU network, let X be a rank-one data matrix.

Original: Then, strong duality holds, i.e., P ReLU ( t ) =
Revised: Then, strong duality holds, i.e., P ReLU ( t ) =

Original: D ReLU ( t ) .
Revised: D ReLU ( t ) .

Original: Suppose that λ ∗ is the optimal solution to the dual problem D ReLU ( t ) , then the optimal weights for each layer can be formulated as
Revised: Suppose that λ ∗ is the optimal solution to the dual problem D ReLU ( t ) , then the optimal weights for each layer can be formulated as

Original: [Equation]
Revised: [Equation]

Original: Here ρ 0 = a 0 / (cid:107) a 0 (cid:107) 2 and ρ 1 ∈ R m l + satisﬁes (cid:107) ρ 1 (cid:107) = 1 .
Revised: Here ρ 0 = a 0 / (cid:107) a 0 (cid:107) 2 and ρ 1 ∈ R m l + satisﬁes (cid:107) ρ 1 (cid:107) = 1 .

Original: Now, we consider the case where the data matrix is whitened, i.e., XX T = I n and Y has orthogonal columns.
Revised: Now, we consider the case where the data matrix is whitened, i.e., XX T = I n and Y has orthogonal columns.

Original: We leave the proof of Theorem 4 and the characterization of optimal solutions in Appendix
Revised: We leave the proof of Theorem 4 and the characterization of optimal solutions in Appendix

Original: C.3.
Revised: C.3.

Original: Theorem 4 Let { X , Y } be a dataset such that XX T = I n and Y has orthogonal columns.
Revised: Theorem 4 Let { X , Y } be a dataset such that XX T = I n and Y has orthogonal columns.

Original: Then, the optimal value of P ReLU ( t ) and D ReLU ( t ) are given by
Revised: Then, the optimal value of P ReLU ( t ) and D ReLU ( t ) are given by

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Suppose that Y is the one-hot encoding of the label.
Revised: Suppose that Y is the one-hot encoding of the label.

Original: Then, we note that y j ≥ 0 and (cid:107) ( y j ) + (cid:107) 2 is the square root of the number of data points in the j -th class.
Revised: Then, we note that y j ≥ 0 and (cid:107) ( y j ) + (cid:107) 2 is the square root of the number of data points in the j -th class.

Original: Therefore, we recover the result about the duality gap of standard three-layer ReLU networks.
Revised: Therefore, we recover the result about the duality gap of standard three-layer ReLU networks.

Original: 4. P ARALLEL DEEP R E LU NETWORKS
Revised: 4. P ARALLEL DEEP R E LU NETWORKS

Original: For the parallel architecture, we show that there is no duality gap for arbitrary deep ReLU network.
Revised: For the parallel architecture, we show that there is no duality gap for arbitrary deep ReLU network with large enough number of branches.

Original: Consider the following minimal norm problem:
Revised: Consider the following minimal norm problem:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proposition 6 The problem (28) can be reformulated as
Revised: Proposition 6 The problem (28) can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: The dual problem (29) is a convex problem deﬁned as
Revised: The dual problem (29) is a convex problem deﬁned as

Original: [Equation]
Revised: [Equation]

Original: For deep ReLU network with parallel architecture, the strong duality holds.
Revised: For deep ReLU network with parallel architecture, the strong duality holds.

Original: Theorem 

Revised: Theorem

Original: The strong duality holds for (29) in the sense that P prlReLU =
Revised: The strong duality holds for (29) in the sense that P prlReLU =

Original: D prlReLU  .
Revised: D prlReLU for m ≥ m ∗ , where m ∗ is upper bounded by KN + 1 .

Original: Similar to case of parallel deep linear networks, the parallel deep ReLU network also achieves zeroduality gap.
Revised: Similar to case of parallel deep linear networks, the parallel deep ReLU network also achieves zeroduality gap.

Original: Therefore, to ﬁnd the global optimum for parallel deep ReLU network is equivalent to solve a convex program.
Revised: Therefore, to ﬁnd the global optimum for parallel deep ReLU network is equivalent to solve a convex program.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present the convex duality framework for standard neural networks, considering both multi-layer linear networks and three-layer ReLU networks with rank-1 or whitened data.
Revised: We present the convex duality framework for standard neural networks, considering both multi-layer linear networks and three-layer ReLU networks with rank-1 or whitened data.

Original: In stark contrast to the two-layer case, the duality gap can be non-zero for neural networks with depth three or more.
Revised: In stark contrast to the two-layer case, the duality gap can be non-zero for neural networks with depth three or more.

Original: Meanwhile, for neural networks with parallel architecture, with the regularization of L -th power of
Revised: Meanwhile, for neural networks with parallel architecture, with the regularization of L -th power of

Original: Frobenius norm in the parameters, we show that strong duality holds and the duality gap reduces to zero.
Revised: Frobenius norm in the parameters, we show that strong duality holds and the duality gap reduces to zero.

Original: A limitation of our work is that we primarily focus on minimal norm interpolation problems.
Revised: A limitation of our work is that we primarily focus on minimal norm interpolation problems.

Original: We expect to generalize our results to general regularized training problems.  
Revised: We expect to generalize our results to general regularized training problems.  A CKNOWLEDGEMENTS

Revised: This work was partially supported by the National Science Foundation under grants ECCS-2037304,

Revised: DMS-2134248, the Army Research Ofﬁce.

Original: A S TAIRS OF DUALITY GAP FOR STANDARD DEEP LINEAR NETWORKS
Revised: A S TAIRS OF DUALITY GAP FOR STANDARD DEEP LINEAR NETWORKS

Original: Now we consider partially dualizing the non-convex optimization problem by exchanging a subset of the minimization problems with respect to the hidden layers.
Revised: Now we consider partially dualizing the non-convex optimization problem by exchanging a subset of the minimization problems with respect to the hidden layers.

Original: Consider the Lagrangian for the primal problem of standard deep linear network
Revised: Consider the Lagrangian for the primal problem of standard deep linear network

Original: [Equation]
Revised: [Equation]

Original: By changing the order of L − 2 min s and the max in (31) [Equation] for l = 0 , 1 , . . .
Revised: By changing the order of L − 2 min s and the max in (31), for l = 0 , 1 , . . .

Original: , L − 2 , we can deﬁne the l -th partial “dual” problem
Revised: , L − 2 , we can deﬁne the l -th partial “dual” problem

Original: [Equation]
Revised: [Equation]

Original: For l = 0 , D ( l ) lin ( t ) corresponds the primal problem P lin ( t ) , while for l = L − 2 , D ( l ) lin ( t ) is the dual problem D lin ( t ) .
Revised: For l = 0 , D ( l ) lin ( t ) corresponds the primal problem P lin ( t ) , while for l = L − 2 , D ( l ) lin ( t ) is the dual problem D lin ( t ) .

Original: From the following proposition, we illustrate that the dual problem of D ( l ) lin ( t ) corresponds to a minimal norm problem of a neural network with parallel structure.
Revised: From the following proposition, we illustrate that the dual problem of D ( l ) lin ( t ) corresponds to a minimal norm problem of a neural network with parallel structure.

Original: Proposition 7 There exists m ∗ < ∞ such that the problem D ( l ) lin ( t ) is equivalent to the “bi-dual” problem
Revised: Proposition 7 There exists m ∗ ≤ KN + 1 such that the problem D ( l ) lin ( t ) is equivalent to the “bidual” problem

Original: [Equation]
Revised: [Equation]

Original: 33) where the variables are W i ∈
Revised: 33) where the variables are W i ∈

Original: R m i − 1 ×
Revised: R m i − 1 ×

Original: m i for i ∈
Revised: m i for i ∈

Original: [ l ] , W i,j ∈ R m i − 1 × m i for i =
Revised: [ l ] , W i,j ∈ R m i − 1 × m i for i =

Original: l + 1 , . . .
Revised: l + 1 , . . .

Original: , L − 2 , j ∈ [ m ∗ ] , W L − 1 ∈ R m L − 2 × m ∗ and W L ∈ R m ∗ × m L .
Revised: , L − 2 , j ∈ [ m ∗ ] , W L − 1 ∈ R m L − 2 × m ∗ and W L ∈ R m ∗ × m L .

Original: We can interpret the problem (33) as the minimal norm problem of a linear network with parallel structures in ( l + 1) -th to ( L − 2) -th layers.
Revised: We can interpret the problem (33) as the minimal norm problem of a linear network with parallel structures in ( l + 1) -th to ( L − 2) -th layers.

Original: This indicates that for l = 0 , 1 , . . .
Revised: This indicates that for l = 0 , 1 , . . .

Original: , L − 2 , the bi-dual formulation of D ( l ) lin ( t ) can be viewed as an interpolation from a network with standard structure to a network with parallel structure.
Revised: , L − 2 , the bi-dual formulation of D ( l ) lin ( t ) can be viewed as an interpolation from a network with standard structure to a network with parallel structure.

Original: Now, we calculate the exact value of D ( l ) lin ( t ) .
Revised: Now, we calculate the exact value of D ( l ) lin ( t ) .

Original: Proposition 8 The optimal value D ( l ) lin ( t ) follows
Revised: Proposition 8 The optimal value D ( l ) lin ( t ) follows

Original: [Equation]
Revised: [Equation]

Original: Suppose that the eigenvalues X † Y are not identical to each other.
Revised: Suppose that the eigenvalues X † Y are not identical to each other.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: In Figure 3, we plot D (l )
Revised: In Figure 3, we plot D (

Original: [Table]
Revised: [Table]

Original: B P ROOFS OF MAIN RESULTS FOR LINEAR NETWORKS B. P ROOF OF P ROPOSITION
Revised: B P ROOFS OF MAIN RESULTS FOR LINEAR NETWORKS B. P ROOF OF P ROPOSITION

Original: Consider the Lagrangian function
Revised: Consider the Lagrangian function

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here Λ ∈ R N × K is the dual variable.
Revised: Here Λ ∈ R N × K is the dual variable.

Original: We note that
Revised: We note that

Original: [Equation]
Revised: [Equation]

Original: Here I ( A ) is 0 if the statement A is true.
Revised: Here I ( A ) is 0 if the statement A is true.

Original: Otherwise it is + ∞ .
Revised: Otherwise it is + ∞ .

Original: For ﬁxed W 1 , . . .
Revised: For ﬁxed W 1 , . . .

Original: , W L − 1 , the constraint on W L is linear so we can exchange the order of max Λ and min W L in the second line of (37).
Revised: , W L − 1 , the constraint on W L is linear so we can exchange the order of max Λ and min W L in the second line of (37).

Original: By exchanging the order of min and max , we obtain the dual problem
Revised: By exchanging the order of min and max , we obtain the dual problem

Original: [Equation]
Revised: [Equation]

Original: Now we derive the bi-dual problem.
Revised: Now we derive the bi-dual problem.

Original: The dual problem can be reformulated as
Revised: The dual problem can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: Here the set Θ is deﬁned as
Revised: Here the set Θ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: By writing θ =
Revised: By writing θ =

Original: ( W
Revised: ( W

Original: [Equation]
Revised: [Equation]

Original: , W L − 2 , w L − 1 )
Revised: , W L − 2 , w L − 1 )

Original: [Equation]
Revised: [Equation]

Original: the dual of the problem (39) is given by
Revised: the dual of the problem (39) is given by

Original: [Equation]
Revised: [Equation]

Original: Here µ : Σ → R K is a signed vector measure and Σ is a σ -ﬁeld of subsets of Θ .
Revised: Here µ : Σ → R K is a signed vector measure and Σ is a σ -ﬁeld of subsets of Θ .

Original: The norm (cid:107) µ (cid:107) TV is the total variation of µ
Revised: The norm (cid:107) µ (cid:107) TV is the total variation of µ

Original: [Equation]
Revised: [Equation]

Original: which can be calculated by
Revised: which can be calculated by

Original: [Equation]
Revised: [Equation]

Original: where we write µ =  µ
Revised: where we write µ =  

Original: [Equation]
Revised: [Equation]

Original: µ K 
Revised: µ K 

Original:  .
Revised:  .

Original: The formulation in (41) has inﬁnite width in each layer.
Revised: The formulation in (41) has inﬁnite width in each layer.

Original: According to
Revised: According to

Original: Theorem 9 in Appendix E, the measure µ in the integral can be represented by ﬁnitely many Dirac delta functions.
Revised: Theorem 9 in Appendix E, the measure µ in the integral can be represented by ﬁnitely many Dirac delta functions.

Original: Therefore, we can rewrite the problem (41) as
Revised: Therefore, we can rewrite the problem (41) as

Original: [Equation]
Revised: [Equation]

Original: Here the variables are W i,j for i ∈
Revised: Here the variables are W i,j for i ∈

Original: [ L − 2] and j ∈ [ m ∗ ] , W L − 1 and W L .
Revised: [ L − 2] and j ∈ [ m ∗ ] , W L − 1 and W L .

Original: As the strong duality holds for the problem (43) and (39), we can reformulate the problem of D lin ( t ) as the bi-dual problem (43).
Revised: As the strong duality holds for the problem (43) and (39), we can reformulate the problem of D lin ( t ) as the bi-dual problem (43).

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: We restate Proposition 2 with details.
Revised: We restate Proposition 2 with details.

Original: Proposition 9 Suppose that W ∈ R d × K with rank r is given.
Revised: Proposition 9 Suppose that W ∈ R d × K with rank r is given.

Original: Consider the following optimization problem:
Revised: Consider the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: in variables W i ∈
Revised: in variables W i ∈

Original: R m i − 1 × m i .
Revised: R m i − 1 × m i .

Original: Here m 

Revised: Here m

Original: = d , m L = K and m i ≥ r for i = 1 , . . .
Revised: = d , m L = K and m i ≥ r for i = 1 , . . .

Original: , L − 1 .
Revised: , L − 1 .

Original: Then, the optimal value of the problem (44) is given by
Revised: Then, the optimal value of the problem (44) is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Suppose that W = UΣV T .
Revised: Suppose that W = UΣV T .

Original: The optimal value can be achieved when
Revised: The optimal value can be achieved when

Original: [Equation]
Revised: [Equation]

Original: Here U i ∈
Revised: Here U i ∈

Original: R r × m i satisﬁes that U
Revised: R r × m i satisﬁes that U

Original: Ti U
Revised: Ti U

Original: i
Revised: i

Original: =
Revised: =

Original: I .
Revised: I .

Original: We start with two lemmas.
Revised: We start with two lemmas.

Original: Lemma 1 Suppose that A ∈ S n × n is a positive semi-deﬁnite matrix.
Revised: Lemma 1 Suppose that A ∈ S n × n is a positive semi-deﬁnite matrix.

Original: Then, for any 0 < p < 1 , we have
Revised: Then, for any 0 < p < 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Here λ i is the i -th largest eigenvalue of A .
Revised: Here λ i is the i -th largest eigenvalue of A .

Original: Lemma 2 Suppose that P ∈ R d × d is a projection matrix.
Revised: Lemma 2 Suppose that P ∈ R d × d is a projection matrix.

Original: Then, for arbitrary W ∈ R d × K , we have
Revised: Then, for arbitrary W ∈ R d × K , we have

Original: [Equation]
Revised: [Equation]

Original: where σ i ( W ) represents the i -th largest singular value of W .
Revised: where σ i ( W ) represents the i -th largest singular value of W .

Original: Now, we present the proof for Proposition 2.
Revised: Now, we present the proof for Proposition 2.

Original: For L = 1 , the statement apparently holds.
Revised: For L = 1 , the statement apparently holds.

Original: Suppose that for L = l this statement holds.
Revised: Suppose that for L = l this statement holds.

Original: For L = l + 1 , by writing A = W 2 . . .
Revised: For L = l + 1 , by writing A = W 2 . . .

Original: W l +1 , we have
Revised: W l +1 , we have

Original: [Equation]
Revised: [Equation]

Original: Suppose that t is ﬁxed.
Revised: Suppose that t is ﬁxed.

Original: It is sufﬁcient to consider the following problem:
Revised: It is sufﬁcient to consider the following problem:

Original: [Equation]
Revised: [Equation]

Original: t. Suppose that there exists W 1 and A such that W = W 1 A .
Revised: Suppose that there exists W 1 and A such that W = W 1 A .

Original: Then, we have WA † A =
Revised: Then, we have WA † A =

Original: W 1 AA † A
Revised: W 1 AA † A

Original: = W .
Revised: = W .

Original: As WA † = W 1 AA † , according to Lemma 2, (cid:107) WA † (cid:107) F ≤ (cid:107) W 1 (cid:107) F ≤ t .
Revised: As WA † = W 1 AA † , according to Lemma 2, (cid:107) WA † (cid:107) F ≤ (cid:107) W 1 (cid:107) F ≤ t .

Original: Therefore, ( WA † , A ) is also feasible for the problem (49).
Revised: Therefore, ( WA † , A ) is also feasible for the problem (49).

Original: Hence, the problem (49) is equivalent to
Revised: Hence, the problem (49) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Assume that W is with rank r .
Revised: Assume that W is with rank r .

Original: Suppose that A = UΣV T , where Σ ∈ R r 0 × r 0 .
Revised: Suppose that A = UΣV T , where Σ ∈ R r 0 × r 0 .

Original: Here r 0 ≥ r .
Revised: Here r 0 ≥ r .

Original: Then, we have A † =
Revised: Then, we have A † =

Original: VΣ − 1 U T .
Revised: VΣ − 1 U T .

Original: We note that
Revised: We note that

Original: [Equation]
Revised: [Equation]

Original: Denote G ( V ) =
Revised: Denote G ( V ) =

Original: V T W T WV .
Revised: V T W T WV .

Original: This implies that r (cid:88) i =
Revised: This implies that r (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: i ( A )
Revised: i ( A )

Original: [Equation]
Revised: [Equation]

Original: ( G ( V )) ii ≤ t 2 .
Revised: ( G ( V )) ii ≤ t 2 .

Original: Therefore, we have (cid:32)
Revised: Therefore, we have (cid:32)

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: i ( A )
Revised: i ( A )

Original: [Equation]
Revised: [Equation]

Original: ( G ( V
Revised: ( G ( V

Original: [Equation]
Revised: [Equation]

Original: ) ii (cid:33)
Revised: ) ii (cid:33)

Original: [Equation]
Revised: [Equation]

Original: ( A
Revised: ( A

Original: [Equation]
Revised: [Equation]

Original: (cid:33)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ( G ( V ))
Revised: ( G ( V ))

Original: [Equation]
Revised: [Equation]

Original: ( l +
Revised: ( l +

Original: [Equation]
Revised: [Equation]

Original: (cid:33)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: As WV T V = W , the non-zero eigenvalues of G ( V ) are exactly the non-zero eigenvalues of
Revised: As WV T V = W , the non-zero eigenvalues of G ( V ) are exactly the non-zero eigenvalues of

Original: WVV T W T = WW T , i.e., the square of non-zero singular values of W .
Revised: WVV T W T = WW T , i.e., the square of non-zero singular values of W .

Original: From Lemma 1, we have
Revised: From Lemma 1, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This also implies that
Revised: This also implies that

Original: [Equation]
Revised: [Equation]

Original: Suppose that W = (cid:80) ri =1 u i σ i v Ti is the SVD of W .
Revised: Suppose that W = (cid:80) ri =1 u i σ i v Ti is the SVD of W .

Original: We can let
Revised: We can let

Original: [Equation]
Revised: [Equation]

Original: Here (cid:107) ρ i (cid:107) 2 = 1 and ρ
Revised: Here (cid:107) ρ i (cid:107) 2 = 1 and ρ

Original: Ti ρ j = 

Revised: Ti ρ j =

Original: for i (cid:54) =
Revised: for i (cid:54) =

Original: j .
Revised: j .

Original: Then, W 1 A = W and (cid:107) W 1 (cid:107) F ≤ t .
Revised: Then, W 1 A = W and (cid:107) W 1 (cid:107) F ≤ t .

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: In summary, we have
Revised: In summary, we have

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: We ﬁrst compute the optimal value P of the primal problem.
Revised: We ﬁrst compute the optimal value P of the primal problem.

Original: From Proposition 2, the minimal norm problem (7) is equivalent to
Revised: From Proposition 2, the minimal norm problem (7) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: in variable W ∈ R d × K .
Revised: in variable W ∈ R d × K .

Original: According to Lemma 2, for any feasible W satisfying XW = Y , because
Revised: According to Lemma 2, for any feasible W satisfying XW = Y , because

Original: X † XW = X † Y and X † X is a projection matrix, we have
Revised: X † XW = X † Y and X † X is a projection matrix, we have

Original: [Equation]
Revised: [Equation]

Original: We also note that XX † Y =
Revised: We also note that XX † Y =

Original: XX † XW = XW = Y .
Revised: XX † XW = XW = Y .

Original: Therefore, X † Y is also feasible for the problem (57).
Revised: Therefore, X † Y is also feasible for the problem (57).

Original: This indicates that P lin =
Revised: This indicates that P lin =

Original: L 2 (cid:107) X † Y (cid:107) 

Revised: L 2 (cid:107) X † Y (cid:107)

Original: /LS 2 /L
Revised: /LS 2 /L

Original: On the other hand, for a feasible point ( W 1 , . . . , W L ) for P lin ( t ) , we note that ( W 1 /t, . . .
Revised: On the other hand, for a feasible point ( W 1 , . . . , W L ) for P lin ( t ) , we note that ( W 1 /t, . . .

Original: , W L − 2 /t, W L − 1 , t L − 2 W L ) is feasible for P lin (1) .
Revised: , W L − 2 /t, W L − 1 , t L − 2 W L ) is feasible for P lin (1) .

Original: This implies that t L − 2 P lin ( t ) =
Revised: This implies that t L − 2 P lin ( t ) =

Original: P lin (1) , or equivalently, P lin ( t ) =
Revised: P lin (1) , or equivalently, P lin ( t ) =

Original: t − ( L − 2)
Revised: t − ( L − 2)

Original: P lin (1) .
Revised: P lin (1) .

Original: Recall that
Revised: Recall that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This implies that P lin (1) =
Revised: This implies that P lin (1) =

Original: (cid:107) X † Y (cid:107) S 2 /L and
Revised: (cid:107) X † Y (cid:107) S 2 /L and

Original: P lin (
Revised: P lin (

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: For the dual problem D lin ( t ) deﬁned in (38), we note that
Revised: For the dual problem D lin ( t ) deﬁned in (38), we note that

Original: [Equation]
Revised: [Equation]

Original: The equality can be achieved when W l = t u l u Tl +1 for l ∈
Revised: The equality can be achieved when W l = t u l u Tl +1 for l ∈

Original: [ L − 2] , where (cid:107) u l (cid:107) 2 = 1 for l = 1 , . . .
Revised: [ L − 2] , where (cid:107) u l (cid:107) 2 = 1 for l = 1 , . . .

Original: , L − 1 .
Revised: , L − 1 .

Original: Speciﬁcally, we set u L − 1 = w L − 1 and let u 0 as right singular vector corresponds to the largest singular value of Λ T X .
Revised: Speciﬁcally, we set u L − 1 = w L − 1 and let u 0 as right singular vector corresponds to the largest singular value of Λ T X .

Original: Therefore, the constraints on Λ is equivalent to
Revised: Therefore, the constraints on Λ is equivalent to

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus, according to the Von Neumann’s trace inequality, it follows
Revised: Thus, according to the Von Neumann’s trace inequality, it follows

Original: [Equation]
Revised: [Equation]

Original: As a result, we have D lin ( t ) =
Revised: As a result, we have D lin ( t ) =

Original: t − ( L − 2) (cid:107) X † Y (cid:107) ∗ < t − ( L − 2) (cid:107) X † Y (cid:107) S 2 /L
Revised: t − ( L − 2) (cid:107) X † Y (cid:107) ∗ < t − ( L − 2) (cid:107) X † Y (cid:107) S 2 /L

Original: =
Revised: =

Original: P lin ( t ) .
Revised: P lin ( t ) .

Original: Namely, the duality gap exists for the standard neural network.
Revised: Namely, the duality gap exists for the standard neural network.

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Revised: For simplicity, we write W L − 1 ,j = w col L − 1 ,j and W L,j = w row L,j for j ∈ [ m ] .

Revised: For the j -th branch of the parallel network, let ˆ W l,j = α l,j W l,j for l ∈

Original: For the problem (17){+[ L ] . Here α l,j > 0 for l ∈ [ L ] and they satisﬁes that (cid:81) Ll =
+}
Revised: {+[ L ] . Here α l,j > 0 for l ∈ [ L ] and they satisﬁes that (cid:81) Ll =
+}

Revised: α l,j = 1 for j ∈ [ m ] .

Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: we consider the Lagrangian function 
Revised: This implies that { ˆ W l,j } l ∈ [ L ] ,j ∈ [ m ] is also feasible for the problem (14). According to the the inequality of arithmetic and geometric means, the objective function in (14) is lower bounded by

Original: [Equation]
Revised: [Equation]

Original: The primal problem is equivalent to
Revised: The equality is achieved when α l,j = (cid:81) Ll =1 (cid:107) W l,j (cid:107) [Equation] /L

Revised: [Equation]

Original: P
Revised: (cid:107) W l,j (cid:107) [Equation] for l ∈ [ L ] and j ∈ [ m ] . As the scaling operation does not change (cid:81) Ll =1 (cid:107) W l,j (cid:107) 2 /LF , we can simply let (cid:107) W l,j (cid:107) F = 1 and the lower bound becomes

Original: [Equation]
Revised: [Equation]

Revised: This completes the proof.

Revised: B. P ROOF OF P ROPOSITION

Revised: We ﬁrst show that the problem (16) is equivalent to (17).

Original:  = 
Revised: The proof is analogous to the proof of Proposition 3. For simplicity, we write W L − 1 ,j = w col L − 1 ,j and W L,j = w row L,j for j ∈ [ m ] . Let α l,j > 0 for l ∈

Revised: [ L ] and they satisﬁes that (cid:81)

Original: {+Ll =
+}
Revised: {+Ll =
+}

Revised: α l,j = 1 for j ∈ [ m ] .

Revised: Consider another parallel network { ˆ W l,j

Revised: } l ∈ [ L ] ,j ∈ [ m ] whose j -th branch is deﬁned by ˆ W l,j = α l,j W l,j for l ∈

Revised: [ L ] .

Revised: As (cid:81)

Revised: Ll =1 α l,j = 1 , we have

Revised: [Equation]

Revised: This implies that { ˆ W l,j } l ∈ [ L ] ,j ∈ [ m ] is also feasible for the problem (16).

Revised: According to the the inequality of arithmetic and geometric means, the objective function in (14) is lower bounded by

Revised: [Equation]

Revised: The equality is achieved when α l,j =

Revised: [Equation]

Revised: F for l ∈ [ L ] and j ∈ [ m ] .

Revised: As the scaling operation does not change (cid:81) L l =1 (cid:107) W l,j (cid:107) F , we can simply let (cid:107) W l,j (cid:107) F = 1 and the lower bound becomes L 2 (cid:80) mi =1 (cid:107) W L,j (cid:107) F = L

Revised: [Equation]

Revised: (cid:80) mi =1 (cid:107) w row L,j (cid:107) 2 .

Revised: Hence, the problem (16) is equivalent to (17).

Revised: For the problem (17), we consider the Lagrangian function

Revised: [Equation]

Revised: The primal problem is equivalent to

Original: [Equation]
Revised: [Equation]

Original: The dual problem follows
Revised: The dual problem follows

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: We can rewrite the dual problem as
Revised: We can rewrite the dual problem as

Original: [Equation]
Revised: [Equation]

Original: where the set Θ is deﬁned as
Revised: where the set Θ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: By writing θ =
Revised: By writing θ =

Original: ( W 1 , . . .
Revised: ( W 1 , . . .

Original: , W L − 2 , w L − 1 ) , the bi-dual problem, i.e., the dual problem of (67), is given by
Revised: , W L − 2 , w L − 1 ) , the bi-dual problem, i.e., the dual problem of (71), is given by

Original: [Equation]
Revised: [Equation]

Original: Here µ : Σ → R K is a signed vector measure, where Σ is a σ -ﬁeld of subsets of Θ and (cid:107) µ (cid:107) TV is its total variation.
Revised: Here µ : Σ → R K is a signed vector measure, where Σ is a σ -ﬁeld of subsets of Θ and (cid:107) µ (cid:107) TV is its total variation.

Original: The formulation in (69) has inﬁnite width in each layer.
Revised: The formulation in (73) has inﬁnite width in each layer.

Original: According to Theorem 9 in
Revised: According to Theorem 9 in

Original: Appendix E, the measure µ in the integral can be represented by ﬁnitely many Dirac delta functions.
Revised: Appendix E, the measure µ in the integral can be represented by ﬁnitely many Dirac delta functions.

Original: Therefore, there exists m ∗ < ∞ (more precisely, m ∗ ≤ n + 1) such that we can rewrite the problem (69) as
Revised: Therefore, there exists m ∗ < KN + 1 such that we can rewrite the problem (73) as

Original: [Equation]
Revised: [Equation]

Original: Here the variables are W l,j for l ∈
Revised: Here the variables are W l,j for l ∈

Original: [ L − 2] and j ∈ [ m ∗ ] , W L − 1 and W L .
Revised: [ L − 2] and j ∈ [ m ∗ ] , W L − 1 and W L .

Original: This is equivalent to (17).
Revised: This is equivalent to (17).

Original: As the strong duality holds for the problem (67) and (69), the primal problem (17) is equivalent to the dual problem (67) as long as m ≥ m ∗ .
Revised: As the strong duality holds for the problem (71) and (73), the primal problem (17) is equivalent to the dual problem (71) as long as m ≥ m ∗ .

Original: Now, we compute the optimal value of D prllin .
Revised: Now, we compute the optimal value of D prllin .

Original: Similar to the proof of Theorem 1, we can show that the constraints in the dual problem (67) is equivalent to
Revised: Similar to the proof of Theorem 1, we can show that the constraints in the dual problem (71) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: This implies that P prllin =
Revised: This implies that P prllin =

Original: D prllin =
Revised: D prllin =

Original: L 2 (cid:107) X [Equation] Y (cid:107) ∗ .
Revised: L 2 (cid:107) X † Y (cid:107) ∗ .

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: We note that
Revised: We note that

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can rewrite D ( l ) lin ( t)( t ) as
Revised: Therefore [Equation] we can rewrite D ( l ) lin ( t ) as

Original: [Equation]
Revised: [Equation]

Original: From the equation (11) [Equation] we note that
Revised: From the equation (11), we note that

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: C P ROOFS OF MAIN RESULTS FOR R E LU NETWORKS C. P ROOF OF P ROPOSITION
Revised: C P ROOFS OF MAIN RESULTS FOR R E LU NETWORKS C. P ROOF OF P ROPOSITION

Original: For the problem of P ( t ) [Equation] introduce the Lagrangian function
Revised: For the problem of P ( t ) , introduce the Lagrangian function

Original: [Equation]
Revised: [Equation]

Original: According to the convex duality of two-layer ReLU network, we have
Revised: According to the convex duality of two-layer ReLU network

Original: [Equation]
Revised: [Equation]

Original: By changing the min and max
Revised: we have

Original: [Equation]
Revised: [Equation]

Original:  we obtain the dual problem
Revised: By changing the min and max, we obtain the dual problem.

Original: [Equation]
Revised: [Equation]

Original: The dual of the dual problem writes
Revised: The dual of the dual problem writes

Original: [Equation]
Revised: [Equation]

Original: Here µ is a signed vector measure and (cid:107) µ (cid:107) TV is its total variation.
Revised: Here µ is a signed vector measure and (cid:107) µ (cid:107) TV is its total variation.

Original: Similar to the proof of Proposition 1, we can ﬁnd a ﬁnite representation for the optimal measure and transform this problem to
Revised: Similar to the proof of Proposition 1, we can ﬁnd a ﬁnite representation for the optimal measure and transform this problem to

Original: [Equation]
Revised: [Equation]

Revised: Here m ∗ ≤ KN + 1 .

Original: This completes the proof.
Revised: This completes the proof.

Original: C. P ROOF OF T HEOREM
Revised: C. P ROOF OF T HEOREM

Original: For rank-1 data matrix that X
Revised: For rank-1 data matrix that X

Original: = ca T 0 , suppose that A 1 = ( XW 1 ) + .
Revised: = ca T 0 , suppose that A 1 = ( XW 1 ) + .

Original: It is easy to observe that
Revised: It is easy to observe that

Original: [Equation]
Revised: [Equation]

Original: Here we let a 1 , + =
Revised: Here we let a 1 , + =

Original: ( W 1 a 0 )
Revised: ( W 1 a 0 )

Original: + and a 0 , − = ( − W 1 a 0 )
Revised: + and a 0 , − = ( − W 1 a 0 )

Original: + .
Revised: + .

Original: For a three-layer network, i.e., L = 3 , suppose that λ ∗ is the optimal solution to the dual problem
Revised: For a three-layer network, i.e., L = 3 , suppose that λ ∗ is the optimal solution to the dual problem

Original: D ReLU ( t ) .
Revised: D ReLU ( t ) .

Original: We consider the extreme points
Revised: We consider the extreme points

Original: [Equation]
Revised: [Equation]

Original: For ﬁxed W 1 , because a T 1 , + a 1 , − = 0 , suppose that[Equation] −
Revised: For ﬁxed W 1 , because a T 1 , + a 1 , − = 0 , suppose that

Original: [Equation]
Revised: [Equation]

Original: where r T a 1 , + = r T a 1 , − = 

Revised: where r T a 1 , + = r T a 1 , − =

Original: and (cid:107) r (cid:107) 2 = 1 .
Revised: and (cid:107) r (cid:107) 2 = 1 .

Original: The maximization problem on w 2 becomes
Revised: The maximization problem on w 2 becomes

Original: [Equation]
Revised: [Equation]

Original: 22 + u 23 ≤ 1 .
Revised: [Equation]

Original: If ( λ ∗ ) T ( c ) + and ( λ ∗ ) T ( − c ) + have different signs, then the optimal value is
Revised: If ( λ ∗ ) T ( c ) + and ( λ ∗ ) T ( − c ) + have different signs, then the optimal value is

Original: [Equation]
Revised: [Equation]

Original: And the corresponding optimal w L − 1 is w L −
Revised: And the corresponding optimal w L − 1 is w L −

Original: [Equation]
Revised: [Equation]

Revised: 2 , + /

Original: [Equation]
Revised: [Equation]

Original:  L 
Revised: a L − 2 , +

Original: [Equation]
Revised: [Equation]

Original: , + (cid:107) or w L −
Revised: or w L −

Original: [Equation]
Revised: [Equation]

Original: a L − 2 , − / (cid:107) a L − 2 , − (cid:107) .
Revised: a L − 2 , − / (cid:107) a L − 2 , − (cid:107) .

Original: Then, the problem becomes
Revised: Then, the problem becomes

Original: [Equation]
Revised: [Equation]

Original: We note that
Revised: We note that

Original: [Equation]
Revised: [Equation]

Original: Thus the optimal W 1 follows
Revised: Thus the optimal W 1 follows

Original: [Equation]
Revised: [Equation]

Original: Here ρ 0 = a 0 / (cid:107) a 0 (cid:107) 2 and ρ 1 ∈ R m l + satisﬁes (cid:107) ρ 1 (cid:107) = 1 .
Revised: Here ρ 0 = a 0 / (cid:107) a 0 (cid:107) 2 and ρ 1 ∈ R m l + satisﬁes (cid:107) ρ 1 (cid:107) = 1 .

Original: This also gives the optimal w 2 = ρ 1 .
Revised: This also gives the optimal w 2 = ρ 1 .

Original: On the other hand, if ( λ ∗ ) T ( c ) + and ( λ ∗ ) T ( − c ) + have same signs, then, the optimal w 2 follows
Revised: On the other hand [Equation] if ( λ ∗ ) T ( c ) + and ( λ ∗ ) T ( − c ) + have same signs, then, the optimal w 2 follows

Original: [Equation]
Revised: [Equation]

Original: The maximization problem is equivalent to
Revised: The maximization problem is equivalent to

Original: [Equation]
Revised: [Equation]

Original: By noting that
Revised: By noting that

Original: [Equation]
Revised: [Equation]

Original:  the optimal W 1 follows 
Revised: 22 ≤ (cid:107) W 1 (cid:107) 22 (cid:107) a 0 (cid:107) 22 ≤ ( t ∗ ) 2 (cid:107) a 0 (cid:107) 22 , the optimal W 1 follows W 1 = t sign

Original: [Equation]
Revised: [Equation]

Original: Here ρ 0 = a 0 / (cid:107) a 0 (cid:107) 2 and ρ 1 ∈ R m 1 + satisﬁes (cid:107) ρ 1 (cid:107) = 1 .
Revised: Here ρ 0 = a 0 / (cid:107) a 0 (cid:107) 2 and ρ 1 ∈ R m 1 + satisﬁes (cid:107) ρ 1 (cid:107) = 1 .

Original: This also gives the optimal w 2 = ρ 1 .
Revised: This also gives the optimal w 2 = ρ 1 .

Original: C. P ROOF OF T HEOREM
Revised: C. P ROOF OF T HEOREM

Original: We restate Theorem 4 with details as follows.
Revised: We restate Theorem 4 with details as follows.

Original: Theorem 6 Let { X , Y } be a dataset such that XX T = I n and Y has orthogonal columns.
Revised: Theorem 6 Let { X , Y } be a dataset such that XX T = I n and Y has orthogonal columns.

Original: Then, the optimal value of P ReLU ( t ) and D ReLU ( t ) are given by
Revised: Then, the optimal value of P ReLU ( t ) and D ReLU ( t ) are given by

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: For the bi-dual formulation of D ReLU ( t ) deﬁned in (25) , the optimal weight matrices for each layer can be constructed as
Revised: For the bi-dual formulation of D ReLU ( t ) deﬁned in (25) , the optimal weight matrices for each layer can be constructed as

Original: [Equation]
Revised: [Equation]

Original: for r = 1 , . . .
Revised: for r = 1 , . . .

Original: , 2 K .
Revised: , 2 K .

Original: Here ( φ 0 , 2 j − 1 , φ 0 , 2 j) =
Revised: Here ( φ 0 , 2 j − 1 , [Equation] y j

Original: (cid:0) X T ( y j ) + , X T ( − y j ) + (cid:1) , (cid:107) φ 1 ,r (cid:107) =
Revised: [Equation]

Original: t , φ 1 ,r ∈ R m 1 + .
Revised: + [Equation]

Original: For P ReLU ( t) , the optimal weight matrices for each layer write
Revised: t

Original: [Equation]
Revised: [Equation]

Original: . , φ[-1 , 2 K (cid:3) . Here g 2 j +
 =-]
Revised: φ[-1 , 2 K (cid:3) . Here g 2 j +
 =-]

Original: [Equation]
Revised: [Equation]

Revised: r ∈ R m 1 + .

Original: (cid:16)(cid:80) [Equation] [Equation] /, g
Revised: For P ReLU ( t ) , the optimal weight matrices for each layer write

Original: [Equation]
Revised: [Equation]

Original: j +
Revised: Here g

Original: [Equation]
Revised: [Equation]

Original: = [Equation] We require that (cid:107) φ 1 ,j (cid:107) 2 = 1 for j ∈ [2 K ] and φ T 1 ,i φ 1 ,j = 0 for i (cid:54) =
Revised: We require that (cid:107) φ 1 ,j (cid:107) 2 = 1 for j ∈ [2 K ] and φ T 1 ,i φ 1 ,j = 0 for i (cid:54) =

Original: j .
Revised: j .

Original: Since XX T = I n , we can characterize the set as
Revised: Since XX T = I n , we can characterize the set as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here A ( W 1 ) is deﬁned in (23).
Revised: Here A ( W 1 ) is deﬁned in (23).

Original: For one thing, for z ∈ R N , (cid:107) z (cid:107) 2 ≤ t , we can let W 

Revised: For one thing, for z ∈ R N , (cid:107) z (cid:107) 2 ≤ t , we can let W

Original: =
Revised: =

Original: X T zv T and w 2 = v , where v ∈ R m 1 + with (cid:107) v (cid:107) 2 = 1 .
Revised: X T zv T and w 2 = v , where v ∈ R m 1 + with (cid:107) v (cid:107) 2 = 1 .

Original: Then, (cid:107) W 1 (cid:107) F ≤ t , (cid:107) w 2 (cid:107) 2 ≤ 1 and ( XW 1 ) + w 2 )
Revised: Then, (cid:107) W 1 (cid:107) F ≤ t , (cid:107) w 2 (cid:107) 2 ≤ 1 and ( XW 1 ) + w 2 )

Original: + = ( z ) + .
Revised: + = ( z ) + .

Original: For another, for any (cid:107) W 1 (cid:107) F ≤ t and (cid:107) w 2 (cid:107) ≤ 1 , we note that (cid:107) (
Revised: For another, for any (cid:107) W 1 (cid:107) F ≤ t and (cid:107) w 2 (cid:107) ≤ 1 , we note that (cid:107) [Equation] XW 1 )

Original: [Equation]
Revised: [Equation]

Original: ) + w 2 (cid:107) 2 ≤ (cid:107) ( XW 1 ) + (cid:107) 2 ≤ (cid:107) ( XW 1 ) + (cid:107) F ≤ (cid:107) XW 1 (cid:107) F ≤ (cid:107) W 1 (cid:107) F ≤ t.
Revised: w [Equation] 2 ≤ (cid:107) ( XW 1 ) + (cid:107) 2 ≤ (cid:107) ( XW 1 ) + (cid:107) F ≤ (cid:107) XW 1 (cid:107) F ≤ (cid:107) W 1 (cid:107) F ≤ t.

Original: Therefore, ∪ (cid:107) W (cid:107) F ≤ t A ( W ) = { ( z ) + | z ∈ R N , (cid:107) z (cid:107) 2 ≤ t } .
Revised: Therefore, ∪ (cid:107) W (cid:107) F ≤ t A ( W ) = { ( z ) + | z ∈ R N , (cid:107) z (cid:107) 2 ≤ t } .

Original: Thus, the constraint on Λ in the dual problem (24) is equivalent to say that t (cid:107) ( Λ ∗ ) T ( z ) + (cid:107) 2 ≤ 1 for all z ∈ R N satisfying (cid:107) z (cid:107) 2 ≤ 1 .
Revised: Thus, the constraint on Λ in the dual problem (24) is equivalent to say that t (cid:107) ( Λ ∗ ) T ( z ) + (cid:107) 2 ≤ 1 for all z ∈ R N satisfying (cid:107) z (cid:107) 2 ≤ 1 .

Original: As Y has orthogonal columns, we note thattr
Revised: As Y has orthogonal columns, we note that

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation] ≤ [Equation] Suppose that ( y j ) + (cid:54) = 0 and ( − y j ) + (cid:54) = 0 for all j = 1 , . .
Revised: Suppose that ( y j ) + (cid:54) = 0 and ( − y j ) + (cid:54) = 0 for all j = 1 , . .

Original: , K .
Revised: , K .

Original: Then, the dual problem is minimized by 
Revised: Then, the dual problem is minimized by Λ ∗ =

Original: [Equation]
Revised: t − 1 (cid:20) ( y 1 ) + (cid:107) ( y 1 ) + (cid:107) 2 − ( − y 1 ) + (cid:107) ( − y 1 ) + (cid:107) 2 , . . . , ( y K ) + (cid:107) ( y K ) + (cid:107) 2 − ( − y K ) + (cid:107) ( − y K ) + (cid:107)(cid:21) .

Original: We can also verify that
Revised: We can also verify that

Original: [Equation]
Revised: [Equation]

Original:  Suppose that ( z ) + = (cid:80) Kj =1 α j, 1 ( y j ) + (cid:107) ( y j ) + (cid:107) 2 + α j, 2 ( − y j ) + (cid:107) ( − y j ) + (cid:107) 2 + r , where r ∈ R N + is orthogonal to ( y j ) + and ( − y j ) + .
Revised: ≤ 1 . Suppose that ( z ) + = (cid:80) Kj [Equation]

Original: Here α j, 1, α j, 2 ≥ 0 .
Revised: 1 α [Equation] 2 + [Equation]

Original: As (cid:107) ( z ) + (cid:107) ≤
Revised: + (cid:107) [Equation] y

Original: [Equation]
Revised: [Equation]

Original: z (cid:107) 2 ≤ 1 , we have
Revised: + (cid:107) 2 + r , where r ∈ R N + is orthogonal to ( y j ) + and ( − y j )

Original: K
Revised: + . Here α j, 1 , α j, 2 ≥ 0 . As (cid:107) ( z ) + (cid:107) ≤ (cid:107) z

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Revised: 2 ≤ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: We note that
Revised: We note that

Original: [Equation]
Revised: [Equation]

Original: Thus, Λ ∗ satisﬁes the constraint (85).
Revised: Thus, Λ ∗ satisﬁes the constraint (89).

Original: We can characterize the optimal layer weight to the bi-dual problem as the extreme points that solves
Revised: We can characterize the optimal layer weight to the bi-dual problem as the extreme points that solves

Original: [Equation]
Revised: [Equation]

Original: These extreme points correspond to the constraints
Revised: These extreme points correspond to the constraints

Original: [Equation]
Revised: [Equation]

Original: In other words, the dual problem is equivalent to
Revised: In other words, the dual problem is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Now, we consider an arbitrary matrix W 1 satisfying (cid:107) W 1 (cid:107) F ≤ t .
Revised: Now, we consider an arbitrary matrix W 1 satisfying (cid:107) W 1 (cid:107) F ≤ t .

Original: Denote
Revised: Denote

Original: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Suppose that Λ ∗ is the optimal solution.
Revised: Suppose that Λ ∗ is the optimal solution.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: For a vector λ ∈ R N , deﬁne
Revised: For a vector λ ∈ R N , deﬁne

Original: [Equation]
Revised: [Equation]

Original: For a given u ∈ R N + , we want to estimate
Revised: For a given u ∈ R N + , we want to estimate

Original: [Equation]
Revised: [Equation]

Original: The following lemma gives an upper bound.
Revised: The following lemma gives an upper bound.

Original: Lemma 3 Suppose that u ∈ R N satisfying (cid:107) u (cid:107) 2 = 1 .
Revised: Lemma 3 Suppose that u ∈ R N satisfying (cid:107) u (cid:107) 2 = 1 .

Original: Then, for arbitrary λ satisfying g ( λ ; W 1 ) ≤ 1 , we have | u T λ | ≤ 1 /g ( u ; W 1 ) .
Revised: Then, for arbitrary λ satisfying g ( λ ; W 1 ) ≤ 1 , we have | u T λ | ≤ 1 /g ( u ; W 1 ) .

Original: Denote z 2 j +

Revised: Denote z 2 j +

Original: =
Revised: =

Original: ( y j ) + (cid:107) ( y j ) + (cid:107) 2 and z 2 j +2 =
Revised: ( y j ) + (cid:107) ( y j ) + (cid:107) 2 and z 2 j +2 =

Original: ( − y j ) + (cid:107) ( − y j ) + (cid:107) 2 .
Revised: ( − y j ) + (cid:107) ( − y j ) + (cid:107) 2 .

Original: For simplicity, we write g ( z ; W 1 ) =
Revised: For simplicity, we write g ( z ; W 1 ) =

Original: g ( z ) .
Revised: g ( z ) .

Original: We note that λ ∗ j satisﬁes that g ( λ ∗ j ) ≤ 1 .
Revised: We note that λ ∗ j satisﬁes that g ( λ ∗ j ) ≤ 1 .

Original: According to Lemma 2, we have
Revised: According to Lemma 2, we have

Original: [Equation]
Revised: [Equation]

Original: This serves as an upper bound for P ∗ ( W 1 ) .
Revised: This serves as an upper bound for P ∗ ( W 1 ) .

Original: We note that for u ∈ R N + , we have
Revised: We note that for u ∈ R N + , we have

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: According to the Holder’s inequality
Revised: According to the Holder’s inequality

Original: [Equation]
Revised: [Equation]

Original: we have 
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Thus, it follows
Revised: Thus, it follows

Original: [Equation]
Revised: [Equation]

Original: (cid:107) ( y j ) + (cid:107) 2 /g ( z 2 j +2 )) ≥ t −  K[-(cid:88) j = (cid:16) (cid:107) ( y j ) + (cid:107) 2 / 3+ (cid:107) ( − y j ) + (cid:107) 2 / 3 
-]
Revised: K[-(cid:88) j = (cid:16) (cid:107) ( y j ) + (cid:107) 2 / 3+ (cid:107) ( − y j ) + (cid:107) 2 / 3 
-]

Original: (cid:17) 3 /.
Revised: [Equation]

Original: The optimal g ( z j ) follows
Revised: The optimal g ( z j ) follows

Original: [Equation]
Revised: [Equation]

Original: This bound can be achieved for
Revised: This bound can be achieved for

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) φ 1 ,j (cid:107) 2 = 1 for j ∈ [2 K ] and φ T 1 ,i φ 1 ,j = 0 for i (cid:54) =
Revised: where (cid:107) φ 1 ,j (cid:107) 2 = 1 for j ∈ [2 K ] and φ T 1 ,i φ 1 ,j = 0 for i (cid:54) =

Original: j .
Revised: j .

Original: C. P ROOF OF P ROPOSITION
Revised: C. P ROOF OF P ROPOSITION

Original: Similar to the proof of Proposition 4, we can reformulate (28) into (29).
Revised: Analogous to the proof of Proposition 4, we can reformulate (28) into (29).

Original: The rest of the proof is analogous to the proof of Proposition 4.
Revised: The rest of the proof is analogous to the proof of Proposition 4.

Original: For the problem (29), we consider the Lagrangian function
Revised: For the problem (29), we consider the Lagrangian function

Original: L ( W 1 , . .

Original: , W L

Original: [Equation]
Revised: [Equation]

Original: The primal problem is equivalent to
Revised: The primal problem is equivalent to

Original: P prlReLU

Original: [Equation]
Revised: [Equation]

Original: By exchanging the order of min and max
Revised: By exchanging the order of min and max

Original: [Equation]
Revised: [Equation]

Original: the dual problem follows
Revised: the dual problem follows

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF T HEOREM
Revised: C. P ROOF OF T HEOREM

Original: The proof is analogous to the proof of Theorem 2.
Revised: The proof is analogous to the proof of Theorem 2.

Original: We can rewrite the dual problem as
Revised: We can rewrite the dual problem as

Original: [Equation]
Revised: [Equation]

Original: where the set Θ is deﬁned as
Revised: where the set Θ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: By writing θ =
Revised: By writing θ =

Original: ( W 1 , . . .
Revised: ( W 1 , . . .

Original: , W L − 2 , w L − 1 ) , the bi-dual problem, i.e., the dual problem of (67), is given by
Revised: , W L − 2 , w L − 1 ) , the bi-dual problem, i.e., the dual problem of (71), is given by

Original: [Equation]
Revised: [Equation]

Original: Here µ : Σ → R K is a signed vector measure, where Σ is a σ -ﬁeld of subsets of Θ and (cid:107) µ (cid:107) TV is its total variation.
Revised: Here µ : Σ → R K is a signed vector measure, where Σ is a σ -ﬁeld of subsets of Θ and (cid:107) µ (cid:107) TV is its total variation.

Original: The formulation in (97) has inﬁnite width in each layer.
Revised: The formulation in (101) has inﬁnite width in each layer.

Original: According to Theorem 9 in
Revised: According to Theorem 9 in

Original: Appendix E, the measure µ in the integral can be represented by ﬁnitely many Dirac delta functions.
Revised: Appendix E, the measure µ in the integral can be represented by ﬁnitely many Dirac delta functions.

Original: Therefore, there exists m ∗< ∞ (more precisely, m ∗ ≤ n + 1) such that we can rewrite the problem (97) as
Revised: Therefore, there exists m ∗ ≤ KN + 1 such that we can rewrite the problem (101) as

Original: [Equation]
Revised: [Equation]

Original: Here the variables are W l,j for l ∈
Revised: Here the variables are W l,j for l ∈

Original: [ L − 2] and j ∈ [ m ∗ ] , W L − 1 and W L .
Revised: [ L − 2] and j ∈ [ m ∗ ] , W L − 1 and W L .

Original: This is equivalent to (29).
Revised: This is equivalent to (29).

Original: As the strong duality holds for the problem (95) and (97), the primal problem (29) is equivalent to the dual problem (67) as long as m ≥ m ∗ .
Revised: As the strong duality holds for the problem (99) and (101), the primal problem (29) is equivalent to the dual problem (71) as long as m ≥ m ∗ .

Original: D P ROOFS OF AUXILIARY RESULTS D. P ROOF OF L EMMA
Revised: D P ROOFS OF AUXILIARY RESULTS D. P ROOF OF L EMMA

Original: Denote a ∈ R n such that a i =
Revised: Denote a ∈ R n such that a i =

Original: A ii and denote b ∈ R n such that b i =
Revised: A ii and denote b ∈ R n such that b i =

Original: λ
Revised: λ

Original: i ( A ) .
Revised: i ( A ) .

Original: We can show that a is majorized by b , i.e., for k ∈ [ n − 1] , we have k (cid:88) i =a ( i )
Revised: We can show that a is majorized by b , i.e., for k ∈ [ n − 1] , we have k [Equation] =

Original: [Equation]
Revised: [Equation]

Original: k (cid:88) i =b ( i
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (99) and (cid:80) ni
Revised: (103) and (cid:80) ni

Original: =1 a i = (cid:80) ni
Revised: =1 a i = (cid:80) ni

Original: =1 b i .
Revised: =1 b i .

Original: Here a ( i ) is the i -th largest entry in a .
Revised: Here a ( i ) is the i -th largest entry in a .

Original: We ﬁrst note that n (cid:88) i =
Revised: We ﬁrst note that n (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: ( A ) 
Revised: ( A ) =

Original: [Equation]

Original: n (cid:88)i
Revised: n (cid:88)

Original:  = λ
Revised: i = λ

Original: i ( A ) .
Revised: i ( A ) .

Original: On the other hand, for k ∈ [ n − 1] , we havek (cid:88) [Equation]
Revised: On the other hand, for k ∈ [ n − 1] , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, a is majorized by b .
Revised: Therefore, a is majorized by b .

Original: As f ( x ) = − x p is a convex function, according to the Karamata’s inequality, we have n (cid:88) i = 
Revised: As f ( x ) = − x p is a convex function, according to the Karamata’s inequality, we have [Equation] i = f ( a i

Original: f ( a i ) ≤ n (cid:88) i =
Revised: [Equation] ≤ n (cid:88) i =

Original: f ( b i ) .
Revised: f ( b i ) .

Original: This completes the proof.
Revised: This completes the proof.

Original: D. P ROOF OF L EMMA
Revised: D. P ROOF OF L EMMA

Original: According to the min-max principle for singular value, we have
Revised: According to the min-max principle for singular value, we have

Original: [Equation]
Revised: [Equation]

Original: As P is a projection matrix, for arbitrary x ∈ R d , we have (cid:107) PWx (cid:107) 2 ≤ (cid:107)
Revised: As P is a projection matrix, for arbitrary x ∈ R d , we have (cid:107) PWx (cid:107) 2 ≤ (cid:107)

Original: Wx (cid:107) 2 .
Revised: Wx (cid:107) 2 .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: D. P ROOF OF L EMMA
Revised: D. P ROOF OF L EMMA

Original: Suppose that λ ( u ) is optimal solution to
Revised: Suppose that λ ( u ) is optimal solution to

Original: [Equation]
Revised: [Equation]

Original: Denote n e is the number of extreme points of
Revised: Denote n e is the number of extreme points of

Original: [Equation]
Revised: [Equation]

Original: Denote v 1 , . .
Revised: Denote v 1 , . .

Original: . , v n e as extreme points of (101).
Revised: . , v n e as extreme points of (105).

Original: We note that (101) is the dual problem for a scalar output three-layer neural network.
Revised: We note that (105) is the dual problem for a scalar output three-layer neural network.

Original: By applying the previous duality theorem, we can write u = (cid:80) n e i
Revised: By applying the previous duality theorem, we can write u = (cid:80) n e i

Original: =
Revised: =

Original: 1 a i v i , where v i ∈
Revised: 1 a i v i , where v i ∈

Original: A ( W 1 ) .
Revised: A ( W 1 ) .

Original: We also have | u T λ ( u ) | = (cid:80) n e i =1 | a i | .
Revised: We also have | u T λ ( u ) | = (cid:80) n e i =1 | a i | .

Original: Besides, we note that {+1 = (cid:107) u (cid:107) 
 = n e (cid:88) i+}
Revised: Besides, we note that {+1 = (cid:107) u (cid:107) 
 = n e (cid:88) i+}

Original: [Equation]
Revised: = a i u T v i ≤ n e (cid:88) i = | a i | g ( u ; W 1 ) = | u T λ ( u ) | g ( u ; W 1 ) .

Original: This completes the proof.
Revised: This completes the proof.

Original: E C ARATHEODORY ’ S THEOREM AND FINITE REPRESENTATION
Revised: E C ARATHEODORY ’ S THEOREM AND FINITE REPRESENTATION

Original: We ﬁrst review a generalized version of Caratheodory’s theorem introduced in (Rosset et al., 2007).
Revised: We ﬁrst review a generalized version of Caratheodory’s theorem introduced in (Rosset et al., 2007).

Original: Theorem 7 Let µ be a positive measure supported on a bounded subset D ⊆ R N .
Revised: Theorem 7 Let µ be a positive measure supported on a bounded subset D ⊆ R N .

Original: Then, there exists a measure ν whose support is a ﬁnite subset of D , { z 1 , . .
Revised: Then, there exists a measure ν whose support is a ﬁnite subset of D , { z 1 , . .

Original: z k } , with k ≤ N + 1 such that 
Revised: z k } , with k ≤ N + 1 such that (cid:90)

Original: [Equation]
Revised: [Equation]

Original:  and (cid:107) µ (cid:107) TV = (cid:107) ν (cid:107) TV .
Revised: (106) and (cid:107) µ (cid:107) TV = (cid:107) ν (cid:107) TV .

Original: We can generalize this theorem to signed vector measures.
Revised: We can generalize this theorem to signed vector measures.

Original: Theorem 8 Let µ : Σ → R K be a signed vector measure supported on a bounded subset D ⊆ R N .
Revised: Theorem 8 Let µ : Σ → R K be a signed vector measure supported on a bounded subset D ⊆ R N .

Original: Here Σ is a σ -ﬁeld of subsets of D .
Revised: Here Σ is a σ -ﬁeld of subsets of D .

Original: Then, there exists a measure ν whose support is a ﬁnite subset of D , { z 1 , . .
Revised: Then, there exists a measure ν whose support is a ﬁnite subset of D , { z 1 , . .

Original: , z k } , with k ≤ KN + 1 such that
Revised: , z k } , with k ≤ KN + 1 such that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and (cid:107) ν (cid:107) TV = (cid:107) µ (cid:107) TV .
Revised: and (cid:107) ν (cid:107) TV = (cid:107) µ (cid:107) TV .

Original: P ROOF Let µ be a signed vector measure supported on a bounded subset D ⊆ R N .
Revised: P ROOF Let µ be a signed vector measure supported on a bounded subset D ⊆ R N .

Original: Consider the extended set ˜ D
Revised: Consider the extended set ˜ D

Original: = { zu T | z ∈ D, u ∈ R K , (cid:107) u (cid:107) 2 = 1 } .
Revised: = { zu T | z ∈ D, u ∈ R K , (cid:107) u (cid:107) 2 = 1 } .

Original: Then, µ corresponds to a scalar-valued measure ˜ µ on the set ˜ D and (cid:107) µ (cid:107) TV =
Revised: Then, µ corresponds to a scalar-valued measure ˜ µ on the set ˜ D and (cid:107) µ (cid:107) TV =

Original: (cid:107) ˜
Revised: (cid:107) ˜

Original: µ
Revised: µ

Original: (cid:107) TV .
Revised: (cid:107) TV .

Original: We note that ˜ D is also bounded.
Revised: We note that ˜ D is also bounded.

Original: Therefore, by applying Theorem 7 to the set ˜ D and the measure ˜ µ , there exists a measure ˜ ν whose support is a ﬁnite subset of ˜ D , { z 1 u T 1 , . .
Revised: Therefore, by applying Theorem 7 to the set ˜ D and the measure ˜ µ , there exists a measure ˜ ν whose support is a ﬁnite subset of ˜ D , { z 1 u T 1 , . .

Original: z k u Tk } , with k ≤ KN + 1 such that 
Revised: z k u Tk } , with k ≤ KN + 1 such that (cid:90)

Original: [Equation]
Revised: [Equation]

Original:  and (cid:107) ˜ µ (cid:107) TV = (cid:107) ˜
Revised: , (108) and (cid:107) ˜ µ (cid:107) TV = (cid:107) ˜

Original: ν (cid:107) TV .
Revised: ν (cid:107) TV .

Original: We can deﬁne ν as the signed vector measure whose support is a ﬁnite subset { z 1 , . . .
Revised: We can deﬁne ν as the signed vector measure whose support is a ﬁnite subset { z 1 , . . .

Original: , z k } and dν ( z i )
Revised: , z k } and dν ( z i )

Original: =
Revised: =

Original: u i d ˜
Revised: u i d ˜

Original: ( z i u i ) .
Revised: ( z i u i ) .

Original: Then, (cid:107) ν (cid:107) TV = (cid:107) ˜
Revised: Then, (cid:107) ν (cid:107) TV = (cid:107) ˜

Original: ν (cid:107) TV = (cid:107) ˜ µ (cid:107) TV
Revised: ν (cid:107) TV = (cid:107) ˜ µ (cid:107) TV

Original: = (cid:107) µ (cid:107) TV .
Revised: = (cid:107) µ (cid:107) TV .

Original: This completes the proof.
Revised: This completes the proof.

Original: Now we are ready to present the theorem about the ﬁnite representation of a signed-vector measure.
Revised: Now we are ready to present the theorem about the ﬁnite representation of a signed-vector measure.

Original: Theorem 9 Suppose that θ is the parameter with a bounded domain Θ ⊆ R p and φ ( X , θ ) : R N × d × Θ
Revised: Theorem 9 Suppose that θ is the parameter with a bounded domain Θ ⊆ R p and φ ( X , θ ) : R N × d × Θ

Original: →
Revised: →

Original: R
Revised: R

Original: N is an embedding of the parameter into the feature space.
Revised: N is an embedding of the parameter into the feature space.

Original: Consider the following optimization problem
Revised: Consider the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: Assume that an optimal solution to (105) exists.
Revised: Assume that an optimal solution to (109) exists.

Original: Then, there exists an optimal solution ˆ µ supported on at most KN + 1 features in Θ .
Revised: Then, there exists an optimal solution ˆ µ supported on at most KN + 1 features in Θ .

Original: P ROOF Let ˆ µ be an optimal solution to (105).
Revised: P ROOF Let ˆ µ be an optimal solution to (109).

Original: We can deﬁne a measure ˆ P on R N as the pushforward of ˆ µ by ˆ P ( B ) =
Revised: We can deﬁne a measure ˆ P on R N as the pushforward of ˆ µ by ˆ P ( B ) =

Original: ˆ µ ( { θ | φ ( X, θ ) ∈ B } ) .
Revised: ˆ µ ( { θ | φ ( X, θ ) ∈ B } ) .

Original: Denote D = { φ ( X, θ ) | θ ∈ Θ } .
Revised: Denote D = { φ ( X, θ ) | θ ∈ Θ } .

Original: We note that ˆ P is supported on D and D is bounded.
Revised: We note that ˆ P is supported on D and D is bounded.

Original: By applying Theorem (8) to the set D and the measure ˆ P , we can ﬁnd a measure Q whose support is a ﬁnite subset of D , { z 1 , . . .
Revised: By applying Theorem (8) to the set D and the measure ˆ P , we can ﬁnd a measure Q whose support is a ﬁnite subset of D , { z 1 , . . .

Original: , z k } with k ≤ 2 K ( n + 1) .
Revised: , z k } with k ≤ 2 K ( n + 1) .

Original: For each z i ∈ D , we can ﬁnd
Revised: For each z i ∈ D , we can ﬁnd

Original: θ i such that φ ( X, θ i ) =
Revised: θ i such that φ ( X, θ i ) =

Original: z i .
Revised: z i .

Original: Then, ˜ µ = (cid:80) ki
Revised: Then, ˜ µ = (cid:80) ki

Original: =1 δ ( θ − θ i ) dQ
Revised: =1 δ ( θ − θ i ) dQ

Original: ( z i ) is an optimal solution 
Revised: ( z i ) is an optimal solution to (109) with at most 2 C ( N + 1) features and (cid:107) ˜ µ (cid:107) TV = (cid:107) µ (cid:107) TV . Here δ ( · ) is the Dirac delta measure.