Original: D EEP L AYERS AS S TOCHASTIC S OLVERS
Revised: D EEP L AYERS AS S TOCHASTIC S OLVERS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We provide a novel perspective on the forward pass through a block of layers in a deep network.
Revised: We provide a novel perspective on the forward pass through a block of layers in a deep network.

Original: In particular, we show that a forward pass through a standard dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex optimization objective with a single iteration of a τ -nice
Revised: In particular, we show that a forward pass through a standard dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex optimization objective with a single iteration of a τ -nice

Original: Proximal Stochastic Gradient method.
Revised: Proximal Stochastic Gradient method.

Original: We further show that replacing standard
Revised: We further show that replacing standard

Original: Bernoulli dropout with additive dropout is equivalent to optimizing the same convex objective with a variance-reduced proximal method.
Revised: Bernoulli dropout with additive dropout is equivalent to optimizing the same convex objective with a variance-reduced proximal method.

Original: By expressing both fullyconnected and convolutional layers as special cases of a high-order tensor product, we unify the underlying convex optimization problem in the tensor setting and derive a formula for the Lipschitz constant L used to determine the optimal step size of the above proximal methods.
Revised: By expressing both fullyconnected and convolutional layers as special cases of a high-order tensor product, we unify the underlying convex optimization problem in the tensor setting and derive a formula for the Lipschitz constant L used to determine the optimal step size of the above proximal methods.

Original: We conduct experiments with standard convolutional networks applied to the CIFAR-10 and CIFAR-100 datasets and show that replacing a block of layers with multiple iterations of the corresponding solver, with step size set via L , consistently improves classiﬁcation accuracy.
Revised: We conduct experiments with standard convolutional networks applied to the CIFAR-10 and CIFAR-100 datasets and show that replacing a block of layers with multiple iterations of the corresponding solver, with step size set via L , consistently improves classiﬁcation accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has revolutionized computer vision and natural language processing and is increasingly applied throughout science and engineering (LeCun et al., 2015).
Revised: Deep learning has revolutionized computer vision and natural language processing and is increasingly applied throughout science and engineering (LeCun et al., 2015).

Original: This has motivated  mathematical analysis of various aspects of deep networks, such as the capacity and uniqueness of their representations (Soatto & Chiuso, 2014; Papyan et al., 2018) and their global training convergence properties (Haeffele & Vidal, 2017).
Revised: This has motivated the mathematical analysis of various aspects of deep networks, such as the capacity and uniqueness of their representations (Soatto & Chiuso, 2014; Papyan et al., 2018) and their global training convergence properties (Haeffele & Vidal, 2017).

Original: However, a complete characterization of deep networks remains elusive.
Revised: However, a complete characterization of deep networks remains elusive.

Original: For example, Bernoulli dropout layers are widely celebrated in improving generalization (Srivastava et al., 2014); however, a complete theoretical understanding of their behavior remains an open problem.
Revised: For example, Bernoulli dropout layers are widely celebrated in improving generalization (Srivastava et al., 2014); however, a complete theoretical understanding of their behavior remains an open problem.

Original: While basic dropout layers have proven to be effective, there are many other types of dropout with various desirable properties that have been proposed (Molchanov et al., 2017).
Revised: While basic dropout layers have proven to be effective, there are many other types of dropout with various desirable properties that have been proposed (Molchanov et al., 2017).

Original: This raises many questions.
Revised: This raises many questions.

Original: Can the fundamental block of layers that consists of a dropout layer followed by a linear transformation and a non-linear activation be further improved for better generalization?
Revised: Can the fundamental block of layers that consists of a dropout layer followed by a linear transformation and a non-linear activation be further improved for better generalization?

Original: Can the choice of dropout layer be made independently from the linear transformation and non-linear activation?
Revised: Can the choice of dropout layer be made independently from the linear transformation and non-linear activation?

Original: Are there systematic ways to propose new types of dropout?
Revised: Are there systematic ways to propose new types of dropout?

Original: We attempt to address some of these questions by establishing a strong connection between the forward pass through a block of layers in a deep network and the solution of convex optimization problems of the following form:
Revised: We attempt to address some of these questions by establishing a strong connection between the forward pass through a block of layers in a deep network and the solution of convex optimization problems of the following form:

Original: [Equation]
Revised: [Equation]

Original: Note that when f i ( a (cid:62) i x ) = 12 ( a (cid:62) i x − y i ) 2 and g ( x ) = (cid:107) x (cid:107) 22 , Eq.
Revised: Note that when f i ( a (cid:62) i x ) = 12 ( a (cid:62) i x − y i ) 2 and g ( x ) = (cid:107) x (cid:107) 22 , Eq.

Original: (1) is standard ridge regression.
Revised: (1) is standard ridge regression.

Original: When g ( x ) = (cid:107) x (cid:107) 1 , Eq.
Revised: When g ( x ) = (cid:107) x (cid:107) 1 , Eq.

Original: (1) has the form of Lasso regression.
Revised: (1) has the form of LASSO regression.

Original: We show that a block of layers that consists of dropout followed by a linear transformation (fullyconnected or convolutional) and a non-linear activation has close connections to applying stochastic solvers to (1).
Revised: We show that a block of layers that consists of dropout followed by a linear transformation (fullyconnected or convolutional) and a non-linear activation has close connections to applying stochastic solvers to (1).

Original: Interestingly, the choice of the stochastic optimization algorithm gives rise to commonly used dropout layers, such as Bernoulli and additive dropout, and to a family of other types of dropout layers that have not been explored before.
Revised: Interestingly, the choice of the stochastic optimization algorithm gives rise to commonly used dropout layers, such as Bernoulli and additive dropout, and to a family of other types of dropout layers that have not been explored before.

Original: As a special case, the stochastic algorithm reduces to a deterministic one, when the block in question does not include dropout.
Revised: As a special case, the stochastic algorithm reduces to a deterministic one, when the block in question does not include dropout.

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: ( i )
Revised: ( i )

Original: We show that a forward pass through a block that consists of Bernoulli dropout followed by a linear transformation and a non-linear activation is equivalent to a single iteration of τ -nice Proximal Stochastic Gradient, Prox-SG (Xiao & Zhang, 2014) when it is applied to an instance of (1).
Revised: We show that a forward pass through a block that consists of Bernoulli dropout followed by a linear transformation and a non-linear activation is equivalent to a single iteration of τ -nice Proximal Stochastic Gradient, Prox-SG (Xiao & Zhang, 2014) when it is applied to an instance of (1).

Original: We provide various conditions on g that recover (either exactly or approximately) common non-linearities used in practice.
Revised: We provide various conditions on g that recover (either exactly or approximately) common non-linearities used in practice.

Original: ( ii ) We show that the same block with an additive dropout instead of Bernoulli dropout is equivalent to a single iteration of mS2GD (Koneˇcn`y et al., 2016) – a mini-batching form of variance-reduced SGD (Johnson & Zhang, 2013) – applied to an instance of (1).
Revised: ( ii ) We show that the same block with an additive dropout instead of Bernoulli dropout is equivalent to a single iteration of mS2GD (Koneˇcn`y et al., 2016) – a mini-batching form of variance-reduced SGD (Johnson & Zhang, 2013) – applied to an instance of (1).

Original: ( iii )
Revised: ( iii )

Original: By expressing both fully-connected and convolutional layers (referred to as linear throughout) as special cases of a high-order tensor product (Bibi & Ghanem, 2017), we derive a formula for the Lipschitz constant L of ∇ F ( x ) .
Revised: By expressing both fully-connected and convolutional layers (referred to as linear throughout) as special cases of a high-order tensor product (Bibi & Ghanem, 2017), we derive a formula for the Lipschitz constant L of ∇ F ( x ) .

Original: As a consequence, we can compute the optimal step size for the stochastic solvers that correspond to blocks of layers.
Revised: As a consequence, we can compute the optimal step size for the stochastic solvers that correspond to blocks of layers.

Revised: We note that concurrent work (Sedghi et al., 2019) used a different analysis strategy to derive an equivalent result for computing the singular values of convolutional layers.

Original: ( iv ) We validate our theoretical analysis experimentally by replacing blocks of layers in standard image classiﬁcation networks with corresponding solvers, with principled setting of the step size as implied by our derivation of L , and showing that this improves the accuracy of the models.
Revised: ( iv ) We validate our theoretical analysis experimentally by replacing blocks of layers in standard image classiﬁcation networks with corresponding solvers and show that this improves the accuracy of the models.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Optimization algorithms can provide insight and guidance in the design of deep network architectures (Vogel & Pock, 2017; Kobler et al., 2017; Yang et al., 2016; Zhang & Ghanem, 2018).
Revised: Optimization algorithms can provide insight and guidance in the design of deep network architectures (Vogel & Pock, 2017; Kobler et al., 2017; Yang et al., 2016; Zhang & Ghanem, 2018).

Original: For example, Yang et al.
Revised: For example, Yang et al.

Original: (2016) have proposed a deep network architecture for compressed sensing.
Revised: (2016) have proposed a deep network architecture for compressed sensing.

Original: Their network, dubbed ADMM-Net, is inspired by ADMM updates (Boyd et al., 2011) on the compressed sensing objective.
Revised: Their network, dubbed ADMM-Net, is inspired by ADMM updates (Boyd et al., 2011) on the compressed sensing objective.

Original: Similarly, Zhang & Ghanem (2018) demonstrated that unrolling a proximal gradient descent solver (Beck & Teboulle, 2009) on the same problem can further improve performance.
Revised: Similarly, Zhang & Ghanem (2018) demonstrated that unrolling a proximal gradient descent solver (Beck & Teboulle, 2009) on the same problem can further improve performance.

Original: Moreover, the work of Kobler et al.
Revised: Moreover, the work of Kobler et al.

Original: (2017) demonstrated a relation between incremental proximal methods and ResNet blocks.
Revised: (2017) demonstrated a relation between incremental proximal methods and ResNet blocks.

Original: Based on this observation, theyhave proposed a new architecture (variational networks) for the task of image reconstruction with an exciting performance.
Revised: Based on this observation, they proposed a new architecture (variational networks) for the task of image reconstruction.

Original: Amos & Kolter (2017) proposed to embed optimization problems, in particular linearly-constrained quadratic programs, as structured layers in deep networks.
Revised: Amos & Kolter (2017) proposed to embed optimization problems, in particular linearly-constrained quadratic programs, as structured layers in deep networks.

Original: Meinhardt et al.
Revised: Meinhardt et al.

Original: (2017) replaced proximal operators in optimization algorithms by neural networks.
Revised: (2017) replaced proximal operators in optimization algorithms by neural networks.

Original: Huang & Van Gool (2017) proposed a new matrix layer, dubbed ReEig, that applies a thresholding operation to the eigenvalues of intermediate feature representations that are stacked in matrix form.
Revised: Huang & Van Gool (2017) proposed a new matrix layer, dubbed ReEig, that applies a thresholding operation to the eigenvalues of intermediate feature representations that are stacked in matrix form.

Original: ReEig can be tightly connected to a proximal operator of the set of positive semi-deﬁnite matrices.
Revised: ReEig can be tightly connected to a proximal operator of the set of positive semi-deﬁnite matrices.

Original: Sulam et al.
Revised: Sulam et al.

Original: (2018) proposed a new architecture based on a sparse representation construct,
Revised: (2018) proposed a new architecture based on a sparse representation construct,

Original: Multi-Layer Convolutional Sparse Coding (ML-CSC), initially introduced by Papyan et al.(2016).
Revised: Multi-Layer Convolutional Sparse Coding (ML-CSC), initially introduced by Papyan et al.

Original: Sparsity on the intermediate representations was enforced by a multi-layer form of basis pursuit.
Revised: Sparsity on the intermediate representations was enforced by a multi-layer form of basis pursuit.

Original: This body of work has demonstrated the merits of connecting the design of deep networks with optimization algorithms in the form of structured layers.
Revised: This body of work has demonstrated the merits of connecting the design of deep networks with optimization algorithms in the form of structured layers.

Original: Still, with few exceptions (Amos & Kolter, 2017; Sulam et al., 2018), previous works propose specialized architectures for speciﬁc tasks.
Revised: Still, with few exceptions (Amos & Kolter, 2017; Sulam et al., 2018), previous works propose specialized architectures for speciﬁc tasks.

Original: Our work aims to contribute to a uniﬁed framework that connects optimization algorithms to deep layers.
Revised: Our work aims to contribute to a uniﬁed framework that connects optimization algorithms to deep layers.

Original: A line of work aims to provide rigorous interpretation for dropout layers.
Revised: A line of work aims to provide rigorous interpretation for dropout layers.

Original: For example, Wager et al.
Revised: For example, Wager et al.

Original: (2013) showed that dropout is linked to an adaptively balanced (cid:96) 2 -regularized loss.
Revised: (2013) showed that dropout is linked to an adaptively balanced (cid:96) 2 -regularized loss.

Original: Wang & Manning (2013) showed that approximating the loss with a normal distribution leads to a faster form of dropout.
Revised: Wang & Manning (2013) showed that approximating the loss with a normal distribution leads to a faster form of dropout.

Original: Gal & Ghahramani (2016a;b) developed an astute framework that connects dropout with approximate variational inference in Bayesian models.
Revised: Gal & Ghahramani (2016a;b) developed a framework that connects dropout with approximate variational inference in Bayesian models.

Original: We provide a complementary perspective, in which dropout layers arise naturally in an optimization-driven framework for network design.
Revised: We provide a complementary perspective, in which dropout layers arise naturally in an optimization-driven framework for network design.

Original:  U NIFIED F RAMEWORK
Revised:  U NIFIED F RAMEWORK

Original: This section is organized as follows.
Revised: This section is organized as follows.

Original: We introduce our notation and preliminaries in Section 3.1.
Revised: We introduce our notation and preliminaries in Section 3.1.

Original: In Section 3.2, we present a motivational example relating a single iteration of proximal gradient descent (Prox-GD) on (1) to the forward pass through a fully-connected layer followed by a nonlinear activation.
Revised: In Section 3.2, we present a motivational example relating a single iteration of proximal gradient descent (Prox-GD) on (1) to the forward pass through a fully-connected layer followed by a nonlinear activation.

Original: We will show that several commonly used non-linear activations can be exactly or approximately represented as proximal operators of g ( x ) .
Revised: We will show that several commonly used non-linear activations can be exactly or approximately represented as proximal operators of g ( x ) .

Original: In Section 3.3, we unify fully-connected and convolutional layers as special cases of a high-order tensor product.
Revised: In Section 3.3, we unify fully-connected and convolutional layers as special cases of a high-order tensor product.

Original: We propose a generic instance of (1) in a tensor setting, where we provide a formula for the Lipschitz constant L of the ﬁnite sum structure of (1).
Revised: We propose a generic instance of (1) in a tensor setting, where we provide a formula for the Lipschitz constant L of the ﬁnite sum structure of (1).

Original: In Section 3.4, we derive an intimate relation between stochastic solvers, namely τ -nice Prox-SG and mS2GD, and two types of dropout layers.
Revised: In Section 3.4, we derive an intimate relation between stochastic solvers, namely τ -nice Prox-SG and mS2GD, and two types of dropout layers.

Original: Figure 1 shows an overview of the connections that will be developed.
Revised: Figure 1 shows an overview of the connections that will be developed.

Original: 3. N OTATION AND P RELIMINARIES
Revised: 3. N OTATION AND P RELIMINARIES

Original: As we will be working with tensors, we will follow the tensor notation of Kolda & Bader (2009).
Revised: As we will be working with tensors, we will follow the tensor notation of Kolda & Bader (2009).

Original: The order of a tensor is the number of its dimensions.
Revised: The order of a tensor is the number of its dimensions.

Original: In particular, scalars are tensors of order zero, vectors are tensors of order one, and matrices are tensors of order two.
Revised: In particular, scalars are tensors of order zero, vectors are tensors of order one, and matrices are tensors of order two.

Original: We denote scalars by lowercase letters a , vectors by bold lowercase letters a , and matrices by bold capital letters A .
Revised: We denote scalars by lowercase letters a , vectors by bold lowercase letters a , and matrices by bold capital letters A .

Original: We use subscripts a i to refer to individual elements in a vector.
Revised: We use subscripts a i to refer to individual elements in a vector.

Original: Tensors of order three or more will be denoted by cursive capital letters A ∈ R J 1 × J 2 ×···× J n .
Revised: Tensors of order three or more will be denoted by cursive capital letters A ∈ R J 1 × J 2 ×···× J n .

Original: Throughout the paper, we will handle tensors that are of at most order four.
Revised: Throughout the paper, we will handle tensors that are of at most order four.

Original: High-order tensors with a second dimension of size equal to one are traditionally called vector tensors and denoted as (cid:126)
Revised: High-order tensors with a second dimension of size equal to one are traditionally called vector tensors and denoted as (cid:126)

Original: A ∈ R J 1 × 1 × J 3 × J 4 .
Revised: A ∈ R J 1 × 1 × J 3 × J 4 .

Original: We use A ( i, j, k, z ) to refer to an element in a tensor and A
Revised: We use A ( i, j, k, z ) to refer to an element in a tensor and A

Original: ( i, j, k, :) to refer to a slice of a tensor.
Revised: ( i, j, k, :) to refer to a slice of a tensor.

Original: The inner product between tensors of the same size is denoted as (cid:104)A , B(cid:105) =
Revised: The inner product between tensors of the same size is denoted as (cid:104)A , B(cid:105) =

Original: (cid:80) i 1 ,...,
Revised: (cid:80) i 1 ,...,

Original: i N A ( i 1 , .
Revised: i N A ( i 1 , .

Original: , i N ) B ( i 1 , . . .
Revised: , i N ) B ( i 1 , . . .

Original: , i N ) .
Revised: , i N ) .

Original: The squared Frobenius norm of a tensor A is deﬁned as (cid:107)A(cid:107) 2 F = (cid:104)A , A(cid:105) .
Revised: The squared Frobenius norm of a tensor A is deﬁned as (cid:107)A(cid:107) 2 F = (cid:104)A , A(cid:105) .

Original: Lastly, the superscripts (cid:62) and H are used to denote the transpose and the Hermitian transpose, respectively.
Revised: Lastly, the superscripts (cid:62) and H are used to denote the transpose and the Hermitian transpose, respectively.

Original: 3. M OTIVATIONAL I NSIGHT : N ON -L INEAR A CTIVATIONS AS P ROXIMAL O PERATORS
Revised: 3. M OTIVATIONAL I NSIGHT : N ON -L INEAR A CTIVATIONS AS P ROXIMAL O PERATORS

Original: As a motivating example, we consider a simple linear layer in a deep network that is followed by a non-linear activation ρ , i.e. x l = ρ ( Ax l − 1 + b ) , where A ∈ R n 2 × n 1 and b ∈ R n 2 are the weights and biases of the layer and x l − 1 and x l are the input and output activations, respectively.
Revised: As a motivating example, we consider the l th linear layer in a deep network that is followed by a non-linear activation ρ , i.e. x l = ρ ( Ax l − 1 + b ) , where A ∈ R n 2 × n 1 and b ∈ R n 2 are the weights and biases of the layer and x l − 1 and x l are the input and output activations, respectively.

Original: Now consider an instance of (1) with a convex function g ( x ) and
Revised: Now consider an instance of (1) with a convex function g ( x ) and

Original: [Equation]
Revised: [Equation]

Original: where A (cid:62) ( i, :) is the i th row of A (cid:62) .
Revised: where A (cid:62) ( i, :) is the i th row of A (cid:62) .

Original: Such an objective can be optimized iteratively  using Prox-GD with the following update equation:
Revised: Such an objective can be optimized iteratively in x l using Prox-GD with the following update equation:

Original: [Equation]
Revised: [Equation]

Original: where the Lipschitz constant L = λ max (cid:0) AA (cid:62) (cid:1) and λ max ( . ) is the maximum eigenvalue function.
Revised: where the Lipschitz constant L = λ max (cid:0) AA (cid:62) (cid:1) and λ max ( . ) denotes the maximum eigenvalue.

Original: By initializing the iterative optimization at x l = 0 , it becomes clear that a single iteration of (3) is equivalent to a fully-connected layer followed by a non-linearity that is implemented by the proximal operator (Fawzi et al., 2015).
Revised: By initializing the iterative optimization at x l = 0 , it becomes clear that a single iteration of (3) is equivalent to a fully-connected layer followed by a non-linearity that is implemented by the proximal operator (Fawzi et al., 2015).

Original: The choice of g ( x ) determines the speciﬁc form of the non-linearity ρ .
Revised: The choice of g ( x ) determines the speciﬁc form of the non-linearity ρ .

Original: Several popular activation functions can be traced back to their corresponding g ( x ) .
Revised: Several popular activation functions can be traced back to their corresponding g ( x ) .

Original: The ReLU, which enforces non-negative output activations, corresponds to  g ( x ) = 1 x ≥ 0 ; the corresponding instance of problem (1) is a non-negative quadratic program.
Revised: The ReLU, which enforces non-negative output activations, corresponds to the indicator function g ( x ) = 1 x ≥ 0 ; the corresponding instance of problem (1) is a non-negative quadratic program.

Original: Similar observations for the ReLU have been made in other contexts (Amos & Kolter, 2017; Papyan et al., 2017). Weobserve that
Revised: Similar observations for the ReLU have been made in other contexts (Amos & Kolter, 2017; Papyan et al., 2017). We

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Root of cubic polynomial
Revised: Root of cubic polynomial

Original: [Table]
Revised: [Table]

Revised: Activation

Original: [Equation]
Revised: [Equation]

Original: can be found in many other activation functions ﬁt this framework.
Revised: observe that many other activation functions ﬁt this framework.

Original: For example, when g ( x ) is a squared hinge loss, i.e. γ 2 (cid:80) i max 2 ( − x
Revised: For example, when g ( x ) is a squared hinge loss, i.e. γ 2 (cid:80) i max 2 ( − x

Original: i − λ, 0) , a single update of (3) is equivalent to a linear layer followed by a Leaky ReLU.
Revised: i − λ, 0) , a single update of (3) is equivalent to a linear layer followed by a Leaky ReLU.

Original: Table 1 lists some other choices of g ( x ) and their induced activations.
Revised: Table 1 lists some other choices of g ( x ) and their induced activations.

Original: Note that g ( x ) is not required to exhibit a simple, coordinate-wise separable structure.
Revised: Note that g ( x ) is not required to exhibit a simple, coordinate-wise separable structure.

Original: More complex functions can be used, as long as the proximal operator is easy to evaluate.
Revised: More complex functions can be used, as long as the proximal operator is easy to evaluate.

Original: Interesting examples arise when the output activations have matrix structure.
Revised: Interesting examples arise when the output activations have matrix structure.

Original: For instance, one can impose nuclear norm regularization g ( X )
Revised: For instance, one can impose nuclear norm regularization g ( X )

Original: = (cid:107) X (cid:107) ∗ to encourage X to be low rank.
Revised: = (cid:107) X (cid:107) ∗ to encourage X to be low rank.

Original: Alternatively, one can enforce positive semi-deﬁnite structure on the matrix X by deﬁning g ( X ) = 1 X (cid:23) 0 .
Revised: Alternatively, one can enforce positive semi-deﬁnite structure on the matrix X by deﬁning g ( X ) = 1 X (cid:23) 0 .

Original: A similar activation has been used for higher-order pooling (Huang & Van Gool, 2017).
Revised: A similar activation has been used for higher-order pooling (Huang & Van Gool, 2017).

Original: In what follows, we will show that this connection can be further extended to explain dropout layers.
Revised: In what follows, we will show that this connection can be further extended to explain dropout layers.

Original: Interestingly, speciﬁc forms of dropout do not arise from particular forms of objective (1), but from different stochastic optimization algorithms that are applied to it.
Revised: Interestingly, speciﬁc forms of dropout do not arise from particular forms of objective (1), but from different stochastic optimization algorithms that are applied to it.

Original: 3. U NIFYING F ULLY -C ONNECTED AND C ONVOLUTIONAL L AYERS
Revised: 3. U NIFYING F ULLY -C ONNECTED AND C ONVOLUTIONAL L AYERS

Original: Before presenting our main results on the equivalence between a forward pass through a block of layers and solving (1) with stochastic algorithms, we provide some key lemmas.
Revised: Before presenting our main results on the equivalence between a forward pass through a block of layers and solving (1) with stochastic algorithms, we provide some key lemmas.

Original: These lemmas will be necessary for a uniﬁed treatment of fully-connected and convolutional layers as generic linear layers.
Revised: These lemmas will be necessary for a uniﬁed treatment of fully-connected and convolutional layers as generic linear layers.

Original: This generic treatment will enable efﬁcient computation of the Lipschitz constant for both fully-connected and convolutional layers.
Revised: This generic treatment will enable efﬁcient computation of the Lipschitz constant for both fully-connected and convolutional layers.

Original: Lemma 1.
Revised: Lemma 1.

Original: Consider the l th convolutional layer in a deep network with some non-linear activation, e.g. Prox g ( . ) , where the weights A ∈ R n 2 × n 1 × W × H , biases (cid:126) B ∈ R n 2 × 1 × W × H , and input activations (cid:126)
Revised: Consider the l th convolutional layer in a deep network with some non-linear activation, e.g. Prox g ( . ) , where the weights A ∈ R n 2 × n 1 × W × H , biases (cid:126) B ∈ R n 2 × 1 × W × H , and input activations (cid:126)

Original: X l − 1 ∈ R n 1 × 1 × W × H are stacked into 4 th -order tensors.
Revised: X l − 1 ∈ R n 1 × 1 × W × H are stacked into 4 th -order tensors.

Original: We can describe the layer as
Revised: We can describe the layer as

Original: [Equation][Equation]
Revised: [Equation]

Original: where (cid:126) HO is the high-order tensor product.
Revised: where (cid:126) HO is the high-order tensor product.

Original: Here n 1 is the number of input features, n 2 is the number of output features (number of ﬁlters), and W and H are the spatial dimensions of the features.
Revised: Here n 1 is the number of input features, n 2 is the number of output features (number of ﬁlters), and W and H are the spatial dimensions of the features.

Original: As a special case, a fully-connected layer follows naturally, since (cid:126) HO reduces to a matrix-vector multiplication when W = H = 1 .
Revised: As a special case, a fully-connected layer follows naturally, since (cid:126) HO reduces to a matrix-vector multiplication when W = H = 1 .

Original: The proof can be found in supplementary material.
Revised: The proof can be found in supplementary material.

Original: Note that the order of the dimensions is essential in this notation, as the ﬁrst dimension in A corresponds to the number of independent ﬁlters while the second corresponds to the input features that will be aggregated after the 2D convolutions.
Revised: Note that the order of the dimensions is essential in this notation, as the ﬁrst dimension in A corresponds to the number of independent ﬁlters while the second corresponds to the input features that will be aggregated after the 2D convolutions.

Original: Also note that according to the deﬁnition of (cid:126) HO in (Bibi & Ghanem, 2017), the spatial size of the ﬁlters in A , namely W and H , has to match the spatial dimensions of the input activations (cid:126)
Revised: Also note that according to the deﬁnition of (cid:126) HO in (Bibi & Ghanem, 2017), the spatial size of the ﬁlters in A , namely W and H , has to match the spatial dimensions of the input activations (cid:126)

Original: X l − 1 , since the operator (cid:126) HO performs 2D circular convolutions while convolutions in deep networks are 2D linear convolutions.
Revised: X l − 1 , since the operator (cid:126) HO performs 2D circular convolutions while convolutions in deep networks are 2D linear convolutions.

Original: This is not a restriction, since one can perform linear convolution through a zero-padded circular convolution.
Revised: This is not a restriction, since one can perform linear convolution through a zero-padded circular convolution.

Original: Lastly, we assume that the values in (cid:126) B are replicated along the spatial dimensions W and H in order to recover the behaviour of biases in deep networks.
Revised: Lastly, we assume that the values in (cid:126) B are replicated along the spatial dimensions W and H in order to recover the behaviour of biases in deep networks.

Original: Given this notation, we will refer to either a fully-connected or a convolutional layer as a linear layer throughout the rest of the paper.
Revised: Given this notation, we will refer to either a fully-connected or a convolutional layer as a linear layer throughout the rest of the paper.

Original: Since we are interested in a generic linear layer followed by a non-linearity, we will consider the tensor quadratic version of F ( x ) , denoted as F ( (cid:126) X ) :
Revised: Since we are interested in a generic linear layer followed by a non-linearity, we will consider the tensor quadratic version of F ( x ) , denoted as F ( (cid:126) X ) :

Original: [Equation]
Revised: [Equation]

Original: Note that if A ∈ R n 2 × n 1 × W × H , then A H ∈ R n 1 × n 2 × W × H , where each of the frontal slices of
Revised: Note that if A ∈ R n 2 × n 1 × W × H , then A H ∈ R n 1 × n 2 × W × H , where each of the frontal slices of

Original: A (: , : , i, j ) is transposed and each ﬁlter,
Revised: A (: , : , i, j ) is transposed and each ﬁlter,

Original: A ( i, j, : , :) , is rotated by 180 ◦ .
Revised: A ( i, j, : , :) , is rotated by 180 ◦ .

Original: This means that A H (cid:126) HO (cid:126)
Revised: This means that A H (cid:126) HO (cid:126)

Original: X aggregates the n 2 ﬁlters after performing 2D correlations.
Revised: X aggregates the n 2 ﬁlters after performing 2D correlations.

Original: This is performed n 1 times independently.
Revised: This is performed n 1 times independently.

Original: This operation is commonly referred to as a transposed convolution.
Revised: This operation is commonly referred to as a transposed convolution.

Original: Details can be found in supplementary material.
Revised: Details can be found in supplementary material.

Original: Next, the following lemma provides a practical formula for the computation of the Lipschitz constant
Revised: Next, the following lemma provides a practical formula for the computation of the Lipschitz constant

Original: L of the ﬁnite sum part of (5):
Revised: L of the ﬁnite sum part of (5):

Original: Lemma 2.
Revised: Lemma 2.

Original: The Lipschitz constant L of ∇ F ( (cid:126) X ) as deﬁned in (5) is given by
Revised: The Lipschitz constant L of ∇ F ( (cid:126) X ) as deﬁned in (5) is given by

Original: [Equation]
Revised: [Equation]

Original: where ˆ A is the 2 D discrete Fourier transform along the spatial dimensions W and H .
Revised: where ˆ A is the 2 D discrete Fourier transform along the spatial dimensions W and H .

Original: The proof can be found in supplementary material.
Revised: The proof can be found in supplementary material.

Original: Lemma 2 states that the Lipschitz constant L is the maximum among the set of maximum eigenvalues of all the possible W ×
Revised: Lemma 2 states that the Lipschitz constant L is the maximum among the set of maximum eigenvalues of all the possible W ×

Original: H combinations of the outer product of frontal slices ˆ A (: , : , i, j ) ˆ
Revised: H combinations of the outer product of frontal slices ˆ A (: , : , i, j ) ˆ

Original: A H (: , : , i, j ) .
Revised: A H (: , : , i, j ) .

Original: Note that if
Revised: Note that if

Original: W = H
Revised: W = H

Original: = 1 , then ˆ A =
Revised: = 1 , then ˆ A =

Original: A ∈ R n 2 × n 1 since the 2D discrete Fourier transform of scalars (i.e. matrices of size 1 × 1 ) is an identity mapping.
Revised: A ∈ R n 2 × n 1 since the 2D discrete Fourier transform of scalars (i.e. matrices of size 1 × 1 ) is an identity mapping.

Original: As a consequence, we can simplify (6) to
Revised: As a consequence, we can simplify (6) to

Original: L = max i =
Revised: L = max i =

Original: j =1 { λ max (cid:0)
Revised: j =1 { λ max (cid:0)

Original: A (: , : , i, j )
Revised: A (: , : , i, j )

Original: A H (: , : , i, j ) (cid:1) }
Revised: A H (: , : , i, j ) (cid:1) }

Original: = λ max (cid:0) AA (cid:62) (cid:1) , which recovers the Lipschitz constant for fully-connected layers.
Revised: = λ max (cid:0) AA (cid:62) (cid:1) , which recovers the Lipschitz constant for fully-connected layers.

Original: 3. D ROPOUT L AYERS AS V ARIANTS OF S TOCHASTIC S OLVERS
Revised: 3. D ROPOUT L AYERS AS V ARIANTS OF S TOCHASTIC S OLVERS

Original: In this subsection, we present two propositions.
Revised: In this subsection, we present two propositions.

Original: The ﬁrst shows the relation between standard
Revised: The ﬁrst shows the relation between standard

Original: Bernoulli dropout, BerDropout p (Srivastava et al., 2014), and τ -nice Prox-SG.
Revised: Bernoulli dropout ( p is the dropout rate), BerDropout p (Srivastava et al., 2014), and τ -nice Prox-SG.

Original: The second proposition relates additive dropout, AddDropout, to mS2GD (Kone ˇ cn ` y et al., 2016).
Revised: The second proposition relates additive dropout, AddDropout, to mS2GD (Kone ˇ cn ` y et al., 2016).

Original: We will ﬁrst introduce a generic notion of sampling from a set.
Revised: We will ﬁrst introduce a generic notion of sampling from a set.

Original: This is essential as the stochastic algorithms sample unbiased function estimates from the set of n 1 functions in (5).
Revised: This is essential as the stochastic algorithms sample unbiased function estimates from the set of n 1 functions in (5).

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: (Gower et al., 2018).
Revised: (Gower et al., 2018).

Original: A sampling is a random set-valued mapping with values being the subsets of [ n 1 ] = { 1 , . . . , n 1 } .
Revised: A sampling is a random set-valued mapping with values being the subsets of [ n 1 ] = { 1 , . . . , n 1 } .

Original: A sampling S is τ -nice if it is uniform, i.e.
Revised: A sampling S is τ -nice if it is uniform, i.e.

Original: Prob ( i ∈ S )
Revised: Prob ( i ∈ S )

Original: =
Revised: =

Original: Prob ( j ∈ S ) ∀ i, j , and assigns equal probabilities to all subsets of [ n 1 ] of cardinality τ and zero probability to all others.
Revised: Prob ( j ∈ S ) ∀ i, j , and assigns equal probabilities to all subsets of [ n 1 ] of cardinality τ and zero probability to all others.

Original: Various other types of sampling can be found in (Gower et al., 2018).
Revised: Various other types of sampling can be found in (Gower et al., 2018).

Original: We are now ready to present our ﬁrst proposition.
Revised: We are now ready to present our ﬁrst proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: A single iteration of Prox-SG with τ -nice sampling S on (5) with τ = (1 − p ) n 1 , zero initialization, and unit step size can be shown to exhibit the update
Revised: A single iteration of Prox-SG with τ -nice sampling S on (5) with τ = (1 − p ) n 1 , zero initialization, and unit step size can be shown to exhibit the update

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to a forward pass through a BerDropout p( p is the dropout rate) layer that drops exactly n 1 p input activations followed by a linear layer and a non-linear activation.
Revised: which is equivalent to a forward pass through a BerDropout p layer that drops exactly n 1 p input activations followed by a linear layer and a non-linear activation.

Original: We provide a simpliﬁed sketch for fully-connected layers here.
Revised: We provide a simpliﬁed sketch for fully-connected layers here.

Original: The detailed proof is in the supplement.
Revised: The detailed proof is in the supplement.

Original: To see how (7) reduces to the functional form of BerDropout p followed by a fully-connected layer and a non-linear activation, consider W = H = 1 .
Revised: To see how (7) reduces to the functional form of BerDropout p followed by a fully-connected layer and a non-linear activation, consider W = H = 1 .

Original: The argument of Prox 1 L g in (7) (without the bias term) reduces ton
Revised: The argument of Prox 1 L g in (7) (without the bias term) reduces to

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst equality follows from the deﬁnition of (cid:126) HO , while the second equality follows from trivially reparameterizing the sum, with BerDropout p ( . ) being equivalent to a mask that zeroes out exactly pn 1 input activations.
Revised: The ﬁrst equality follows from the deﬁnition of (cid:126) HO , while the second equality follows from trivially reparameterizing the sum, with BerDropout p ( . ) being equivalent to a mask that zeroes out exactly pn 1 input activations.

Original: Note that if τ -nice Prox-SG was replaced with Prox-GD, i.e. τ = n 1 , then this corresponds to having a BerDropout p layer with dropout rate p = 0 ; thus, (8) reduces to
Revised: Note that if τ -nice Prox-SG was replaced with Prox-GD, i.e. τ = n 1 , then this corresponds to having a BerDropout p layer with dropout rate p = 0 ; thus, (8) reduces to

Original: A BerDropout p ( (cid:126) X l − 1 ) =
Revised: A BerDropout p ( (cid:126) X l − 1 ) =

Original: A (cid:126)
Revised: A (cid:126)

Original: X l − 1 , which recovers our motivating example (3) that relates ProxGD with the forward pass through a fully-connected layer followed by a non-linearity.
Revised: X l − 1 , which recovers our motivating example (3) that relates ProxGD with the forward pass through a fully-connected layer followed by a non-linearity.

Original: Note that
Revised: Note that

Original: Proposition 1 directly suggests how to apply dropout to convolutional layers.
Revised: Proposition 1 directly suggests how to apply dropout to convolutional layers.

Original: Speciﬁcally, complete input features from n 1 should be dropped and the 2D convolutions should be performed only on the τ -sampled subset, where τ = (1 − p ) n 1 .
Revised: Speciﬁcally, complete input features from n 1 should be dropped and the 2D convolutions should be performed only on the τ -sampled subset, where τ = (1 − p ) n 1 .

Original: Similarly, the following proposition shows that a form of additive dropout, AddDropout, can be recovered from a different choice of stochastic solver.
Revised: Similarly, the following proposition shows that a form of additive dropout, AddDropout, can be recovered from a different choice of stochastic solver.

Original: Proposition 2.
Revised: Proposition 2.

Original: A single outer-loop iteration of mS2GD (Koneˇcn`y et al., 2016) with unit step size and zero initialization is equivalent to a forward pass through an AddDropout layer followed by a linear layer and a non-linear activation.
Revised: A single outer-loop iteration of mS2GD (Koneˇcn`y et al., 2016) with unit step size and zero initialization is equivalent to a forward pass through an AddDropout layer followed by a linear layer and a non-linear activation.

Original: The proof is given in the supplement.
Revised: The proof is given in the supplement.

Original: It is similar to Proposition 1, with mS2GD replacing τ -nice
Revised: It is similar to Proposition 1, with mS2GD replacing τ -nice

Original: Prox-SG.
Revised: Prox-SG.

Original: Note that any variance-reduced algorithm where one full gradient is computed at least once can be used here as a replacement for mS2GD.
Revised: Note that any variance-reduced algorithm where one full gradient is computed at least once can be used here as a replacement for mS2GD.

Original: For instance, one can show that the serial sampling version of mS2GD, S2GD (Kone ˇ cn ` y et al., 2016), and SVRG (Johnson & Zhang, 2013) can also be used.
Revised: For instance, one can show that the serial sampling version of mS2GD, S2GD (Kone ˇ cn ` y et al., 2016), and SVRG (Johnson & Zhang, 2013) can also be used.

Original: Other algorithms such as Stochastic Coordinate Descent (Richt ´ arik & Tak ´ a ˇ c, 2016) with arbitrary sampling are discussed in the supplement.
Revised: Other algorithms such as Stochastic Coordinate Descent (Richt´arik & Tak´aˇc, 2016) with arbitrary sampling are discussed in the supplement.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: A natural question arises as a consequence of our framework: If common layers in deep networks can be understood as a single iteration of an optimization algorithm, what happens if the algorithm is applied for multiple iterations?
Revised: A natural question arises as a consequence of our framework: If common layers in deep networks can be understood as a single iteration of an optimization algorithm, what happens if the algorithm is applied for multiple iterations?

Original: We empirically answer this question in our experiments.
Revised: We empirically answer this question in our experiments.

Original: In particular, we embed solvers as a replacement to their corresponding blocks of layers and show that this improves the accuracy of the models without an increase in the number of network parameters.
Revised: In particular, we embed solvers as a replacement to their corresponding blocks of layers and show that this improves the accuracy of the models without an increase in the number of network parameters.

Original: Experimental setup.
Revised: Experimental setup.

Original: We perform experiments on CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009).
Revised: We perform experiments on CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009).

Original: In all experiments, training was conducted on 90% of the training set while 10% was left for validation.
Revised: In all experiments, training was conducted on 90% of the training set while 10% was left for validation.

Original: The networks used in the experiments are variants of LeNet (LeCun et al., 1999),
Revised: The networks used in the experiments are variants of LeNet (LeCun et al., 1999),

Original: AlexNet (Krizhevsky et al., 2012), and VGG16 (Simonyan & Zisserman, 2014).
Revised: AlexNet (Krizhevsky et al., 2012), and VGG16 (Simonyan & Zisserman, 2014).

Original: We used stochastic gradient descent with a momentum of 0 .
Revised: We used stochastic gradient descent with a momentum of 0 .

Original: 9 and a weight decay of 5 ×
Revised: 9 and a weight decay of 5 ×

Original: 10 − 4 .
Revised: 10 − 4 .

Original: The learning rate was set to (10 − 2 , 10 − 3 , 10 − 4 ) for the ﬁrst, second, and third 100 epochs, respectively.
Revised: The learning rate was set to (10 − 2 , 10 − 3 , 10 − 4 ) for the ﬁrst, second, and third 100 epochs, respectively.

Original: For ﬁnetuning, the learning rate was initially set to 10 − 3 and reduced to 10 − 4 after 100 epochs.
Revised: For ﬁnetuning, the learning rate was initially set to 10 − 3 and reduced to 10 − 4 after 100 epochs.

Original: Moreover, when a block of layers is replaced with a deterministic solver, i.e. Prox-GD, the step size is set to the optimal constant 1 /L , where L is computed according to Lemma 2 and updated every epoch.
Revised: Moreover, when a block of layers is replaced with a deterministic solver, i.e. Prox-GD, the step size is set to the optimal constant 1 /L , where L is computed according to Lemma 2 and updated every epoch without any zero padding as a circular convolution operator approximates a linear convolution in large dimensions (Zhu & Wakin, 2017).

Original: In Prox-SG, a decaying step size is necessary for convergence; therefore, the step size is exponentially decayed as suggested by Bottou (2012), where the initial step size is again set according to Lemma 2.
Revised: In Prox-SG, a decaying step size is necessary for convergence; therefore, the step size is exponentially decayed as suggested by Bottou (2012), where the initial step size is again set according to Lemma 2.

Original: Finally, to guarantee convergence of the stochastic solvers, we add the strongly convex function λ 2 (cid:107) (cid:126) X(cid:107) 2 F to the ﬁnite sum in (5), where we set λ
Revised: Finally, to guarantee convergence of the stochastic solvers, we add the strongly convex function λ 2 (cid:107) (cid:126) X(cid:107) 2 F to the ﬁnite sum in (5), where we set λ

Original: = 10 − 3 in all experiments.
Revised: = 10 − 3 in all experiments.

Original: Note that for networks that include a stochastic solver, the network will be stochastic at test time.
Revised: Note that for networks that include a stochastic solver, the network will be stochastic at test time.

Original: We thus report the average accuracy and standard deviation over 20 trials.
Revised: We thus report the average accuracy and standard deviation over 20 trials.

Original: Replacing fully-connected layers with solvers.
Revised: Replacing fully-connected layers with solvers.

Original: In this experiment, we demonstrate that ( i ) training networks with solvers replacing one or more blocks of layers can improve accuracy when trained from scratch.
Revised: In this experiment, we demonstrate that ( i ) training networks with solvers replacing one or more blocks of layers can improve accuracy when trained from scratch.

Original: ( ii )
Revised: ( ii )

Original: The improvement is consistently present when one or more blocks are replaced with solvers at different layers in the network.
Revised: The improvement is consistently present when one or more blocks are replaced with solvers at different layers in the network.

Original: To do so, we train a variant of LeNet on the CIFAR10 dataset with two BerDropout p layers.
Revised: To do so, we train a variant of LeNet on the CIFARdataset with two BerDropout p layers.

Original: The last two layers are fully-connected layers with ReLU activation.
Revised: The last two layers are fully-connected layers with ReLU activation.

Original: We consider three variants of this network: Both fully-connected layers are augmented with BerDropout p (LeNet-D-D), only the last layer is augmented with BerDropout p (LeNet-ND-D), and ﬁnally only the penultimate layer is augmented with BerDropout p (LeNet-D-ND).
Revised: We consider three variants of this network: Both fully-connected layers are augmented with BerDropout p (LeNet-D-D), only the last layer is augmented with BerDropout p (LeNet-ND-D), and ﬁnally only the penultimate layer is augmented with BerDropout p (LeNet-D-ND).

Original: In all cases, we set the dropout rate to p = 0 .
Revised: In all cases, we set the dropout rate to p = 0 .

Original: We replace the BerDropout p layers with their corresponding stochastic solvers and run them for 10 iterations with τ = n 1 / 2 (the setting corresponding to a dropout rate of p = 0 . 5 ).
Revised: We replace the BerDropout p layers with their corresponding stochastic solvers and run them for 10 iterations with τ = n 1 / 2 (the setting corresponding to a dropout rate of p = 0 . 5 ).

Original: We train these networks from scratch using the same procedure as the baseline networks.
Revised: We train these networks from scratch using the same procedure as the baseline networks.

Original: The results are summarized in Table 2.
Revised: The results are summarized in Table 2.

Original: It can be seen that replacing BerDropout p with the corresponding stochastic solver ( τ -nice Prox-SG) improves performance signiﬁcantly, for any choice of layer.
Revised: It can be seen that replacing BerDropout p with the corresponding stochastic solver ( τ -nice Prox-SG) improves performance signiﬁcantly, for any choice of layer.

Original: The results indicate that networks that incorporate stochastic solvers can be trained stably and achieve desirable generalization performance.
Revised: The results indicate that networks that incorporate stochastic solvers can be trained stably and achieve desirable generalization performance.

Original: [Table]
Revised: [Table]

Original: Convolutional layers and larger networks.
Revised: Convolutional layers and larger networks.

Original: We now demonstrate that solvers can be used to improve larger networks.
Revised: We now demonstrate that solvers can be used to improve larger networks.

Original: We conduct experiments with variants of AlexNet 1 and VGG16 on both
Revised: We conduct experiments with variants of AlexNet 1 and VGG16 on both

Original: CIFAR-10 and CIFAR-100.
Revised: CIFAR-10 and CIFAR-100.

Original: We start by training strong baselines for both AlexNet and VGG16, achieving 77 .
Revised: We start by training strong baselines for both AlexNet and VGG16, achieving 77 .

Original: 3% and 92 .
Revised: 3% and 92 .

Original: 56% test accuracy on CIFAR-10, respectively.
Revised: 56% test accuracy on CIFAR-10, respectively.

Original: Note that performance on this dataset is nearly saturated.
Revised: Note that performance on this dataset is nearly saturated.

Original: We then replace the ﬁrst convolutional layer in AlexNet with the deterministic Prox-GD solver, since this layer is not preceded by a dropout layer.
Revised: We then replace the ﬁrst convolutional layer in AlexNet with the deterministic Prox-GD solver, since this layer is not preceded by a dropout layer.

Original: The results are summarized in Table 3.
Revised: The results are summarized in Table 3.

Original: We observe that ﬁnetuning the baseline network with the solver leads to an improvement of ≈ 1 . 2% , without any change in the network’s capacity.
Revised: We observe that ﬁnetuning the baseline network with the solver leads to an improvement of ≈ 1 . 2% , without any change in the network’s capacity.

Original: A similar improvement is observed on the harder CIFAR-100 dataset.
Revised: A similar improvement is observed on the harder CIFAR-100 dataset.

Original: [Table]
Revised: [Table]

Original: of AlexNet by the deterministic Prox-GD solver yields consistent improvement in test accuracy on CIFAR-10 and CIFAR-100.
Revised: AlexNet by the deterministic Prox-GD solver yields consistent improvement in test accuracy on CIFAR-10 and CIFAR-100.

Original: Results on VGG16 are summarized in Table 4.
Revised: Results on VGG16 are summarized in Table 4.

Original: Note that VGG16 has two fully-connected layers, which are preceded by a BerDropout  layer with dropout rate p = 0 .
Revised: Note that VGG16 has two fully-connected layers, which are preceded by a BerDropout p layer with dropout rate p = 0 .

Original: We start by replacing only the last layer with Prox-SG with 30 iterations and τ = n 1 / 2 (VGG16-Prox-SG-ND-D).
Revised: We start by replacing only the last layer with Prox-SG with 30 iterations and τ = n 1 / 2 (VGG16-Prox-SG-ND-D).

Original: We further replace both fully-connected layers that include BerDropout p with solvers (VGG16-Prox-SG-D-D).
Revised: We further replace both fully-connected layers that include BerDropout p with solvers (VGG16-Prox-SG-D-D).

Original: We observe comparable performance for both settings on CIFAR-10.
Revised: We observe comparable performance for both settings on CIFAR-10.

Original: We conjecture that this might be due to the dataset being close to saturation.
Revised: We conjecture that this might be due to the dataset being close to saturation.

Original: On CIFAR-100, a more pronounced increase in accuracy is observed, where VGG-16-Prox-SG-ND-D outperforms the baseline by about 0.7%.
Revised: On CIFAR-100, a more pronounced increase in accuracy is observed, where VGG-16-Prox-SG-ND-D outperforms the baseline by about 0.7%.

Original: We further replace the stochastic solver with a deterministic solver and leave the dropout layers unchanged.
Revised: We further replace the stochastic solver with a deterministic solver and leave the dropout layers unchanged.

Original: We denote this setting as VGG16-Prox-GD in Table 4.
Revised: We denote this setting as VGG16-Prox-GD in Table 4.

Original: Interestingly, this setting performs the best on CIFAR-10 and comparably to VGG16-Prox-SG-ND-D on CIFAR-100.
Revised: Interestingly, this setting performs the best on CIFAR-10 and comparably to VGG16-Prox-SG-ND-D on CIFAR-100.

Original: [Table]
Revised: [Table]

Original: Dropout rate vs. τ -nice sampling.
Revised: Dropout rate vs. τ -nice sampling.

Original: In this experiment, we demonstrate that the improvement in performance is still consistently present across varying dropout rates.
Revised: In this experiment, we demonstrate that the improvement in performance is still consistently present across varying dropout rates.

Original: Since Proposition 1 has established a tight connection between the dropout rate p and the sampling rate τ in (5), we observe that for different choices of dropout rate the baseline performance improves upon replacing a block of layers with a stochastic solver with the corresponding sampling rate τ .
Revised: Since Proposition 1 has established a tight connection between the dropout rate p and the sampling rate τ in (5), we observe that for different choices of dropout rate the baseline performance improves upon replacing a block of layers with a stochastic solver with the corresponding sampling rate τ .

Original: We conduct experiments with
Revised: We conduct experiments with

Original: VGG16 on CIFAR-100.
Revised: VGG16 on CIFAR-100.

Original: We train four different baseline models with varying choices of dropout rate p ∈ { 0 , 0 . 1 , 0 . 9 . 0 .
Revised: We train four different baseline models with varying choices of dropout rate p ∈ { 0 , 0 . 1 , 0 . 9 . 0 .

Original: 95 } for the last layer.
Revised: 95 } for the last layer.

Original: We then replace this block with a stochastic solver with a sampling rate τ and ﬁnetune the network.
Revised: We then replace this block with a stochastic solver with a sampling rate τ and ﬁnetune the network.

Original: Table 5 reports the accuracy of the baselines(BerDropout p ) for varying dropout rates  and compares to the accuracy of the stochastic solver with corresponding τ (Prox-SG).
Revised: Table 5 reports the accuracy of the baselines for varying dropout rates p and compares to the accuracy of the stochastic solver with corresponding τ (Prox-SG).

Original: Note that since the last layer has n 1 = 512 input activations, the corresponding sampling rates τ are { 512 , 461 , 256 , 51 , 26 } where with p = 0 , this is a deterministic solver Prox-GD.

Original: With a high dropout rate p , the performance of the baseline network drops drastically.
Revised: With a high dropout rate, the performance of the baseline network drops drastically.

Original: When using the stochastic solver, we observe a much more graceful drop in performance.
Revised: When using the stochastic solver, we observe a much more graceful drop in performance.

Original: For example, with a sampling rate τ that corresponds to an extreme dropout rate of p = 0 .
Revised: For example, with a sampling rate τ that corresponds to an extreme dropout rate of p = 0 .

Original: 95 (i.e. 95% of all input activations are masked out), the baseline network with BerDropout p suffers a 56% reduction in accuracy while the stochastic solver declines by only 5%.
Revised: 95 (i.e. 95% of all input activations are masked out), the baseline network with BerDropout p suffers a 56% reduction in accuracy while the stochastic solver declines by only 5%.

Original: [Table]
Revised: [Table]

Original: In summary, our experiments show that replacing common layers in deep networks with stochastic solvers can lead to better performance without increasing the number of parameters in the network.
Revised: In summary, our experiments show that replacing common layers in deep networks with stochastic solvers can lead to better performance without increasing the number of parameters in the network.

Original: The resulting networks are stable to train and exhibit high accuracy in cases where standard dropout has proven to be problematic, such as high dropout rates.
Revised: The resulting networks are stable to train and exhibit high accuracy in cases where standard dropout has proven to be problematic, such as high dropout rates.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have presented equivalences between layers in deep networks and stochastic solvers, and have shown that this can be leveraged to improve accuracy.
Revised: We have presented equivalences between layers in deep networks and stochastic solvers, and have shown that this can be leveraged to improve accuracy.

Original: The presented relationships open many doors for future work.
Revised: The presented relationships open many doors for future work.

Original: For instance, our framework shows an intimate relation between a dropout layer and the sampling S from the set [ n 1 ] in a stochastic algorithm.
Revised: For instance, our framework shows an intimate relation between a dropout layer and the sampling S from the set [ n 1 ] in a stochastic algorithm.

Original: As a consequence, one can borrow theory from the stochastic optimization literature to propose new types of dropout layers.
Revised: As a consequence, one can borrow theory from the stochastic optimization literature to propose new types of dropout layers.

Original: For example, consider a serial importance sampling strategy with Prox-SG to solve (5) (Zhao & Zhang, 2015; Xiao & Zhang, 2014), where serial sampling is the sampling that satisﬁes
Revised: For example, consider a serial importance sampling strategy with Prox-SG to solve (5) (Zhao & Zhang, 2015; Xiao & Zhang, 2014), where serial sampling is the sampling that satisﬁes

Original: Prob ( i ∈ S, j ∈ S ) = 0 .
Revised: Prob ( i ∈ S, j ∈ S ) = 0 .

Original: A serial importance sampling S from the set of functions
Revised: A serial importance sampling S from the set of functions

Original: f i ( (cid:126) X ) is the sampling such that Prob ( i ∈ S )
Revised: f i ( (cid:126) X ) is the sampling such that Prob ( i ∈ S )

Original: ∝ (cid:107)∇ f i ( (cid:126) X ) (cid:107) ∝ L
Revised: ∝ (cid:107)∇ f i ( (cid:126) X ) (cid:107) ∝ L

Original: i , where L i is the Lipschitz constant of ∇ f i ( (cid:126) X ) , i.e. each function from the set [ n 1 ] is sampled with a probability proportional to the norm of the gradient of the function.
Revised: i , where L i is the Lipschitz constant of ∇ f i ( (cid:126) X ) , i.e. each function from the set [ n 1 ] is sampled with a probability proportional to the norm of the gradient of the function.

Original: This sampling strategy is the optimal serial sampling S that maximizes the rate of convergence solving (5)
Revised: This sampling strategy is the optimal serial sampling S that maximizes the rate of convergence solving (5)

Original: (Zhao & Zhang, 2015).
Revised: (Zhao & Zhang, 2015).

Original: From a deep layer perspective, performing
Revised: From a deep layer perspective, performing

Original: Prox-SG with importance sampling for a single iteration is equivalent to a forward pass through the same block of layers with a new dropout layer.
Revised: Prox-SG with importance sampling for a single iteration is equivalent to a forward pass through the same block of layers with a new dropout layer.

Original: Such a dropout layer will keep each input activation with a non-uniform probability proportional to the norm of the gradient.
Revised: Such a dropout layer will keep each input activation with a non-uniform probability proportional to the norm of the gradient.

Original: This is in contrast to
Revised: This is in contrast to

Original: BerDropout p where all input activations are kept with an equal probability 1 − p .
Revised: BerDropout p where all input activations are kept with an equal probability 1 − p .

Original: Other types of dropout arise when considering non-serial importance sampling where | S | = τ > 1 .
Revised: Other types of dropout arise when considering non-serial importance sampling where | S | = τ > 1 .

Original: In summary, we have presented equivalences between stochastic solvers on a particular class of convex optimization problems and a forward pass through a dropout layer followed by a linear layer and a non-linear activation.
Revised: In summary, we have presented equivalences between stochastic solvers on a particular class of convex optimization problems and a forward pass through a dropout layer followed by a linear layer and a non-linear activation.

Original: Inspired by these equivalences, we have demonstrated empirically on multiple datasets and network architectures that replacing such network blocks with their corresponding stochastic solvers improves the model’s accuracy.
Revised: Inspired by these equivalences, we have demonstrated empirically on multiple datasets and network architectures that replacing such network blocks with their corresponding stochastic solvers improves the accuracy of the model.

Original: We hope that the presented framework will contribute to a principled understanding of the theory and practice of deep network architecture.
Revised: We hope that the presented framework will contribute to a principled understanding of the theory and practice of deep network architectures. Acknowledgments.

Revised: This work was partially supported by the King Abdullah University of Science and Technology (KAUST)

Revised: Ofﬁce of Sponsored Research.

Original: A L EAKY R E LU AS A P ROXIMAL O PERATOR
Revised: A L EAKY R E LU AS A P ROXIMAL O PERATOR

Original: Proposition 3.
Revised: Proposition 3.

Original: The Leaky ReLU is the solution of the proximal operator of the function g ( x ) = γ 2 (cid:80) i max 2 ( − x i − λ, 0) with slope 11+ γ and shift λ .
Revised: The Leaky ReLU is the solution of the proximal operator of the function g ( x ) = γ 2 (cid:80) i max 2 ( − x i − λ, 0) with slope 11+ γ and shift λ .

Original: Proof.
Revised: Proof.

Original: The proximal operator is deﬁned as
Revised: The proximal operator is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Note that the problem is both convex and smooth.
Revised: Note that the problem is both convex and smooth.

Original: The optimality conditions are given by:
Revised: The optimality conditions are given by:

Original: [Equation]
Revised: [Equation]

Original: Since the problem is separable in coordinates, we have:
Revised: Since the problem is separable in coordinates, we have:

Original: [Equation]
Revised: [Equation]

Original: The Leaky ReLU is deﬁned as
Revised: The Leaky ReLU is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: which shows that Prox g is a generalized form of the Leaky ReLU with a shift of λ and a slope α = 11+ γ .
Revised: which shows that Prox g is a generalized form of the Leaky ReLU with a shift of λ and a slope α = 11+ γ .

Original: With λ = 0 and γ = 1 − αα the standard form of the Leaky ReLU is recovered.
Revised: With λ = 0 and γ = 1 − αα the standard form of the Leaky ReLU is recovered.

Original: B A PPROXIMATION OF A S OFT P LUS AS A P ROXIMAL O PERATOR.
Revised: B A PPROXIMATION OF A S OFT P LUS AS A P ROXIMAL O PERATOR

Original: Proposition 4.
Revised: Proposition 4.

Original: The proximal operator to g ( x ) = − γ (cid:80) i log ( x i ) approximates the SoftPlus activation
Revised: The proximal operator to g ( x ) = − γ (cid:80) i log ( x i ) approximates the SoftPlus activation

Original: Proof.
Revised: Proof.

Original: The proximal operator is deﬁned as
Revised: The proximal operator is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Note that the function g ( x ) is elementwise separable, convex, and smooth.
Revised: Note that the function g ( x ) is elementwise separable, convex, and smooth.

Original: By equating the gradient to zero and taking the positive solution of the resulting quadratic polynomial, we arrive at the closedform solution:
Revised: By equating the gradient to zero and taking the positive solution of the resulting quadratic polynomial, we arrive at the closedform solution:

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes elementwise multiplication.
Revised: where (cid:12) denotes elementwise multiplication.

Original: It is easy to see that this operator is close to zero for x i << 0 , and close to x i for x i >> 0 , with a smooth transition for small | x i | .
Revised: It is easy to see that this operator is close to zero for x i << 0 , and close to x i for x i >> 0 , with a smooth transition for small | x i | .

Original: Note that the function Prox g ( a ) approximates the activation SoftPlus = log (1 + exp ( a )) very well.
Revised: Note that the function Prox g ( a ) approximates the activation SoftPlus = log (1 + exp ( a )) very well.

Original: An illustrative example is shown in Figure 2.
Revised: An illustrative example is shown in Figure 2.

Original: C A PPROXIMATION OF T AN H AND S IGMOID AS A P ROXIMAL O PERATOR
Revised: C A PPROXIMATION OF T AN H AND S IGMOID AS A P ROXIMAL O PERATOR

Original: Proposition 5.
Revised: Proposition 5.

Original: The proximal operator to g ( x ) = − γ (cid:80) i log (1 − x i (cid:12) x i ) approximates the Tanh non-linearity.
Revised: The proximal operator to g ( x ) = − γ (cid:80) i log (1 − x i (cid:12) x i ) approximates the Tanh non-linearity.

Original: [Figure]
Revised: [Figure]

Original: Proof.
Revised: Proof.

Original: To simplify the exposition, we derive the proximal operator for the case γ = 1 .
Revised: To simplify the exposition, we derive the proximal operator for the case γ = 1 .

Original: The general case for γ > 0 follows analogously.
Revised: The general case for γ > 0 follows analogously.

Original: The proximal operator is deﬁned as
Revised: The proximal operator is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Note that the logarithm is taken element wise, and the objective is convex and smooth.
Revised: Note that the logarithm is taken element wise, and the objective is convex and smooth.

Original: By equating the gradient to zero, it can be seen that the optimal solution is a root of a cubic equation:
Revised: By equating the gradient to zero, it can be seen that the optimal solution is a root of a cubic equation:

Original: [Equation]
Revised: [Equation]

Original: which is deﬁned in each coordinate i separately.
Revised: which is deﬁned in each coordinate i separately.

Original: Let
Revised: Let

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since q 2 − p 3 < 0 , ∀ a i ∈ R , it is guaranteed that all roots are real and distinct.
Revised: Since q 2 − p 3 < 0 , ∀ a i ∈ R , it is guaranteed that all roots are real and distinct.

Original: Consequently, the roots can be described as
Revised: Consequently, the roots can be described as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since g ( x ) is only deﬁned on x ∈ [ − 1 , 1] d , the root that minimizes (11) has to satisfy
Revised: Since g ( x ) is only deﬁned on x ∈ [ − 1 , 1] d , the root that minimizes (11) has to satisfy

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: − 1 (cid:18) a 3 i 27 √ p 3 (cid:19) implies 0 ≤ y ≤ π .
Revised: − 1 (cid:18) a 3 i 27 √ p 3 (cid:19) implies 0 ≤ y ≤ π .

Original: Let
Revised: Let

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: It is straightforward to check that
Revised: It is straightforward to check that

Original: [Equation]
Revised: [Equation]

Original: By substituting f k into (14) and checking inequality (15) it becomes clear that the root corresponding to k = 2 minimizes (11).
Revised: By substituting f k into (14) and checking inequality (15) it becomes clear that the root corresponding to k = 2 minimizes (11).

Original: By using trigonometric identities the root corresponding to k = 2 can be further simpliﬁed to
Revised: By using trigonometric identities the root corresponding to k = 2 can be further simpliﬁed to

Original: [Equation]
Revised: [Equation]

Original: which has the approximate shape of the Tanh activation.
Revised: which has the approximate shape of the Tanh activation.

Original: An example of this operator is shown in Figure 2.
Revised: An example of this operator is shown in Figure 2.

Original: The proximal operator corresponding to the
Revised: The proximal operator corresponding to the

Original: Sigmoid activation can be derived in a similar fashion by setting g ( x ) = −
Revised: Sigmoid activation can be derived in a similar fashion by setting g ( x ) = −

Original: γ log( x ) − γ log( 1 − x ) .
Revised: γ log( x ) − γ log( 1 − x ) .

Original: D T ENSOR P RELIMINARIES
Revised: D T ENSOR P RELIMINARIES

Original: The exposition presented in the paper requires some deﬁnitions related to tensors and operators on tensors.
Revised: The exposition presented in the paper requires some deﬁnitions related to tensors and operators on tensors.

Original: We summarize the material here.
Revised: We summarize the material here.

Original: In all subsequent deﬁnitions, we assume
Revised: In all subsequent deﬁnitions, we assume

Original: D ∈ R n 1 × n 2 × n 3 × n 4 and (cid:126) X ∈ R n 2 × 1 × n 3 × n 4 .
Revised: D ∈ R n 1 × n 2 × n 3 × n 4 and (cid:126) X ∈ R n 2 × 1 × n 3 × n 4 .

Original: Deﬁnition D.1.
Revised: Deﬁnition D.1.

Original: (Bibi & Ghanem, 2017)
Revised: (Bibi & Ghanem, 2017)

Original: The t-product between high-order tensors is deﬁned as
Revised: The t-product between high-order tensors is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (19) where circ HO ( D ) ∈ R n 1 n 3 n 4 × n 2 n 3 n 4 and MatVec HO (cid:16) (cid:126) X (cid:17) ∈ R n 2 n 3 n 4 × 1 .
Revised: (19) where circ HO ( D ) ∈ R n 1 n 3 n 4 × n 2 n 3 n 4 and MatVec HO (cid:16) (cid:126) X (cid:17) ∈ R n 2 n 3 n 4 × 1 .

Original: The operator circ HO ( . ) unfolds an input tensor into a structured matrix.
Revised: The operator circ HO ( . ) unfolds an input tensor into a structured matrix.

Original: On the other hand,
Revised: On the other hand,

Original: MatVec HO ( . ) unfolds an input tensor into a vector.
Revised: MatVec HO ( . ) unfolds an input tensor into a vector.

Original: The fold and unfold procedures are detailed in Bibi & Ghanem (2017).
Revised: The fold and unfold procedures are detailed in Bibi & Ghanem (2017).

Original: Deﬁnition D.2.
Revised: Deﬁnition D.2.

Original: (Bibi & Ghanem, 2017)
Revised: (Bibi & Ghanem, 2017)

Original: The operator bdiag ( D [Equation] =  
Revised: The operator bdiag ( D ) =  

Original: D 
Revised: D (: , : , 1 , 1) 0 n

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: n. . . 0 n 1 × n. . .

Original: [Equation] [Equation] [Equation] [Equation]
Revised: D (: , : , i, j ) . . .

Original: D
Revised: D

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: j
Revised: n

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation] [Equation]
Revised: n

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: (20) where bdiag ( . ) :
Revised: (20) where bdiag ( . ) :

Revised: C

Original: [Equation]
Revised: [Equation]

Original: n 3 n 4 , maps a tensor to a block diagonal matrix of all the frontal faces D (: , : , i, j ) .
Revised: 3 n 4 , maps a tensor to a block diagonal matrix of all the frontal faces D (: , : , i, j ) .

Original: Note that if n 3 = n 4 = 1 , bidag () is an identity mapping.
Revised: Note that if n 3 = n 4 = 1 , bdiag () is an identity mapping.

Original: Moreover if n 1 = n 2 = 1 , bidag ( . ) is a diagonal matrix.
Revised: Moreover if n 1 = n 2 = 1 , bdiag ( . ) is a diagonal matrix.

Original: Due to the structure of the tensor unfold of circ HO ( . ) , the resultant matrix circ HO ( D ) exhibits the following blockwise diagonalization:
Revised: Due to the structure of the tensor unfold of circ HO ( . ) , the resultant matrix circ HO ( D ) exhibits the following blockwise diagonalization:

Original: [Equation]
Revised: [Equation]

Original: (21) where F n is the n × n Normalized Discrete Fourier Matrix.
Revised: (21) where F n is the n × n Normalized Discrete Fourier Matrix.

Original: Note that ˆ D has the dimensions n 3 and n 4 replaced with the corresponding 2D Discrete Fourier Transforms.
Revised: Note that ˆ D has the dimensions n 3 and n 4 replaced with the corresponding 2D Discrete Fourier Transforms.

Original: That is ˆ D ( i, j, : , :) is the 2D
Revised: That is ˆ D ( i, j, : , :) is the 2D

Original: Discrete Fourier Transform of D ( i, j, : , :) .
Revised: Discrete Fourier Transform of D ( i, j, : , :) .

Original: For more details,  reader is advised to start with third order tensors in the work of Kilmer & Martin (2011) and move to the work of Bibi & Ghanem (2017) for extension to higher orders.
Revised: For more details, the reader is advised to start with third order tensors in the work of Kilmer & Martin (2011) and move to the work of Bibi & Ghanem (2017) for extension to higher orders.

Original: E P ROOF OF L EMMA
Revised: E P ROOF OF L EMMA

Original: Since non-linear activations commonly used in deep learning are elementwise, we only need to show that A (cid:126) HO (cid:126) X l − 1 performs a convolution.
Revised: Since non-linear activations commonly used in deep learning are elementwise, we only need to show that A (cid:126) HO (cid:126) X l − 1 performs a convolution.

Original: In particular we need to show, that A (cid:126) HO (cid:126)
Revised: In particular we need to show, that A (cid:126) HO (cid:126)

Original: X l −is equivalent to ( i ) performing 2D convolutions spatially along the third and fourth dimensions.
Revised: X l −is equivalent to ( i ) performing 2D convolutions spatially along the third and fourth dimensions.

Original: ( ii )
Revised: ( ii )

Original: It aggregates the result along the feature dimension n 1 .
Revised: It aggregates the result along the feature dimension n 1 .

Original: ( iii )
Revised: ( iii )

Original: It repeats the procedure for each of the n 2 ﬁlters independently.
Revised: It repeats the procedure for each of the n 2 ﬁlters independently.

Original: We will show the following using direct manipulation of the properties of (cid:126)
Revised: We will show the following using direct manipulation of the properties of (cid:126)

Original: HO (Bibi & Ghanem, 2017).
Revised: HO (Bibi & Ghanem, 2017).

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that Equation (22) shows that features are aggregated along the n 1 dimension.
Revised: Note that Equation (22) shows that features are aggregated along the n 1 dimension.

Original: Now, by showing that A (: , i, : , :) (cid:126) HO (cid:126) X l − 1 ( i, : , : , :) performs n 2 independent 2D convolutions along on the i th channel, the Lemma 1 is proven.
Revised: Now, by showing that A (: , i, : , :) (cid:126) HO (cid:126) X l − 1 ( i, : , : , :) performs n 2 independent 2D convolutions along on the i th channel, the Lemma 1 is proven.

Original: For ease of notation, consider two tensors (cid:126) U ∈ R n 2 × 1 × W × H and (cid:126) Y ∈ R 1 × 1 × W × H
Revised: For ease of notation, consider two tensors (cid:126) U ∈ R n 2 × 1 × W × H and (cid:126) Y ∈ R 1 × 1 × W × H

Original: then we have the following:[Equation]
Revised: then we have the following:

Original: Y

Original: [Equation]

Original: (21) = fold HO 

Original: [Equation]

Original: = fold

Original: [Equation]

Original: fold

Original: [Equation]
Revised: [Equation]

Original:  
Revised: 

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: 
Revised: 

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original:    
Revised:    

Original: Note that (cid:126) ˆ G is the elementwise product of the 2D Discrete Fourier Transform between a feature of an input activation (cid:126) Y and the 2D Discrete Fourier Transform of every ﬁlter of the n 2 in (cid:126) Y .
Revised: Note that (cid:126) ˆ G is the elementwise product of the 2D Discrete Fourier Transform between a feature of an input activation (cid:126) Y and the 2D Discrete Fourier Transform of every ﬁlter of the n 2 in (cid:126) Y .

Original: Since ( F H ⊗ F W ⊗ I n 2 ) is the inverse 2D Fourier transform along each of the n 2 ﬁlters resulting in (cid:126) ˆ G .
Revised: Since ( F H ⊗ F W ⊗ I n 2 ) is the inverse 2D Fourier transform along each of the n 2 ﬁlters resulting in (cid:126) ˆ G .

Original: Thus (cid:126) U (cid:126) HO (cid:126)
Revised: Thus (cid:126) U (cid:126) HO (cid:126)

Original: Y performs 2D convolutions independently along each of the n 2 ﬁlters, combined with (22); thus, Lemma 1 is proven.
Revised: Y performs 2D convolutions independently along each of the n 2 ﬁlters, combined with (22); thus, Lemma 1 is proven.

Original: F P ROOF OF L EMMA
Revised: F P ROOF OF L EMMA

Original: Since,
Revised: Since,

Original: [Equation]
Revised: [Equation]

Original: The second eqality also follows from the separability of (cid:107) .
Revised: The second equality also follows from the separability of (cid:107) .

Original: (cid:107) 2 F where we dropped fold HO .
Revised: (cid:107) 2 F where we dropped fold HO .

Original: The last equality follows from the linearity of (cid:126) HO .
Revised: The last equality follows from the linearity of (cid:126) HO .

Original: Moreover, from (19), we have:
Revised: Moreover, from (19), we have:

Original: [Equation]
Revised: [Equation]

Original: HO
Revised: HO

Original: [Equation]
Revised: [Equation]

Original: ( F H ⊗ F W ⊗ I n
Revised: ( F H ⊗ F W ⊗ I n

Original: 2 ) H (cid:35) H (cid:34) ( F H ⊗ F W ⊗ I n 1 ) bdiag (cid:16) ˆ A H (cid:17) ( F H ⊗ F W ⊗ I n 2 ) H (cid:35) =
Revised: 2 ) H (cid:35) H (cid:34) ( F H ⊗ F W ⊗ I n 1 ) bdiag (cid:16) ˆ A H (cid:17) ( F H ⊗ F W ⊗ I n 2 ) H (cid:35) =

Original: ( F H ⊗ F W ⊗ I n 2 ) bdiag (cid:16) ˆ A (cid:17) bdiag (cid:16) ˆ A H (cid:17) ( F H ⊗ F W ⊗ I n 2 ) H (20) =
Revised: ( F H ⊗ F W ⊗ I n 2 ) bdiag (cid:16) ˆ A (cid:17) bdiag (cid:16) ˆ A H (cid:17) ( F H ⊗ F W ⊗ I n 2 ) H (20) =

Original: ( F H ⊗ F W ⊗ I n 2 )   ˆ A (: , : , 1 , 1)
Revised: ( F H ⊗ F W ⊗ I n 2 )   ˆ A (: , : , 1 , 1)

Original: ˆ A H (: , : , 1 , 1)   
Revised: ˆ A H (: , : , 1 , 1)

Original: ˆ A (: , : , 1 , 2)
Revised: ˆ A (: , : , 1 , 2)

Original: ˆ A H (: , : , 1 , 2)...
Revised: ˆ A H (: , : , 1 , 2)...

Original: 
Revised: 

Original:  ( F H ⊗ F W ⊗ I n 2 ) H =
Revised:  ( F H ⊗ F W ⊗ I n 2 ) H =

Original: ( F H ⊗ F W ⊗ I n 2 )  
Revised: ( F H ⊗ F W ⊗ I n 2 )  

Original: U. . . U  . . .
Revised: U. . . U  . . .

Original:  (cid:124) (cid:123)(cid:122) (cid:125) eigenvectors   Σ   Σ . . .
Revised:   (cid:124) (cid:123)(cid:122) (cid:125) eigenvectors   Σ   Σ . . .

Original:    
Revised:    

Original: U H. . . U H  . . .
Revised: U H. . . U H  . . .

Original: 
Revised: 

Original:  ( F H ⊗ F W ⊗ I n 2 ) H .
Revised:  ( F H ⊗ F W ⊗ I n 2 ) H .

Original: The second equality follows from the orthogonality of F H ⊗ F W ⊗ I n 1 .
Revised: The second equality follows from the orthogonality of F H ⊗ F W ⊗ I n 1 .

Original: The fourth equality follows from ˆ A (: , : , i, j ) ˆ A H (: , : , i, j ) =
Revised: The fourth equality follows from ˆ A (: , : , i, j ) ˆ A H (: , : , i, j ) =

Original: U ij Σ ij U H ij .
Revised: U ij Σ ij U H ij .

Original: Thus by combining (23) and (24), we have:L [Equation]
Revised: Thus by combining (23) and (24), we have:

Original:  max (cid:16) ∇ [Equation] F ( (cid:126) X ) 
Revised: L = λ max (cid:16) ∇ 2 F ( (cid:126) X ) (cid:17)

Original: [Equation]
Revised: = λ

Original: max     Σ   Σ
Revised: max     Σ   Σ

Original: [Equation]
Revised: [Equation]

Original: G P ROOF OF P ROPOSITION
Revised: G P ROOF OF P ROPOSITION

Original: Lemma 3.
Revised: Lemma 3.

Original: For τ -nice Prox-SG,
Revised: For τ -nice Prox-SG,

Original: [Equation]
Revised: [Equation]

Original: is an unbiased estimator to F ( (cid:126) X )
Revised: is an unbiased estimator to F ( (cid:126) X )

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: (cid:34)(cid:88)
Revised: (cid:34)(cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: X ) .
Revised: X ) .

Original: The ﬁrst equality follows by introducing an indicator function where 1 i ∈ S
Revised: The ﬁrst equality follows by introducing an indicator function where 1 i ∈ S

Original: = 1 if i ∈ S and zero otherwise.
Revised: = 1 if i ∈ S and zero otherwise.

Original: The last equality follows from the uniformity across elements of the τ -nice S .
Revised: The last equality follows from the uniformity across elements of the τ -nice S .

Original: From Lemma
Revised: From Lemma

Original: [Equation]
Revised: [Equation]

Original: and with zero initialization it follows 
Revised: and with zero initialization it follows that

Original: [Equation]
Revised: [Equation]

Original: is an unbiased estimator of ∇ F ( (cid:126) X ) (cid:12) (cid:12) (cid:126) X
Revised: is an unbiased estimator of ∇ F ( (cid:126) X ) (cid:12) (cid:12) (cid:126) X

Original: = 0 .
Revised: = 0 .

Original: The last iteration follows by noting that A = (cid:0) A H (cid:1) H .
Revised: The last iteration follows by noting that A = (cid:0) A H (cid:1) H .

Original: Therefore, the ﬁrst iteration of τ -nice Prox-SGD with zero initialization and unit step size is:
Revised: Therefore, the ﬁrst iteration of τ -nice Prox-SGD with zero initialization and unit step size is:

Original: [Equation]
Revised: [Equation]

Original: Note that the previous stochastic sum in (26) with τ = (1 − p ) n 1 can be reparameterized as follows:
Revised: Note that the previous stochastic sum in (26) with τ = (1 − p ) n 1 can be reparameterized as follows:

Original: [Equation]
Revised: [Equation]

Original: where M ∈ R n 1 × 1 × W × H is a mask tensor.
Revised: where M ∈ R n 1 × 1 × W × H is a mask tensor.

Original: Note that since τ = (1 − p ) n 1 , M has exactly pn 1 slices M ( i, : , : , :) that are completely zero.
Revised: Note that since τ = (1 − p ) n 1 , M has exactly pn 1 slices M ( i, : , : , :) that are completely zero.

Original: This equivalent to a dropout layer where the layer drops exactly pn 1 input activations.
Revised: This equivalent to a dropout layer where the layer drops exactly pn 1 input activations.

Original: It follows that (27) is equivalent to a forward pass through a
Revised: It follows that (27) is equivalent to a forward pass through a

Original: BerDropout p layer followed by a linear layer and non-linear activation.
Revised: BerDropout p layer followed by a linear layer and non-linear activation.

Original: H P ROOF OF P ROPOSITION
Revised: H P ROOF OF P ROPOSITION

Original: mS2GD (Koneˇcn`y et al., 2016) with zero initialization at the ﬁrst epoch deﬁnes the following update:
Revised: mS2GD (Koneˇcn`y et al., 2016) with zero initialization at the ﬁrst epoch deﬁnes the following update:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Note with zero initialization at the ﬁrst epoch we have Y = 0 ; therefore,
Revised: With zero initialization at the ﬁrst epoch we have Y = 0 , therefore

Original: [Equation]
Revised: [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation]

Original: H ( i

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: :) (cid:1) MatVec (cid:16) (cid:126)

Original: [Equation]

Original: (cid:17) −

Original: [Equation]

Original: (cid:16) (cid:126)

Original: [Equation]

Original: (cid:17)(cid:105)

Original: [Equation]

Original: (cid:126) B − (cid:34)

Original: [Equation]

Original: B

Original: [Equation]

Original: =

Original: [Equation]

Original: τ
Revised: τ

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation] i [Equation] Substituting both (29) and (30) into (28) we have(cid:126) X ← Prox 1 L g (cid:32) (cid:126) X
Revised: Substituting both (29) and (30) into (28) we have

Original: + A (cid:126)

Original: HO (cid:126)

Original: X l

Original: [Equation]
Revised: [Equation]

Original: 1 + (cid:126) B − (cid:32) n [Equation] τ (cid:88) i ∈ S A (: , i, : , :) (cid:126) HO
Revised: Fully-connected/Convolutional

Original: [Equation]
Revised: [Equation]

Original: H ( i, : , : , :) (cid:126) HO (cid:126) X (cid:33)(cid:33) = Prox 1 L g   A (cid:126) [Equation] (cid:126)
Revised: 

Original: X l −

Original: [Equation]
Revised: [Equation]

Original: (cid:126) B (cid:124) (cid:123)(cid:122) (cid:125)

Original: Fully-connected/Convolutional + (cid:32) [Equation] Note that I ∈ R n 2 × n 2 × W × H is the identity tensor in which all frontal slices are identity I (: , : , i, j ) 
Revised: Note that I ∈ R n 2 × n 2 × W × H is the identity tensor in which all frontal slices are identity I (: , : , i, j ) =

Original: = I n 2 × n 2 and all other slices are zeros.
Revised: I n 2 × n 2 and all other slices are zeros.

Original: Note that following the deﬁnition in (19), we have
Revised: Following the deﬁnition in (19), we have

Original: I (cid:126) HO (cid:126) X
Revised: I (cid:126) HO (cid:126) X

Original: = (cid:126) X .
Revised: = (cid:126) X .

Original: I R ANDOMIZED C OORDINATE D ESCENT P ERMUTES D ROPOUT AND L INEAR L AYER
Revised: I R ANDOMIZED C OORDINATE D ESCENT P ERMUTES D ROPOUT AND L INEAR L AYER

Original: We present an additional insight to the role of stochastic solvers on (5) in network design.
Revised: We present an additional insight to the role of stochastic solvers on (5) in network design.

Original: In particular, we show that performing a randomized coordinate descent, RCD, on (5) ignoring the ﬁnite sum structure, is equivalent to a linear transformation followed by BerDropout p and a non-linear activation.
Revised: In particular, we show that performing a randomized coordinate descent, RCD, on (5) ignoring the ﬁnite sum structure, is equivalent to a linear transformation followed by BerDropout p and a non-linear activation.

Original: That is, performing RCD permutes the order of linear transformation and dropout.
Revised: That is, performing RCD permutes the order of linear transformation and dropout.

Original: For ease of notation, we show this under the special case of fully connected layers.
Revised: For ease of notation, we show this under the special case of fully connected layers.

Original: Proposition 6.
Revised: Proposition 6.

Original: A single iteration of Randomized Coordinate Descent, e.g. NSync (Richt´arik &
Revised: A single iteration of Randomized Coordinate Descent, e.g. NSync (Richt´arik &

Original: Tak´aˇc, 2016), with τ -nice sampling of coordinates of (5) with τ = (1 − p ) n 2 , unit step sizes along each partial derivative, and with zero initialization is equivalent to:
Revised: Tak´aˇc, 2016), with τ -nice sampling of coordinates of (5) with τ = (1 − p ) n 2 , unit step sizes along each partial derivative, and with zero initialization is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to a forward pass through a linear layer followed by a BerDropout p layer (that drops exactly n 2 p output activations) followed by a non-linear activation.
Revised: which is equivalent to a forward pass through a linear layer followed by a BerDropout p layer (that drops exactly n 2 p output activations) followed by a non-linear activation.

Original: Proof.
Revised: Proof.

Original: We provide a sketch of the proof on the simple quadratic F ( x ) = 12 (cid:107) A[Equation] x −
Revised: We provide a sketch of the proof on the simple quadratic F ( x ) = 12 (cid:107) A

Original: [Equation]
Revised: [Equation]

Original: b
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x where the linear layer is a fully-connected layer.
Revised: where the linear layer is a fully-connected layer.

Original: Considering a randomized coordinate descent, e.g.
Revised: Considering a randomized coordinate descent, e.g.

Original: NSync, with τ -nice sampling of the coordinates we have the following:
Revised: NSync, with τ -nice sampling of the coordinates we have the following:

Original: [Equation]
Revised: [Equation]

Original: Note that e i is a vector of all zeros except the i th coordinate which is equal to 1.
Revised: Note that e i is a vector of all zeros except the i th coordinate which is equal to 1.

Original: Moreover, since the step sizes along each partial derivative is 1, v = 1 .
Revised: Moreover, since the step sizes along each partial derivative is 1, v = 1 .

Original: Equation (32) is equivalent to a forward pass through a linear layer followed by a BerDropout p layer and a non-linear activation.
Revised: Equation (32) is equivalent to a forward pass through a linear layer followed by a BerDropout p layer and a non-linear activation.

Original: J R ES N ET B LOCKS
Revised: J R ES N ET B LOCKS

Original: In here, we brieﬂy discuss how a ResNet block can be incorporated as an iteration of a solver.
Revised: In here, we brieﬂy discuss how a ResNet block can be incorporated as an iteration of a solver.

Original: The key observation is that a ResNet block has two non-linearities connected through a skip connection.
Revised: The key observation is that a ResNet block has two non-linearities connected through a skip connection.

Original: In particular, consider the l th ResNet block with two linear operators denotes as A (1) and A (2) with biases as B (1) and B (2) , respectively.
Revised: In particular, consider the l th ResNet block with two linear operators denotes as A (1) and A (2) with biases as B (1) and B (2) , respectively.

Original: The input activations to the ResNet block form the previous layer Therefore, a forward pass through a ResNet block are X k − 1 where X l are the output activations.
Revised: The input activations to the ResNet block form the previous layer. Therefore, a forward pass through a ResNet block are X k − 1 where X l are the output activations.

Original: A forward pass through a ResNet block exhibits the following functional form:
Revised: A forward pass through a ResNet block exhibits the following functional form:

Original: [Equation]
Revised: [Equation]

Original: Note that this can be seen as two blocks of linear followed by nonlinear composed such that:
Revised: This can be seen as two blocks of linear followed by nonlinear composed such that:

Original: [Equation]
Revised: [Equation]

Original: Note that a a batch normalization layer is linear during the forward pass through the network; therefore, it can be absorbed in A and B .
Revised: Note that a a batch normalization layer is linear during the forward pass through the network and thus can be absorbed in A and B .

Original: Thereafter, a forward pass through a ResNet block can be viewed as performing a single iteration of two consecutive stochastic or deterministic solvers, depending on if there are Dropout layers, on (5).K R ESPONSE T O R  We thank R1 for the positive comments and review. (1) Deﬁnition of λ max in Eq (3).
Revised: Consequently, a forward pass through a ResNet block can be viewed as performing a single iteration of two consecutive stochastic or deterministic solvers (depending on the presence or absence of Dropout layers) on (5).

Original: We have adjusted the text below Eq (3) to clearly state that λ max ( . ) is the maximum eigenvalue function.

Original: (2) On the implementation of solvers.

Original: The block of layers (dropout followed by linear and nonlinear layers) is replaced with an iterative solver, i.e. recurrent layer, that performs the Prox operator several times before generating an output.

Original: This is for the forward pass through the network.

Original: As for backpropagation, it is performed by simply unrolling the layers.

Original: This is commonly referred to as backpropagation through time in recurrent neural networks.

Original: While this is not particularly efﬁcient in general, there are several potential ways to improve this by taking gradients implicitly through the argmin operator [1,2].

Original: We leave this to future work.

Original: (3) On the number of iterations.

Original: The number of iterations was always kept constant.

Original: It was set to 10 as stated at the end of page 6 for all small networks.

Original: As for larger networks such as VGG16, this number is ﬁxed to 30 iterations as discussed in page 7 (just below Table 3).

Original: At present we do not have a universal criterion for choosing the number of iterations; this is treated as a hyperparameter.

Original: [1] “Techniques smooth

Original: Peter Ochs, Ren Ranftl, Thomas Brox, Thomas Pock.

Original: ] “On

Original: , Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa

Original: Cruz, Edison Guo.

Original: L R ESPONSE TO R

Original: * We recommend reading this response in the revised PDF uploaded to OpenReview.

Original: This response is in Appendix L.

Original: The mathematical notation is easier to read in the PDF.

Original: We thank R2 for the comments and the positive feedback on the novelty of our approach.

Original: R2 raised several issues regarding the proofs.

Original: We have proofread all the proofs and there are no factual errors.

Original: In fact, some of the main results, e.g. Lemma 2, have also been veriﬁed numerically.

Original: However, there were some minor typos and non-standard notation that may have been confusing.

Original: We have corrected these typos in the revised version uploaded to OpenReview.

Original: Below is a detailed answer to all of R2’s concerns.

Original: General Comments.

Original: (1) Unclear technical details and mistakes in the proofs.

Original: There are no mistakes in the proofs.

Original: We have thoroughly checked them and they are all correct.

Original: There were some typos that may have been behind R2’s confusion.

Original: We have corrected these typos and improved the notation.

Original: (2) Limitations of the current method.

Original: What about ResNets and BatchNorm?

Original: Our framework can be directly applied to ResNets.

Original: A ResNet block can be viewed as two consecutive stochastic solvers.

Original: We have added Appendix J discussing this in the supplementary material.

Original: This approach is somewhat simplistic; there is room for exciting future work.

Original: Normalization layers are easy to handle.

Original: Note that during test time normalization layers are linear; thus they can be combined with the fully-connected or convolutional layer as a single linear layer.

Original: Detailed Comments

Original: (1) Deﬁnition of g ( x ) and f i ( x ) .

Original: The structure in problem (1) is standard in the machine learning and optimization communities [1,2,3,4].

Original: As suggested by R2 and R3, we have added further details and examples in the ﬁrst page.

Original: Note that for instance, if f i ( a (cid:62) i x ) = 12 ( a (cid:62) i x − y i ) 2 and g ( x ) = 12 (cid:107) x (cid:107) 22 , we recover ridge regression, while with g ( x )

Original: [-= (cid:107) x (cid:107) 
-]
Revised: [-= (cid:107) x (cid:107) 
-]

Original: we recover LASSO regression.

Original: (2) The motivation and some details of Function (2).

Original: In addition, x should be corrected as x l .

Original: The missing superscript was corrected.

Original: It is not clear to us what R2 is asking for in terms of extra motivation and details

Original: Could you clarify, so that we can alleviate this concern in the paper?

Original: (3) Is Equation (3) wrong?

Original: No, Eq (3) is correct.

Original: Note that x l is the optimization variable while x l − 1 is a ﬁxed optimization parameter.

Original: R2 is confusing x l − 1 (the activation output of the l − 1 layer) with the optimization variable x l (the output activations of layer l ).

Original: Thus, unlike what is stated by R2, the

Original: Prox-GD update has the following form: x l ← Prox (cid:0) x l − 1 L ∇ F ( x l ) (cid:1) .

Original: Consequently, we have x l ← Prox (cid:0) x

Original: l − 1 L (cid:0) AA (cid:62) x l − Ax l − 1 − b (cid:1)(cid:1) , which is identical to Eq (3).

Original: Lastly, R2 is asking to prove “the Lipschitz constant w.r.t.

Original: maximal eigenvalue”.

Original: We do not understand the request.

Original: It is well-known that the maximum eigenvalue of the Hessian of F ( x l ) induces the tightest quadratic upper bound for the smooth part F ( x ) .

Original: This leads to the optimal largest step size 1 L in Prox-GD.

Original: This is well-known and straightforward to show [5].

Original: (4) Deﬁnition of fold H0 ( . ) ?

Original: Is the dimensionality of bdiag ( D ) wrong?

Original: Why is bdiag ( D ) an identity mapping when n 3 = n 4 ?

Original: a) The deﬁnition of fold HO ( . ) can be found in [11] and in [12] for third-order tensors.

Original: The operator fold H0 ( . )

Original: reshapes the elements of a matrix

Original: into a tensor.

Original: The precise reshape procedure can be best described as the inverse reshape to the tensor-to-matrix unfold operator MatVec HO ( . )

Original: such that the following holds: fold HO ( MatVec HO ( A ))

Original: =

Original: A .

Original: For example, for a third-order tensor

Original: A ∈ R n 1 × n 2 × n 3 , MatVec HO ( A ) ∈ R n 1 n 3 × n 2 , which is a matrix, while fold HO ( MatVec HO ( A )) reshapes that matrix into the original tensor A ∈ R n 1 × n 2 × n 3 .

Original: b) Regarding bdiag ( D ) , there are indeed two typos.

Original: The dimensionality of bdiag ( D ) is C n 1 n 3 n 4 × n 2 n 3 n 4 .

Original: If n 3 = n 4 = 1 then bdiag ( D ) is an identity mapping.

Original: We have corrected the typos in the revision.

Original: (5) Some issues in Eqs (7,8).

Original: Does Eq (25) miss the operator fold H0 ( . ) in Appendix G?

Original: Eqs (7,8,25) are correct with nothing missing.

Original: Eq (25) is not missing any operator.

Original: The reason fold HO ( . ) did not appear in Eq (25) simply follows from the fact that the Frobenius norm of a tensor and the Frobenius norm of the unfolded tensor are identical since fold HO ( . )

Original: performs reordering of the elements.

Original: That is, (cid:107) MatVec HO

Original: ( A ) (cid:107) 2 F = (cid:107) fold HO ( MatVec HO ( A ))

Original: (cid:107) 2 F .

Original: This has also been discussed in the text below Eq (23) as this identity appears there too.

Original: (6) On the proof of Lemma 2 and the orthogonality of F H ⊗ F W ⊗ I n 1 .

Original: Is the third and fourth equalities in (24) wrong?

Original: On the fourth equality in (24).

Original: The proof in Lemma 2 is correct.

Original: First, note that the normalized DFT matrices F H and F W and the identity matrix I are orthogonal/unitary.

Original: It is trivial to prove using properties of the Kronecker product that the Kronecker product of orthogonal matrices is an orthogonal matrix.

Original: Note that for matrices A , B , C , D of appropriate sizes, ( A ⊗ B ) (cid:62)

Original: =

Original: A (cid:62) ⊗ B (cid:62) and ( A ⊗ B )( C ⊗ D )

Original: =

Original: AC ⊗

Original: BD .

Original: Using the previous properties, the proof follows trivially.

Original: The only typo is in the text below

Original: Eq (24).

Original: [Equation]

Original: Σ

Original: [Equation]

Original: That is, we are performing an eigendecomposition of the faces of the tensors which are matrices.

Original: We have corrected the typo and changed the notation a bit in Eq (24) for further clarity.

Original: Comments on Experiments.

Original: (1) Training Networks is equivalent to optimizing proximal solvers.

Original: Why can training networks with solvers replacing blocks of layers improve accuracy?

Original: Reasonable explanations should be provided.

Original: In general, we do not have theoretical generalization bounds justifying the improvement.

Original: However, similar observations [6,7] have been made in various contexts upon designing networks.

Original: It seems that the recurrent structure in hidden activations has merits in improving generalization but no one has thorough theoretical foundations for this yet.

Original: Such behaviour has also been observed very recently in [8] where ODE solvers are used rather than optimization solvers.

Original: Our hypothesis is that

Original: since

Original: the

Original: output of a

Original: feed-forward network approximates the optimal solution of some convex optimization

Original: in each

Original: layer

Original: , attaining better solutions by performing more iterations of the corresponding solver may improve accuracy.

Original: (2) Optimizing a convex objective can easily obtain the optimal solution.

Original: What happens if solvers are used to replace more blocks of layers?

Original: Complexity analysis for these should be provided.

Original: Optimizing convex objectives is easier than other problems in the sense of global optimality guarantees and analysis of various deterministic and stochastic algorithms.

Original: However, achieving high accuracy solutions may still require performing a large number of iterations (e.g., > 10 3 ).

Original: In our framework, larger number of iterations will increase the computational complexity of the network, particularly during training.

Original: On the other hand, replacing more blocks of layers with solvers (each with a small number of iterations) has the potential to improve the performance as highlighted in Table (4).

Original: However, replacing more blocks of feed-forward networks with solvers will suffer from diminishing returns, i.e. the improvements will tend to be smaller with more blocks replaced with solvers.

Original: This is ﬁrstly because many networks are already close to saturating datasets (e.g.,

Original: VGG16 on CIFAR10).

Original: That is to say that the baseline networks are already achieving high accuracy ( > 90% ).

Original: Secondly, some networks do not have enough capacity to improve on certain datasets (e.g., AlexNet on CIFAR-100).

Original: Note that replacing blocks of layers with solvers does not increase capacity; thus the improvement is attributed to the new structure and not to any form of over-parameterization of the network.

Original: As for complexity, adding t layers of a solver is as expensive as performing t feed-forward passes through a single layer.

Original: (3) The experiments are only conducted on two datasets (i.e., CIFAR-10, CIFAR-100).

Original: It would be better to compare more baselines on other datasets, such as ImageNet.

Original: We were mostly interested in the theory around connecting stochastic solvers to feed-forward passes through networks and generalizing linear layers through tensor theory, which will potentially inspire the design of new dropout layers.

Original: Many machine learning and optimization papers conduct experiments on MNIST only, or MNIST and CIFAR-10 [8,9,10].

Original: We have conducted experiments on both

Original: CIFAR-10 and CIFAR-100, which match or exceed the complexity of datasets used in multiple comparable papers published in NIPS/ICML/ICLR in recent years.

Original: We do not think that it is necessary for our work to conduct experiments on ImageNet

Original: M R ESPONSE TO R

Original: * We recommend reading this response in the revised PDF uploaded to OpenReview.

Original: This response is in Appendix M.

Original: The mathematical notation is easier to read in the PDF.

Original: We thank R3 for the positive review and feedback.

Original: Below are our responses to all concerns.

Original: On the major concerns

Original: Some

Original: clarity on the forward/backward pass of networks with Prox solvers

Original: R3’s description of the forward pass through the network with a Prox solver is correct.

Original: In general, the best way to understand how a network with a Prox solver operates in both forward and backward passes is to think of that layer with a Prox solver as a recurrent neural network.

Original: Thus, asking how one performs a backward pass through such a layer is equivalent to asking how one would perform a backward pass through a recurrent neural network.

Original: The backward pass through the Prox solver is still performed: not through the original network as R3 thought, but through the same network with the Prox solver, akin to backpropagation-through-time (BPTT).

Original: This resembles the parameter update procedure in recurrent neural networks.

Original: (2) Adjusting baselines to perform the same amount of computation for a fair comparison.

Original: R3 is correct about the fact that networks with Prox solvers do in fact perform more computation.

Original: However, the capacity of both networks (baseline and the Prox solver network) is identical.

Original: This means that both networks have the same exact number of parameters and there is no advantage of the

Original: Prox solvers in terms of capacity over the baseline.

Original: This is the essential factor for a fair comparison, since accuracy is often considered as a function of network capacity rather than the amount of computation.

Original: Moreover, note that to the best of our knowledge we have reported the best results for the baselines in comparison to any publicly available online repository that performs training without using any test data statistics (i.e. proper training).

Original: For instance, the results of VGG16 on

Original: CIFAR-10 are comparable to or better than some ResNet architectures on the same dataset.

Original: We are not aware of better numbers for the corresponding networks on the corresponding datasets.

Original: On the minor issues

Original: (1) Missing deﬁnitions.

Original: We have addressed this in the revised version.

Original: We provided several examples of g ( x ) when it was ﬁrst introduced in the ﬁrst page.

Original: We have explicitly, as suggested, deﬁned p in Proposition 1.

Original: (2) Give examples of where the prox problems in Table 1 show up in practice (outside of activation functions in neural networks)

Original: We are only aware of applications in activation functions in neural networks.

Original: But this is sufﬁcient motivation for us.

Original: (3) On the statement “for different choices of dropout rate the baseline can always be improved by...”

Original: in the Experiments.

Original: We have softened the statement in the revision.

Original: The statement is now “we observe that for different choices of dropout rate the baseline performance improves upon replacing ...”.

Original: (4) Include results for Dropout rate p=0 in Table 5.

Original: We have added this experiment to Table 5.

Original: N C OMMENT ON A CONCURRENT ICLR SUBMISSION

Original: We would like to bring to the attention of the reviewers a well-rated concurrent ICLR submission, “The Singular Values of Convolutional Layers”.

Original: The main result of that paper (Theorem 6) is equivalent to Lemma 2 in our submission.

Original: In our submission, this is a supporting result, the proof of which is relegated to the supplement.

Original: Our proof is simple and is one page long.

Original: This is enabled by

Original: Lemma 1, which enables the use of tensor theory in connecting convolutional and fully-connected layers in a uniﬁed framework.

Original: We believe this speaks to the value of our work as a whole.