P ROXIMAL B ACKPROPAGATION
A BSTRACT
We propose proximal backpropagation (ProxProp) as a novel algorithm that takes implicit instead of explicit gradient steps to update the network parameters during neural network training.
Our algorithm is motivated by the step size limitation of explicit gradient descent, which poses an impediment for optimization.
ProxProp is developed from a general point of view on the backpropagation algorithm, currently the most common technique to train neural networks via stochastic gradient descent and variants thereof.
Speciﬁcally, we show that backpropagation of a prediction error is equivalent to sequential gradient descent steps on a quadratic penalty energy, which comprises the network activations as variables of the optimization.
We further analyze theoretical properties of ProxProp and in particular prove that the algorithm yields a descent direction in parameter space and can therefore be combined with a wide variety of convergent algorithms.
Finally, we devise an efﬁcient numerical implementation that integrates well with popular deep learning frameworks.
We conclude by demonstrating promising numerical results and show that ProxProp can be effectively combined with common ﬁrst order optimizers such as Adam.
I NTRODUCTION
In recent years neural networks have gained considerable attention in solving difﬁcult correlation tasks such as classiﬁcation in computer vision (Krizhevsky et al., 2012) or sequence learning (Sutskever et al., 2014) and as building blocks of larger learning systems (Silver et al., 2016).
Training neural networks is accomplished by optimizing a nonconvex, possibly nonsmooth, nested function of the network parameters.
Since the introduction of stochastic gradient descent (SGD) (Robbins & Monro, 1951; Bottou, 1991), several more sophisticated optimization methods have been studied.
One such class is that of quasi-Newton methods, as for example the comparison of
L-BFGS with SGD in (Le et al., 2011), Hessian-free approaches (Martens, 2010), and the Sum of
Functions Optimizer in (Sohl-Dickstein et al., 2013).
Several works consider speciﬁc properties of energy landscapes of deep learning models such as frequent saddle points (Dauphin et al., 2014) and well-generalizable local optima (Chaudhari et al., 2017a).
Among the most popular optimization methods in currently used deep learning frameworks are momentum based improvements of classical SGD, notably Nesterov’s Accelerated Gradient (Nesterov, 1983; Sutskever et al., 2013), and the
Adam optimizer (Kingma & Ba, 2015), which uses estimates of ﬁrst and second order moments of the gradients for parameter updates.
Nevertheless, the optimization of these models remains challenging, as learning with SGD and its variants requires careful weight initialization and a sufﬁciently small learning rate in order to yield a stable and convergent algorithm.
Moreover, SGD often has difﬁculties in propagating a learning signal deeply into a network, commonly referred to as the vanishing gradient problem (Hochreiter et al., 2001). L y n
X nz 1 φ na 1 σ nz 2 φ n L −z L −n L −a L − 2 σ φ
Training neural networks can be formulated as a constrained optimization problem by explicitly introducing the network activations as variables of the optimization, which are coupled via layerwise constraints to enforce a feasible network conﬁguration.
The authors of (Carreira-Perpi˜n´an &
Wang, 2014) have tackled this problem with a quadratic penalty approach, the method of auxiliary coordinates (MAC).
Closely related, (Taylor et al., 2016) introduce additional auxiliary variables to further split linear and nonlinear transfer between layers and propose a primal dual algorithm for optimization.
From a different perspective, (LeCun, 1988) takes a Lagrangian approach to formulate the constrained optimization problem.
In this work, we start from a constrained optimization point of view on the classical backpropagation algorithm.
We show that backpropagation can be interpreted as a method alternating between two steps.
First, a forward pass of the data with the current network weights.
Secondly, an ordered sequence of gradient descent steps on a quadratic penalty energy.
Using this point of view, instead of taking explicit gradient steps to update the network parameters associated with the linear transfer functions, we propose to use implicit gradient steps (also known as proximal steps, for the deﬁnition see (6)).
We prove that such a model yields a descent direction and can therefore be used in a wide variety of (provably convergent) algorithms under weak assumptions.
Since an exact proximal step may be costly, we further consider a matrix-free conjugate gradient (CG) approximation, which can directly utilize the efﬁcient pre-implemented forward and backward operations of any deep learning framework.
We prove that this approximation still yields a descent direction and demonstrate the effectiveness of the proposed approach in PyTorch.
M ODEL AND NOTATION
We propose a method to train a general L -layer neural network of the functional form
[Equation]
Here, J ( θ ; X, y ) denotes the training loss as a function of the network parameters θ , the input data
X and the training targets y .
As the ﬁnal loss function L y we choose the softmax cross-entropy for our classiﬁcation experiments.
φ is a linear transfer function and σ an elementwise nonlinear transfer function.
As an example, for fully-connected neural networks θ =
( W, b ) and φ ( θ, a ) =
Wa + b 1 .
While we assume the nonlinearities σ to be continuously differentiable functions for analysis purposes, our numerical experiments indicate that the proposed scheme extends to rectiﬁed linear units (ReLU), σ ( x ) = max(0 , x ) .
Formally, the functions σ and φ map between spaces of different dimensions depending on the layer.
However, to keep the presentation clean, we do not state this dependence explicitly.
Figure 1 illustrates our notation for the fully-connected network architecture.
Throughout this paper, we denote the Euclidean norm for vectors and the Frobenius norm for matrices by ||·|| , induced by an inner product (cid:104)· , ·(cid:105) .
We use the gradient symbol ∇ to denote the transpose of the Jacobian matrix, such that the chain rule applies in the form “inner derivative times outer derivative”.
For all computations involving matrix-valued functions and their gradient/Jacobian, we uniquely identify all involved quantities with their vectorized form by ﬂattening matrices in a column-ﬁrst order.
Furthermore, we denote by A ∗ the adjoint of a linear operator A .
Input: Current parameters θ k .
// Forward pass.
for l = 1 to L − 2 do z kl
= φ ( θ kl , a kl − 1 ) , // a 0 = a kl = σ ( z kl ) .
end for // Perform minimization steps on
a (cid:13) grad.
step on wrt
2 ) for to do and
end for
Output: New parameters θ +1 .
Input: Current parameters θ k .
// Forward pass.
for l = 1 to L − 2 do
= φ θ
− 1 ) , // a end for // Perform
− do − c
Output New
+1 .
The gradient descent iteration on a nested function J ( θ ; X, y ) , [Equation] (2) is commonly implemented using the backpropagation algorithm (Rumelhart et al., 1986).
As the basis for our proposed optimization method, we derive a connection between the classical backpropagation algorithm and quadratic penalty functions of the form [Equation]
The approach of (Carreira-Perpi ˜ n ´ an & Wang, 2014) is based on the minimization of (3), as under mild conditions the limit ρ, γ → ∞ leads to the convergence of the sequence of minimizers of E to the minimizer of J (Nocedal & Wright, 2006, Theorem 17.1).
In contrast to (Carreira-Perpi ˜ n ´ an & Wang, 2014) we do not optimize (3), but rather use a connection of (3) to the classical backpropagation algorithm to motivate a semi-implicit optimization algorithm for the original cost function
J .
Indeed, the iteration shown in Algorithm 1 of forward passes followed by a sequential gradient descent on the penalty function E is equivalent to the classical gradient descent iteration.
Proposition 1.
Let L y , φ and σ be continuously differentiable.
For ρ = γ = 1 /τ and θ k as the input to Algorithm 1, its output θ k +1 satisﬁes (2) , i.e., Algorithm 1 computes one gradient descent iteration on J .
Proof.
For this and all further proofs we refer to Appendix A.
P ROXIMAL BACKPROPAGATION
The interpretation of Proposition 1 leads to the natural idea of replacing the explicit gradient steps a (cid:13) , b (cid:13) and c (cid:13) in Algorithm 1 with other – possibly more powerful – minimization steps.
We propose
Proximal Backpropagation (ProxProp) as one such algorithm that takes implicit instead of explicit gradient steps to update the network parameters θ in step c (cid:13) .
This algorithm is motivated by the step size restriction of gradient descent.
4. G RADIENT DESCENT AND PROXIMAL MAPPINGS
Explicit gradient steps pose severe restrictions on the allowed step size τ : Even for a convex, twice continuously differentiable, L -smooth function f : R n → R , the convergence of the gradient descent algorithm can only be guaranteed for step sizes 0 < τ < 2 / L .
The Lipschitz constant L of the gradient ∇ f is in this case equal to the largest eigenvalue of the Hessian H .
With the interpretation of backpropagation as in Proposition 1, gradient steps are taken on quadratic functions.
As an example for the ﬁrst layer,
[Equation]
In this case the Hessian is H =
XX (cid:62) , which is often ill-conditioned.
For the CIFAR-10 dataset the largest eigenvalue is 6 . 7 · 10 6 , which is seven orders of magnitude larger than the smallest eigenvalue.
Similar problems also arise in other layers where poorly conditioned matrices a l pose limitations for guaranteeing the energy E to decrease.
The proximal mapping (Moreau, 1965) of a function f : R n → R is deﬁned as:
[Equation]
By rearranging the optimality conditions to (5) and taking y = x k , it can be interpreted as an implicit gradient step evaluated at the new point x k +1 (assuming differentiability of f ):
[Equation]
The iterative algorithm (6) is known as the proximal point algorithm (Martinet, 1970).
In contrast to explicit gradient descent this algorithm is unconditionally stable , i.e. the update scheme (6) monotonically decreases f for any τ > 0 , since it holds by deﬁnition of the minimizer x k +1 that f ( x k +1 )
+ 12 τ || x
k +1 − x k || 2 ≤ f ( x k ) .
Thus proximal mappings yield unconditionally stable subproblems in the following sense: The update in θ l provably decreases the penalty energy E ( θ , a k , z k ) from (3) for ﬁxed activations ( a k , z k ) for any choice of step size.
This motivates us to use proximal steps as depicted in Algorithm 2.
4. P ROX P ROP
We propose to replace explicit gradient steps with proximal steps to update the network parameters of the linear transfer function.
More precisely, after the forward pass
[Equation]
we keep the explicit gradient update equations for z l and a l .
The last layer update is
[Equation]
[Equation]
and for all other layers,
[Equation]
[Equation]
where we use a k + 1 / 2 l and z k + 1 / 2 l to denote the updated variables before the forward pass of the next iteration and multiplication in (9) is componentwise.
However, instead of taking explicit gradient steps to update the linear transfer parameters θ l , we take proximal steps
[Equation]
This update can be computed in closed form as it amounts to a linear solve (for details see Appendix B).
While in principle one can take a proximal step on the ﬁnal loss L y , for efﬁciency reasons we choose an explicit gradient step, as the proximal step does not have a closed form solution in many scenarios (e.g. the softmax cross-entropy loss in classiﬁcation problems).
Speciﬁcally,
[Equation]
Note that we have eliminated the step sizes in the updates for z l and a l − 1 in (9) and (10), as such updates correspond to the choice of ρ = γ = 1 τ in the penalty function (3) and are natural in the sense of Proposition 1.
For the proximal steps in the parameters θ in (11) we have introduced a step size τ θ which – as we will see in Proposition 2 below – changes the descent metric opposed to τ which rather rescales the magnitude of the update.
We refer to one sweep of updates according to equations (7) - (12) as ProxProp , as it closely resembles the classical backpropagation (BackProp), but replaces the parameter update by a proximal mapping instead of an explicit gradient descent step.
In the following subsection we analyze the convergence properties of ProxProp more closely.
4.2. C ONVERGENCE OF P ROX P ROP
ProxProp inherits all convergence-relevant properties from the classical backpropagation algorithm, despite replacing explicit gradient steps with proximal steps: It minimizes the original network energy J ( θ ; X, y ) as its ﬁxed-points are stationary points of J ( θ ; X, y ) , and the update direction θ k +1 − θ k is a descent direction such that it converges when combined with a suitable optimizer.
In particular, it is straight forward to combine ProxProp with popular optimizers such as Nesterov’s accelerated gradient descent (Nesterov, 1983) or Adam (Kingma & Ba, 2015).
In the following, we give a detailed analysis of these properties.
Proposition 2.
For l = 1 , . . .
, L − 2 , the update direction θ k +1 − θ k computed by ProxProp meets
[Equation]
In other words, ProxProp multiplies the gradient ∇ θ l J with the inverse of the positive deﬁnite, symmetric matrix
[Equation] [Equation]
which depends on the activations a kl − 1 of the forward pass.
Proposition 2 has some important implications:
Proposition 3.
For any choice of τ > 0 and τ θ > 0 , ﬁxed points θ ∗ of ProxProp are stationary points of the original energy J ( θ ; X, y ) .
Moreover, we can conclude convergence in the following sense.
Proposition 4.
The ProxProp direction θ k +1 − θ k is a descent direction.
Moreover, under the (weak) assumption that the activations a kl remain bounded, the angle α k between −∇ θ J ( θ k ; X, y ) and θ k +1 − θ k remains uniformly bounded away from π/ 2 , i.e.
[Equation]
for some constant c .
Proposition 4 immediately implies convergence of a whole class of algorithms that depend only on a provided descent direction.
We refer the reader to (Nocedal & Wright, 2006, Chapter 3.2) for examples and more details.
Furthermore, Proposition 4 states convergence for any minimization scheme in step c (cid:13) of Algorithm 2 that induces a descent direction in parameter space and thus provides the theoretical basis for a wide range of neural network optimization algorithms.
Considering the advantages of proximal steps over gradient steps, it is tempting to also update the auxiliary variables a and z in an implicit fashion.
This corresponds to a proximal step in b (cid:13) of
Algorithm 2.
However, one cannot expect an analogue version of Proposition 3 to hold anymore.
For example, if the update of a L − 2 in (8) is replaced by a proximal step, the propagated error does not correspond to the gradient of the loss function L y , but to the gradient of its Moreau envelope.
Consequently, one would then minimize a different energy.
While in principle this could result in an optimization algorithm with, for example, favorable generalization properties, we focus on minimizing the original network energy in this work and therefore do not further pursue the idea of implicit steps on a and z .
4.2. I NEXACT SOLUTION OF PROXIMAL STEPS
As we can see in Proposition 2, the ProxProp updates differ from vanilla gradient descent by the variable metric induced by the matrices ( M kl ) − 1 with M kl deﬁned in (14).
Computing the ProxProp update direction v kl := 1 τ ( θ k +1 l − θ kl ) therefore reduces to solving the linear equation
[Equation]
(16) which requires an efﬁcient implementation.
We propose to use a conjugate gradient (CG) method, not only because it is one of the most efﬁcient methods for iteratively solving linear systems in general, but also because it can be implemented matrix-free : It merely requires the application of the linear operator M kl which consists of the identity and an application of ( ∇ φ ( · , a kl − 1 ))
( ∇ φ ( · , a kl − 1 )) ∗ .
The latter, however, is preimplemented for many linear transfer functions φ in common deep learning frameworks, because φ ( x, a kl − 1 ) =
( ∇ φ ( · , a kl − 1 )) ∗ ( x ) is nothing but a forward-pass in φ , and φ ∗ ( z, a kl − 1 )
=
( ∇ φ ( · , a kl − 1 ))
( z ) provides the gradient with respect to the parameters θ if z is the backpropagated gradient up to that layer.
Therefore, a CG solver is straight-forward to implement in any deep learning framework using the existing, highly efﬁcient and high level implementations of φ and φ ∗ .
For a fully connected network φ is a matrix multiplication and for a convolutional network the convolution operation.
As we will analyze in more detail in Section 5.1, we approximate the solution to (16) with a few CG iterations, as the advantage of highly precise solutions does not justify the additional computational effort in obtaining them.
Using any number of iterations provably does not harm the convergence properties of ProxProp:
Proposition 5.
The direction ˜ v kl one obtains from approximating the solution v kl of (16) with the
CG method remains a descent direction for any number of iterations.
4.2. C ONVERGENCE IN THE STOCHASTIC SETTING
While the above analysis considers only the full batch setting, we remark that convergence of ProxProp can also be guaranteed in the stochastic setting under mild assumptions.
Assuming that the activations a kl remain bounded (as in Proposition 4), the eigenvalues of ( M kl ) − 1 are uniformly contained in the interval [ λ, τ θ ] for some ﬁxed λ > 0 .
Therefore, our ProxProp updates fulﬁll Assumption 4.3 in (Bottou et al., 2016), presuming the classic stochastic gradient fulﬁlls them.
This guarantees convergence of stochastic ProxProp updates in the sense of (Bottou et al., 2016, Theorem 4.9), i.e. for a suitable sequence of diminishing step sizes.
N UMERICAL EVALUATION
ProxProp generally ﬁts well with the API provided by modern deep learning frameworks, since it can be implemented as a network layer with a custom backward pass for the proximal mapping.
We chose PyTorch for our implementation 1 .
In particular, our implementation can use the API’s GPU compute capabilities; all numerical experiments reported below were conducted on an NVIDIA
Titan X GPU.
To directly compare the algorithms, we used our own layer for either proximal or gradient update steps (cf.
step c (cid:13) in Algorithms 1 and 2).
A ProxProp layer can be seamlessly integrated in a larger network architecture, also with other parametrized layers such as BatchNormalization.
5. E XACT AND APPROXIMATE SOLUTIONS TO PROXIMAL STEPS
We study the behavior of ProxProp in comparison to classical BackProp for a supervised visual learning problem on the CIFAR-10 dataset.
We train a fully connected network with architecture 3072 − 4000 − 1000 − 4000 − 10 and ReLU nonlinearities.
As the loss function, we chose the crossentropy between the probability distribution obtained by a softmax nonlinearity and the ground-truth labels.
We used a subset of 45000 images for training while keeping 5000 images as a validation set.
We initialized the parameters θ l uniformly in [ − 1 / √ n l − 1 , 1 / √ n l − 1 ] , the default initialization of PyTorch.
[Figure] Figure 2 shows the decay of the full batch training loss over epochs (left) and training time (middle) for a Nesterov momentum 2 based optimizer using a momentum of µ = 0 .
95 and minibatches of size 500.
We used τ θ = 0 .
for the ProxProp variants along with τ = 1 .
For BackProp we chose τ = 0 .
as the optimal value we found in a grid search.
As we can see in Figure 2, using implicit steps indeed improves the optimization progress per epoch.
Thanks to powerful linear algebra methods on the GPU, the exact ProxProp solution is competitive with BackProp even in terms of runtime.
The advantage of the CG-based approximations, however, is that they generalize to arbitrary linear transfer functions in a matrix-free manner, i.e. they are independent of whether the matrices M kl can be formed efﬁciently.
Moreover, the validation accuracies (right plot in Figure 2) suggest that these approximations have generalization advantages in comparison to BackProp as well as the exact
ProxProp method.
Finally, we found the exact solution to be signiﬁcantly more sensitive to changes of τ θ than its CG-based approximations.
We therefore focus on the CG-based variants of ProxProp in the following.
In particular, we can eliminate the hyperparameter τ θ and consistently chose τ θ =for the rest of this paper, while one can in principle perform a hyperparameter search just as for the learning rate τ .
Consequently, there are no additional parameters compared with BackProp.
5. S TABILITY FOR LARGER STEP SIZES
We compare the behavior of ProxProp and BackProp for different step sizes.
Table 1 shows the ﬁnal full batch training loss after 50 epochs with various τ .
The ProxProp based approaches remain stable over a signiﬁcantly larger range of τ .
Even more importantly, deviating from the optimal step size τ by one order of magnitude resulted in a divergent algorithm for classical BackProp, but still provides reasonable training results for ProxProp with 3 or 5 CG iterations.
These results are in accordance with our motivation developed in Section 4.1.
From a practical point of view, this eases hyperparameter search over τ .
[-τ    0 .0 .0 .
 · 10 −5 · 10 −-][Table]
BackProp – – – –
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Figure] [Figure]
5. P ROX P ROP AS A FIRST - ORDER ORACLE
We show that ProxProp can be used as a gradient oracle for ﬁrst-order optimization algorithms.
In this section, we consider Adam (Kingma & Ba, 2015).
Furthermore, to demonstrate our algorithm on a generic architecture with layers commonly used in practice, we trained on a convolutional neural network of the form:
Conv[16 × 32 × 32] → ReLU → Pool[16 × 16 × 16]
→ Conv[20 × 16 × 16] → ReLU → Pool[20 × 8 × 8] → Conv[20 × 8 × 8] → ReLU → Pool[20 × 4 × 4] → FC + Softmax[10 × 1 × 1]
Here, the ﬁrst dimension denotes the respective number of ﬁlters with kernel size 5 × 5 and max pooling downsamples its input by a factor of two.
We set the step size τ = 10 − 3 for both BackProp and ProxProp.
The results are shown in Fig.
Using parameter update directions induced by ProxProp within
Adam leads to a signiﬁcantly faster decrease of the full batch training loss in epochs.
While the running time is higher than the highly optimized backpropagation method, we expect that it can be improved through further engineering efforts.
We deduce from Fig.
3 that the best validation accuracy ( 72 . 9% ) of the proposed method is higher than the one obtained with classical backpropagation ( 71 . 7% ).
Such a positive effect of proximal smoothing on the generalization capabilities of deep networks is consistent with the observations of Chaudhari et al.
(2017b).
Finally, the accuracies on the test set after 50 epochs are 70 .
7% for ProxProp and 69 .
6% for BackProp which suggests that the proposed algorithm can lead to better generalization.
C ONCLUSION
We have proposed proximal backpropagation (ProxProp) as an effective method for training neural networks.
To this end, we ﬁrst showed the equivalence of the classical backpropagation algorithm with an algorithm that alternates between sequential gradient steps on a quadratic penalty function and forward passes through the network.
Subsequently, we developed a generalization of BackProp, which replaces explicit gradient steps with implicit (proximal) steps, and proved that such a scheme yields a descent direction, even if the implicit steps are approximated by conjugate gradient iterations.
Our numerical analysis demonstrates that ProxProp is stable across various choices of step sizes and shows promising results when compared with common stochastic gradient descent optimizers.
We believe that the interpretation of error backpropagation as the alternation between forward passes and sequential minimization steps on a penalty functional provides a theoretical basis for the development of further learning algorithms.
A PPENDIX A T HEORETICAL RESULTS
Proof of Proposition 1.
We ﬁrst take a gradient step on
[Equation]
with respect to ( θ L − 1 , a L − 2 ) .
The gradient step with respect to θ L − 1 is the same as in the gradient descent update,
[Equation]
[Equation]
since J depends on θ L − 1 only via L y ◦ φ .
The gradient descent step on a L − 2 in a (cid:13) yields
[Equation]
where we use a k + 1 / 2 L − 2 to denote the updated variable a L − 2 before the forward pass of the next iteration.
To keep the presentation as clear as possible we slightly abused the notation of a right multiplication with ∇ a φ ( θ kL − 1 , a kL − 2 ) : While this notation is exact in the case of fully connected layers, it represents the application of the corresponding linear operator in the more general case, e.g. for convolutions.
For all layers l ≤ L − 2 note that due to the forward pass in Algorithm 1 we have
[Equation]
and we therefore get the following update equations in the gradient step b (cid:13)
[Equation]
and in the gradient step c (cid:13)
w.r
[Equation]
t. a l
[Equation] [Equation]
Equations (21) and (22) can be combined to obtain:
[Equation]
The above formula allows us to backtrack the differences of the old z k l and the updated z k + 1 / 2 l up to layer L − 2 , where we can use equations (21) and (19) to relate the difference to the loss.
Altogether, we obtain
[Equation]
By inserting (24) into the gradient descent update equation with respect to θ l in c (cid:13) ,
[Equation]
we obtain the chain rule for update (18)
[Equation]
Proof of Proposition 2.
Since only the updates for θ l , l = 1 , . . .
, L − 2 , are performed implicitly, one can replicate the proof of Proposition 1 exactly up to equation (24).
Let us denote the right hand side of (24) by g kl , i.e. z k + 1 / 2 l
= z kl − g kl and note that
[Equation]
holds by the chain rule (as seen in (25)).
We have eliminated the dependence of ∇ θ φ ( θ kl , a kl − 1 ) on θ kl and wrote ∇ θ φ ( · , a kl − 1 ) instead, because we assume φ to be linear in θ such that ∇ θ φ does not depend on the point θ where the gradient is evaluated anymore.
We now rewrite the ProxProp update equation of the parameters θ as follows
[Equation]
where we have used that φ is linear in θ .
The optimality condition yields
[Equation]
Again, due to the linearity of φ in θ
[Equation]
one has
[Equation]
where ∗ , denotes the adjoint of a linear operator.
We conclude
[Equation]
which yields the assertion.
Proof of Proposition 3.
Under the assumption that θ k converges, θ k → ˆ θ , one ﬁnds that a kl → ˆ a l and z kl → ˆ z l = φ (ˆ θ l , ˆ a l − 1 ) converge to the respective activations of the parameters ˆ θ due to the forward pass and the continuity of the network.
As we assume J ( · ; X, y ) to be continuously differentiable, we deduce from (30) that lim k →∞ ∇ θ l J ( θ k ; X, y )
=
for all l = 1 , . . .
, L − 2 .
The parameters of the last layer θ L − 1 are treated explicitly anyways, such that the above equation also holds for l =
L − 1 , which then yields the assertion.
Proof of Proposition 4.
As the matrices
[Equation]
(31) (with the convention M kL − 1 = I ) are positive deﬁnite, so are their inverses, and the claim that θ k +1 − θ k is a descent direction is immediate,
[Equation]
We still have to guarantee that this update direction does not become orthogonal to the gradient in the limit k → ∞ .
The largest eigenvalue of ( M kl ) − 1 is bounded from above by τ θ .
If the a kl −remain bounded, then so does ∇ φ ( · , a kl − 1 ) and the largest eigenvalue of ∇ φ ( · , a kl − 1 ) ∇ φ ( · , a kl − 1 ) ∗ must be bounded by some constant ˜ c .
Therefore, the smallest eigenvalue of ( M kl ) − 1 must remain bounded from from below by ( 1 τ θ + ˜ c ) − 1 .
Abbreviating v = ∇ θ l J ( θ k ; Y, x ) , it follows that
[Equation]
which yields the assertion.
Proof of Proposition 5.
According to (Nocedal & Wright, 2006, p. 109, Thm. 5.3) and (Nocedal &
Wright, 2006, p. 106, Thm. 5.2) the k -th iteration x k of the CG method for solving a linear system
Ax = b with starting point x 0 = 0 meets
[Equation]
i.e. is optimizing over an order- k Krylov subspace.
The starting point x 0 = 0 can be chosen without loss of generality.
Suppose the starting point is ˜ x 0 (cid:54) = 0 , then one can optimize the variable x = ˜ x − ˜ x 0 with a starting point x 0 = 0 and b =
˜ b + A ˜ x 0 .
We will assume that the CG iteration has not converged yet as the claim for a fully converged CG iteration immediately follow from Proposition 4.
Writing the vectors b, Ab, . .
A k − 1 b as columns of a matrix K k , the condition x ∈ span ( b, Ab, . . .
A k − 1 b ) can equivalently be expressed as x =
K k α for some α ∈ R k .
In terms of α our minimization problem becomes
[Equation]
leading to the optimality condition
[Equation]
Note that A is symmetric positive deﬁnite and can therefore be written as √ A T √ A , leading to
[Equation]
[Equation]
being symmetric positive deﬁnite.
Hence, the matrix (( K k ) T
A K k ) − 1 is positive deﬁnite, too, and
[Equation]
Note that ( K k ) T b is nonzero if b is nonzero, as (cid:107) b (cid:107) 2 is its ﬁrst entry.
To translate the general analysis of the CG iteration to our speciﬁc case, using any number of CG iterations we ﬁnd that an approximate solution ˜ v kl of
[Equation]
leads to
[Equation]
i.e., to ˜ v kl being a descent direction.
B P ROXIMAL OPERATOR FOR LINEAR TRANSFER FUNCTIONS
In order to update the parameters θ l of the linear transfer function, we have to solve the problem (11),
[Equation]
Since we assume that φ is linear in θ for a ﬁxed a k , there exists a matrix A k such that
[Equation]
(41) and the optimality condition yields
[Equation]
1 ( vec ( θ k ) +
( A k ) T vec ( z k + 1 / 2 )) .
In the main paper we sometimes use the more abstract but also more concise notion of ∇ φ ( · , a k ) , which represents the linear operator
[Equation]
To also make the above more speciﬁc, consider the example of φ ( θ, a k )
=
θa k .
In this case the variable θ may remain in a matrix form and the solution of the proximal mapping becomes
[Equation]
Since a k ∈ R n × N for some layer size n and batch size N , the size of the linear system is independent of the batch size.