T RAINING A UTOENCODERS BY A LTERNATING M INIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We present DANTE, a novel method for training neural networks, in particular autoencoders, using the alternating minimization principle.
DANTE provides a distinct perspective in lieu of traditional gradient-based backpropagation techniques commonly used to train deep networks.
It utilizes an adaptation of quasi-convex optimization techniques to cast autoencoder training as a bi-quasi-convex optimization problem.
We show that for autoencoder conﬁgurations with both differentiable (e g. sigmoid) and non-differentiable (e g. ReLU) activation functions, we can perform the alternations very effectively.
DANTE effortlessly extends to networks with multiple hidden layers and varying network conﬁgurations. In experiments on standard datasets, autoencoders trained using the proposed method were found to be very promising when compared to those trained using traditional backpropagation techniques, both in terms of training speed, as well as feature extraction and reconstruction performance
I NTRODUCTION
For much of the recent march of deep learning, gradient-based backpropagation methods, e.g.
Stochastic Gradient Descent (SGD) and its variants, have been the mainstay of practitioners.
The use of these methods, especially on vast amounts of data, has led to unprecedented progress in several areas of artiﬁcial intelligence.
On one hand, the intense focus on these techniques has led to an intimate understanding of hardware requirements and code optimizations needed to execute these routines on large datasets in a scalable manner.
Today, myriad off-the-shelf and highly optimized packages exist that can churn reasonably large datasets on GPU architectures with relatively mild human involvement and little bootstrap effort.
However, this surge of success of backpropagation-based methods in recent years has somewhat overshadowed the need to continue to look for options beyond backprogagation to train deep networks.
Despite several advancements in deep learning with respect to novel architectures such as encoderdecoder networks and generative adversarial models, the reliance on backpropagation methods remains.
While reinforcement learning methods are becoming increasingly popular, their scope is limited to a particular family of settings such as agent-based systems or reward-based learning.
Recent efforts have studied the limitations of SGD-based backpropagation, including parallelization of SGDbased techniques that are inherently serial Taylor et al.
(2016), vanishing gradients, especially for certain activation functions Hochreiter & Schmidhuber (1997), convergence of stochastic techniques to local optima Anandkumar & Ge (2016), and many more.
For a well-referenced critique of gradient-based methods, we point the reader to Taylor et al.
From another perspective, there has been marked progress in recent years in the area of non-convex optimization (beyond deep learning), which has resulted in scalable methods such as iterated hard thresholding Blumensath & Davies (2009) and alternating minimization Jain et al.
as methods of choice for solving large-scale sparse recovery, matrix completion, and tensor factorization tasks.
Several of these methods not only scale well to large problems, but also offer provably accurate solutions.
In this work, we investigate a non-backpropagation strategy to train neural networks, leveraging recent advances in quasi-convex optimization.
Our method is called DANTE (Deep AlterNations for Training autoEncoders), and it offers an alternating minimization-based technique for training neural networks - in particular, autoencoders.
DANTE is based on a simple but useful observation that the problem of training a single hidden-layer autoencoder can be cast as a bi-quasiconvex optimization problem (described in Section 3.1).
This observation allows us to use an alternating optimization strategy to train the autoencoder, where each step involves relatively simple quasi-convex problems.
DANTE then uses efﬁcient solvers for quasi-convex problems including normalized gradient descent Nesterov (1984) and stochastic normalized gradient descent Hazan et al.
(2015) to train autoencoder networks.
The key contributions of this work are summarized below:
• We show that viewing each layer of an a neural network as applying an ensemble of generalized linear transformations, allows the problem of training the network to be cast as a bi-quasi-convex optimization problem (exact statement later). • We exploit this intuition by employing an alternating minimization strategy DANTE that reduces the problem of training the layers to quasi-convex optimization problems.
• We utilize state-of-the-art Stochastic Normalized Gradient Descent (SNGD) technique Hazan et al. (2015) for quasi-convex optimization to provide an efﬁcient implementation of DANTE for networks with sigmoidal activation functions. However, a limitation of SNGD is its inability to handle link non-differentiable functions such as the ReLU. • To overcome this limitation, we introduce the generalized ReLU , a variant of the popular ReLU activation function and show how the SNGD technique can be applied with the generalized ReLU function.
This presents an augmentation in the state of the art in quasiconvex optimization and may be of independent interest.
This allows DANTE to train AEs with both differentiable and non-differentiable activation functions, including ReLUs and sigmoid.
• We show that the SNGD method offers provably more rapid convergence with the generalized ReLU function than it does even for the sigmoidal activation. This is corroborated in experiments as well.
A key advantage of our approach is the ability to exploit these theoretical results to set learning rates and batch sizes without any ﬁne tuning/cross-validation required.
• We also show DANTE can be easily extended to train deep AEs with multiple hidden layers. • We empirically validate DANTE with both the generalized ReLU and sigmoid activations and establish that DANTE does provide comparable or better test errors, reconstructions and classiﬁcation performance (with the learned representations), when compared to an identical network train using standard used mini-batch SGD-based backpropagation.
R ELATED W ORK
Backpropagation-based techniques date back to the early days of neural network research Rumelhart et al.
; Chauvin & Rumelhart (1995) but remain to this day, the most commonly used methods for training a variety of neural networks including multi-layer perceptrons, convolutional neural networks, autoencoders, recurrent networks and the like.
Recent years have seen the development of other methods, predominantly based on least-squares approaches, used to train neural networks.
Carreira-Perpinan and Wang Carreira-Perpinan & Wang (2014) proposed a least-squares based method to train a neural network.
In particular, they introduced the Method of Auxiliary Constraints (MAC), and used quadratic penalties to enforce equality constraints.
Patel et al. Patel et al.
(2015) proposed an Expectation-Maximization (EM) approach derived from a hierarchical generative model called the Deep Rendering Model (DRM), and also used least-squared parameter updates in each of the EM steps.
They showed that forward propagation in a convolutional neural network was equivalent to the inference on their DRM.
Unfortunately, neither of these methods has publicly available implementations or even published training results to compare against.
More recently, Taylor et al. proposed a method to train neural networks using the Alternating
Direction Method of Multipliers (ADMM) and Bregman iterations Taylor et al.
The focus of this method, however, was on scaling the training of neural networks to a distributed setting on multiple cores across a computing cluster.
Jaderberg also proposed the idea of ’synthetic gradients’ in
Jaderberg et al.
While this approach is interesting, this work is more focused towards a more efﬁcient way to carry out gradient-based parameter updates in a neural network.
However, in our work, we focus on an entirely new approach to training neural networks – in particular, autoencoders – using alternating optimization, quasi-convexity and SNGD, and show that this approach shows promising results on the a range of datasets.
Although alternating minimization has found much appeal in areas such as matrix factorization Jain et al.
(2013), to the best of our knowledge, this is the ﬁrst such effort in using alternating principles to train neural networks with related performance guarantees.
DANTE: D EEP A LTER N ATIONS FOR T RAINING AUTO E NCODERS
In this section, we will ﬁrst set notation and establish the problem setting, then present details of the
DANTE method, including the SNGD algorithm.
For sake of simplicity, we consider networks with just a single hidden layer.
We then offer some theoretical insight intro DANTE’s inner workings, which also allow us to arrive at the generalized ReLU activation function, and ﬁnally describe
how DANTE can be extended to deep networks with multiple hidden layers. 3. P ROBLEM F ORMULATION Consider a neural network with L layers.
Each layer l ∈ { 1 , 2 , . . .
, L } has n l nodes and is characterized by a linear operator W l ∈ R n l − 1 × n l and a non-linear activation function φ l : R n l → R n l .
The activations generated by the layer l are denoted by a l ∈ R n l .
We denote by a 0 , the input activations and n 0 to be the number of input activations i.e. a 0 ∈ R n 0 .
Each layer uses activations being fed into it to compute its own activations as a l = φ l (cid:104) W l , a l − 1 (cid:105) ∈ R n l , where φ (cid:104) .
(cid:105) denotes φ ( (cid:104) .
(cid:105) ) for simplicity of notation.
A multi-layer neural network is formed by nesting such layers to form a composite function f given as follows:
[Equation]
(1) where W = { W l } is the collection of all the weights through the network, and x = a 0 contains the input activations for each training sample.
Given m data samples { ( x i , y i ) } mi =1 from some distribution D , the network is trained by tuning the weights W to minimize a given loss function, J :
[Equation]
Note that a multi-layer autoencoder is trained similarly, but with the loss function modiﬁed as below:
[Equation]
For purpose of simplicity and convenience, we ﬁrst consider the case of a single-layer autoencoder, represented as f ( W ; x ) =
φ 2 (cid:104) W 2 , φ 1 (cid:104) W 1 , x (cid:105)(cid:105) to describe our methodology.
We describe in a later section on how this idea can be extended to deep multi-layer autoencoders.
(Note that our deﬁnition of a single-layer autoencoder is equivalent to a two-layer neural network in a classiﬁcation setting, by nature of the autoencoder.)
A common loss function used to train autoencoders is the squared loss function which, in our simpliﬁed setting, yields the following objective.
[Equation]
Now denote
[Equation]
[Equation] An important observation here is that if we ﬁx W 1 , then Eqn (5) turns into a Generalized Linear
Model problem with φ 2 as the activation function, i.e.
[Equation]
where z = φ 1 (cid:104) W 1 , x (cid:105) .
Similarly, ﬁxing W 2 turns the problem into yet another Generalized Linear
Model problem, this time with W 1 as the parameter (note that φ W 2 (cid:104)·(cid:105) =
φ 2 (cid:104) W 2 , φ 1 (cid:104)·(cid:105)(cid:105) ).
[Equation]
Input : Number of iterations T , training data S =
{ ( x i , y i ) } mi =1 ∈ R d × R , learning rate η , minibatch size b , Initialization parameters w1 for t = 1 to T do
Sample { ( x i , y i ) } bi =1 ∼ Unif ( S ) //Select
a random mini-batch of training points
Let f t ( w ) = 1 b (cid:80) bi =
( y i − φ (cid:104) w , x i (cid:105) )
Let g t = ∇ f t ( w t ) , and ˆ g ( t )
=
g t (cid:107) g t (cid:107) w t
+
=
w t − η · ˆ g t 6 endOutput : Model given by w T
Thus, a single-layer autoencoder is a combination of two Generalized Linear Model (GLM) problems, and we exploit this key observation in this work.
In particular, we leverage a recent result by Hazan et al. (2015) that shows that GLMs with nice, differentiable link functions such as sigmoid (or even a combination of sigmoids such as φ W 2 ( · ) ), satisfy a property the authors name Strict Locally
Quasi-Convexity (SLQC), which allows techniques such as SNGD to solve the GLM problems effectively.
This is quite advantageous for us since it allows us to solve each sub-problem of the alternating setup efﬁciently. In a subsequent section, we will show that GLMs with non-differentiable activation – in particular, a generalized Rectiﬁed Linear Unit (ReLU) – can also satisfy the SLQC property, thus allowing us to extend the proposed alternating strategy, DANTE, to ReLU-based autoencoders too.
We note that while we have developed this idea to train autoencoders in this work (since our approach relates closely to the greedy layer-wise training in autoencoders), DANTE can be used to train standard multi-layer neural networks too (discussed in Section 5).
M ETHODOLOGY
We begin our presentation of the proposed method by brieﬂy reviewing the Stochastic Normalized
Gradient Descent (SNGD) method, which is used to execute the inner steps of DANTE.
We explain in the next subsection, the rationale behind the choice of SNGD as the optimizer.
We stress that although
DANTE does use stochastic gradient-style methods internally (such as the SNGD algorithm), the overall strategy adopted by DANTE is not a descent-based strategy, rather an alternating-minimization strategy.
Stochastic Normalized Gradient Descent (SNGD):
Normalized Gradient Descent (NGD) is an adaptation of traditional Gradient Descent where the updates in each iteration are purely based on the direction of the gradients, while ignoring their magnitudes. This is achieved by normalizing the gradients.
SNGD is the stochastic version of NGD, where weight updates are performed using individual (randomly chosen) training samples, instead of the complete set of samples.
Mini-batch SNGD generalizes this by applying updates to the parameters at the end of every mini-batch of samples, as does mini-batch Stochastic Gradient Descent (SGD).
In the remainder of this paper, we refer to mini-batch SNGD as SNGD itself, as is common for SGD.
Algorithm 1 describes the SNGD methodology for a generic GLM problem.
DANTE:
Given this background, Algorithm 2 outlines the proposed method, DANTE.
Consider the autoencoder problem below for a single hidden layer network:
[Equation]
Upon ﬁxing the parameters of the lower layer i.e. W 1 , it is easy to see that we are left with a GLM problem:
[Equation]
where z = φ 1 (cid:104) W 1 , x (cid:105) .
DANTE solves this intermediate problem using SNGD steps by sampling several mini-batches of data points and performing updates as dictated by Algorithm 1. Similarly, [Table] [Table]
2 while | f ( W t 1 , W t 2 ) − f
( W t − 1, W t −
[Table] or t < T AM do [Table] ﬁxing the parameters of the upper layer, i.e. W 2 , we are left with another GLM problem:
[Equation]
where φ W 2 (cid:104)·(cid:105) = φ 2 (cid:104) W 2 , φ 1 (cid:104)·(cid:105)(cid:105) .
This is once again solved by mini-batch SNGD, as before.
3. R ATIONALE
To describe the motivation for our alternating strategy in DANTE, we ﬁrst deﬁne key terms and results that are essential to our work.
We present the notion of a locally quasi-convex function (as introduced in Hazan et al.
and show that under certain realizability conditions, empirical objective functions induced by Generalized Linear Models (GLMs) are locally quasi-convex.
To this end, we introduce a new activation function, the generalized ReLU, and show that the GLM with the generalized ReLU also satisﬁes this property.
We then cite a result that shows that SNGD converges to the optimum solution provably for locally quasi-convex functions, and extend this result to the newly introduced activation function.
Deﬁnition 3.1 ( Local-Quasi-Convexity ) .
Let x , z ∈ R d , κ, (cid:15) > 0 and let f : R d → R be a differentiable function.
Then f is said to be ( (cid:15), κ, z ) -Strictly-Locally-Quasi-Convex
(SLQC) in x , if at least one of the following applies:
f ( x ) − f ( z ) ≤ (cid:15) 2.
(cid:107)∇ f ( x ) (cid:107) > 0 , and ∀ y ∈ B ( z , (cid:15)/κ ) , (cid:104)∇ f ( x ) , y − x (cid:105) ≤
where B ( z , (cid:15)/κ ) refers to a ball centered at z with radius (cid:15)/κ .
Deﬁnition 3.2 ( Idealized and Noisy Generalized Linear Model (GLM) ) .
Given an (unknown) distribution D and an activation function φ : R → R , an idealized GLM is deﬁned by the existence of a w ∗ ∈ R d such that y i =
φ (cid:104) w ∗ , x i (cid:105)∀ i ∈ [ m ] where w ∗ is the global minimizer of the error function:
[Equation]
Similarly, a noisy GLM is deﬁned by the existence of a w ∗ ∈ R d , which is the global minimizer of the error function: E ( x ,y )
∼D ( y | x ) =
φ ( (cid:104) w ∗ , x (cid:105) ) .
the expected error is then:
[Equation]
(Hazan et al., 2015, Lemma 3.2) show that if we draw m ≥ Ω (cid:16) exp(2 (cid:107) w ∗ (cid:107)
[Equation]
(cid:15)log 1 δ (cid:17) samples { ( x i , y i ) } mi =1 from a GLM with the sigmoid activation function, then with probability at least 1 − δ , the empirical error function
[Equation]
is ( (cid:15), e (cid:107) w ∗ (cid:107) 2 , w ∗ ) -SLQC in w .
However, this is a bit restrictive since proof of this result critically uses properties of the sigmoid that are not satisﬁed by other popular activation functions such as the
ReLU. In order to go beyond the sigmoid activation function, we introduce a new generalized ReLU activation function.
Deﬁnition 3.3.
(Generalized ReLU)
The generalized ReLU function f : R → R , 0 < a < b , a, b ∈ R is deﬁned as:
[Equation]
This function is differentiable at every point except 0.
Note that this deﬁnition also subsumes variants of ReLU such as the leaky ReLU
Xu et al.
We deﬁne the function g that gives a valid subgradient at all x for the generalized ReLU to be
[Equation]
While SLQC is primarily deﬁned for differentiable functions, we now show that with the above deﬁnition of the subgradient, the GLM with the generalized ReLU is also SLQC.
Theorem 3.4.
In the idealized GLM with generalized ReLU activation, assuming || w ∗ || ≤ W , ˆ err ( w ) is (cid:16) (cid:15), 2 b 2 Wa , w ∗ (cid:17) − SLQC in w for all w ∈ B d (0 , W ) .
Proof.
Consider || w || ≤ W such that
ˆ err m ( w) = 1 m (cid:80) mi =1 ( y i − φ (cid:104) w , x i (cid:105) )
2 ≥ (cid:15) , where m is the total number of samples.
Also let v be a point (cid:15)/κ -close to minima w ∗ with κ = 2 b 2 Wa .
Let g be the subgradient of the generalized ReLU activation and G be the subgradient of ˆ err m ( w) .
(Note that as before, g (cid:104) .
(cid:105) denotes g ( (cid:104) .
(cid:105) ) ).
Then: (cid:104) G ( w ) , w − v (cid:105) =m m (cid:88) i = g (cid:104) w , x i (cid:105) (
[Equation]
(cid:104)
[Equation]
x i (cid:105)
[Equation]
y i ) (cid:104) x i , ( w
[Equation]
) (cid:105) =m m [Equation] i = g
[Equation]
(cid:105) [Equation] (cid:104)
[Equation]
(cid:105)
[Equation]
[ (cid:104) x
[Equation]
, w
[Equation]
(cid:105) [Equation] (cid:104) x
[Equation]
, w
[Equation]
(cid:105) ] (Step 1) ≥m m (cid:88) i = g (cid:104) w
[Equation]
x
[Equation]
(cid:105) (cid:104)
[Equation]
[Equation] (cid:104) w
[Equation]
x
[Equation]
(cid:105)
[Equation]
(cid:104)
[Equation]
(cid:104)
[Equation]
x
[Equation]
(cid:105) ) (cid:104) x
[Equation]
w
[Equation]
(cid:105) (cid:105) (Step 2) ≥m m (cid:88) i = g (cid:104) w , x i (cid:105) ( b
[Equation]
1 ( φ (cid:104) w , x i (cid:105) − φ (cid:104) w ∗ , x i (cid:105) ) 2 − | φ (cid:104) w , x i (cid:105) − φ (cid:104) w ∗ , x i (cid:105)||| x i |||| w ∗ − v || ≥m m
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
In the above proof
[Table]
we ﬁrst use the fact (in Step 1) that in the GLM [Table] there is some w ∗ such that φ (cid:104) w ∗ , x i (cid:105) =
y i .
Then, we use the fact (in Steps 2 and 4) that the generalized ReLU function is b -Lipschitz [Table] and the fact that the minimum value of the quasigradient of g is a (Step 3).
Subsequently, in Step 5, we simply use the given bounds on the variables x i , w , w ∗ due to the setup of the problem ( w ∈ B d (0 , W ) , and x i ∈ B d , the unit d -dimensional ball, as in Defn 2.6).
We also prove a similar result for the Noisy GLM below.
Theorem 3.5.
In the noisy GLM with generalized ReLU activation, assuming || w ∗ || ≤ W , given w ∈ B (0 , W ) , then with probability ≥ 1 − δ after m ≥ 4 Wb 2 a(cid:15) 2 log (1 /δ ) /(cid:15)
2 samples, ˆ err ( w ) is (cid:16) (cid:15), 2 b 2 Wa , w ∗ (cid:17) − SLQC in w .
The proof for Theorem 3.5 is included in Appendix A (Supplementary Section).
Now, we review a result from Hazan et al.
(2015) which shows that SNGD provably converges to the optimum for SLQC functions, and hence, with very high probability, for empirical objective functions induced by noisy GLM instances too.
Theorem 3.6 (Hazan et al.
Let (cid:15), δ, G, M, κ > 0 , let f : R d → R and w ∗ =
arg min w f ( w ) .
Assume that for b ≥ b 0 ( (cid:15), δ, T ) , with probability ≥ 1 − δ , f t deﬁned in Algorithm 1 is ( (cid:15), κ, w ∗ ) -SLQC ∀ w , and | f t | ≤ M ∀ t ∈ { 1 , · · · , T } .
If we run SNGD with T ≥ κ 2 || w 1 − w ∗ || 2 (cid:15)and η = (cid:15)κ , and b ≥ max (cid:26) M 2 log ( 4 T δ ) 2 (cid:15), b 0 ( (cid:15), δ, T ) (cid:27) , with probability 1 − 2 δ , f ( w ) − f ( w ∗ ) ≤ 3 (cid:15) .
Note that by performing each DANTE alternation with a freshly sampled batch of data points, we can easily establish SLQC properties of the problem, and as a consequence, harness the convergence guarantees provided by Theorem 3.6.
Thus, given a single-layer autoencoder with either a sigmoid activation function or a generalized ReLU activation function, DANTE uses SNGD to solve a SLQC problem in each alternating step, which converges to the optima solution in each step with very high probability, as shown above in Theorem 3.6.
Note that this allows us to cover a large family of activation functions, since the tanh activation is simply a rescaled sigmoid and the generalized ReLU includes many variants of ReLUs
More importantly, note that the convergence rate of SNGD depends crucially on the κ parameter.
Whereas the GLM error function with sigmoid activation has κ = e W Hazan et al.
, we obtain κ = 2 b 2 Wa (i.e. linear in W ) for the generalized ReLU setting, an exponential improvement!
This is signiﬁcant as in Theorem 3.6, the number of iterations T depends on κ 2 .
This shows that SNGD offers accelerated convergence with generalized ReLU GLMs than sigmoid GLMs.
3. E XTENDING TO A M ULTI -L AYER A UTOENCODER
In the previous sections, we illustrated how a single hidden-layer autoencoder can be cast as a set of
SLQC problems and proposed an alternating minimization method, DANTE.
This approach can be generalized to deep autoencoders by considering the greedy layer-wise approach to training a neural network (Bengio et al.
In this approach, each pair of layers of a deep stacked autoencoder is successively trained in order to obtain the ﬁnal representation.
Each pair of layers considered in this paradigm is a single hidden-layer autoencoder, which can be cast as a pair of SLQC problems that can be trained using DANTE.
Therefore, training a deep autoencoder using greedy layer-wise approach can be modeled as a series of SLQC problem pairs.
Algorithm 3 summarizes the proposed approach to use DANTE for a deep autoencoder, and Figure 1 illustrates the approach.
Note that it may be possible to use other schemes to use DANTE for multi-layer autoencoders such as a round-robin scheme, where each layer is trained separately one after the other in the sequence in which the layers appear in the network.
E XPERIMENTS AND R ESULTS
We validated DANTE by training autoencoders on an expanded 32 × 32 variant of the standard MNIST dataset LeCun et al.
(1998) as well as other datasets from the UCI repository.
We also conducted experiments with multi-layer autoencoders, as well as studies with varying number of hidden neurons on single-layer autoencoders.
Our experiments on MNIST used the standard benchmarking setup of
[Figure]
Input : Encoder e with weights U , Decoder d with weights V , Number of hidden layers 2 n − 1 , Learning rates η , Stopping threshold (cid:15) , Number of iterations of alternating minimization T AM , initial values U 0 , V 0 , minibatch size b 1 t :=2 for l = 1 to n do while | f ( U t , V t ) − f ( U t − 1 , V t − 1 ) | ≥ (cid:15) or t < T AM do //Use SNGD for minimizations
[Equation]
t := t [Equation] end 7 endOutput :U , V the dataset 1 , with 60 , 000 data samples used for training and 10 , 000 samples for testing.
Experiments were conducted using Torch 7 Collobert et al.
Autoencoder with Sigmoid Activation: A single-layer autoencoder (equivalent to a neural network with one hidden layer) was trained using DANTE and SGD with 600 hidden units, a sigmoid activation, a learning rate of 0.001, standard Mean-Squared Error loss function and a minibatch size of 500 (same setup was maintained for SGD and the SNGD used inside DANTE for fair comparison; one could optimize both SGD and SNGD to improve the absolute result values).
The results are shown in Figure 2a.
While DANTE takes slightly (negligibly) longer to reach a local minimum, it obtains a better solution than SGD.
Autoencoder with ReLU Activation: A single-layer autoencoder was trained using DANTE and
SGD with 600 hidden units, a Leaky ReLU activation (with leakiness parameter 0.01), a learning rate of 0.001, Mean-Squared Error loss function and a minibatch size of 500.
The results are shown in
Figure 2b.
The results for ReLU showed an improvement, and DANTE was consistently superior to
SGD across the iterations.
In Figure 3, we also show the reconstructions obtained by both trained models (DANTE and SGD) for the autoencoder with the Generalized ReLU activation. The model trained using DANTE shows [Figure]
[Table] comparable performance as a model trained by SGD under the same settings, in this case.
We also conducted experiments to study the effectiveness of the feature representations learned using the models trained using DANTE and SGD in the same setting.
After training, we passed the dataset through the autoencoder, extracted the hidden layer representations, and then trained a linear SVM.
The classiﬁcation accuracy results using the hidden representations are given in Table 1.
The table clearly shows the superior performance of DANTE on this task.
Experiments on other datasets: We also studied the performance of proposed method on other standard datasets 2 , viz.
Ionosphere (34 dimensions, 351 datapoints), SVMGuide4 (10 dimensions, 300 datapoints), and USPS (256 dimensions, 7291 datapoints).
Figure 4 and Table 1 show the performance of the proposed method vs SGD on the abovementioned datasets.
It can be clearly seen that the proposed DANTE method demonstrates superior generalization performance on most datasets.
Varying Number of Hidden Neurons:
Given the decomposable nature of the proposed solution to learning autoencoders, we also studied the effect of varying hyperparameters across the layers,
[Figure]
[Figure]
in particular, the number of hidden neurons in a single-layer autoencoder.
The results of these experiments are shown in Figure 5.
The plots show that when the number of hidden neurons is low, DANTE reaches its minumum value much sooner (considering this is a subgradient method, one can always choose the best iterate over training) than SGD, although SGD ﬁnds a slightly better solution. However, when the number of hidden neurons increases, DANTE starts getting consistently better. This can be attributed to the fact that the subproblem is relatively more challenging for an alternating optimization setting when the number of hidden neurons is lesser.
[Figure] Multi-Layer Autoencoder: We also studied the performance of the proposed multi-layer DANTE method (Algorithm 3) for the MNIST dataset.
Figure 6 shows the results obtained by stacking two single-layer autoencoders, each with the generalized (leaky) ReLU activation (note that a two singlelayer autoencoder corresponds to 4 layers in the overall network, as mentioned in the architecture on the ﬁgure).
Evidently, the ﬁgure shows promising performance for DANTE in this experiment.
Note that Figure 6b shows two spikes: one when the training for the next pair of layers in the autoencoder begins, and another when the end-to-end ﬁnetuning process is done.
This is not present in Figure 6a, since the 500 → 500 layer in between is only randomly initialized, and is not trained using DANTE or SGD.
C ONCLUSIONS AND F UTURE W ORK
In this work, we presented a novel methodology, Deep AlterNations for Training autoEncoders (DANTE), to efﬁciently train autoencoders using alternating minimization, thus providing an effective alternative to backpropagation.
We formulate training each layer of an autoencoder as a Generalized
Linear Model (GLM) problem with an activation function, and leverage recent results to use Stochastic
Normalized Gradient Descent (SNGD) to train each step of the autoencoder.
While recent work showed that a GLM with a sigmoid activation function is SLQC (which SNGD solves with a provable convergence bound), we introduced a new generalized ReLU activation function, and showed that a GLM with this activation function is also SLQC, thus allowing us to expand the applicability of the proposed method to autoencoders with both sigmoid and ReLU family of activation functions.
In particular, we extended the deﬁnitions of local quasi-convexity to use subgradients in order to prove that the GLM with generalized ReLU activation is (cid:16) (cid:15), b 2 W 2 a , w ∗ (cid:17) − SLQC , which improves the convergence bound for the corresponding GLM with the generalized ReLU (as compared to a GLM with sigmoid). We also showed how DANTE can be extended to train multi-layer autoencoders.
We empirically validated DANTE with both sigmoidal and ReLU activations on standard datasets as well as in a multi-layer setting, and observed that it performs comparably or better than the standard SGD approach to train autoencoders.
DANTE can not only be used to train autoencoders, but can be extended to train standard multi-layer neural networks too.
One could use DANTE to train a neural network layer-wise in a round robin fashion, and then ﬁnetune end-to-end using SGD.
In case of autoencoders with tied weights, one would ideally use DANTE to learn the weights of the required layers, and then ﬁnetune end-to-end using a method such as SGD. Our future work will involve a more careful study of the proposed method for deeper autoencoders, as well as in theoretically analyzing the end-to-end convergence of the method for deep multi-layer autoencoders.
A N OISY GLM WITH G ENERALIZED R E LU A CTIVATION F UNCTION
The theorem below is a continuation of our discussion from Section ?? .
Theorem A.1.
In the noisy GLM with generalized ReLU activation, assuming || w ∗ || ≤ W , given w ∈ B (0 , W ) , then with probability ≥ 1 − δ after m ≥ 4 Wb 2 a(cid:15) 2 log (1 /δ ) /(cid:15)
2 samples, ˆ err ( w ) is (cid:16) (cid:15), 2 b 2 Wa , w ∗ (cid:17) − SLQC in w .
Proof.
Here, ∀ i, y i ∈ [0 , 1] , the following holds:
[Equation]
where { ξ i } mi =1 are zero mean, independent and bounded random variables, i.e. ∀ i ∈ [ m ] ,
|| ξ i || ≤ 1 .
Then, ˆ err m ( w ) may be written as follows (expanding y i as in Eqn 6):
[Equation]
Therefore, we also have (by deﬁnition of noisy GLM in Defn 3.2):
[Equation]
Consider || w || ≤ W such that ˆ err m ( w) − ˆ err m ( w ∗ ) ≥ (cid:15)
[Equation]
Also
[Equation]
let v be a point (cid:15)/κ -close to minima w ∗ , κ = 2 b 2 Wa .
Let g be the subgradient of the generalized ReLU activation and G be the subgradient of ˆ err m ( w) .
Then:
[Equation]
Step
[Equation]
+
[Equation]
m
[Equation]
Step
[Equation]
[Table]
+ 1 m m (cid:88) i =
ξ i λ i ( w ) (Step 7) ≥ ab − 1 (cid:15) + 1 m m (cid:88) i =
ξ i λ i ( w ) (Step 8) (7)
Here, λ i ( w ) = 2 a ( (cid:104) w , x i (cid:105) − (cid:104) w ∗ , x i (cid:105) )
+ 2 b − 1 ( φ (cid:104) w ,
x i (cid:105) − φ (cid:104) w ∗ , x i (cid:105) ) , and | ξ i λ i ( w ) | ≤ 2 a ( |(cid:104) w , x i (cid:105) − (cid:104) w ∗ , x i (cid:105)| + 2 b − 1 | φ (cid:104) w , x i (cid:105) − φ (cid:104) w ∗ , x i (cid:105)| ) ≤ 2 a (2 |(cid:104) w , x i (cid:105) − (cid:104) w ∗ , x i (cid:105)| ) ≤
2 a (4 W ) = 8 aW
The above proof uses arguments similar to the proof for the idealized GLM (please see the lines after the proof of Theorem 3.4, viz.
the b -Lipschitzness of the generalized ReLU, and the problem setup).
Now, when
[Equation]
our model is SLQC.
By simply using the Hoeffding’s bound, we get that the theorem statement holds for m ≥
Wb 2 a(cid:15) 2 log (1 /δ ) /(cid:15) 2 .