T RAINING A UTOENCODERS BY A LTERNATING M INIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work, we present a novel method, DANTE, to train neural networks - in particular, autoencoders - using alternating minimization.
This method provides a different perspective in lieu of traditional gradient-based backpropagation commonly used to train deep networks such as autoencoders.
DANTE utilizes an adaptation of quasi-convex optimization techniques to cast autoencoder training as a bi-quasi-convex optimization problem.
We show that for autoencoder conﬁgurations with both differentiable (e.g. sigmoid) and non-differentiable (e.g. ReLU) activation functions, we can perform alternations effectively.
In experiments on standard datasets, autoencoders trained using DANTE were found to be very promising when compared to those trained using traditional backpropagation techniques, both in terms of solutions as well as feature extraction and reconstruction performance. We also extended DANTE to multi-layer settings and showed that the proposed method performs promisingly in this setting too.
I NTRODUCTION
For much of the recent march of deep learning, gradient-based backpropagation methods (e.g.
Stochastic Gradient Descent, SGD) have been the mainstay of practitioners.
The use of these methods, especially on vast amounts of data, has led to unprecedented progress in several areas of artiﬁcial intelligence.
On one hand, the intense focus on these techniques has led to an intimate understanding of hardware requirements and code optimizations needed to execute these routines on large datasets in a scalable manner.
Today, myriad off-the-shelf and highly optimized packages exist that can churn reasonably large datasets on GPU architectures with relatively mild human involvement and little bootstrap effort.
However, this surge of success of backpropagation-based methods in recent years has overshadowed the need to continue to look for options beyond backprogagation to train deep networks.
Despite several advancements in deep learning with respect to novel architectures such as encoder-decoder networks and generative adversarial models, the reliance on backpropagation methods remains.
While reinforcement learning methods are becoming increasingly used [REF] today, their scope is limited to a particular family of settings such as agent-based systems or reward-based learning.
Recent efforts have studied the limitations of SGD-based backpropagation, including parallelization of SGD-based techniques that are inherently serial Taylor et al.
(2016), vanishing gradients, especially for certain activation functions Hochreiter & Schmidhuber (1997), convergence of stochastic techniques to local optima Anandkumar & Ge (2016), and many more.
For a well-referenced critique of gradient-based methods, we request the reader to refer Taylor et al.
From another perspective, there has been marked progress in the area of non-convex optimization (beyond deep learning) in recent years, which has resulted in scalable methods such as iterated hard thresholding Blumensath & Davies (2009) and alternating minimization Jain et al.
as methods of choice for solving large-scale sparse recovery, matrix completion, and tensor factorization tasks.
Several of these methods not only scale well to large problems, but also offer provably accurate solutions.
In this work, we seek to formulate a non-backpropagation strategy to train neural networks, leveraging a recent result in quasi-convex optimization.
Our method is called DANTE (Deep AlterNations for Training autoEncoders), an alternating minimization-based technique for training neural networks - in particular, autoencoders.
DANTE is based on our observation that the problem of training a single hidden-layer autoencoder can be cast as a bi-quasiconvex optimization problem (described in Section 3.1).
This observation allows us to use an alternating optimization strategy to train the autoencoder, where each step involves relatively simple quasi-convex problems.
DANTE then uses efﬁcient solvers for quasi-convex problems including normalized gradient descent Nesterov (1984) and the more recent stochastic normalized gradient descent Hazan et al.
(2015) to train an autoencoder.
The key contributions of this work are summarized below:
• We introduce DANTE, a new method to efﬁciently train neural networks using alternating minimization. DANTE trains Autoencoder (AE) networks through alternating optimization of quasi-convex - Strictly-Locally-Quasi-Convex (SLQC), to be speciﬁc - problems using an efﬁcient stochastic solver. • DANTE views each layer of a neural network as a Generalized Linear Model (GLM).
While it has been recently shown that a GLM with sigmoid activation function is SLQC Hazan et al. (2015), we introduce a generalized version of the popularly used ReLU activation function, and show that a GLM with such non-differentiable activation functions can also be shown to be SLQC.
This allows us to apply our methodology to AEs with both differentiable and non-differentiable activation functions, including ReLUs and sigmoid.
• We show that the Stochastic Normalized Gradient Descent (SNGD) method offers provably more rapid convergence under this newly introduced activation function for an idealized GLM problem than it does for the sigmoidal activation function.
A key advantage of our approach is that the theoretical result provides a direction to set the learning rate and batch size to provide an acceptable level of convergence while training the AE.
• We empirically validate DANTE with both generalized ReLU and sigmoid activations and establish that the proposed method provides comparable or better test error, reconstructions and classiﬁcation performance (with the learned representations), when compared with a regularly used mini-batch SGD setup. • We also show that the proposed methodology can be extended to train deep AEs, and our results on deep AEs show promise in this setting too.
R ELATED W ORK
Backpropagation-based techniques date back to the early days of neural network research Rumelhart et al.
; Chauvin & Rumelhart (1995) but remain to this day, the most commonly used methods for training a variety of neural networks including multi-layer perceptrons, convolutional neural networks, autoencoders, recurrent networks and the like.
Recent years have seen the development of other methods, predominantly based on least-squares approaches, used to train neural networks.
Carreira-Perpinan and Wang Carreira-Perpinan & Wang (2014) proposed a least-squares based method to train a neural network.
In particular, they introduced the Method of Auxiliary Constraints (MAC), and used quadratic penalties to enforce equality constraints.
Patel et al. Patel et al.
(2015) proposed an Expectation-Maximization (EM) approach derived from a hierarchical generative model called the Deep Rendering Model (DRM), and also used least-squared parameter updates in each of the EM steps.
They showed that forward propagation in a convolutional neural network was equivalent to the inference on their DRM.
Unfortunately, neither of these methods has publicly available implementations or even published training results to compare against.
More recently, Taylor et al. proposed a method to train neural networks using the Alternating
Direction Method of Multipliers (ADMM) and Bregman iterations Taylor et al.
The focus of this method, however, was on scaling the training of neural networks to a distributed setting on multiple cores across a computing cluster.
Jaderberg also proposed the idea of ’synthetic gradients’ in
Jaderberg et al.
While this approach is interesting, this work is more towards a more efﬁcient way to carry out gradient-based parameter updates in a neural network.
In this work, we focus on a new approach to train neural networks - in particular, autoencoders - using alternating optimization, quasi-convexity and SNGD, and show that this approach shows promising results on the considered datasets.
To the best of our knowledge, this is the ﬁrst such effort in using alternating SNGD to train neural networks with related performance guarantees.
DANTE: D EEP A LTER N ATIONS FOR T RAINING AUTO E NCODERS 3. P ROBLEM F ORMULATION
Consider a neural network with L layers.
Each layer l ∈ { 1 , 2 , . . .
, L } has n l nodes and is characterized by a linear operator W l ∈ R n l − 1 × n l and a non-linear activation function φ l : R n l → R n l .
The activations generated by the layer l are denoted by a l ∈ R n l .
We denote by a 0 , the input activations and n 0 to be the number of input activations i.e. a 0 ∈ R n 0 .
Each layer uses activations being fed into it to compute its own activations as a l = φ l (cid:104) W l , a l − 1 (cid:105) , where φ (cid:104) .
(cid:105) denotes φ ( (cid:104) .
(cid:105) ) for simplicity of notation.
A multi-layer neural network is formed by nesting such layers to form a composite function f given as follows:
[Equation]
where W = { W l } is the collection of all the weights through the network, and x = a 0 contains the input activations for each training sample.
Given m data samples { ( x i , y i ) } mi =1 from some distribution D , the network is trained by tuning the weights W to minimize a given loss function, J :
[Equation]
Note that a multi-layer autoencoder is trained similarly, but with the loss function modiﬁed as below:
[Equation]
For purposes of simplicity and convenience, we consider the case of a single-layer autoencoder, represented as f ( W ; x ) =
φ 2 (cid:104) W 2 , φ 1 (cid:104) W 1 , x (cid:105)(cid:105) to describe our methodology.
We describe in a later section on how this idea can be extended to deep multi-layer autoencoders.
(Note that our deﬁnition of a single-layer autoencoder is equivalent to a two-layer neural network in a classiﬁcation setting, by nature of the autoencoder.)
A common loss function used to train autoencoders is the squared loss function which, in our simpliﬁed setting, yields the following objective.
[Equation]
Now denote
[Equation]
An important observation here is that if we ﬁx W 1 , then Eqn (5) turns into a Generalized Linear
Model problem with φ 2 as the activation function, i.e.
[Equation]
where z = φ 1 (cid:104) W 1 , x (cid:105) .
Similarly, inverting the network and ﬁxing W 2 turns the problem into yet another Generalized Linear Model problem, this time with W 1 as the parameter. [Equation]
where φ (cid:104)·(cid:105) =
φ 2 (cid:104) W 2 , φ 1 (cid:104)·(cid:105)(cid:105) .
Thus, a single-layer autoencoder is a combination of two Generalized Linear Model (GLM) problems, and we exploit this key observation in this work. In particular, we leverage a recent result in Hazan et al. that a GLM with a differentiable activation, such as sigmoid, is Strictly Locally Quasi-Convex (SLQC), which allows us to use SNGD to solve each sub-problem of the alternating setup efﬁciently (with performance guarantees).
We further show that a GLM with a non-differentiable activation – in particular, a generalized Rectiﬁed Linear Unit (ReLU) – is also SLQC, thus allowing us to extend the proposed alternating strategy, DANTE, to ReLU-based autoencoders too. We note that while we have developed this idea to train autoencoders in this work (since our approach relates closely to the greedy layerwise training in autoencoders), DANTE can be used to train standard multi-layer neural networks too (discussed in Section 5).
Input : Number of iterations T , { ( x i , y i ) } mi =1 ∈ R d , learning rate η , minibatch size b , Initial parameters w1 for t = 1 to T do
Sample { ( x i , y i ) } bi =1 ∼ D
Let f t ( w ) = 1 b (cid:80) bi =
( y i − φ (cid:104) w , x i (cid:105) )
Let g t = ∇ f t ( w t ) , and ˆ g ( t )
=
g t (cid:107) g t (cid:107) w t
+
=
w t − η · ˆ g t 6 endOutput : Model given by w T
Input : Stopping threshold (cid:15) , Number of iterations of alternating minimization T AM , Number of iterations for each SNGD T SNGD , initial values w 01 , w 02 , learning rate η , minibatch size b 1 t :=2 while | f ( W t 1 , W t 2 ) − f
( W t − 1, W t − 1) | ≥ (cid:15) or t < T AM do W t 2 ← arg min W
E x ∼D (cid:107) φ 2 (cid:104) W, φ 1 (cid:104) W t − 1, x (cid:105)(cid:105) − x (cid:107)//use SNGD
W t 1 ← arg min W
E x ∼D (cid:107) φ 2 (cid:104) W t 2 , φ 1 (cid:104) W, x (cid:105)(cid:105) − x (cid:107)//use SNGD t := t +6 endOutput :w 1 , w
3. M ETHODOLOGY
We begin our presentation of the proposed method by brieﬂy reviewing the Stochastic Normalized
Gradient Descent (SNGD) method, which is used in each step of our alternating strategy.
(We explain in the next section, the rationale for the choice of SNGD).
Stochastic Normalized Gradient Descent (SNGD):
Normalized Gradient Descent (NGD) is an adaptation of traditional Gradient Descent where the updates in each iteration are purely based on the direction of the gradients, and not the gradients themselves (achieved by normalizing the gradients).
SNGD is the stochastic version of NGD, where weight updates are performed at the end of every training sample, instead of the complete set of samples.
Mini-batch SNGD updates the parameters at the end of every mini-batch of samples, as in mini-batch Stochastic Gradient Descent (SGD).
In the remainder of this paper, we refer to mini-batch SNGD as SNGD itself, as is common for SGD.
Algorithm 1 describes the SNGD methodology for a generic GLM problem.
DANTE:
Given this background, Algorithm 2 outlines the proposed method, DANTE.
Consider the autoencoder problem below:
[Equation]
Upon ﬁxing the parameters of the lower layer i.e. W 1 , it is easy to see that we are left with a GLM problem:
[Equation]
where z = φ 1 (cid:104) W 1 , x (cid:105) .
DANTE solves this intermediate problem using SNGD steps by sampling several mini-batches of data points and performing updates as dictated by Algorithm 1.
Similarly, upon ﬁxing the parameters of the upper layer, i.e. W 2 , we are left with another GLM problem:
[Equation]
where v = φ 2 (cid:104) W 2 , x (cid:105) .
3. R ATIONALE
To describe the motivation for our alternating strategy in DANTE, we ﬁrst deﬁne key terms and results that are essential for our work.
We present the notion of a locally quasi-convex function (as introduced in Hazan et al.
and show that under certain realizability conditions, empirical objective functions induced by Generalized Linear Models (GLMs) are locally quasi-convex.
To this end, we introduce a new activation function, the generalized ReLU, and show that the GLM with the generalized ReLU also satisﬁes this property.
We then cite a result that shows that SNGD converges to the optimum solution provably for locally quasi-convex functions, and extend this result to the newly introduced activation function.
Deﬁnition 3.1 ( Local-Quasi-Convexity ) .
Let x , z ∈ R d , κ, (cid:15) > 0 and let f : R d → R be a differentiable function.
Then f is said to be ( (cid:15), κ, z ) -Strictly-Locally-Quasi-Convex
(SLQC) in x , if at least one of the following applies: f
[Equation] − [Equation] ≤ (cid:15) 2.
(cid:107)∇ f ( x ) (cid:107) > 0 , and ∀ y ∈ B ( z , (cid:15)/κ ) , (cid:104)∇ f ( x ) , y − x (cid:105) ≤
where B ( z , (cid:15)/κ ) refers to a ball centered at z with radius (cid:15)/κ .
Deﬁnition 3.2 ( Idealized and Noisy Generalized Linear Model (GLM) ) .
Given an (unknown) distribution D and an activation function φ : R → R , an idealized GLM is deﬁned by the existence of a w ∗ ∈ R d such that y i =
φ (cid:104) w ∗ , x i (cid:105)∀ i ∈ [ m ] where w ∗ is the global minimizer of the error function:
[Equation]
m (cid:88) [Equation] [Equation] Similarly, a noisy GLM is deﬁned by the existence of a w ∗ ∈ R d , which is the global minimizer of the error function: E ( x ,y )
∼D ( y | x ) =
φ ( (cid:104) w ∗ , x (cid:105) ) .
the expected error is then:
[Equation]
(Hazan et al., 2015, Lemma 3.2) showed that if we draw m ≥ Ω (cid:16) exp(
[Equation]
∗
[Equation]
log 1 δ (cid:17) samples { ( x i , y i ) } mi =1 from a GLM with the sigmoid activation function, then with probability at least 1 − δ , the empirical error function
[Equation]
is ( (cid:15), e (cid:107) w ∗ (cid:107) 2 , w ∗ ) -SLQC in w .
In order to go beyond the sigmoid activation function, we now introduce a new generalized ReLU activation function.
Deﬁnition 3.3.
(Generalized ReLU)
The generalized ReLU function f : R → R , 0 < a < b , a, b ∈ R is deﬁned as:
[Equation]
This function is differentiable at every point except 0.
Note that this deﬁnition also subsumes variants of ReLU such as the leaky ReLU
Xu et al.
We deﬁne the function g that gives a valid subgradient at all x for the generalized ReLU to be
[Equation]
While SLQC is primarily deﬁned for differentiable functions, we now show that with the above deﬁnition of the subgradient, the GLM with the generalized ReLU is also SLQC.
Theorem 3.4.
In the idealized GLM with generalized ReLU activation, assuming || w ∗ || ≤ W , ˆ err ( w ) is (cid:16) (cid:15), 2 b 2 W a , w ∗ (cid:17) − SLQC in w for all w ∈ B d (0 , W ) .
Proof.
Consider || w || ≤ W such that
ˆ err m ( w) = 1 m (cid:80) mi =1 ( y i − φ (cid:104) w , x i (cid:105) )
2 ≥ (cid:15) , where m is the total number of samples.
Also let v be a point (cid:15)/κ -close to minima w ∗ with κ = 2 b 2 Wa .
Let g be the subgradient of the generalized ReLU activation and G be the subgradient of ˆ err m ( w) .
(Note that as before, g (cid:104) .
(cid:105) denotes g ( (cid:104) .
(cid:105) ) ).
Then: (cid:104) G ( w ) , w − v (cid:105)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
In the above proof
[Equation]
we ﬁrst use the fact (in Step 1) that in the GLM, there is some w ∗ such that φ (cid:104) w ∗ , x i (cid:105) =
y i .
Then, we use the fact (in Steps 2 and 4) that the generalized ReLU function is b -Lipschitz, and the fact that the minimum value of the quasigradient of g is a (Step 3).
Subsequently, in Step 5, we simply use the given bounds on the variables x i , w , w ∗ due to the setup of the problem ( w ∈ B d (0 , W ) , and x i ∈ B d , the unit d -dimensional ball, as in Defn 2.6).
We also prove a similar result for the Noisy GLM below.
Theorem 3.5.
In the noisy GLM with generalized ReLU activation, assuming || w ∗ || ≤ W , given w ∈ B (0 , W ) , then with probability ≥ 1 − δ after m ≥ 4 Wb 2 a(cid:15) 2 log (1 /δ ) /(cid:15)
2 samples, ˆ err ( w ) is (cid:16) (cid:15), 2 b 2 Wa , w ∗ (cid:17) − SLQC in w .
The proof for Theorem 3.5 is included in Appendix A (Supplementary Section).
Now, we review a result from Hazan et al.
(2015) which shows that SNGD provably converges to the optimum for SLQC functions, and hence, with very high probability, for empirical objective functions induced by noisy GLM instances too.
Theorem 3.6 (Hazan et al.
Let (cid:15), δ, G, M, κ > 0 , let f : R d → R and w ∗ =
arg min w f ( w ) .
Assume that for b ≥ b 0 ( (cid:15), δ, T ) , with probability ≥ 1 − δ , f t deﬁned in Algorithm 1 is ( (cid:15), κ, w ∗ ) -SLQC ∀ w , and | f t | ≤ M ∀ t ∈ { 1 , · · · , T } .
If we run SNGD with T ≥ κ 2 || w 1 − w ∗ || 2 (cid:15)and η = (cid:15)κ , and b ≥ max (cid:26) M 2 log ( 4 Tδ ) 2 (cid:15), b 0 ( (cid:15), δ, T ) (cid:27) , with probability 1 − 2 δ , f ( w ) − f ( w ∗ ) ≤ 3 (cid:15) .
Thus, given a single-layer autoencoder with either a sigmoid activation function or a generalized
ReLU activation function (since tanh is a rescaled sigmoid and the generalized ReLU includes many variants of ReLUs, this covers a large family of activation functions), DANTE uses SNGD to solve a
SLQC problem in each alternating step, which converges to the optima solution in each step with [Figure] Algorithm
Input : Encoder e with weights U , Decoder d with weights V , Number of hidden layers 2 n − 1 ,
Learning rates η , Stopping threshold (cid:15) , Number of iterations of alternating minimization
T AM , initial values U 0 , V 0 , minibatch size b 1 t :=2 for l = 1 to n do while | f ( U t , V t ) − f
( U t − 1 , V t − 1 ) | ≥ (cid:15) or t < T AM do //Use SNGD for minimizations [Equation] t := t
[Equation]
end 7 endOutput :U , V very high probability, as shown above in Theorem 3.6.
Furthermore, while the GLM error function with sigmoid activation has κ = e W Hazan et al.
, we obtain κ = 2 b 2 W a (linear in W ) for the generalized ReLU setting.
This is signiﬁcant as in Theorem 3.6, the number of iterations T is lower-bounded by an expression with κ in the numerator.
3. E XTENDING TO A M ULTI -L AYER A UTOENCODER
In the previous sections, we illustrated how a single hidden-layer autoencoder can be cast as a set of
SLQC problems and proposed an alternating minimization method, DANTE.
This approach can be generalized to deep autoencoders by considering the greedy layer-wise approach to training a neural network (Bengio et al.
In this approach, each pair of layers of a deep stacked autoencoder is successively trained in order to obtain the ﬁnal representation.
Each pair of layers considered in this paradigm is a single hidden-layer autoencoder, which can be cast as a pair of SLQC problems that can be trained using DANTE.
Therefore, training a deep autoencoder using greedy layer-wise approach can be modeled as a series of SLQC problem pairs.
Algorithm 3 summarizes the proposed approach to use DANTE for a deep autoencoder, and Figure 1 illustrates the approach.
Note that it may be possible to use other schemes to use DANTE for multi-layer autoencoders such as a round-robin scheme, where each layer is trained separately one after the other in the sequence in which the layers appear in the network.
[Figure]
[Figure]
E XPERIMENTS AND R ESULTS
We validated DANTE by training autoencoders on the standard MNIST dataset LeCun et al.
(1998) as well as other datasets from the UCI repository.
We also conducted experiments with multi-layer autoencoders as well as with single-layer autoencoders with varying number of hidden neurons.
Each of these experiments and the corresponding results are described below.
Our ﬁrst set of experiments were carried out with the standard benchmarking setup of the MNIST dataset 1 , with 60 , 000 data samples used for training and 10 , 000 samples for testing.
Experiments were conducted using Torch 7 Collobert et al.
Autoencoder with Sigmoid Activation: A single-layer autoencoder (equivalent to a neural network with one hidden layer) was trained using DANTE and SGD with 600 hidden units, a sigmoid activation, a learning rate of 0.001, standard Mean-Squared Error loss function and a minibatch size of 500.
The results are shown in Figure 2a.
While DANTE takes slightly (negligibly) longer to reach a local minimum, it obtains a better solution than SGD.
Autoencoder with ReLU Activation: A single-layer autoencoder was trained using DANTE and
SGD with 600 hidden units, a Leaky ReLU activation (with leakiness parameter 0.01), a learning rate of 0.001, Mean-Squared Error loss function and a minibatch size of 500.
The results are shown in
Figure 2b.
The results for ReLU showed an improvement, and DANTE was consistently superior to
SGD across the iterations.
In Figure 4, we also show the best reconstructions - both qualitatively and quantitatively - obtained by both trained models (DANTE and SGD) for the autoencoder with the leaky ReLU activation.
The model trained using DANTE shows qualitatively better reconstructions, when compared to reconstructions obtained using a model trained by SGD under the same settings.
We also conducted experiments to study the effectiveness of the feature representations learned using the models trained using DANTE and SGD in the same setting.
After training, we passed the dataset through the autoencoder, extracted the hidden layer representations, and then trained a linear SVM. The
[Table] classiﬁcation accuracy results using the hidden representations on various datasets are given in Table 1.
Varying Number of Hidden Neurons:
Given the decomposable nature of the proposed solution to learning autoencoders, we also studied the effect of varying hyperparameters across the layers, in particular, the number of hidden neurons in a single-layer autoencoder.
The results of these experiments are shown in Figure 6.
The plots ﬁrstly show that DANTE performs competitively when compared to SGD.
When the number of hidden neurons is relatively lower, both DANTE and SGD show comparable performance (SGD takes longer to reach the same solution, but ﬁnds a slightly better solution at the end); however, when the number of hidden neurons increases, DANTE is consistently better.
This can be attributed to the fact that the subproblem is relatively more challenging for an alternating optimization setting when the number of hidden neurons is lesser.
Experiments on other datasets: We also studied the performance of proposed method on other standard datasets 2 , viz.
Ionosphere (34 dimensions, 351 datapoints), Satimage (36 dimensions, 4435 datapoints), SVMGuide4 (10 dimensions, 300 datapoints), and USPS (256 dimensions,datapoints).
Figure 3 shows the performance of the proposed method vs SGD on the above mentioned datasets.
It can be clearly seen that the proposed DANTE method demonstrates superior generalization performance on most datasets. [Figure]
Multi-Layer Autoencoder:
We also studied the performance of the proposed multi-layer DANTE method (Algorithm 3) for the MNIST dataset.
Figure 5 shows the results obtained by stacking two single-layer autoencoders, each with the generalized (leaky) ReLU activation (note that a two singlelayer autoencoder corresponds to 4 layers in the overall network, as mentioned in the architecture on the ﬁgure).
Evidently, the ﬁgure shows promising performance for DANTE in this experiment too.
D ISCUSSIONS AND C ONCLUSION
In this work, we presented a novel methodology, Deep AlterNations for Training autoEncoders (DANTE), to efﬁciently train autoencoders using a non-backpropagation method, in particular, using alternating minimization.
The proposed method formulates training an autoencoder as a set of
[Figure] Generalized Linear Model (GLM) problems, which allows us to use an efﬁcient solver proposed in Hazan et al.
(2015) in an alternating fashion to train deep autoencoders.
While the earlier work was proposed only for GLMs with sigmoidal activation functions, we propose an extension of localquasi-convexity to a wider class of activation functions - in particular, the ReLU. We introduce a generalized ReLU activation which allows the Stochastic Normalized Gradient Descent method to have provably faster convergence, when compared to a sigmoidal activation.
We empirically validated
DANTE with both sigmoidal and ReLU activations on standard datasets as well as in a multi-layer setting, and observed that it performs comparably or better than the standard SGD approach to train autoencoders.
Our future work will involve a more careful study of the proposed method for deeper autoencoders, as well as in variations of the autoencoder such as the Convolutional Autoencoder Masci et al.
In this work, we have extended the deﬁnitions of quasi-convexity to use subgradients in order to enable us to prove that the GLM with generalized ReLU activation is (cid:16) (cid:15), b 2 W 2 a , w ∗ (cid:17) − SLQC , mathematically validating that ReLU is far superior to Sigmoid as an activation function.
Additionally, we have proposed a new algorithm for training autoencoders involving alternating minimization techniques and Stochastic Normal Gradient Descent. We showed experimentally, on using hyperparameters suggested theoretically, our proposed algorithm drastically outperforms traditional methods, with the case of ReLUs showing immense improvement.
In this process of experimental validation, we have demonstrated the sensitivity of our algorithm to learning rate and batch size. A N OISY GLM WITH G ENERALIZED R E LU A CTIVATION F UNCTION
The theorem below is a continuation of our discussion from Section ?? .
Theorem A.1.
In the noisy GLM with generalized ReLU activation, assuming || w ∗ || ≤ W , given w ∈ B (0 , W ) , then with probability ≥ 1 − δ after m ≥ 4 Wb 2 a(cid:15) 2 log (1 /δ ) /(cid:15)
2 samples, ˆ err ( w ) is (cid:16) (cid:15), 2 b 2 Wa , w ∗ (cid:17) − SLQC in w .
Proof.
Here, ∀ i, y i ∈ [0 , 1] , the following holds:
[Equation]
where { ξ i } mi =1 are zero mean, independent and bounded random variables, i.e. ∀ i ∈ [ m ] ,
|| ξ i || ≤ 1 .
Then, ˆ err m ( w ) may be written as follows (expanding y i as in Eqn 6):
[Equation]
Therefore, we also have (by deﬁnition of noisy GLM in Defn 3.2):
[Equation]
Consider || w || ≤ W such that ˆ err m ( w) − ˆ err m ( w ∗ ) ≥ (cid:15)
[Equation]
Also
[Equation]
let v be a point (cid:15)/κ -close to minima w ∗ , κ = 2 b 2 Wa .
Let g be the subgradient of the generalized ReLU activation and G be the subgradient of ˆ err m ( w) .
Then:
[Equation]
Step
[Equation]
+
[Equation]
m
[Equation]
Step
[Equation]
[Table]
+ 1 m m (cid:88) i =
ξ i λ i ( w ) (Step 7) ≥ ab − 1 (cid:15) + 1 m m (cid:88) i =
ξ i λ i ( w ) (Step 8) (7)
Here, λ i ( w ) = 2 a ( (cid:104) w , x i (cid:105) − (cid:104) w ∗ , x i (cid:105) )
+ 2 b − 1 ( φ (cid:104) w ,
x i (cid:105) − φ (cid:104) w ∗ , x i (cid:105) ) , and | ξ i λ i ( w ) | ≤ 2 a ( |(cid:104) w , x i (cid:105) − (cid:104) w ∗ , x i (cid:105)| + 2 b − 1 | φ (cid:104) w , x i (cid:105) − φ (cid:104) w ∗ , x i (cid:105)| ) ≤ 2 a (2 |(cid:104) w , x i (cid:105) − (cid:104) w ∗ , x i (cid:105)| ) ≤
2 a (4 W ) = 8 aW
The above proof uses arguments similar to the proof for the idealized GLM (please see the lines after the proof of Theorem 3.4, viz.
the b -Lipschitzness of the generalized ReLU, and the problem setup).
Now, when
[Equation]
our model is SLQC.
By simply using the Hoeffding’s bound, we get that the theorem statement holds for m ≥
Wb 2 a(cid:15) 2 log (1 /δ ) /(cid:15) 2 .