L EARNING REPRESENTATIONS FOR BINARYCLASSIFICATION WITHOUT BACKPROPAGATION
A BSTRACT
The family of feedback alignment (FA) algorithms aims to provide a more biologically motivated alternative to backpropagation (BP), by substituting the computations that are unrealistic to be implemented in physical brains.
While FA algorithms have been shown to work well in practice, there is a lack of rigorous theory prooﬁng their learning capabilities.
Here we introduce the ﬁrst feedback alignment algorithm with provable learning guarantees.
In contrast to existing work, we do not require any assumption about the size or depth of the network except that it has a single output neuron, i.e., such as for binary classiﬁcation tasks.
We show that our FA algorithm can deliver its theoretical promises in practice, surpassing the learning performance of existing FA methods and matching backpropagation in binary classiﬁcation tasks.
Finally, we demonstrate the limits of our FA variant when the number of output neurons grows beyond a certain quantity.
I NTRODUCTION
A key factor enabling the successes of Deep Learning is the backpropagation of error (BP) algorithm
(Rumelhart et al., 1986).
Since it has been introduced, BP has sparked several discussions on whether physical brains are realizing BP-like learning or not (Grossberg, 1987; Crick, 1989).
Today most researchers consent that two distinct characteristics of BP render the idea of a BP based learning in brains as implausible: 1) The usage of symmetric forward and backward connections and 2) the strict separation of activity and error propagation (Bartunov et al., 2018).
These two objections have lead researchers to search for more biologically motivated alternatives to BP
The three most inﬂuential families of BP alternatives distilled so far are Contrastive Hebbian Learing (CHL) (Movellan, 1991), target-propagation (TP) (LeCun, 1986; Hinton, 2007; Bengio, 2014) and feedback Alignment (FA) (Lillicrap et al., 2016).
The idea of CHL is to propagate the target activities, instead of the errors, backward through the network.
For this reason, a temporal dimension is added to the neuron activities.
Each neuron then adapts its parameters based on the temporal differences of its ”forward” and ”backward” activity.
The two signiﬁcant critic points of CHL are the requirement for symmetric ”forward-backward” connections and the use of alternating ”forward” and ”backward” phases (Baldi & Pineda, 1991;
Bartunov et al., 2018).
TP shares the idea with Contrastive Hebbian Learning of propagating target activities instead of errors.
However, rather than keeping symmetric forward and backward paths, the reciprocal propagation of the activities are realized through learned connections.
Consequently, each layer has assigned two objectives: Learning the inverse of the layer’s forward function and minimizing the difference to the back-projected target activity.
Variants of TP differ in how exactly the target activity is projected backward (LeCun, 1986; Bengio, 2014; Bartunov et al., 2018).
Theoretical guarantees of TP rely on the assumption that each reciprocal connection implements the perfect inverse of the corresponding forward function.
This issue of an imperfect inverse was also found to be the ”bottleneck” of TP in practice (Bartunov et al., 2018).
When the output of a layer has a signiﬁcant lower dimension than its input, reconstructing the input from the output becomes challenging, resulting in poor learning performance.
Feedback alignment algorithms eliminate the weight sharing implausibility of BP by replacing the symmetric weights in the error backpropagation path by random matrices.
The second objection, i.e., separate activity and error channels, is attenuated by Direct Feedback Alignment (Nøkland, 2016) which drastically reduces the number of channels carrying an error signal.
While feedback alignment algorithms work well on small and medium-sized benchmarks, a recent study identiﬁed that they are unable to provide learning on more challenging datasets like ImageNet (Bartunov et al., 2018).
Another criticism of FA algorithms is the lack of rigorous mathematical justiﬁcation and convergence guarantees of the performed computations.
In this work, we investigate feed-forward networks where the weights of all, expect the ﬁrst, layers are constrained to positive values.
We prove that this constraint does not invalidate the universal approximation capabilities of neural networks.
Next, we show that, in combination with monotonic activation functions, all layers from the second layer on realize monotonically increasing functions.
The backpropagation of a scalar error signal through these layers only affects the magnitude of the error signal but does not change its sign.
Consequently, we prove that algorithms that bypass the error backpropagation steps, such as Direct Feedback Alignment, can compute the sign of the true gradient with respect to the weights of our constraint networks without the need for backpropagation.
Finally, we show that our algorithm, which we call monotone Direct Feedback Alignment, can deliver its theoretical promises in practice, by surpassing the learning performance of existing feedback alignment algorithms in binary classiﬁcation task, i.e., when the error signal is scalar, and provide decent performance even when the error signal is not scalar.
We make the following key contributions:
• First FA algorithm that has provable learning capabilities for non-linear networks of arbitrary depth • Experimental evaluation showing that our FA algorithm outperforms the learning performance of existing FA algorithms and match backpropagation in binary classiﬁcation tasks • We make an efﬁcient TensorFlow implementation of all tested algorithms publicly available
B ACKPROPAGATION AND F EEDBACK A LIGNMENT
We consider the feed-forward neural network
[Equation]
where f is the non-linear activation function, x the input and h L the output of the network.
For classiﬁcation tasks, h L is usually transformed into a probability distribution with discrete support by a sigmoid or softmax function.
During training, the parameters W l , b l , l = 1 , . . .
L are adjusted to minimize a loss function
L ( y, h L ) on samples of a giving training distribution p ( y, x ) .
This is usually done by performing gradient descent
[Equation]
with respect to the parameters θ l ∈ { W l , b l } , 1 ≤ l ≤ n of the network.
[Figure]
y y
y y x
x x x hhhh
W
W W
W T W T
W
W W
B B
W W
W B
B
W
W + W +
B + B +
Weights that are learned are denoted as W i and ﬁxed weights are denoted by B i .
Weights that are restricted to only positive values are annotated by a + .
a) Backpropagation, b) Feedback alignment, c) Direct Feedback Alignment, d) monotone Direct Feedback Alignment.
2. B ACKPROPAGATION
Backpropagation (Rumelhart et al., 1986) is the primary method to compute the gradients needed by the updates in equation (2) by iteratively applying the chain-rule
[Equation]
[Equation] [Equation]
A graphical representation of how information ﬁrst ﬂow forward and then backward in BP through each layer is shown in Figure (1) a.
Two major concerns argue against the idea that biological neural networks are implementing BPbased learning.
I)
The weight matrix W l of the forward path is reused in the backward path in the form of W Tl ( weight sharing ), and II) the strict separation of activity carrying forward and error carrying backward connections ( reciprocal error transport )
. 2. F EEDBACK A LIGNMENT ALGORITHMS
Feedback alignment addresses the implausibility of reusing W Tl in the backward path by replacing
W Tl by a ﬁxed random matrix B l .
Lillicrap et al.
(2016) showed that this somewhat counterintuitive approach works remarkably well in practice.
The term ”feedback alignment” originates from the observations made by Lillicrap et al.
(2016) that the angle between the FA update vector and the true gradient starts to decrease, i.e., align , after a few epochs of the training algorithm.
Theoretical groundwork on this alignment principle of FA relies on strong assumptions such as a linearized network with one hidden layer (Lillicrap et al., 2016).
FA avoids any weight sharing but does not address the reciprocal error transport implausibility, due to its strict separation of forward and backward pathways, as shown in Figure (1) b. Direct Feedback
Alignment (DFA) (Nøkland, 2016) relaxes this issue by replacing all backward paths with a direct feed from the output layer
error-gradient d L dh L .
Consequently, there is only a single error signal that is distributed across the entire network, which is arguably more biologically plausible than reciprocal error connections.
The resulting parameter updates of DFA are of the form
[Equation]
, where B l ∈ R n L × n l is a random matrix.
A graphical schematic of DFA is shown in Figure (1) c.
Similar to FA, DFA shows a decent learning performance in mid sized classiﬁcation tasks (Nøkland, 2016), but fails on more complex datasets such as ImageNet (Bartunov et al., 2018).
Theory on adapting the alignment principle to DFA shows that under the strong assumptions of constant DFA update directions and a layer-wise criterion minimization, the DFA update vector will align with the true gradient (Nøkland, 2016;
Gilmer et al., 2017).
Recently, Frenkel et al.
(2019) proposed to combine ideas from feedback alignment and targetpropagation in their Direct Random Target Projection (DRTP) algorithm.
While DRTP shows decent empirical performance, theoretical guarantees about DRTP rely on linearized networks.
2. S IGN - SYMMETRY ALGORITHMS
Liao et al.
(2016) introduced the sign-symmetry algorithm, a hybrid of BP and FA.
Sign-symmetry locks the signs of the feedback weight B l to have the same signs as W Tl , but random absolute value.
The authors showed that this approach drastically improves learning performance compared to standard FA.
Furthermore, Moskovitz et al.
(2018) and Xiao et al.
(2019) demonstrated that the sign-symmetry algorithm is even able to match backpropagation for training deep network architectures and large datasets such as ImageNet.
While these empirical observations suggest that the polarity of the error feedback is more important than its magnitude, the mathematical justiﬁcation of sign-symmetry remains absent.
Similar to
FA, sign-symmetry relaxes the strict weight sharing implausibility, but still relies on an unrealistic reciprocal error transport.
MONOTONE D IRECT F EEDBACK A LIGNMENT
In this section, we ﬁrst introduce a new class of feed-forward networks, where all, except the ﬁrst, layers are constrained to realize monotone functions.
We call such networks mono-nets and show that they are as expressive as unconstrained feed-forward networks.
Next, we prove that for our mono-nets with single output tasks, e.g., binary-classiﬁcation, feedback alignment algorithms provide the sign of the gradient.
The sign of the gradient is interesting for learning, as it tells us if the value of a parameter should be increased or decreased in order to reduce the loss.
At the end of this section, we will highlight similarities to algorithms from literature, which can provide resilient learning by only relying on the sign of the gradient.
Neural networks with monotonic constraints have been already studied in literature (You et al., 2017), however not in the context of learning algorithms.
Deﬁnition 1 (mono-net) .
A mono-net is a feed-forward neural network with L layers h 1 , . . .
h L , each layer l composted of n l units and the semantics
[Equation]
where R + are the positive reals, i.e. R + = { x ∈ R | x ≥ 0 } , f is a non-linear monotonic increasing activation function, x the input and h L the output of the network.
The major difference between mono-nets and general feed-forward neural networks is the restriction to only positive weight values in layers from the second layer on.
Combined with the monotonic increasing activation function, this means that each layer h l ( h l − 1 ) , l ≥ 2 realizes a monotone increasing function.
Because functional composition preserves monotonicity, the complete network up to the ﬁrst layer
[Equation]
12) implements a monotone increasing function.
mono-nets are Universal Approximators At ﬁrst glance, this restriction seems counterproductive, as it might interfere with the expressiveness of the networks.
However, we proof in Theorem 1 that our mono-nets with tangent hyperbolic activation are universal approximators, meaning that we can approximate any continuous function arbitrarily close.
A potential drawback of the monotonicity constraint is that we might need a larger number of units in the hidden layers to achieve the same expressiveness as a general feed-forward network, as illustrated in our proof of Theorem 1.
Theorem 1 (mono-nets are Universal Approximators) .
Let I n be the n -dimensional unit hypercube [0 , 1] n and C ( I n ) denote the set of continuous functions f : I n → R .
We deﬁne (cid:107) f (cid:107) ∞ as the supremum norm of f ∈ C ( I n ) over its domain I n .
For any given f ∈ C ( I n ) and ε > 0 , there exist a function m : I n → R of the form
[Equation]
with ¯ v ∈ R + M , ¯ w i ∈ R + n , ˆ w i ∈ R + n , ¯ b ∈ R M , c ∈ R and M < ∞ such that
[Equation]
In essence, the set of functions m ( x ) of the form given in (19) is dense in C ( I n ) .
Proof.
See supplementary materials 3.
mDFA provides the sign of the gradient Here, we prove that for 1-dimensional outputs DFA applied on a mono-net, which we will call simply mDFA, provides the sign of the true gradient.
Note that we focus our methods on DFA instead of ”vanilla” FA, due to the superiority of DFA in terms of biological plausibility and empirical performance (Nøkland, 2016; Bartunov et al., 2018).
Theorem 2 (For 1-dimensional outputs mDFA computes the sign of the gradient) .
Let L ( y, h L ) be a loss function, m ( x ) :
=
h L ◦ h
L − 1 ◦ · · · ◦ h 2 ◦ h 1 ◦
h 0 ( x ) be a mono-net according to Deﬁnition 1 with parameters Θ :=
{ W l , b l (cid:12)(cid:12) l = 1 , . . .
L } .
We denote δθ the update value computed by mDFA and ∇ θ as the gradient ∂ L ∂θ for any θ ∈ { W l , b l } with 1 ≤ l ≤ L .
If n L = 1 , it follows that
[Equation]
for each coordinate ( i, j ) of θ .
Proof.
See supplementary materials.
A graphical illustration of how activities and errors propagate in mDFA is shown in Figure (1) d.
Literature on learning by relying only on the sign of the gradient Two learning concepts related to mDFA are RPROP (Riedmiller & Braun, 1993; Riedmiller, 1994) and signSGD
(Bernstein et al., 2018).
RPROP aims to build a more resilient alternative to gradient descent by decoupling the amplitude of the gradient from the step size of parameter updates.
In essence, for each coordinate
RPROP adapts the step size based on the sign of the most recent gradients computed.
Riedmiller &
Braun (1993) showed that their approach could stabilize the training of a neural network compared to standard gradient descent.
Performing gradient descent with taking the sign of each gradient coordinate is on an algorithmic level equivalent to the well-known steepest descent method with L ∞ norm (Boyd & Vandenberghe, 2004; Bernstein et al., 2018).
signSGD
(Bernstein et al., 2018) studies convergence properties of the stochastic approximation of this algorithm.
What about networks with more than one output neuron?
Theorem 2 applies only to networks with scalar output.
As a natural consequence, one may ask whether such theoretical guarantees can be extended to more dimensional output variables.
The simple answer is, unfortunately not.
In the supplementary materials section A.3 we provide a counterexample showing that Theorem 2 naively extended to two output neurons does not hold anymore.
We want to note that the requirement of a neural network to have only a single output neuron is biologically unjustiﬁed.
It is known that sub-circuits of biological neuronal networks can feed to multiple motor neuron groups Cook et al.
How does mDFA relate to the non-negative matrix factorization?
A seemingly related concept to mDFA is the non-negative matrix factorization (NMF) algorithm.
NMF decomposes an observation matrix V into a weight matrix W and a latent variable matrix H such that V ≈ WH .
In contrast to other decomposition-based unsupervised learning methods, all three matrices V, W and H are restricted to non-negative entries.
While NMF can model data that is inherently non-negative, such has semantic features of images and text, effectively Yuan & Oja (2005);
Shahnaz et al.
(2006), the method is unable to learn subtractive and non-linear structures that are present in the data Lee &
Seung (1999).
Semi-non-negative matrix factorization Ding et al.
(2008) relaxes the original restriction to nonnegative observations of NMF, by only constraining the weight matrix W to be non-negative.
Deep semi-NMF Trigeorgis et al.
(2014) further enhances the expressiveness of NMF by adding multiple layers and non-linearities between them to the decomposition.
Concerning this work, the semantics of mono-nets from the second layer on is equivalent to that of deep semi-NMF models.
However, the unconstrained ﬁrst layer of mono-nets provides universal approximation capabilities, enabling mono-nets to learn subtractive and non-monotonic input dependencies.
Moreover, while deep NMF models are mostly trained via layer-wise learning in an unsupervised context Trigeorgis et al.
Yu et al.
(2018), the sole purpose of mono-nets is to investigate alternatives to backpropagation for training multi-layer classiﬁers.
E XPERIMENTS
So far, we have only proven the learning capabilities of mDFA.
What remains unclear is whether mDFA can deliver its theoretical promises in practice.
In this section, we experimentally evaluate the learning performance of mDFA on a set of empirical benchmarks.
We aim to answer the following two questions:
• How well does mDFA perform compared to DFA, FA, and backpropagation in ”natural conditions,” i.e., in binary classiﬁcation tasks, and • how much does the performance of mDFA degrade in multi-class classiﬁcation tasks?
Our performance measure is the achieved classiﬁcation accuracy of a network trained by a particular method.
First, we report the highest achieved accuracy on the training set, which tells us how well the algorithm could ﬁt the model to the training data.
Secondly, for each method, we tuned the hyperparameters on a separate validation set and selected the best performing conﬁguration to be evaluated on the test data.
The obtained test accuracy tells us how well the model generalizes to data outside the training set.
We evaluate fully-connected networks (FC) and Convolutional networks (CNNs) in form of modiﬁed all-convolutional-nets (Springenberg et al., 2015) with tangent hyperbolic, ReLU (Nair & Hinton, 2010), and the hard-tanh non-linearity.
The hard-tanh function is deﬁned as
[Equation]
Hyperparameters For all training methods, we ﬁxed the batch size to 64, applied no regularization, no normalization, and no data augmentation.
Optimizer, i.e., { ”Vanilla” Gradient Descent, Adam (Kingma & Ba, 2014), Rmsprop (Tieleman & Hinton, 2012) } , learning rate, training epochs, and weight initialization method were tuned on the validation set.
We tested three different weight initialization schemes; all zeros, a scaled uniform, and a normal distribution.
Note that all zeros was only tested on the forward weights.
Our uniform initialization followed the methodology of Nøkland (2016), i.e., scaling the bounds of the distribution inversely by the square-root of the number of incoming connections of a neuron.
In order to comply with the weight constraints of mono-nets, for mDFA the lower bound of the uniform distribution was set to ε = 10 − 3 .
Moreover, for mDFA we post-processed the initial weights of the normal distribution by taking their absolute values.
Input variables are scaled to [0,1].
Detailed network architectures and a brief discussion about the best performing hyperparameter conﬁgurations are listed in the supplementary materials in section B. and section B.3.
4. B INARY CLASSIFICATION
We created a series of binary classiﬁcation benchmarks by randomly sampling two classes from the well-studied CIFAR-100 and ImageNet datasets.
We then train and test a 1-vs-1 classiﬁer on the samples of the two classes.
For each dataset, we create ﬁve such ”binarized” sub-datasets and report the mean and standard deviation over these experiments.
CIFAR-100 (Krizhevsky et al., 2014) poses a challenging classiﬁcation dataset, consisting of 32-by-32 RGB images in 100 real-world object categories.
ImageNet (Russakovsky et al., 2015) is a large-scale object classiﬁcation benchmark and de-facto standard to asses new deep learning methods.
Each sample is a high-resolution image representing one out of 1000 possible object classes.
We pre-processed all samples by cropping and resizing each image to 224-by-224 pixels.
Because ImageNet lacks a public test set, we will report the validation accuracy, i.e., as it was reported by Bartunov et al.
The results on the binarized benchmarks are shown in Table 1 for CIFAR-100 and Table 2 for
ImageNet.
mDFA could bring the training error to zero for fully-connected networks, and match the test/validation accuracy of backpropagation for convolutional networks.
Poor learning performance for ReLU networks One surprising characteristic in our results is that mDFA fails to provide the same level of learning performance as backpropagation for ReLU networks.
Recall that Theorem 1 proves the universal approximation capabilities only for mono-net with tanh activation function.
A mono-net with ReLU non-linearity restricts both, the activation and the weight values, to positive values.
These constraints arguably limit the approximation capabilities of mono-net in combination with ReLU and thus explain the poor performance of mDFA for ReLU networks.
We could validate our hypothesis by testing a rectiﬁer non-linearity that also contains negative values in its image.
Coincidentally, the hard-tanh function matches exactly this criterion.
Therefore, the decent learning performance of mDFA for hard-tanh networks conﬁrms our hypothesis.
Note that also FA and DFA expressed an improvement in performance when switching from ReLU to hard-tanh activation.
This observation suggests that feedback alignment algorithms in general beneﬁt from symmetric activation functions.
Discrepancy between fully-connected and convolutional networks Though mDFA achieved the same training accuracy as backpropagation for fully-connected networks, the generalization ability, i [Table] e., test and validation accuracy, slightly lacks behind BP in our binarized-CIFAR-100 benchmark.
This observation aligns with the studies by Nøkland (2016); Bartunov et al.
For convolutional neural networks, this effect is reversed, i.e., a decent test/validation accuracy but a higher training error than BP.
We speculate that the two tested initialization schemes for the feedback weights caused this discrepancy.
Both backpropagation and feedback alignment have been shown to be sensitive to the employed initialization method (Zhang et al., 2019; Bartunov et al., 2018).
The restriction to positive values of the weights used in mDFA, requires a re-thinking of the initialization methods examined in the literature.
However, we will leave this study open for future work.
[Table] [Table]
[Table]
4. M ULTI - CLASS CLASSIFICATION
Here we modify the classiﬁcation benchmark creation procedure used above, by randomly sampling n classes from the datasets instead of just two.
Due to their compelling results in our binary classiﬁcation benchmark, we restrict our evaluation to networks with tanh activation.
Table 3 and Tableshow the results on our n -class CIFAR-100 benchmark for fully-connected and CNNs respectively.
The results on n -class ImageNet can be found in the supplementary materials in Table 5 and 6.
mDFA can provide learning for networks with more than one output neuron Though we do not have a complete theory on mDFA for multi-dimensional outputs, our experiments indicate that mDFA can provide learning for networks with more than one output neuron.
In particular, the achieved accuracies of mDFA outperforms other feedback alignment methods for convolutional networks with ten or fewer output neurons.
However, mDFA falls behind standard DFA for networks with more than ten output neurons.
This observation suggests that our restriction to positive weights can be beneﬁcial even for multi-class tasks but eventually hurts learning performance when the number of classes grows larger.
Feedback alignment algorithms, in general, tend to struggle with increasing output dimension
Our results express the trend that with an increase in the number of classes, feedback alignment algorithms struggle to ﬁt the training data.
While BP can reduce the training error to almost zero independently of the output dimension, the training errors achieved by FA algorithms are signiﬁcantly higher and correlate with the number of output neurons.
Our observations suggest that the dimension of the error signal affects the training convergence for FA algorithms, in contrast to BP, which appears to be less affected by the dimension of the error signal.
This relation potentially provides a step towards understanding why FA algorithms fail on challenging datasets as described by Bartunov et al.
[Table]
C ONCLUSION
Feedback alignment algorithms are promising biologically motivated alternatives to backpropagation.
While existing literature provides empirical evidence that FA algorithms can work well in practice, there is still a lack of rigorous theory formalizing their learning capabilities.
Here we contributed to the ﬁeld of biologically motivated learning algorithms, by introducing the ﬁrst feedback alignment algorithm that provable provides learning for non-linear networks of arbitrary depth and single output neuron.
We showed that our FA algorithm outperforms existing FA algorithms in binary classiﬁcation tasks, and even provide decent learning on multi-class problems.
Limitations We demonstrated on empirical benchmarks as well as theoretical examples that our method is limited to networks with scalar output.
Indeed, uncovering the mathematical principles behind the decent learning performance of FA and DFA on multi-class tasks remains an open challenge.
Is this really useful?
In terms of scientiﬁc signiﬁcance, this work provided theoretical contributions toward understanding the capabilities and limits of feedback alignment algorithms.
From a practical point of view, our mDFA algorithm has an advantage over backpropagation concerning training latency, as all weight updates can be computed in parallel, i.e., see Figure 1.
Furthermore, we have shown that mDFA is superior to other feedback alignment algorithms in binary classiﬁcation tasks.
Consequently, mDFA provides an effective solution for binary classiﬁcation problems with training latency constraints.
Dynamic branch prediction in microprocessor pipelines poses such problem instance, where program speciﬁc binary branch outcomes, i.e., branch taken/not taken, need to be learned in real-time.
Because of this real-time constraint, existing branch predictors often employ only shallow perceptron modules (Jim ´ enez & Lin, 2002; Egan et al., 2003).
mDFA could enable deeper branch predictor networks to be learned in real-time.
A CKNOWLEDGMENTS
This research was supported in part by the Austrian Science Fund (FWF) under grant Z211-N (Wittgenstein Award).
A A PPENDIX A. MONO - NETS ARE U NIVERSAL A PPROXIMATORS
We denote
[Equation]
the sigmoid function,
[Equation]
the hyperbolic tangent and
[Equation]
the indicator function .
From the deﬁnitions above we can derive the equalities
[Equation]
, which we will need for our proof.
Lemma 1 (Universal Approximation Theorem) .
For any given f ∈ C ( I n ) and ε > 0 , there exist a function g : I n → R of the form
[Equation]
with v ∈ R N , w i ∈ R n , b ∈ R N and N < ∞ such that
[Equation]
In essence, the set of functions g ( x ) of the form given in (18) is dense in C ( I n ) .
Proof.
See Hornik et al.
Theorem 3 (mono-nets are Universal Approximators) .
For any given f ∈ C ( I n ) and ε > 0 , there exist a function m : I n → R of the form
[Equation]
=
[Equation]
19) with v ∈ R + M , w i ∈ R n , ¯ b ∈ R M , c ∈ R and M < ∞ such that
[Equation]
In essence, the set of functions m ( x ) of the form given in (19) is dense in C ( I n ) .
Proof.
By Lemma 1 we know that there exist a g ( x ) with (cid:107) g ( x ) − f ( x ) (cid:107) ∞ < ε that has the form g ( x
[Equation]
= N (cid:88)
[Equation]
We will show that we can reformulate g ( x ) to the form in equation (19).
Our basic idea is to propagate all negative weight entries into to the ﬁrst layer where negative weight values are allowed.
[Equation]
with
[Equation]
Therefore, we showed that there exist a m ( x ) with a form as in equation (19) that satisfy
[Equation]
This proof showed us how to translate any neural network with sigmoid activation function and one hidden layer of size N into a mono-net with 2 N hidden units.
A. F OR 1- DIMENSIONAL OUTPUTS M DFA COMPUTES THE SIGN OF THE GRADIENT
Lemma 2 (The gradient of a monotone layer is non-negative) .
Let m : R n 0 → R with m ( x ) :=
h L ◦ h
L − 1 ◦ · · · ◦ h 2 ◦ h 1 ◦
h 0 ( x ) be a mono-net according to Deﬁnition 1.
Then
[Equation]
20) for any l, i, j with 2 ≤ l ≤ L and 1 ≤ i ≤ n l and 1 ≤ j ≤ n l −
Proof.
We have to distinguish two cases:
Case 1: l = L , i.e. there is no activation function.
We have
[Equation]
[Equation]
according to the deﬁnition in Equation (10).
Case 2: l < L , i.e. there is an activation function.
We have
[Equation]
where f (cid:48) is the derivative of the activation function.
Because f is a monotonic function, its derivative is non-negative everywhere.
As a result we have a sum of a product of non-negative values.
Ergo
[Equation]
[Equation]
Lemma
(The gradient of a composition of monotone layers is non-negative) .
Let m : R n 0 → R with m ( x ) :=
h L ◦ h
L − 1 ◦ · · · ◦ h 2 ◦ h 1 ◦
h 0 ( x ) be a mono-net according to Deﬁnition 1.
Then
[Equation]
for any l, k, i, j with 2 ≤ l ≤ L and 1 ≤ k < l and 1 ≤ i ≤ n l and 1 ≤ j ≤ n k
Proof.
By applying the chain rule we get
[Equation]
According to Lemma 2 we have a product of non-negative values.
Because a product of non-negative values is non-negative itself, we have
[Equation]
Theorem 4 (For 1-dimensional outputs mDFA computes the sign of the gradient) .
Let L ( y, h L ) be a loss function, m ( x ) :
=
h L ◦ h
L − 1 ◦ · · · ◦ h 2 ◦ h 1 ◦
h 0 ( x ) be a mono-net according to Deﬁnition 1 with parameters Θ :=
{ W l , b l (cid:12) (cid:12) l = 1 , . . .
L } .
We denote δθ the update value computed by mDFA and ∇ θ as the gradient ∂ L ∂θ for any θ ∈ { W l , b l } with 1 ≤ l ≤ L .
If n L = 1 , it follows that
[Equation]
for each coordinate ( i, j ) of θ .
Proof.
We distinguish two cases:
Case 1: l = L , i.e. θ is a parameter of the last layer.
From the deﬁnition of mDFA we have
[Equation]
For the gradient by applying the chain rule we get
[Equation]
Thus, in the last layer the mDFA update equals the gradient.
Case 2: l < L , i.e θ is a parameter of a hidden layer.
From the deﬁnition of mDFA we have
[Equation]
) with B ∈ R + n L × n l .
Next, we expand the multiplication,
[Equation]
[Equation]
We assumed that the output dimension is 1, i.e. n L = 1 .
Therefore,
[Equation]
For the gradient by applying the chain rule we get
[Equation]
Like above, we expand the multiplication,
[Equation]
We assumed that the output dimension is 1, i.e. n L = 1 .
Therefore,
[Equation]
For Equation (28) we get by applying Lemma 3,
[Equation] [Equation]
A. F OR k ≥ 2 - DIMENSIONAL OUTPUTS M DFA UPDATES MAY NOT ALIGN WITH THE GRADIENT
Theorem 5 (For k ≥ 2 -dimensional outputs mDFA updates may not align with the gradient) .
Let
L ( y, h L ) be a loss function, m ( x ) :
=
h L ◦ h
L − 1 ◦ · · · ◦ h 2 ◦ h 1 ◦
h 0 ( x ) be a mono-net according to
Deﬁnition 1 with parameters Θ := { W l , b l (cid:12)(cid:12) l = 1 , . . .
L } .
We denote δθ the update value computed by mDFA and ∇ θ as the gradient ∂ L ∂θ for any θ ∈ { W l , b l } with 1 ≤ l < L .
If n L ≥ 2 , there exists the possibility that
[Equation]
for at least one coordinate ( i, j ) of θ .
Proof.
We construct a minimal counterexample consisting of a network with two outputs and one hidden layer of two neurons.
Let
[Equation]
(cid:19) (44)
B
[Equation]
(cid:18)
[Equation]
(cid:19) (45)
[Equation]
L ∂h
[Equation]
(cid:18) 0 −(cid:19) (46)
[Equation]
∂θ
[Equation]
(cid:18)
[Equation]
0(cid:19) (47) (48)
Then we have
[Equation]
= (cid:18) 0 −(cid:19) (cid:18) 01(cid:19) (cid:18) 01(cid:19) (50) =
(cid:18) − 1  (cid:19) (51) and δθ
[Equation]
d L dh L Bdh l dθ (52) =
(cid:18) 0 −(cid:19) (cid:18) 10(cid:19) (cid:18) 01(cid:19) (53) =
(cid:18) 0 −(cid:19) (54) (55) , which are orthogonal.
B E XPERIMENT S ETUP B. N ETWORK ARCHITECTURES
[Table]
Fully-connected
Convolutional
CIFAR1024, (96,5,2),(96,3,2),(96,3,1)
ImageNet 1024,1024,1024, (96,3,2),(96,5,1),(128,3,2), (192,3,1),(192,3,2),(384,3,1)
Network architectures, layers are separated by commas.
Fully-connected column speciﬁes the number of neurons of each layer.
Convolutional column speciﬁes the number of ﬁlters, kernel size, and stride for each layer.
[Table] [Table]
B. D ISCUSSION ON HYPERPARAMETERS
We observed that all FA algorithms yield a more stable convergence with ”vanilla” stochastic gradient descent, i.e., no post-processing of the FA updates, than with Adam (Kingma & Ba, 2014) or
Rmsprop (Tieleman & Hinton, 2012).
This may be non-surprising as these acceleration methods have been developed for gradient-based optimization, whereas FA updates roughly align with the gradients at best.
Furthermore, we observed that FA algorithms achieve a descent validation accuracy after the ﬁrst few training epochs.
However, in contrast to BP, these methods may require over a hundred training epochs to converge fully.
This ”fast-start-slow-convergence” aligns with the observations made by
Lillicrap et al.
We found that FA, DFA, and mDFA perform best when all forward weights are initialized to zero.
Notable exceptions are networks with ReLU activation function which express poor performance with the all-zeros initialization scheme.
This poor performance with all-zeros initialization partially explains the poor observed performance of FA and DFA for ReLU networks.
In contrast to Nøkland (2016), we observed that backward weights initialized by a normal distribution perform slightly better than the scaled uniform proposed by Nøkland (2016).
We conﬁrmed the observation made by Bartunov et al.
(2018), that feedback alignment algorithms are in general relatively sensitive to hyperparameter choice.