S CALING F ORWARD G RADIENT W ITH L OCAL L OSSES
A BSTRACT
Forward gradient learning computes a noisy directional gradient and is a biologically plausible alternative to backprop for learning deep neural networks.
However, the standard forward gradient algorithm, when applied naively, suffers from high variance when the number of parameters to be learned is large.
In this paper, we propose a series of architectural and algorithmic modiﬁcations that together make forward gradient learning practical for standard deep learning benchmark tasks.
We show that it is possible to substantially reduce the variance of the forward gradient estimator by applying perturbations to activations rather than weights.
We further improve the scalability of forward gradient by introducing a large number of local greedy loss functions, each of which involves only a small number of learnable parameters, and a new MLPMixer-inspired architecture, LocalMixer, that is more suitable for local learning.
Our approach matches backprop on MNIST and CIFARand signiﬁcantly outperforms previously proposed backprop-free algorithms on ImageNet.
Code is released at https://github.com/google-research/ google-research/tree/master/local_forward_gradient .
I NTRODUCTION
Most deep neural networks today are trained using the backpropagation algorithm (a.k.a. backprop)
(Werbos, 1974; LeCun, 1985; Rumelhart et al., 1986), which efﬁciently computes the gradients of the weight parameters by propagating the error signal backwards from the loss function to each layer.
Although artiﬁcial neural networks were originally inspired by biological neurons, backprop has always been considered as “biologically implausible” as the brain does not form symmetric backward connections or perform synchronized computations.
From an engineering perspective, backprop is incompatible with a massive level of model parallelism, and restricts potential hardware designs.
These concerns call for a drastically different learning algorithm for deep networks.
In the past, there have been attempts to address the above weight transport problem by introducing random backward weights (Lillicrap et al., 2016; Nøkland, 2016), but they have been found to scale poorly on larger datasets such as ImageNet (Bartunov et al., 2018).
Addressing the issue of global synchronization, several papers showed that greedy local loss functions can be almost as good as end-to-end learning (Belilovsky et al., 2019; Löwe et al., 2019; Xiong et al., 2020).
However, they still rely on backprop for learning a number of internal layers within each local module.
Approaches based on weight perturbation, on the other hand, directly send the loss signal back to the weight connections and hence do not require any backward weights.
In the forward pass, the network adds a slight perturbation to the synaptic connections and the weight update is then multiplied by the negative change in the loss.
Weight perturbation was previously proposed as a biologically plausible alternative to backprop (Xie & Seung, 1999; Seung, 2003; Fiete & Seung, 2006).
Instead of directly perturbing the weights, it is also possible to use forward-mode automatic differentiation (AD) to compute a directional gradient of the ﬁnal loss along the perturbation direction (Pearlmutter, 1994).
Algorithms based on forward-mode AD have recently received renewed interest in the context of deep learning (Baydin et al., 2022; Silver et al., 2022).
However, existing approaches suffer from the curse of dimensionality, and the variance of the estimated gradients is too high to effectively train large networks.
In this paper, we revisit activity perturbation (Le Cun et al., 1988; Widrow & Lehr, 1990; Fiete & Seung, 2006) as an alternative to weight perturbation.
As previous works focused on speciﬁc settings, we explore the general applicability to large networks trained on challenging vision tasks.
We prove that activity perturbation yields lower-variance gradient estimates than weight perturbation, and provide a continuous-time rate-based interpretation of our algorithm.
We directly address the scalability issue of forward gradient learning by designing an architecture with many local greedy loss functions, isolating the network into local modules and hence reducing the number of learnable parameters per loss.
Unlike prior work that only adds local losses along the depth dimension, we found that having patch-wise and channel group-wise losses is also critical.
Lastly, inspired by the design of MLPMixer (Tolstikhin et al., 2021), we designed a network called LocalMixer, featuring a linear token mixing layer and grouped channels for better compatibility with local learning.
We evaluate our local greedy forward gradient algorithm on supervised and self-supervised image classiﬁcation problems.
On MNIST and CIFAR-10, our learning algorithm performs comparably with backprop, and on ImageNet, it performs signiﬁcantly better than other biologically plausible alternatives using asymmetric forward and backward weights.
Although we have not fully matched backprop on larger-scale problems, we believe that local loss design could be a critical ingredient for biologically plausible learning algorithms and the next generation of model-parallel computation.
R ELATED W ORK
Ever since the perceptron era, the design of learning algorithms for neural networks, especially algorithms that could be realized by biological brains, has been a central interest.
Review papers by Whittington & Bogacz (2019) and Lillicrap et al.
(2020) have systematically summarized the progress of biologically plausible deep learning.
Here, we discuss related work in the following subtopics.
Forward gradient and reinforcement learning.
Our work leverages forward-mode automatic differentiation (AD), which was ﬁrst proposed by Wengert (1964).
Later it was used to learn recurrent neural networks (Williams & Zipser, 1989) and to compute Hessian vector products (Pearlmutter, 1994).
Computing the true gradient using forward-mode AD requires the full Jacobian, which is often large and expensive to compute.
Recently, Baydin et al.
(2022) and Silver et al.
(2022) proposed to update the weights based on the directional gradient along a random or learned perturbation direction.
They found that this approach is sufﬁcient for small-scale problems.
This general family of algorithms is also related to reinforcement learning (RL) and evolution strategies (ES), since in each case the network receives a global reward.
RL and ES have a long history of application in neural networks (Whitley, 1993; Stanley & Miikkulainen, 2002; Salimans et al., 2017), and they are effective for certain continuous control and decision-making tasks.
Clark et al.
(2021) found global credit assignment can also work well in vector neural networks where weights are only present between vectorized groups of neurons.
Greedy local learning.
There have been numerous attempts to use local greedy learning objectives for training deep neural networks.
Greedy layerwise pretraining (Bengio et al., 2006; Hinton et al., 2006; Vincent et al., 2010) trains individual layers or modules one at a time to greedily optimize an objective.
Local losses are typically applied to different layers or residual stages, using common supervised and self-supervised loss formulations (Belilovsky et al., 2019; Nøkland & Eidnes, 2019;
Löwe et al., 2019; Belilovsky et al., 2020).
Xiong et al.
; Gomez et al.
(2020) proposed to use overlapped losses to reduce the impact of greedy learning.
Patel et al.
(2022) proposed to split a network into neuron groups.
Laskin et al.
(2020) applied greedy local learning on model parallelism training, and Wang et al.
(2021) proposed to add a local reconstruction loss for preserving information.
However, most local learning approaches proposed in the last decade rely on backprop to compute the weight updates within a local module.
One exception is the work of Nøkland & Eidnes (2019), which avoided backprop by using layerwise objectives coupled with a similarity loss or a feedback alignment mechanism.
Gated linear networks and their variants (Veness et al., 2017; 2021; Sezener et al., 2021) ask every neuron to make a prediction, and have shown interesting results on avoiding catastrophic forgetting.
From a theoretical perspective, Baldi & Sadowski (2016) provided insights and proofs on why local learning can be worse than global learning.
Asymmetric feedback weights.
Backprop relies on weight symmetry: the backward weights are the same as the forward weights.
Past research has looked at whether this constraint is necessary.
Lillicrap et al.
(2016) proposed feedback alignment (FA) that uses random and ﬁxed backward weights and found it can support error driven learning in neural networks.
Direct FA (Nøkland, 2016) uses a single backward layer to wire the loss function back to each layer.
There have also been methods that aim to explicitly update backward weights.
Recirculation (Hinton & McClelland, 1987) and target propagation (TP) (Bengio, 2014; Lee et al., 2015; Bartunov et al., 2018) use local reconstruction objective to learn separate forward and backward weights as approximate inverses of each other.
Ladder networks (Rasmus et al., 2015) found local reconstruction objectives and asymmetric weights can help achieve strong semi-supervised learning performance.
However, Bartunov et al.
(2018) reported both FA and TP algorithms do not scale to larger problems such as ImageNet, where their error rates are over 90%.
Liao et al.
; Xiao et al.
(2019) proposed sign symmetry (SS) where each backward connection weight share the same sign as the forward counterpart.
Akrout et al.
(2019) proposed weight mirroring and the modiﬁed Kolen-Pollack algorithm (Kolen & Pollack, 1994) to align forward and backward weights.
Woo et al.
(2021) proposed to update using activities from several layers below to avoid bidirectional connections.
Compared to these works, we circumvent the issue of weight symmetry, and more generally network symmetry, by using only reward (and the change rate thereof), instead of backward weights.
Biologically plausible perturbation learning.
Forward gradient is related to perturbation learning in the biology context.
Traditionally, neural plasticity learning rules focus on deriving weight updates as a function of the input and output activity of a neuron (Hebb, 1949; Widrow & Hoff, 1960; Oja, 1982; Bienenstock et al., 1982; Abbott & Nelson, 2000).
Weight perturbation learning (Jabri & Flower, 1992), on the other hand, is much more general as it permits any form of global reward (Schultz et al., 1997).
It was developed in both rated-based and spiking-based formuations (Xie & Seung, 1999; Seung, 2003).
Activity (or node) perturbation was proposed in shallow networks (Le Cun et al., 1988; Widrow & Lehr, 1990) and later in a spike-based continuous time network (Fiete &
Seung, 2006), where it was interpreted as the perturbation of the conductance of neurons.
Werfel et al.
(2003) showed that backprop has a faster convergence rate than perturbation learning, and activity perturbation wins over weight perturbation by another factor.
In our work, we show activity perturbation has lower gradient estimation variance compared to weight perturbation.
F ORWARD G RADIENT L EARNING
In this section, we review and establish the technical background for our learning algorithm.
We ﬁrst review the technique of forward-mode automatic differentiation (AD).
Second, we formulate two different types of perturbation in the weight space or activity space.
3. F ORWARD - MODE AUTOMATIC DIFFERENTIATION (AD)
Let f : R m (cid:55)→ R n .
The Jacobian of f , J f , is a matrix of size n × m .
Forward-mode AD computes the matrix-vector product J f v , where v ∈ R m .
It is deﬁned as the directional gradient along v evaluated at x :
[Equation]
For comparison, backprop, also known as reverse-mode AD, computes the vector-Jacobian product v J f , where v ∈ R n , which corresponds to the last term in the chain rule.
In contrast to reverse-mode
AD, forward-mode AD only requires one forward pass, which is augmented with the derivative information.
To compute the Jacobian vector product of a node in a computation graph, ﬁrst the input node will be augmented with v , which is the vector to be multiplied.
Then for other nodes, we send in a tuple of ( x , x (cid:48) ) as inputs and compute a tuple ( y , y (cid:48) ) as outputs, where x (cid:48) and y (cid:48) are the intermediate derivatives at node x and node y , i.e . y (cid:48)
=
d y d x x (cid:48) , and d y d x is the Jacobian between y and x .
In the JAX library (Bradbury et al., 2018), forward-mode AD is implemented as jax.jvp .
3. W EIGHT - PERTURBED FORWARD GRADIENT
Weight perturbation to generate weight updates was originally explored in (Barto et al., 1983; Xie & Seung, 1999; Seung, 2003).
Baydin et al.
(2022) uses the technique of forward-mode AD to implement weight perturbation, which is better than ﬁnite differences in terms of numerical stability.
Let w ij be the weight connection between unit i and j , and f be the loss function.
We can estimate the gradient by sampling a random matrix with iid elements v ij drawn from a zero-mean unit-variance
Gaussian distribution.
The estimator is
[Equation]
Intuitively, this estimator samples a random perturbation direction v ij and tests how it aligns with the true gradient ∇ w i (cid:48) j (cid:48) by using forward-mode to perform the dot product, and then multiplies the scalar alignment with the perturbation direction again.
Baydin et al.
(2022) referred this form of gradient estimation using forward-mode AD as “forward gradient”.
To distinguish with another form
[Table]
of perturbation we detail later, we refer this to as “weight-perturbed forward gradient”, or simply as “weight perturbation”.
3. A CTIVITY - PERTURBED FORWARD GRADIENT
An alternative to perturbing the weights is to instead perturb the activities, which can reduce the number of perturbation dimensions per example.
Activity perturbation was originally explored in Le Cun et al.
(1988); Widrow & Lehr (1990)
under restrictive assumptions.
Here, we introduce a general way to estimate gradients using activity perturbation.
It is potentially biologically plausible, since it could correspond to perturbation of the conductance in each neuron (Fiete & Seung, 2006).
Here, we focus on a discrete-time rate-based formulation for simplicity.
Let x i denote the activity of the i -th presynaptic neuron and z j denote that of the j -th post-synaptic neuron before the non-linear activation function, and u
j be the perturbation of z j .
The activity-perturbed forward gradient estimator is
[Equation]
(3) where the inner product between ∇ z and u is again computed by using forward-mode AD.
3. T HEORETICAL PROPERTIES
In this section we aim to analyze the expectation and variance properties of forward gradient estimators.
We focus our analysis on the gradient of one weight matrix { w ij } , but the conclusion holds for a network with many weight matrices too.
Table 1 summarizes the theoretical results 1 .
With a batch size of N , independent perturbation can achieve 1 /N reduction of variance, whereas shared perturbation has a constant variance term dominated by the squared gradient norm.
However, when performing independent weight perturbation, matrix multiplications cannot be batched because each example’s activation vector is multiplied with a different weight matrix.
By contrast, independent activity perturbation admits batched matrix multiplications.
Moreover, activity perturbation enjoys a factor of fan-in ( p ) times smaller variance compared to weight perturbation since the number of perturbed elements is the number of output units instead of the size of the whole weight matrix.
The only drawback of activity perturbation is the memory required for storage of intermediate activations, in exchange for a factor of Np reduction in variance.
However, for both activity and weight perturbation, the variance still grows with larger networks.
In Section 4 we will further reduce the variance by introducing local loss functions.
3. C ONTINUOUS - TIME RATE - BASED MODELS
Forward-mode AD can be viewed as computing the ﬁrst-order time derivative in a continuous-time physical system.
Suppose the tuples passed between nodes of the computation graph are ( x , ˙ x ) , where ˙ x is the change in x over time.
The computation is then the same as forward-mode AD.
For each node, ˙ y =
d y d x ˙ x , where d y d x is the Jacobian between the output and the input.
Note that in a physical system we don’t have to explicitly perform the differentiation operation by running two forward passes.
Instead the ﬁrst-order derivative information is readily available in the analog signal, and we only need to plug the output signal into a differentiator circuit.
The activity-perturbed learning rule for a continuous time system is thus ˙ w ij ∝ x i ˙
y j ˙ r , where x i is the pre-synaptic activity, and ˙
y j is the rate of change in the post-synaptic activity, which is the perturbation direction for a small period of time, and ˙ r is the rate of change of reward (or the negative loss).
The reward controls whether learning is Hebbian or anti-Hebbian.
Both Hinton et al.
(2007) and Bengio et al.
(2017) propose to use a product of pre-synaptic activity and the rate of change of postsynaptic activity.
However, they did not consider using the rate of change of reward as a modulator and instead relied on another set of feedback weights to communicate the error signal through inputs.
In contrast, we show that by broadcasting the rate of change of reward, we can actually bypass the weight transport problem.
ChannelMixing
TokenMixing
ChannelMixing +
TokenMixing
ChannelMixing
[Figure]
[Figure]
LN+FC+LN
[Figure]
P a t
[Figure]
h e s
[Figure]
LN+
[Figure]
LN [Figure] LN+A
[Figure]
N
[Figure]
C [Figure] G
N x HW [Figure] G … …
A
A CTIVATION SPARSITY AND NORMALIZATION FUNCTIONS
In networks with ReLU activations, we can leverage ReLU sparsity to achieve further variance reduction, because the inactivated units will have zero gradient and therefore we should not perturb these units, and set the perturbation to be zero.
Normalization layers are often added in deep neural networks after the linear layer.
To compute the correct gradient in activity perturbation, we also need to account for normalization in the weight update rule.
Since there is no backward weight connections, one option is to simply apply backprop on normalization layers.
However, we also found that it is also ﬁne to ignore the gradient of normalization layer when using layer normalization.
S CALING WITH L OCAL L OSSES
As we have explained in the previous section, perturbation learning can suffer from a curse of dimensionality: the variance grows with the number of perturbation dimensions, and in deep networks there are often millions of parameters changing at the same time.
One way to limit the number of learnable dimensions is to divide the network into submodules, each with a separate loss function.
In this section, we will explore several ways to increase the number of local losses to tame the variance.
1) Blockwise loss.
First, we will divide the network into modules in depth.
Each module consists of several layers.
At the end of each module, we compute a loss function, and that loss is used to update the parameters in that module.
This approach is equivalent of adding a “stop gradient” operator in between modules.
Such local greedy losses were previously explored in Belilovsky et al.
and
Löwe et al.
2) Patchwise loss.
Sensory input signals such as images have spatial dimensions.
We will apply a separate loss patchwise along these spatial dimensions.
In the Vision Transformer architecture (Vaswani et al., 2017; Dosovitskiy et al., 2021), each spatial token represents a patch in the image.
In modern deep networks, parameters in each spatial location are often shared to improve data efﬁciency and reduce memory bandwidth utilization.
Although naive weight sharing is not biologically plausible, we still consider shared weights in this work.
It may be possible to mimic the effect of weight sharing by adding knowledge distillation (Hinton et al., 2015) losses in between patches.
3) Groupwise loss.
Lastly, we turn to the channel dimension.
To create multiple losses, we split the channels into a number of groups, and each group is attached to a loss function (Patel et al., 2022).
To prevent groups from communicating between each other, channels are only connected to other channels within the same group.
A grouped linear layer is computed as z g,j = (cid:80) i w g,i,j x g,i , for individual group g .
Whereas previous work used channel groups to improve computational efﬁciency (Krizhevsky et al., 2012; Ioannou et al., 2017; Xie et al., 2017), in our work, adding groups contributes to the total number of losses and thus reduces variance.
Feature aggregators.
Naively applying losses separately to the spatial and channel dimensions leads to suboptimal performances, since each dimension contains only local information.
For losses of
G r o u p s
Stack + StopGradient
P a t c h e s
AvgPool + StopGradient
Avg Avg
Avg Avg
N x HW x G x C/G N x HW x G x C
N x HW x HW x G x C N x HW x G x C
[Figure]
Patches
Avg
N x C N x HW x C standard tasks such as classiﬁcation, the model needs a global view of the inputs to make a decision.
Standard architectures obtain this global view by performing global average pooling layer before the ﬁnal classiﬁcation layer.
We therefore explore strategies for aggregating information from other groups and spatial patches before the local loss function.
We would prefer to perform aggregation without reducing the total number of dimensions.
We thus propose a replicated design for feature aggregation, shown in Figure 3.
First, channel groups are copied and communicated to one another, but every group except the active group itself is masked with stop gradient so that other groups do not affect the forward gradient computation:
[Equation]
(4) where p and g index the patches and groups respectively.
Similarly, each spatial location is also copied, communicated, and masked and then averaged locally:
[Equation]
The output of feature aggregation is the same as that of the conventional global average pooling layer.
The difference is that here the loss is replicated and different patch groups are activated in each loss.
Learning objectives.
We consider the supervised classiﬁcation loss and the contrastive InfoNCE loss (van den Oord et al., 2018;
Chen et al., 2020), which are the two most commonly used losses in image representation learning.
For supervised classiﬁcation, we attach a shared linear layer (shared across p, g ) on top of the aggregated features for a cross entropy loss: L sp,g = − (cid:80) k
t k log softmax( W l x p,g ) k .
The loss is of the same value across each group and patch location.
For contrastive learning, the linear layer becomes a linear feature projector.
Suppose x (1) n and x (2) n are the two different views of the n -th example, the InfoNCE loss for contrastive learning is:
[Figure]
[Equation]
Note that we add a stop gradient operator on the second view.
It is usually unnecessary to add this stop gradient in the InfoNCE loss; however, we found that perturbationbased methods require a stop gradient and otherwise the loss will not go down.
This is likely because we share the perturbations on both views, and having the same perturbation will increase the dot product between the two views but is not desired from a representation learning perspective.
Figure 4 shows a comparison of the loss curves.
Non-shared perturbations also work but are worse than stop gradient.
[Table]
I MPLEMENTATION
Network architecture.
We propose the LocalMixer architecture that is more suitable for local learning.
It takes inspiration from MLPMixer (Tolstikhin et al., 2021), which consists of fully connected networks and residual blocks.
We leverage the fully connected networks so that each spatial patch performs computations without interfering with other patches, which is more compatible with our local learning objective.
An image is divided into non-overlapping patches ( i.e . tokens), and each block consists of token and channel mixing layers.
Figure 1 shows the high level architecture, and
Figure 2 shows the detailed diagram for one residual block.
We add a linear projector/classiﬁcation layer to attach a loss function at the end of each block.
The last layer always uses backprop to update weights.
For token mixing layers, we use one linear fully connected layer instead of an MLP, since we would like to make each block as shallow as possible.
Before the last channel mixing layer, features are reshaped into a number of groups, and the last layer is fully connected within each feature group.
Table 2 shows architectural details for the different sizes of models we investigate.
Normalization.
There are many ways of performing normalization within a neural network across different tensor dimensions (Krizhevsky et al., 2012; Ioffe & Szegedy, 2015; Ba et al., 2016;
Ren et al., 2017; Wu & He, 2018).
We opted for a local variant of layer normalization that normalizes within each local spatial patch of features (Ren et al., 2017).
For grouped linear layers, each group is normalized separately (Wu & He, 2018).
Empirically, we found such local normalization performs better on contrastive learning experiments and about the same as layer normalization on supervised experiments.
Local normalization is also more biologically plausible as it does not perform global communication.
Conventionally, normalization layers are placed after linear layers.
In MLPMixer (Tolstikhin et al., 2021), layer normalization is placed at the beginning of each residual block.
We found it is the best to place normalization before and after each linear layer, as shown in Figure 2.
Empirically this design choice does not make much difference for backprop, but it allows forward gradient learning to learn much faster and achieve lower training errors.
[Figure]
Efﬁcient implementation of replicated losses.
Due to the design of feature aggregation and replicated losses, a naïve implementation of groups can be very inefﬁcient in terms of both memory consumption and compute.
However, each spatial group actually computes the same aggregated feature and loss function.
This means that it is possible to share most of the computation across loss functions when performing both backprop and forward gradient.
We implemented our custom JAX JVP/VJP functions (Bradbury et al., 2018) and observed signiﬁcant memory savings and compute speed-ups for replicated losses, which would otherwise not be feasible to run on modern hardware.
The results are reported in Figure 5.
A code snippet is included in Appendix 12.
E XPERIMENTS
We compare our proposed algorithm to a set of alternatives: Backprop, Feedback Alignment and other global variants of Forward Gradient.
Backprop is a biologically implausible oracle, since it computes true gradients whereas we compute noisy gradients.
Feedback alignment computes approximate gradients by using a set of random backward weights.
We explain each method below.
1) Backprop (BP).
We include the standard backprop algorithm as well as its local variants.
Local Backprop (L-BP) adds local losses as proposed, but still permits gradient to ﬂow in an end-to-end fashion.
Local Greedy Backprop (LG-BP) in addition adds stop gradient operators in between blocks.
This is to provide a comparison to our methods by computing true local gradients.
LG-BP is similar in spirit to recent local learning algorithms (Belilovsky et al., 2019; Löwe et al., 2019).
[Table] [Table]
2) Feedback Alignment (FA).
The standard FA algorithm (Lillicrap et al., 2016) adds a set of random and ﬁxed backward weights.
We assume that the gradients to normalization layers and activation functions are known since they do not have weight connections.
Local Feedback Alignment ( L-FA ) adds local losses as proposed, but still permits error signals to ﬂow back.
Local Greedy
Feedback Alignment ( LG-FA ) adds a stop gradient to prevent error signals from ﬂowing back, similar to the
backprop-free algorithm in Nøkland & Eidnes (2019)
3) Forward Gradient (FG).
This family of methods comprises our proposed algorithm and related approaches.
Weight-perturbed forward gradient ( FG-W ) was proposed by Baydin et al.
In this paper, we propose the activity perturbation variant ( FG-A ).
We further add local objective functions, producing LG-FG-W and LG-FG-A , which stand for Local Greedy Forward Gradient
Weight/Activity-Perturbed.
For local perturbation to work, we have to add a stop gradient in between blocks so each perturbation has a single corresponding loss.
We expect LG-FG-A to achieve the best performance among other variants because it can leverage the variance reduction beneﬁt from both activity perturbation and local losses.
Datasets.
We use standard image classiﬁcation datasets to benchmark the learning algorithms.
MNIST (LeCun, 1998) contains 70,000 28 × 28 handwritten digit images of class 0-9.
CIFAR
(Krizhevsky et al., 2009) contains 60,000 32 × 32 natural images of 10 semantic classes.
ImageNet (Deng et al., 2009) contains 1.3 million natural images of 1000 classes, which we resized to 224 × 224.
For CIFAR-10 and ImageNet, we applied both supervised learning and contrastive learning.
For MNIST, we applied supervised learning only.
We designed different conﬁgurations of the LocalMixer architecture for each dataset, listed in Table 2.
Data augmentation.
For MNIST and CIFAR-10 supervised experiments, we do not apply data augmentation.
Data augmentation on ImageNet follows the open source implementation by Grill
[Figure] [Figure]
et al.
Because forward gradient suffers from variance, we apply weaker augmentations for contrastive learning experiments, increasing the area lower bound for random crops from 0.08 to 0.3-0.5.
We ﬁnd that this change has relatively little effect on the performance of backprop.
Main results.
Our main results are shown in Table 3 and Table 4.
In supervised experiments, there is almost no cost of introducing local greedy losses, and our local forward gradient method can match the test error of backprop on MNIST and CIFAR.
Note that LG-FG-A fails to overﬁt the training set to 0% error when trained without data augmentation.
This suggests that variance could still be an issue.
For CIFAR-10 contrastive learning, our method obtains an error rate approaching that obtained by backprop (26.81% vs. 17.53%), and most of the gap is due to greedy learning vs. gradient estimation (6.09% vs. 3.19%).
On ImageNet, we achieve reasonable performance compared to backprop (58.37% vs. 36.82% for supervised and 73.24% vs. 55.66% for contrastive).
However, we ﬁnd that the error due to greediness grows as the problem gets more complex and requires more layers to cooperate.
We signiﬁcantly outperform the FA family on ImageNet (by 25% for supervised and 10% for contrastive).
Interestingly, local greedy FA also performs better than global feedback alignment, which suggests that the beneﬁt of local learning transfers to other types of gradient approximation.
TP-based methods were evaluated in Bartunov et al.
and were found to be worse than FA on ImageNet.
In sum, although there is still some noticeable gap between our method and backprop, we have made a large stride forward compared to backprop-free algorithms.
More results are included in the Appendix 14.
Effect of local losses.
In Figure 6 we ablate the beneﬁt of placing local losses at different locations: blockwise, patchwise and groupwise.
A combination of all three is the strongest.
Global perturbation learning fails to learn as the accuracy is similar to initializing with random weights.
Effect of groups.
In Figure 7 we investigate the effect of different number of groups by showing the training curves.
Adding more groups bring signiﬁcant improvement to local perturbation learning in terms of lowering both training and test errors, but the effect vanishes around 8 channels / group.
C ONCLUSION
It is often believed that perturbation-based learning cannot scale to large and deep networks.
We show that this is to some extent true because the gradient estimation variance grows with the number of hidden dimensions for activity perturbation, and is even worse for shared weight perturbation.
But more optimistically, we show that a huge number of local greedy losses can help forward gradient learning scale much better.
We explored blockwise, patchwise, and groupwise local losses, and a combination of all three, with a total of a quarter of a million losses in one of the larger networks, performs the best.
Local activity-perturbed forward gradient performs better than previous backpropfree algorithms on larger networks.
The idea of local losses opens up opportunities for different loss designs and sheds light on the search for biologically plausible learning algorithms in the brain and alternative computing devices.
A CKNOWLEDGMENT
We thank Timothy Lillicrap for his helpful feedback on our earlier draft.
P ROOFS OF U NBIASEDNESS
In this section, we show the unbiasedness of g w ( w ij ) and g a ( w ij ) .
The ﬁrst proof was given by Baydin et al.
Proposition 1.
g w ( w ij ) is an unbiased gradient estimator if { v ij } are independent zero-mean uni-variance random variables (Baydin et al., 2022).
Proof.
We can rewrite the weight perturbation estimator as
[Equation]
Note that since each dimension of v is an independent zero-mean uni-variance random variable, E [ v ij ] = 0 , E [ v 2 ij ] =
Var [ v ij ] + E [ v ij ] 2 = 1 + 0 = 1 , and E [ v ij v i (cid:48) j (cid:48) ]
=
if ij (cid:54) = i (cid:48) j
[Equation]
Proposition 2.
g a ( w ij ) is an unbiased gradient estimator if { u j } are independent zero-mean univariance random variables.
Proof.
The true gradient to the weights ∇ w ij is the product between x j and ∇ z k .
Therefore, we can rewrite the weight perturbation estimator as
[Equation]
Since each dimension of u is an independent zero-mean uni-variance random variable, E [ u j ] = 0 , E [ u 2 j ] =
Var [ u j ] + E [ u j ] 2 = 1 + 0 = 1 , and E [ u j u j (cid:48) ]
=
if j (cid:54) = j (cid:48) .
[Equation]
[Equation]
P ROOFS OF V ARIANCES
We followed Wen et al.
and show that the variance of the gradient estimators can be decomposed.
Lemma 1.
The variance of the gradient estimator can be decomposed into three parts:
Var ( g ( w ij ) | x ) =
Z 1 + Z 2 + Z 3 , where Z 1 = 1 N V 1 Var x ( ∇ w ij | x ) , Z 2 = 1 N E x
[Var v ( g ( w ij ) | x )] ,
[Equation]
Proof.
By the law of total variance
[Equation]
The ﬁrst term comes from the gradient variance from data sampling
[Equation]
and it vanishes as batch size grows:
[Equation]
The second term comes from the gradient estimation variance:
[Equation]
Remark.
Z 2 is the variance of the gradient estimator in the deterministic case, and Z 3 measures the correlation between different gradient estimation within the batch.
The Z 3 is zero if the perturbations are independent, and non-zero if the perturbations are shared within the mini-batch.
Proposition 3.
Let p × q be the size of the weight matrix, the element-wise average variance of the weight perturbed gradient estimator with a batch size N is pq +2 N V + ( pq + 1) S if the perturbations are shared across the batch [Equation] and pq +2 N V + pq +1 N S if they are independent, where V is the element-wise average variance of the true gradient [Equation]
and S is the element-wise average squared gradient.
Proof.
We ﬁrst derive Z 2 .
[Equation]
(cid:54) =
ij
[Equation]
w i (cid:48) j (cid:48)
[Equation]
v [ v ij v i (cid:48)
j (cid:48) ] 
 
 (39) =
N E x [Equation] (cid:48)
[Equation]
(cid:48) (cid:54)
[Equation]
i (cid:48)
[Equation]
(cid:48)
[Equation] ij
[Equation]
i (cid:48) j (cid:48) 

[Equation] [Equation] (cid:48)
[Equation]
(cid:48) (cid:54) =
ij [Equation]
ij [Equation] i (cid:48)
[Equation]
(cid:48) · [Equation]
[Equation] w ij ·  (cid:88) i (cid:48) j (cid:48) (cid:54) =
ij
[Equation]
w i (cid:48) j (cid:48) ·  
 (41) =
N E x
[Equation]
v
[Equation]
(cid:48) [Equation] (cid:48) (cid:54) = ij
[Equation]
i (cid:48) j (cid:48) v ij
[Equation]
(cid:48)
[Equation]
(cid:48) [Equation] (42) = N E x
[Equation]
(cid:48) [Equation] (cid:48) [Equation]
(cid:48)
[Equation]
(cid:48)
[Equation]
(cid:48) ) [Equation] = N E x
[Equation] [Equation]
[Table]
[Equation] [Equation] [Equation] [Table]
[Equation]
[Table] [Equation] [Equation] (cid:48)
[Equation]
(cid:48)
[Equation]
 [Equation] [Equation]
[Equation]
Lastly, we average the variance across all weight dimensions: [Equation] [Equation]
[Equation]
N
[Equation]
 (cid:88)
n (cid:88) m (cid:54) = n ∇ w
[Equation]
ij 
 (100) =
N
[Equation]

[Equation]
(cid:88) x ( n ) [Equation] B (cid:88) x ( m ) ∈ B \{ x ( n ) } ∇ w ij | x ( n ) ∇ w ij | x ( m )
[Equation]
v (cid:104) v ( n ) [Equation] ij (cid:105)
[Equation]
v (cid:104) v ( m )2 ij (cid:105) − (101) N (cid:88) n (cid:88) m (cid:54) = n ∇ w
[Equation]
ij 
 (102) =
N
[Equation]
(cid:88) x ( n )
[Equation]
(cid:88) x ( m )
[Equation]
[Equation]
( n ) } ∇ w ij | x ( n )
[Equation]
w ij | x ( m ) 
 − 1 N
[Equation]
 (cid:88)
n (cid:88) m (cid:54) = n ∇ w [Equation]
ij 
 (103) =
N (cid:88) n (cid:88) m (cid:54) = n E x [ ∇ w ij | x ]
[Equation]
  − 1 N
[Equation]
 (cid:88) n (cid:88) m
(cid:54) = n ∇ w 2 ij 

(104) =0 .
Then the average variance becomes:
[Equation]
Proposition 4.
Let p × q be the size of the weight matrix
[Equation]
the element-wise average variance of the activity perturbed gradient estimator with a batch size N is q +2 N V
+ ( q + 1) S if the perturbations are shared across the batch
[Equation]
and q +2 N V + q +1 N S if they are independent, where V is the element-wise average variance of the true gradient
[Equation]
and S is the element-wise average squared gradient
[Equation]
Proof.
[Equation]

[Equation]
(114) =
N
[Equation]
[Equation]
(116) =
N
[Equation]
[Equation]
118) =
N E
[Equation]

[Equation]

 +
[Equation]
119) 2 (cid:88)
[Equation] [Equation] [Equation] Z 3 is nonzero if the perturbations are shared within a batch.
Assuming that the perturbations are shared,
[Equation] [Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation] |
[Equation]
n
[Equation]

 
[Equation]
N
[Equation]

[Equation]
n
[Equation]
n
[Equation]
(146) =
[Equation]
[Equation]

[Equation]
n
[Equation]
[Equation]
[Equation]
n
[Equation]
m
[Equation]
n
[Equation]
  (150) [Equation] Then we compute the average variance across all weight dimensions (for shared perturbation):
[Equation]
If the perturbations are independent [Equation] we show that Z 3 is [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
N
[Equation]
 [Equation] = n ∇ w ij | x ( n ) ∇ w ij | x ( m )   
 (175) = N 2 E B   (cid:88) x ( n ) ∈ B (cid:88)
x ( m ) ∈ B \{ x ( n ) }
E u (cid:104) ∇ w ij | x ( n ) ∇ w ij | x ( m ) u ( n )2 j u ( m )2 j + (176) (cid:88) j (cid:48) (cid:54) =
j ∇ w ij (cid:48)
| x ( m ) ∇ w ij (cid:48)
| x ( n ) u ( m ) j (cid:48) u ( n ) j (cid:48) u
( m ) j u ( n ) j   
 − (177)
N  E x ( n ) E x ( m )  (cid:88) n (cid:88) m (cid:54) = n ∇ w ij | x ( n ) ∇ w ij | x ( m ) 
 
 (178) =
N 2 E B   (cid:88) x ( n ) ∈ B (cid:88)
x ( m ) ∈ B \{ x ( n ) }
E u (cid:104) ∇ w ij | x ( n ) ∇ w ij | x ( m ) u ( n )2 j u ( m )2 j (cid:105) − (179)
N (cid:88) n (cid:88) m (cid:54) =
n ∇ w 2 ij 
 (180) = N 2 E B   (cid:88) x ( n ) ∈ B (cid:88)
x ( m ) ∈ B \{ x ( n ) } ∇ w ij | x ( n ) ∇ w
[Equation]
| x ( m ) E u (cid:104) u ( n )2 j (cid:105)
[Equation]
[Equation]
[Equation]
Then the average variance becomes:
[Equation]
[Figure]
N UMERICAL S IMULATION OF V ARIANCES
In Figure 8, we ran numerical simulation experiments to verify our analytical variance properties.
We used a multi-layer network with 4 input units, 4 hidden units, 1 output unit, a tanh activation function, and the mean squared error loss.
We varied the batch size ( N ) between 1 and 4096.
We tested the gradient estimator of the ﬁrst layer weights using 5000 random samples.
We also calculated the theoretical variance by applying the gradient norm and gradient variance constants found by backprop, from 5000 mini-batch true gradients.
We then ﬁxed the batch size to be 4 and vary the number of input units ( p , fan in) and the number of hidden units ( q , fan out) between 1 and 256.
The theoretical variance for backprop was only computed for the batch size experiment since it is an inverse relationship ( 1 N ), but for fan in and fan out, we do not aim to analyze the theoretical variances here.
“wt perturb” stands for weight perturbation with shared noise; “ind wt perturb” stands for weight perturbation with independent noise; and “act perturb” stands for activity perturbation with independent noise.
Note that indepedent weight perturbation is much more costly to compute in neural networks.
As shown in the ﬁgure, the empirical variances match very well with our theoretical predictions.
T RAINING D ETAILS
Here we provide more training details.
MNIST.
We use a batch size of 128, and the SGD optimizer with learning rate 0.01 and momentum 0.9 for a total of 1000 epochs with no data augmentation and a linear learning rate decay schedule.
CIFAR-10.
For the supervised experiments, we use a batch size of 128 and the SGD optimizer with learning rate 0.01 and momentum 0.9 for a total of 200 epochs with no data augmentation and a linear learning rate decay schedule.
For the contrastive M/8 experiments, we use a batch size ofand the SGD optimizer with learning rate 1.0 and momentum 0.9 for a total of 1000 epochs with
BYOL data augmentation using area crop lower bound to be 0.5 and a cosine decay schedule with a warm-up period of 10 epochs.
For the contrastive L/8 experiments, we use a batch size ofand the SGD optimizer with learning rate 4.0 and momentum 0.9 for a total of 1000 epochs with
BYOL data augmentation (Grill et al., 2020) using area crop lower bound to be 0.3 and a cosine decay schedule with a warm-up period of 10 epochs.
ImageNet.
For the supervised experiments, we use a batch size of 256 and the SGD optimizer with learning rate 0.05 and momentum 0.9 for a total of 120 epochs with BYOL data augmentation (Grill et al., 2020) using area crop lower bound to be 0.3 and a cosine learning rate decay schedule with a warm-up period of 10 epochs.
For the contrastive experiments, we use a batch size of 2048 and the
LARS optimizer with learning rate 0.1 and momentum 0.9 for a total of 800 epochs with BYOL data augmentation (Grill et al., 2020) using area crop lower bound to be 0.08 and a cosine learning rate decay schedule with a warm-up period of 10 epochs. F USED
[Equation]
/VJP D ETAILS
In Algorithm 1, we provide a JAX code snippet implementing fused operators for the supervised cross entropy loss.
“Fused” here means that we package several operations into one function.
In the supervised cross entropy loss, we combine average pooling, channel concatenation, a linear classiﬁer layer, and cross entropy all together.
Key steps and expected tensor shapes are annotated in the comments.
The fused InfoNCE loss implementation will be included in our full code release.
L OCAL M IXER A RCHITECTURE
In Algorithm 2, we provide code in JAX style that implements our proposed LocalMixer architecture.
A DDITIONAL R ESULTS
In this section we provide additional experimental results.
Normalization scheme.
Table 5 compares different normalization schemes.
Layer normalization (LN) is often better than batch normalization (BN) on our mixer architecture.
Local LN is better on contrastive learning experiments and achieves lower error rates using forward gradient learning.
Although in our main paper, backprop were used in normalization layers, backprop is not necessary for Local LN, c.f . “NG” (No Gradient) columns in Table 5.
Place of normalization.
We investigate the places where we add normalization layers.
Traditionally, normalization is added after linear layers.
In MLPMixer, LN is added at the beginning of each block.
With our forward gradient learning, it is now a question of which location is the optimal design.
Adding it after the linear layer has the advantage of shaping the activations to be more well behaved, which can make perturbation learning more effective.
Adding it before the linear layer can also help reduce the variance since the inputs always get multiplied with the gradient of the output activity.
The results are reported in Table 6.
Adding normalization both before and after the linear layer helps forward gradient to achieve lower training errors.
While this could result in some overﬁtting on supervised learning, it is good for contrastive learning which needs more model capacity.
This is reasonable as forward gradient introduce a lot of variances, and more normalization layers help achieve better training performance.
[Table]
import jax jax numpy def
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
return
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
t
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Effect of groups.
We provide additional results summarizing the training and test performance of adding more groups in Figure 9.
Backprop and local greedy backprop always achieve zero training error with increasing number of groups on CIFAR-10 supervised, but adding groups has a signiﬁcant beneﬁt lowering training errors for forward gradient.
This suggests that the main opponent here is still the gradient estimation variance, and lowering training errors can generally make test errors lower too; on the other hand adding groups have negligible effect on backprop.
For contrastive learning, here the task requires higher model capacity, and adding groups effectively reduce the model capacity by introducing sparsity in the weight matrix.
As a result, we observe a slight drop of less than 5% performance on both backprop and local greedy backprop.
By contrast, forward gradient gains over 10% of performance by adding 16 groups.