D EEP R EWIRING : T RAINING VERY SPARSE DEEP NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neuromorphic hardware tends to pose limits on the connectivity of deep networksthat one can run on them.
But also generic hardware and software implementations of deep learning run more efﬁciently on sparse networks.
Several methodsexist for pruning connections of a neural network after it was trained without connectivity constraints.
We present an algorithm, DEEP R, that enables us to traindirectly a sparsely connected neural network.
DEEP R automatically rewires thenetwork during supervised training so that connections are there where they aremost needed for the task, while its total number is all the time strictly bounded.
We demonstrate that DEEP R can be used to train very sparse feedforward andrecurrent neural networks on standard benchmark tasks with just a minor loss inperformance.
DEEP R is based on a rigorous theoretical foundation that viewsrewiring as stochastic sampling of network conﬁgurations from a posterior.
I NTRODUCTION
Network connectivity is one of the main determinants for whether a neural network can be efﬁciently implemented in hardware or simulated in software.
For example, it is mentioned in Jouppiet al.
(2017) that in Google’s tensor processing units (TPUs), weights do not normally ﬁt in on-chipmemory for neural network applications despite the small 8 bit weight precision on TPUs.
Memoryis also the bottleneck in terms of energy consumption in TPUs and FPGAs (Han et al., 2017; Iandolaet al., 2016).
For example, for an implementation of a long short term memory network (LSTM),memory reference consumes more than two orders of magnitude more energy than ALU operations(Han et al., 2017).
The situation is even more critical in neuromorphic hardware, where either hardupper bounds on network connectivity are unavoidable (Schemmel et al., 2010; Merolla et al.
, 2014)or fast on-chip memory of local processing cores is severely limited, for example the 96 MByte local memory of cores in the SpiNNaker system (Furber et al., 2014).
This implementation bottleneckwill become even more severe in future applications of deep learning when the number of neuronsin layers will increase, causing a quadratic growth in the number of connections between them.
Evolution has apparently faced a similar problem when evolving large neuronal systems such as thehuman brain, given that the brain volume is dominated by white matter, i.e., by connections betweenneurons.
The solution found by evolution is quite convincing.
Synaptic connectivity in the brainis highly dynamic in the sense that new synapses are constantly rewired, especially during learning(Holtmaat et al., 2005; Stettler et al., 2006; Chambers & Rumpel, 2017).
In other words, rewiring isan integral part of the learning algorithms in the brain, rather than a separate process.
We are not aware of previous methods for simultaneous training and rewiring in artiﬁcial neuralnetworks, so that they are able to stay within a strict bound on the total number of connectionsthroughout the learning process.
There are however several heuristic methods for pruning a largernetwork (Han et al., 2015b;a; Collins & Kohli, 2014; Yang et al., 2015; Srinivas & Babu, 2015), thatis, the network is ﬁrst trained to convergence, and network connections and / or neurons are prunedonly subsequently.
These methods are useful for downloading a trained network on neuromorphichardware, but not for on-chip training.
A number of methods have been proposed that are capableof reducing connectivity during training (Collins & Kohli, 2014; Jin et al., 2016; Narang et al.,2017).
However, these algorithms usually start out with full connectivity.
Hence, besides reducingcomputational demands only partially, they cannot be applied when computational resources (suchas memory) is bounded throughout training.
Inspired by experimental ﬁndings on rewiring in the brain, we propose in this article deep rewiring(DEEP R), an algorithm that makes it possible to train deep neural networks under strict connectivityconstraints.
In contrast to many previous pruning approaches that were based on heuristic arguments,
DEEP R is embedded in a thorough theoretical framework.
DEEP R is conceptually different fromstandard gradient descent algorithms in two respects.
First, each connection has a predeﬁned sign.
Speciﬁcally, we assign to each connection k
a connection parameter θ k and a constant sign s k ∈{− 1 , 1 } .
For non-negative θ k , the corresponding network weight is given by w k = s k θ k .
In standardbackprop, when the absolute value of a weight is moved through 0 , it becomes a weight with theopposite sign.
In contrast, in DEEP R a connection vanishes in this case ( w k = 0 ), and a randomlydrawn other connection is tried out by the algorithm.
Second, in DEEP R, gradient descent iscombined with a random walk in parameter space.
This modiﬁcation leads to important functionaldifferences.
In fact, our theoretical analysis relates DEEP R to a process that samples network conﬁgurations (i.e.,network connectivity, weights, and biases) from the posterior distribution of conﬁgurations, that is,the distribution that combines the data likelihood and the connectivity prior in a Bayes optimalmanner.
As a result, the algorithm continues to rewire connections even when the performance hasconverged.
We show that this feature enables DEEP R to adapt the network connectivity structureonline when the task demands are drifting.
We show on several benchmark tasks that with DEEP R, the connectivity of several deep architectures — fully connected deep networks, convolutional nets, and recurrent networks (LSTMs) — canbe constrained to be extremely sparse throughout training with a marginal drop in performance.
Inone example, a standard feed forward network trained on the MNIST dataset, we achieved goodperformance with 1 % of the connectivity of the fully connected counterpart.
We show that DEEP
R reaches a similar performance level as state-of-the-art pruning algorithms where training startswith the full connectivity matrix.
If the target connectivity is very sparse (a few percent of the fullconnectivity), DEEP R outperformed these pruning algorithms.
R EWIRING IN DEEP NEURAL NETWORKS
Stochastic gradient descent (SGD) and its modern variants (Kingma & Ba, 2014; Tieleman & Hinton, 2012) implemented through the Error Backpropagation algorithm is the dominant learningparadigm of contemporary deep learning applications.
For a given list of network inputs X andtarget network outputs Y ∗ , gradient descent iteratively moves the parameter vector θ in the direction of the negative gradient of an error function E X , Y ∗ ( θ ) such that a local minimum of E X , Y ∗ ( θ )is eventually reached.
A more general view on neural network training is provided by a probabilistic interpretation of thelearning problem (Bishop, 2006; Neal, 1992).
In this probabilistic learning framework, the deterministic network output is interpreted as deﬁning a probability distribution p N ( Y | X , θ ) over outputs
Y for the given input X and the given network parameters θ .
The goal of training is then to ﬁnd parameters that maximize the likelihood p N ( Y ∗ | X , θ ) of the training targets under this model (maximum likelihood learning).
Training can again be performed by gradient descent on an equivalent error function that is usually given by the negative log-likelihood E X , Y ∗ ( θ )
= − log p N ( Y ∗ |
X , θ ) .
Going one step further in this reasoning, a full Bayesian treatment adds prior beliefs about the network parameters through a prior distribution p S ( θ ) (we term this distribution the structural prior for reasons that will become clear below) over parameter values θ and the training goal is formulated via the posterior distribution over parameters θ .
The training goal that we consider in this article is to produce sample parameter vectors which have a high probability under the posterior distribution p ∗ ( θ | X , Y ∗ )
∝ p S ( θ ) · p N ( Y ∗ | X , θ ) .
More generally, we are interested in a target distribution p ∗ ( θ ) that is a tempered version of the posterior, i.e., we deﬁne the training goal as follows:
[Equation]
sample parameter vectors θ with high probability
[Equation]
where Z is a normalizing constant and T is a temperature parameter.
For T = 1 we recover the posterior distribution, for T > 1 the peaks of the posterior are ﬂattened, and for T < 1 the distributionis sharpened, leading to higher probabilities for parameter settings with better performance.
The DEEP R algorithm:
In many situations, network connectivity is strictly limited during training, for instance because of hardware memory limitations.
Then the limiting factor for a trainingalgorithm is the maximal connectivity ever needed during training.
DEEP R guaranties such a hardlimit.
DEEP R achieves the learning goal (1) on network conﬁgurations , that is, it not only samples the network weights and biases, but also the connectivity under the given constraints.
This isachieved by introducing the following mapping from network parameters θ to network weights w :
A connection parameter θ k and a constant sign s k ∈ {− 1 , 1 } are assigned to each connection k .
Ifθ k is negative, we say that the connection k is dormant , and the corresponding weight is w k = 0 .
Otherwise, the connection is considered active , and the corresponding weight is w k
=
s k θ k .
Hence,each θ k encodes (a) whether the connection is active in the network, and (b) the weight of theconnection if it is active.
Note that we use here a single index k for each connection / weight insteadof the more usual double index that deﬁnes the sending and receiving neuron.
This connectioncentric indexing is more natural for our rewiring algorithms where the connections are in the focusrather than the neurons.
Using this mapping, sampling from the posterior over θ is equivalent tosampling from the posterior over network conﬁgurations, that is, the network connectivity structureand the network weights.1 for i in [1 , N iterations ] do for all active connections k
( θ k ≥ 0) do
[Equation]
; if θ k < 0 then set connection k dormant ; end while number of active connections lower than K do select a dormant connection with uniform probability and activate it; θ k (cid:48) ← end 10 endAlgorithm of unit variance independently for each active and each update step.
Note that the gradient of the error
E X , Y ∗ ( θ ) is computed by backpropagation over a mini-batch in practice.
DEEP R is deﬁned in Algorithm 1.
Gradient updates are performed only on parameters of activeconnections (line 3).
The derivatives of the error function ∂∂θ k E X , Y ∗ ( θ ) can be computed in theusual way, most commonly with the backpropagation algorithm.
Since we consider only classiﬁcation problems in this article, we used the cross-entropy error for the experiments in this article.
Thethird term in line 3 ( − ηα ) is an (cid:96) 1 regularization term, but other regularizers could be used as well.
A conceptual difference to gradient descent is introduced via the last term in line 3.
Here, noise 108 √ 2 ηT ν k is added to the update, where the temperature parameter T controls the amount of noiseand ν k is sampled from a zero-mean Gaussian of unit variance independently for each parameter andeach update step.
The last term alone would implement a random walk in parameter space.
Hence,the whole line 3 of the algorithm implements a combination of gradient descent on the regularizederror function with a random walk.
Our theoretical analysis shows that this random walk behaviorhas an important functional consequence, see the paragraph after the next for a discussion on thetheoretical properties of DEEP R.
The rewiring aspect of the algorithm is captured in lines 4 and 6–9 in Algorithm (1).
Whenevera parameter θ k becomes smaller than 0 , the connection is set dormant, i.e., it is deleted from thenetwork and no longer considered for updates (line 4).
For each connection that was set to thedormant state, a new connection k (cid:48) is chosen randomly from the uniform distribution over dormantconnections, k (cid:48) is activated and its parameter is initialized to 0 .
This rewiring strategy (a) ensuresthat exactly K connections are active at any time during training (one initializes the network with
K active connections), and (b) that dormant connections do not need any computational demandsexcept for drawing connections to be activated.
Note that for sparse networks, it is efﬁcient to keeponly a list of active connections and none for the dormant connections.
Then, one can efﬁcientlydraw connections from the whole set of possible connections and reject those that are already active.
We provide a theoretical analysis of this algorithm in Section 4.
Since the updates are stochastic,one cannot predict the exact network conﬁguration after any number of updates.
However, one can
[Figure]
describe the evolution of the distribution of network conﬁgurations.
Our analysis in Section 4 showsthat under some simplifying assumptions, this distribution converges to a unique stationary distribution, that is, a distribution that will no longer change under the updates.
Further, the stationarydistribution is given by the target distribution of network conﬁgurations (1) where the prior p S ( θ )is given by a uniform distribution over connectivity matrices that fulﬁll the connectivity constraintof K active connections and an exponential prior on the weights of these active connections.
E XPERIMENTS
Rewiring in fully connected and in convolutional networks: We ﬁrst tested the performanceof DEEP R on MNIST and CIFAR-10.
For MNIST, we considered a fully connected feed-forwardnetwork used in Han et al.
(2015b) to benchmark pruning algorithms.
It has two hidden layers of300 and 100 neurons respectively and a 10-fold softmax output layer.
On the CIFAR-10 dataset, weused a convolutional neural network (CNN) with two convolutional followed by two fully connectedlayers.
For reproducibility purposes the network architecture and all parameters of this CNN weretaken from the ofﬁcial tutorial of Tensorﬂow.
On CIFAR-10, we used a decreasing learning rateand a cooling schedule to reduce the temperature parameter T over iterations (see Appendix A fordetails on all experiments).
For each task, we performed four training sessions.
First, we trained a network with DEEP R. Inthe CNN, the ﬁrst convolutional layer was kept fully connected while we allowed rewiring of thesecond convolutional layer.
Second, we tested another algorithm, soft-DEEP R, which is a simpliﬁedversion of DEEP R that does however not guarantee a strict connectivity constraint (see Section 4for a description).
Third, we trained a network in the standard manner without any rewiring or a u d i o i n p u t s p
e c t r o g r a m "tofu is made from processed soybeans"
TIMIT
A
B        p h
o n e m e e rr o r r
a t e ( % ) highest connectivity met during training (%)
p h
o n e m e e rr o r r
a t e ( % )
[Figure]
pruning to obtain a baseline performance.
Finally, we trained a network with a connectivity thatwas randomly chosen before training and kept ﬁxed during the optimization.
The connectivity washowever not completely random.
Rather each layer received a number of connections that was thesame as the number found by soft-DEEP R.
The performance of this network is expected to be muchbetter than a network where all layers are treated equally.
Fig.
1 shows the performance of these algorithms on MNIST (panel A) and on CIFAR-10 (panel
B).
DEEP R reaches a classiﬁcation accuracy of 96 . 2 % when constrained to 1 .
3 % connectivity.
To evaluate precisely the accuracy that is reachable with 1 . 0 % connectivity, we did an additionalexperiment where we doubled the number of training epochs.
DEEP R reached a classiﬁcation accuracy of 96 .
3% (less than 2 % drop in comparison to the fully connected baseline).
Training on ﬁxedrandom connectivity performed surprisingly well for connectivities around 10 %, possibly due to thelarge redundancy in the MNIST images.
Soft-DEEP R does not guarantee a strict upper bound onthe network connectivity.
When considering the maximum connectivity ever seen during training,soft-DEEP R performed consistently worse than DEEP R for networks where this maximum connectivity was low.
On CIFAR-10, classiﬁcation accuracy of DEEP R was 84.1 % at a connectivitylevel of 5 %.
The performance of DEEP R at 20 % connectivity was close to the performance of thefully connected network.
To study the rewiring properties of DEEP R, we monitored the number of newly activated connections per iteration (i.e., connections that changed their status from dormant to active in that iteration).
We found that after an initial transient, the number of newly activated connections converged to astable value and remained stable even after network performance has converged, see Appendix B.
Rewiring in recurrent neural networks:
In order to test the generality of our rewiring approach,we also considered the training of recurrent neural networks with backpropagation through time(BPTT).
Recurrent networks are quite different from their feed forward counterparts in terms of theirdynamics.
In particular, they are potentially unstable due to recurrent loops in inference and trainingsignals.
As a test bed, we considered an LSTM network trained on the TIMIT data set.
In ourrewiring algorithms, all connections were potentially available for rewiring, including connectionsto gating units.
From the TIMIT audio data, MFCC coefﬁcients and their temporal derivatives werecomputed and fed into a bi-directional LSTMs with a single recurrent layer of 200 cells followed bya softmax to generate the phoneme likelihood (Graves & Schmidhuber, 2005), see Appendix A.
We considered as ﬁrst baseline a fully connected LSTM with standard BPTT without regularizationas the training algorithm.
This algorithm performed similar as the one described in Greff et al.(2017).
It turned out however that performance could be signiﬁcantly improved by including a regularizer in the training objective.
We therefore considered the same setup with (cid:96) 2 regularization(cross-validated).
This setup achieved a phoneme error rate of 28 . 3 %.
We note that better resultshave been reported in the literature using the CTC cost function and deeper networks (Graves et al.,2013).
For the sake of easy comparison however, we sticked here to the much simpler setup with amedium-sized network and the standard cross-entropy error function.
We found that connectivity can be reduced signiﬁcantly in this setup with our algorithms, see Fig. 2.
Both algorithms, DEEP R and soft-DEEP R, performed even slightly better than the fully connectedbaseline at connectivities around 10 %, probably due to generalization issues.
DEEP R outperformedsoft-DEEP R at very low connectivities and it outperformed BPTT with ﬁxed random connectivityconsistently at any connectivity level considered.
Comparison to algorithms that cannot be run on very sparse networks: We wondered howmuch performance is lost when a strict connectivity constraint has to be taken into account during
a cc u r a c y ( % ) epoch number
A
B
C c o rr e l a t i o n ( % ) epoch number     weight matrices similarity between consecutive tasks epoch number  9 c o rr e l a t i o n ( % )  neural response similarity between consecutive tasks training as compared to pruning algorithms that only achieve sparse networks after training.
Tothis end, we compared the performance of DEEP R and soft-DEEP R to recently proposed pruning algorithms: (cid:96) 1 -shrinkage (Tibshirani, 1996; Collins & Kohli, 2014) and the pruning algorithmproposed by Han et al. (2015b).
(cid:96) 1 -shrinkage uses simple (cid:96) 1 -norm regularization and ﬁnds networksolutions with a connectivity that is comparable to the state of the art (Collins & Kohli, 2014; Yuet al., 2012).
We chose this one since it is relatively close to DEEP R with the difference that it doesnot implement rewiring.
The pruning algorithm from Han et al.
(2015b) is more complex and uses a projection of network weights on a (cid:96) 0 constraint.
Both algorithms prune connections starting from the fully connected network.
The hyper-parameters such as learning rate, layer size, and weightdecay coefﬁcients were kept the same in all experiments.
We validated by an extensive parametersearch that these settings were good settings for the comparison algorithms, see Appendix A.
Results for the same setups as considered above (MNIST, CIFAR-10, TIMIT) are shown in Fig.
Despite the strict connectivity constraints, DEEP R and soft-DEEP R performed slightly better thanthe unconstrained pruning algorithms on CIFAR-10 and TIMIT at all connectivity levels considered.
On MNIST, pruning was slightly better for larger connectivities.
On MNIST and TIMIT, pruningand (cid:96) 1 -shrinkage failed completely for very low connectivities while rewiring with DEEP R or softDEEP R still produced reasonable networks in this case.
One interesting observation can be made for the error rate evolution of the LSTM on TIMIT(Fig.
3D).
Here, both (cid:96) 1 -shrinkage and pruning induced large sudden increases of the error rate,possibly due to instabilities induced by parameter changes in the recurrent network.
In contrast, weobserved only small glitches of this type in DEEP R.
This indicates that sparsiﬁcation of networkconnectivity is harder in recurrent networks due to potential instabilities, and that DEEP R is bettersuited to avoid such instabilities.
The reason for this advantage of DEEP R is however not clear.
Transfer learning is supported by DEEP R: If the temperature parameter T is kept constantduring training, the proposed rewiring algorithms do not converge to a static solution but explorecontinuously the posterior distribution of network conﬁgurations.
As a consequence, rewiring isexpected to adapt to changes in the task in an on line manner.
If the task demands change in anonline learning setup, one may hope that a transfer of invariant aspects of the tasks occurs such thatthese aspects can be utilized for faster convergence on later tasks (transfer learning).
To verify thishypothesis, we performed one experiment on the MNIST dataset where the class to which each output neuron should respond to was changed after each training epoch (class-shufﬂed MNIST task).
Fig.
4A shows the performance of a network trained with DEEP R in the class-shufﬂed MNISTtask.
One can observe that performance recovered after each shufﬂing of the target classes.
Moreimportantly, we found a clear trend of increasing classiﬁcation accuracy even across shufﬂes.
Thisindicates a form of transfer learning in the network such that information about the previous tasks(i.e.
, the previous target-shufﬂed MNIST instances) was preserved in the network and utilized inthe following instances.
We hypothesized for the reason of this transfer that early layers developedfeatures that were invariant to the target shufﬂing and did not need to be re-learned in later taskinstances.
To verify this hypothesis, we computed the following two quantities.
First, in order to quantify the speed of parameter dynamics in different layers, we computed the correlation betweenthe layer weight matrices of two subsequent training epoch (Fig. 4B).
Second, in order to quantifythe speed of change of network dynamics in different layers, we computed the correlation betweenthe neuron outputs of a layer in subsequent epochs (Fig. 4C).
We found that the correlation betweenweights and layer outputs increased across training epochs and were signiﬁcantly larger in early layers.
This supports the hypothesis that early network layers learned features invariant to the shufﬂedcoding convention of the output layer.
C ONVERGENCE PROPERTIES OF DEEP R AND SOFT -DEEP R
The theoretical analysis of DEEP R is somewhat involved due to the implemented hard constraints.
We therefore ﬁrst introduce and discuss here another algorithm, soft-DEEP R where the theoreticaltreatment of convergence is more straight forward.
In contrast to standard gradient-based algorithms,this convergence is not a convergence to a particular parameter vector, but a convergence to the targetdistribution over network conﬁgurations Eq.
, N iterations ] of the soft-DEEP R algorithm min < 0 is a constant that deﬁnes a lower boundary for negative θ k s.
Convergence properties of soft-DEEP R: The soft-DEEP R algorithm is given in Algorithm 2.
Note that the updates for active connections are the same as for DEEP R (line 3).
Also the mappingfrom parameters θ k to weights w k is the same as in DEEP R.
The main conceptual difference to
DEEP R is that connection parameters continue their random walk when dormant (line 7).
Due tothis random walk, connections will be re-activated at random times when they cross zero.
Therefore,soft-DEEP R does not impose a hard constraint on network connectivity but rather uses the (cid:96) 1 normregularization to impose a soft-constraint.
Since dormant connections have to be simulated, this algorithm is computationally inefﬁcient forsparse networks.
An approximation could be used where silent connections are re-activated at aconstant rate, leading to an algorithm very similar to DEEP R. DEEP R adds to that the additionalfeature of a strict connectivity constraint.
The central result for soft-DEEP R has been proven in the context of spiking neural networks in(Kappel
et al., 2015) in order to understand rewiring in the brain from a functional perspective.
Thesame theory however also applies to standard deep neural networks.
To be able to apply standardmathematical tools, we consider parameter dynamics in continuous time.
In particular, consider thefollowing stochastic differential equation (SDE)
[Equation]
where β is the equivalent to the learning rate and ∂∂θ k log p ∗ ( θ | X , Y ∗ ) (cid:12)(cid:12)(cid:12) θ t denotes the gradient ofthe log parameter posterior evaluated at the parameter vector θ t at time t .
The term d W k denotesthe inﬁnitesimal updates of a standard Wiener process.
This SDE describes gradient ascent on thelog posterior combined with a random walk in parameter space.
We show in Appendix C that theunique stationary distribution of this parameter dynamics is given by p ∗ ( θ )
=
1 Z p ∗ ( θ | X , Y ∗ ) 1 T .
Since we considered classiﬁcation tasks in this article, we interpret the network output as a multinomial distribution over class labels.
Then, the derivative of the log likelihood is equivalent to thederivative of the negative cross-entropy error.
Together with an (cid:96) 1 regularization term for the prior,and after discretization of time, we obtain the update of line 3 in Algorithm 2 for non-negative parameters.
For negative parameters, the ﬁrst term in Eq.
(2) vanishes since the network weight isconstant zero there.
This leads to the update in line 7.
Note that we introduced a reﬂecting boundaryat θ min < 0 in the practical algorithm to avoid divergence of parameters (line 8).
Convergence properties of DEEP R: A detailed analysis of the stochastic process that underliesthe algorithm is provided in Appendix D.
Here we summarize the main ﬁndings of this analysis.
Each iteration of DEEP R in Algorithm 1 consists of two parts: In the ﬁrst part (lines 2-5) allconnections that are currently active are advanced, while keeping the other parameters at 0.
In thesecond part (lines 6-9) the number of connections that became dormant during the ﬁrst step arerandomly replenished.
To describe the connectivity constraint over the K connections we introduce the binary constraintvector c ∈ { 0 , 1 } K which represents the set of active connections, i.e., element c k of c is 1 ifconnection k is allowed to be active and zero else.
In Theorem 2 of Appendix D, we link DEEP Rto a theoretical framework which realizes a sampling process that simultaneously samples fromthe posterior distribution (1) under the constraint c of maintaining M active connections and theconstraint vector from a prior distribution p ( c ) , i.e., from the stationary distribution given by thejoint probability
[Equation]
(3) where p ( c ) is a uniform prior over all possible connectivities with M active connections andp ∗ ( θ | c , X , Y ∗ ) is the posterior for the given constraint vector c .
We conclude that DEEP R solvesa constraint optimization problem by sampling parameter vectors θ with high learning performanceand with constrained connectivity.
The algorithm will therefore spend most time in network conﬁgurations where the connectivity supports the parameter learning, such that, connections with largesupport under the objective function (1) will be maintained active with high probability, while otherconnections are randomly tested and discarded if found not useful.
C ONCLUSIONS
We have presented a method for modifying backprop and backprop-through-time so that not only theweights of connections, but also the connectivity graph is simultaneously optimized during training.
This can be achieved while staying always within a given bound on the total number of connections.
When the absolute value of a weight is moved by backprop through 0 , it becomes a weight with theopposite sign.
In contrast, in DEEP R a connection vanishes in this case (more precisely: becomesdormant), and a randomly drawn other connection is tried out by the algorithm.
This setup requiresthat, like in neurobiology, the sign of a weight does not change during learning.
Another essentialingredient of DEEP R is that it superimposes the gradient-driven dynamics of each weight witha random walk.
This feature can be viewed as another inspiration from neurobiology (Mongilloet al., 2017).
An important property of DEEP R is that — in spite of its stochastic ingredient — itsoverall learning dynamics remains theoretically tractable: Not as gradient descent in the usual sense,but as convergence to a stationary distribution of network conﬁgurations which assigns the largestprobabilities to the best-performing network conﬁgurations.
An automatic beneﬁt of this ongoingstochastic parameter dynamics is that the training process immediately adjusts to changes in thetask, while simultaneously transferring previously gained competences of the network (see Fig. 4).
We have demonstrated in Fig.
1–3 that DEEP R endows sparse and very sparse networks with powerful learning capabilities.
Due to its stochastic nature, various methods from stochastic optimizationare in principle applicable to the algorithm.
For example, while we kept the noise level (i.e., thetemperature T) ﬁxed during training for MNIST and TIMIT, a cooling schedule like in simulatedannealing enhanced the learning performance for CIFAR-
R EFERENCES
A M ETHODS
Choosing hyper-parameters for DEEP R: The learning rate η is deﬁned for each task independently (see task descriptions below).
Considering that the number of active connections is given asa constraint, the remaining hyper parameters are the regularization coefﬁcient α and the temperature
T .
We found that the performance of DEEP R does not depend strongly on the temperature T .
Yet,the choice of α has to be done more carefully.
For each dataset there was an ideal value of α : oneorder of magnitude higher or lower typically lead to a substantial loss of accuracy.
In MNIST, 96.3% accuracy under the constraint of 1% connectivity was achieved with α = 10 − 4and T chosen so that T =
η 2 10 − 12 .
In TIMIT, α = 0 .
03 and T =
(higher values of T couldimprove the performance slightly but it did not seem very signiﬁcant).
In CIFAR-10 a different αwas assigned to each connectivity matrix.
To reach 84.1% accuracy with 5% connectivity we usedin each layer from input to output α = [0 , 10 − 7 , 10 − 6 , 10 − 9 , 0] .
The temperature is initialized with
T =
η α 2 18 and decays with the learning rate (see paragraph of the methods about CIFAR-10).
Choosing hyper-parameters for soft-DEEP R: The main difference between soft-DEEP R and
DEEP R is that the connectivity is note given as a global constraint.
This is a considerable drawbackif one has strict constraint due to hardware limitation but it is also an advantage if one simply wantsto generate very sparse network solutions without having a clear idea on the connectivities that arereachable for the task and architecture considered.
In any cases, the performance depends on the choice of hyper-parameters α , T and θ min , but alsounlike in DEEP R - these hyper parameters have inter-dependent relationships that one cannot ignore(as for DEEP R, the learning rate η is deﬁned for each task independently).
The reason why softDEEP R depends more on the temperature is that the rate of re-activation of connections is drivenby the amplitude of the noise whereas they are decoupled in DEEP R. To summarize the results ofan exhaustive parameter search, we found that √ 2 Tη should ideally be slightly below α .
In generalhigh θ min leads to high performance but it also deﬁnes an approximate lower bound on the smallestreachable connectivity.
This lower bound can be estimated by computing analytically the stationarydistribution under rough approximations and the assumption that the gradient of the likelihood iszero.
If p min is the targeted lower connectivity bound, one needs θ min ≈ − T (1 − p min ) αp min .
For MNIST we used α = 10 − 5 and T =
η α 2 18 for all data points in Fig.
1 panel A and a rangeof values of θ min to scope across different ranges of connectivity lower bounds.
In TIMIT and
CIFAR-10 we used a simpler strategy which lead to a similar outcome, we ﬁxed the relationships:α = 3 (cid:113) 2 Tη = − 13 θ min
and we varied only α to produce the solutions shown in Fig.
1 panel B and
Fig.
Re-implementing pruning and (cid:96) 1 -shrinkage: To implement (cid:96) 1 -shrinkage (Tibshirani, 1996;
Collins & Kohli, 2014), we applied the (cid:96) 1 -shrinkage operator θ ← relu ( | θ | − ηα ) sign ( θ ) after eachgradient descent iteration.
The performance of the algorithm is evaluated for different α varying ona logarithmic scale to privilege a sparse connectivity or a high accuracy.
For instance for MNISTin Figure 3.A we used α of the form 10 − n 2 with n going from 4 to 12 .
The optimal parameter wasn = 9 .
We implemented the pruning described in Han et al.
(2015b).
This algorithm uses several phases:training - pruning - training, or one can also add another pruning iteration: training - pruning - training - pruning - training.
We went for the latter because it increased performance.
Each ”training”phase is a complete training of the neural network with (cid:96) 2 -regularization 1 .
At each ”pruning” phase,the standard deviation of weights within a weight matrix w std is computed and all active weightswith absolute values smaller than qw std are pruned ( q is called the quality parameter).
Grid search isused to optimize the (cid:96) 2 -regularization coefﬁcient and quality parameter.
The results for MNIST arereported in Figure 5.
[Figure]
MNIST:
We used a standard feed forward network architecture with two hidden layers with 200neurons each and rectiﬁed linear activation functions followed by a 10-fold softmax output.
For allalgorithms we used a learning rate of 0.05 and a batch size of 10 with standard stochastic gradientdescent.
Learning stopped after 10 epochs.
All reported performances in this article are based onthe classiﬁcation error on the MNIST test set.
CIFAR-10:
The ofﬁcial tutorial for convolutional networks of tensorﬂow 2 is used as a referenceimplementation.
Its performance out-of-the-box provides the fully connected baseline.
We used thevalues given in the tutorial for the hyper-parameters in all algorithms.
In particular the layer-speciﬁcweight decay coefﬁcients that interact with our algorithms were chosen from the tutorial for DEEP
R, soft-DEEP R, pruning, and (cid:96) 1 -shrinkage.
In the fully connected baseline implementation, standard stochastic gradient descent was used witha decreasing learning rate initialized to 1 and decayed by a factor 0 .
1 every 350 epochs.
Trainingwas performed for one million iterations for all algorithms.
For soft-DEEP R, which includes atemperature parameter, keeping a high temperature as the weight decays was increasing the rateof re-activation of connections.
Even if intermediate solution were rather sparse and efﬁcient thesolution after convergence were always dense.
Therefore, the weight decay was accompanied byannealing of the temperature T .
This was done by setting the temperature to be proportional to thedecaying η .
This annealing was used for DEEP R and soft-DEEP R.
TIMIT:
The TIMIT dataset was preprocessed and the LSTM architecture was chosen to reproducethe results from Greff et al.
Input time series were formed by 12 MFCC coefﬁcients and thelog energy computed over each time frame.
The inputs were then expanded with their ﬁrst andsecond temporal derivatives.
There are 61 different phonemes annotated in the TIMIT dataset, toreport an error rate that is comparable to the literature we performed a standard grouping of thephonemes to generate 39 output classes (Lee & Hon, 1989; Graves et al., 2013; Greff et al., 2017).
As usual, the dialect speciﬁc sentences were excluded (SA ﬁles).
A validation set and early stopping were necessary to train a network with dense connectivity matrixon TIMIT because the performance was sometimes instable and it suddenly dropped during trainingas seen in Fig.
3D for (cid:96) 1 -shrinkage.
Therefore a validation set was deﬁned by randomly selecting5% of the training utterances.
All algorithms were trained for 40 epochs and the reported test errorrate is the one at minimal validation error.
To accelerate the training in comparison the reference from Greff et al.
(2017) we used mini-batchesof size 32 and the ADAM optimizer (Kingma & Ba (2014)).
This was also an opportunity to test theperformance of DEEP R and soft-DEEP R with such a variant of gradient descent.
The learning ratewas set to 0 .
and we kept the default momentum parameters of ADAM, yet we found that changing the (cid:15) parameter (as deﬁned in Kingma & Ba (2014)) from 10 − 8 to 10 − 4 improved the stability offully connected networks during training in this recurrent setup.
As we could not ﬁnd a reference thatimplemented (cid:96) 1 -shrinkage in combination with ADAM, we simply applied the shrinkage operatorafter each iteration of ADAM which might not be the ideal choice in theory.
It worked well inpractice as the minimal error rate was reached with this setup.
The same type of (cid:96) 1 regularizationin combination with ADAM was used for DEEP R and soft-DEEP R which lead to very sparse andefﬁcient network solutions.
Initialization of connectivity matrices:
We found that the performance of the networks dependedstrongly on the initial connectivity.
Therefore, we followed the following heuristics to generateinitial connectivity for DEEP R, soft-DEEP R and the control setup with ﬁxed connectivity.
First, for the connectivity matrix of each individual layer, the zero entries were chosen with uniformprobability.
Second, for a given connectivity constraint we found that the learning time increasedand the performance dropped if the initial connectivity matrices were not chosen carefully.
Typicallythe performance dropped drastically if the output layer was initialized to be very sparse.
Yet in mostnetworks the number of parameters is dominated by large connectivity matrices to hidden layers.
Abasic rule of thumb that worked in our cases was to give an equal number of active connections tothe large and intermediate weight matrices, whereas smaller ones - typically output layers - shouldbe densely connected.
We suggest two approaches to reﬁne this guess: One can either look at the statistics of the connectivity matrices after convergence of DEEP R or soft-DEEP R, or, if possible, the second alternative isto initialize once soft-DEEP R with a dense matrix and observe the connectivity matrix after convergence.
In our cases the connectivity at convergence were coherent with the rule of thumb describedabove and we did not need to pursue intensive search for ideal initial connectivity matrices.
For MNIST, the number of parameters in each layers are 235k, 30k and 1k from input to output.
Using our rule of thumb, for a given global connectivity p 0 , the layers were respectively initializedwith connectivity 0 .
75 p 0 , 2 . 3 p 0 and 22 .
8 p 0 .
For CIFAR-10, the baseline network has two convolutional layers with ﬁlters of shapes 5 × 5 × 3 × 64and 5 × 5 × 64 × 64 respectively, followed by two fully connected layer with weight matrices of shape2304 × 384 and 384 × 192 .
The last layer is then projected into a softmax over 10 output classes.
The number of parameters per connectivity matrices are therefore 5k, 102k, 885k, 738k and 2k frominput to output.
The connectivity matrices were initialized with connectivity 1 , 8 p 0 , 0 .
8 p 0 , 8 p 0 , and1 .
For TIMIT, the connection matrix from the input to the hidden layer is of size 39 × 800 , the recurrentmatrix has size 200 × 800 and the size of the output matrix is 200 × 39 .
Each of these threeconnectivity matrices were initialized with a connectivity of 3 p 0 , p 0 , and 10 p 0 respectively.
Initialization of weight matrices: For CIFAR-10 the initialization of matrix coefﬁcient are givenby the reference implementation.
For MNIST and TIMIT, the weight matrices were initialized withθ = 1 √ n in N (0 , 1) c where n in is the number of afferent neurons, N (0 , 1) samples from a centeredgaussian with unit variance and c is a binary connectivity matrix.
It would not be good to initialize the parameters of all dormant connections to zero in soft-DEEP R.
After a single noisy iteration, half of them would become active which would fail to initialize thenetwork with a sparse connectivity matrix.
To balance out this problem we initialized the parametersof dormant connections uniformly between the clipping value θ min and zero in soft-DEEP R.
Parameters for Figure 4 This is a variant of our MNIST experiment were the target labels areshufﬂed after every training epoch.
To make visible the generalization capability of DEEP R overa small number of epochs, we enhanced the noise exploration by setting a batch to 1 so that theconnectivity matrices were updated at every time step.
Also we used a larger network with 400neurons in each hidden layer.
The remaining parameters were similar to those used previously: theconnectivity was constrained to 1% and the connectivity matrices were initialized with respectiveconnectivities: 0 . 01 , 0 . 01 , and 0 .
The parameters of DEEP R were set to η = 0 .
05 , α = 10 − 5 and
T =
η α 2 2 .
[Figure]
B R EWIRING DURING TRAINING ON MNIST
Fig.
6 shows the rewiring behavior of DEEP R per network layer for the feed-forward neural networktrained on MNIST and the training run indicated by the small gray box around the green dot in
Fig.
1A. Since it takes some iterations until the weights of connections that do not contribute to areduction of the error are driven to 0 , the number of newly established connections K ( l ) new in layer lis small for all layers initially.
After this initial transient, the number of newly activated connectionsstabelized to a value that is proportional to the total number of potential connections in the layer(Fig.
1B).
DEEP R continued to rewire connections even late in the training process.
C D ETAILS TO : C ONVERGENCE PROPERTIES OF SOFT -DEEP R
Here we provide additional details to the convergence properties of the soft-DEEP R parameterupdate provided in Algorithm 2.
We reiterate here Eq.
(2):dθ
[Equation]
Discrete time updates can be recovered from the set of SDEs (4) by integration over a short timeperiod ∆ t
[Equation]
where the learning rate η is given by η = β ∆ t .
We prove that the stochastic parameter dynamics Eq.
(4) converges to the target distribution p ∗ ( θ )given in Eq.
The proof is analogous to the derivation given in Kappel et al.
Wereiterate the proof here for the special case of supervised learning.
The fundamental property ofthe synaptic sampling dynamics Eq.
(4) is formalized in Theorem 1 and proven below.
Before westate the theorem, we brieﬂy discuss its statement in simple terms.
Consider some initial parametersetting θ 0 .
Over time, the parameters change according to the dynamics (4).
Since the dynamicsinclude a noise term, the exact value of the parameters θ ( t ) at some time t > 0 cannot be determined.
However, it is possible to describe the exact distribution of parameters for each time t .
We denote thisdistribution by p FP ( θ , t ) , where the “FP” subscript stands for “Fokker-Planck” since the evolutionof this distribution is described by the Fokker-Planck equation (6) given below.
Note that we makethe dependence of this distribution on time explicit in this notation.
It can be shown that for thedynamics (6), p FP ( θ , t ) converges to a well-deﬁned and unique stationary distribution in the limit oflarge t .
To prove the convergence to the stationary distribution we show that it is kept invariant bythe set of SDEs Eq.
(4) and that it can be reached from any initial condition.
We now state Theorem 1 formally.
To simplify notation we drop in the following the explicit timedependence of the parameters θ .
Theorem 1.
Let p ∗ ( θ | X , Y ∗ ) be a strictly positive, continuous probability distribution over parameters θ , twice continuously differentiable with respect to θ , and let β > 0 .
Then the set ofstochastic differential equations Eq.
(4) leaves the distribution p ∗ ( θ ) (1) invariant.
Furthermore,p ∗ ( θ ) is the unique stationary distribution of the sampling dynamics.
Proof.
The stochastic differential equation Eq.
(4) translates into a Fokker-Planck equation (Gardiner, 2004) that describes the evolution of the distribution over parameters θ
[Equation]
where p FP ( θ , t ) denotes the distribution over network parameters at time t .
To show that p ∗ ( θ ) leaves the distribution invariant, we have to show that ∂∂t p FP ( θ , t )
=
(i.e., p FP ( θ , t ) does not change) if we set p FP ( θ , t ) to p ∗ ( θ ) .
Plugging in the presumed stationary distribution p ∗ ( θ ) for p FP ( θ , t ) on the right hand side of Eq.
(6), one obtains
[Equation]
which by inserting p ∗ ( θ ) =
1 Z p ∗ ( θ | X , Y ∗ ) 1 T , with normalizing constant Z , becomes
[Equation]
This proves that p ∗ ( θ ) is a stationary distribution of the parameter sampling dynamics Eq.
Sinceβ is strictly positive, this stationary distribution is also unique (see Section 3.7.2 in Gardiner (2004)).
The unique stationary distribution of Eq.
(6) is given by p ∗ ( θ )
=
1 Z p ∗ ( θ | X , Y ∗ ) 1 T , i.e., p ∗ ( θ ) is theonly solution for which ∂∂t p FP ( θ , t ) becomes 0 , which completes the proof.
The updates of the soft-DEEP R algorithm (Algorithm 2) can be written as
[Equation]
Eq.
(7) is a special case of the general discrete parameter dynamics (5).
To see this we apply Bayes’ rule to expand the derivative of the log posterior into the sum of the derivatives of the prior and the likelihood:
[Equation]
such that we can rewrite Eq.
[Equation]
To include automatic network rewiring in our deep learning model we adopt the approach described in Kappel et al. (2015).
Instead of using the network parameters θ directly to determine the synaptic weights of network N , we apply a nonlinear transformation w k = f
( θ k ) to each connection k , given by the function
[Equation]
where s k ∈ { 1 , − 1 } is a parameter that determines the sign of the connection weight and γ > 0 is a constant parameter that determines the smoothness of the mapping.
In the limit of large γ Eq.
(9) converges to the rectiﬁed linear function
[Equation]
such that all connections with θ k < 0 are not functional.
Using this, the gradient of the log-likelihood function ∂∂θ k log p N ( Y ∗ | X , θ ) in Eq.
(8) can be written as ∂∂θ k log p N ( Y ∗ | X , θ ) =
∂∂θ k f ( θ k ) ∂∂θ k E X , Y ∗ ( θ ) which for our choice of f ( θ k ) , Eqs.
(9), becomes
[Equation] [Equation]
where σ ( x ) = 1 1+ e − x denotes the sigmoid function.
The error gradient ∂∂θ k E X , Y ∗ ( θ ) can be computed using standard Error Backpropagation Neal (1992);
Rumelhart et al.
Theorem 1 requires that Eq.
(11) is twice differentiable, which is true for any ﬁnite value for γ .
In our simulations we used the limiting case of large γ such that dormant connections are actually mapped to zero weight.
In this limit, one approaches the simple expression
[Equation]
θ ) else .
Thus, the gradient (12) vanishes for dormant connections ( θ k < 0 ).
Therefor changes of dormantconnections are independent of the error gradient.
This leads to the parameter updates of the soft-DEEP R algorithm given by Eq.
The term 559 √ 2 Tη ν k results from the diffusion term W k integrated over ∆ t , where ν k is a Gaussian randomvariable with zero mean and unit variance.
The term − ηα results from the exponential prior distribution p S ( θ ) (the (cid:96) 1 -regularization).
Note that this prior is not differentiable at 0.
In (7) weapproximate the gradient by assuming it to be zero at θ k = 0 and below.
Thus, parameters on thenegative axis are only driven by a random walk and parameter values might therefore diverge to−∞ .
To ﬁx this problem we introduced a reﬂecting boundary at θ min (parameters were clipped atthis value).
Another potential solution would be to use a different prior distribution that also effectsthe negative axis, however we found that Eq.
(7) produces very good results in practice.
D A NALYSIS OF CONVERGENCE OF THE DEEP R ALGORITHM
endAlgorithm 3: A reformulation of Algorithm 1 that is used for the proof in Theorem 2.
Markov transition operators T θ ( θ | θ (cid:48) , c (cid:48) ) and T c ( c | θ ) are applied for parameter updates in each iteration.
The transition operator T θ ( θ | θ (cid:48) , c (cid:48) ) updates θ and corresponds to line 3, T c ( c | θ ) updates the connectivity constraint vector c and corresponds to lines 4,7 and 8 of Algorithm 1.
θ (cid:48) and c (cid:48) denote the parameter vector and connectivity constraint of the previous time step, respectively.
Here we provide additional details to the convergence properties of the DEEP R algorithm.
To doso we formulate the algorithm in terms of a Markov chain that evolves the parameters θ and the connectivity constraints (listed in Algorithm 3).
Each application of the Markov transition operatorscorresponds to one iteration of the DEEP R algorithm.
We show that the distribution of parametersand network connectivities over the iterations of DEEP R converges to the stationary distribution
Eq.
Each iteration of DEEP R corresponds to two update steps, which we formally describe in Algorithm 3 using the Markov transition operators T θ and T c and the binary constraint vector c ∈ { 0 , 1 } Kwith elements c k , where c k = 1 represents an active connection k .
c is a constraint on the dynamics, i.e., all connections k for which c k = 0 have to be dormant in the evolution of the parameters.
The transition operators are conditional probability distributions from which in each iteration newsamples for θ and c are drawn for given values from the previous time step ( θ (cid:48) and c (cid:48) ).
Parameter update : The transition operator T θ ( θ | θ (cid:48) , c (cid:48) ) updates all parameters θ k for which
c k = 1 (that are active).
The operator is realized by advancing the SDE (2) for a small time
step ∆ t (line 3 of Algorithm 3).
Connectivity update : for all parameters θ k that are dormant, set c k = 0 and randomly select
an element c l which is currently 0 and set it to 1.
This corresponds to line 3 of Algorithm
and is realized by drawing a new c from T c ( c | θ ) .
The constraint imposed by c on θ is formalized through the deterministic binary function C ( θ , c ) ∈{ 0 , 1 } which is 1 if the parameters θ are compatible with the constraint vector c and 0 otherwise.
This is expressed as (with ⇒ denoting the Boolean implication):
[Equation]
if for all
[Equation]
0 else .
The constraint C ( θ , c ) is fulﬁlled if all connections k with c k = 0 are dormant ( θ k < 0 ).
Note that the transition operator T c ( c | θ ) depends only on the parameter vector θ .
It samples a newc with uniform probability among the constraint vectors that are compatible with the current set ofparameters θ .
We write the number of possible vectors c that are compatible with θ as µ ( θ ) , givenby the binomial coefﬁcient (the number of possible selections that fulﬁll the constraint of new activeconnections)
[Equation]
where | c | denotes the number of non-zero elements in c and χ is the set of all binary vectors withexactly M elements of value 1 .
Using this we can deﬁne the operator T c ( c | θ ) as:
[Equation]
where δ denotes the vectorized Kronecker delta function, with δ ( 0 )
= 1 and 0 else.
Note that
Eq.
(15) assigns non-zero probability only to vectors c that are zero for elements k for which θ k < 0is true (assured by the second term).
In addition vectors c have to fulﬁll | c | =
M .
Therefore,sampling from this operator introduces randomly new connection for the number of missing ones inθ .
This process models the connectivity update of Algorithm 3.
The transition operator T θ ( θ | θ (cid:48) , c (cid:48) ) in Eq.
(17) evolves the parameter vector θ under the constraintc , i.e., it produces parameters conﬁned to the connectivity constraint.
We ﬁrst consider a generaloperator that leaves some constrained distribution π ( θ | c ) invariant and we will later discuss thespecial case of using a constrained version of the posterior distribution p ∗ ( θ | c , X , Y ∗ ) .
We assumethat we can write the stationary distribution of the operator T θ as
[Equation]
16) with normalizer Z and where π ( θ ) is some distribution over θ that may not obey the constraint
C ( θ , c ) .
The Markov transition operators are performed one after the other so that the total update can bewritten in terms of the compound operator T
[Equation]
Applying the compound operator T given by Eq.
(17) corresponds to advancing the parameters fora single iteration of Algorithm 3.
Using these deﬁnitions a general theorem can be enunciated for arbitrary distributions π ( θ | c ) ofthe form (16).
The following theorem states that the distribution of variable pairs c and θ that is leftstationary by the operator T is the product of Eq.
(16) and a uniform prior p C ( c ) over the constraintvectors which have M active connections.
This prior is formally deﬁned as
[Equation] [Equation]
with χ as deﬁned in (14).
The main theorem to analyze the dynamics of Algorithm 3 can now be written as
Theorem 2.
Let T θ ( θ | θ (cid:48) , c ) be the transition operator of a Markov chain over θ with stationarydistribution π ( θ | c ) of the form (16) .
Let T c
( c | θ ) be deﬁned by Eq.
Then the Markov chainover θ and c with transition operator
[Equation]
[Equation]
leaves the stationary distribution
[Equation]
invariant.
Under the assumption that T θ ( θ | θ (cid:48) , c ) maintains for each parameter θ k a non-zero probability of reaching negative values, the stationary distribution is also unique.
Proof.
Theorem 2 holds for T c in combination with any operator T θ that updates θ that can bewritten in the form (16).
We prove Theorem 2 by proving the following equality to show that Tleaves (20) invariant:
[Equation]
We expand the left-hand term using Eq.
(19) and Eq.
[Equation]
Since T c does not depend on θ (cid:48)
[Equation]
one can pull it out of the integral and then marginalize over θ (cid:48) by observing that π ( θ | c (cid:48) ) is by deﬁnition the stationary distribution of T
θ ( θ | θ (cid:48) , c (cid:48) ) :
[Equation]
[Equation]
What remains to be done is to marginalize over c (cid:48) and to relate the result to the stationary distribution p ∗ ( θ , c )
=
π ( θ | c ) p C ( c ) .
First we replace T c with its deﬁnition Eq.
[Equation]
We can now replace the sum over ξ using Eq.
[Equation]
p C ( c (cid:48) ) .
From Eq.
(14), Eq.
(16) and Eq.
(18) we ﬁnd the equalities (cid:80) c (cid:48)
π ( θ | c (cid:48) )
p C ( c (cid:48) )
= 1 Z π ( θ ) (cid:80) c (cid:48)
C
( θ , c (cid:48) )
p C ( c (cid:48) )
= 1 Z µ ( θ ) | χ | π ( θ ) .
Using this we get
[Equation]
c
[Equation]
c | θ ) (cid:88) c
[Equation]
Finally using again Eq.
(16), i.e. 1 Z π ( θ ) C ( θ , c
[Equation]
=
[Equation]
This shows that the stationary distribution Eq.
(20) is invariant under the compound operator (19).
Under the assumption that T θ ( θ | θ (cid:48) , c (cid:48) ) allows each parameter θ k to become negative with non-zeroprobability, the stationary distribution is also unique.
This can be seen by noting that under thisassumption each connection will become dormant sooner or later and thus each state in c can bereached from any state c (cid:48) .
The Markov chain is therefore irreducible and the stationary distributionis unique.
What remains to be shown is that the transition operator T θ ( θ | θ (cid:48) , c (cid:48) ) with stationary distributionπ ( θ | c (cid:48) ) of the form Eq.
(16) can be constructed for our case of supervised learning π ( θ | c (cid:48) )
=p ∗ ( θ | c (cid:48) , X , Y ∗ ) and that it corresponds to the implementation of DEEP R.
A transition operator T θ ( θ | θ (cid:48) ) without the constraint, with stationary distribution p ∗ ( θ | X , Y ∗ )
canbe constructed by letting the parameters θ advance according to (5) for a short time ∆ t .
Thereforethe transition operator T θ
( θ | θ (cid:48) ) for this process is given by the Fokker-Planck equation (6) evaluatedat ∆ t with parameters set initially to θ (cid:48) .
This corresponds to the updates for the soft-DEEP Ralgorithm.
For the constrained transition operator T θ ( θ | θ (cid:48) , c (cid:48) ) in Eq.
(17) we use that by construction the dynamics of dormant connections is independent from the error gradient in DEEP R for small ∆ t , andin fact all dormant connections are independent from each other.
This can be seen by noting that theerror gradient disappears in the update equations for dormant connections in (7), which is a resultof our choice of mapping function from θ to w given in Eq.
Therefore, under the constraint
C ( θ , c ) it is sufﬁcient to compute the updates only for θ k with c k = 1 and simply redraw parametervalues for connections with c k = 0 from the support of the stationary distribution p ∗ ( θ | X , Y ∗ )on the negative axis once they become active again.
The stationary distribution of this transitionoperator is given by the constrained posterior distribution
[Equation]
The constraint C ( θ , c ) for this process is fulﬁlled since connections become dormant only if they gobelow 0 and the process keeps them at this value.
For negative parameters values, the stationary distribution depends only on the choice for the priordistribution p S ( θ ) and can be constructed in a way that is easily computable.
In Algorithm (1) weimplement this by instantaneously setting connections to 0 as soon as they become active.
We foundthat this process works very well in practice since it makes most effective use of the connections andfor a strong enough (cid:96) 1 regularization, connections that are not useful will disappear soon after theirintroduction.
Furthermore, it is also not necessary to memorize the values of dormant parameters for the DEEP Ralgorithm.
This also follows from our choice of mapping function from θ to w given in Eq.
Allparameters θ k ≤ 0 are mapped to w k =
and they only become active once θ k randomly crosses tothe positive side.
Since, by construction parameter updates for dormant synapses are independent ofthe error gradient, it is sufﬁcient to randomly introduce these connections according to the stationarydistribution over parameter values on the negative axis