C RITICAL INITIALISATION IN CONTINUOUS APPROXIMATIONS OF BINARY NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The training of stochastic neural network models with binary ( ± 1 ) weights and activations via continuous surrogate networks is investigated.
We derive new surrogates using a novel derivation based on writing the stochastic neural network as a Markov chain.
This derivation also encompasses existing variants of the surrogates presented in the literature.
Following this, we theoretically study the surrogates at initialisation. We derive, using mean ﬁeld theory, a set of scalar equations describing how input signals propagate through the randomly initialised networks. The equations reveal whether so-called critical initialisations exist for each surrogate network, where the network can be trained to arbitrary depth.
Moreover, we predict theoretically and conﬁrm numerically, that common weight initialisation schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance.
This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to ± 1 , for deeper networks to be trainable.
I NTRODUCTION
The problem of learning with low-precision neural networks has seen renewed interest in recent years, in part due to the deployment of neural networks on low-power devices.
Currently, deep neural networks are trained and deployed on GPUs, without the memory or power constraints of such devices.
Binary neural networks are a promising solution to these problems.
If one is interested in addressing memory usage, the precision of the weights of the network should be reduced, with the binary case being the most extreme. In order to address power consumption, networks with both binary weights and neurons can deliver signiﬁcant gains in processing speed, even making it feasible to run the neural networks on CPUs Rastegari et al.
Of course, introducing discrete variables creates challenges for optimisation, since the networks are not continuous and differentiable.
Recent work has opted to train binary neural networks directly via backpropagation on a differentiable surrogate network, thus leveraging automatic differentiation libraries and GPUs.
A key to this approach is in deﬁning an appropriate differentiable surrogate network as an approximation to the discrete model.
A principled approach is to consider binary stochastic variables and use this stochasticity to “smooth out” the non-differentiable network.
This includes the cases when (i) only weights, and (ii) both weights and neurons are stochastic and binary.
In this work we study two classes of surrogates, both of which make use of the Gaussian central limit theorem (CLT) at the receptive ﬁelds of each neuron.
In either case, the surrogates are written as differentiable functions of the continuous means of stochastic binary weights, but with more complicated expressions than for standard continuous networks.
One approximation, based on analytic integration, yields a class of deterministic surrogates Soudry et al.
The other approximation is based on the local reparameterisation trick (LRT) Kingma & Welling (2013), which yields a class of stochastic surrogates Shayer et al. Previous works have relied on heuristics to deal with binary neurons Peters & Welling (2018), or not backpropagated gradients correctly. Moreover, none of these works considered the question of initialisation, potentially limiting performance.
The seminal papers of Saxe et al.
(2013), Poole et al. (2016), Schoenholz et al.
(2016) used a mean ﬁeld formalism to explain the empirically well known impact of initialization on the dynamics of learning in standard networks. From one perspective the formalism studies how signals propagate forward and backward in wide, random neural networks, by measuring how the variance and correlation of input signals evolve from layer to layer, knowing the distributions of the weights and biases of the network. By studying these moments the authors in Schoenholz et al. (2016) were able to explain how heuristic initialization schemes avoid the “vanishing and exploding gradients problem”
Glorot & Bengio (2010), establishing that for neural networks of arbirary depth to be trainable they must be initialised at “criticality”, which corresponds to initial correlation being preserved to any depth. The paper makes three contributions. The ﬁrst contribution is the presentation of new algorithms, with a new derivation able to encompass both surrogates, and all choices of stochastic binary weights, or neurons. The derivation is based on representing the stochastic neural network as a
Markov chain, a simplifying and useful development. As an example, using this representation we are easily able to extend the LRT to the case of stochastic binary neurons, which is new. This was not possible in Shayer et al.
(2017), who only considered stochastic binary weights.
As a second example, the deterministic surrogate of Soudry et al. (2014) is easily derived, without the need for Bayesian message passing arguments. Moreover, unlike Soudry et al.
(2014) we correctly backpropagate through variance terms, as we discuss. The second contribution is the theoretical analysis of both classes of surrogate at initialisation, through the prism of signal propagation theory Poole et al.
(2016), Schoenholz et al. This analysis is achieved through novel derivations of the dynamic mean ﬁeld equations, which hinges on the use of self-averaging arguments Mezard et al.
The results of the theoretical study, which are supported by numerical simulations and experiment, establish that for a surrogate of arbitrary depth to be trainable, it must be randomly initialised at “criticality”. In practical terms, criticality corresponds to using initialisations that avoid the “vanishing and exploding gradients problem”
Glorot & Bengio (2010).
We establish the following key results: • For networks with stochastic binary weights and neurons, the deterministic surrogate can achieve criticality, while the LRT cannot. • For networks with stochastic binary weights and continuous neurons, the LRT surrogate can achieve criticality (no deterministic surrogate exists for this case)
In both cases, the critical initialisation corresponds to randomly initialising the means of the binary weights close to ± 1 , a counter intuitive result.
A third contribution is the consideration of the signal propagation properties of random binary networks, in the context of training a differentiable surrogate network. We derive these results, which are partially known, and in order to inform our discussion of the experiments.
This paper provides insights into the dynamics and training of the class of binary neural network models.
To date, the initialisation of any binary neural network algorithm has not been studied, although the effect of quantization levels has been explored through this perspective Blumenfeld et al. Currently, the most popular surrogates are based on the so-called “Straight-Through” estimator Bengio et al.
(2013), which relies on heuristic deﬁnitions of derivatives in order to deﬁne a gradient. However, this surrogate typically requires the use of batch normalization, and other heuristics.
The contributions in this paper may help shed light on what is holding back the more principled algorithms, by suggesting practical advice on how to initialise, and what to expect during training. Paper outline: In section 2 we present the binary neural network algorithms considered. In subsection 2.1 we deﬁne binary neural networks and subsection 2.2 their stochastic counterparts.
In subsection 2.3 we use these deﬁnitions to present new and existing surrogates in a coherent framework, using the Markov chain representation of a neural network to derive variants of both the deterministic surrogate, and the LRT-based surrogates.
We derive the LRT for the case of stochastic binary weights, and both LRT and deterministic surrogates for the case of stochastic binary weights and neurons.
In section 3 we derive the signal propagation equations for both the deterministic and stochastic LRT surrogates.
This includes deriving the explicit depth scales for trainability, and solving the equations to ﬁnd the critical initialisations for each surrogate, if they exist.
In section 4 we present the numerical simulations of wide random networks, to validate the mean ﬁeld description, and experimental results to test the trainability claims.
Finally in section 5 we summarize the key results, and provide a discussion of the insights they provide.
B INARY NEURAL NETWORK ALGORITHMS 2. C ONTINUOUS NEURAL NETWORKS AND BINARY NEURAL NETWORKS
A neural network model is typically deﬁned as a deterministic non-linear function.
We consider a fully connected feedforward model, which is composed of N (cid:96) × N (cid:96) − 1 weight matrices W (cid:96) and bias vectors b (cid:96) in each layer (cid:96) ∈ { 1 , . . .
, L } , with elements W (cid:96)ij ∈ R and b (cid:96)i ∈ R .
Given an input vector x 0 ∈ R N 0 , the network is deﬁned in terms of the following recursion,
[Equation]
where the pointwise non-linearity is, for example, φ (cid:96) ( · )
= tanh( · ) .
We refer to the input to a neuron, such as h (cid:96) , as the pre-activation ﬁeld.
A deterministic binary neural network simply has weights W (cid:96)ij ∈ {± 1 } and φ (cid:96) ( · ) = sign( · ) , and otherwise the same propagation equations. Of course, this is not differentiable, thus we deﬁne stochastic binary variables in order to smooth out the non-differentiable network. The product of training a surrogate of a stochastic binary network is ideally a deterministic binary network that is able to generalise from its training set.
We could also use the stochastic binary network, but this is not as computationally advantageous in standard hardware. 2. S TOCHASTIC BINARY NEURAL NETWORKS In stochastic binary neural networks we denote the matrices as S (cid:96) with all weights 1 S (cid:96)ij ∈ {± 1 } being independently sampled binary variables with probability is controlled by the mean M (cid:96)ij = E S (cid:96)ij . Neuron activation in this model are also binary random variables, due to pre-activation stochasticity and to inherent noise.
We consider parameterised neurons such that the mean activation conditioned on the pre-activation is given by some function taking values in [ − 1 , 1] , i.e. E [ x (cid:96)j | h (cid:96)i ] = φ ( h (cid:96)i ) , for example φ ( · ) = tanh( · ) . We write the propagation rules for the stochastic network as follows:
[Equation] Notice that the distribution of x (cid:96) factorizes when conditioning on x (cid:96) − 1 .
The form of the neuron’s mean function φ ( · ) depends on the underlying noise model. We can express a binary random variable x ∈ {± 1 } with x ∼ p ( x ; θ ) via its latent variable formulation x = sign( θ + α L ) . In this form θ is referred to as a “natural” parameter, and the term L is a latent random noise, whose cumulative distribution function σ ( · ) determines the form of the non-linearity since φ ( · ) = 2 σ ( · ) − 1 . In general the form of φ ( · ) will impact on the surrogates’ performance, including within and beyond the mean ﬁeld description presented here.
However, a result from the analysis in Section 3 is that choosing a deterministic binary neuron, ie. the sign( · ) function, or a stochastic binary neuron, produces the same signal propagation equations, up to a scaling constant [Equation] 2. D ERIVATIONS OF NEW AND EXISTING SURROGATE NETWORKS
The idea behind several recent papers Soudry et al.
(2014), Baldassi et al.
(2018), Shayer et al.
(2017), Peters & Welling (2018) is to adapt the mean of the binary stochastic weights, with the stochastic model essentially used to “smooth out” the discrete variables and arrive at a differentiable function, open to the application of continuous optimisation techniques.
We now derive both the deterministic surrogate and LRT-based surrogates, in a common framework.
We consider a supervised classiﬁcation task, with training set D = { x µ , y µ } Pµ =1 , with y µ the label. we deﬁne a loss function for our surrogate model via
[Equation]
For a given input x µ and a realization of weights, neuron activations and biases in all layers, denoted by ( S , x , b ) , the stochastic neural network produces a probability distribution over the classes. Expectations over weights and activations are given by the mean values, E S (cid:96) = M (cid:96) and E [ x (cid:96) | h (cid:96) ] = φ ( h (cid:96) ) .
This objective can be recognised as a (minus) marginal likelihood, thus this method could be described as Type II maximum likelihood, or empirical Bayes.
The starting point for our derivations comes from rewriting the expectation equation 3 as the marginalization of a Markov chain, with layers indexes corresponding to time indices (cid:96) ∈ { 1 , . . . , L } . Markov chain representation of stochastic neural network:
[Equation]
where in the second line we dropped from the notation p ( S (cid:96) ; M (cid:96) ) the dependence on M (cid:96) for brevity. Therefore, for a stochastic network the forward pass consists in the propagation of the joint distribution of layer activations, p ( x (cid:96) | x µ ) , according to the Markov chain. We drop the explicit dependence on the initial input x µ from now on.
The ﬁrst step to obtaining a differentiable surrogate is to introduce continuous random variables.
We take the limit of large layer width and appeal to the central limit theorem to model the ﬁeld h (cid:96) as
Gaussian, with mean ¯ h (cid:96) and covariance matrix Σ (cid:96) .
Assumption 1: (CLT for stochastic binary networks)
In the large N limit, under the Lyapunov central limit theorem, the ﬁeld h (cid:96) =
[Equation]
converges to a Gaussian random variable with mean ¯ h (cid:96)i =
[Equation]
and covariance matrix Σ (cid:96) with diagonal Σ
[Equation]
While this assumption holds true for a large enough network, the second assumption is much stronger.
Assumption 2: (correlations are zero)
We assume the independence of the pre-activation ﬁeld h (cid:96) between any two dimensions.
Speciﬁcally, we assume the following covariance Σ MF ,
[Equation]
(5) with MF denoting the mean ﬁeld (factorized) assumption.
This assumption approximately holds assuming the neurons in each layer are not strongly correlated.
In the ﬁrst layer this is certainly true, since the input neurons are not random variables 2 . In subsequent layers, since the ﬁelds h (cid:96)i and h (cid:96)j share stochastic neurons from the previous layer, this cannot be assumed to be true. We expect this correlation to not play a signiﬁcant role, since the weights act to decorrelate the ﬁelds, and the neurons are independently sampled. However, the choice of surrogate inﬂuences the level of dependence.
The sampling procedure used in within the local reparametrization trick reduces correlations since variables are sampled, while the deterministic surrogate entirely discards them.
We obtain either surrogate model by successively approximating the marginal distributions, p ( x (cid:96) ) = (cid:82) d h (cid:96) p ( x (cid:96) | h (cid:96) ) ≈ ˆ p ( x (cid:96)i ) , starting from the ﬁrst layer.
We can do this by either (i) marginalising over the Gaussian ﬁeld using analytic integration, or (ii) sampling from the Gaussian.
After this, we use the approximation ˆ p ( x (cid:96)i ) to form the Gaussian approximation for the next layer, and so on.
[Equation]
Deterministic surrogate: We ﬁrst recall the analytic form of the neuron probability mass function, p ( x (cid:96) +1 i | h (cid:96) ) =
σ ( x (cid:96)i h (cid:96)i ) , with σ ( · ) a sigmoidal function, such as the logistic or Gaussian CDF. The approximation ˆ p ( x (cid:96)i ) is obtained using the well known Gaussian integral of the Gaussian cumulative distribution function, ˆ p ( x (cid:96)i ) =
(cid:82) dh σ ( x (cid:96)i h ) N ( h ; ¯ h (cid:96)i , Σ (cid:96)MF,ii ) = σ ( κ
[Equation]
We step through this derivation carefully in the Appendices, where we show that we can write down analogous forward propagation equations for this surrogate,
[Equation]
This is a slightly more general formulation than that in Soudry et al.
(2014), which considered sign activations.
We show in the appendices we obtain this as a particular limit.
Furthermore, in all implementations we backpropagate through the variance terms Σ − 12 MF , which were ignored in the previous work of Soudry et al. Note that the derivation here is simpler as well, not requiring complicated Bayesian message passing arguments, and approximations therein.
LRT surrogate:
The basic idea here is to rewrite the incoming Gaussian ﬁeld h ∼ N ( µ, Σ) as h = µ + √ Σ (cid:15) where (cid:15) ∼ N (0 , I ) .
Thus expectations over h can be written as expectations over (cid:15) and approximated by sampling.
The resulting network is thus differentiable, albeit not deterministic.
The forward propagation equations for this surrogate are
[Equation]
The local reparameterisation trick (LRT) Kingma & Welling (2013) has been previously used to obtain differentiable surrogates for binary networks. The authors of Shayer et al. (2017) considered only the case of stochastic binary weights, since they did not write the network as a Markov chain. The authors of Peters & Welling (2018) considered stochastic binary weights and neurons, but relied on other approximations to deal with the neurons, again having not used the Markov chain representation.
The result of each approximation, applied successively from layer to layer by either propagating means and variances or by, produces a differentiable function of the parameters M (cid:96)ij . It is then possible to perform gradient descent with respect to the M and b . Ideally, at the end of training we obtain a binary network that attains good performance.
This network could be a stochastic network, where we sample all weights and neurons, or a deterministic binary network. A deterministic network might be chosen taking the most likely weights, therefore setting W (cid:96)ij = sign( M (cid:96)ij ) , and replacing the stochastic neurons with sign( · ) activations.
S IGNAL PROPAGATION THEORY FOR CONTINUOUS SURROGATES
Since all the surrogates still retain the basic neural network structure of layerwise processing, crucially applying backpropagation for optimisation, it is reasonable to expect that surrogates are likely to inherit similar “training problems” as standard neural networks.
In this section we apply this formalism to the surrogates considered, given random initialisation of the means M (cid:96)ij and biases b (cid:96)i .
We are able to solve for the conditions of critical initialisation for each surrogate, as well as deriving the depth scales that limit trainability.
3. F ORWARD SIGNAL PROPAGATION FOR STANDARD CONTINUOUS NETWORKS
We ﬁrst recount the formalism developed in Poole et al. (2016).
Assume the weights of a standard continuous network are initialised with W (cid:96)ij ∼ N (0 , σ 2 w ) , biases b (cid:96) ∼ N (0 , σ 2 b ) , and input signal x 0 a has zero mean E x 0 = 0 and variance E [ x 0 a · x 0 a ] = q 0 aa , and with a denoting a particular input pattern.
As before, the signal propagates via Equation 1 from layer to layer.
We are interested in computing, from layer to layer, the variance q (cid:96)aa = 1 N (cid:96) (cid:80) i
( h (cid:96)i ; a ) 2 from a particular input x 0 a , and also the covariance between the pre-activations q (cid:96)ab = 1 N (cid:96) (cid:80) i
h
(cid:96)i ; a h (cid:96)i ; b , arising from two different inputs x 0 a and x 0 b with given covariance q 0 ab .
The mean ﬁeld approximation used here replaces each element in the pre-activation ﬁeld h (cid:96) i by a Gaussian random variable whose moments are matched.
Assuming also independence within a layer; E h (cid:96)i ; a h (cid:96)j ; a = q (cid:96)aa δ ij and
E h (cid:96)i ; a h (cid:96)j ; b = q (cid:96)ab δ ij , one can derive recurrence relations from layer to layer,
[Equation]
with Dz = dz √ 2 π e − z 2 2 the standard Gaussian measure.
The recursion for the covariance is given by
[Equation]
where u a = (cid:112) q (cid:96) − 1 aa z 1 , u b = (cid:113) q [Equation] 1 bb [Equation]
c
[Equation]
ab
[Equation]
+
[Equation]
1 − ( c [Equation] 1 ab ) 2 z 2 (cid:1) , and we identify c (cid:96)ab as the correlation in layer (cid:96) .
The other important quantity is the slope of the correlation recursion equation or mapping from layer to layer, denoted as χ , which is given by: [Equation]
[Equation]
We denote χ at the ﬁxed point c ∗ = 1 as χ 1 .
As discussed Poole et al.
(2016), when χ 1 = [-
 the system is at a critical point where-]1 , correlations can propagate to arbitrary depth.
Furthermore, χ 1 is equivalent to the mean square singular value of the Jacobian matrix for a single layer J ij = ∂h
(cid:96)i ∂h (cid:96) − 1 j , as explained in Poole et al.
Therefore controlling χ 1 will prevent the gradients from either vanishing or growing exponentially with depth.
We thus deﬁne critical initialisations as follows. Deﬁnition 1: Critical initialisations of a neural network are the points ( σ 2 b , σ 2 w ) corresponding to χ 1 = 1 .
This deﬁnition also holds for the surrogates which we now study.
3. S IGNAL PROPAGATION THEORY FOR DETERMINISTIC SURROGATES
For the deterministic surrogate model we assume at initialization that the binary weight means M (cid:96)ij are drawn independently and identically from a distribution P ( M ) , with mean zero and variance of the means given by σ 2 m .
For instance, a valid distribution could be a clipped Gaussian 3 , or another stochastic binary variable, for example P ( M ) =
12 δ ( M + σ m ) + 12 δ
( M − σ m ) , whose variance is σ 2 m .
The biases at initialization are distributed as b (cid:96) ∼ N (0 , σ 2 b ) .
Recall from the deterministic surrogate’s forward propagation equations, signals essentially propagate through the mean activations, ¯ x (cid:96)i =
E x (cid:96)i .
We show in Appendix B that the stochastic and deterministic binary neuron cases reduce to the same signal propagation equations, up to scaling constants.
In the deterministic sign( · ) neuron case, the equations are slightly simpler, and the ﬁeld is given by:
[Equation]
which we can be read from the Eq. 6.
As in the continuous case we are interested in computing the variance q (cid:96)aa = 1 N (cid:96) (cid:80)
i ( h (cid:96)i ; a ) 2 and covariance E h (cid:96)i ; a h (cid:96)j ; b
= q (cid:96)ab δ ij , via recursive formulae.
The key to the derivation is recognising that the denominator (cid:113) Σ (cid:96)MF,ii is a self-averaging [-quantity, [Equation] where we have used the properties that the M (cid:96)ij and h (cid:96) − 
-]quantity Mezard et al. This means it concentrates in probability to its expected value for large N .
Therefore we can safely replace it with its expectation.
Following this self-averaging argument, we can take expectations more readily as shown in the appendices. We ﬁnd the variance recursion to be
[Equation]
Based on this expression, and assuming q aa = q bb , the correlation recursion can be written as [Equation]
[Equation]
The slope of the correlation mapping from layer to layer, when the normalized length of each input is at its ﬁxed point q (cid:96)aa = q (cid:96)bb = q ∗ ( σ m , σ b ) , denoted as χ , is given by:
[Equation]
where u a and u b are deﬁned exactly as in the continuous case.
Refer to the appendices for full details of the derivation.
3.2. C RITICAL INITIALISATION : DETERMINISTIC SURROGATE
The condition for critical initialisation is χ 1 = 1 , since this determines the stability of the correlation map ﬁxed point c ∗ = 1 .
Note that for the deterministic surrogate this is always a ﬁxed point.
We can solve for the hyper-parameters ( σ 2 b , σ 2 m ) that satisfy this condition, using the dynamical equations of the network.
Claim 1: The points ( σ 2 b , σ 2 m ) corresponding to critical initialisation are given by σ 2 m = 1 / E [ (cid:0) φ (cid:48) ( √ q ∗ z ) (cid:1) 2 ] + E
[ φ 2 ( √ q ∗ z )] and ﬁnding σ 2 b that satisﬁes
[Equation]
This can be established by rearranging Equations 12 and 14.
We solve for σ 2 b numerically, as shown in Figure 4, for different neuron noise models and hence non-linearities φ ( · ) .
We ﬁnd that the critical initialisation for any of these design choices is close to the point ( σ 2 m , σ 2 b ) =
However, it is not just the singleton point, as for example in Hayou et al.
for the ReLu case for standard networks.
We plot the solutions in the Appendix.
3.2. A SYMPTOTIC EXPANSIONS AND DEPTH SCALES
The depth scales, as derived in Schoenholz et al.
(2016) provide a quantitative indicator to the number of layers correlations will survive for, and thus how trainable a network is.
Similar depth scales can be derived for these deterministic surrogates.
Asymptotically in network depth (cid:96) , we expect that | q (cid:96)aa − q ∗ | ∼ exp( − (cid:96)ξ q ) and | c (cid:96)ab − c ∗ | ∼ exp( − (cid:96)ξ c ) , where the terms ξ q and ξ c deﬁne the depth scales over which the variance and correlations of signals may propagate.
We are most interested in the correlation depth scale, since it relates to χ . The derivation is identical to that of Schoenholz et al.
One can expand the correlation c (cid:96)ab =
c ∗ + (cid:15) (cid:96) , and assuming q (cid:96)aa = q ∗ , it is possible to write
[Equation]
The depth scale ξ − 1 c are given by the log ratio log (cid:15) (cid:96) +
[Equation]
[Equation]
We plot this depth scale in Figure 2. We derive the variance depth scale in the appendices, since it is different to the standard continuous case, but not of prime practical importance
[Equation]
3. S IGNAL PROPAGATION THEORY FOR LOCAL REPARAMETERIZATION TRICK SURROGATES
From Equation 7, the pre-activation ﬁeld for the perturbed surrogate with both stochastic binary weights and neurons is given by,
[Equation]
where we recall that (cid:15) ∼ N (0 , 1) .
The non-linearity φ ( · ) can of course be derived from any valid binary stochastic neuron model.
Appealing to the same self-averaging arguments used in the previous section, we ﬁnd the variance map to be
[Equation]
Interestingly, we see that the variance map does not depend on the variance of the means of the binary weights.
This is a counter intuitive result, not immediately obvious from the pre-activation ﬁeld deﬁnition.
In the covariance map however, we do not have such simpliﬁcation, since the perturbation (cid:15) i,a in uncorrelated between inputs a and b , thus we recover Equation 9 similarly for the standard continuous case.
Thus the correlation map is given by
[Equation]
[Equation]
3. C RITICAL INITIALISATION : LRT SURROGATES
Claim 2: There is no critical initialisation for the local reparameterisation trick based surrogate, for a network with binary weights and neurons.
Proof: The conditions for a critical initialisation are that c ∗ = 1 to be a ﬁxed point and χ 1 = 1 .
No such ﬁxed point exists.
We have a ﬁxed point c ∗ = {+
+} if and only if σ 2 m = 1 E [ φ 2 ( h l − 1 j,a )] . Note that σ 2 m ≤ 1 .
For any φ ( z ) which is the mean of the stochastic binary neuron, the expectation E [ φ 2 ( z )] ≤ 1 .
For example, consider φ ( z )
=
tanh( κz ) for any ﬁnite 4 .
We also considered the LRT surrogate with continuous ( tanh( · ) ) neurons and stochastic binary weights.
The derivations are very similar to the previous case, as we show in the appendix. The variance and correlation maps are given by
[Equation]
This leads to the following result, Claim 3: The critical initialisation for the LRT surrogate, for the case of continuous tanh( · ) neurons and stochastic binary weights is the singleton ( σ 2 b , σ 2 m ) = Proof: From the correlation map we have a ﬁxed point c ∗ = 1 if and only if σ 2 m = 1 , by inspection. In turn, the critical initialisation condition χ 1 = 1 holds if E [( φ (cid:48)
( h l − 1 j,a )) 2 ] = 1 σ 2 m = 1 . Thus, to ﬁnd the edge of chaos, we need to ﬁnd a value of q aa = E φ 2 ( h l − 1 j,a ) + σ 2 b that satisﬁes this ﬁnal condition.
In the case that φ ( · ) = tanh( · ) , then the function ( φ (cid:48) ( h l − 1 j,a )) 2 ≤ 1 , taking the value 1 at the origin only, this requires q aa → 0 . Thus we have the singleton point ( σ 2 b , σ 2 m ) = (0 , 1) as the only solution.
N UMERICAL AND EXPERIMENTAL RESULTS 4. S IMULATIONS
We ﬁrst verify that the theory accurately predicts the average behaviour of randomly initialised networks.
We present simulations for the deterministic surrogate in Figure 1. We see that the average behaviour of random networks are well predicted by the mean ﬁeld theory.
Estimates of the variance and correlation are plotted, with dotted lines corresponding to empirical means and the shaded area corresponding to one standard deviation. Theoretical predictions are given by solid lines, with strong agreement for even ﬁnite networks.
Similar plots can be produced for the LRT surrogate.
In Appendix D we present the variance and correlation depth scales as a function of σ m , and different curves corresponding to different bias variance values σ b . [Figure] [Figure] scale ξ c .
4. T RAINING PERFORMANCE FOR DIFFERENT MEAN INITIALISATION σ 2 m
Here we test experimentally the predictions of the mean ﬁeld theory by training networks to overﬁt a dataset in the supervised learning setting, having arbitrary depth and different initialisations.
We consider ﬁrst the performance of the deterministic and LRT surrogates, not their corresponding binary networks.
We use the MNIST dataset with reduced training set size ( 50% ) and record the training performance (percentage of the training set correctly labeled) after 10 epochs of gradient descent over the training set, for various network depths L < 70 and different mean variances σ 2 m ∈ [0 , 1) .
The optimizer used was Adam Kingma & Ba (2014) with learning rate of 2 × 10 − 4 chosen after simple grid search, and a batch size of 64 .
We see that the experimental results match the correlation depth scale derived, which are overlaid as dotted curves. A proportion of 6 ξ c was found to indicate the maximum attenuation in signal strength before trainability becomes difﬁcult, for continuous networks.
The deterministic surrogate appears to share this, but a different scaling was found for the LRT surrogate.
The reason we see the trainability not diverging in Figure 2 is that training time increases with depth, on top of requiring smaller learning rates for deeper networks, as described in detail in Saxe et al.
The experiment here used the same number of epochs regardless of depth, meaning shallower networks actually had an advantage over deeper networks.
Note that this theory does not specify for how many steps of training the effects of critical initialisation will persist.
Therefore, the number of steps we trained the network for is an arbitrary choice, and thus the experiments validate the theory in a more qualitative than quantitative way.
Results were similar for other optimizers, including SGD, SGD with momentum, and RMSprop.
Note that these networks were trained without dropout, batchnorm or any other heuristics.
In Figure 3 we present the training performance for the deterministic surrogate and its stochastic binary counterpart. The results for a deterministic binary network were similar to a single Monte
Carlo sample.
Once again, we test our algorithms on the MNIST dataset and plot results afterepochs.
We see that the performance of the stochastic network matches more closely the performance of the continuous surrogate as the number of samples increases, from N = 5 to N =samples.
We can report that the number of samples necessary to achieve better classiﬁcation, at least for more shallow networks, appears to depends on the number of training epochs.
In some way, this is a sensible relationship, since during the course of training we might expect the means of the weights to polarise, moving closer to the bounds ± 1 .
Likewise, from experience continuous with neural networks, the neurons, which initially have zero mean pre-activations, are expected to “saturate” during training, that is, they become either always “on” ( +1 ) or “off” ( − 1 ).
A stochastic network being “closer” to deterministic would require fewer samples overall.
We can again report that this phenomena was observed.
D ISCUSSION
This ﬁrst study of two classes of surrogate networks, and the derivation of their initialisation theories has yielded results of practical signiﬁcance. Based on the results of Section 3, in particular Claims 1-3, we can offer the following advice. If a practitioner is interested in training networks with binary weights and neurons, one should use the deterministic surrogate, not the LRT surrogate, since the latter has no critical initialisation.
If a practitioner is interested in binary weights only,the LRT in this case does have a critical initialisation (and is the only choice from amongst these two classes of surrogate). Furthermore, both networks are critically initialised when σ 2 b → 0 and by setting the means of the weights to ± 1 .
Interesting results were uncovered for the binary neural networks corresponding to the trained surrogate.
It was seen that during training, when evaluating the stochastic binary counterparts concurrently with the surrogate, the performance of binary networks is worse than the continuous model, especially as depth increases.
We reported that the stochastic binary network, with more samples, outperformed the deterministic binary network. This makes sense since the objective optimised is the expectation over an ensemble of stochastic binary networks.
A study of random binary networks, included in the Appendices, and published recently Blumenfeld et al.
for a different problem, showed that binary networks are always in a chaotic phase.
Of course, when evaluating any binary network which is trained via gradient descent on a given surrogate model, signals have different average behaviour through the corresponding binary network.
It makes sense that the closer one is to the early stages of the training process, the closer the signal propagation behaviour is to the randomly initialised case.
It is likely that as training progresses the behaviour of the binary counterparts approaches that of the trained surrogate.
Any such difference would not be observed for a heuristic surrogate as used in Courbariaux & Bengio (2016) or Rastegari et al.
(2016), which has no continuous forward propagation equations.
A D ERIVATION OF DETERMINISTIC SURROGATE NETWORKS A. I NTEGRATING OVER STOCHASTIC OR DETERMINISTIC BINARY NEURONS
The form of each neuron’s probability distribution depends on the underlying noise model.
We can express a stochastic binary random variable S ∈ {± 1 } with S ∼ p ( S ; θ ) via its latent variable formulation,
[Equation]
In this form θ is referred to as a “natural” parameter, from the statistics literature on exponential families.
The term L is a latent random noise, which determines the form of the probability distribution.
We also introduce a scaling α to control the variance of the noise, so that as α → 0 the neuron becomes a deterministic sign function.
Letting α = 1 for simplicity, we see that the probability of the binary variable taking a positive value is
[Equation]
where p ( L ) is the known probability density function for the noise L .
The two common choices of noise models are Gaussian or logistic noise.
The Gaussian of course has shifted and scaled erf( · ) function as its cumulative distribution.
The logistic random variable has the classic “sigmoid” or logistic function as its CDF, σ ( z ) = [Equation] +
[Equation]
z . Thus, the probability of a the variable being positive is a function of the CDF.
In the Gaussian case, this is Φ( θ ) .
By symmetry, the probability of p ( S = − 1) =
Φ( − θ ) .
Thus, we see the probability distribution for the binary random variable in general is the CDF of the noise L , and we write p ( S ) =
Φ( S θ ) .
In the logistic noise case, we have p ( S ) =
σ ( S θ )
For the stochastic neurons, the natural parameter is the incoming ﬁeld h (cid:96)i = (cid:80) j S (cid:96)i,j x (cid:96) − 1 j + b (cid:96)i .
Assuming this is approximately Gaussian in the large layer width limit, we can successively marginalise over the stochastic inputs to each neuron, calculating an approximation of each neuron’s probability distribution, ˆ p ( x (cid:96)i ) .
This approximation is then used in the central limit theorem for the next layer, and so on.
For the case of neurons with latent Gaussian noise as part of the binary random variable model, the integration over the pre-activation ﬁeld (assumed to be Gaussian) is exact.
Explicitly,
[Equation]
where Φ( · ) is the CDF of the Gaussian distribution.
We have again Σ MF denoting the mean ﬁeld approximation to the covariance between the stochastic binary pre-activations.
The Gaussian expectation of the Gaussian CDF is a known identity, which we state in more generality in the next section, where we also consider neurons with logistic noise.
This new approximate probability distribution ˆ p ( x (cid:96)i ) can then used as part of the Gaussian CLT applied at the next layer, since it determines the means of the neurons in the next layer,
[Equation]
If we follow these setps from layer to layer, we see that we are actually propagating approximate means for the neurons, combined non-linearly with the means of the weights.
Given the approximately analytically integrated loss function, it is possible to perform gradient descent with respect to the means and biases, M (cid:96) ij and b (cid:96) i .
In the case of deterministic sign() neurons we obtain particularly simple expressions.
In this case the “probability” of a neuron taking, for instance, positive is just Heaviside step function of the incoming ﬁeld.
Denoting the Heaviside with Θ( · ) , we have
[Equation]
We can write out the network forward equations for the case of deterministic binary neurons, since it is a particularly elegant result.
In general we have
[Equation]
26) where φ ( · ) = erf( · ) is the mean of the next layer of neurons, being a scaled and shifted version of the neuron’s noise model CDF.
The constant is η = 1 √ 2 , standard for the Gaussian CDF to error functin conversion.
A. E XACT AND APPROXIMATE G AUSSIAN INTEGRATION OF SIGMOIDAL FUNCTIONS
We now present the integration of stochastic neurons with logistic as well as Gaussian noise as part of their latent variable models.
The logistic case is an approximation built on the Gaussian case, motivated by approximating the logistic CDF with the Gaussian CDF.
The reason we may be interested in using logistic CDFs, rather than just considering latent Gaussian noise models which integrate exactly, is not justiﬁed in any rigorous or experimental way.
Any such analysis would likely consider the effect of the tails of the logistic versus the Gaussian distributions, where the logistic tails are much heavier than those of the Gaussian.
One historic reason for considering the logistic function, we note, is the prevalence of logistic-type functions (such as tanh( · ) ) in the neural network literature.
The computational cost of evaluating either logistic or error functions is similar, so there is no motivation from the efﬁciency side.
Instead it seems a historic preference to have logistic type functions used with neural networks.
As we saw in the previous subsection, the integration over the analytic probability distribution for each neuron gave a function which allows us to calculate the means of the neurons in the next layer.
Therefore, we directly calculate the expression for the means.
The Gaussian integral of the Gaussian CDF was used in the previous section to derive the exact probability distribution for the stochastic binary neuron in the next layer.
The result is well known, and can be stated in generality as follows,
[Equation]
We can integrate a logistic noise binary neuron using this result as well.
The idea is to approximate the logistic noise with a suitably scaled Gaussian noise.
However, since the overall network approximation results in propagating means from layer to layer, we can equivalently need to approximate the tanh( · ) with the with the erf .
Speciﬁcally, if we have f ( x ; α ) = tanh( xα ) , an approximation is g ( x ; α ) = erf( √ π 2 α x ) , by requiring equality of derivatives at the origin.
In order to establish this, consider
[Equation]
and
[Equation]
Equating these, gives σ 2 = 4 α 2 π , thus σ = 2 α √ π .
The approximate integral over the stochastic binary neuron mean is then
[Equation]
If we so desire, we can approximate this again with a tanh( · ) using the tanh( · ) to erf( · ) approximation in reverse.
The scale parameter of this tanh( · ) will be α 2 =
π 4 αγ .
If α = 1 as is standard, then
[Equation]
B E QUIVALENCE OF DETERMINISTIC AND STOCHASTIC NEURONS FOR DETERMINISTIC SURROGATE
Assume a stochastic neuron with some latent noise, as per the previous appendix, with mean ¯ x (cid:96)i =
E p ( x i ) x
(cid:96)i = φ ( h (cid:96) − 1 i ) .
The ﬁeld is given by
[Equation] [Equation]
We see that the expression for the variance of the ﬁeld simpliﬁes as follows,
[Equation]
By similar steps, we ﬁnd that in the deterministic binary neuron case, we would obtain the same expression, albeit with a different scaling constant.
This is easily seen by inspection of the ﬁeld term in the deterministic neuron case,
[Equation]
which again was derived in the previous appendix.
C D ERIVATION OF SIGNAL PROPAGATION EQUATIONS IN DETERMINISTIC SURROGATE NETWORKS
Here we present the derivations for the signal propagation in the continuous network models studied in the paper.
C. V ARIANCE PROPAGATION
We ﬁrst calculate the variance given a signal:
[Equation]
Where for us:
[Equation]
and
[Equation]
Where
[Equation]
E φ 2 (cid:0) h l − 1 j,a (cid:1) can be written explicitly
[Equation]
taking into account that h l − 1 j,a ∼ N (0 , q aa ) :
[Equation]
We can now perform the following change of variable:
[Equation]
[Equation]
Then:
[Equation] [Equation]
In the ﬁrst layer, input neurons are not stochastic: they are samples drawn from the Gaussian distribution x 0 ∼ N (cid:0) 0 , q 0 (cid:1) :
C.1. C ORRELATION PROPAGATION
To determine the correlation recursion we start from its deﬁnition:
[Equation]
where q lab represents the covariance of the pre-activations h li,a and h li,b , related to two distinct input signals and therefore deﬁned as:
[Equation]
Replacing the pre-activations with their expressions provided in eq.
(40) and taking advantage of the self-averaging argument
[Equation]
we can then write:
[Equation]
At this point
[Equation] given that q laa and q lbb quite quickly approach the ﬁxed point [Equation] we can conveniently assume q laa =
q lbb .
Moreover, exploiting eq.(46) [Equation] we can ﬁnally write the expression for the correlation recursion:
[Equation]
D ERIVATION OF THE SLOPE OF THE CORRELATIONS AT THE FIXED POINT
To check the stability at the ﬁxed point, we need to compute the slope of the correlations mapping from layer to layer at the ﬁxed point:
[Equation]
where we get rid of σ b because independent from c l − 1 ab .
Replacing the deﬁnition of u a and u b provided in the continuous model, we can explicitly compute the derivative with respect to c l − 1 ab : [Equation]
[Equation]
where we have deﬁned A and B as:
[Equation]
We can focus on B ﬁrst.
Integrating by parts over z b we get:
[Equation]
Then, integrating by parts over z a
[Equation]
we the get:
[Equation]
Replacing A and B in eq.
[Equation]
we then obtain the closest expression for the stability at the variance ﬁxed point, namely:
[Equation]
[Equation]
C. V ARIANCE DEPTH SCALE
As pointed out in the main text
[Equation]
it should hold asymptotically that:
[Equation]
57) with ξ q deﬁning the variance depth scale.
To compute it we can expand over small perturbations around the ﬁxed point, namely:
[Equation]
Expanding the square root for small (cid:15) l [Equation] we can then write:
[Equation]
We can now expand the activation function φ around small perturbations and then computing the square getting rid of higher order terms in (cid:15) l , thus ﬁnally obtaining:
[Equation]
Comparing this expression with the one in eq.
[Equation]
we can then write:
[Equation]
Integrating by parts over z , we then obtain:
[Equation]
Given that it holds eq.
[Equation]
and noticing that χ evaluated at the correlation ﬁxed point c ∗ = 1 is given by:
[Equation]
we can ﬁnally get:
[Equation]
Given that we expect (57) to hold asymptotically
[Equation]
that is:
[Equation]
we can ﬁnally obtain the variance depth scale:
[Equation]
D S UPPLEMENTARY F IGURES D. E DGE OF CHAOS SIMULATIONS : DETERMINISTIC SURROGATE CASE
We see in Figure 4 that the edges exist in the plane, for but σ 2 b > 10 − 20 all the corresponding mean variances σ + m 2 > 1 which is not possible.
D. D EPTH SCALES
We see in Figure 5 the depth scales for the deterministic surrogate.
Note the divergence as one expects following the simulations in Figure 4. [Figure] [Figure]
D. J ACOBIAN MEAN SQUARED SINGULAR VALUE AND M EAN F IELD G RADIENT B ACKPROPAGATION
An alternative perspective on critical initialisation, to be contrasted with the forward signal propagation theory, is that we are simply attempting to control the mean squared singular value of the input-output Jacobain matrix of the entire network, which we can decompose into the product of single layer Jacobian matrices.
In standard networks, the single layer Jacobian mean squared singular value is equal to the derivative of the correlation mapping χ as established in Poole et al.
For the Gaussian model studied here this is not true, and corrections must be made to calculate the true mean squared singular value.
This can be seen by observing the terms arising from denominator of the pre-activation ﬁeld,
[Equation]
[Equation] Since Σ ii is a quantity that scales with the layer width N (cid:96) , it is clear that when we consider squared quantities, such as the mean squared singular value, the second term, from the derivative of the denominator, will vanish in the large layer width limit.
Thus the mean squared singular value of the single layer Jacobian approaches χ .
We will proceed as if χ is the exact quantity we are interested in controlling.
The analysis involved in determining whether the mean squared singular value is well approximated by χ essentially takes us through the mean ﬁeld gradient backpropagation theory as described in Schoenholz et al.
This idea provides complementary depth scales for gradient signals travelling backwards.
E R EPARAMETERISATION TRICK SURROGATE E. S IGNAL PROPAGATION EQUATIONS
We present, in slightly more detail, the signal propagation equations for the case of continuous neurons and stochastic binary weights yields the variance map,
[Equation]
Thus, once again, the variance map does not depend on the variance of the means of the binary weights.
The covariance map however does retain a dependence on σ 2 m ,
[Equation]
) with the same expression as before.
The correlation map is given by
[Equation]
and we have the derivative of the correlation map given by
[Equation]
E. D ETERMINING THE CRITICAL INITIALISATION CONDITIONS
We recount the argument from the paper here.
Since the mean variance σ 2 m does not appear in the variance map, we must once again consider different conditions for the edge of chaos.
Speciﬁcally, from the correlation map we have a ﬁxed point c ∗ = 1 if and only if
[Equation]
In turn, the edge of chaos condition χ 1 = 1 holds if
[Equation]
Thus, to ﬁnd the edge of chaos, we need to ﬁnd a value of q aa =
E φ 2 ( h l − 1 j,a ) + σ 2 b that satisﬁes this ﬁnal condition.
In the case that φ ( · ) = tanh( · ) , then the function ( φ (cid:48) ( h l − 1 j,a )) 2 ≤ 1 , taking the value 1 at the origin only, this requires q aa → 0 .
Thus the ‘edge of chaos’ is the singleton point ( σ 2 b , σ 2 m ) =
This is conﬁrmed by experiment, as we report in the subsequent sections.
It is of course possible to investigate this perturbed surrogate for different noise models.
For example, given different noise scaling κ , as in the previous chapter, there will be a corresponding σ 2 b that satisfy the edge of chaos condition.
We leave such an investigation to future work, given the case of binary weights and continuous neurons does not appear to be of a particular interest over the binary neuron case.
In the next section we present experiments conﬁrming the success of the ‘edge of chaos’ initialisation for the perturbed surrogate with continuous neurons.
F S IGNAL PROPAGATION OF BINARY NETWORKS F. F ORWARD SIGNAL PROPAGATION
In this neural network, it should be understood that all neurons are simply sign( · ) functions of their input, and all weights W (cid:96)ij ∈ {± 1 } are randomly distributed according to
[Equation]
thus maintaining a zero mean.
The pre-activation ﬁeld is given by
[Equation]
So, the length map is:
[Equation]
Interestingly, this is the same value as for the perturbed Gaussian with stochastic binary weights and neurons.
The covariance evolves as
[Equation]
(79) we again have a correlation map:
[Equation]
where as in the paper, u a =
[Equation]
We can ﬁnd this correlation in closed form.
First we rewrite our integral with h , for a joint density p ( h a , h b ) , and then rescale the h a such that the variance is 1, so that dh a = √ q aa
[Equation]
where p ( v a , v b ) is a joint with the same correlation c ab (which is now equal to its covariance), and the capital P ( v 1 , v 2 ) corresponds to the (cumulative) distribution function.
A standard result for standard bivariate normal distributions with correlation ρ ,
[Equation]
So we then have that
[Equation]
Thus the correlation map is:
[Equation]
Since, from before we have q aa = 1 + σ 2 b , we then obtain
[Equation]
Recall that sin − 1 (1) =
π 2 , so we have that c ∗ = 1 is a ﬁxed point always.
We will now derive its slope, denoted as χ =
∂c
(cid:96)ab ∂c (cid:96) − 1 ab , but by ﬁrst integrating over the φ () = sign() non-linearities, and then taking the derivative.
Now we are in a place to take the derivative :
[Equation]
We can see that the derivative χ diverges at c (cid:96)ab = 1 , meaning that there is no ‘edge of chaos’ for this system.
This of course means that correlations will not propagate to arbitrary depth in deterministic binary networks, as one might have expected.
F. S TOCHASTIC WEIGHTS AND NEURONS
We begin again with the variance map,
[Equation]
[Equation]
where in this the ﬁeld is given by
[Equation]
where x h l − 1 j,a denotes a stochastic binary neuron whose natural parameter is the pre-activation from the previous layer.
The expectation for the length map is deﬁned in terms of nested conditional expectations, since we wish to average over all random elements in the forward pass,
[Equation]
Once again, this is the same value as for the perturbed Gaussian with stochastic binary weights and neurons.
Similarly, the covariance map gives us,
[Equation]
with phi ( · ) being the mean function, or a shifted and scaled version of the cumulative distribution function for the stochastic binary neurons, just as in previous Chapters.
This expression is equivalent to the perturbed surrogate for stochastic binary weights and neurons, with a mean variance of σ 2 m = 1 .
Following the arguments for that surrogate, no edge of chaos exists.
F. S TOCHASTIC BINARY WEIGHTS AND CONTINUOUS NEURONS
In this case, as we show in the appendix, the resulting equations are
[Equation]
which are, once again, the same as for the perturbed surrogate in this case, with σ 2 m = 1 .
This means that this model does have an edge of case, at the point ( σ 2 m , σ 2 b ) =
F. C ONTINUOUS WEIGHTS AND STOCHASTIC BINARY NEURONS
Similar arguments to the above show that the equations for this case are exactly equivalent to the perturbed surrogate model.
This means that no edge of chaos exists in this case either.
G M ISCELLANEOUS COMMENTS G. R EMARK : V ALDITY OF THE CLT FOR THE FIRST LEVEL OF MEAN FIELD
A legitimate immediate concern with initialisations that send σ 2 m → 1 may be that the binary stochastic weights S (cid:96)ij are no longer stochastic, and that the variance of the Gaussian under the central limit theorem would no longer be correct.
First recall the CLT’s variance is given by
Var ( h (cid:96) ) =
(cid:80) j (1 − m 2 j x 2 j ) .
If the means m j → ± 1 then variance is equal in value to (cid:80) j m 2 j (1 − x 2 j ) , which is the central limit variance in the case of only stochastic binary neurons at initialisation.
Therefore, the applicability of the CLT is invariant to the stochasticity of the weights.
This is not so of course if both neurons and weights are deterministic, for example if neurons are just tanh() functions.