N EURAL NETWORKS ARE a priori BIASED TOWARDS B OOLEAN FUNCTIONS WITH LOW ENTROPY
Anonymous authors
Paper under double-blind review
A BSTRACT
Understanding the inductive bias of neural networks is critical to explaining their ability to generalise.
Here, for one of the simplest neural networks – a single-layer perceptron with n input neurons, one output neuron, and no threshold bias term – we prove that upon random initialisation of weights, the a priori probability P ( t ) that it represents a Boolean function that classiﬁes t points in { 0 , 1 } n as 1 has a remarkably simple form: P ( t ) = 2 − n for 0 ≤ t < 2 n .
Since a perceptron can express far fewer Boolean functions with small or large values of t (low “entropy”) than with intermediate values of t (high “entropy”) there is, on average , a strong intrinsic a-priori bias towards individual functions with low entropy.
Furthermore, within a class of functions with ﬁxed t , we often observe a further intrinsic bias towards functions of lower complexity.
Finally, we prove that, regardless of the distribution of inputs, the bias towards low entropy becomes monotonically stronger upon adding ReLU layers, and empirically show that increasing the variance of the bias term has a similar effect.
I NTRODUCTION
In order to generalise beyond training data, learning algorithms need some sort of inductive bias.
The particular form of the inductive bias dictates the performance of the algorithm.
For one of the most important machine learning techniques, deep neural networks (DNNs) (LeCun et al., 2015), sources of inductive bias can include the architecture of the networks, e.g. the number of layers, how they are connected, say as a fully connected network (FCN) or as a convolutional neural net (CNN), and the type of optimisation algorithm used, e.g. stochastic gradient descent (SGD) versus full gradient descent (GD).
Many further methods such as dropout (Srivastava et al., 2014), weight decay (Krogh & Hertz, 1992) and early stopping (Morgan & Bourlard, 1990) have been proposed as techniques to improve the inductive bias towards desired solutions that generalise well.
What is particularly surprising about DNNs is that they are highly expressive and work well in the heavily overparameterised regime where traditional learning theory would predict poor generalisation due to overﬁtting (Zhang et al., 2016).
DNNs must therefore have a strong intrinsic bias that allows for good generalisation, in spite of being in the overparameterised regime.
Here we study the intrinsic bias of the parameter-function map for neural networks, deﬁned in (ValleP´erez et al., 2018) as the map between a set of parameters and the function that the neural network represents.
In particular, we deﬁne the a-priori probability P ( f ) of a DNN as the probability that a particular function f is produced upon random sampling (or initialisation) of the weight and threshold bias parameters. The prior at initialization, P ( f ) , should inform the inductive bias of
SGD-trained neural networks, as long as SGD approximates Bayesian inference with P ( f ) as prior sufﬁciently well Valle-P´erez et al. We explain this connection further, and give some evidence supporting this behavior of SGD, in Appendix L. This supports the idea studying neural networks with random parameters Poole et al. ; Lee et al.
; Schoenholz et al. Garriga-Alonso et al. ; Novak et al. (2018) is not just relevant to ﬁnd good initializations for optimization, but also to understand their generalization.
A naive null-model for P ( f ) might suggest that without further information, one should expect that all functions are equally likely.
However, recent very general arguments (Dingle et al., 2018) based on the coding theorem from Algorithmic Information Theory (AIT) (Li et al., 2008) have instead suggested that for a wide range of maps M that obey a number of conditions such as being simple (they have a low Kolmogorov complexity K ( M ) ) and redundancy (multiple inputs map to the same output) then if they are sufﬁciently biased , they will be exponentially biased towards outputs of low Kolmogorov complexity.
The parameter-function map of neural networks satisﬁes these conditions, and it was found empirically (Valle-P´erez et al., 2018) that, as predicted in (Dingle et al., 2018), the probability P ( f ) of obtaining a function f upon random sampling of parameter weights satisﬁes the following simplicity-bias bound
[Equation]
where (cid:101) K ( f ) is a computable approximation of the true Kolmogorov complexity K ( f ) , and a and b are constants that depend on the network, but not on the functions.
It is widely expected that real world data is highly structured, and so has a relatively low Kolmogorov complexity (Hinton & Van Camp, 1993; Schmidhuber, 1997).
The simplicity bias described above may therefore be an important source of the inductive bias that allows DNNs to generalise so well (and not overﬁt) in the highly over-parameterised regime (Valle-P´erez et al., 2018).
Nevertheless, this bound has limitations.
Firstly, the only rigorously proven result is for the true Kolmogorov complexity version of the bound in the case of large enough K ( f ) .
Although it has been found to work remarkably well for small systems and computable approximations to Kolmogorov complexity (Valle-P ´ erez et al., 2018; Dingle et al., 2018), this success is not yet fully understood theoretically.
Secondly, it does not explain why models like DNNs are biased; it only explains that, if they are biased, they should be biased towards simplicity.
Also, the AIT bound is very general – it predicts a probability P ( f ) that depends mainly on the function, and only weakly on the network.
It may therefore not capture some variations in the bias that are due to details of the network architecture, and which may be important for practical applications.
For these reasons it is of interest to obtain a ﬁner quantitative understanding of the simplicity bias of neural networks.
Some work has been done in this direction, showing that inﬁnitely wide neural networks are biased towards functions which are robust to changes in the input (De Palma et al., 2018), showing that “ﬂatness” is connected to function smoothness (Wu et al., 2016), or arguing that low Fourier frequencies are learned ﬁrst by a ReLU neural network (Rahaman et al., 2018;
Yang & Salman, 2019).
All of these papers take some notion of “smoothness” as tractable proxy for the complexity of a function.
One generally expects smoother functions to be simpler, although this is clearly a very rough measure of the Kolmogorov complexity.
S UMMARY OF KEY RESULTS
In this paper we study how likely different Boolean functions, deﬁned as f : { 0 , 1 } n → { 0 , 1 } , are obtained upon randomly chosen weights of neural networks.
Our key results are aimed at ﬂeshing out with more precision and rigour what the inductive biases of (very) simple neural networks are, and how they arise For this, we study the prior distribution over functions P ( f ) , upon random initialization of the parameters, which reﬂects the inductive bias of training algorithms that approximate Bayesian inference (see Appendix L for a detailed explanation, and data on how well SGD follows this behaviour).
We focus our study on a notion of complexity, namely the “entropy,” H ( f ) , of a Boolean function f , deﬁned as the binary entropy of the fraction of possible inputs to f that f maps to 1 .
This quantity essentially measures the amount of class imbalance of the function, and is complementary to previous works studying notions of smoothness as a proxy for complexity.
In Section 4 we study a simple perceptron with no threshold bias term, and with weights w sampled from a distribution which is symmetric under reﬂections along the coordinate axes.
Let the random variable T correspond to the number of points in { 0 , 1 } n which that fall above the decision boundary of the network (i.e. T= |{ x ∈ { 0 , 1 } n : (cid:104) w, x (cid:105) > 0 }| ) upon i.i.d.
random initialisation of the weights.
We prove that T is distributed uniformly, i.e. P ( T = t )
= 2 − n for 0 ≤ t < 2 n .
Let F t be the set of all functions with T = t that the perceptron can produce and let | F t | be its size (cf.
Deﬁnition 3.4).
We expect | F t | for t ∼ 2 n − 1 (high entropy) to be (much) larger than | F t | for extreme values of t (low entropy).
The average probability of obtaining a particular function f which maps t inputs to 1 is 2 − n / | F t | .
The perceptron therefore shows a strong bias towards functions with low entropy, in the sense that individual functions with low entropy have, on average, higher probability than individual functions with high entropy.
In Section 4.3, we show that within the sets F t , there is a further bias, and in some cases this is clearly towards simple functions which correlates with Lempel-Ziv complexity (Lempel & Ziv, 1976; Dingle et al., 2018), as predicted in (Valle-P´erez et al., 2018).
In Section 4.4, we show that adding a threshold bias term to a perceptron signiﬁcantly increases the bias towards low entropy.
In Section 5.1, we provide a new expressivity bound for Boolean functions: DNNs with input size n , l hidden layers each with width n +2 n − 1 − log 2 l +1 and a single output neuron can express all 2 2 n Boolean functions over n variables.
In Section 5.2 we generalise our results to neural networks with multiple layers, proving (in the inﬁnite-width limit) that the bias towards low entropy increases with the number of ReLU-activated layers.
In Appendix J, we also show some empirical evidence that the results derived in this paper seem to generalize beyond the assumptions of our theoretical analysis, to more complicated data distributions (MNIST, CIFAR) and architectures (CNNs). Finally, in Appendix M, we show preliminary results on the effect of entropy-like biases in P ( f ) on learning class-imbalanced data.
D EFINITIONS , T ERMINOLOGY , AND N OTATION
Deﬁnition 3.1 (DNNs) .
Fully connected feed-forward neural networks with activations σ and a single output neuron form a parameterised function family f ( x ) on inputs x ∈ R n .
This can be deﬁned recursively, for L hidden layers for 1 ≤ l ≤ L , as
[Equation]
where 1 ( X ) is the Heaviside step function deﬁned as 1 if X > 0 and 0 otherwise, and σ is an activation function that acts element-wise.
The w l ∈ R n l + l × n l are the weights, and b l ∈ R n l +1 are the threshold bias weights at layer l , where n l is the number of hidden neurons in the l -th layer.
n L +1 is the number of outputs ( 1 in this paper), and n 0 is the dimension of the inputs (which we will also refer to as n ).
We will refer to the whole set of parameters ( w l and b l , 1 ≤ l ≤ L ) as θ .
In the case of perceptrons we use f θ ( x ) = σ ( (cid:104) w, x (cid:105) + b ) to specify a network.
We deﬁne the parameter-function map as in (Valle-P ´ erez et al., 2018) below.
Deﬁnition 3.2 (Parameter-function map) .
Consider a parameterised supervised model, and let the input space be X and the output space be Y .
The space of functions the model can express is
F ⊂ Y | X | .
If the model has p real valued parameters, taking values within a set Θ ⊆ R p , the parameter function map M is deﬁned
[Equation]
where f θ is the function corresponding to parameters θ .
In this paper we are interested in the Boolean functions that neural networks express.
We consider the 0 - 1 Boolean hypercube { 0 , 1 } n as the input domain.
Deﬁnition 3.3.
The function T ( f ) is deﬁned as the number of points in the hypercube { 0 , 1 } n that are mapped to 1 by the action of a neural network f .
For example, for a perceptron this function is deﬁned as,
[Equation]
We will sometimes use T ( w, b ) if the neural network is a perceptron.
Deﬁnition 3.4 ( F t and P ( t ) ) .
We deﬁne the set F t to be the set of functions expressible by some model M (e.g. a perceptron, a neural network) which all have the same value of T ( f ) ,
[Equation]
, where F M is the set of all functions expressible by M .
Given a probability measure P on the weights θ , we deﬁne the probability measure
[Equation]
We can also deﬁne T ( f ) and P ( T = t ) in the natural way for sets of input points other than { 0 , 1 } n , the context making clear what deﬁnition is being used.
Deﬁnition 3.5.
The entropy H ( f ) of a Boolean function f : { 0 , 1 } ∗ → { 0 , 1 } is deﬁned as H ( f )
= − p log 2 p − (1 − p ) log 2 (1 − p ) , where p =
T f / 2 n . It is the binary entropy of the fraction p of possible inputs to f that f maps to 1 or equivalently, the binary entropy of the fraction of 1 ’s in the right-hand column of the truth table of f .
Deﬁnition 3.6.
We deﬁne the Boolean complexity K Bool ( f ) of a function f as the number of binary connectives in the shortest Boolean formula that expresses f .
Note that Boolean complexity can be deﬁned in other ways as well.
For example, K Bool ( f ) is sometimes deﬁned as the number of connectives (rather than binary connectives) in the shortest formula that expresses f , or as the depth of the most shallow formula that expresses f .
These deﬁnitions tend to give similar values for the complexity of a given function, and so they are largely interchangeable in most contexts.
We use the deﬁnition above because it makes our calculations easier.
I NTRINSIC BIAS IN A PERCEPTRON ’ S PARAMETER - FUNCTION MAP
In this section we study the parameter-function map of the perceptron (Rosenblatt, 1958), in many ways the simplest neural network.
While it famously cannot express many Boolean functions – including XOR – it remains an important model system.
Moreover, many DNN architectures include layers of perceptrons, so understanding this very basic architecture may provide important insight into the more complex neural networks used today.
4. E NTROPY BIAS IN A SIMPLE PERCEPTRON WITH b = 0 ( NO THRESHOLD BIAS TERM )
Here we consider perceptrons f θ ( x ) = 1 ( (cid:104) w, x (cid:105) + b ) without threshold bias terms, i.e. b = 0 .
The following theorem shows that under certain conditions on the weight distribution, a perceptron with no threshold bias has a uniform P ( θ : T ( f θ )
= t ) .
The class of weight distributions includes the commonly used isotropic multivariate Gaussian with zero mean, a uniform distribution on a centred cuboid, and many other distributions.
The full proof of the theorem is in Appendix A.
Theorem 4.1.
For a perceptron f θ with b = 0 and weights w sampled from a distribution which is symmetric under reﬂections along the coordinate axes, the probability measure P ( θ : T ( f θ )
= t ) is given by
[Equation]
Proof sketch.
We consider the sampling of the normal vector w as a two-step process: we ﬁrst sample the absolute values of the elements, giving us a vector w pos with positive elements, and then we sample the signs of the elements.
Our assumption on the probability distribution implies that each of the 2 n sign assignments is equally probable, each happening with a probability 2 − n .
The key of the proof is to show that for any w pos , each of the sign assignments gives a distinct value of
T (and because there are 2 n possible sign assignments, for any value of T , there is exactly one sign assignment resulting in a normal vector with that value of T ).
This implies that, provided all sign assignments of any w pos are equally likely, the distribution on T is uniform.
A consequence of Theorem 4.1 is that the average probability of the perceptron producing a particular function f with T ( f ) =
t is given by
[Equation]
where F t denotes the set of Boolean functions that the perceptron can express which satisfy T ( f )
= t , and (cid:104)·(cid:105) t denotes the average (under uniform measure) over all functions f ∈ F t .
We expect | F t | to be much smaller for more extreme values of t , as there are fewer distinct possible functions with extreme values of t .
This would imply a bias towards low entropy functions.
By way of an example, | F 0 | = 1 and | F 1 | = n (since the only Boolean functions f a perceptron can express which satisfy T ( f ) = 1 have f ( x ) = 1 for a single one-hot x ∈ { 0 , 1 } n ), implying that (cid:104) P ( f ) (cid:105)
= 2 − n and (cid:104) P ( f ) (cid:105) 1 = 2 − n /n .
Nevertheless, the probability of functions within a set F t is unlikely to be uniform.
We ﬁnd that, in contrast to the overall entropy bias, which is independent of the shape of the distribution (as long as it satisﬁes the right symmetry conditions), the probability P ( f ) of obtaining function f within a set F t can depend on distribution shape.
Nevertheless, for a given distribution shape, the probabilities P ( f ) are independent of scale of the shape, e.g. they are independent of the variance of the Gaussian, or the width of the uniform distribution.
This is because the function is invariant under scaling all weights by the same factor (true only in the case of no threshold bias).
We will address the probabilities of functions within a given F t further in Section 4.3.
4. S IMPLICITY BIAS OF THE b = 0 PERCEPTRON
The entropy bias of Theorem 4.1 entails an overall bias towards low Boolean complexity.
In Theorem B.1 in Appendix B we show that the Boolean complexity of a function f is bounded by
[Equation]
Using Theorem 4.1 and Equation (4), we have that the probability that a randomly initialised perceptron expresses a function f of Boolean complexity k or greater is upper bounded by
[Equation]
Uniformly sampling functions would result in P ( K Bool ( f ) ≥ k )
≈ 1 − 2 k − 2 n
which for intermediate k is much larger than Equation (5).
Thus from entropy bias alone, we see that the perceptron is much more likely to produce simple functions than complex functions: it has an inductive bias towards simplicity.
This derivation is complementary to the AIT arguments from simplicity bias (Dingle et al., 2018; Valle-P ´ erez et al., 2018), and has the advantage that it also proves that bias exists, whereas AIT-based simplicity bias arguments presuppose bias.
To empirically study the inductive bias of the perceptron with b = 0 , we sampled over many random initialisations with weights drawn from Gaussian or uniform distributions and input size n = 7 .
As can be seen in Figure 1a and Figure 1b, the probability P ( f ) that function f obtains varies over many orders of magnitude.
Moreover, there is a clear simplicity bias upper bound on this probability, which, as predicted by Eq.
1, decreases with increasing Lempel-Ziv complexity ( K LZ ( f ) ) (using a version from (Dingle et al., 2018) applied to the Boolean functions represented as strings of bits, see Appendix E).
Similar behaviour was observed in (Valle-P´erez et al., 2018) for a FCN network.
Moreover it was also shown there that Lempel-Ziv complexity for these Boolean functions correlates with approximations to the Boolean complexity K Bool .
A one-layer neural network ( Figure 1c) shows stronger bias than the perceptron, which may be expected because the former has a much larger expressivity.
A rough estimate of the slope a in Eq.
1 from (Dingle et al., 2018) suggests that a ∼ log 2 ( N O ) /
max f ∈ O
( ˜ K ( f ))
where O is the set of all Boolean functions the model can produce, and N O is the number of such functions.
The maximum K ( f ) may not differ that much between the one layer network and the perceptron, but N O will be much larger in former than in the latter.
In Appendix D we also show rank plots for the networks from Figure 1.
Interestingly, at larger rank, they all show a Zipf like power-law decay, which can be used to estimate N O , the total number of
Boolean functions the network can express.
We also note that the rank plots for the perceptron with b = 0 with Gaussian or uniform distributions of weights are nearly indistinguishable, which may be because the overall rank plot is being mainly determined by the entropy bias.
[Figure]
4. B IAS WITHIN F t
In Figure 2 we compare a rank plot for all functions expressed by an n = 7 perceptron with b =to the rank plots for functions with T ( f ) = 47 and T ( f ) = 64 .
To deﬁne the rank, we order the functions by decreasing probability, and then the rank of a function f is the index of f under this ordering (so the most probable function has rank 1, the second rank 2 and so on).
The highest probability functions in F 64 have higher probability than the highest in F 47 because the former allows for simpler functions (such as 0101 .. ), but for both sets, the maximum probability is still considerably lower than the maximum probability functions overall.
In Appendix E we present further empirical data that suggests that these probabilities are bounded above by Lempel-Ziv complexity (in agreement with (Valle-P ´ erez et al., 2018)).
However, in contrast to Theorem 4.1 which is independent of the parameter distribution (as long as they are symmetric), the distributions within F t are different for the Gaussian and uniform parameter distributions, with the latter showing less simplicity bias within a class of ﬁxed t (see Appendix E.1).
[Figure]
In Appendix F, we give further arguments for simplicity bias, based on the set of constraints that needs to be satisﬁed to specify a function.
Every function f can be speciﬁed by a minimal set of linear conditions on the weight vector of the perceptron, which correspond to the boundaries of the cone in weight space producing f .
The Kolmogorov complexity of conditions should be close to that of the functions they produce as they are related to the functions in a one-to-one fashion, via a simple procedure.
In Appendix F, we focus on conditions which involve more than two weights, and show that within each set F t there exists one function with as few as 1 such conditions, and that there exists a function with as many as n − 2 such conditions.
We also compute the set of necessary conditions (up to permutations of the axes) explicitly for functions with small t , and ﬁnd that the range in the number and complexity of the conditions appears to grow with t , in agreement, with what we observe in Figure 2 for the range of complexities.
More generally, we ﬁnd that complex functions typically need more conditions than simple functions do.
Intuitively, the more conditions needed to specify a function, the smaller the volume of parameters that can generate the function, so the lower its a-priori probability.
4. E FFECT OF b ( THE THRESHOLD BIAS TERM ) ON P ( t )
We next study the behaviour of the perceptron when we include the threshold bias term b , sampled from N (0 , σ b ) , while still initialising the weights from N (0 , 1) , as in Section 4.1.
We present results for n = 7 in Figure 3.
Interestingly, for inﬁnitesimal σ b , P ( T = 0) is less than for b =
(See
Appendix C),
but then for increasing σ b it rapidly grows larger than 1 / 2 n and in the limit of large σ b asymptotes to 1 / 2 (see Figure 3b).
It’s not hard to see where this asymptotic behaviour comes from, a large positive or negative b means all inputs are mapped to true (1) or false (0) respectively. [Equation] selected
[Figure]
E NTROPY BIAS IN M ULTI - LAYER N EURAL N ETWORKS
We next extend results from Section 4 to multi-layer neural networks, with the aim to comment on the behaviour of P ( T = t ) as we add hidden layers with ReLU activations.
To study the bias in the parameter-function map of neural networks, it is important to ﬁrst understand the expressivity of the networks.
In Section 5.1, we produce a (loose) upper bound on the minimum size of a network with ReLU activations and l layers that is maximally expressive over Boolean functions.
We comment on how sufﬁciently large expressivity implies a larger bias towards low entropy for models with similarly shaped distribution over T (when compared to the perceptron).
In Section 5.2, we prove, in the limit of inﬁnite width, that adding ReLU activated layers causes the moments of P ( T = t ) to increase, .
This entails a lower expected entropy for neural networks with more hidden layers.
We empirically observe that the distribution of T becomes convex (with input { 0 , 1 } n ) with the addition of ReLU activated layers for neural networks with ﬁnite width.
5. E XPRESSIVITY CONDITIONS FOR DNN S
We provide upper bounds on the minimum size of a DNNs that can model all Boolean functions.
We use the notation (cid:104) n 0 , n 1 , . . .
, n L , n L +1 (cid:105) to denote a neural network with ReLU activations and of the form given in Deﬁnition 3.1.
Lemma 5.1.
A neural network with layer sizes (cid:104) n, 2 n − 1 , 1 (cid:105) , threshold bias terms, and ReLU activations can express all Boolean functions over n variables (also found in (Raj, 2018)).
See Appendix G for proof.
Lemma 5.2.
A neural network with l hidden layers, layer sizes (cid:104) n, ( n + 2 n − 1 /l + 1) , . . .
, ( n + 2 n − 1 /l + 1) , 1 (cid:105) , threshold bias terms, and ReLU activations can express all Boolean functions over n variables.
See Appendix G for proof.
Note that neither of these bounds are (known to be) tight.
Lemma 5.1 says that a network with one hidden layer of size 2 n − 1 can express all Boolean functions over n variables.
We know that a perceptron with n input neurons (and a threshold bias term) can express at most 2 n 2 Boolean functions ((Anthony, 2001), Theorem 4.3), which is signiﬁcantly less than the total number of Boolean functions over n variables, which is 2 2 n .
Hence there is a very large number of Boolean functions that the network with a (sufﬁciently wide) hidden layer can express, but the perceptron cannot.
The vast majority of these functions have high entropy (as almost all Boolean functions do).
Moreover, we observe that the measure P ( T = t ) is convex in the case of the more expressive neural networks, as discussed in section Section 5.2.
This suggests that the networks with hidden layers have a much stronger relative bias towards low entropy functions than the perceptron does, which is also consistent with the stronger simplicity bias found in Figure 1.
We further observe from Lemma 5.2 that the number of neurons can be kept constant and spread over multiple layers without loss of expressivity for a Boolean classiﬁer (provided the neurons are evenly spread across the layers).
5. H OW MULTIPLE LAYERS AFFECT THE BIAS
We next consider the effect of addition of ReLU activated layers on the distribution P ( t ) .
Of course adding even just one layer hugely increases expressivity over a perceptron.
Therefore, even if the distribution of P ( t ) would not change, the average probability of functions in a given F t could drop signiﬁcantly due to the increase in expressivity.
However, we observe that for inputs { 0 , 1 } n , P ( t ) becomes more convex when more ReLUactivated hidden layers are added, see Figure 4.
The distribution appears to be monotone on either side of t = 2 n − 1 and relatively ﬂat in the middle, even with the addition of 8 intermediate layers 2 .
In particular, we show in Figure 4 that for large number of layers, or large σ b , the probabilities for
P ( t = 0) (and by symmetry, in the inﬁnite width limit, also P ( t = 2 n ) ) each asymptotically reach 12 , and thus take up the vast majority of the probability weight.
We now prove some properties of the distribution P ( t ) for DNNs with several layers.
Lemma 5.3.
The probability distribution on T for inputs in { 0 , 1 } n of a neural network with linear activations and i.i.d.
initialisation of the weights is independent of the number of layers and the layer widths, and is equal to the distribution of a perceptron.
See Appendix H for proof.
While it is trivial that such a linear network has the same expressivity as a perceptron, it may not be obvious that the entropy bias is identical.
Lemma 5.4.
Applying a ReLU function in between each layer produces a lower bound on P ( T = 0) such that P ( T = 0) ≥ 2 − n .
See Appendix H for proof.
This lemma shows that a DNN with ReLU functions is no less biased towards the lowest entropy function than a perceptron is.
We prove a more general result in the following theorem which concerns the behaviour of the average entropy (cid:104) H ( t ) (cid:105) (where the average upon random sampling of parameters) as the number of layers grows.
The theorem shows that the bias towards low entropy becomes stronger as we increase the number of layers, for any distribution of inputs.
We rely See [Figure] on previous work that shows that in the inﬁnite width limit, neural networks approach a Gaussian process (Lee et al.
(2018); Garriga-Alonso et al.
; Novak et al.
; Matthews et al. Yang (2019)), which for the case of fully-connected ReLU networks, has an analytic form (Lee et al., 2018).
Theorem 5.5.
Let S be a set of m = | S | input points in R n .
Consider neural networks with i.i.d.
Gaussian weights with variances σ 2 w / √ n and biases with variance σ b , in the limit where the width of all hidden layers n goes to inﬁnity.
Let N
and N 2 be such neural networks with L and
L + 1 inﬁnitely wide hidden layers, respectively, and no bias.
Then, the following holds: (cid:104) H ( T ) (cid:105) is smaller than or equal for N 2 than for N 1 .
It is strictly smaller if there exist pairs of points in S with correlations less than 1 .
If the networks have sufﬁciently large threshold bias ( σ b > 1 is a sufﬁcient condition), the result above also holds.
For smaller bias, the result holds only for a sufﬁciently large number of layers.
See Appendix H for a proof of Theorem 5.5.
Theorem 5.5 is complementary to Theorem 4.1, in that the former only proves that the bias towards low entropy increases with depth, and the later proves conditions on the data that guarantee bias toward low entropy on the “base case” of 0 layers.
We show in Figure 4 that when σ b = 0 , the bias towards low entropy indeed becomes monotonically stronger as we increase the number of ReLU layers, for both inputs in { 0 , 1 } n as well as for centered data {− 1 , 1 } n .
For centered inputs {− 1 , 1 } n , the perceptron with b = 0 shows rather unusual behaviour.
The distribution is completely peaked around t = 2 n − 1 because every input mapping to 1 has the opposite input mapping to 0 .
Not surprisingly, its expressivity is much lower than the equivalent perceptron with { 0 , 1 } n (as can be seen in Figure 6a in Appendix D).
Nevertheless, in Figure 4d we see that as the number of layers increases, the behaviour rapidly resembles that of uncentered data (In Appendix K we also show that the bias toward low entropy is also recovered as we increase σ b ). So far this is the only exception we have found to the general bias to low entropy we observe for all other systems (see also Appendix J).
We therefore argue that this is a singular result brought about by particular symmetries of the perceptron with zero bias term. The fact that there is an exception does not negate our general result which we ﬁnd holds much more generally.
The insets of in Figure 4 show that the two trivial functions asymptotically dominate in the limit of large numbers of layers.
We note that recent work ((Lee et al., 2018; Luther & Seung, 2019)) has also pointed out that for fully-connected ReLU networks in the inﬁnite-width inﬁnite-depth limit, all inputs become asymptotically correlated, so that the networks will tend to compute the constant function.
Here we give a quantitative characterisation of this phenomenon for any number of layers.
Some interesting recent work (Yang & Salman, 2019) has shown that certain choices of hyperparameters lead to networks which are a priori unbiased, that is the P ( f ) appears to be uniform.
In Appendix I we show that this result is due to a choice of hyperparameters that lie deep in the chaotic region deﬁned in (Poole et al., 2016).
The effect therefore depends on the choice of activation function (it can occur for say tanh and erf, but most likely not ReLU), and we are studying it further.
D ISCUSSION AND FUTURE WORK
In Section 4 Theorem 4.1, we have proven the existence of an intrinsic bias towards Boolean functions of low entropy in a perceptron with no threshold bias term, such that P ( T = t ) =
2 − n for 0 ≤ t < 2 n .
This result puts an upper bound on the probability that a perceptron with no threshold bias term will be initialised to a Boolean function with at least a certain Boolean complexity.
Adding a threshold term in general increases the bias towards low entropy.
We also study how the entropy bias is affected by adding a threshold bias term or ReLU-activated hidden layers.
One of our main results, Theorem 5.5, proves that adding layers to a feed-forward neural network with ReLU activations makes the bias towards low entropy stronger.
We also show empirically that the bias towards low entropy functions is further increased when a threshold bias term with high enough variance is added.
Recently, (Luther & Seung, 2019) have argued that batch normalisation (Ioffe & Szegedy, 2015) makes ReLU networks less likely to compute the constant function (which has also been experimentally shown in (Page, 2019)).
If batch norm increases the probability of high entropy functions, it could help explain why batch norm improves generalisation for (typically class balanced) datasets.
We leave further exploration of the effect of batch normalisation on a-priori bias to future work.
Simplicity bias within the set of constant t functions F t is affected by the choice of initialisation, even when the entropy bias is unaffected.
This indicates that there are further properties of the parameter-function map that lead to a simplicity bias.
In Section 4.3, we suggest that the complexity of the conditions on w producing a function should correlate with the complexity of the function, and we conjecture that more complex conditions correlate with a lower probability.
We note that the a priori inductive bias we study here is for a randomly initialised network.
If a network is trained on data, then the optimisation procedure (for example SGD) may introduce further biases.
In Appendix L, we give some evidence that the bias at initialization is the main driver of the inductive bias on SGD-trained networks. Furthermore, in Appendix M, we show preliminary results on how bias in P ( f ) can affect learning in class-imbalanced problems. This suggests that understanding properties of P ( f ) (like those we study in this paper), can help design architectures with desired inductive biases.
Simplicity bias in neural networks (Valle-P ´ erez et al., 2018) offers an explanation of why DNNs work in the highly overparameterised regime.
DNNs can express an unimaginably large number of functions that will ﬁt the training data, but almost all of these will give extremely poor generalisation.
Simplicity bias, however, means that a DNN will preferentially choose low complexity functions, which should give better generalisation.
Here we have shown some examples where changing hyperparameters can affect the bias further.
This raises the possibility of explicitly designing biases to optimise a DNN for a particular problem.
A P ROOF OF UNIFORMITY
For convenience we repeat some notation we use in this section.
Let { 0 , 1 } n be the set of vertices of the n -dimensional hypercube.
We use (cid:104)· , ·(cid:105) to refer to the standard inner product in R n .
Deﬁne the function T : R n → N as the number of vertices of the hypercube that are above the hyperplane with normal vector w and that passes through the origin.
Formally T ( w ) =
| x ∈ { 0 , 1 } n : (cid:104) w, x (cid:105) > 0 | .
We use (cid:12) for element-wise multiplication of two vectors.
We slightly abuse notation and denote the probability density function corresponding to a probability measure P , with the same symbol, P .
The arguments of the function or context should make clear which one is meant.
Proof strategy.
We consider the sampling of the normal vector w as a two-step process: we ﬁrst sample the absolute values of the elements, giving us a vector w pos with positive elements 3 , and then we sample the signs of the elements.
Our assumption on the probability distribution implies that each of the 2 n sign assignments is equally probable, each happening with a probability 2 − n .
The key of the proof is to show that for any w pos , each of the sign assignments gives a distinct value of
T (and because there are 2 n possible sign assignments, for any value of T , there is exactly one sign assignment resulting in a normal vector with that value of T ).
This implies that, provided all sign assignments of any w pos are equally likely, the distribution on T is uniform.
Theorem 4.1 Let P be a probability measure on R n , which is symmetric under reﬂections along the coordinate axes, so that P ( x ) = P ( Rx ) , where R is a reﬂection matrix (a diagonal matrix with elements in {− 1 , 1 } ).
Let the weights of a perceptron without bias, w , be distributed according to
P .
Then P ( T = t ) is the uniform measure.
Before proving the theorem, we ﬁrst need a deﬁnition and a lemma.
Deﬁnition We deﬁne the function mapping a vector from {− 1 , 1 } n (which we interpret as the signature of the weight vector), and a vector of nonnegative reals (which we interpret as the absolute values of the elements of the weight vector) to the value of t of the corresponding weight vector:
[Equation]
Lemma A.1.
The function K is bijective with respect to its ﬁrst argument, for any value of its second argument except for a set of measure 0 .
Proof of Lemma A.1.
Because the cardinality of the codomain of K is the same as the domain of its ﬁrst argument, it is enough to prove injectivity of K with respect to its ﬁrst argument.
Fix a ∈ R n ≥ 0 satisfying that the following set has cardinality 3 n :
[Equation]
Note that the set of a in which some pair of elements in the deﬁnition of S a is equal has measure zero, because their equality implies that a lies within a hyperplane in R n .
Let us also deﬁne the set of subsums of elements of a :
[Equation]
which has cardinality 2 n for the a considered.
Now, consider a natural bijection J a : {− 1 , 1 } n → S a induced by the bijective mapping of signatures σ ∈ {− 1 , 1 } n to vertices of the hypercube { 0 , 1 } n by mapping − 1 to 0 and 1 to 1 .
To be more precise, J a ( σ ) = (cid:80)
ni
=
a i ( σ ( i )+1) .
Then, we claim that
[Equation]
This implies that for the a we have ﬁxed K is injective, for if two σ mapped to the same value, their corresponding value of J a ( σ ) should be the same, giving a contradiction.
So it only remains to prove equation 7.
Let us also ﬁrst denote, for x ∈ { 0 , 1 } n and s, s (cid:48)
∈ S
a
[Equation]
where we interpret elements of { 0 , 1 } n as subsets of { 1 , ..., n } .
The notation above lets us interpret subsums in S a as subsets of entries of a .
Note that Σ − 1 is well deﬁned for the ﬁxed a we are considering.
Now, let σ ∈ {− 1 , 1 } n , s (cid:48)
=
J a ( σ ) , and consider an s ∈ S a such that s < s (cid:48) .
Then s ∩ s (cid:48)
+ s ∩ ¯ s (cid:48)
=
s < s (cid:48)
so
[Equation]
Now, let the operation ∗ (we omit dependence on s (cid:48) ) be deﬁned for any u ∈ S
a as u ∗ := u
∩ ¯ s (cid:48)
+ ( s (cid:48) − u ∩ s (cid:48) )
∈ S a .
Since s (cid:48)
=
J a ( σ ) we have
[Equation]
Using equation 8,
[Equation]
Therefore, all the points Σ − 1 ( s ∗ ) for s < s (cid:48) are above the hyperplane with normal ( σ (cid:12) a ) , and all points Σ − 1 ( s ∗ ) for s ≥ s (cid:48) are below or precisely on the hyperplane.
All that is left is to show the converse, all points which are above the hyperplane are Σ − 1 ( s ∗ ) for one and only one s < s (cid:48) .
It sufﬁces to show that the operation ∗ is injective for all s (as bijectivity follows from the domain and codomain being the same).
By contradiction, let s and u map to the same value under ∗ , then s ∩ s (cid:48) − s ∩ ¯ s (cid:48) =
u ∩ s (cid:48) − u
∩ ¯ s (cid:48) , which implies s ∩ s (cid:48)
=
u ∩ s (cid:48) and s ∩ ¯ s (cid:48)
=
u
∩ ¯ s (cid:48) , for the a we are considering, and so s = u .
Therefore ∗ is injective, and equation 7 follows.
Proof of Theorem 4.1.
[Equation]
Now, we can divide the integral into the quadrants corresponding to different signatures of w , and we can let P ( w ) = 12 n ˜ P ( | w | ) , because it is symmetric under reﬂections of the coordinate axes.
[Equation]
The third equality follows from Lemma A.1.
Indeed, bijectivity implies that for any a , except for a set of measure 0 , there is one and only one signature which results in t (cid:48) .
B B OUNDING B OOLEAN FUNCTION COMPLEXITY , K Bool , WITH t
Theorem B.1.
n × min( t, 2 n − t ) − 1 is an upper bound on the complexity of Boolean functions f for which T ( f ) = t .
Proof.
Let f : { 0 , 1 } n → { 0 , 1 } be a function
s.t. T ( f ) = t .
Let x 1 . . .
x n be propositional variables and let each assignment to x 1 . . .
x n correspond to a vector in { 0 , 1 } n in the straightforward way.
Let φ be the Boolean formula (cid:87) { (cid:86) { if v i = 1 then x i else ¬ x i | v i ∈ v } | v ∈ { 0 , 1 } n , f ( v ) = 1 } .
The formula φ expresses f as a Boolean formula in Disjunctive Normal Form (DNF).
Let ψ be the Boolean formula (cid:86) { (cid:87) { if v i =
then x i else ¬ x i | v i ∈ v } | v ∈ { 0 , 1 } n , f ( v ) = 0 } .
The formula ψ expresses f as a Boolean formula in Conjunctive Normal Form (CNF).
Since f maps t out of the 2 n vectors in { 0 , 1 } n to 1 it must be the case that φ has t clauses and ψ has 2 n − t clauses.
Each clause contains n − 1 binary connectives, and there is one connective between each clause.
Hence φ contains n × t − 1 binary connectives and ψ contains n × (2 n − t ) − 1 binary connectives.
Therefore f is expressed by some Boolean formula of complexity n × min( t, 2 n − t ) − 1 .
Since f was chosen arbitrarily, if a function f : { 0 , 1 } n → { 0 , 1 } maps t inputs to 1 then the complexity of f is at most n × min( t, 2 n − t ) − 1 .
Theorem B.2.
Let C be a deﬁned recursively as follows;
[Equation]
(cid:98) t/ 2 (cid:99) )
+
Then C( n, t ) is an upper bound on the complexity of Boolean functions f for which T ( f ) = t .
Proof.
Let P( n ) be that C ( n, t ) is an upper bound on the complexity of Boolean functions f over n variables
s.t. T ( f ) = t .
Base case P( 1 ): If φ is a Boolean formula deﬁned over 1 variable then φ is equivalent to True, False, x 1 , or ¬ x 1 .
We can see by exhaustive enumeration that P( 1 ) holds in each of these four cases.
Inductive step P( n ) → P( n + 1 ): Let φ be a Boolean formula deﬁned over n + 1 variables s.t.
T ( φ ) = t .
Case 1.
t = 0 : If t = 0 then φ ≡ False, and so the complexity of φ is 0 .
Hence the inductive step holds.
Case 2.
t = 2 n : If t = 2 n then φ ≡ True, and so the complexity of φ is 0 .
Hence the inductive step holds.
Case 3.
t = 1 : If t = 1 then φ has just a single satisfying assignment.
If this is the case then φ can be expressed as a formula of length n written in Disjunctive Normal Form, and hence the inductive step holds.
Case 4.
t = 2 n − 1 : If t = 2 n − 1 then φ has just a single non-satisfying assignment.
If this is the case then φ can be expressed as a formula of length n written in Conjunctive Normal Form, and hence the inductive step holds.
Case 5.
1 < t and t < 2 n − 1 : If φ is a Boolean formula deﬁned over n + 1 variables then φ is logically equivalent to a formula ( x n +1 ∧ ψ 1 ) ∨ ( ¬ x n +1 ∧ ψ 2 ) , where ψ 1 and ψ 2 are deﬁned over x 1 . . .
x n .
Let t 1 be the number of assignments to x 1 . . .
x n that are mapped to 1 by ψ 1 , and let tbe the corresponding value for ψ 2 .
By the inductive assumption the complexity of ψ 1 and ψ 2 is bounded by C( n, t 1 ) and C( n, t 2 ) respectively.
Therefore, since φ ≡ ( x n +1 ∧ ψ 1 ) ∨ ( ¬ x n +1 ∧ ψ 2 ) it follows that the complexity of φ is at most C ( n, t 1 )
+ C ( n, t 2 ) + 2 .
Since t 1 + t 2 = t , and since C ( n, t a ) < C ( n, t b ) if t b is closer to 2 n − 1 than t a is (lemma B.3), it follows that the complexity of φ is bounded by
C ( n, t ) =
C ( n − 1 , (cid:100) t/ 2 (cid:101) )
+ C ( n − 1 , (cid:98) t/ 2 (cid:99) )
+ 2 .
Since Case 1 to 5 are exhaustive the inductive step holds.
Lemma B.3.
If t + 1 ≤ 2 n − 1 then C ( n, t ) < C ( n, t + 1) .
Proof.
Let P( n ) be that if t + 1 ≤ 2 n − 1 then C ( n, t ) < C ( n, t + 1) .
Inductive step P( n ) → P( n + 1 ):
Case 1.
t is even :
[Equation]
If t is even and t + 1 ≤ 2 ( n +1) − 1 then t/ 2 + 1 ≤ 2 n − 1 .
Hence C ( n, t/ 2) − C ( n, t/ 2 + 1) < 0 by the inductive assumption, and so C ( n + 1 , t ) − C ( n + 1 , t + 1) < 0 .
Case 2.
t is odd :
[Equation]
If t is odd and t +1 ≤ 2 ( n +1) − 1 then ( t +1) / 2 ≤ 2 n − 1 .
Hence C ( n, ( t − 1) / 2) − C ( n, ( t +1) / 2) <by the inductive assumption, and so C ( n + 1 , t ) − C ( n + 1 , t + 1) < 0 .
Since Case 1 and 2 are exhaustive the inductive step holds.
[Equation]
t = 0) FOR PERCEPTRON WITH INFINITESIMAL b
If b is sampled uniformly from [ − (cid:15), (cid:15) ] , then only if |(cid:104) w, x (cid:105)| < (cid:15) can some x be classiﬁed differently from a perceptron without a threshold bias term.
The set of weight vectors which change the classiﬁcation of non-zero x becomes vanishingly small as (cid:15) goes to 0 , but for x = 0 , we have
P ( 1 ( (cid:104) w, 0 (cid:105) + b ) = 0) =
P ( 1 ( (cid:104) w, 0 (cid:105) + b ) = 1) = 1 / 2 . Consider some function f , and deﬁne g where f → g under the addition of an inﬁnitesimal bias.
Then with even probability the origin remains mapped to 0 (meaning T ( g )
=
T ( f ) ), or is mapped to 1 (meaning T ( g )
=
T ( f ) + 1 ) as the rest of f is unchanged, to O ( (cid:15) ) .
As this is true of all f , P ( T = t ) b ∼ ( − (cid:15),(cid:15) )
= 12 P ( T = t )+ 12 P ( T = t − 1) , leading to:
P ( T = t ) = (cid:26) 2 − ( n +1) if t = 0 or t = 2 n 2 − n otherwise .
For larger σ b P ( t = 0) or P ( t = 2 n ) increases with increasing σ b as can be seen in Figure 3 of the main text.
D Z IPF ’ S LAW IN A PERCEPTRON WITH b =
In (Valle-P´erez et al., 2018) it was shown empirically that the rank plot for a simple DNN exhibited a Zipf like power law scaling for larger ranks.
Zipf’s law occurs in many branches of science (and probably for many reasons).
In this section we check whether this scaling also occurs for the
Perceptron.
In Figure 5, we compare a rank plot of the probability P ( f ) for individual functions for the simple perceptron with b = 0 , the perceptron, and for a one layer FCN.
While all architectures have n = 7 , the perceptrons can of course express far fewer functions.
Nevertheless, both the perceptrons and the more complex FCN show similar phenomenology, with a Zipf law like tail at larger ranks (i.e. a power law).
[Figure]
While the orginal formulations for Zipf’s law only allows for a powerlaw with exponent 1, in practice the terminology of Zipf’s law is used for other powers, such that p =
b × rank − a for some positive a, b .
If we assume that this scaling persists, then we can relate the total number of functions a perceptron can express, to the constant b , because the total probability must integrate to 1 .
For the simplest case with a = 1 , this leads to an equation for the probability as function of rank given by
[Equation]
where N 0 is the total number of functions expressible.
The FCN appears to show such simple scaling.
And as the FCN of width 64 is fully expressive (see Lemma 5.1) , there are N O = 2 2
≈ 3 × 10 38 possible Boolean functions.
We plot the Zipf law prediction of Equation (10) next to the empirically estimated probabilities in Figure 5c.
We observe that the curve is described well at higher values of the rank Zipf’s law.
Note that the mean probability uniformly sampled over functions for this FCN is (cid:104) P ( f ) (cid:105)
= 1 /N
O ≈ 3 × 10 − 39 so that we only measure a tiny fraction of the functions with extremely high probabilities, compared to the mean.
Also, most functions have probabilities less than the mean, and only order 2 − n have probability larger than the mean.
A least-squares linear ﬁt on the log-log graph was consistent within experimental error for the ansatz.
For the perceptron with a bias term Figure 5b, we observe that the gradient differs substantially from − 1 , and a linear ﬁt gives log 10 ( p ) = − 0 .
85 log 10 ( rank ) − 2 . 32 .
Using the same arguments for calculating N O as made in Equation (10), we obtain a prediction of N
= 7 .
63 × 10 9 , which is 91% [Figure] of the known value 4 .
A linear ﬁt for the perceptron with no threshold bias term gives log 10 ( p ) = − 0 .
81 log 10 ( rank ) − 2 . 31 , leading to a prediction of N 0 = 3 .
97 × 10 8 .
which is, as expected, signiﬁcantly lower than a perceptron with no threshold bias term.
We expect there to be some discrepancy between the pure Zipf law prediction, and the true N O , because the probability seems to deviate from the Zipf-like behaviour at the highest rank, which we observe for n = 5 in Figure 6a, as in this case the number of functions is small enough that it becomes feasible to sample all of them.
It is also worth mentioning that a rank-probability plot for a perceptron with weights sampled from a uniform distribution (Figure 5d) is almost indistinguishable from the Gaussian case (Figure 5b), which is interesting, because when plotted against LZ complexity, as in Figure 1 of the main text, there is a small but discernible difference between the two types of initialisation.
Finally, in Figure 6a we compare the rank plot for centered and uncentered data for a smaller n = 5 , σ b = 0 perceptron where we can ﬁnd all functions.
Note that for the centered data, only functions with t = 16 can be expressed, which is somewhat peculiar, and of course means signiﬁcantly less functions.
Nevertheless, this systems still has clear bias within this one entropy class, and this bias correlates with the LZ complexity (Figure 6b) as also observed for the perceptron with centered data.
E F URTHER RESULTS ON THE DISTRIBUTION WITHIN F t
In this appendix we will denote the output function of the perceptron evaluated on { 0 , 1 } n by way of a bit string f , whose i ’th bit is given by
[Equation]
where bin ( i ) takes an integer i and maps it to a point x ∈ { 0 , 1 } n according to its binary representation (so bin (5) = (1 , 0 , 1) and bin (1) =
(0 , 0 , 1) when n = 3 ).
E. E MPIRICAL RESULTS
We sample 10 8 initialisations of the perceptron, divide the list of functions into F t , and present probability-complexity plots for several values of t in Figure 7.
We use the Lempel-Ziv complexity (Lempel & Ziv, 1976; Dingle et al., 2018) of the output bit string as the approximation to the Kolmogorov complexity of the function (Dingle et al., 2018).
As in (Dingle et al., 2018; Valle-P´erez et al., 2018), this complexity measure is denoted K LZ ( f ) .
To avoid ﬁnite size effects (as noted in (Valle-P´erez et al., 2018)), we cut off all frequencies less than or equal to 2 .
[Figure]
As can be seen in Figure 1 the probability-complexity graph satisﬁes the simplicity bias bound Equation (1) for all functions.
Now, in Figure 7) we observe subsets F f of the overall set of functions.
Firstly, we observe, as expected, that smaller t means a smaller range in K LZ , since high complexity functions are not possible at low entropy.
Conversely, low complexity functions are possible at high entropy (say for 010101...), and so a larger range of probabilities and complexities is observed for t = 64 .
For these larger entropies, an overall simplicity bias within the set of ﬁxed t can be observed.
The larger range observed in t = 64 and t = 16 (compared to t = 41 and t = 9 ) can be explained by the presence of highly ordered functions having those t values - for example, in t = 64 , there is f = 0101 . . .
and its symmetries; and in t = 16 there are functions such as f = 00010001 .
and its symmetries.
The other two t values do not divide 2 7 , so there will be no functions with such low block entropy (implying low K LZ ).
We also demonstrate differences in the variation in P ( f ) vs K LZ ( f ) when w is sampled from uniform distributions, in Figure 8, and compare these pots to those in Figure 7.
Whilst we know from Theorem 4.1 that sampling w from a uniform distribution will not affect P ( t ) , it is not hard to see that there will be some variation in function probability within the classes F t .
We observe that the simple functions which have high probability for t = 64 when the perceptron is initialised from a Gaussian (Figure 7d) have lower probabilities in the uniform case (Figure 8d).
We comment further on this behaviour in Appendix E.2.
However, we see limited differences in their respective rank-probability plots ( Figure 5a and Figure 5d).
[Figure]
E. S UBSTRUCTURE WITHIN { 0 , 1 } n
Consider a subset H m ⊂ { 0 , 1 } n such that 0 ∈ H m .
We have (cid:0) nm (cid:1) such subsets.
Then the marginal distribution over H m is given by
[Equation]
This is again independent of the distribution of w (provided it’s symmetric about coordinate planes).
We give two example applications of Equation (12) in Equation (13).
We use ∗ to mean any allowed value, and sum over all allowed values,
[Equation]
We can apply the same argument that we applied in Section 4 to any set of bits in f deﬁned by some
H m , to show that there is an “entropy bias” within each of these substrings.
However, these identities imply a strong bias within each F t .
For the case of full expressivity, assuming each value of t has probability 2 − n , and every string with the same value of t is equally likely, one gets probabilities very close to those in Equation (13) (although slightly lower).
However, the perceptron is not fully expressive, so it
It is difﬁcult to calculate the exact probabilities of any function for Gaussian initialisation 5 .
It may not be possible to ﬁne-grain probabilities analytically further than Equation (12) (although we can use the techniques in Section 4.3 to come up with analytic expressions for P ( f ) for all f ).
However, we can calculate some probabilities quite easily when w is sampled from a uniform distribution, w ∼ U n ( − 1 , 1) .
By way of example, we calculate P ( f = (cid:101) f ) for (cid:101) f = ”0101 . . . 0101” .
The conditions for (cid:101) f are as follows: w n > 0 , w i < 0 ∀
i (cid:54) = n and (cid:80)
j w j > 0 ,
so
[Equation]
Using Equation (3), we can calculate how much more likely the function is than expected,
[Equation]
For n = 5 , we can calculate Equation (15) using 6 | F t =2 n − 1 | = 370 and w ∼ U n ( − 1 , 1) and obtain P ( f = (cid:101) f ) / (cid:104) P ( f t =2 n − 1 ) (cid:105)
= 3 .
This clearly shows that (cid:101) f is signiﬁcantly more likely than expected just by using results from Equation (3).
Empirical results further suggest that for Gaussian initialisation of w , P ( f = (cid:101) f ) / (cid:104) P ( f t =2 n − 1 ) (cid:105)
≈ 10 .
This, plus data in Appendix E.1 suggest that
Gaussian initialisation may lead to more simplicity bias.
F T OWARDS UNDERSTANDING THE SIMPLICITY BIAS OBSERVED WITHIN F t
Here we offer some intuitive arguments that aim to explain why there should be further bias towards simpler functions within F t .
We ﬁrst need several deﬁnitions.
We deﬁne a set A ⊂ R n ≥ 0 such that a ∈
A iff a i < a j ∀ i < j , interpreted as the absolute values of the weight vector.
We now deﬁne four sets, Γ Σ and Υ , which classify the types of linear conditions on the weights of a perceptron acting on an n -dimensional hypercube:
Γ , the set of permutations in { 1 , . . . , n } (which we can interpret as permutations of the axes in R n or as possible orders of the absolute values of the weights if no two of them are equal).
Σ = {− 1 , +1 } n (which is interpreted as the signature of the weight vector) 3.
Υ is the set of linear inequality conditions on components of a ∈ A , which include more than two elements of a (so they exclude the conditions deﬁning A .
A unique weight vector can be speciﬁed giving its signature σ ∈ Σ , the order of its absolute values γ ∈ Γ , and a value of a ∈ A .
On the other hand, a unique function can be speciﬁed by a set of linear conditions on the weight vector w .
These conditions can be divided into three types: Σ (signs), Γ (ordering), and Υ (any other condition).
The intuition to understand the variation in complexity and probability over different functions is the following.
Each function corresponds to a unique set of necessary and sufﬁcient conditions on the weight vector (corresponding to the faces of the cone in weight space producing that function).
We argue that different functions have conditions which vary a lot in complexity, and we conjecture that this correlates with their probability, as we discuss in Section 4.3 in the main text.
As a ﬁrst approach in understanding this, we consider the role of symmetries under permutations of dimensions.
Any string that is symmetric under a permutation of k dimensions 7 , can’t have necessary conditions that represent relative orderings of those k dimensions.
Furthermore the set of 5 Except conditions must be invariant under these permutations.
This strongly constraints the sets of conditions that highly symmetric strings like “010101...” or “11111...00000...” can have, to be relatively simple sets.
We now consider the set of necessary conditions in Υ for different functions.
We expect that conditions in Υ are more complex than those in Γ and Σ .
Furthermore, we ﬁnd that functions have a similar number of minimal conditions 8 , so that more conditions in Υ seems to imply fewer conditions in Γ and Σ , and therefore, a more complex set of conditions overall.
We are going to explicitly study the functions within each set of constant t , for some small values of t , and ﬁnd their corresponding conditions in Υ and Σ .
We ﬁx Γ to be the identity for simplicity.
The conditions for a particular function should include the conditions we ﬁnd here plus their corresponding conditions under any permutation of the axes which leaves the function unchanged.
This means that on top of the describing the conditions for a ﬁxed Γ , we would need to describe the set of axes which can be permuted.
This will result in a small change to the Kolmogorov complexity of the set of conditions, specially small for functions with many symmetries or very few symmetries.
We ﬁnd that the conditions appear to be arranged in a decision tree (a consequence of Theorem 4.1) with a particular form.
First, we will prove the following simple lemma which bounds the value of t that is possible for some special cases of σ .
Lemma F.1.
We deﬁne t min ( σ ) to be the minimum possible value of T ( σ (cid:12) a ) for ﬁxed σ over all a .
We deﬁne t max similarly.
Consider σ = ( − 1 , . .
. , −(cid:124) (cid:123)(cid:122) (cid:125) k −, +1 , − 1 , . . . , −(cid:124) (cid:123)(cid:122) (cid:125) n − k ) .
Then:
t max (
[Equation]
2 k −2.
t min ( σ ) =
k 3.
Changing σ i = { +1 } for some
i < k will lead to an increase in t min ( σ )
Proof.
For any a , f ( x ) = 1 for all (exactly k ) points x ∈ { 0 , 1 } n which satisfy ( x k = 1) and ( x i = 1 for exactly one i ≤ k ) and ( x j =
if j (cid:54) =
i, k )
We can set f ( x ) = 0 for all other x ∈ { 0 , 1 } n by imposing the condition a 1 + a 2 > a k .
Thus t min ( σ ) = k .
We can restrict the values of a i for i < k to be arbitrarily smaller than a k provided they satisfy the ordering condition on a , and thus if we impose the condition
[Equation]
then for any a , f ( x ) = 1 for all x ∈ { 0 , 1 } n which satisfy ( x k = 1) and ( x l = 0 if l > k )
We can see that, for all a , f ( x ) = 0 for any x which does not satisfy these conditions, because a k < a l for all k < l and σ k is the only positive element in σ .
Thus t max = 2 k −3.
On changing σ such that σ i = { +1 } , all x ∈ { 0 , 1 } n which previously satisﬁed f ( x ) = 1 will remain mapped to 1 , plus at least the one-hot vector with a i = 1 .
We will now sketch a procedure that allows one to enumerate the conditions in Υ and Σ (corresponding to conditions on a and on σ respectively)
such that they satisfy T ( σ (cid:12) a )
=
t , for any given t .
From Lemma F.1 we see that all σ i = − 1 for i > t in order for t min ( σ ) ≤ t . 2.
Iterate through all 2 t distinct σ which satisfy 1., and retain only those which also satisfy t min ( σ ) ≤ t .
t min ( σ ) can be computed by counting the
number of inputs x ∈ { 0 , 1 } n such that for every x i = 1 with σ i = − 1 , there exists a x j with j > i such that σ j = 1 .
These are all the x such that σ and a imply they are mapped to 1 without further conditions on a .
Find conditions on these σ such that T ( σ (cid:12) a )
= t .
We can ﬁnd the possible values of
T ( σ (cid:12) a ) by ﬁrst ordering the x in a way that satisﬁes x < x (cid:48) if x has less 1 s than x (cid:48) .
We then traverse the decision tree corresponding to mapping each of the x , in order, to either 1 or 0 .
At each step, we propagate the decision by ﬁnding all x not yet assigned an output, which can be constructed as a linear combination of x (cid:48) s with assignments, where the coefﬁcients in the combination have opposite signs for x (cid:48) mapped to different outputs.
We stop the traversal if at some point more than t points are mapped to 1 .
Each of the decisions in the tree, correspond to a new condition on a .
We denote these conditions by υ .
For small values of t one can perform this search by hand.
For example, we consider t = 4 .
We ﬁnd that all signatures with σ i> 4 = − 1 are the only set that have t min ≤ 4 .
As an example we consider the signature σ = { +1 , +1 , − 1 , ..., − 1) .
For this signature to result in T ( σ (cid:12) a ) = 4 , we need conditions on x 1 = (1 , 1 , 0 , 1 . . . )
and x 2 = (1 , 1 , 1 , 0 . . . )
which are ( a 4 > a 1 + a 2 ) and ( a 3 < a 1 + a 2 ) .
Figure 9 shows the full sets for t = 4 and t = 5 .
We observe that each branching corresponds to complementary conditions - which is to be expected, as there exists a signature producing t for any a , as per Theorem 4.1.
[Equation]
− + a 5 > a 1 + aa 4 < a 1 + a+ + − − − a 4 > a 1 + aa 4 < a 1 + a− − − + − a 4 > a 1 + a+ − + − −
[Figure]
t=
Each distinct condition on a and σ produces a unique function f , as the constructive procedure above produces the inequalities on a i by specifying the outputs of each input not already implied by the set of conditions, thus uniquely specifying the output of every input.
We now show that there is a large range in the number of conditions in Υ required to specify different function.
First, as the analysis in the proof of Lemma F.1 shows, for every t there exists a signature which only requires one further condition,
[Equation]
Furthermore, we prove in Lemma F.2 that for all n there is at least one function which has n −conditions in Υ (which are neither in Σ or Γ ).
This implies that there is a large range in the number of conditions in Υ for large n .
Lemma F.2.
There exists a function with n − 2 inequalities in Υ , that is not including those induced by Σ or in Γ .
Proof.
We prove this by induction.
Base case n = 2 : There exists a function with minimal conditions corresponding to (1 , 0) mapping to 1 and (1 , 1) mapping to 0 .
Assume that in n dimensions, there exists a function with n minimal conditions corresponding to points { p i } n i =1 with p i = (1 , . .
, 0) with 1 s in the ﬁrst i positions, and 0 in the last n − i positions, being mapped to f ( p i )
= (1 + ( − 1) i +1 ) / 2 .
Now, in n + 1 dimensions, we can extend each of those n conditions (planes) by adding a 0 to the n th coordinate of each point p i .
We now consider a cone in n dimensions bounded by these planes and bounded by the w n +1 = 0 plane, with w n +1 > 0 if n is even or w n +1 < 0 if n is odd.
If n is even, we can keep increasing w n + until we cross the plane p n +1 = 0 , with p n +1 =
We do the same, decreasing w n + if n is odd.
After we crossed the p n +1 = 0 plane, the boundaries of the plane will be { p i } n +1 i = with p i =
, 1 , 0 , dots, 0) with 1 s in the ﬁrst i positions, and 0 in the last n + 1 − i positions, ﬁnishing the induction.
We conjecture that more conditions in Υ (i.e. more complex conditions) correlates with lower probability (if w is sampled from a Gaussian distribution).
If this is true for higher t , we should expect to see high probabilities correlating with lower complexities, which if true would explain the “simplicity bias” we observe beyond the entropy bias.
G E XPRESSIVITY CONDITIONS
Lemma 5.1.
A neural network with layer sizes (cid:104) n, 2 n − 1 , 1 (cid:105) can express all Boolean functions over n variables.
Proof.
Let f : { 0 , 1 } n → { 0 , 1 } be any Boolean function over n variables, and let t be the number of vectors in { 0 , 1 } n that f maps to 1 .
Let f be the negation of f , that is, f ( x ) = 1 − f ( x ) .
It is possible to express f as a Boolean formula φ in Disjunctive Normal Form (DNF) using t clauses, and f as a Boolean formula ψ using 2 n − t clauses (see Appendix B).
Let φ and ψ be speciﬁed over the variables x 1 . . .
x n .
We can specify a neural network N φ = (cid:104) W φ 1 , b φ 1 , W φ 2 , b φ 2 (cid:105) with layer sizes (cid:104) n, t, 1 (cid:105) that expresses f by mimicking the structure of φ .
Let W φ 1[ i,j ] = 1 if x i is positive in the j ’th clause of φ ,
W φ 1[ i,j ] = − 1 if x i is negative in the j ’th clause of φ , and W φ 1[ i,j ] = 0 if x i does not occur in the j ’th clause of φ (note that if the construction in Appendix B is followed then every variable occurs in every clause in φ ).
Let b φ 1[ j ] = 1 − γ , where γ is the number of positive literals in the j ’th clause of φ , let b φ 2 = 0 , and let W φ 2[ j ] = 1 for all j .
Now N φ computes f .
Intuitively every neuron in the hidden layer corresponds to a clause in φ , and the output neuron corresponds to an OR-gate.
We can similarly specify a neural network N ψ = (cid:104) W ψ 1 , b ψ 1 , W ψ 2 , b ψ 2 (cid:105) with layer sizes (cid:104) n, 2 n − t, 1 (cid:105) that expresses f by mimicking the structure of ψ .
Using this network we can specify a third network
N θ = (cid:104) W θ 1 , b θ 1 , W θ 2 , b θ 2 (cid:105) with the same layer sizes as N ψ that expresses f by letting W θ 1 =
W ψ 1 , b θ 1 = b ψ 1 , W θ 2 = − W ψ 2 , and b θ 2 = − b ψ 2 + 0 . 5 .
Intuitively N θ simply negates the output of N ψ .
Therefore, any Boolean function f over n variables is expressed by a neural network N φ with layer sizes (cid:104) n, t, 1 (cid:105) and by a neural network N θ with layer sizes (cid:104) n, 2 n − t, 1 (cid:105) .
Since either t ≤ 2 n − 1 or 2 n − t ≤ 2 n − 1 it follows that any Boolean function over n variables can be expressed by a neural network with layer sizes (cid:104) n, 2 n − 1 , 1 (cid:105) .
Lemma 5.2.
A neural network with l hidden layers and layer sizes (cid:104) n, ( n +2 n − 1 − log 2 l +1) , . . .
( n + 2 n − 1 − log 2 l + 1) , 1 (cid:105) can express all Boolean functions over n variables.
Proof.
Let f : { 0 , 1 } n → { 0 , 1 } be any Boolean function over n variables, and let t be the number of vectors in { 0 , 1 } n that f maps to 1 .
Let f be the negation of f , that is, f ( x ) = 1 − f ( x ) .
It is possible to express f as a Boolean formula φ in Disjunctive Normal Form (DNF) using t clauses, and f as a Boolean formula ψ using 2 n − t clauses (see Appendix B).
Let φ and ψ be speciﬁed over the variables x 1 . . .
x n .
Let N φ = (cid:104) W φ 1 , b φ 1 , . . .
W φl +2 , b φl +2 (cid:105) be a neural network with layer sizes (cid:104) n, ( n + (cid:100) t/l (cid:101) + 1) , . . .
, ( n + (cid:100) t/l (cid:101) + 1) , 1 (cid:105) .
For 2 ≤ k ≤ l :
• Let W φkA be an identity matrix of size n × n , and b φkA a zero vector of length i .
Let W φkB be a matrix of size n × (cid:100) t/l (cid:101) such that W φkB
[ i,j ] = 1 if x i is positive in the (( k − 1) × ( (cid:100) t/l (cid:101) ) + j ) ’
th clause of φ , − 1 if it is negative, and 0 if x i does not occur in the (( k − 1) × ( (cid:100) t/l (cid:101) ) + j ) ’
th clause of φ (note that if the construction in Appendix B is followed then every variable occurs in every clause in φ ).
Let b φkB be a vector of length (cid:100) t/l (cid:101) such that b φkB [ j ] = 1 − γ , where γ is the number of positive literals in the (( k − 1) ×
( (cid:100) t/l (cid:101) )
+ j ) ’
th clause of φ .
• Let W φkC be a unit matrix of size ( (cid:100) t/l (cid:101) + 1) × 1 , and let b φkC = [0] .
• Let W φk and b φk be the concatenation of W φkA , W φkB , W φkC and b φkA , b φkB , b φkC respectively in the following way:
[Equation]
Let W φ 1 and b φ 1 be the concatenation of W φkA , W φkB , and b φkA , b φkB respectively, where these are constructed as above.
Let W φ ( l +2) be a unit matrix of size ( (cid:100) t/l (cid:101) + 1) × 1 , and let b φ ( l +2)
=
Now N φ computes f .
Intuitively each hidden layer is divided into three parts; n neurons that store the value of the input to the network, (cid:100) t/l (cid:101) neurons that compute the value of some of the clauses in φ , and one neuron that keeps track of whether some clause that has been computed so far is satisﬁed.
We can similarly specify a neural network N ψ = (cid:104) W ψ 1 , b ψ 1 , . . .
W ψl +2 , b ψl +2 (cid:105) with layer sizes (cid:104) n, ( n + (cid:100) (
2 n − t ) /l
(cid:101) + 1) , . . .
, ( n + (cid:100) (2 n − t ) /l
(cid:101) + 1) , 1 (cid:105) that expresses f .
Using this network we can specify a third network N θ =
(cid:104) W θ 1 , b θ 1 , . .
W θl +2 , b θl +2 (cid:105) with the same layer sizes as N ψ that expresses f by letting W θk
= W ψk , b θk
=
b ψk for k ≤ l and W θ ( l +2)
= − W ψ ( l +2) , and b θ ( l +2)
=
− b ψ ( l +2) + 0 . 5 .
Intuitively N θ simply negates the output of N ψ .
Therefore, any Boolean function f over n variables is expressed by a neural network N φ with layer sizes (cid:104) n, ( n + (cid:100) t/l (cid:101) + 1) , . . .
, ( n + (cid:100) t/l (cid:101) + 1) , 1 (cid:105) and by a neural network N θ with layer sizes (cid:104) n, ( n + (cid:100)
(2 n − t ) /l
(cid:101) +1) , . . .
, ( n + (cid:100) (2 n − t ) /l
(cid:101) +1) , 1 (cid:105) .
Since either t ≤ 2 n − 1 or 2 n − t ≤ 2 n − 1 it follows that any Boolean function over n variables can be expressed by a neural network with layer sizes (cid:104) n, ( n + (cid:100) 2 n − 1 /l (cid:101) +1) , . . .
, ( n + (cid:100) 2 n − 1 /l (cid:101) +1) , 1 (cid:105) =
(cid:104) n, (2 n − 1 − log 2 l + n +1) , . . .
(2 n − 1 − log 2 l + n + 1) , 1 (cid:105) .
H T HEOREMS ASSOCIATED WITH ENTROPY INCREASE FOR DNN S
We deﬁne the data matrix for a general set of input points below.
Deﬁnition H.1 (Data matrix) .
For a general set of inputs, { x ( i ) } i =1 ,...,m , x i ∈ R n , we deﬁne the data matrix
X ∈ R m × n which has elements X ij = x ( i ) j , the j -th component of the i -th point.
Lemma 5.3.
For any set of inputs S , the probability distribution on T of a fully connected feedforwad neural network with linear activations, no bias, and i.i.d.
initialisation of the weights is equivalent to an perceptron with no bias and i.i.d.
weights.
Proof.
Consider a neural network with L layers and weight matrices w 0 . . .
w L (notation in Section 5) acting on the set of points S .
The output of the network on an input point x ∈ S equals (cid:101) wx where (cid:101) w = w L w L − 1 . . .
w 1 w 0 .
As the weight matrices w i are i.i.d., their distributions are spherically symmetric P ( w i = aR )
=
P ( w i = a ) for any rotation matrix R in R n i and matrix a ∈ R n i +1 × n i .
This implies that P ( (cid:101) w = aR ) =
P ( (cid:101) w = a ) .
Because the value of T is independent of the magnitude of the weight, | (cid:101) w | , this means that P ( T = t ) is equivalent to that of an perceptron with i.i.d.
(and thus spherically symmetric) weights.
One can make Lemma 5.3 stronger, by only requiring the ﬁrst layer weights w 0 to have a spherically symmetric distribution, and be independent of the rest of the weights.
If the set of inputs is the hypercube, S = H n , one needs even weaker conditions, namely that the distribution of w 0 is symmetric under reﬂections along the coordinate axes (as in Theorem 4.1) and has signs independent of the rest of the layer’s weights.
This implies that the condition of Theorem 4.1 is satisﬁed by (cid:101) w .
Lemma 5.4.
Applying a ReLU function in between each layer produces a lower bound on P ( T = 0) such that P ( T = 0) ≥ 2 − n .
Proof.
Consider the action of a neural network (cid:104) n, l 1 , . . .
, l p , 1 (cid:105) with ReLU activation functions on { 0 , 1 } n .
After passing { 0 , 1 } n through l 1 . . .
l p and applying the ﬁnal ReLU function after l p , all points must lie in R n ≥ 0 by the deﬁnition of the ReLU function.
Then, if w is sampled from a distribution symmetric under reﬂection in coordinate planes:
[Equation]
This result also follows from Corollary H.7.
We observe P ( T = 2 n − 1)
≈ P ( T = 0) because the two states are symmetric except in cases where multiple points are mapped to the origin.
This happens with zero probability for inﬁnite width hidden layers.
Theorem 5.5 Let S be a set of m = | S | input points in R n .
Consider neural networks with i.i.d.
Gaussian weights with variances σ 2 w / √ n and biases with variance σ b , in the limit where the width of all hidden layers n goes to inﬁnity.
Let N
and N 2 be such a neural networks with L and
L + 1 inﬁnitely wide hidden layers, respectively, and no bias.
Then, the following holds: (cid:104) H ( T ) (cid:105) is smaller than or equal for N 2 than for N 1 .
It is strictly smaller if there exist pairs of points in S with correlations less than 1 .
If the networks has sufﬁciently large bias ( σ b > 1 is a sufﬁcient condition), the result still holds.
For smaller bias, the result holds only for sufﬁciently large number of layers L .
Proof.
The covariance matrix of the activations of the last hidden layer of a fully connected neural network in the limit of inﬁnite width has been calculated and are given by the following recurrence relation for the covariance of the outputs 9 at layer l
[[cite]]:
[Equation]
For the variance the equation simpliﬁes to
[Equation]
The correlation at layer l is ρ l ( x, x (cid:48) )
=
[Equation]
K
[Equation]
x
[Equation]
) , can be obtained using Equation (16) above recursively as
[Equation]
where
[Equation]
For the case when σ b = 0 , this simply becomes to ρ l ( x, x (cid:48) )
=
ρ l 0 ( x, x (cid:48) ) .
This function is 1 when ρ l − 1 ( x, x (cid:48) )
= 1 , and has a positive derivative less than 1 for 0 ≤ ρ l − 1 ( x, x (cid:48) ) < 1 , which implies that it is greater than ρ l − 1 ( x, x (cid:48) ) .
Therefore the correlation between any pair of points increases if ρ l − 1 ( x, x (cid:48) ) < 1 or stays the same if ρ l − 1 ( x, x (cid:48) )
= 1 , as you add one hidden layer.
By Lemma H.2, this then implies the theorem, for the case of no bias.
When σ b > 0 , we can write, after some algebraic manipulation
[Equation]
where γ = [Equation] b > 0 and a = ρ l 0 ( x,x (cid:48) )
ρ l − 1 0 ( x,x (cid:48) ) > 1 , and the inequality follows from K l ( x, x ) ≥ σ 2 b for any x , which follows from Equation (16).
We will study the behaviour of φ ( γ ) as a function of γ for different values of a and σ 2 b .
For γ = 0 , this function equals 1 .
If 1 a < σ 2 b < a , its derivative is positive, and therefore is greater than 1 for γ > 0 .
If a < σ 2 b , there is a unique maximum for γ > 0 at γ = aσ 2 b − 1 σ 2 b − a .
Because the function tends to a as γ → ∞ , if it went below 1 , then at some γ > 0 it should cross 1 (by the intermediate value theorem), and by the mean value theorem, therefore it would have an extremum below one, and thus a local minimum, giving a contradiction.
Thus the function is always greater than 1 when σ b > 1 a .
When σ b < 1 a , φ ( γ ) can be less than 1 , thus the decreasing the correlations, for some values of γ .
We know that if γ ≥ 2 ( 1 − aσ 2 b ) ( a 2 − 1) σ 2 b the function is greater than or equal to 1 .
However, because of the inequality in Equation (19), we can’t say what happens when γ is smaller than this.
From Equation (18), we know that if σ 2 w ≥ 2 , K l − 1 ( x, x ) and K l − 1 ( x (cid:48) , x (cid:48) ) grow unboundedly as l grows.
By the above arguments applied to the expression in Equation (19) before taking the inequality, this implies that after some sufﬁciently large l , ρ l ( x,x (cid:48) )
ρ l − 1 ( x,x (cid:48) ) will be > 1 .
If σ 2 w < 2 ,
K l − 1 ( x, x ) and K l − 1 ( x (cid:48) , x (cid:48) ) tend to σ 2 b 1 − σ 2 w / 2 which also becomes the ﬁxed point of the equation for K l ( x, x (cid:48) ) , Equation (16), implying that ρ l ( x, x (cid:48) )
→ 1 as l → ∞ , so that the correlation must increase with layers after a sufﬁcient number of layers, and thus the moments by Lemma H.2.
Finally, applying Lemma H.8, the theorem follows.
Lemma H.2.
Consider two sets of m input points to an n -dimensional perceptron without bias,
U and V with data matrices U and V respectively (see Deﬁnition H.1).
If for all i, j = 1 , ..., m , ( V V T ) ij / (cid:112) ( V V T ) ii ( V V T )
jj ≥ ( UU T ) ij / (cid:112) ( UU T ) ii ( UU T )
jj , then every moment (cid:104) t q (cid:105) of the distribution P ( T = t ) is greater for the set of points V than the set of points U .
Proof of Lemma H.2.
We write T as:
[Equation]
The moments of the distribution P ( T s = t ) can be calculated by the following integral:
[Equation]
Where S =
( S 1 , . . .
, S q ) .
Taking the sum outside the integral, =
[Equation]
The distribution for U is of the equivalent form.
From corolloray Corollary H.7, we have that
P ( (cid:104) w
[Equation]
s i
[Equation]
> 0 , . .
[Equation]
q
[Equation]
>
[Equation]
> 0) .
Thus we have Equation (22) for all q .
[Equation]
Lemma H.3.
Consider an 2 -dimensional Gaussian random variable with mean µ and covariance Σ .
If the correlation Σ ij / (cid:112) Σ ii Σ jj increases, then
[Equation]
increases
Proof.
We can write the non-centered orthant probability as a Gaussian integral
[Equation]
Without loss of generality, we consider Σ ii =
Σ jj = 1 .
Otherwise, we can rescale the variables x and obtain a new mean vector.
The covariance matrix Σ thus has eigenvalues λ + = 1 + Σ ij and λ − = 1 − Σ ij with corresponding eigenvectors (1 , 1) and (1 , − 1) .
We can rotate the axis so that (1 , 1) becomes (1 , 0) .
We can then rescale the x axis by 1 / (cid:112) λ
+ and the y axis by 1 / (cid:112) λ − .
The positive orthant becomes a cone C centered around the origin and with opening angle α given by
[Equation]
which increases when Σ ij increases.
The integral in polar coordinates becomes
[Equation]
Therefore, all that’s left to show is that the range of θ for any r increases when α increases.
See Figure 10 for the illustration.
Call γ the angle between one boundary of the cone C and the position vector of the center of the Gaussian in the transformed coordinates.
We can ﬁnd the length of the chord between the two points of intersection between the circle and the boundary of the cone as the difference between the distances c − , c + , of the segments OA and OB, respectively.
Using the cosine angle formula, we ﬁnd c ± =
d cos γ ± (cid:112) r 2 − d 2 sin 2 γ , and the chord length is (cid:112) r 2 − d 2 sin 2 γ , which decreases as γ increases.
Furthermore, γ increases as α increases.
Using the same argument for the other boundary of the cone, concludes the proof.
The following lemma shows that for a vector of n Gaussian random variables, the probability of two variables being simultaneously greater than 0 , given that all other any ﬁxed signs, increases if their correlation increases.
Lemma H.4.
Consider an n -dimensional Gaussian random variable with mean 0 and covariance Σ , x ∼ N (0 , Σ) .
Consider, for any σ ∈ {− 1 , 1 } n − 2 , the following probability
P
[Equation]
If Σ ij / (cid:112) Σ ii Σ jj increases, then P ij 00 increases.
Proof.
We can write P ij 00 =
E [ P ( x i > 0 , x j > 0 | ˆ x i,j )] , where ˆ x i,j is the vector of x without the i th and j th elements, and the expectation is over the distribution of ˆ x i,j conditioned on the condition ∀ k / ∈
i, jσ k x k > 0 .
The conditional distribution P ( x i , x j | ˆ x i,j ) is also a Gaussian with a, generally non-zero mean µ , and a covariance matrix given by
[Equation]
where ˆΣ is independent of Σ ij .
This means that increasing the Σ ji (keeping Σ ii and Σ jj ﬁxed) will increase the correlation in ¯Σ .
Therefore, by Lemma H.3, P ( x i > 0 , x j > 0 | ˆ x i,j ) increases, and thus
P ij 00 increases.
Corollary H.5.
An immediate consequence of Lemma H.4 is that P ( ∀ i, x i > 0) =
P ( x i > 0 , x j > 0 |∀ k / ∈ { i, j } x k > 0) P ( ∀ k / ∈ { i, j } x k > 0) increases when Σ ij / (cid:112) Σ ii Σ jj increases, as
[Table]
Lemma H.6.
In the same setting as Lemma H.4, for covariance matrices Σ and Σ (cid:48) of full rank, the following holds: for every i, j , Σ ij = Σ (cid:48) ij implies P ijK, 00 =
P ijK (cid:48) . 00 and Σ ij / (cid:112) Σ ii Σ jj > Σ (cid:48) ij / (cid:113) Σ (cid:48) ii Σ (cid:48)
jj implies P ijK. 00 > P ijK (cid:48) , 00 , where P K is the probability measure corresponding to covariance K .
Proof.
The set S of all symmetric positive deﬁnite matrices is an open convex subset of the set of all matrices.
Therefore, it is path-connected.
We can traverse the path between Σ and Σ (cid:48) through a sequence of points such that the distance between point x i and x i +1 is smaller than the radius of a ball centered around x i and contained in the set S .
Within this ball, one can move between x i and x i +1 in coordinate steps that only change one element of the matrix.
Therefore we can apply
Lemma H.4 to each step of this path, which implies the theorem.
Corollary H.7.
Consider any set of m points x ( i ) ∈ R n with elements x ( i ) j .
Let X be the m × n data matrix with X ij = x ( i ) j .
For a weight vector w ∈ R n let y =
Xw be the vector of realvalued outputs of an perceptron, with weights sampled from an isotropic Gaussian N (0 , I n ) .
If the correlation between any two inputs increases, then P ( T = 0) =
P ([ (cid:80) s 1 ( s )]
= 0) increases
Proof.
The vector y =
(cid:80) i X i · w i is a sum of Gaussian vectors (with covariances of rank 1 ), and therefore is itself Gaussian, with a covariance given by Σ
= XX T .
If the correlation product between any two inputs increases, then applying Lemma H.6 and Corollary H.5 at each step, the theorem follows.
Lemma H.8.
If the uncentered moments of the distribution P ( T = t ) increase, except for its mean (which is 2 n − 1 ), then (cid:104) H ( t ) (cid:105) increases.
Proof.
We consider the deﬁnition of the entropy, H , of a string (Deﬁnition 3.5).
We deﬁne the ﬁrst and second terms by h 1 = ( t/t max ) ln( t/t max ) and h 2 = (1 − t/t max ) ln(1 − t/t max ) .
We taylor expand h 2 about t = 0 :
[Equation]
By symmetry, we see that h 1 ( t ) =
h 2 ( t max − t ) ), and we can thus Taylor expand h 1 around 0 t max , to give:
[Equation]
Because every a 2 k = 1 2 k (2 k − 1) is positive, we see that increasing every even moment increases the average entropy, (cid:104) H ( t ) (cid:105) .
I B IAS DISAPPEARS IN THE CHAOTIC REGIME
In the paper we have analyzed fully connected networks with ReLU activations, and ﬁnd a general bias towards low entropy for a wide range of parameters.
In a stimulating new study, Yang et al.
((Yang & Salman, 2019)) recently showed an example of a network using an erf activation function where bias disappeared with increasing number of layers.
Here we argue that the reason for this behaviour lies in the emergence of a chaotic regime, which does not occur for ReLU activations, but does for some other activation functions.
In particular, the erf activation function is very similar to the tanh activation function used in an important series of recent papers that studied the propagation of correlations between hidden layer activations through neural networks, which they call “deep information propagation” ((Poole et al., 2016; Schoenholz et al., 2017; Lee et al., 2018)).
They ﬁnd that the activation function can have a dramatic impact on the behaviour of correlations.
In particular, these papers show that for FCNs with tanh activation, there are two distinct parameter regimes in the asymptotic limit of inﬁnite depth: One in which inputs approach perfect correlation (the ordered regime ), and one in which the inputs approachcorrelation (the chaotic regime ).
FCNs with ReLU activation do not appear to exhibit this chaotic regime, although they nevertheless have different dynamic regimes.
The particular example in (Yang & Salman, 2019) was for σ w = 4 . 0 , and σ b = 0 . 0 , a choice of hyperparameters that, for sufﬁcient depth, lies deep in the chaotic regime.
In this regime, inputs with initial correlations > − 1 and < 1 will become asymptotically uncorrelated for sufﬁcient depth, while initial correlations with equal to ± 1 stay ﬁxed, as we show below.
The network is therefore equally likely to produce any odd function (on the {− 1 , 1 } n Boolean hypercube).
To conﬁrm our conjecture above, we performed experiments to calculate the bias of tanh networks in both the chaotic and ordered regime, which we show in Figure 11.
We obtain the expected results: in the chaotic regime, the bias gets weaker with number of layers, while in the ordered regime, just as was found for the ReLU activation in the main text, the bias remains, and the trivial functions gain more probability.
These experiment illustrate for tanh activation that in either the chaotic or ordered regime, the a-priori bias of the network becomes asymptotically degenerate with depth, either by becoming unbiased, or too biased.
[Figure]
We now explain that a simple extension of the analysis of (Poole et al., 2016) shows that, for the special case of σ b = 0 . 0 and tanh activation function, initial correlations equal to ± 1 stay ﬁxed.
This happens because the RHS in Equation 5.
describing the propagation of correlation in (Poole et al., 2016) (which we replicate in Equation (23) below) is odd on the correlation q l − 1 12 (which represents the covariance between a pair of activations for inputs 1 and 2 at layer l − 1 ) when σ b = 0 .
In addition to the ﬁxed point they identiﬁed for the correlation being +1 there is therefore another unstable ﬁxed point at − 1 .
[Equation]
(24) where c l 12 = q l 12 ( q l 11 q l 22 ) − 1 , and z 1 , z 2 are independent standard Gaussian variables, and the q 11 and q 22 are the variances of the activations for input 1 and input 2, respectively.
This ﬁxed point at −ensures that points which are parallel but opposite (like opposite corners in the {− 1 , 1 } n hypercube) will stay perfectly anti-correlated.
This agrees with the expectation that the erf/tanh network with σ b = 0 can only produce odd functions, as the activations are odd functions.
However, any other pair of points becomes uncorrelated, and this explains why every (real valued) function, up to the oddness constraint, is equally likely.
This also implies that every odd Boolean function is equally likely, as the region of function space satisfying the oddness constraint has the same shape within every octant corresponding to an odd Boolean function.
This is because the region in one octant is related to that on another octant by simply changing signs of elements of the function vector (a reﬂection transformation).
It will be interesting to study the effect of these different dynamic regimes, for different activation functions, on simplicity bias.
J B IAS TOWARDS LOW ENTROPY IN REALISTIC DATASETS AND ARCHITECTURES
In the main text we demonstrate bias towards low entropy only for the Perceptron, where we can prove certain results rigorously, and for a fully connected network (FCN).
An obvious question is: does this bias persist for other architectures or data sets?
In this Appendix we show that similar bias indeed occurs for more realistic datasets and architectures.
We perform experiments whereby we sample the parameters of a convolutional neural network (CNN) (with a single Boolean output), and evaluate the functions they obtain on subsets of the standard vision datasets MNIST or CIFAR10.
Our results suggest that the bias towards low entropy (high class imbalance) is a generic property of ReLU-activated neural networks.
This has in fact been pointed out previously, either empiricallyPage (2019), or for the limit of inﬁnite depthLee et al.
Extending our analytic results to these more complicated architectures and datasets is probably extremely challenging, but perhaps possible for some architectures, by analyzing the inﬁnite-width Gaussian process (GP) limit.
[Figure]
[Figure]
K E FFECT OF THE BIAS TERM ON THE PERCEPTRON ON CENTERED DATA
Here we show that the bias towards low entropy is recovered for the perceptron on centered ( {− 1 , 1 } n ) inputs, when the bias term is increased.
L C ONNECTION BETWEEN THE BIAS AT INITIALIZATION , INDUCTIVE BIAS AND GENERALIZATION
In this appendix we describe more formally the connection between the distribution over functions at initialization and the inductive biases, when training with an optimizer like SGD, and the link to generalisation.
Following the ideas in (Valle-P ´ erez et al., 2018), we will ﬁrst formally introduce an exact Bayesian classiﬁer for classiﬁcation.
Let P ( θ ) be a prior distribution over parameters, which induces a prior distribution over Boolean functions given by P ( f ) :
=
P (Θ f ) , where we deﬁne Θ f to be the set of parameter values that produce the function f .
We assume a 0-1 likelihood P ( D | f ) for data D
= { ( x i , y i ) } mi =1 , deﬁned as
{+P ( D | f ) = (cid:26) 
+}
if ∀ if ( x i ) =
{+y i 
+}
otherwise
The likelihood on parameter space is deﬁned as P ( D | θ ) :
=
P ( D | f θ ) , where f θ is the function produced by parameter θ .
The Bayesian posterior on function space is then
[Equation]
P
[Equation]
D ) , where P ( D ) =
(cid:80) f P ( D | f ) P ( f ) is the marginal likelihood of the data.
In parameter space, the posterior is just P ( θ | D ) =
P ( D | θ ) P ( θ ) P
[Equation]
For an exact Bayesian algorithm, like the one described above, the inductive bias (the preference of some functions versus others, for a given dataset), is fully encoded in the prior P ( f ) .
The connection with generalisation for the Bayesian learner described above can be expressed through the PAC-Bayes theorem (McAllester, 1999; Valle-P´erez et al., 2018) which gives bounds on the expected generalization performance that depend on the marginal likelihood, which is just the probability of the labelling of the data under the prior distribution.
In this picture, the distribution
P ( f ) at initialization determines the full inductive bias, but only to extent to which the training algorithm approximates the Bayesian posterior, with P ( f ) as its prior.
In (Valle-P´erez et al., 2018) it was shown that PAC-Bayes bounds work well for an FCN and a CNN trained on CIFAR, and also for an FCN on Boolean data.
The success of these bounds is indirect evidence that the training algorithm (here different variants of SGD) is indeed behaving somewhat like a Bayesian sampler.
We are aware that the claim in the paragraph above may be controversial, given that there is also a literature arguing that SGD itself is an important source of the inductive bias of deep learning.
So it is important to also ﬁnd direct evidence that that stochastic gradient descent (SGD) approximates the Bayesian posterior with prior P ( f ) .
In(Valle-P´erez et al., 2018), Fig 4, the probability P ( f ) was compared to the probability that two variants of SGD generate functions, for the Boolean system.
Good agreement was found, suggesting that SGD indeed samples functions with a probability close to P ( f ) , at least on the log scale and for these problems.
Here we provide further evidence for this behaviour in Figure 16, where we compare the probabilities of ﬁnding different Boolean functions upon training a fully connected neural network with SGD to learn a relatively simple Boolean function versus the probabilities of obtaining those functions by randomly sampling parameters until they ﬁt the data (which we refer to as approximate Bayesian inference, or ABI).
We see that most functions in the sample show very similar probabilities to be obtained by either SGD or ABI.
Although we don’t yet have a formal theoretical explanation of this correlation we can make the following heuristic argument: The parameter-function map (deﬁned in Section 3) is hugely biased, typically over many orders of magnitude.
The basins of attraction of the functions are also likely to vary over many orders of magnitude in size so that SGD is much more likely to ﬁnd functions with larger P ( f ) , which have larger basins, than functions with smaller P ( f ) (See also (Wu et al., 2017)).
This argument would be sufﬁcient to show that the bias upon initialisation (or equivalently upon uniform random sampling of parameters) has an important effect on the functions that SGD ﬁnds.
However, empirically we ﬁnd a stronger result, which is that SGD approximately samples functions with a probability directly proportional to P ( f ) .
This agreement suggests another stronger ansatz:
As long as SGD samples the parameters more or less uniformly (within the region of likelihood 1 ), then on function space the functions are samples with probabilities of the same orders of magnitude as P ( f | D ) .
Finally, preliminary results on MNIST and CIFAR data sets (using the GP approximation) exhibit a similar scaling to what is observed in Figure 16.
We are currently working further on this important and complex question of how SGD samples functions.
M E FFECT OF BIAS ON LEARNING
The effect of inductive biases such as the ones we discuss on learning is a vast and open research project.
In particular, we expect that bias towards low entropy should play a bigger role when trying to learn class-imbalanced data.
In class-imbalanced problems, one is often interested in quantities beyond the raw test accuracy.
For example, the sensitivity (the fraction of test-set misclassiﬁcations on the rare class), and the speciﬁcity (the fraction of test-set misclassiﬁcations on the common class).
Furthermore, the use of tricks, like oversampling the rare class during optimisation, makes the formal analysis of practical cases even more challenging.
In this section, we show preliminary results illustrating some of the effects that entropy bias in P ( f ) can have on learning.
In the experiments, we shift the bias term of the output neuron b (cid:55)→ b (cid:48)
=
b + shift.
This causes the functions at initialisation to have distributions P ( T ) which may be biased towards higher or lower values of T (mapping most inputs to 0 or 1 ).
We measure the average (cid:104) T (cid:105) for different choices of the shift hyperparameter, and train the network, using SGD, on a class-imbalanced dataset.
Here we perform this experiment on a CNN with 4 layers (and no pooling), and a FCN with 1 layer.
The dataset consists of a sample of either MNIST (10 classes) or balanced EMNIST (47 classes), where a single class is labelled as 0, and the rest are labelled as 1.
We see in Figure 17 that values of the shift hyperparameter producing large values (cid:104) T (cid:105) give higher test accuracy.
We suggest that this could be because the new parameterisation induced by the shifted bias term 10 , causes the inductive bias is more “atuned” to the correct target function (which has a class imbalanced of 1:10 and 1:47, respectively for MNIST and balanced EMNIST).
However, the detailed shape of the curve seems architecture and data dependent.
For instance, for the FCN (with
ReLU or tanh activation) trained on MNIST we ﬁnd a clear accuracy peak around the true value of (cid:104) T (cid:105) , but not for the others.
We also measured the sensitivity and 99% speciﬁcity (found by ﬁnding the smallest threshold value of the sigmoid unit, giving 99% speciﬁcity).
Sensitivity seems to follow a less clear pattern.
For two of the experiments, it followed the accuracy quite closely, while for FCN trained on MNIST it peaked around a positive value of the ship hyperparameter.
These results suggest that looking at architectural changes that affect the entropy bias of P ( f ) can have a signiﬁcant effect on learning performance for class-imbalanced tasks.
For example, for the tanh-activated FCN, increasing the shift hyperparameter above 0 improves both the accuracy and sensitivity signiﬁcantly.
As we mentioned at the beginning, understanding the full effect of entropy bias, and other biases on P ( f ) , on learning is a large research project.
In this paper, we focused on explaining properties of P ( f ) , while only showing preliminary results on the effect of these properties on learning.
[Figure]