Additive function approximation in the brain
Abstract
Many biological learning systems such as the mushroom body, hippocampus, and cerebellum are built from sparsely connected networks of neurons.
For a new understanding of such networks, we study the function spaces induced by sparse random features and characterize what functions may and may not be learned.
A network with d inputs per neuron is found to be equivalent to an additive model of order d , whereas with a degree distribution the network combines additive terms of different orders.
We identify three speciﬁc advantages of sparsity: additive function approximation is a powerful inductive bias that limits the curse of dimensionality, sparse networks are stable to outlier noise in the inputs, and sparse random features are scalable.
Thus, even simple brain architectures can be powerful function approximators.
Finally, we hope that this work helps popularize kernel theories of networks among computational neuroscientists.
Introduction
Kernel function spaces are popular among machine learning researchers as a potentially tractable framework for understanding artiﬁcial neural networks trained via gradient descent [e g. 1, 2, 3, 4, 5, 6].
Artiﬁcial neural networks are an area of intense interest due to their often surprising empirical performance on a number of challenging problems and our still incomplete theoretical understanding.
Yet computational neuroscientists have not widely applied these new theoretical tools to describe the ability of biological networks to perform function approximation.
The idea of using ﬁxed random weights in a neural network is primordial, and was a part of
Rosenblatt’s perceptron model of the retina [7].
Random features have then resurfaced under many guises: random centers in radial basis function networks [8], functional link networks [9], Gaussian processes (GPs) [10, 11] and so-called extreme learning machines [12]; see [13] for a review.
Random feature networks, where the neurons are initialized with random weights and only the readout layer is trained, were proposed by Rahimi and Recht in order to improve the performance of kernel methods [14, 15] and can perform well for many problems [13].
In parallel to these developments in machine learning, computational neuroscientists have also studied the properties of random networks with a goal towards understanding neurons in real brains.
To a ﬁrst approximation, many neuronal circuits seem to be randomly organized [16, 17, 18, 19, 20].
However, the recent theory of random features appears to be mostly unknown to the greater computational neuroscience community.
Here, we study random feature networks with sparse connectivity : the hidden neurons each receive input from a random, sparse subset of input neurons.
This is inspired by the observation that the connectivity in a variety of predominantly feedforward brain networks is approximately random and sparse.
These brain areas include the cerebellar cortex, invertebrate mushroom body, and dentate gyrus of the hippocampus [21].
All of these areas perform pattern separation and associative learning.
The cerebellum is important for motor control, while the mushroom body and dentate gyrus are
[Figure]
[Figure]
True
Estimate be general learning and memory areas for invertebrates and vertebrates, respectively, and may have evolved from a similar structure in the ancient bilaterian ancestor [22].
Recent work has argued that the sparsity observed in these areas may be optimized to balance the dimensionality of representation with wiring cost [20].
Sparse connectivity has been used to compress artiﬁcial networks and speed up computation [23, 24, 25], whereas convolutions are a kind of structured sparsity [26, 27].
We show that sparse random features approximate additive kernels [28, 29, 30, 31] with arbitrary orders of interaction.
The in-degree of the hidden neurons d sets the order of interaction.
When the degrees of the neurons are drawn from a distribution, the resulting kernel contains a weighted mixture of interactions.
These sparse features offer advantages of generalization in high-dimensions, stability under perturbations of their input, and computational and biological efﬁciency.
Background: Random features and kernels
Now we will introduce the mathematical setting and review how random features give rise to kernels.
The simplest artiﬁcial neural network contains a single hidden layer, of size m , receiving input from a layer of size l (Figure 1).
The activity in the hidden layer is given by, for i ∈ [ m ] ,
[Equation]
Here each φ i is a feature in the hidden layer, h is the nonlinearity, W = ( w 1 , w 2 , . . . , w m
) ∈ R l × m are the input to mixed weights, and b ∈ R m are their biases.
We can write this in vector notation as φ ( x ) =
h ( W (cid:124) x − b ) , where φ : R l → R m .
Random features networks draw their input-hidden layer weights at random.
Let the weights w i and biases b i in the feature expansion (1) be sampled i.i.d.
from a distribution µ on R l +1 .
Under mild assumptions, the inner product of the feature vectors for two inputs converges to its expectation
[Equation]
We identify the limit (2) with a reproducing kernel k ( x , x (cid:48) ) induced by the random features, since the limiting function is an inner product and thus always positive semideﬁnite [14].
The kernel deﬁnes an associated reproducing kernel Hilbert space (RKHS) of functions.
For a ﬁnite network of width m , the inner product 1 m φ ( x ) (cid:124) φ ( x (cid:48) ) is a randomized approximation to the kernel k ( x , x (cid:48) ) .
Sparsely connected random feature kernels
We now turn to our main result: the general form of the random feature kernels with sparse, independent weights.
For simplicity, we start with a regular model and then generalize the result to networks with varying in-degree.
Two kernels that can be computed in closed form are highlighted.
Fix an in-degree d , where 1 ≤ d ≤ l , and let µ | d be a distribution on R d which induce, together with some nonlinearity h , the kernel k d ( z , z (cid:48) ) on z , z (cid:48) ∈ R d (for the moment, d is not random).
Sample a sparse feature i ∈ [ m ] in two steps: First, pick d neighbors from all (cid:0) ld (cid:1) uniformly at random.
Let
N i ⊆
[ l ] denote this set of neighbors.
Second, sample w ji ∼ µ | d if j ∈ N i and otherwise set w ji = 0 .
We ﬁnd that the resulting kernel
[Equation]
Here x N denotes the length d vector of x restricted to the neighborhood N , with the other l − d entries in x ignored.
More generally, the in-degrees may be chosen independently according to a degree distribution , so that d becomes a random variable.
Let D ( d ) be the probability mass function of the hidden node in-degrees.
Conditional on node i having degree d
i , the in-neighborhood N i is chosen uniformly at random among the (cid:0) ld i (cid:1) possible sets.
Then the induced kernel becomes
[Equation]
For example, if every layer-two node chooses its inputs independently with probability p , the D ( d i ) is the probability mass function of the binomial distribution Bin( l, p ) .
The regular model (3) is a special case of (4) with D ( d (cid:48) )
=
I { d (cid:48)
=
d } .
Extending the proof techniques in [14] yields:
Claim The random map 1 m φ ( x ) (cid:124) φ ( x (cid:48) ) with κ -Lipschitz nonlinearity uniformly approximates k dist D ( x , x (cid:48) ) to error (cid:15) using m = O ( lκ 2 (cid:15) 2 log C(cid:15) )
many features (the proof is contained in Appendix C).
Two simple examples With Gaussian weights and regular d = 1 , we ﬁnd that (see Appendix B)
[Equation]
if h = step function, and
[Equation] [Equation]
if h = sign function .
[Equation]
[Equation]
Advantages of sparse connectivity 4.1 Additive modeling
The regular degree kernel (3) is a sum of kernels that only depend on combinations of d inputs, making it an additive kernel of order d .
The general expression for the degree distribution kernel (4) illustrates that sparsity leads to a mixture of additive kernels of different orders.
These have been referred to as additive GPs [30], but these kind of models have a long history as generalized additive models [e.g. 28, 32].
For the regular degree model with d = 1 , the sum in (3) is over neighborhoods of size one, simply the individual indices of the input space.
Thus, for any two input neighborhoods
N 1 and N 2 , we have |N 1 ∩ N 2 | = ∅ , and the RKHS corresponding to k reg1 ( x , x (cid:48) ) is the direct sum of the subspaces H
= H 1 ⊕ . . .
⊕ H l .
Thus regular d = 1 deﬁnes a ﬁrst-order additive model, where f ( x ) = f 1 ( x 1 ) + . . .
+ f l ( x l ) .
When d > 1 we allow interactions between subsets of d variables, e.g. regular d = 2 leads to f ( x ) = f 12 ( x 1 , x 2 ) + . . .
+ f l − 1 ,l ( x l − 1 , x l ) , all pairwise terms.
These interactions are deﬁned by the structure of the terms k d ( x N , x (cid:48)N ) .
Finally, the degree distribution
D ( d ) determines how much weight to place on different degrees of interaction.
Generalization from fewer examples in high dimensions Stone proved that ﬁrst-order additive models do not suffer from the curse of dimensionality [33, 34], as the excess risk does not depend on the dimension l .
Kandasamy and Yu [31] extended this result to d th-order additive models and found a bound on the excess risk of O ( l 2 d n − 2 s 2 s + d ) or O ( l 2 d C d /n ) for kernels with polynomial or exponential eigenvalue decay rates ( n is the number of samples and the constants s and C parametrize rates).
Without additivity, these weaken to O ( n − 2 s 2 s + l ) and O ( C l /n ) , much worse when l (cid:29) d .
Similarity to dropout Dropout regularization [35, 36] in deep networks has been analyzed in a kernel/GP framework [37], leading to (4) with D = Bin( l, p ) for a particular base kernel.
Dropout may thus improve generalization by enforcing approximate additivity, for the reasons above.
4.2 Stability: robustness to noise or attacks affecting a few inputs
Equations (5) and (6) are similar: They differ only by the presence of an (cid:96) 0 -“norm” versus an (cid:96) 1 - norm and the presence of the sign function.
Both norms are stable to outlying coordinates in an input x .
This property also holds for different nonlinearities and 1 < d (cid:28) l , since every feature φ i ( x ) only depends on d inputs, and therefore only a minority of the m features will be affected by the few outliers.
1 Sufﬁciently sparse features will then be less affected by sparse noise than a fully-connected network, offering denoising advantages [e.g. 20].
A regressor f ( x ) = α (cid:124) φ ( x ) built from these features is stable so long as (cid:107) α (cid:107) p is small, since | f ( x ) − f ( x (cid:48) )
| ≤ (cid:107) α (cid:107) p (cid:107) φ ( x ) − φ ( x (cid:48) ) (cid:107) q for any Hölder conjugates 1 /p + 1 /q
= 1 .
Thus if x (cid:48) =
x + e where e contains a small number of nonzero entries, then f ( x (cid:48) )
≈ f
( x ) since φ ( x ) ≈ φ ( x (cid:48) ) .
Stability also may guarantee the robustness of these networks to sparse adversarial attacks [38, 39, 40], although exactly the conditions under which these approximations hold ( p = ∞ , q = 1 is an interesting case) we leave for future work.
4.3 Scalability: computational and biological
Computational Sparse random features give potentially huge improvements in scaling.
Direct implementations of additive models incur a large cost for d > 1 , since (3) requires a sum over (cid:0) ld (cid:1)
=
O ( l d ) neighborhoods.
This leads to O ( n 2 l d ) time to compute the Gram matrix of n examples and O ( nl d ) operations to evaluate f ( x ) .
In our case, since the random features method is primal, we need to perform O ( nmd ) computations to evaluate the feature matrix and the cost of evaluating f ( x ) remains O ( md ) .
3 Sparse matrix-vector multiplication makes evaluation faster than the O ( ml ) time it takes when connectivity is dense.
For ridge regression, we have the usual advantages that computing an estimator takes O ( nm 2 + nmd ) time and O ( nm + md ) memory, rather than O ( n 3 ) time and O ( n 2 ) memory for a naïve kernel ridge method.
Biological In a small animal such as a ﬂying insect, space is extremely limited.
Sparsity offers a huge advantage in terms of wiring cost [20].
Additive approximation also means that such animals can learn much more quickly, as seen in the mushroom body [41, 42, 43].
While the previous computational points do not apply as well to biology, since real neurons operate in parallel, fewer operations translate into lower metabolic cost for the animal.
Discussion
Inspired by their ubiquity in biology, we have studied sparse random networks of neurons using the theory of random features, ﬁnding the advantages of additivity, stability, and scalability.
This theory shows that sparse networks such as those found in the mushroom body, cerebellum, and hippocampus can be powerful function approximators.
Kernel theories of neural circuits may be more broadly applicable in the ﬁeld of computational neuroscience.
Expanding the theory of dimensionality in neuroscience Learning is easier in additive function spaces because they are low-dimensional , a possible explanation for few-shot learning in biological systems.
Our theory is complementary to existing theories of dimensionality in neural systems [16, 44, 45, 46, 47, 20, 48, 49, 50], which deﬁned dimensionality using a skewness measure of covariance eigenvalues.
Kernel theory extends this concept, measuring dimensionality similarly [51] in the space of nonlinear functions spanned by the kernel.
Limitations We model biological neurons as simple scalar functions, completely ignoring time and neuromodulatory context.
It seems possible that a kernel theory could be developed for timeand context-dependent features.
Our networks suppose i.i.d.
weights, but weights that follow Dale’s law should also be considered.
We have not studied the sparsity of activity, postulated to be relevant in cerebellum.
It remains to be demonstrated how the theory can make concrete, testable predictions, e.g. whether this theory may explain identity versus concentration encoding of odors or the discrimination/generalization tradeoff under experimental conditions.
Acknowledgments KDH was supported by a Washington Research Foundation postdoctoral fellowship.
Thank you to Rajesh Rao for support during this project and to Bing Brunton for support and many helpful comments.
A Test problems
We have implemented sparse random features in Python to demonstrate the properties of learning in this basis.
Our code, which provides a scikit-learn style SparseRFRegressor and SparseRFClassifier estimators, is available from https://github.com/kharris/ sparse-random-features .
A.1 Additive function approximation A.1.1 Comparison with datasets from Kandasamy and Yu [1]
As said in the main text, Kandasamy and Yu [1] created a theory of the generalization properties of higher-order additive models.
They supplemented this with an empirical study of a number of datasets using their Shrunk Additive Least Squares Approximation (SALSA) implementation of the additive kernel ridge regression (KRR).
Their data and code were obtained from https: //github.com/kirthevasank/salsa .
We compared the performance of SALSA to the sparse random feature approximation of the same kernel.
We employ random sparse
Fourier features with Gaussian weights N (0 , σ 2 I ) with σ = 0 .
05 · √ dn 1 / 5 in order to match the Gaussian radial basis function used by Kandasamy and Yu [1].
We use m = 300 l features for every problem, with regular degree d selected equal to the one chosen by SALSA.
The regressor on the features is cross-validated ridge regression ( RidgeCV from scikit-learn ) with ridge penalty selected from 5 logarithmically spaced points between 10 − 4 · n and 10 2 · n .
In Figure 2, we compare the performance of sparse random features to SALSA.
Generally, the training and testing errors of the sparse model are slightly higher than for the kernel method, except for the forestfires dataset.
A.1.2 Polynomial test function shows generalization from fewer examples
We studied the speed of learning for a test function as well.
The function to be learned f ( x ) was a sparse polynomial plus a linear term:
[Equation]
The linear term took a ∼ N (0 , I ) , the polynomial p was chosen to have 3 terms of degree 3 with weights drawn from N (0 , 1) .
The inputs x are drawn from the uniform distribution over [0 , 1] 16 .
Gaussian noise (cid:15) with variance 0 .
05 2 was added to generate observations
y i =
f ( x i ) + (cid:15) i .
Constants c 1 and c 2 were tuned by setting c 1 = 1 σ lin
[Equation]
and c
[Equation]
2 , where α = 0 .and
σ lin and σ nonlin were the standard deviations of the linear and nonlinear terms alone.
For this problem we use random features of varying regular degrees d = 1 , 3 , 10 , 16 and number of data points n .
The features use a Fourier nonlinearity h ( · ) =
(sin · , cos · ) , weights w ij ∼
N (0 , d − 1 / 2 ) , and biases b i ∼ U ([ − π, π ]) , leading to an RBF kernel in d dimensions.
The output regression model is again ridge regression with the penalty selected via cross-validation on the training set from 7 logarithmically spaced points between 10 − 4 and 10 2 .
In Figure 3, we show the test error as well as the selected ridge penalty for different values of d and n .
With a small amount of data ( n < 250 ), the model with d = 1 has the lowest test error, since this “simplest” model is less likely to overﬁt.
On the other hand, in the intermediate data regime ( 250 < n < 400 ), the model with d = 3 does best.
For large amounts of data ( n > 400 ), all of the models with interactions d ≥ 3 do roughly the same.
Note that with the RBF kernel the RKHS
H d ⊆ H d (cid:48) whenever d ≤ d (cid:48) , so d > 3 can still capture the degree 3 polynomial model.
However, we see that the more complex models have a higher ridge penalty selected.
The penalty is able to adaptively control this complexity given enough data.
A.2 Stability with respect to sparse input noise
Here we show that sparse random features are stable for spike-and-slab input noise.
In this example, the truth follows a linear model, where we have random input points x i ∼ N (0 , I ) and linear observations y i
=
x (cid:124) i β for i = 1 , . . . , n and β ∼ N (0 , I ) .
However, we only have access to sparsely corruputed inputs w i =
x i + e i , where e i = 0 with probability 1 − p and e i = (cid:15) x − x i with probability p , (cid:15) x ∼ N (0 , σ 2 I ) .
That is, the corrupted inputs are replaced with pure noise.
We use p = 0 .
03 (cid:28) 1 and σ = 6 (cid:29) 1 so that the noise is sparse but large when it occurs.
In Table 1 we show the performance of various methods on this regression problem given the corrupted data ( W , y ) .
Note that if the practitioner has access to the uncorrupted data X , linear regression succeeds with a perfect score of 1.
Using kernel ridge regression with k ( x , x (cid:48) )
= 1 − 1 l (cid:107) x − x (cid:48) (cid:107) 1 , the kernel that arises from sparse random features with d = 1 and sign nonlinearity, leads to improved performance over naïve linear regression on the corrupted data or a robust Huber loss function.
The
Model
Training score
Testing score
[Table]
best performance is attained by trimming the outliers and then performing linear regression.
However, this is meant to illustrate our point that sparse random features and their corresponding kernels may be useful when dealing with noisy inputs in a learning problem.
In Figure 4 we show another way of measuring this stability property.
We compute the eigenvalues of the kernel matrix on a ﬁxed dataset of size n = 800 points both with noise and without noise.
Plotted are the ratio of the noisy to noiseless eigenvalues, in decibels, which we call the ampliﬁcation and is a measure of how corrupted the kernel matrix is by this noise.
The main trend that we see is, for ﬁxed p = 3 , changing the amplitude of the noise σ does not lead to signiﬁcant ampliﬁcation, especially of the early eigenvalues which are of largest magnitude.
On the other hand, making the outliers denser does lead to more ampliﬁcation of all the eigenvalues.
The eigenspace spanned by the largest eigenvalues is the most “important” for any learning problem.
B Kernel examples
B.1 Fully-connected weights
We will now describe a number of common random features and the kernels they generate with fully-connected weights.
Later on, we will see how these change as sparsity is introduced in the input-hidden connections.
Translation invariant kernels The classical random features [2] sample Gaussian weights w ∼
N (0 , σ − 2 I ) , uniform biases b ∼ U [ − a, a ] , and employ the Fourier nonlinearity h ( · ) = cos( · ) .
This leads to the Gaussian radial basis function kernel
[Equation]
for x , x (cid:48) ∈ [ − a, a ] l .
In fact, every translation-invariant kernel arises from Fourier nonlinearities for some distributions of weights and biases (Bôchner’s theorem).
Moment generating function kernels The exponential function is more similar to the kinds of monotone ﬁring rate curves found in biological neurons.
In this case, we have
[Equation]
We can often evaluate this expectation using moment generating functions.
For example, if w and b are independent, which is a common assumption, then
[Equation]
where E (exp( w (cid:124)
( x + x (cid:48) )) is the moment generating function for the marginal distribution of w , and E exp(2 b ) is just a constant that scales the kernel.
For multivariate Gaussian weights w ∼ N ( m , Σ ) this becomes
[Equation]
This equation becomes more interpretable if m = 0 and Σ = σ − 2 I and the input data are normalized: (cid:107) x (cid:107)
= (cid:107) x (cid:48) (cid:107) = 1 .
Then,
[Equation]
This result highlights that dot product kernels k ( [Equation]
[Equation] x (cid:48) ) , where v : R → R , are radial basis functions on the sphere S l − 1 = { x ∈ R l : (cid:107) x (cid:107) 2 = 1 } .
The eigenbasis of these kernels are the spherical harmonics [3, 4].
Arc-cosine kernels
This class of kernels is also induced by monotone “neuronal” nonlinearities and leads to different radial basis functions on the sphere [3, 5, 6].
Consider standard normal weights w ∼ N (0 , I ) and nonlinearities which are threshold polynomial functions h [Equation] z ) = Θ
[Equation]
p for p ∈ Z + , where Θ( · ) is the Heaviside step function.
The kernel in this case is given by
[Equation]
) , for a known function J p ( θ ) where θ = arccos (cid:16) x (cid:124) x
(cid:48) (cid:107) x
(cid:107)(cid:107) x [Equation] (cid:107) (cid:17) .
Note that arc-cosine kernels are also dot product kernels.
Also, if the weights are drawn as w ∼ N (0 , σ − 2 I ) , the terms x are replaced by x /σ , but this does not affect θ .
With p = 0 , corresponding to the step function nonlinearity, we have
J 0 ( θ ) =
π − θ , and the resulting kernel does not depend on (cid:107) x (cid:107) or (cid:107) x (cid:48) (cid:107) :
[Equation]
Sign nonlinearity We also consider a shifted version of the step function nonlinearity, the sign function sgn( z ) , equal to {++
+} when z > 0 , − 1 when z < 0 , and zero when z = 0 .
Let b ∼ U
([ a 1 , a 2 ]) and w ∼ P , where P is any spherically symmetric distribution, such as a Gaussian.
Then,
[Equation]
where e = ( x − x (cid:48) ) / (cid:107) x − x (cid:48) (cid:107) 2 .
The factor E ( | w (cid:124) e | ) in front of the norm is just a function of the radial part of the distribution P , which we should set inversely proportional to √ l to match the scaling of (cid:107) x − x (cid:48) (cid:107) 2 .
For w ∼ N (0 , σ 2 l − 1 I ) , we obtain
[Equation]
B.2 Sparse weights
The sparsest networks possible have d = 1 , leading to ﬁrst-order additive kernels.
Here we look at two simple nonlinearities where we can perform the sum and obtain an explicit formula for the additive kernel.
In both cases, the kernels are simply related to a robust distance metric.
This suggests that such kernels may be useful in cases where there are outlier coordinates in the input data.
Step function nonlinearity We again consider the step function nonlinearity h ( · ) =
Θ( · ) , which in the case of fully-connected Gaussian weights leads to the degree p = [-
-]0 arc-cosine kernel k ( x , x (cid:48) )
= 1 − θ ( x , x (cid:48) )
π .
When d = 1 , x N = x i and x (cid:48)N = x (cid:48)
i are scalars.
For a scalar a , normalization leads to a/ (cid:107) a (cid:107) = sgn ( a ) .
Therefore, θ = arccos (sgn( x i ) sgn( x (cid:48) i ))
=
if sgn( x i ) = sgn
( x (cid:48) i ) and π otherwise.
Performing the sum in (3), we ﬁnd that the kernel becomes
[Equation]
This kernel is equal to one minus the normalized Hamming distance of vectors sgn( x ) and sgn( x (cid:48) ) .
The fully-connected kernel, on the other hand, uses the full angle between the vectors x and x (cid:48) .
The sparsity can be seen as inducing a “quantization,” via the sign function, on these vectors.
Finally, if the data are in the binary hypercube, with x and x (cid:48) ∈ {− 1 , +1 } l , then the kernel is exactly one minus the normalized Hamming distance.
Sign nonlinearity We now consider a slightly different nonlinearity, the sign function.
It will turn out that the kernel is quite different than for the step function.
This has h ( · ) = sgn( · )
= 2Θ( · ) − 1 .
Let b ∼ U
([ a 1 , a 2 ]) and w ∼ P .
Then,
[Equation]
Choosing P ( w ) =
12 δ ( w + 1) + 12 δ ( w − 1) and a 2 = − a 1 = a recovers the “random stump” result of Rahimi and Recht [2].
Despite the fact that sign is just a shifted version of the step function, the kernels are quite different: the sign nonlinearity does not exhibit the quantization effect and depends on the (cid:96) 1 -norm rather than the (cid:96) 0 -“norm”.
C Kernel approximation results
We now show a basic uniform convergence result for any random features, not necessarily sparse, that use Lipschitz continuous nonlinearities.
Recall the deﬁnition of a Lipschitz function:
Deﬁnition 1.
A function f : X → R is said to be L -Lipschitz continuous (or Lipschitz with constant L )
if
[Equation]
holds for all x , y ∈ X .
Here, (cid:107) · (cid:107) is a norm on X (the (cid:96) 2 -norm unless otherwise speciﬁed).
Assuming that h is Lipschitz and some regularity assumptions on the distribution µ , the random feature expansion approximates the kernel uniformly over X .
As far as we are aware, this result has not been stated previously, although it appears to be known (see Bach [7]) and is very similar to
Claim 1 in Rahimi and Recht [2] which holds only for random Fourier features (see also Sutherland and Schneider [8] and Sriperumbudur and Szabo [9] for improved results in this case).
The rates we obtain for Lipschitz nonlinearities are not essentially different than those obtained in the Fourier features case.
As for the examples we have given, the only ones which are not Lipschitz are the step function (order 0 arc-cosine kernel) and sign nonlinearities.
Since these functions are discontinuous, their convergence to the kernel occurs in a weaker than uniform sense.
However, our result does apply to the rectiﬁed linear nonlinearity (order 1 arc-cosine kernel), which is non-differentiable at zero but 1-Lipschitz and widely applied in artiﬁcial neural networks.
The proof of the following Theorem appears at the end of this section.
Theorem 1 (Kernel approximation for Lipschitz nonlinearities) .
Assume that x ∈ X ⊂ R l and that X is compact, ∆ = diam( X ) , and the null vector 0 ∈ X .
Let the weights and biases ( w , b ) follow the distribution µ on R l +1 with ﬁnite second moments.
Let h ( · ) be a nonlinearity which is
L -Lipschitz continuous and deﬁne the random feature φ : R l → R by φ ( x ) =
h ( w (cid:124) x − b ) .
We assume that the following hold for all x ∈ X : | φ ( x ) | ≤ κ almost surely, E | φ ( x ) | 2 < ∞ , and E φ ( x ) φ ( x (cid:48) )
=
k ( x , x (cid:48) ) .
Then sup x , x (cid:48) ∈X (cid:12)(cid:12) 1 m φ ( x ) (cid:124) φ ( x (cid:48) )
− k ( x , x (cid:48) )
(cid:12)(cid:12) ≤ (cid:15) with probability at least 1 − 2(cid:32)
κL ∆ (cid:112) E (cid:107) w (cid:107) 2 + 3( E (cid:107) w (cid:107) )(cid:15) (cid:33)exp (cid:18) − m(cid:15)8( l + 1) κ(cid:19) .
Sample complexity Theorem 1 guarantees uniform approximation up to error (cid:15) using m = O (cid:16) lκ 2 (cid:15) 2 log C(cid:15) (cid:17) features.
This is precisely the same dependence on l and (cid:15) as for random Fourier features.
Note that [10] also found that m should scale linearly with l to minimize error in a particular classiﬁcation task.
A limitation of Theorem 1 is that it only shows approximation of the limiting kernel rather than direct approximation of functions in the RKHS.
A more detailed analysis of the convergence to
RKHS is contained in the work of Bach [7], whereas Rudi and Rosasco [11] directly analyze the generalization ability of these approximations.
Sun et al.
[12] show even faster rates which also apply to SVMs, assuming that the features are compatible (“optimized”) for the learning problem.
Also, the techniques of Sutherland and Schneider [8] and Sriperumbudur and Szabo [9] could be used to improve our constants and prove convergence in other L p norms.
In the sparse case, we must extend our probability space to capture the randomness of (1) the degrees, (2) the neighborhoods conditional on the degree, and (3) the weight vectors conditional on the degree and neighborhood.
The degrees are distributed independently according to d i ∼ D , with some abuse of notation since we also use D ( d ) to represent the probability mass function.
We shall always think of the neighborhoods N ∼ ν | d as chosen uniformly among all d element subsets, where ν | d represents this conditional distribution.
Finally, given a neighborhood of some degree, the nonzero weights and bias are drawn from a distribution ( w , b ) ∼ µ | d on R d +1 .
For simpler notation, we do not show any dependence on the neighborhood here, since we will always take the actual weight values to not depend on the particular neighborhood N .
However, strictly speaking, the weights do depend on N because that determines their support.
Finally, we use E to denote expectation over all variables (degree, neighborhood, and weights), whereas we use E µ | d for the expectation under µ | d for a given degree.
Corollary 2 (Kernel approximation with sparse features) .
Assume that x ∈ X ⊂ R l and that
X is compact, ∆ = diam( X ) , and the null vector 0 ∈ X .
Let the degrees d follow the degree distribution D on [ l ] .
For every d ∈ [ l ] , let µ | d denote the conditional distributions for ( w , b ) on R d +1 and assume that these have ﬁnite second moments.
Let h ( · ) be a nonlinearity which is
L -Lipschitz continuous, and deﬁne the random feature φ : R l → R by φ ( x ) =
h ( w (cid:124) x − b ) , where w follows the degree distribution model.
We assume that the following hold for all x N ∈ X N with |N| =
d , and for all 1 ≤ d ≤ l : | φ ( x N ) | 2 ≤ κ almost surely under µ | d , E (cid:2) | φ ( x N ) | 2 | d (cid:3) < ∞ , and
E [ φ ( x N ) φ ( x (cid:48)N ) | d ] =
k reg d ( x N , x (cid:48)N ) .
Then sup x , x (cid:48) ∈X (cid:12)(cid:12) 1 m φ ( x ) (cid:124) φ ( x (cid:48) )
− k dist D ( x , x (cid:48) )
(cid:12)(cid:12) ≤ (cid:15), with probability at least
[Equation]
[Equation]
[-− 
-]+
[Equation]
(cid:19) . The kernels k reg d ( z , z (cid:48) ) and k dist D ( x , x (cid:48) ) are given by equations (3) and (4) .
Proof.
It sufﬁces to show that conditions (1–3) on the conditional distributions µ | d , d ∈ [ l ] , imply conditions (1–3) in Theorem 1.
Conditions (1) and (2) clearly hold, since the distribution D has ﬁnite support.
By construction, E φ ( x ) φ ( x (cid:48) )
=
[Equation]
[Equation]
x
[Equation]
) , which concludes the proof.
Differences of sparsity The only difference we ﬁnd with sparse random features is in the terms E (cid:107) w (cid:107) 2 and E (cid:107) w (cid:107) , since sparsity adds variance to the weights.
This suggests that scaling the weights so that E µ | d (cid:107) w (cid:107) 2 is constant for all d is a good idea.
For example, setting ( w i ) N i ∼ N (0 , σ 2 d − 1 i I d i ) , the random variables (cid:107) w i (cid:107) 2 ∼ σ 2 d − 1 i χ 2 ( d i ) and (cid:107) w
i
(cid:107) ∼ σd − 1 / 2 i χ ( d i ) .
Then E (cid:107) w i (cid:107) 2 = σirregardless of d i and E (cid:107) w i (cid:107) =
σ (1 + o ( d i )) .
With this choice, the number of sparse features needed to achieve an error (cid:15) is the same as in the dense case, up to a small constant factor.
This is perhaps remarkable since there could be as many as 2 l terms in the expression of k dist D ( x , x (cid:48) ) .
However, the random feature expansion does not need to approximate all of these terms well, just their average.
Proof of Theorem 1.
We follow the approach of Claim 1 in [2], a similar result for random Fourier features but which crucially uses the fact that the trigonometric functions are differentiable and bounded.
For simplicity of notation, let ξ =
( x , x (cid:48) ) and deﬁne the direct sum norm on X + = X ⊕ X as (cid:107) ξ (cid:107) + = (cid:107) x (cid:107) + (cid:107) x (cid:48) (cid:107) .
Under this norm X
+ is a Banach space but not a Hilbert space, however this will not matter.
For i = 1 , . . .
, m , let
[Equation]
and note that these g i are i.i.d., centered random variables.
By assumptions (1) and (2), f i and g i are absolutely integrable and k ( x , x (cid:48) )
=
E φ i ( x ) φ i ( x (cid:48) ) .
Denote their mean by
[Equation]
Our goal is to show that | ¯ g ( ξ ) | ≤ (cid:15) for all ξ ∈ X + with sufﬁciently high probability.
The space X
+ is compact and 2 l -dimensional, and it has diameter at most twice the diameter of X under the sum norm.
Thus we can cover X + with an (cid:15) -net using at most T =
(4∆ /R ) 2 l balls of radius R [13].
Call the centers of these balls ξ i for i = 1 , . .
, T , and let ¯ L denote the Lipschitz constant of ¯ g with respect to the sum norm.
Then we can show that | ¯ g ( ξ ) | ≤ (cid:15) for all ξ ∈ X
+ if we show that
¯ L ≤ (cid:15) 2 R , and 2.
| ¯ g ( ξ i ) | ≤ (cid:15) 2 for all i .
First, we bound the Lipschitz constant of g i with respect to the sum norm (cid:107)·(cid:107) + .
Since h is L -Lipschitz, we have that φ i is Lipschitz with constant L (cid:107)
w i (cid:107) .
Thus, letting ξ (cid:48) =
ξ + ( δ , δ (cid:48) ) ,
[Equation]
we have that f i has Lipschitz constant κL (cid:107) w i (cid:107) .
This implies that g i has Lipschitz constant ≤ κL ( (cid:107) w i (cid:107) + E (cid:107) w (cid:107) ) .
Let ¯ L denote the Lipschitz constant of ¯ g .
Note that E ¯ L ≤ 2 κL E (cid:107) w (cid:107) .
Also,
[Equation]
Markov’s inequality states that Pr[¯ L 2 > t 2 ] ≤ E [¯ L 2 ] /t 2 .
Letting t = (cid:15) 2 R , we ﬁnd that
[Equation]
Now we would like to show that | ¯ g ( ξ
[Equation]
) | ≤ (cid:15)/ 2 for all i = 1 , . . .
, T anchors in the (cid:15) -net.
A straightforward application of Hoeffding’s inequality and a union bound shows that
[Equation] [Equation]
since | f i ( ξ ) | ≤ κ 2 .
Combining equations (11) and (12) results in a probability of failure
Pr
[Equation]
[Equation]
Set R =
( a/b )
1 2 l +2 , so that the probability (13) has the form, 2 a 2 2 l +2 b 2 l 2 l +2 .
Thus the probability of failure satisﬁes
Pr
[Equation]
for all l ∈ N , assuming ∆ κL (cid:112) E (cid:107) w (cid:107) 2 + 3( E (cid:107) w (cid:107) ) 2 > (cid:15) .
Considering the complementary event concludes the proof.