Neural Networks as Inter-Domain Inducing Points
Abstract
Equivalences between inﬁnite neural networks and Gaussian processes have been established for explaining the functional prior and training dynamics of deep learning models.
In this paper we cast the hidden units of ﬁnite-width neural networks as the inter-domain inducing points of a kernel, then a one-hidden-layer network becomes a kernel regression model.
For dot-product kernels on both R d and S d − 1 , we derive the kernel functions for inducing points.
Empirically we conduct toy experiments to validate the proposed approaches
1. Introduction
Connections between Gaussian processes (GP) and deep neural networks (DNN) have been drawn to explain the success of deep learning.
Speciﬁcally, inﬁnite-width neural networks have been demonstrated equivalent to Gaussian processes of a NNGP kernel (Neal, 1995;
Lee et al., 2018), which state that the prior of a inﬁnite random neural network is a GP prior over functions.
Furthermore, neural tangent kernels (NTK) (Jacot et al., 2018; Arora et al., 2019) are shown to govern the training dynamics of inﬁnite neural networks.
Given these equivalences between inﬁnite neural networks and Gaussian processes, ﬁnite-width neural networks can be regarded as random feature approximations (Rahimi and Recht, 2008;
Ghorbani et al., 2020) of the corresponding GP.
For a ﬁnite-width one-hidden-layer network, the theory of random feature approximations requires the weights of the ﬁrst layer to be ﬁxed after initialization, which is contrary to the real training and cannot learn adaptive representations of the data manifold.
In this paper we cast an alternative perspective, formulating the one-hidden-layer networks as conducting Nystrom approximations (Drineas and Mahoney, 2005) instead of random feature approximations.
Speciﬁcally, for a kernel k , each hidden unit σ ( w (cid:62) m x ) corresponds to an inter-domain inducing point z m (Titsias, 2009; Lázaro-Gredilla and Figueiras-Vidal, 2009).
The overall network f ( x ) = (cid:80) Mm =1 a i σ ( w (cid:62) m x ) is then equivalently f ( x )
= (cid:80) Mm =
a i k ( x , z m ) .
In such way, the fully connected network is formulated as a kernel regression model, where the weights { w m } Mm =1 in the ﬁrst layer are M inducing locations and the weights a in the second layer are linear coeﬃcients.
2. Background: Inter-domain GP and Variational Fourier Features
Inter-domain Gaussian processes (Lázaro-Gredilla and Figueiras-Vidal, 2009) introduce the inducing points u z by integrating with an inducing function z ( x ) ,
[Equation]
Depending on the inducing function z ( x ) , the inter-domain inducing points lead to various kernel inﬂuences k x ( · ) , which can be obtained via,
[Equation]
Evaluating the kernels
k ( z, x ) , k ( z, z (cid:48) ) requires integrations, which are usually intractable.
Variational Fourier Features (VFF) (Hensman et al., 2017) propose a variation of the interdomain GPs by directly specifying the kernel function k
( z, x ) .
In concrete, let the inducing function z ∈ H belong to the reproducing kernel Hilbert space H of k , VFF deﬁnes the inducing points by the RKHS inner product instead of the (cid:96) 2 inner product,
[Equation]
[Equation]
Since the RKHS inner product is also a linear operator, the kernel functions can be computed,
[Equation]
The kernel function k
( z, x ) is exactly the inducing function z , and k ( z, z (cid:48) ) is the RKHS inner product between z and z (cid:48) .
Therefore, how to choose the inducing functions z 1 , ..., z M aﬀects the accuracy of variational approximations and the computational cost.
Speciﬁcally,
Hensman et al.
(2017) applies sinusoidal Fourier features on a one-dimensional bounded segment H [ a,b ] .
Furthermore, Burt et al.
(2020) propose orthogonal features for stationary kernels
k ( x , x ) = κ ( x − x (cid:48) ) , whose resulting kernel matrix K zz is diagonal.
Dutordoir et al.
(2020) propose to use the spherical harmonics as inducing functions for zonal kernels k ( x , x ) = κ
( x (cid:62) x (cid:48) ) .
Because the spherical harmonics are eigenfunctions of zonal kernels, they are also orthogonal to each other.
However, the aforementioned variational features are all data-independent, which might suﬀer the curse-of-dimensionality issue.
3. Neural Networks as Inter-domain Inducing Points
In this section we adopt the theory of variational Fourier features and reformulate neural networks as inter-domain inducing points.
Speciﬁcally, we can explain each neuron σ ( w (cid:62) x ) as one inducing function z in VFF.
Compared to previous variational features (Hensman et al., 2017; Burt et al., 2020; Dutordoir et al., 2020), our proposed features are fully trainable and naturally adapt to high dimensions.
Speciﬁcally, for d -dimensional inputs, a one-hidden-layer neural network is deﬁned as the function,
[Equation]
where σ is the activation function and M is the number of hidden units.
W =
[ w 1 , ..., w M ] ∈
R d × M and a ∈ R M are trainable parameters.
We assume a dot-product kernel k ( x , x (cid:48) )
=
τ ( x (cid:62) x (cid:48) ) , where τ is a positive-type function (Berlinet and Thomas-Agnan, 2011).
For w ∈ R d , we deﬁne the inducing function z as σ ( w (cid:62) x ) .
Then the network has M inducing functions.
3.1. Dot-Product Kernels on R d
In this subsection we assume the input domain X = R d .
As shown by Smola et al.
, the analytic kernel k is positive semideﬁnite if and only if τ admits a Taylor expansion with nonnegative coeﬃcients, i.e., τ ( t )
=
(cid:80) ∞
j =0 β j t j , β j ≥ 0 .
We further assume the Taylor expansion of the activation, σ ( t ) =
(cid:80) ∞
j =
α j t j .
Given the Taylor expansions, the kernel admits the feature map,
[Equation]
[Equation]
Where the feature map ϕ ( x ) =
[ √ β j x i 1 · · · x i j ] j,i 1 ,...,i j .
The variational feature σ ( w (cid:62) x ) can be represented as an inner product with the feature map ϕ .
Speciﬁcally,
[Equation]
where the weights ζ ( w ) =
[ α j √ β j w i 1 · · · w i j ] j,i 1 ,...,i j .
Because each term of the feature map ϕ ( x ) is one independent monomial, the RKHS inner product between σ ( w (cid:62) x ) and σ ( w (cid:48)(cid:62) x ) equals to the (cid:96) 2 inner product of ζ ( w ) and ζ ( w (cid:48) ) . [Equation]
[Equation]
where we deﬁne ρ ( t ) = (cid:80) ∞
j =
α
j β j t j .
Therefore, the kernel function k ( σ ( w (cid:62) · ) , x ) = σ ( w (cid:62) x ) and k ( σ ( w (cid:62) · ) , σ ( w (cid:48)(cid:62) · ))
=
ρ ( w (cid:62) w (cid:48) ) .
3.2. Dot-Product Kernels on S d −
In this subsection we assume the input lies on the sphere S d − 1 = { x | x ∈ R d , (cid:107) x (cid:107) 2 = 1 } .
Compared to the previous subsection, the change of input domain has substantial inﬂuences.
For example, the neural network kernel (NNGP) (Neal, 1995; Lee et al., 2018) and the neural tangent kernel (NTK) (Jacot et al., 2018) depend on the norms x (cid:62) x , x (cid:48)(cid:62) x on R d , thus the previous analyses do not apply.
Furthermore, if we restrict x ∈ S d − 1 , the previous feature map terms √ β j x i 1 · · · x i j in ϕ ( x ) are not independent monomials anymore, thus the derived RKHS inner products are not applicable either.
The theory of spherical harmonics (Thomson and Tait, 1888; Morimoto, 1998) is an important tool for analyzing functions on sphere, which we will adopt for computing the
RKHS inner products.
Denote by P dn ( ξ ) the associated Legendre polynomials on [ − 1 , 1] , which form a complete orthogonal basis for functions on S d − 1 .
The Addition formula (see e.g. Gallier, 2009, Proposition 1.18) states that, if { Y n,j } n,j are a set of spherical harmonics on S d − 1 , for any x , x (cid:48)
∈ S
d − 1 ,
The Addition Formula: P dn ( x · x (cid:48) ) =
| S d − 1 | N ( d, n ) N ( d,n ) (cid:88) j =
Y n,j ( x ) Y n,j ( x (cid:48) ) .
Where | S d − 1 | is the area of S d − 1 , and N ( d, n ) = (cid:0) n + d − 1 n (cid:1) − (cid:0) n + d − 3 n − 2 (cid:1) is the number of spherical harmonics of degree n .
Because { P dn ( ξ ) } ∞ n =0 is a complete orthogonal basis, the Funk-Hecke formula (Smola et al., 2001; Gallier, 2009; Müller, 2012) states, for any analytic function f on [ − 1 , 1] ,
The Funk-Hecke Formula: f (
[Equation]
where ˆ f ( n ) = (cid:82) 1 − 1 f ( ξ (cid:48) )
P dn ( ξ (cid:48) )
(1 − ξ (cid:48) 2 )
d − 32 dξ (cid:48) 1 .
Now we can use these established results in spherical harmonics for the proposed variational features.
Since k ( x , x (cid:48) )
=
τ ( x (cid:62) x (cid:48) ) , we apply the Funk-Hecke formula for τ and rewrite P dn using the Addition formula,
[Equation]
x
[Equation]
x
[Equation]
(15) where ϕ ( x ) =
[ (cid:112) | S d − 2 | ˆ τ ( n ) Y n,j ( x )] n,j .
Furthermore, we deﬁne ˜ w = w (cid:107) w (cid:107) 2 so that
˜ w ∈ S
d − 1 , and the function ˜ σ (cid:107) w (cid:107) 2 ( x (cid:62) ˜ w ) :
= σ ( x (cid:62) w ) .
Then we can expand the function ˜ σ (cid:107) w (cid:107) 2 using the Funk-Hecke formula,
[Equation]
16) where ζ ( w ) =
[ (cid:112) | S d − 2 | / ˆ τ ( n )ˆ˜ σ (cid:107) w (cid:107) 2 ( n ) Y n,j ( ˜ w )] n,j , and ˆ˜ σ (cid:107) w (cid:107) 2 ( n ) is the integral in Funk-Hecke formula for ˜ σ (cid:107) w (cid:107) 2 .
Because the spherical harmonics { Y n,j } n,j are complete orthonormal basis of the function space on S d − 1 , the RKHS inner product can be computed as,
[Equation]
where we used the Addition formula again in the last equality.
Therefore, we have derived the
RKHS inner products for the proposed variational features.
Because the associated Legendre polynomials can be obtained using standard libraries, such as Scipy (Virtanen et al., 2020), what we need are only one-dimensional integrals for κ, ˜ σ (cid:107) w (cid:107) 2 , ˜ σ [Equation] w
[Equation]
(cid:107) 2 .
Furthermore, we can use a truncated summation n = 0 , ..., n max to approximate the inﬁnite summation.
3.3. Unifying Radial Basis Function and Feed Forward Neural Networks
A radial basis function (RBF) network (Chen et al., 1991) implements a mapping f : R d → R , [Equation] rbf (
[Equation]
18) where { w i } Mm =1 are RBF centers and φ is a given function.
The hidden units depend on the radial distances, thus they are called radial basis function networks.
Speciﬁcally, when φ ( (cid:107) x − w i (cid:107) )
=
k ( x , w i ) is an instantiation from a stationary kernel k , the network is equivalently written as f rbf ( x ) = (cid:80) mi
=1 a i k
( x , w i ) .
For example, φ ( t ) =
e − t 2 / 2 is associated to the Squared Exponential kernel.
Feed forward neural networks (FFN) have been seen parallel with RBF networks.
However, our variational features provide a uniﬁed perspective: both FFN and RBF networks are kernel regression models.
Speciﬁcally, a FFN is represented as, f ffn ( x ) =
(cid:80) mi
=
a i σ (cid:0) w (cid:62) i x (cid:1) .
Treating each hidden unit z i := σ (cid:0) w (cid:62) i · (cid:1) as a variational feature for a dot-product kernel k ( x , x (cid:48) )
= κ ( x (cid:62) x (cid:48) ) , then the network is equivalently written as f ffn ( x ) = (cid:80) mi
=1 a i k
( x , z i ) as well.
Therefore, both the RBF networks and the feed-forward neural networks are instantiations of kernel regressions.
If we regularize the data loss with the RKHS norm (cid:107) f (cid:107) 2 H = a (cid:62) K zz a , then we result in the kernel ridge regression, where K zz is the kernel matrix between inducing points,
[Equation]
3.4. Nystrom Approximation Residuals as Predictive Variances
We can use the proposed variational features for stochastic variational Gaussian process (SVGP)
(Titsias, 2009; Hensman et al., 2015) as well.
Denote by N ( K zz a , S ) the variational posterior for inducing points u , and k zx =
[ σ ( w (cid:62) 1 x ) , ..., σ ( w (cid:62) M x )] (cid:62) , the posterior predictive mean and variance can be written as, [Equation]
[Equation]
We observe that the predictive mean function resembles the feed forward neural network f .
In this way, the weights in the ﬁrst layer parameterize inducing locations and the weights in the second layer parameterize the linear coeﬃcients.
Instead of training ( a , S ) from scratch using SVGP objectives, we propose to directly generate predictive variances from a post-trained network f .
Suppose we have obtained a
M -hidden-unit neural network f trained using the standard losses, such as the squared loss or the cross entropy.
The neurons σ ( w (cid:62) 1 x ) , ..., σ ( w (cid:62) M x ) are inducing points for the kernel k , whose k zx and K zz can be obtained directly.
We compute the Nystrom approximation residual of the inducing points, and use it as the predictive variance,
[Equation]
The Nystrom approximation residual is a lower bound of σ 2 ( x ) .
However, since S is usually small, ˆ σ 2 ( x ) is an accurate estimate of σ 2 ( x ) .
Here we present experimental results for dot-product kernels on R and S 2 , where we ﬁrstly train the one-hidden-layer network, and then output the Nystrom approximation residuals as the predictive variances. [Figure] ( a ) τ = cosh , σ = cos , X = R
[Figure]
x ) from the trained network. (b) We used the neural tangent kernel corresponding to one-hidden-layer networks with Erf activations. The RKHS inner products can be computed as we introduced in Subsection 3.2.
For any input-target pair (( x 1 , x 2 ) , y ) in the training set, y ∈ { 0 , 1 } .
We normalized x 1 ∈ [0 , π ] , x 2 ∈ [0 , π 2 ] , and embeded it on the sphere as (sin( x 1 ) cos( x 2 ) , sin( x 1 ) sin( x 2 ) , cos( x 1 )) ∈ S 2 . We trained a 100 -hidden-unit neural network using the squared loss, which outputs the predictive mean µ ( x ) . Given the trained network, we computed the Nystrom approximation residual for predictive variances ˆ σ 2 ( x ) .
In the ﬁgure, the solid line represents µ ( x ) = 0 .
5 and the dashed lines represent µ ( x ) ± ˆ σ ( x ) = 0 .