Universal approximation and model compression for radial neural networks
Anonymous Author(s)
AfﬁliationAddress
Abstract
We introduce a class of fully-connected neural networks whose activation functions, rather than being pointwise, rescale feature vectors by afunction depending only on their norm.
We call such networks radialneural networks, extending previous work on rotation equivariant networks that considers rescaling activations in less generality.
We proveuniversal approximation theorems for radial neural networks, includingin the more difﬁcult cases of bounded widths and unbounded domains.
Our proof techniques are novel, distinct from those in the pointwise case.
Additionally, radial neural networks exhibit a rich group of orthogonalchange-of-basis symmetries on the vector space of trainable parameters.
Factoring out these symmetries leads to a practical lossless model compression algorithm.
Optimization of the compressed model by gradientdescent is equivalent to projected gradient descent for the full model
Introduction
Inspired by biological neural networks, the theory of artiﬁcial neural networks has largelyfocused on pointwise (or “local”) nonlinear layers [ 46 , 14 ], in which the same functions : R !
R is applied to each coordinate independently:
[Equation]
In networks with pointwise nonlinearities, the standard basis vectors in R n can be interpreted as “neurons” and the nonlinearity as a “neuron activation.”
Research has generallyfocused on ﬁnding functions s which lead to more stable training, have less sensitivity toinitialization, or are better adapted to certain applications [ 42 , 38 , 37 , 10 , 29 ].
Many s havebeen considered, including sigmoid, ReLU, arctangent, ELU, Swish, and others.
However, by setting aside the biological metaphor, it is possible to consider a muchbroader class of nonlinearities, which are not necessarily pointwise, but instead dependsimultaneously on many coordinates.
Freedom from the pointwise assumption allows one to design activations that yield expressive function classes with speciﬁc advantages.
Additionally, certain choices of non-pointwise activations maximize symmetry in the parameter space of the network, leading to compressibility and other desirable properties.
In this we introduce radial neural networks which employ non-pointwise nonlin- earities called radial rescaling activations.
Such networks enjoy several provable properties high model compressibility symmetry optimization σ σ σ σ ρ ||·|| λ
W iW i
W iW i depends only on the norm of the vector:
[Equation]
where l is a scalar-valued function of the norm.
Whereas in the pointwise setting, only thelinear layers mix information between different components of the latent features, for radialrescaling, all coordinates of the activation output vector are affected by all coordinates ofthe activation input vector.
The inherent geometric symmetry of radial rescalings makesthem particularly useful for designing equivariant neural networks [ 55 , 47 , 56 , 57 ].
We note that radial neural networks constitute a simple and previously unconsidered typeof multilayer radial basis functions network [ 4 ], namely, one where the number of hiddenactivation neurons (often denoted N ) in each layer is equal to one.
Indeed, pre-composingequation 1 .
2 with a translation and post-composing with a linear map
, one obtains a specialcase of the local linear model extension of a radial basis functions network.
In our ﬁrst set of main results, we prove that radial neural networks are in fact universalapproximators .
Speciﬁcally, we demonstrate that any asymptotically afﬁne function can beapproximated with a radial neural network, suggesting potentially good extrapolationbehavior.
Moreover, this approximation can be done with bounded width.
Our approachto proving these results departs markedly from techniques used in the pointwise case. Additionally, our result is not implied by the universality property of radial basis functionsnetworks in general, and differs in signiﬁcant ways, particularly in the bounded widthproperty and the approximation of asymptotically afﬁne functions.
In our second set of main results, we exploit parameter space symmetries of radial neuralnetworks to achieve model compression .
Using the fact that radial rescaling activationscommute with orthogonal transformations, we develop a practical algorithm to systematically factor out orthogonal symmetries via iterated QR decompositions.
This leads toanother radial neural network with fewer neurons in each hidden layer.
The resultingmodel compression algorithm is lossless : the compressed network and the original networkboth have the same value of the loss function on any batch of training data.
Furthermore, we prove that the loss of the compressed model after one step of gradientdescent is equal to the loss of the original model after one step of projected gradient descent .
As explained below, projected gradient descent involves zeroing out certain parametervalues after each step of gradient descent.
Although training the original network mayresult in a lower loss function after fewer epochs, in many cases the compressed networktakes less time per epoch to train and is faster in reaching a local minimum.
To summarize, our main contributions are:
• A formalization of radial neural networks, a new class of neural networks; • Universal approximations results for radial neural networks, including: a) approxi- mation of asymptotically afﬁne functions, and b) bounded width approximation;
• Implementation of a lossless compression algorithm for radial neural networks;
• A theorem providing the precise relationship between gradient descent optimization of the original and compressed networks.
Related work
Radial rescaling activations.
As noted, radial rescaling activations are a special case of theactivations used in radial basis functions networks [ 4 ].
Radial rescaling functions have thesymmetry property of preserving vector directions, and hence exhibit rotation equivariance.
Consequently, examples of such functions, such as the squashing nonlinearity and NormReLU, feature in the study of rotationally equivariant neural networks [ 55 , 47 , 56 , 57 , 26 ].
However, previous works apply the activation only along the channel dimension, andconsider the orthogonal group O ( n ) only for n = 2, 3.
In contrast, we consider a radialrescaling activation across the entire hidden layer, and O ( n ) -equivariance where n is thehidden layer dimension.
Our constructions echo the vector neurons formalism [ 15 ], inwhich the output of a nonlinearity is a vector rather than a scalar.
Universal approximation.
Neural networks of arbitrary width and sigmoid activationshave long been known to be universal approximators [ 14 ].
Universality can also be achievedby bounded width networks with arbitrary depth [ 36 ], and generalizes to other activationsand architectures [ 24 , 60 , 43 , 50 ].
While most work has focused on compact domains,some recent work also considers non-compact domains [ 28 , 54 ].
The techniques used forpointwise activations do not generalize to radial rescaling activations, where all activationoutput coordinates are affected by all input coordinates. Consequently, individual radialneural network approximators of two different functions cannot be easily combined to anapproximator of the sum of the functions. The standard proof of universal approximationfor radial basis functions networks requires an unbounded increase the number of hiddenactivation neurons, and hence does not apply to the case of radial neural networks [ 40 ].
Groups and symmetry.
Appearances of symmetry in machine learning have generallyfocused on symmetric input and output spaces.
Most prominently, equivariant neuralnetworks incorporate symmetry as an inductive bias and feature weight-sharing constraintsbased on equivariance.
Examples include G -convolution, steerable CNN, and ClebschGordon networks [ 13 , 55 , 11 , 9 , 30 , 2 , 58 , 12 , 57 , 16 , 31 , 44 ].
By contrast, our approach toradial neural networks does not depend on symmetries of the input domain, output space,or feedforward mapping.
Instead, we exploit parameter space symmetries and thus obtainmore general results that apply to domains with no apparent symmetry.
Model compression.
A major goal in machine learning is to ﬁnd methods to reducethe number of trainable parameters, decrease memory usage, or accelerate inference andtraining [ 8 , 61 ].
Our approach toward this goal differs signiﬁcantly from most existingmethods in that it is based on the inherent symmetry of network parameter spaces.
One prior method is weight pruning , which removes redundant weights with little lossin accuracy [ 20 , 3 , 27 ].
Pruning can be done during training [ 18 ] or at initialization[ 34 , 53 ].
Gradient-based pruning removes weights by estimating the increase in loss resultingfrom their removal [ 33 , 22 , 17 , 39 ].
A complementary approach is quantization , whichdecreases the bit depth of weights [ 59 , 25 , 19 ].
Knowledge distillation identiﬁes a small modelmimicking the performance of a larger model [ 5 , 23 , 1 ].
Matrix Factorization methods replacefully connected layers with lower rank or sparse factored tensors [ 6 , 7 , 52 , 32 , 45 , 35 ] andcan often be applied before training.
Our method involves a type of matrix factorizationbased on the QR decomposition; however, rather than aim for rank reduction, we leveragethis decomposition to reduce hidden widths via change-of-basis operations on the hiddenrepresentations.
Close to our method are lossless compression methods which removestable neurons in ReLU networks [ 49 , 48 ] or exploit permutation parameter space symmetryto remove neurons [ 51 ]; our compression instead follows from the symmetries of the radialrescaling activation.
Finally, the compression results of [ 26 ], while conceptually similar toours, are weaker, as ( 1 ) the unitary group action is on disjoint layers instead of movingthrough all layers, and ( 2 ) the results are only stated for the squashing nonlinearity.
[Equation]
[Equation]
[Equation]
function
Radial neural networks
In this section, we deﬁne radial rescaling functions and radial neural networks.
Leth : R !
R be a function.
For any n  1, set:
[Equation]
| v | for v 6 = 0, and h ( n ) ( 0 ) = 0.
A function r : R n !
R n is called a radial rescaling function ifr = h ( n ) for some piecewise differentiable h : R !
R .
Hence, r sends each input vector toa scalar multiple of itself, and that scalar depends only on the norm of the vector 1 .
It iseasy to show that radial rescaling functions commute with orthogonal transformations.
Example 1 .
( 1 ) Step-ReLU, where h ( r ) =
r if r  1 and 0 otherwise.
In this case, the radialrescaling function is given by
[Equation]
( 2 ) The squashing function, where h ( r ) =
r 2 / ( r 2 + 1 ) .
( 3 ) Shifted ReLU, where h ( r ) =max ( 0, r  b ) for r > 0 and b is a real number.
See Figure 2 .
We refer to [ 55 ] and thereferences therein for more examples and discussion of radial functions.
A radial neural network with L layers consists of a positive integer n i indicating the width ofeach layer i = 0, 1, . . .
, L ; the trainable parameters, comprising of a matrix W i 2 R n i ⇥ n i  1of weights and a bias vector b i 2 R n i for each i = 1, . . .
, L ; and a radial rescaling functionr i : R n i !
R n i for each i = 1, . . .
, L .
We refer to the tuple n = ( n 0 , n 1 , . . .
, n L ) as the widthsvector of the neural network.
The hidden widths vector is n hid =
( n 1 , n 2 , . .
. , n L  1 ) .
Thefeedforward function F : R n 0 !
R n L of a radial neural network is deﬁned in the usual wayas an iterated composition of afﬁne maps and activations.
Explicitly, set F 0 = id R n 0 andrecursively deﬁne the partial feedforward functions for i = 1, . . . , L :
[Equation]
Then the feedforward function is F = F L .
Radial neural networks are a special type ofradial basis functions network; we explain the connection in Appendix F.
Remark 2 .
If b i = 0 for all i , then the feedforward function takes the form F ( x ) =
W ( µ ( x ) x )where µ : R n !
R is a scalar-valued function and W =
W L W L  1 · · · W 1 2 R n L ⇥ n 0 is theproduct of the weight matrices.
If any of the biases are non-zero, then the feedforwardfunction lacks such a simple form.
[Equation]
Universal Approximation
In this section, we consider two universal approximation results.
The ﬁrst approximates asymptotically afﬁne functions with a network of unbounded width.
The secondgeneralizes to bounded width networks.
Proofs appear in Appendix B. Throughout,
B r ( c ) = { x 2 R n : | x  c | < r } denotes the r -ball around a point c , and an afﬁne map 152 R n !
R m is one of the from L ( x ) =
Ax + b for a matrix A 2 R m ⇥ n and b 2 R m .
4 . 1 Approximation of asymptotically afﬁne functions
A continuous function f : R n !
R m is said to be asymptotically afﬁne if there exists anafﬁne map L : R n !
R m such that, for every e > 0, there is a compact subset K of R n suchthat | L ( x )  f ( x ) | < e for all x 2 R n \ K .
In particular, continuous functions with compactsupport are asymptotically afﬁne.
The continuity of f and compactness of K imply that,for any e > 0, there exist c 1 , . . .
, c N 2 K and r 1 , . .
r N 2 ( 0, 1 ) such that, ﬁrst, the unionof the balls B r i ( c i ) covers K and, second, for all i , we have f ( B r i ( c i ) \ K ) ✓ B e ( f ( c i )) .
Let N ( f , K , e ) be the minimal 2 choice of N .
Theorem 3 (Universal approximation) .
Let f : R n !
R m be an asymptotically afﬁne function.
For any e > 0 , there exists a compact set K ⇢ R n and a function F : R n !
R m such that:
F is the feedforward function of a radial neural network with N = N ( f , K , e ) layers whose
hidden widths are ( n + 1, n + 2, . .
. , n + N ) .
For any x 2 R n , we have | F ( x )  f ( x ) | < e .
We note that the approximation in Theorem 3 is valid on all of R n .
To give an ideaof the proof, ﬁrst ﬁx c 1 , . .
c N 2 K and r 1 , . .
, r N 2 ( 0, 1 ) as above.
Let e 1 , . .
, e N beorthonormal basis vectors extending R n to R n + N .
For i = 1, . . .
, N deﬁne afﬁne maps T i : R n + i  1 ! R n + i and S i : R n + i ! R n + i by
[Equation]
where h 2 i = 1  r 2 i and h e i , z i is the coefﬁcient of e i in z .
Setting r
i to be Step-ReLU(Equation 3 . 1 ) on R n + i , these maps are chosen so that the composition S i  r i  T
i mapsthe points in B r i
( c i ) to c
i + e i , while keeping points outside this ball the same.
We nowdescribe a radial neural network with widths ( n , n + 1, . . . , n + N , m ) whose feedforwardfunction approximates f .
For i = 1, . . .
, N the afﬁne map from layer i  1 to layer i is givenby
z 7!
T i  S
i  1 ( z ) , with S
= id R n .
The activation at each hidden layer is Step-ReLU.
Let
L be the afﬁne map such that | L  f | < e on R n \ K .
The afﬁne map from layer N to theoutput layer is F  S N where F : R n + N !
R m is the unique afﬁne map determined byx 7!
L ( x ) if x 2 R n , and e i 7!
f ( c i )  L ( c i ) .
This construction is illustrated in Figure 3 .
Corollary 4 .
Radial neural networks are dense in the space of all continuous functions with respectto the topology of compact convergence, and hence satisfy cc-universality.
4 . 2 Bounded width approximation
We now turn our attention to a bounded width universal approximation result.
Theorem 5 .
Let f : R n !
R m be an asymptotically afﬁne function.
For any e > 0 , there exists acompact set K ⇢ R n and a function F : R n !
R m such that:
F is the feedforward function of a radial neural network with N = N ( f , K , e ) hiddenlayers whose widths are all n + m + 1 .
For any x 2 R n , we have | F ( x )  f ( x ) | < e .
The proof, which is more involved than that of Theorem 3 , relies on using orthogonaldimensions to represent the domain and the range of f , together with an indicator l
R p n 2 e m n .
R n
K ccccccccc 2 + ec
S 2  r  T
R m f ( c 2 )
F dimension to distinguish the two.
We regard points in R n + m + 1 as triples ( x , y , q ) wherex 2 R n , y 2 R m and q 2 R .
The proof of Theorem 5 parallels that of Theorem 3 , but insteadof mapping points in B r
i ( c i ) to c
i + e i , we map the points in B r i (( c i , 0, 0 )) to ( 0, f ( c i )  L ( 0 ) s , 1 ) ,where s is chosen such that different balls do not interfere.
The ﬁnal layer then uses anafﬁne map ( x , y , q ) 7!
L ( x ) + sy , which takes ( x , 0, 0 ) to L ( x ) , and ( 0, f ( c i )  L ( 0 ) s , 1 ) to f ( c i ) .
We remark on several additional results; see Appendix B for full statements and proofs.
The bound of Theorem 5 can be strengthened to max ( n , m ) + 1 in the case of functionsf : K !
R m deﬁned on a compact domain K ⇢ R n (i.e., ignoring asymptotic behavior).
Furthermore, with more layers, it is possible to reduce that bound to max ( n , m ) .
Model compression
In this section, we prove a model compression result.
Speciﬁcally, we provide an algorithmwhich, given any radial neural network, computes a different radial neural network withsmaller widths.
The resulting compressed network has the same feedforward functionas the original network, and hence the same value of the loss function on any batch oftraining data.
In other words, our model compression procedure is lossless .
Althoughour algorithm is practical and explicit, it reﬂects more conceptual phenomena, namely, achange-of-basis action on network parameter spaces (Section 5 . 1 ).
5 . 1 The parameter space
Suppose a fully connected network has L layers and widths given by the tuple n =( n 0 , n 1 , n 2 , . .
. , n L  1 , n L ) .
In other words, the i -th layer has input width n i  1 and outputwidth n i .
The parameter space is deﬁned as the vector space of all possible choices ofparameter values.
Hence, it is given by the following product of vector spaces:
[Equation]
We denote an element therein as a pair of tuples ( W , b ) where
W =
( W i 2 R n i ⇥ n i  1 ) Li = 1are the weights and b =
( b i 2 R n i ) Li = 1 are the biases.
To describe certain symmetries ofthe parameter space, consider the following product of orthogonal groups, with sizescorresponding to the widths of the hidden layers:
[Equation]
There is a change-of-basis action of O ( n hid ) on the parameter space Param ( n ) .
Explicitly,the tuple of orthogonal matrices Q =
( Q i )
L  1 i = 1 2 O ( n hid ) transforms the parameter values( W , b ) to Q · W : =
⇣
Q
i W i Q  1 i ⌘
L i = 1 and Q · b : =
( Q i b i )
Li = 1 , where Q 0 = id n 0 and Q L = id n L .
5 . 2 Model compression
In order to state the compression result, we ﬁrst deﬁne the reduced widths.
Namely,the reduction n red = ( n red0 , n red1 , . .
. , n red L ) of a widths vector n is deﬁned recursively bysetting n red0 = n 0 , then n red i =
min ( n i , n red
i
1 + 1 ) for i = 1, . . .
, L  1, and ﬁnally n red L =n L .
For a tuple r = ( r i : R n i !
R n i ) Li
= 1 of radial rescaling functions, we write r red =
⇣ r red i : R n red i !
R n red i ⌘ for the corresponding tuple of restrictions, which are all radialrescaling functions.
The following result relies on Algorithm 1 below.
Theorem 6 .
Let ( W , b , r ) be a radial neural network with widths n .
Let W red and b red be theweights and biases of the compressed network produced by Algorithm 1 .
The feedforward functionof the original network ( W , b , r ) coincides with that of the compressed network ( W red , b red , r red ) .
Algorithm 1 : QR Model Compression ( QR-compress ) input : W , b 2 Param ( n ) output : Q 2 O ( n hid ) and W red , b red 2 Param ( n red )
Q , W red , b red [ ] , [ ] , [ ] // initialize output lists
A[ b
W 1 ] // matrix of size n 1 ⇥ ( n 0 + 1 ) for i 1 to L  1 do // iterate through layers
Q i , R
i
QR-decomp( A i , mode = ‘complete’) //
A i =
Q i Inc i R
i
Append Q i to Q
Append ﬁrst column of R i to b red // reduced bias for layer
i
Append remainder of R i to W red //
reduced weights for layer
i
Set A i +[ b i +
W i + 1 Q i Inc i ] // matrix of size n i + 1 ⇥ ( n red i + 1 ) end
Append the ﬁrst column of A L to b red // reduced bias for last layer
Append the remainder of A L to W red // reduced weights for last layer return Q , W red , b red
We explain the notation of the algorithm.
The inclusion matrix Inc i 2 R n i ⇥ n red
i hasones along the main diagonal and zeros elsewhere.
The method QR-decomp with mode =‘complete’ computes the complete QR decomposition of the n i ⇥ ( 1 + n red
i
1 ) matrix A ias Q
i Inc i R i where Q i 2 O ( n i ) and R i is upper-triangular of size n red
i ⇥ ( 1 + n red i  1 ) .
Thedeﬁnition of n red i implies that either n red i = n red
i  1 + 1 or n red i = n i .
The matrix R i is of sizen red
i ⇥ n red i in the former case and of size n i ⇥ ( 1 + n red i  1 ) in the latter case.
Example 7 .
Suppose the widths of a radial neural network are ( 1, 8, 16, 8, 1 ) .
Then it hasÂ 4 i = 1 ( n i  1 + 1 ) n i = 305 trainable parameters.
The reduced network has widths ( 1, 2, 3, 4, 1 )and Â 4 i = 1 ( n red
i  1 + 1 )( n red i )
=
34 trainable parameters.
Another example appears in Figure 4 .
We note that the tuple of matrices Q produced by Algorithm 1 does not feature in thestatement of Theorem 6 , but is important in the proof (which appears in Appendix C).
R
R 4 R R 4 R
R 2 664 •••• 3r 2 664 • • • • • • • • • • • • • • • • 3 775 r ⇥ • • • • ⇤
R
R 2 R
R 4 R
R  ••  r 2 664 • • • • • • • • 3r ⇥ • • • • ⇤
R
R 2 R
R 3 R
R  ••  r 2 4 • • • • • • 3r ⇥ • • • ⇤
Namely, an induction argument shows that the i -th partial feedforward function of theoriginal and reduced models are related via the matrices Q i and Inc i .
A crucial ingredientin the proof is that radial rescaling activations commute with orthogonal transformations.
Projected gradient descent
The typical use case for model compression algorithms is to produce a smaller versionof the fully trained model which can be deployed to make inference more efﬁcient.
Itis also worth considering whether compression can be used to accelerate training.
Forexample, for some compression algorithms, the compressed and full models have the samefeedforward function after a step of gradient descent is applied to each, and so one cancompress before training and still reach the same minimum.
Unfortunately, in the contextof radial neural networks, compression using Algorithm 1 and then training does notnecessarily give the same result as training and then compression (see Appendix D. 6 for acounterexample).
However, QR-compress does lead to a precise mathematical relationshipbetween optimization of the two models: the loss of the compressed model after one stepof gradient descent is equivalent to the loss of (a transformed version of) the original modelafter one step of projected gradient descent.
Proofs appear in Appendix D.
To state our results, ﬁx a tuple of widths n and a tuple r =
( r i : R n i !
R n i ) Li
= 1 of radialrescaling functions.
The loss function L : Param ( n ) !
R associated to a batch of trainingdata { ( x j , y j ) } ✓ R n 0 ⇥ R n L is deﬁned as taking parameter values ( W , b ) to the sumÂ j C ( F ( x j ) , y j )
where C : R n L ⇥ R n L !
R is a cost function on the output space, and
F = F ( W , b , r ) is the feedforward of the radial neural network with parameters ( W , b ) andactivations r .
Similarly, we have a loss function L red on the parameter space Param ( n red )with reduced widths vector.
For any learning rate h > 0, we obtain gradient descent maps:g : Param ( n )
[Equation]
We will also consider, for k  0, the k -fold composition
g k = g  g  · · ·  g and similarly 265 for g red .
The projected gradient descent map on Param ( n ) is given by:
[Equation]
where the map Proj zeroes out all entries in the bottom left ( n i  n red i ) ⇥ n red i  1 submatrix of
W i  r W i L , and the bottom ( n i  n red i ) entries in b i  r b i L , for each i .
Schematically:
[Equation]
To state the following theorem, let W red , b red , Q = QR-compress ( W , b ) be the outputsof Algorithm 1 applied to ( W , b ) 2 Param ( n ) .
Hence ( W red , b red ) 2 Param ( n red ) arethe parameters of the compressed model, and Q 2 O ( n hid ) is an orthogonal parametersymmetry.
We also consider the action (Section 5 . 1 ) of Q  1 applied to ( W , b ) .
Theorem 8 .
Let W red , b red , Q = QR-compress ( W , b ) be the outputs of Algorithm 1 applied to( W , b ) 2 Param ( n ) .
Set U
= Q  1 · ( W , b )  ( W red , b red ) .
For any k  0 , we have:
[Equation]
We conclude that gradient descent with initial values ( W , b ) is equivalent to gradientdescent with initial values Q  1 · ( W , b ) since at any stage we can apply Q ± 1 to move fromone to the other.
Furthermore, projected gradient descent with initial values Q  1 · ( W , b )is equivalent to gradient descent on Param ( n red ) with initial values ( W red , b red ) since atany stage we can move from one to the other by ± U .
Neither Q nor U depends on k .
Experiments
In addition to the theoretical results in this work, we provide an implementation of
Algorithm 1 , in order to validate the claims of Theorems 6 and 8 empirically, as well as toquantify real-world performance.
Full experimental details are in Appendix E.( 1 ) Empirical veriﬁcation of Theorem 6 .
We learn the function f
( x ) =
e  x 2 from samplesusing a radial neural network with widths n = ( 1, 6, 7, 1 ) and activation the radial shiftedsigmoid h ( x ) =
1/ ( 1 + e  x + s ) .
Applying QR-compress gives a compressed radial neuralnetwork with widths n red =
Theorem 6 implies that the respective neuralfunctions F and F red are equal.
Over 10 random initializations, the mean absolute error isnegligible up to machine precision: ( 1/ N ) Â j | F ( x j )  F red ( x j ) | = 1.31 · 10  8 ± 4.45 · 10  9 .
( 2 ) Empirical veriﬁcation of Theorem 8 .
The claim is that training the transformed modelwith parameters Q  1 · ( W , b ) and objective L by projected gradient descent coincideswith training the reduced model with parameters ( W red , b red ) and objective L red byusual gradient descent.
We veriﬁed this on synthetic data as above.
Over 10 randominitializations, the loss functions after training match:
|L  L red | = 4.02 · 10  9 ± 7.01 · 10  9 .
The compressed model trains faster.
Our compression method may be applied beforetraining to produce a smaller model class which trains faster without sacriﬁcing accuracy.
We demonstrate this in learning the function f : R 2 !
R 2 sending ( t 1 , t 2 ) to ( e  t 21 , e  t 22 )using a radial neural network with widths n = ( 2, 16, 64, 128, 16, 2 ) and activation theradial sigmoid h ( r ) =
1/ ( 1 + e  r ) .
Applying QR-compress gives a compressed networkwith widths n red =
We trained both models until the training loss was 0.01.
Over 10 random initializations on our system, the reduced network trained in15.32 ± 2.53 seconds and the original network trained in 31.24 ± 4.55 seconds.
Conclusions and Discussion
This paper demonstrates that radial neural networks are universal approximators and thattheir parameter spaces exhibit a rich symmetry group, leading to a model compressionalgorithm.
The results of this work combine to build a theoretical foundation for the use ofradial neural networks, and suggest that radial neural networks hold promise for widerpractical applicability.
Furthermore, this work makes an argument for considering theadvantages of non-pointwise nonlinearities in neural networks.
There are two main limitations of our results, each providing an opportunity for futurework.
First, our universal approximation constructions currently work only for Step-ReLUradial rescaling radial activations; it would be desirable to generalize to other activations.
Additionally, Theorem 6 achieves compression only for networks whose widths satisfyn i > n i  1 + 1 for some i .
Neural networks which do not have increasing widths anywherein their architecture, such as encoders, would not be compressible.
Further extensions of this work include: First, little is currently known about the stability properties of radial neural networks during training, as well as their sensitivity toinitialization.
Second, radial rescaling activations provide an extreme case of symmetry;there may be beneﬁts to combining radial and pointwise activations within a single network, for example, through ‘block’ radial rescaling functions.
Our techniques may yieldweaker compression properties for more general radial basis functions networks; radialneural networks may be the most compressible such networks.
Third, the parameter spacesymmetries may provide a key ingredient in analyzing the gradient ﬂow dynamics ofradial neural networks and computation of conserved quantities.
Fourth, radial rescalingactivations can be used within convolutional or group-equivariant NNs.
Finally, basedon the theoretical advantages laid out in this paper, future work will explore empiricallyapplications in which we expect radial networks to outperform alternate methods.
Suchpotential applications include data spaces with circular or distance-based class boundaries.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect
the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 8 .
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work is theoretical and does not hold speciﬁc risks of negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper
conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Most of the
proofs appear in the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the
main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, howthey were chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after runningexperiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E.4 .
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally
identiﬁable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screen-
shots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation?
[N/A]