Universal approximation and model compression for radial neural networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We introduce a class of fully-connected neural networks whose activation functions, rather than being pointwise, rescale feature vectors by afunction depending only on their norm.
We call such networks radialneural networks, extending previous work on rotation equivariant networks that considers rescaling activations in less generality.
We proveuniversal approximation theorems for radial neural networks, includingin the more difﬁcult cases of bounded widths and unbounded domains.
Our proof techniques are novel, distinct from those in the pointwise case.
Additionally, radial neural networks exhibit a rich group of orthogonalchange-of-basis symmetries on the vector space of trainable parameters.
Factoring out these symmetries leads to a practical lossless model compression algorithm.
Optimization of the compressed model by gradientdescent is equivalent to projected gradient descent for the full model
Introduction
Inspired by biological neural networks, the theory of artiﬁcial neural networks has largelyfocused on pointwise (or “local”) nonlinear layers [ 46 , 14 ], in which the same functions : R !
R is applied to each coordinate independently:
[Equation]
In networks with pointwise nonlinearities, the standard basis vectors in R n can be interpreted as “neurons” and the nonlinearity as a “neuron activation.”
Research has generallyfocused on ﬁnding functions s which lead to more stable training, have less sensitivity toinitialization, or are better adapted to certain applications [ 42 , 38 , 37 , 10 , 29 ].
Many s havebeen considered, including sigmoid, ReLU, arctangent, ELU, Swish, and others.
However, by setting aside the biological metaphor, it is possible to consider a muchbroader class of nonlinearities, which are not necessarily pointwise, but instead dependsimultaneously on many coordinates.
Neural networks using such nonlinearities may yieldexpressive function classes with different advantages.
One example is radial basis networks[ 4 ], which contain nonlinearities of the form k v  c k , which depend on all coordinates of v .
However, each coordinate output is still independent.
In this paper, we introduce radial neural networks which employ non-pointwise nonlinearities called radial rescaling activations.
Freedom from the pointwise assumption allows us to design activation functions that maximize symmetry in the parameter space of the σ σ σ σ ρ ||·|| λ
W iW i
W iW i neural network.
Such networks enjoy several provable properties including high modelcompressibility, symmetry in optimization, and universal approximation.
These activationsare deﬁned by rescaling each vector by a scalar that depends only on the norm of thevector:
[Equation]
where l is a scalar-valued function of the norm.
Whereas in the pointwise setting, only thelinear layers mix information between different components of the latent features, for radialrescaling, all coordinates of the activation output vector are affected by all coordinates ofthe activation input vector.
The inherent geometric symmetry of radial rescalings makesthem particularly useful for designing equivariant neural networks [ 55 , 47 , 56 , 57 ].
In our ﬁrst set of main results, we prove that radial neural networks are in fact universalapproximators .
Speciﬁcally, we demonstrate that any asymptotically afﬁne function can beapproximated with a radial neural network, suggesting potentially good extrapolationbehavior.
Moreover, this approximation can be done with bounded width.
Our approachto proving these results differs signiﬁcantly from standard techniques used in the caseof pointwise activations due to the fact that coordinates cannot be treated independentlywhen dealing with non-pointwise activations.
In our second set of main results, we exploit parameter space symmetries of radial neuralnetworks to achieve model compression .
Using the fact that radial rescaling activationscommute with orthogonal transformations, we develop a practical algorithm to systematically factor out orthogonal symmetries via iterated QR decompositions.
This leads toanother radial neural network with fewer neurons in each hidden layer.
The resultingmodel compression algorithm is lossless : the compressed network and the original networkboth have the same value of the loss function on any batch of training data.
Furthermore, we prove that the loss of the compressed model after one step of gradientdescent is equal to the loss of the original model after one step of projected gradient descent .
As explained below, projected gradient descent involves zeroing out certain parametervalues after each step of gradient descent.
Although training the original network mayresult in a lower loss function after fewer epochs, in many cases the compressed networktakes less time per epoch to train and is faster in reaching a local minimum.
To summarize, our main contributions are:
• A formalization of radial neural networks, a new class of neural networks;• Two universal approximations results for radial neural networks: a) approximationof asymptotically afﬁne functions, and b) bounded width approximation;•
An implementation of a lossless model compression algorithm for radial neuralnetworks;
• A theorem providing the precise relationship between gradient descent optimiza-
tion of the original and compressed networks.
Related work
Radial rescaling activations.
Radial rescaling functions have the symmetry propertyof preserving vector directions, and hence exhibit rotation equivariance.
Consequently,examples of such functions, such as the squashing nonlinearity and Norm-ReLU, feature inthe study of rotationally equivariant neural networks [ 55 , 47 , 56 , 57 , 26 ].
However, previousworks apply the activation only along the channel dimension, and consider the orthogonalgroup O ( n ) only for n = 2, 3.
In contrast, we consider a radial rescaling activation acrossthe entire hidden layer, and O ( n ) -equivariance where n is the hidden layer dimension.
Ourconstructions echo the vector neurons formalism [ 15 ], in which the output of a nonlinearityis a vector rather than a scalar.
For radial basis networks, each hidden neuron is a radialnonlinear function of the shifted input vector, but the outputs are independent, whereasfor radial rescaling functions, the outputs are also tied together [ 4 ].
Universal approximation.
Neural networks of arbitrary width and sigmoid activationshave long been known to be universal approximators [ 14 ].
Universal approximation canalso be achieved by bounded width networks with arbitrary depth [ 36 ].
Additional workhas generalized to other activation functions and neural architectures [ 24 , 60 , 43 , 50 ].
Whilemost work has focused on compact domains, some recent work also considers non-compactdomains [ 28 , 54 ].
The techniques used for pointwise activation functions generalize toradial basis networks since the outputs of each RBF are independent [ 40 ], but do not easilygeneralize to the radial rescaling activations considered here, because all coordinates of theactivation output vector are affected by all coordinates of the activation input vector.
As aconsequence, individual radial neural network approximators of two different functionscannot be easily combined to give an approximator of the sum of the functions.
Groups and symmetry.
Appearances of symmetry in machine learning have generallyfocused on symmetric input and output spaces.
Most prominently, equivariant neuralnetworks incorporate symmetry as an inductive bias and feature weight-sharing constraintsbased on equivariance with respect to various symmetry groups.
Examples of equivariantarchitectures include G -convolution, steerable CNN, and Clebsch-Gordon networks [ 13 , 55 ,11 , 9 , 30 , 2 , 58 , 12 , 57 , 16 , 31 , 44 ].
By contrast, our approach to radial neural networks doesnot depend on symmetries of the input domain, output space, or feedforward mapping.
Instead, we exploit parameter space symmetries and thus obtain more general results thatapply to domains with no apparent symmetry.
Model compression.
A major goal in machine learning is to ﬁnd methods to reducethe number of trainable parameters, decrease memory usage, or accelerate inference andtraining [ 8 , 61 ].
Our approach toward this goal differs signiﬁcantly from most existingmethods in that it is based on the inherent symmetry of network parameter spaces.
Oneprior method is weight pruning , which removes redundant or small weights from a networkwith little loss in accuracy [ 20 , 3 , 27 ].
Pruning can be done during training [ 18 ] or atinitialization [ 34 , 53 ].
Gradient-based pruning identiﬁes low saliency weights by estimatingthe increase in loss resulting from their removal [ 33 , 22 , 17 , 39 ].
A complementary approachis quantization , which decreases the bit depth of weights [ 59 , 25 , 19 ].
Knowledge distillationidentiﬁes a small model mimicking the performance of a larger model or ensembleof models [ 5 , 23 , 1 ].
Matrix Factorization methods replace fully connected layers withlower rank or sparse factored tensors [ 6 , 7 , 52 , 32 , 45 , 35 ] and can often be appliedbefore training.
Our method involves a type of matrix factorization based on the QRdecomposition; however, rather than aim for a rank reduction of linear layers, we leveragethis decomposition to reduce hidden widths via change-of-basis operations on the hiddenrepresentations.
Close to our method are lossless compression methods which removestable neurons in ReLU networks [ 49 , 48 ] or exploit permutation parameter space symmetryto remove redundant neurons [ 51 ]; our compression instead follows from the symmetriesof the radial rescaling activation.
Finally, the model compression results of [ 26 ], whileconceptually similar to ours, are weaker, as ( 1 ) the unitary group action is on disjoint layersinstead of iteratively moving through all layers, and ( 2 ) the results are only stated for aversion of the squashing nonlinearity.
[Equation] [Equation]
function
Radial neural networks
In this section, we deﬁne radial rescaling functions and radial neural networks.
Leth : R !
R be a function.
For any n  1, set:
[Equation]
| v | for v 6 = 0, and h ( n ) ( 0 ) = 0.
A function r : R n !
R n is called a radial rescaling function ifr = h ( n ) for some piecewise differentiable h : R !
R .
Hence, r sends each input vector toa scalar multiple of itself, and that scalar depends only on the norm of the vector 1 .
It iseasy to show that radial rescaling functions commute with orthogonal transformations.
Example 1 .
( 1 ) Step-ReLU, where h ( r ) =
r if r  1 and 0 otherwise.
In this case, the radialrescaling function is given by
[Equation]
( 2 ) The squashing function, where h ( r ) =
r 2 / ( r 2 + 1 ) .
( 3 ) Shifted ReLU, where h ( r ) =max ( 0, r  b ) for r > 0 and b is a real number.
See Figure 2 .
We refer to [ 55 ] and thereferences therein for more examples and discussion of radial functions.
A radial neural network with L layers consists of a positive integer n i indicating the width ofeach layer i = 0, 1, . . .
, L ; the trainable parameters, comprising of a matrix W i 2 R n i ⇥ n i  1of weights and a bias vector b i 2 R n i for each i = 1, . . .
, L ; and a radial rescaling functionr i : R n i !
R n i for each i = 1, . . .
, L .
We refer to the tuple n = ( n 0 , n 1 , . . .
, n L ) as the widthsvector of the neural network.
The hidden widths vector is n hid =
( n 1 , n 2 , . .
. , n L  1 ) .
Thefeedforward function F : R n 0 !
R n L of a radial neural network is deﬁned in the usual wayas an iterated composition of afﬁne maps and activations.
Explicitly, set F 0 = id R n 0 andrecursively deﬁne the partial feedforward functions:
[Equation]
for i = 1, . . .
, L .
Then the feedforward function is F = F L .
Remark 2 .
If b i = 0 for all i , then the feedforward function takes the form F ( x ) =
W ( µ ( x ) x )where µ : R n !
R is a scalar-valued function and W =
W L W L  1 · · · W 1 2 R n L ⇥ n 0 is theproduct of the weight matrices.
If any of the biases are non-zero, then the feedforwardfunction lacks such a simple form.
Universal Approximation
In this section, we consider two universal approximation results.
The ﬁrst approximates asymptotically afﬁne functions with a network of unbounded width.
The secondgeneralizes to bounded width networks.
Proofs appear in Appendix B. Throughout,
B r ( c ) = { x 2 R n : | x  c | < r } denotes the r -ball around a point c , and an afﬁne map 152 R n !
R m is one of the from L ( x ) =
Ax + b for a matrix A 2 R m ⇥ n and b 2 R m .
4 . 1 Approximation of asymptotically afﬁne functions
A continuous function f : R n !
R m is said to be asymptotically afﬁne if there exists anafﬁne map L : R n !
R m such that, for every e > 0, there is a compact subset K of R nsuch that | L ( x )  f ( x ) | < e for all x 2 R n \ K .
In particular, continuous functions withcompact support are asymptotically afﬁne.
The continuity of f and compactness of K implythat, for any e > 0, there exist c 1 , . . .
, c N 2 K and r 1 , . .
r N 2 ( 0, 1 ) such that, ﬁrst, theunion of the balls B r i ( c i ) covers K and, second, for all i , we have f ( B r i ( c i ) \ K ) ✓ B e ( f ( c i )) .
Let N ( f , K , e ) be the minimal choice of N .
In many cases, the constant N ( f , K , e ) can bebounded explicitly 2 .
Theorem 3 (Universal approximation) .
Let f : R n !
R m be an asymptotically afﬁne function.
For any e > 0 , there exists a compact set K ⇢ R n and a function F : R n !
R m such that:
F is the feedforward function of a radial neural network with
N
= N ( f , K , e ) layers whose
hidden widths are ( n + 1, n + 2, . . . , n
+ N ) .2 .
For any x 2 R n , we have | F ( x )  f ( x ) | < e .
We note that the approximation in Theorem 3 is valid on all of R n .
To give an idea of theproof, ﬁrst ﬁx c 1 , . .
c N 2 K and r 1 , . .
, r N 2 ( 0, 1 ) as above.
Let e 1 , . .
, e N be orthonormalbasis vectors extending R n to R n + N .
For i = 1, . . .
, N deﬁne the following afﬁne maps:
[Equation]
where h i = q 1  r 2 i and h e i ,
z i is the coefﬁcient of e i in z .
Setting r
i to be Step-ReLU(Equation 3 . 1 ) on R n + i , these maps are chosen so that the composition S i  r i  T
i mapsthe points in B r i
( c i ) to c
i + e i , while keeping points outside this ball the same.
We nowdescribe a radial neural network with widths ( n , n + 1, . . . , n + N , m ) whose feedforwardfunction approximates f .
For i = 1, . . .
, N the afﬁne map from layer i  1 to layer i is givenby
z 7!
T i  S
i  1 ( z ) , with S
= id R n .
The activation at each hidden layer is Step-ReLU.
Let
L be the afﬁne map such that | L  f | < e on R n \ K .
The afﬁne map from layer N to theoutput layer is F  S N where F : R n + N !
R m is the unique afﬁne map determined byx 7!
L ( x ) if x 2 R n , and e i 7!
f ( c i )  L ( c i ) .
This construction is illustrated in Figure 3 .
4 . 2 Bounded width approximation
We now turn our attention to a bounded width universal approximation result.
Theorem 4 .
Let f : R n !
R m be an asymptotically afﬁne function.
For any e > 0 , there exists acompact set K ⇢ R n and a function F : R n !
R m such that:
F is the feedforward function of a radial neural network with N = N ( f , K , e ) hiddenlayers whose widths are all n + m + 1 .
For any x 2 R n , we have | F ( x )  f ( x ) | < e .
The proof, which is more involved than that of Theorem 3 , relies on using orthogonaldimensions to represent the domain and the range of f , together with an indicator
R n
K ccccccccc 2 + ec
S 2  r  T
R m f ( c 2 )
F dimension to distinguish the two.
We regard points in R n + m + 1 as triples ( x , y , q ) wherex 2 R n , y 2 R m and q 2 R .
The proof of Theorem 4 parallels that of Theorem 3 , but insteadof mapping points in B
r i ( c i ) to c
i + e i , we map the points in B r i (( c i , 0, 0 )) to ( 0, f ( c i )  L ( 0 ) s , 1 ) ,where s is chosen such that different balls do not interfere.
The ﬁnal layer then uses anafﬁne map ( x , y , q ) 7!
L ( x ) + sy , which takes ( x , 0, 0 ) to L ( x ) , and ( 0, f ( c i )  L ( 0 ) s , 1 ) to f ( c i ) .
We remark on several additional results; see Appendix B for full statements and proofs.
The bound of Theorem 4 can be strengthened to max ( n , m ) + 1 in the case of functionsf : K !
R m deﬁned on a compact domain K ⇢ R n (i.e., ignoring asymptotic behavior).
Furthermore, with more layers, it is possible to reduce that bound to max ( n , m ) .
Model compression
In this section, we prove a model compression result.
Speciﬁcally, we provide an algorithmwhich, given any radial neural network, computes a different radial neural network withsmaller widths.
The resulting compressed network has the same feedforward functionas the original network, and hence the same value of the loss function on any batch oftraining data.
In other words, our model compression procedure is lossless .
Althoughour algorithm is practical and explicit, it reﬂects more conceptual phenomena, namely, achange-of-basis action on network parameter spaces (Section 5 . 1 ).
5 . 1 The parameter space
Suppose a fully connected network has L layers and widths given by the tuple n =( n 0 , n 1 , n 2 , . .
. , n L  1 , n L ) .
In other words, the i -th layer has input width n i  1 and outputwidth n i .
The parameter space is deﬁned as the vector space of all possible choices ofparameter values.
Hence, it is given by the following product of vector spaces:
[Equation]
We denote an element therein as a pair of tuples ( W , b ) where
W =
( W i 2 R n i ⇥ n i  1 ) Li = 1are the weights and b =
( b i 2 R n i ) Li = 1 are the biases.
To describe certain symmetries ofthe parameter space, consider the following product of orthogonal groups, with sizescorresponding to the widths of the hidden layers:
[Equation]
There is a change-of-basis action of O ( n hid ) on the parameter space Param ( n ) .
Explicitly,the tuple of orthogonal matrices Q =
( Q i )
L  1 i = 1 2 O ( n hid ) transforms the parameter values( W , b ) to Q · W : =
⇣
Q
i W i Q  1 i ⌘
L i = 1 and Q · b : =
( Q i b i )
Li = 1 , where Q 0 = id n 0 and Q L = id n L .
We write Q · ( W , b ) for ( Q · W , Q · b ) .
5 . 2 Model compression
In order to state the compression result, we ﬁrst deﬁne the reduced widths.
Namely,the reduction n red = ( n red0 , n red1 , . .
. , n red L ) of a widths vector n is deﬁned recursively bysetting n red0 = n 0 , then n red i =
min ( n i , n red
i
1 + 1 ) for i = 1, . . .
, L  1, and ﬁnally n red L =n L .
For a tuple r = ( r i : R n i !
R n i ) Li
= 1 of radial rescaling functions, we write r red =
⇣ r red i : R n red i !
R n red i ⌘ for the corresponding tuple of restrictions, which are all radialrescaling functions.
The following result relies on Algorithm 1 below.
Theorem 5 .
Let ( W , b , r ) be a radial neural network with widths n .
Let W red and b red be theweights and biases of the compressed network produced by Algorithm 1 .
The feedforward functionof the original network ( W , b , r ) coincides with that of the compressed network ( W red , b red , r red ) .
Algorithm 1 : QR Model Compression ( QR-compress ) input : W , b 2 Param ( n ) output : Q 2 O ( n hid ) and W red , b red 2 Param ( n red )
Q , W red , b red [ ] , [ ] , [ ] // initialize output lists
A[ b
W 1 ] // matrix of size n 1 ⇥ ( n 0 + 1 ) for i 1 to L  1 do // iterate through layers
Q i , R
i
QR-decomp( A i , mode = ‘complete’) //
A i =
Q i Inc i R
i
Append Q i to Q
Append ﬁrst column of R i to b red // reduced bias for layer
i
Append remainder of R i to W red //
reduced weights for layer
i
Set A i +[ b i +
W i + 1 Q i Inc i ] // matrix of size n i + 1 ⇥ ( n red i + 1 ) end
Append the ﬁrst column of A L to b red // reduced bias for last layer
Append the remainder of A L to W red // reduced weights for last layer return Q , W red , b red
We explain the notation of the algorithm.
The inclusion matrix Inc i 2 R n i ⇥ n red
i hasones along the main diagonal and zeros elsewhere.
The method QR-decomp with mode =‘complete’ computes the complete QR decomposition of the n i ⇥ ( 1 + n red
i
1 ) matrix A ias Q
i Inc i R i where Q i 2 O ( n i ) and R i is upper-triangular of size n red
i ⇥ ( 1 + n red i  1 ) .
Thedeﬁnition of n red i implies that either n red i = n red
i  1 + 1 or n red i = n i .
The matrix R i is of sizen red
i ⇥ n red i in the former case and of size n i ⇥ ( 1 + n red i  1 ) in the latter case.
Example 6 .
Suppose the widths of a radial neural network are ( 1, 8, 16, 8, 1 ) .
Then it hasÂ 4 i = 1 ( n i  1 + 1 ) n i = 305 trainable parameters.
The reduced network has widths ( 1, 2, 3, 4, 1 )and Â 4 i = 1 ( n red
i  1 + 1 )( n red i )
=
34 trainable parameters.
Another example appears in Figure 4 .
We note that the tuple of matrices Q produced by Algorithm 1 does not feature in thestatement of Theorem 5 , but is important in the proof (which appears in Appendix C).
R
R 4 R R 4 R
R 2 664 •••• 3r 2 664 • • • • • • • • • • • • • • • • 3 775 r ⇥ • • • • ⇤
R
R 2 R
R 4 R
R  ••  r 2 664 • • • • • • • • 3r ⇥ • • • • ⇤
R
R 2 R
R 3 R
R  ••  r 2 4 • • • • • • 3r ⇥ • • • ⇤
Namely, an induction argument shows that the i -th partial feedforward function of theoriginal and reduced models are related via the matrices Q i and Inc i .
A crucial ingredientin the proof is that radial rescaling activations commute with orthogonal transformations.
Projected gradient descent
The typical use case for model compression algorithms is to produce a smaller versionof the fully trained model which can be deployed to make inference more efﬁcient.
Itis also worth considering whether compression can be used to accelerate training.
Forexample, for some compression algorithms, the compressed and full models have the samefeedforward function after a step of gradient descent is applied to each, and so one cancompress before training and still reach the same minimum.
Unfortunately, in the contextof radial neural networks, compression using Algorithm 1 and then training does notnecessarily give the same result as training and then compression (see Appendix D. 6 for acounterexample).
However, QR-compress does lead to a precise mathematical relationshipbetween optimization of the two models: the loss of the compressed model after one stepof gradient descent is equivalent to the loss of (a transformed version of) the original modelafter one step of projected gradient descent.
Proofs appear in Appendix D.
To state our results, ﬁx a tuple of widths n and a tuple r =
( r i : R n i !
R n i ) Li
= 1 of radialrescaling functions.
The loss function L : Param ( n ) !
R associated to a batch of trainingdata { ( x j , y j ) } ✓ R n 0 ⇥ R n L is deﬁned as taking parameter values ( W , b ) to the sumÂ j C ( F ( x j ) , y j )
where C : R n L ⇥ R n L !
R is a cost function on the output space, and
F = F ( W , b , r ) is the feedforward of the radial neural network with parameters ( W , b ) andactivations r .
Similarly, we have a loss function L red on the parameter space Param ( n red )with reduced widths vector.
For any learning rate h > 0, we obtain gradient descent maps:g : Param ( n )
[Equation]
We will also consider, for k  0, the k -fold composition
g k = g  g  · · ·  g and similarly 264 for g red .
The projected gradient descent map on Param ( n ) is given by:
[Equation]
where the map Proj zeroes out all entries in the bottom left ( n i  n red i ) ⇥ n red i  1 submatrix of
W i  r W i L , and the bottom ( n i  n red i ) entries in b i  r b i L , for each i .
Schematically:
[Equation]
To state the following theorem, let W red , b red , Q = QR-compress ( W , b ) be the outputsof Algorithm 1 applied to ( W , b ) 2 Param ( n ) .
Hence ( W red , b red ) 2 Param ( n red ) arethe parameters of the compressed model, and Q 2 O ( n hid ) is an orthogonal parametersymmetry.
We also consider the action (Section 5 . 1 ) of Q  1 applied to ( W , b ) .
Theorem 7 .
Let W red , b red , Q = QR-compress ( W , b ) be the outputs of Algorithm 1 applied to( W , b ) 2 Param ( n ) .
Set U
= Q  1 · ( W , b )  ( W red , b red ) .
For any k  0 , we have:
[Equation]
We conclude that gradient descent with initial values ( W , b ) is equivalent to gradientdescent with initial values Q  1 · ( W , b ) since at any stage we can apply Q ± 1 to move fromone to the other.
Furthermore, projected gradient descent with initial values Q  1 · ( W , b )is equivalent to gradient descent on Param ( n red ) with initial values ( W red , b red ) since atany stage we can move from one to the other by ± U . Neither Q nor U depends on k .
Experiments
In addition to the theoretical results in this work, we provide an implementation of
Algorithm 1 , in order to validate the claims of Theorems 5 and 7 empirically, as well as toquantify real-world performance.
Full experimental details are in Appendix E.( 1 ) Empirical veriﬁcation of Theorem 5 .
We learn the function f
( x ) =
e  x 2 from samplesusing a radial neural network with widths n = ( 1, 6, 7, 1 ) and activation the radial shiftedsigmoid h ( x ) =
1/ ( 1 + e  x + s ) .
Applying QR-compress gives a compressed radial neuralnetwork with widths n red =
Theorem 5 implies that the respective neuralfunctions F and F red are equal.
Over 10 random initializations, the mean absolute error isnegligible up to machine precision: ( 1/ N ) Â j | F ( x j )  F red ( x j ) | = 1.31 · 10  8 ± 4.45 · 10  9 .
( 2 ) Empirical veriﬁcation of Theorem 7 .
The claim is that training the transformed modelwith parameters Q  1 · ( W , b ) and objective L by projected gradient descent coincideswith training the reduced model with parameters ( W red , b red ) and objective L red byusual gradient descent.
We veriﬁed this on synthetic data as above.
Over 10 randominitializations, the loss functions after training match:
|L  L red | = 4.02 · 10  9 ± 7.01 · 10  9 .
The compressed model trains faster.
Our compression method may be applied beforetraining to produce a smaller model class which trains faster without sacriﬁcing accuracy.
We demonstrate this in learning the function f : R 2 !
R 2 sending ( t 1 , t 2 ) to ( e  t 21 , e  t 22 )using a radial neural network with widths n = ( 2, 16, 64, 128, 16, 2 ) and activation theradial sigmoid h ( r ) =
1/ ( 1 + e  r ) .
Applying QR-compress gives a compressed networkwith widths n red =
We trained both models until the training loss was 0.01.
Over 10 random initializations on our system, the reduced network trained in15.32 ± 2.53 seconds and the original network trained in 31.24 ± 4.55 seconds.
Conclusions and Discussion
This paper demonstrates that radial neural networks are universal approximators and thattheir parameter spaces exhibit a rich symmetry group, leading to a model compressionalgorithm.
The results of this work combine to build a theoretical foundation for the use ofradial neural networks, and suggest that radial neural networks hold promise for widerpractical applicability.
Furthermore, this work makes an argument for considering theadvantages of non-pointwise nonlinearities in neural networks.
There are two main limitations of our results, each providing an opportunity for futurework.
First, our universal approximation constructions currently work only for Step-ReLUradial rescaling radial activations; it would be desirable to generalize to other activations.
Additionally, Theorem 5 achieves compression only for networks whose widths satisfyn i > n i  1 + 1 for some i .
Neural networks which do not have increasing widths anywherein their architecture, such as encoders, would not be compressible.
Further extensions of this work include: First, little is currently known about the stability properties of radial neural networks during training, as well as their sensitivity toinitialization.
Second, radial rescaling activations provide an extreme case of symmetry;there may be beneﬁts to combining radial and pointwise activations within a single network, for example, through ‘block’ radial rescaling functions.
Third, the parameter spacesymmetries may provide a key ingredient in analyzing the gradient ﬂow dynamics ofradial neural networks and computation of conserved quantities.
Fourth, radial rescalingactivations can be used within convolutional or group-equivariant NNs.
Finally, basedon the theoretical advantages laid out in this paper, future work will explore empiricallyapplications in which we expect radial networks to outperform alternate methods.
Suchpotential applications include data spaces with circular or distance-based class boundaries.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect
the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 8 .
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work is theoretical and does not hold speciﬁc risks of negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper
conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Most of the
proofs appear in the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the
main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, howthey were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used(e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E.4 .
If you are using existing assets (e.g., code, data, models) or curating/releasing newassets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as aURL?
[N/A](d)
Did you discuss whether
and how consent was obtained from people whosedata you’re using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally
[Table]
identiﬁable information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screen-
[Table]
shots, if applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional
[Table]
Review Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total
[Table]
amount spent on participant compensation?
[N/A]