Universality of group convolutional neural networks based on ridgelet analysis on groups
Anonymous Author(s)
AffiliationAddress email
Abstract
We investigate the approximation property of group convolutional neural networks(GCNNs)
based on the ridgelet theory.
We regard a group convolution as a matrixelement of a group representation, and formulate a versatile GCNN as a nonlinearmapping between group representations, which covers typical GCNN literaturessuch as a cyclic convolution on a multi-channel image, permutation-invariantdatasets (Deep Sets), and E p n q -equivariant convolutions.
The ridgelet transformis an analysis operator of a depth-2 network, namely, it maps an arbitrary giventarget function f to the weight γ of a network S r γ s so that the network representsthe function as S r γ s “ f .
It has been known only for fully-connected networks,and this study is the first to present the ridgelet transform for (G)CNNs .
Since theridgelet transform is given as a closed-form integral operator, it provides a constructive proof of the cc -universality of GCNNs .
Unlike previous universality argumentson CNNs, we do not need to convert/modify the networks into other universalapproximators such as invariant polynomials and fully-connected networks.
Introduction
In the research field of geometric deep learning [1] group convolutional neural networks (GCNNs)have been developed to capture the inductive bias behind a variety of datasets such as sets and pointclouds [2, 3], graphs [4, 5], manifolds, groups and homogeneous spaces [6–8, 4].
In the theoreticalaspect, however, the approximation property of CNNs is less investigated than that of fully-connectedneural networks (FNNs).
To this date, several authors have shown that some CNNs are universal , thatis, they can approximate some class of continuous maps with any precision [9–15].
These studies arestill limited because (1) the network components such as the activation function and convolution arelimited, and/or (2) the proofs are indirect or existential since they rely on the universality of otheruniversal approximators such as FNNs and invariant polynomials. (See § 6 for more details).
The approximation property of FNNs has been investigated in the 1990s, with gradually increasingthe resolution of proofs from abstract to concrete, starting from existential proofs based on the Hahn-Banach theorem [16] and the Stone-Weierstrass theorem [17], indirect proofs based on the Fourier transform [18, 19], the Radon transform [20, 21], B -splines [22, 23], to constructive proofsbased on the integral representation [24], ridge functions [25], and the ridgelet transform [26–28].
For deep-ReLU-FNNs, further approximation properties have been investigated [29–31] in the 2010s.
In this study, we show the universality of GCNNs by developing the ridgelet transform for GCNNs —an integral transform (or analysis operator) that maps a given function f to the weight parameter γ ina single hidden layer of a neural network. By virtue of the ridgelet transform, our universality proofis more unified and constructive .
Some previous studies have also presented construction algorithmsof network parameters, but they are case-by-case, and if the network architecture changes, theconstruction algorithm has to be manually reprogrammed from scratch.
On the contrary, our ridgelet transform is devised for a wide range of GCNNs in a unified manner, and its discretization (numericalintegration, or finite sum approximation) process automatically turns out to be a construction algorithmof network parameters.
To overview the main contributions, we describe the formulation of GCNNs.
A Typical Convolution Layer for Images.
Given an m 1 ˆ m 2 -dimensional n in -channel inputimage x P R m 1 ˆ m 2 ˆ n in , a typical convolution layer with w 1 ˆ w 2 -dimensional n in ˆ n out -channelfilter a P R w 1 ˆ w 2 ˆ n in ˆ n out and n out -channel bias b P R n out followed by the elementwise activationfunction σ : R Ñ R and the aggregation with output coefficients c P R n out is given by
[Equation]
for each pixel at p i, j q P p m 1 ´ w 1 ` 1 q ˆ p m 2 ´ w 2 ` 1 q .
Here, in this study, the output channels (indexed by n out ) are aggregated soon after the activationfunction, which may be slightly different from an ordinary formulation of CNNs.
We impose thisaggregation for technical reasons, but we may understand this as a part of the subsequent layer.
In the standard formulation of GCNNs, the convolution in the pixel directions p i, j q is reformulatedas a group convolution with respect to the product group G “ Z m 1 ˆ Z m 2 of cyclic groups
Z m ip i “ 1 , 2 q .
So, the input image is understood as a function on a group G (or a homogeneous space
G { H on which G acts).
In addition, the inner product in the channel direction k is also understood asa part of the convolution with respect to the trivial action of G on the ‘fiber’ R n in ([32]).
In this sense,the multi-channel image is understood as a vector-valued function (or a section of a fiber bundle).
The Integral Representation S r γ s of Group Convolution Layer.
Let G be an arbitrary group,σ : R Ñ R be an arbitrary nonlinear function, and H be an arbitrary vector space of feature vectorx and filter a .
In this study, we formulate a group convolution layer in an integral form, called theintegral representation , as
[Equation]
This is an infinite-dimensional linear representation of a depth-2 group convolutional neural network.
Here, each function x ÞÑ σ pp a ˚ x qp g q ´ b q represents a single convolutional neuron, or a feature mapof input x parametrized by p a, b q .
The integration over p a, b q implies that all the possible neuronsare assigned in advance, and thus S r γ s can be understood as a continuous neural network .
We note,however, that if we take γ to be a finite sum of Dirac’s measures such as γ n : “ ř nℓ “ 1 c ℓ δ p a ℓ ,b ℓ q , thenthe integral representation can also exactly reproduce a finite model: S r γ n sp x qp g q “ ř nℓ “ 1 c ℓ σ pp a ℓ ˚x qp g q ´ b ℓ q for x P H , g P G.
In summary, S r γ s is a mathematical model of shallow neural networkswith any width ranging from finite to infinite.
In particular, if the parameter distribution γ is localizedand concentrated, then it means the filters a ℓ are sparse and low-rank. An advantage to use the integral representation is the linearization effect (of a single hidden layer). Whereas a finite network is nonlinear in the original parameters p a, b q , the integral representation islinear in the parameter distribution γ . It is first emerged in the 1990s to investigate the expressivepower of infinitely-wide shallow FNNs [18–21, 24, 26–28]; and it is as well common in today’sdeep learning theory, for example, to investigate the learning dynamics of SGD such as neuraltangent kernel (NTK) [33–35], lazy learning [36], lottery tickets [37], mean field theory [38–42], and
Langevin dynamics [43]. The Ridgelet Transform R r f ; ρ sp a, b q is a right inverse (or pseudo-inverse) operator of the integralrepresentation operator S . It is given as a closed-form integral operator
[Equation]
where f : H Ñ C G is a target vector-valued function to be approximated, e P G is the identityelement, and ρ : R Ñ C is an auxiliary function, called the ridgelet function .
Under mild conditions,it satisfies the reconstruction formula
[Equation]
where pp¨ , ¨qq denote a scalar product of σ and ρ .
Therefore, as long as the product pp σ, ρ qq is neither 0nor 8 , we can normalize ρ to satisfy pp σ, ρ qq “ 1 so
that S r R r f ; ρ ss “ f .
In other words, R and S are analysis and synthesis operators, and thus play the same roles as the
Fourier ( F ) and inverse Fourier ( F ´ 1 ) transforms respectively, in the sense that the reconstructionformula S r R r f ; ρ ss “ pp σ, ρ qq f corresponds to the Fourier inversion formula F ´ 1 r F r f ss “ f . An advantage to use the ridgelet transform is the closed-form expression .
Despite the common beliefthat neural network parameters are a blackbox, the closed-form expression of ridgelet transform clearlydescribes how the network parameters are organized. Previous studies on the CNN universality havealso provided several construction algorithms of parameters, but these are only particular solutionsfor a CNN to represent a target function f , and not necessary related to, for example, deep learningsolutions.
For FNNs , on the other hand, Sonoda et al. [44, 45] have shown that any parameterdistribution γ satisfying S r γ s “ f can always be represented as (not always single but) a linearcombination of ridgelet transforms, and that finite networks trained by regularized empirical riskminimization (RERM) converges to a certain unique ridgelet transform. (We note that other theoriessuch as NTK and Gibbs distribution can also provide the final distribution, but they are more limitedor indirect.)
Furthermore, Savarese et al.
[46] and their followers [47–50] have established therepresenter theorems for ReLU-FNNs by using the ridgelet transform. For CNNs, similar resultshave not yet been presented, but we expect that the ridgelet transform for GCNN will be a key tofacilitate our understanding of deep learning solutions. Challenges and Contributions.
Since the original ridgelet transform was discovered in the 1990sindependently by Murata [26], Candès [27] and Rubin [28], the closed-form expression of the ridgelettransform has been known only for FNNs. (We refer to [51–53] for ridgelet analysis in the 2000s,and [54–56] for more recent results.) One of the difficulties to devise the ridgelet transform for CNNsis that there is no unique way to formulate an “integral representation of CNNs”. (We note that someauthors claim the “equivalence of CNNs and FNNs,” which is simply misleading because such anequivalence holds only for very carefully designed CNNs and FNNs.)
While FNNs are defined on
Euclidean spaces R m , (G)CNNs are defined on a more abstract space of features, denoted H . Forexample, since the convolution on the Euclidean space can be written using Töplitz matrices, onecould consider a formulation such as [Equation] b where the parameter A is a matrix,but this leads to a completely different ridgelet transform covering less symmetries (related to theridgelet transform in the 2000s [51]).
To circumvent this difficulty, we formulate GCNNs as geometric as possible by dealing with thefeature space H , group G , and representation T in a coordinate-free manner.
Eventually, we haveshown the reconstruction formula for a wide range of GCNNs (as instantiated in the Examplessection), with a relatively simple proof.
This study is the first to derive the ridgelet transform for ageneral class of GCNNs. As an application, we show the cc -universality of GCNNs for a generalclass of group equivalent continuous vector-valued functions in a unified and constructive manner.
Preliminaries
2.1 Notation
For any topological space X , C p X q , C 0 p X q and C c p X q denote the collections of all continuousfunctions on X , continuous functions on X that vanish at infinity, and continuous functions on Xwith compact support, respectively.
We note that when X is compact, then these three spaces areequivalent, i.e., C p X q “ C 0 p X q “ C c p X q .
For any integer d ą 0 , S p R d q and S 1 p R d q denote the classes of Schwartz test functions (or rapidlydecreasing functions) and tempered distributions on R d , respectively.
Namely, S 1 is the topologicaldual of S .
In this study, we employ S 1 p R q and S p R q to be classes of activation and ridgelet functions,respectively. We note that S 1 p R q includes truncated power functions σ p b q “ b k ` “ max t b, 0 u k suchas step function for k “ 0 and ReLU for k “ 1 .
Fourier Transform The Fourier transform on the Euclidean space R d and its inversion formulahas been defined on (at least) three different function classes:
L 1 p R d q , L 2 p R d q and S 1 p R d q .
Whenf P L 1 p R d q and p f P L 1 p R d q , the inversion formula holds “at every continuous point x of f ”, which is a pointwise equality.
When f P L 2 p R d q , the inversion formula holds “in L 2 ”, which is not apointwise equality because the equality “ f “ g in L 2 ” is defined as “ f p x q “ g p x q a.e.”.
Similarly,when f P S 1 p R m q , the inversion formula holds “in S 1 ”.
We use the third definition for computing the Fourier transform of activation functions σ P S 1 p R q such as ReLU and tanh .
We refer to [57, 58, 54]for more details on the Fourier transform for tempered distributions ( S 1 ).
2.2 Group Representation
Let G be a group, let H be a vector space over a field K , and let
GL p H
q be the general linear groupon H .
A group representation T of the group G on the vector space H is a group homomorphismfrom G to GL p H q , that is, a map T : G Ñ GL p H q ; g ÞÑ T g satisfying T gh “ T g T h for all g, h P G . When G is a topological group, we further assume that the action G ˆ H Ñ H ; p g, x q ÞÑ T g r x s becontinuous.
Here, H is called the representation space , or sometimes the G -module .
Regular Representation.
Let H be the vector space of all functions on G , i.e., H “ K G .
The (left)regular representation L is a group representation defined on H as
[Equation]
In particular, when G is a locally compact Haussdorf (LCH) group, then it has a (left) invariantmeasure µ , and we can define the collection L 2 p G q of all square integrable functions on G withrespect to the canonical inner product x x, y y L 2 p G q : “ ş
G x p g q y p g q d µ p g q for any measurable functionsx, y : G Ñ C .
It is known that the regular representation on H “ L 2 p G q is a unitary representation.
Dual Representation.
For any group representation T : G Ñ GL p H q , the dual representation
T ˚ : G Ñ GL p H 1 q is a group representation defined on the dual vector space H 1 as the transpose of T g ´ 1 , that is, T ˚ g “ T J g ´ 1 .
When H is a Hilbert space with inner product x¨ , ¨y H , then it satisfies thefollowing relation:
[Equation]
Matrix Element.
For any group representation T : G Ñ GL p H q , the matrix element (or the matrixcoefficient ) of T is a bilinear functional f a,x on G defined by
[Equation]
where x is a vector in H and a P H 1 is a continuous linear functional on H .
When H is a Hilbertspace, then (abusing the notations a P H 1 and a P H ) it can be written as
[Equation]
In the next section, we use this quantity as the generalized form of the group convolution .
2.3 Universality
The notion of universality in machine learning can be rephrased as the density in mathematics, andthus it has several definitions [59, 60].
In this study, we show the so-called cc -univesality , one of thestandard universalities in the neural network theory.cc -Universality.
Let X be a topological space, and let NN be a collection of functions (e.g., neuralnetworks) on X .
The cc -univesality of NN is defined as the density of NN in C
p X q endowed withthe topology of compact convergence , that is, for any compact subset K Ă X , continuous functionf P C p K q , and all ε ą 0 , there exists a function g P NN such that
[Equation]
where g | K denotes the restriction of g to K .
Formulations
We introduce the notions of p G, T q -convolution and p G, T q -equivariance, which are natural extensionsof group convolution and group equivariance, respectively.
Then, we introduce analysis tools forabstract feature vectors f : H Ñ C G such as a norm and a measure.
3.1 p G, T q -Convolution
Definition 1.
Let G be a group, let H be a Hilbert space with inner product x¨ , ¨y H over a field K ,and let T : G Ñ GL
p H q be a representation of G on H .
For any a, x P H and g P G , we define thep G, T q -convolution as
[Equation]
First of all, this is simply a paraphrase of the matrix element of a group representation (see theprevious section).
Nevertheless, we call it a convolution simply because it covers a wide range ofpreviously proposed ‘group convolutions’ in the GCNN literature.
For example, an orthodox group convolution is reproduced when T is the regular representation (of a
LCH group G ) on H “ L 2 p G q , i.e., T ˚ g r a sp h q “ a p g ´ 1 h q .
In fact,
[Equation]
where r a p g q : “ a p g ´ 1 q is an involution. Specifically, the cyclic convolution for an n -channel imagex “ p x k ij q P R m 1 ˆ m 2 ˆ n is understood as the case when G “ Z m 1 ˆ Z m 2 , H “ R m 1 ˆ m 2 ˆ n and
[Equation] p
[Equation]
i, j, k
[Equation]
p
[Equation]
thus x x
[Equation] [Equation]
On the other hand, the ‘input feature space’ H can be an arbitrary abstract vector space, which ismore abstract than the conventional group convolution where the feature space is instantiated as afunction space on G , such as L 2 p G q and L 2 p G { H q .
We consider this as an advantage for geometricunderstanding of CNNs, since the theory becomes free from the specific instantiation of the feature x .
3.2 p G, T q -Equivariance
Associated with the extended convolution, we introduce the notion of equivariance as below.
Definition 2.
We say a function f : H Ñ C G is p G, T
q -equivariant when
[Equation] [Equation]
Here, we restrict the term ‘equivariance’ for a special case of the regular representation, which issimply because our GCNN satisfies this case.
3.3 p G, T q -Equivariant Continuous Vector-Valued Function f : H Ñ C p G q
Let G be a topological group.
For any vector-valued function f : H Ñ C G , put
[Equation]
[Equation]
By C equi p H ; C p G qq , we denote the normed vector space of all p G, T q -equivariant C p G q -valuedcontinuous functions on H .
We note that in the cc -universality argument, we deal with the topology ofcompact convergence, which is weaker than the topology of uniform norm }¨} C p H ; C p G qq .
Specifically, C equi p H ; C p G qq need not be a Banach space.
3.4 Fourier Transform on The Abstract Feature Space H
For the sake of simplicity, we assume that the real dimension m : “ dim R H of H is finite .
Lebesgue Measure on H .
We define the Lebesgue measure λ on H by pushing forward the
Lebesgue measure d x on R m via an isometric linear embedding ϕ : R m Ñ H .
For example, lett e i u i
Pr m s be an orthonormal basis of H , and put a linear embedding ϕ p x q : “ ř i
Pr m s x i e i , then (itpreserves the length and) we can induce the Lebesgue measure λ on H as the push forward measure λ “ ϕ 7 d x so that the volume of a hypercube E “ t ř i Pr m s c
i e i |
c i P r a i , b i su in H is calculated asin R m : λ p E q “ ś i Pr m s |
b i ´ a i | , and the integration is calculated as
[Equation]
ż
[Equation]
(15) for any measurable subset E Ă H and measurable function f : H Ñ C .
As far as there is no risk ofconfusion, we denote d x instead of d λ p x q .
Fourier Transform on H .
Based on λ , we induce the Fourier transform on R m as below: For anyfunction f : H Ñ C ,
[Equation]
Here, the equality ‹ “ holds in at least three different senses (see the comments in § 2.1).
Again, H remains abstract.
Depending on the embedding ϕ , each instantiation of H looks different.
Nevertheless, these share the equivalent structures induced from R m .
We consider this as also anadvantage for geometric understanding of CNNs.
Main Results
Based on the formulations in the previous sections, we define the p G, T q -convolutional neuralnetworks and the corresponding ridgelet transform.
Throughout this section, T : G Ñ GL p H q is a representation of a group G on a (potentially infinitedimensional) Hilbert space H over a field K endowed with an inner product x¨ , ¨y H , and H m Ă H isa finite-dimensional closed subspace of H equipped with the Lebesgue measure λ .
Let P : H Ñ H mdenote the orthogonal projection onto H m , and let H K m denote the orthogonal complement of H m in H . Let m : “ dim R H m be the real dimension of H m , and k : “ dim R K be the real dimension of K ,that is, k “ 1 for K “ R and k “ 2 for K “ C .
4.1 Integral Representation of p G, T q -Convolutional Neural Network
Definition 3.
For any functions γ : H m ˆ K Ñ C and σ : K Ñ C , we define the integralrepresentation of p G, T q -convolutional neural network as a vector-valued function H Ñ C G ,
[Equation]
Here, we call γ a coefficient function, and σ an activation function.
If there is no risk of confusion,we abbreviate d λ p a q as d a .
It is easy to see that a p G, T q -CNN is p G, T q -equivariant.
In fact, for every g, h P G ,
[Equation] [Equation]
In addition, at the identity element g “ e , it is reduced to a fully-connected network: S r γ sp x qp e q “ 227 ş
H m ˆ K γ p
a, b q σ px x, a y H ´
b q d a d b satisfying a projection property :
[Equation]
In order for the p G, T q -convolution a ˚ T x to be well-defined, H needs to be a G -invariant space:
T G r H s : “ t T g r x s | g P G, x P H u Ă H , which is automatically satisfied when T is a grouprepresentation, whereas T G r H m s Ć H m in general.
On the other hand, in order for the Lebesguemeasure λ to be well-defined, H m needs to be a finite-dimensional space.
4.2 Ridgelet Transform and Scalar Product of Activation Function
Definition 4.
For any functions f : H m Ñ C G and ρ : K Ñ C , we define the ridgelet transform as
[Equation]
Here e denotes the identity element of G .
Definition 5.
For any tempered distribution σ P S 1 p K q and function ρ P S p K q , put a scalar productas
[Equation]
Here, ¨ 7 denotes the Fourier transform on K , which is identified with the Fourier transform on R kwith k “ dim R K .
We note that σ 7 is defined in the sense of tempered distributions.
The derivations of the ridgelet transform and the scalar product are clarified in the proof of thereconstruction formula.
Some readers who are familiar with the classical ridgelet transform maynotice that the ridgelet transform for GCNN is formally the same as the one for fully-connected
NNs.
This may look asymmetric, since ρ pp a ˚ T x qp g q ´ b q looks intuitively more symmetric thanρ px x, a y H ´ b q for the ridgelet transform for GCNNs.
In fact, this symmetry is not broken, becauseat the identity element g “ e , the convolution is reduced to the inner product: x x, a y H “ p a ˚ T x qp e q ,thus we can understand the ‘ridgelet transform with ρ pp a ˚ T x qp g q ´ b q ’ is applied at the singlepoint g “ e , which works because (in the main theorem) the target function is assumed to bep G, T q -equivariant.
4.3 Reconstruction Formula
Here, we state the first half of our main results.
For a function f : H m Ñ C G , put f e p x q : “ f p x qp e qfor any x P H m , and let P 7 f : H Ñ C G {+denote a pullback defined by P 
 f p z qp g+} q : “ f p P r z sqp g qfor every z P H and g P G . We note that when H m “ H and thus P “ Id , then P 7 f “ f .
Since P 7 f p x ‘ y q “ f p x ‘ 0 q for any x ‘ y P H m ‘ H K m , P 7 f is an extension of f to H as a constantfunction over H K m . Theorem 1 (Main Theorem 1/2) .
Given a function f : H m Ñ C G , assume (A1) that P 7 f : H Ñ C Gis p G, T q -equivariant, i.e.,
[Equation]
and (A2) that f satisfies at least one of the following conditions: (A2a) both f e and p f e are absoluteintegrable, i.e., f e ,
p f e P L 1 p H m q , (A2b)
f e is square-integrable, i.e., f e P L 2 p H m q , or (A2c)
f e is atempered distribution, i.e., f e P S 1 p H m q .
Then, the following reconstruction formula holds:
[Equation]
where the equality ‹ “ holds at every continuous point x c of f for (A2a), in L 2 for (A2b), and in S 1 for(A2c), respectively.
The proof is given in Appendix A.1.
4.4 cc -Universality
Finally, we state the second half of our main results.
Since the reconstruction formula holds for anarbitrary function f , by discretizing the reconstruction formula S r γ s “ f with γ “ R r f ; ρ s into afinite sum
[Equation]
in a ‘nice’ manner so that the discretized coefficient function γ n : “ ř ni “ 1 c i δ p a
i ,b i q reproducesγ “ R r f ; ρ s in a limit of n Ñ 8 , we can construct a sequence t f n u n P N of finite p G, T q -CNNs thatconverges to an arbitrary target function f ,
[Equation]
This is the primitive idea behind the constructive proof of the following cc -universality of finitep G, T q -CNNs based on ridgelet analysis.
To state an assumption on the activation function σ , we introduce the forward difference operator ∆ nθwith difference θ ą 0 , defined as
[Equation]
Theorem 2 (Main Theorem 2/2) .
For an activation function σ P S 1 p K q , assume (A3)
that there existn P N and θ ą 0 such that ∆ nθ r σ s is bounded and Lipschitz continuous.
Let NN be the collection offinite p G, T q -CNNs, that is, NN : “ Ť n P N t f n p x qp g q “ ř ni “ 1 c i σ pp a i ˚ T x qp
g q ´ b
i q | p a i , b i , c i q P
H m ˆ K ˆ C , i P r n su .
Then, NN is cc -universal, that is, for any p G, T q -equivariant C p G q -valuedcontinuous function f P C equi p H m ; C p G qq , and for any compact sets K Ă H m and L Ă G , thereexists a sequence t f n u n P N Ă NN of finite GCNNs such that
[Equation]
The proof is given in Appendix A.2.
Here, f P C equi p H m ; C p G qq means that P 7 f is p G, T qequivariant.
Remarks
The topological group G covers a quite large class of groups.
For example, it contains all thefinite groups like Z n and S n , compact groups like SO p n q and U p n q , and non-compact groups like R nand E p n q as well.
We assumed this simply for considering the class C p G q of continuous functions.
We note that when G is a finite group, then C p G q is simply identified with a | G | -dimensionalreal/complex-vector space K | G | , because C p G q : “ t x : G Ñ K , continuous u “ K G .
The G -module H is an abstract vector space and thus needs not be a space of functions on G suchas C p G q and L 2 p G { H q , which is one of the major advantages; On the other hand, H m needs to befinite-dimensional, since the proof is based on the Fourier inversion formula on the finite-dimensional
Euclidean space R m .
We conjecture that the extension to an infinite-dimensional setting wouldbe a routine for some specialists in functional analysis.
As we will see in the examples, however,ordinary input vectors are always essentially finite-dimensional due to the limitations in computationalresources, and thus we consider finite-dimensional setting is reasonable.
The representation T needs not to be unitary, irreducible, nor square-integrable, since the proof isbased only a few basic properties of the group homomorphism.
Examples
5.1 Finite Periodic Convolution Layer
Example 1 (For 1-dimensional periodic signals) .
The periodic convolution corresponds to the casewhen K “ R , G “ Z m – r m s “ t 0 , 1 , . . .
, m ´ 1 u , H “ L 2 p G q – R m equipped with the innerproduct x x, y y : “ 1 m ř i
Pr m s x i
y i , and T i r x sp j q : “
x j ´ i thus
[Equation]
Therefore, the ridgelet transform and the reconstruction formula are given by
[Equation]
Example 2 (For 2-dimensional multi-channel periodic images) .
A 2 -dimensional c -channel image isidentified with a vector-valued function x : Z m ˆ Z n Ñ R c , thus H – R m ˆ n ˆ c .
Let x kij denote thep i, j q -th component in the k -th channel of x P H .
Let G “ Z m ˆ Z n , and put T p p,q q r x s kij : “ x ki ´ p,j ´ q .
Therefore, the ridgelet transform and the reconstruction formula are given by [Equation]
Example 3 (Difference operator (with cutoff function)) .
A difference operator on x : r m s Ñ 307 R is given by x “ ř i
Pr m s x i
δ
i ÞÑ [Equation]
p
[Equation]
ř
[Equation]
m s p x i ` 1 ´ x i q δ i , which is p G, T q -equivariant:f
[Equation]
q
[Equation]
´
[Equation]
i
[Equation]
x i ` 1 ´ k ´ x
[Equation]
´ k
[Equation]
f p x qp i ´ k q .
Since f p x qp 0 q “ x 1 ´ x 0 ,
[Equation]
We note since x ÞÑ x 1 ´ x 0 is not integrable in R m , we restrict f to a compact set K Ă R m , andimpose the indicator function 1 K as an auxiliary cutoff function.
5.2 (Deep Sets) Permutation Equivariant Maps on A Finite Set
Example 4.
Let H “ R m , G ď S m and T g r x s “ p x g ´ 1 p 1 q , x g ´ 1 p 2 q , . . .
, x g ´ 1 p m q q .
Thus x a, x y H “1 m ř i
Pr m s a
i x i , and p a ˚ T x qp g q “ 1 m ř p
Pr m s a p x g p p q .
So,
[Equation]
5.3 Continuous Periodic Convolution Layer
Example 5.
Let G “ T : “ R { 2 π Z – t
e iθ | θ P r´ π, π su be the 1-dimensional torus group, whichis one of the most basic continuous group.
As a consequence of the Fourier series expansion, L 2 p T qis spanned by t e inθ | n P N u .
Hence, we can take H to be an m -dimensional subspace H : “t ř | n |ă m x n e inθ | x ´ n “ x n P R u equipped with an inner product x x, y y H : “ ş
T x p θ q y p θ q d θ .
Wenote that the constraint x n “ x ´ n implies ř | n |ă m x n e inθ “ ř | n |ă m x n cos p nθ q and thus any signalx P H is a bandlimited real-valued continuous signal with each coefficient x n being the n -th frequencyspectrum.
Put T α r x sp θ q : “ x p θ ´ α q , then p a ˚ x qp α q “ ş
T a p θ q x p α ´ θ q d θ “ ř | n |ă m a n x n e inα “ 322 ř | n |ă m a n x n cos p nα q (by the convolution theorem and the constraint).
Therefore,
[Equation]
Example 6 (Differential operator (with convergence factor)) .
A differential operator dd θ is calculatedas x “ ř | n |ă m x n e inθ ÞÑ f p x q “ ř | n |ă m nx n e inθ .
Since x ÞÑ f p x qp 0 q “ ř | n |ă m nx n isnot integrable on R m , we impose a convergence factor ϕ t as follows.
f t p x qp θ q : “ f p x q ϕ t p x q “dd θ x p θ q ϕ t p x q “ ř | n |ă m nx n ϕ t p x q e inθ .
Here, p ϕ t q t ą 0 Ă S p H q is a family of convergence factorsthat satisfies (1) the first moment ş
H | x | H | ϕ t p x q| d x exists at every t , (2) ϕ t Ñ 1 in the weak senseas t Ñ 8 , and (3) (continuous and) p G, T q -equivariant.
For example, we can take a Gaussianϕ t p x q “ exp p´| x | 2 H { 4 t q .
Hence,
[Equation]
5.4 Euclidean group E p n q equivariant map
Example 7.
The Euclidean group E p n q is a semidirect product R n ¸ O p n q of the translational group 332 R n and the orthogonal group O p n q , which acts on R n as p U, s q ¨ t : “ Ut ` s for any t P R n andp U, s q P O p n q ˆ R n .
So, put H Ă L 2 p R n q and T p U,s q r x sp t q : “ x p U ´ 1 p t ´ s qq .
Then,
R r f ; ρ sp a, b q : “
[Equation]
We note that a more efficient representation for L 2 p R n q and/or a more general representation such as
L 2 p SO p 2 qq and L 2 p E p 3 qq , have been developed in the context of steerable CNNs [32, 61].
Related Work (on CNN Universality)
Non-Group CNN. Zhou [9, 10] have shown the cc -universality of specific deep ReLU (non-group) CNNs.
This seems to be the earliest result. In [10], he presented (Theorem 1) the cc -universality in C p R d ; R q in the limit of depth J Ñ 8 , and (Theorem 2) an approximation error rate with respect to J .
The CNN is carefully designed so that increasing the depth results in increasing the width. Theactivation function is limited to ReLU, which is essential for the proof. Since the proposed networkis different from standard CNNs (as well as group CNNs), the “universality of CNN” shown in thispaper is at a concept level. Specifically, our result does not directly cover his results.
Finite Group CNN.
Maron et al. [12], Sannai et al.
[62], Keriven and Peyré [63], Ravanbakhsh [64]and Petersen and Voigtlaender [13] presented the cc -(or L p -)universality results of finite -group CNNs. Maron et al. [12] is often cited as one of the earliest publications, where the input space is H “ R n k ˆ a
( a -channel k -th order n -dimensional tensors), the output space is H 1 “ R n l ˆ b ( b -channel l -th ordern -dimensional tensors), the group G is a subgroup of a symmetric group S n , and the group action isthe left-translation (i.e. the group representation T is the left-regular representation). In this setup,they presented the cc -universality of deep-ReLU-GCNNs in the space of continuous G -equivariantfunctions C equi p H ; H 1 q . The proofs are constructive but still existential because they are based onexistential theorems of either invariant polynomials (Stone-Weierstrass) or MLPs (Hahn-Banach). We note that a Stone-Weierstrass theorem states the existence of polynomials without construction,and Taylor/Fourier series expansion is not the one.
These finite-group cases are essentially coveredas Example 4 (Deep Sets). Again, the proof based on the ridgelet transform can further provide theconstruction of parameters, and thus be more constructive. Infinite Group CNN. Yarotsky [11] deals with infinite groups G and infinite-dimensional input/output spaces, and show a version of universalities of carefully designed deep GCNNs in thespace of G -equivariant
continuous functions C equi p L 2 p G ; R d q ; L 2 p G ; R d 1 qq where G is either translation group R d and 2-dimensional roto-translation group SE p 2 q . This study aims to investigate anew convolution-like structure that explicitly encodes the group-equivariance structure as explicitinductive bias, particularly when the input/output feature spaces are infinite-dimensional spaces (i.e.
L 2 p G ; R d q ) and the group G is an infinite group. (We note that eventually the input/output spaces arefinite-dimensional since the spaces are discretized as finite-dimensional Euclidean spaces). Thus, theproposed networks are not directly covered in our study. However, the cases of infinite-dimensionalgroups acting on a function space are covered in Examples 5 and 7.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A]
c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Proofs
Additional Notation In the proofs, we use two symbols p ¨ and ¨ 7 for the Fourier transforms in x P Hand b P K , respectively.
For example,
[Equation]
With a slight abuse of notation, when σ is a tempered distribution (i.e., σ P S 1 p K q ), then σ 7 isunderstood as the Fourier transform of distributions.
Namely, σ 7 is another tempered distributionsatisfying
[Equation]
for any test function ϕ P S p K q .
For any integer d ą 0 and vector v P R d , | v | denotes the Euclidean norm, and x v y : “ a 1 ` | v | 2 .
For any positive number t ą 0 , △ t { 2 and x △ y t denote fractional differential operators defined as
Fourier multipliers: for any ϕ P S 1 p R d q ,
[Equation]
[Equation]
[Equation]
In particular when t “ 2 , △ t { 2 coincides with the ordinary Laplacian on R d .
A.1 Theorem
Proof.
In the following, we fix a representation T : G Ñ GL p H q of a group G on a (potentiallyinfinite-dimensional)
Hilbert space H over a field K equipped with inner product x¨ , ¨y H , whichis a G -invariant vector space: T G r H s “ H , and a finite-dimensional closed subspace H m Ă H equipped with the Lebesgue measure λ .
Let H K
m be the orthogonal complement of H m in H , i.e.,x ‘ y P H m ‘ H K m “ H , and let P : H Ñ H m denote the orthogonal projection onto H m .
Letm : “ dim R H m denotes the real dimension of H m , and let k : “ dim R K denotes the real dimensionof K , which is either 1 or 2 .
Without loss of generality, we can assume (A’) that γ p a, ‚q ˚ σ P S p K q for a.e. a P H m , and(A”) that γ 7 σ 7 P L 1 p H m ˆ K q , which will be eventually justified because later in (34), we setγ 7 p a, ω q “ p f p ωa q ρ 7 p ω q .
Step 1 (Fourier expression).
Using an identity: For any function ϕ P S p K q and b P K , ϕ p b q “1 p 2 π q k ş
K
[Equation]
d ω , namely the Fourier inversion formula, we can turn S into a Fourier expression :
[Equation]
By the assumption (A’), the first equation holds at every point b “ p a ˚ T x qp g q , and by the assumption(A”), the Fourier expression is uniformly absolutely convergent:
[Equation]
[Equation]
for all p x, g q P H ˆ G .
Hence, we can change the order of integration freely.
Step 2 (Reconstruction).
By changing the variables as p a, ω q “ p ξ { ω, ω q with d a d ω “| ω | ´ m d ξ d ω , we have
[Equation]
Hence, using a given f satisfying the assumptions (A1) and (A2), and some function ρ P S p K q ,suppose that γ f,ρ satisfies the following separation-of-variables form:
[Equation]
Then,
[Equation]
[Equation]
[Equation]
38) where we put
[Equation]
Here, the equality ‹ “ holds at every continuous point x c of f for (A2a),
L 2 for (A2b), and in S
(A2c), respectively.
Step 3 (Ridgelet transform).
Since we put
[Equation]
it is calculated as
[Equation]
which is the definition of the ridgelet transform for GCNN.
A.2 Theorem
Proof.
Fix arbitrary compact sets K Ă H m and L Ă G , positive number ε ą 0 , and functionf P C equi p K ; C p G qq .
An n -term finite p G, T q -CNN is given by
[Equation]
with parameters p a i , b i , c i q P H m ˆ K ˆ C .
Observe that any finite p G, T q -CNN is p G, T q -equivariant,that is,
[Equation]
[Equation]
Put K : “ t T g ´ 1 r x s | x P K, g P L u , which is compact because T is continuous, and put f e p x q : “f p x qp e q , which is compactly supported, i.e., f e P C p K q Ă C p K q .
By Theorem 3, there exist a finitenumber N P N and an N -term C -valued fully-connected network F N
[Equation]
y H ´
b i qsatisfying } F N ´ f e } C p K q ă ε .
Put f N p x qp g q : “ F N p T g ´ 1 r x sq .
Then, it is a p G, T q -CNN because
[Equation]
and it is an ε -neighbour of f because
[Equation]
which concludes the assertion.
Theorem 3 ( cc -universality of scalar-valued finite fully-connected NNs on R m ) .
Suppose that1.
H “ H m “ R m ,2.
f P C p H ; C q (not vector-valued C p H ; C G q but scalar-valued), and3.
there exists k ě 0 and θ ą 0 such that ∆ kθ r σ s P L 8 p R q and Lipschitz continuous.
Then, the finite neural networks of the form f n p x q “ ř ni “ 1 c i σ p a i ¨ x ´ b
i q are cc -universal, thatis, for any compact set K Ă R m , positive number ε ą 0 , and continuous function f P C p K q , thereexists a finite network f n such that } f ´ f n } C p K q ă ε .
Proof.
Since ř ni “ 1 c i ∆ kθ r σ sp a i ¨ x ´ b i q is rewritten as another finite model ř n 1 i “ 1 c 1 i σ p a 1 i ¨ x ´ b 1 i q , it suffice to consider the case k “ 0 .
In the following we assume that σ p“ ∆ 0 θ r σ sq is bounded and
Step 1 ( f „ f c ).
By the density of C 8 c p R m q in C p K q with respect to the uniform norm, we cantake a compactly-supported smooth function f c P C 8 c p R m q satisfying } f
´ f c } C p K q ă ε { 3 .
Since f cis sufficiently smooth and integrable, there exists a compactly-supported smooth function ρ P C 8 c p R qsuch that
[Equation]
at every point x P R m .
For example, take a compactly-supported smooth function ρ 0 P C 8 c p K q , write k “ dim R K p“1 or 2 q , and put ρ p b q : “ △ m { 2 b
[Equation]
0 sp b
q “ p 2 π q ´ k ş
K | ω | m ρ 7 0 p ω q e ib ¨ ω d ω .
Then, pp σ, ρ qq “p 2 π q m ´ k ş K σ 7 p ω q ρ 7 p ω q| ω | ´ m d ω “ p 2 π q m ´ k ş K σ 7 p ω q ρ 7 0 p ω q d ω “ p 2 π q m ş K σ p b q ρ 0 p b q d b “x σ, ρ 0 y L 2 p K q , which is an ordinary functional inner product, and it is easy to find a ρ 0 satisfying x σ, ρ 0 y L 2 p K q ‰ 0 .
By normalizing ρ 1 : “ ρ {pp σ, ρ qq , we can find the ρ 1 .
We refer to Sonoda and
Murata [54] and Sonoda et al.
[44] for more details on the scalar product pp σ, ρ qq .
Step 2 ( R r f c ; ρ s ).
To show a discretization f n of the reconstruction formula converges to f c in
C p K q , it is convenient to regard the integration ş
R m ˆ R r¨ ¨ ¨ s d a d b in S as the Bochner integral, andthe integrand γ p a , b q σ p a ¨ x ´ b q as a vector-valued function from R m ˆ R to C p K q .
Since f c is C 8 -smooth, R r f c ; ρ sp a , b q is bounded and decays rapidly in a , and thus R r f c ; ρ s σ p a ¨x ´ b q is Bochner integrable, that is,
[Equation]
To see this, the decay property is estimated as follows.
For any positive numbers s, t ą 1 ,
[Equation]
(53) which asserts the integrability as below
[Equation]
[Equation]
Step 3 ( f c „ f V „ f n ).
Next, take a compact domain ( m ` 1 -dimensional hypercube) V : “tp a , b q P R m ˆ R | | a i | ď δ { 2 , | b | ď δ { 2 u , and put a band-limited function
[Equation]
(55) so that } f c ´ f V } C p K q ă ε { 3 (by letting δ sufficiently large).
Then, let V “ Ů i P I n V ni be adecomposition of the domain V into the union of disjoint family of | I n | “ n m ` 1 cubes withvolume vol p V n q “ p δ { n q m ` 1 and the longest diagonal d n “ ? m ` 1 δ { n .
From each cube, takea point p a ni , b ni q P V ni as a center of gravity, that is, so that c ni “
[Equation]
sp
[Equation]
[Equation]
q
[Equation]
a
[Equation]
R r f c ; ρ sp a ni , b ni q vol p V n q , and put w ni : “ R r f c ; ρ sp a ni , b ni q , then put a finite network as
[Equation]
Step 4 ( f V „ f n ).
We show f n Ñ f V in C p K q .
First, the integrands converge to the limit atalmost every p a , b q P V ni as
[Equation]
[Equation]
Besides, the integrands are uniformly bounded as
[Equation]
Therefore, by the dominated convergence theorem for the Bochner integral, we have
[Equation]
Hence by letting n sufficiently large, we have } f n ´ f V } C p K q ă ε { 3 .
To sum up, we have shonw the cc -universality:
[Equation]
Notes.
In the proof, we employed a naive discretization based on the regular grids in V .
However,since we know the closed-form expression of the ridgelet transform, we can discretize it moreeffectively.
For example, a better discretization scheme is investigated in the so-called Maurey-JonesBarron (MJB) theory and the dimension independent Barron’s bound ([see, e.g., 65]).