Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groups
Abstract
We show the universality of depth-2 group convolutional neural networks (GCNNs) in a unified and constructive manner based on the ridgelet theory.
Despite widespread use in applications, the approximation property of (G)CNNs has not been well investigated.
The universality of (G)CNNs has been shown since the late 2010s.
Yet, our understanding on how (G)CNNs represent functions is incomplete because the past universality theorems have been shown in a case-by-case manner by manually/carefully assigning the network parameters depending on the variety of convolution layers, and in an indirect manner by converting/modifying the (G)CNNs into other universal approximators such as invariant polynomials and fully-connected networks.
In this study, we formulate a versatile depth-2 continuous GCNN S r γ s as a nonlinear mapping between group representations, and directly obtain an analysis operator, called the ridgelet trasform , that maps a given function f to the network parameter γ so that S r γ s “ f .
The proposed GCNN covers typical GCNNs such as the cyclic convolution on multi-channel images, networks on permutation-invariant inputs (Deep Sets), and E p n q -equivariant networks.
The closed-form expression of the ridgelet transform can describe how the network parameters are organized to represent a function.
While it has been known only for fully-connected networks, this study is the first to obtain the ridgelet transform for GCNNs .
By discretizing the closed-form expression, we can systematically generate a constructive proof of the cc -universality of finite GCNNs .
In other words, our universality proofs are more unified and constructive than previous proofs.
Introduction
In the research field of geometric deep learning [1], group convolutional neural networks (GCNNs) have been developed to capture the inductive bias behind a variety of datasets such as sets and point clouds [2, 3], graphs [4, 5], manifolds, groups, and homogeneous spaces [6–8, 4].
Despite the rapid growth of diversity, the approximation property of CNNs is less investigated than that of fully-connected neural networks (FNNs).
To this date, several authors have shown the universality of (G)CNNs. That is, they can approximate some class of continuous maps with any precision [9–15].
These studies are still limited because the proofs are shown (1) in a case-by-case manner by manually assigning the parameters for a network to approximate a given function f , which means that once the network architecture is modified, then we need to reassign the parameters from scratch, and (2) in an indirect manner by converting/modifying the (G)CNNs into other universal approximators such as invariant polynomials and FNNs, which means that we know only indirectly about (G)CNNs.
The approximation property of FNNs has been investigated in the 1990s, with gradually increasing the resolution of proofs from abstract to concrete, starting from purely existential proofs based on the Hahn-Banach theorem [16] and the Stone-Weierstrass theorem [17], indirect proofs based on the
Fourier transform [18, 19], the Radon transform [20, 21], B -splines [22, 23], to more constructive proofs based on the integral representation [24], ridge functions [25], and the ridgelet transform [26–28].
For deep-ReLU-FNNs, further approximation properties have been investigated [29–31] in the 2010s.
In this context, (G)CNN studies are at the stage of case-by-case and indirect proofs.
(See § 6.1 for more details).
In this study, we show the universality of depth-2 GCNNs by devising a general notion of group convolution and developing the ridgelet transform for GCNNs —an analysis operator that maps a given function f to the weight parameter γ in a single hidden layer of a neural network.
Consequently, our universality proof is more unified and constructive because our GCNN covers a wide range of typical GCNNs, and the ridgelet transform can describe how to assign the network parameters.
In the following, we describe the formulation of GCNNs to overview our main contributions.
A Typical Convolution Layer for Images.
Given an m 1 ˆ m 2 -dimensional n in -channel input image x
P R m 1 ˆ m 2 ˆ n in , a typical convolution layer with w 1 ˆ w 2 -dimensional n in ˆ n out -channel filter a P R w 1 ˆ w 2 ˆ n in ˆ n out and n out -channel bias b P R n out followed by an elementwise activation function σ : R Ñ R and the aggregation with output coefficients c P R n out is given by
[Equation]
for each pixel at p i, j q P r m 1 ´ w 1 ` 1 s ˆ r m 2 ´
w
` 1 s .
For technical reasons, we assume that the output channels (indexed by ℓ P r n out s ) are aggregated soon after the activation function, which may be slightly different from an ordinary formulation of
CNNs, but we can understand this as a part of the subsequent layer.
In the standard formulation of GCNNs, a multi-channel image is understood as a vector-valued function on a group G or a homogeneous space G { H , such as a product group G “ Z m 1 ˆ Z m 2 of cyclic groups
Z m i : “ Z { m i Z p i “ 1 , 2 q .
(More geometrically, Cohen and Welling [32] phrased it as ‘a section of a fiber bundle’).
The convolution in the pixel directions p i, j q is reformulated as a group convolution with respect to the product group, and the inner product in the channel direction k is understood as the convolution with respect to the trivial action of G on a ‘fiber’ R n in .
The Integral Representation S r γ s of Group Convolution Layer.
Let G be an arbitrary group, σ : R Ñ R be an arbitrary nonlinear function, X be an arbitrary Hilbert space of feature vector x and filter a , and γ : X ˆ R Ñ C be an arbitrary function, called the parameter distribution .
We formulate a group convolution layer in an integral form, called the integral representation , as
[Equation]
This is an infinite-dimensional reparametrization of a depth-2 GCNN; namely, each function x ÞÑ σ pp a ˚ x qp g q ´ b q represents a single convolutional neuron, or a feature map of input x parametrized by p a, b q , the integration over p a, b q means that all the neurons are assigned, and a single function γ —the parameter distribution —parameterizes the assignment of each parameters p a, b q .
Hence, S r γ s can be understood as a continuous neural network .
We note that, however, if we put γ as a finite sum of Dirac’s measures such as γ n : “ ř nℓ “ 1 c ℓ δ p a ℓ ,b ℓ q , then the integral representation can also represent a finite model
[Equation]
In summary, S r γ s is a mathematical model of shallow neural networks with any width ranging from finite to continuous.
In particular, the sparsity/low-rankness of parameters are reflected as the localization/concentration of parameter distribution γ .
An advantage to use the integral representation is the linearization trick .
Whereas a finite network S r γ n s is nonlinear in the original parameters p a, b q , the integral representation S r γ s is linear in the parameter distribution γ .
It is first emerged in the 1990s to investigate the expressive power of infinitely-wide shallow FNNs [18–21, 24, 26–28]; and it is as well common in today’s deep learning theory, for example, to investigate the learning dynamics of SGD such as neural tangent kernel (NTK) [33–35], lazy learning [36], lottery tickets [37], mean field theory [38–42], and Langevin dynamics [43].
The Ridgelet Transform R r f ; ρ sp a, b q is a right inverse (or pseudo-inverse) operator of the integral representation operator S .
As an outcome of this study, we have obtained its closed-form expression:
[Equation]
where f : X Ñ C G is a target vector-valued nonlinear function to be approximated, called a feature map , e P G is the identity element, and ρ : R Ñ C is an auxiliary function, called the ridgelet function .
Provided that f is group equivariant , then under mild regularity assumptions, it satisfies the reconstruction formula
[Equation]
where pp¨ , ¨qq denote a scalar product of σ and ρ .
Therefore, as long as the product pp σ, ρ qq is neithernor 8 , we can normalize ρ to satisfy pp σ, ρ qq “ 1 so that S r R r f ; ρ ss “ f .
In other words, R and S are analysis and synthesis operators , and thus play the same roles as the
Fourier ( F ) and inverse Fourier ( F ´ 1 ) transforms, respectively.
Particularly, the reconstruction formula S r R r f ; ρ ss “ pp σ, ρ qq f corresponds to the Fourier inversion formula F ´ 1 r F r f ss “ f .
An advantage of the ridgelet transform is the closed-form expression .
Despite the common belief that neural network parameters are a blackbox, the closed-form expression can clearly describe how the network parameters are organized.
Previous studies on the CNN universality have also provided several construction algorithms of parameters, but these are only particular solutions for a CNN to represent a target function f , and not necessary related to, for example, deep learning solutions.
For FNNs , on the other hand, Sonoda et al.
[44] have shown that any parameter distribution γ satisfying S r γ s “ f can always be represented as (not always single but) a linear combination of ridgelet transforms, and they [45] have shown that finite networks trained by regularized empirical risk minimization (RERM) converges to a certain unique ridgelet transform.
(We note that NTK and the Gibbs distribution can also describe the parameter distribution, but NTK is limited to the kernel regime, and the Gibbs distribution is given only implicitly.)
As an application, Savarese et al.
[46] and their followers [47–49] have established the representer theorems for ReLU-FNNs by using the ridgelet transform.
Although the parallel results for CNNs have not yet been published, we anticipate that the ridgelet transform could facilitate our understanding of deep learning solutions.
Challenges and Contributions.
The closed-form expression of the ridgelet transform has been known only for FNNs, which was discovered in the 1990s independently by Murata [26], Candès [27] and Rubin [28].
(We refer to [50–52] for ridgelet analysis in the 2000s, and [53–55] for more recent results.)
One of the difficulties to obtain the ridgelet transform for CNNs is that there is no unique way to formulate an “integral representation of CNNs”.
We note that some authors claim the “equivalence of CNNs and FNNs” (see e.g. [13]), but it is somewhat misleading because such an equivalence holds only when both CNNs and FNNs are very carefully designed.
While FNNs are defined on the Euclidean space R m , GCNNs are defined on a more abstract space X .
For example, since the convolution on the Euclidean space can be written using Töplitz matrices, one could consider a formulation such as
[Equation]
where the parameter A is an k ˆ m -matrix.
However, this only leads to another ridgelet transform that covers less symmetries G .
In fact, it is a version of the so-called k -plane ridgelet transform developed in the 2000s [50].
To circumvent this difficulty, we formulate GCNNs as general as possible by dealing with the feature space X , group G , and representation T in a coordinate-free manner.
Eventually, we have shown the reconstruction formula for a wide range of GCNNs (as displayed in § 5), with a relatively simple proof.
This study is the first to obtain the ridgelet transform for a general class of GCNNs.
As an application, we show the cc -universality of GCNNs for a general class of group equivalent continuous vector-valued functions in a unified and constructive manner.
Notation and Basic Terminologies
Notation.
For any integer n ą 0 , r n s denotes the set t 1 , . . . , n u .
For any sets G and K , K G denotes the collection t G Ñ K u of all mappings from G to K .
For any topological space X , C p X q
and
C c p X q denote the collections of all continuous functions on X , and continuous functions on X with compact support, respectively.
We note that when X is compact, then C p X q “ C c p X q .
For any measure space X and number p P r 1 , 8s , L p p
X q denotes the space of p -integrable functions on X .
2.1 Fourier Analysis on R d
We refer to [56, 57, 53] for more details on Fourier transform and tempered distributions ( S 1 ).
Schwartz Distributions.
For any integer d ą 0 , S p R d q and S 1 p R d q denote the classes of Schwartz test functions (or rapidly decreasing functions) and tempered distributions on R d , respectively.
Namely, S 1 is the topological dual of S .
In this study, S 1 p R q and S p R q are assigned as classes of activation and ridgelet functions, respectively.
We note that S 1 p R q includes truncated power functions σ p b q “ b k ` “ max t b, 0 u k such as step function for k “ 0 and ReLU for k “ 1 .
Fourier Transform.
The Fourier transform on the Euclidean space R d and its inversion formula has been defined on (at least) three different function classes:
L 1 p R d q , L 2 p R d q and S 1 p R d q .
When f P L 1 p R d q and p f P L 1 p R d q , the inversion formula holds “at every continuous point x of f ”, which is a pointwise equation.
When f P L 2 p R d q , the inversion formula holds “in L 2 ”, which is not a pointwise equation because the equation “ f “ g in L 2 ” is defined as “ f p x q “ g p x q a.e.”.
Similarly, when f P S 1 p R m q , the inversion formula holds “in S 1 ”.
We use the third definition for computing the
Fourier transform of activation functions σ P S 1 p R q such as ReLU and tanh .
2.2 Group Representation
Let G be a group, let X be a vector space over a field K , and let GL p X
q be the general linear group on X .
A group representation T of the group G on the vector space
X is a group homomorphism from G to GL p X q , that is, a map T : G Ñ GL p X q ; g ÞÑ T g satisfying T gh “ T g T h for all g, h P G .
When G is a topological group, we further assume that the action G ˆ X Ñ X ; p g, x q ÞÑ T g r x s be continuous.
Here, X is called the representation space . We refer to [58] for more details on group representation.
Regular Representation.
Let X be the vector space of all functions on G , i.e., X “ K G .
The (left) regular representation L is a group representation defined on X as
[Equation]
In particular, when G is a locally compact Haussdorf (LCH) group, then it has a (left) invariant measure µ , and we can define the collection L 2 p G q of all square integrable functions on G with respect to the canonical inner product x x, y y L 2 p G q : “ ş
G x p g q y p g q d µ p g q for any measurable functions x, y : G Ñ C .
It is known that the regular representation on X “ L 2 p G q is a unitary representation.
Dual Representation.
For any group representation T : G Ñ GL p X q , the dual representation
T ˚ : G Ñ GL p X 1 q is a group representation defined on the dual vector space X 1 as the transpose of
T g ´ 1 , that is, T ˚ g “ T J g
´ 1 .
When X is a Hilbert space with inner product x¨ , ¨y X , then it satisfies the following relation:
[Equation]
Matrix Element.
For any group representation T : G Ñ GL p X q , the matrix element (or the matrix coefficient ) of T is a bilinear functional f a,x on G defined by
[Equation]
where x is a vector in X and a P X 1 is a continuous linear functional on X .
When X is a Hilbert space, then (identifying X 1 with X ) it can be written as
[Equation]
In the next section, we use this quantity as the generalized form of the group convolution .
2.3 Universality
The notion of universality in machine learning can be rephrased as the density in mathematics, and thus it has several definitions.
(See, e.g., [59, 60]).
In this study, we show the so-called cc -univesality , one of the standard universalities in the machine learning theory.
cc -Universality.
Let X be a topological space, and let NN be a collection of functions (e.g., neural networks) on X .
The cc -univesality of NN is defined as the density of NN in C
p X q endowed with the topology of compact convergence , that is, for any compact subset K Ă X , continuous function f P C p K q , and all ε ą 0 , there exists a function g P NN such that
[Equation]
) where g | K denotes the restriction of g to K .
Functions on Abstract Hilbert Space X
We introduce an extended group convolution on X , a uniform norm and the group-equivariance for functions on X , an induced measure and an induced Fourier transform on X , and a projection to X m .
3.1 p G, T q -Convolution ˚ T : X ˆ X Ñ C G
Definition 1.
Let G be a group, let X be a Hilbert space with inner product x¨ , ¨y X over a field K , and let T : G Ñ GL p X q be a representation of G on X .
For any a, x P X and g P G , we define the p G, T q -convolution as
[Equation]
We remark (1) that this is simply a paraphrase of the matrix element of a group representation (see the previous section), and (2) that this is not necessarily a binary operation because X ‰ C G in general.
Nevertheless, we call it a convolution simply because it covers a wide range of ‘group convolutions’ in today’s GCNN literature.
Example 1.
An orthodox group convolution is reproduced when T is the regular representation (of a
LCH group G ) on X “ L 2 p G q , i.e., T ˚ g r a sp h q “ a p g ´ 1 h q .
In fact, x x, T ˚ g r a sy L
[Equation]
where r a p g q : “ a p g ´ 1 q is an involution.
Example 2.
The cyclic convolution for an n -channel image x “ p x kij q P R m 1 ˆ m 2 ˆ n is understood as the case when G “ Z m 1 ˆ Z m 2 , X “ R m 1 ˆ m 2 ˆ n , and T ˚p p,q q r a sp i, j, k q “ a ki ´ p,j ´ q , then
[Equation]
While the post-activation feature σ pp a ˚ x qp g q ´ b q is a function on G , the input feature x can be an arbitrary abstract vector, which is more general than typical GCNN formulations where feature x is supposed to be a vector-valued function on G or G { H .
This is an advantage for a more geometric understanding of CNNs, since the theory becomes free from the specification of x .
3.2 Continuous p G, T q -Equivariant Vector-Valued Function f : X Ñ C p G q
Definition 2.
We say a vector-valued function f : X Ñ C G is p G, T q -equivariant when
[Equation]
Here, we restrict the definition for a special case of the regular representation L .
This is simply due to the fact that our GCNN satisfies this case.
Definition 3.
Let G be a topological group.
For any vector-valued function f : X Ñ C G , put
[Equation]
By C equi p X ; C p G qq , we denote the normed vector space of all continuous p G, T q -equivariant C p G qvalued functions on X equipped with the uniform norm } ¨ } C p X ; C p G qq .
We note that the topology of uniform norm } ¨ } C p X ; C p G qq is stronger than the topology of compact convergence, which is employed in the cc -universality argument.
In fact, C equi p X ; C p G qq need not be complete (or Banach) to show the cc -universality.
3.3 Induced Lebesgue Measure λ and Induced Fourier Transform p ¨ on Subspace X m
Let X m denote an m -dimensional subspace of X , and let t e i u i Pr
m s be an orthonormal basis of X m .
Induced Lebesgue Measure on X m .
We induce the Lebesgue measure λ on X m by pushing forward the Lebesgue measure d x on R m via an isometric linear embedding ϕ : R m Ñ X m .
For example, take a linear embedding ϕ
p
x
q : “ ř i
Pr m s x i e i .
Then, it preserves the length, and we can induce the Lebesgue measure λ on X m as the push forward measure λ “ ϕ 7 d x so that the volume of a hypercube Q “ t ř i Pr m s c
i e i |
c i P r a i , b i su in X m is calculated as λ p Q q “ ś i Pr m s |
b i ´ a i | , and the integration of a measurable function f : X m Ñ C over a measurable set E Ă X m is calculated as ż
[Equation]
[Equation]
As far as there is no risk of confusion, we denote d x instead of d λ p x q .
Induced Fourier Transform on X m .
Using λ , we induce the Fourier transform on R m as below:
For any function f : X m Ñ C ,
[Equation]
Here, the equality ‹ “ holds in at least three different senses (see the comments in § 2.1).
We remark (1) that once the subspace X m is fixed, the induced Fourier transform is unique up to the orthogonal transformation of the basis t e i u i Pr m s , and (2) that the induced Fourier transform “on
X ” should not be confused with the Fourier transform “on group G ”.
Especially, this cannot map a convolution x ˚ T a , an element in C G , to a point product such as “ p x ¨ p a ”.
3.4 Projection P : X Ñ X m and Extension Operator P ˚
In order to induce the Lebesgue measure λ , we assume that the dimension of X m to be finite.
As a side effect of this assumption, the image T G r X m s : “ t T g r x s | g P G, x P X m u can extend toward the outside of X m ; that is, X m is not necessarily G -invariant ( T G r X m s Ă X m ).
To avoid an “undefined error” such as to input x outside of X m for a function f defined only on X m , we introduce projection
P and extension P ˚ as below.
When dim X ă 8 , we can omit P by putting X m “ X (so P “ Id ), because by the definition of the group representation, always T G r X s “ X .
Let X K m denote the orthogonal complement of X m in X .
Let P : X Ñ X m denote the orthogonal projection onto X m .
For any function f : X m Ñ C G , put
[Equation]
This extends f (on a subspace X m ) to the entire space X as a constant function on X K m ; that is,
P ˚ f p x ‘ y q “ f p x ‘ 0 q for each x ‘ y P X m ‘ X K m .
Main Results
We introduce the p G, T q -convolutional neural networks and the corresponding ridgelet transform, and present the reconstruction formula for continuous GCNNs and the cc -universality for finite GCNNs.
Throughout this section, we fix a representation T : G Ñ GL p X q of a group G on a (potentially infinite-dimensional)
Hilbert space X over a field K endowed with an inner product x¨ , ¨y X , and fix an m -dimensional closed subspace X m of X equipped with an induced Lebesgue measure λ .
Let k : “ dim R K denote the real dimension of K , that is, k “ 1 for K “ R and k “ 2 for K “ C .
Let e denote the identity element of G .
4.1 Integral Representation of p G, T q -Convolutional Neural Network
Definition 4.
For any functions γ : X m ˆ K Ñ C and σ : K Ñ C , we define the integral representation of p G, T q -convolutional neural network as a vector-valued function X Ñ C G ,
[Equation]
Here, we call γ a parameter distribution, and σ an activation function.
If there is no risk of confusion, we abbreviate d λ p a q as d a .
It is easy to see that a p G, T q -CNN is p G, T q -equivariant.
In fact, for every g, h P G ,
S r γ sp T g
[Equation]
x sqp
[Equation]
q
[Equation]
In addition, at the identity element g “ e , it is reduced to a FNN:
[Equation]
and it satisfies a projection property :
[Equation]
4.2 Ridgelet Transform and Scalar Product of Activation Function
Definition 5.
For any functions f : X m Ñ C G and ρ : K Ñ C , we define the ridgelet transform as
[Equation]
Here e denotes the identity element of G .
Definition 6.
For any tempered distribution σ P S 1 p K q and function ρ P S p K q , put a scalar product as
[Equation]
Here, ¨ 7 denotes the Fourier transform on K , which is identified with the Fourier transform on R k with k “ dim R K .
We note that σ 7 is defined in the sense of tempered distributions.
The derivations of the ridgelet transform and the scalar product are clarified in the proof of the reconstruction formula.
Some readers may notice that the ridgelet transform for GCNN is formally the same as the one for FNNs, and may wonder why inner product x x, a y X instead of group convolution p a ˚ T x qp g q .
Indeed, this is a consequence of two facts (1) that a group convolution at the identity e is reduced to an inner product: p a ˚ T x
qp e q “ x x, a y X , and (2) that when f is p G, T q -equivariant, then the value f p x qp g q at each g P G is determined by translating the value f p x qp e q at the identity.
4.3 Reconstruction Formula, or the Universality of Continuous GCNNs
We state the first half of our main results.
For f : X m Ñ C G , we write f e p x q : “ f p x qp e q for short.
Theorem 1 (Main Theorem 1/2) .
Given a function f : X m Ñ C G , assume (A1) that P ˚ f : X Ñ C G is p G, T q -equivariant, i.e.,
[Equation]
for every z P
[Equation]
and g
[Equation]
h P G
[Equation]
and (A2) that f satisfies at least one of the following conditions: (A2a) both f e and p f e are absoluteintegrable, i.e., f e , p f e P L 1 p
X m q , (A2b) f e is square-integrable, i.e., f e P L 2 p
X m q , or (A2c)
f e is a tempered distribution, i.e., f e P S 1 p X m q .
Then, the following reconstruction formula holds:
[Equation]
where the equality ‹ “ holds at every continuous
The proof is given in Appendix A.1.
4.4 cc -Universality of Finite GCNNs
Finally, we state the second half of our main results.
Let NN be the collection of finite GCNNs, that is,
NN
[Equation]
Since the reconstruction formula S r γ f s “ f with γ f “ R r f ; ρ s holds for an arbitrary function f , we can construct a sequence t f n u n P N of finite p G, T q -CNNs that converges to an arbitrary target function f , namely f n Ñ f as n Ñ
[Equation]
28) by discretizing the continuous network S r γ f s and distribution γ f into finite sums
[Equation]
in a ‘nice’ manner so that γ n Ñ γ f “ R r f ; ρ s as n Ñ 8 .
This is the primitive idea behind the constructive proof of the following cc -universality of finite p G, T q -CNNs based on ridgelet analysis.
To state a regularity assumption on the activation function σ , we introduce the forward difference operator ∆ nθ with difference θ ą 0 , defined as
[Equation]
Theorem 2 (Main Theorem 2/2) .
For an activation function σ P S 1 p K q , assume (A3) that there exist n P N and θ ą 0 such that ∆ nθ r σ s is bounded and Lipschitz continuous.
Then, NN is cc -universal; that is, for any continuous p G, T q -equivariant C p G q -valued function f P C equi p X m ; C p G qq , and for any compact sets K Ă X m and L Ă G , there exists a sequence t f n u n P N Ă NN of finite GCNNs satisfying
[Equation]
The proof is given in Appendix A.2.
Here, f P C equi p X m ; C p G qq means that P ˚ f is p G, T qequivariant.
Examples
We display the ridgelet transforms and reconstruction formulas for a few typical GCNNs.
Besides, we calculated in Examples 5 and 8 the ridgelet transforms of a differential filter, which is often reported to be acquired as a feature map in the first layer of deep CNNs for image recognition [61, 62].
5.1 Finite Periodic Convolution Layer
Example 3 (For 1-dimensional periodic signals) .
The periodic convolution corresponds to the case when K “ R , G “ Z m – r m s “ t 0 , 1 , . . .
, m ´ 1 u , X
“ L 2 p G q – R m equipped with the inner product x x, y y : “ 1 m ř i
Pr m s x i
y i , and T i r x sp j q : “
x j ´ i thus p a ˚ T x qp i q “ 1 m ř j Pr m s a j x
i ` j .
Therefore, the ridgelet transform and the reconstruction formula are given by
[Equation]
[Equation]
Example 4 (For 2-dimensional multi-channel periodic images) .
A 2 -dimensional n -channel image is identified with a vector-valued function x : Z 2 m Ñ R n , thus X – R m 2 ˆ n .
Let x kij denote the p i, j q -th component in the k -th channel of x P X .
Let G “ Z 2 m
[Equation]
and put T p p,q q r x s kij : “ x ki ´ p,j ´ q .
Therefore, the ridgelet transform and the reconstruction formula are given by
[Equation]
Example 5 (Difference operator (with cutoff function)) .
A difference operator on x : r m s Ñ R is given by x “ ř i
Pr m s x i
δ i ÞÑ f p x q “ ř i Pr m s p x
i ` 1 ´ x i q δ i , which is p G, T q -equivariant: f p T ˚
k r x
[Equation]
q “ T ˚ k r
[Equation]
s
[Equation]
T ˚ k r x s i “ x i
[Equation] [Equation]
k ´ x i ´ k “ f p x qp
i ´ k q .
Since f p x qp 0 q “ x 1 ´ x 0 ,
[Equation]
ż
[Equation]
´ 1 m ř
i
[Equation]
m
[Equation]
a i x i ´ b ¯ d x .
We note since x ÞÑ x 1 ´ x 0 is not integrable in R m , we restrict f to a compact set K Ă R m , and impose the indicator function 1 K as an auxiliary cutoff function.
5.2 (Deep Sets) Permutation Equivariant Maps on A Finite Set
Example 6.
Let X “ R m , G ď S m and T g r x s “ p x g ´ 1 p 1 q , x g ´ 1 p 2 q , . . .
, x g ´ 1 p m q q .
Thus x a, x y X “m ř i
Pr m s a i x i , and p a ˚ T x qp g q “ 1 m ř
[Equation]
s a p x g p p q .
So,
[Equation]
b
[Equation] [Equation]
5.3 Continuous Periodic Convolution
Example 7.
Let G “ T : “ R { 2 π Z – t
e iθ | θ P r´ π, π su be the 1-dimensional torus group, which is one of the most basic continuous group.
As a consequence of the Fourier series expansion, L 2 p T q is spanned by t e inθ | n P N u .
Hence, we can take X to be an m -dimensional subspace X : “ t ř | n |ă m x n e inθ | x ´ n “ x n P R u equipped with an inner product x x, y y X : “ ş
T x p θ q y p θ q d θ .
We note that the constraint x n “ x ´ n implies
[Equation]
q and thus any signal x P X is a bandlimited real-valued continuous signal with each coefficient x n being the n -th frequency spectrum.
Put T α r x sp θ q : “ x p θ ´ α q , then p a ˚ x qp α q “ ş
[Equation]
ř | n |ă m a n x n cos p nα q (by the convolution theorem and the constraint).
Therefore,
[Equation]
Example 8 (Differential operator (with convergence factor)) .
A differential operator dd θ is calculated as x “ ř | n |ă m x n e inθ ÞÑ f p x q “ ř | n |ă m nx n e inθ .
Since x ÞÑ f p x qp 0 q “ ř | n |ă m nx n is not integrable on R m , we impose a convergence factor ϕ t as follows.
f t p x qp θ q : “ f p x q ϕ t p x q “ dd θ x p θ q ϕ t p x q “ ř | n |ă m nx n ϕ t p x q e inθ .
Here, p ϕ t q t ą 0 Ă S p X q is a family of convergence factors that satisfies (1) the first moment ş
X | x | X | ϕ t p x q| d x exists at every t , (2) ϕ t Ñ 1 in the weak sense as t Ñ 8 , and (3) (continuous and) p G, T q -equivariant.
For example, we can take a Gaussian ϕ t p x q “ exp p´| x | 2 X { 4 t q .
Hence,
[Equation]
5.4 Euclidean group E p n q equivariant map
Example 9.
The Euclidean group E p n q is a semidirect product R n ¸ O p n q of the translational group R n and the orthogonal group O p n q , which acts on R n as p U, s q ¨ t : “ Ut ` s for any t P R n and p U, s q P O p n q ˆ R n .
So, put X Ă L 2 p R n q and T p U,s q r x sp t q : “ x p U ´ 1 p t ´ s qq .
Then,
R r f ; ρ sp a, b q : “ ż
X f p x qp I, 0 q ρ ˆ ż
R n a p t q x p t q d t ´ b ˙ d x
S r R r f ; ρ ssp x qp U, t q “ ż
[Equation]
R r f ; ρ sp a, b q σ ˆż
[Equation]
d a d b “ f p x qp U, t q
We note that a more memory efficient representation for L 2 p R n q and/or a more general representation such as L 2 p SO p 2 qq and L 2 p E p 3 qq , have been developed in the context of steerable CNNs [32, 63].
Discussion 6.1 Related Works on (G)CNN Universality
Non-Group CNN.
Zhou [9, 10] is the earliest to show the cc -universality of deep ReLU (nongroup) CNNs.
In [10], he presented (Theorem 1) the cc -universality in C p R d ; R q in the limit of depth J Ñ 8 , and (Theorem 2) an approximation error rate with respect to J .
The CNN is carefully designed so that increasing depth also increases width, which is not covered in our GCNN.
Finite Group CNN.
Maron et al.
[12], Sannai et al.
[64], Keriven and Peyré [65], Ravanbakhsh [66] and Petersen and Voigtlaender [13] presented the cc -(or L p -)universality results of finite -group
CNNs.
Maron et al.
[12] is often cited as one of the earliest publications, where the input space is X “ R n k
ˆ a ( a -channel k -th order n -dimensional tensors), the output space is X 1 “ R n l ˆ b ( b -channel l -th order n -dimensional tensors), the group G is a subgroup of a symmetric group S n , and the group action (or representation) T is the left-translation (or left-regular representation).
In this setup, they presented the cc -universality of deep-ReLU-GCNNs in the space of continuous
G -equivariant functions C equi p X ; X 1 q .
The proofs are indirect because they are based on invariant polynomials or MLPs .
The finite group cases are essentially covered as Example 6 (Deep Sets).
Lie Group CNN.
Yarotsky [11] carefully designed deep GCNNs with Lie groups acting on infinitedimensional input/output spaces, and show a version of universality in the space of continuous
G -equivariant functions C equi p
L 2 p G ; R d q ; L 2 p G ; R d 1 qq .
To be precise, G is either a compact group, translation group R d , or 2-dimensional roto-translation group SE p 2 q , and the input/output spaces X and X 1 are square-integrable functions on G .
The proposed networks are not covered in our GCNNs, but several infinite group cases are covered in Examples 7 and 9.
Remarkably, Kumagai et al.
[14] introduced an integral representation that covers LCH groups, and showed the universality.
The proposed integral representation is based on the Haar measure, thus slightly different from ours.
The proofs are indirect because the network is converted to an MLP.
6.2 Review of Assumptions
Group G .
We only assume G to be a topological group, to deal with continuous functions on G .
Thus, a quite large class of groups are covered, for example, all the finite groups such as Z n and S n , compact groups such as SO p n q
and U p n q , and non-compact groups such as R n and E p n q as well.
Representation Space X .
Unlike previous studies, it does not need to be a function space such as C p G q and L 2 p G { H q , but it only needs to be an abstract Hilbert space, which is one of the major advantages for geometric understanding of GCNNs.
On the other hand, we introduce an auxiliary finite-dimensional subspace X m (and projection P ), to use the Fourier inversion formula on the finite-dimensional Euclidean space R m in the proof.
We conjecture that the extension to an infinite-dimensional setting would be a routine for some specialists in functional analysis.
Group Representation T .
It does not need to be unitary, irreducible, nor square-integrable, since the proof is based only on a few basic properties of the linear group representation.
Network Architecture.
The ridgelet theory supports a wide class of activation functions , namely, the tempered distributions ( S 1 ).
The extension to deep GCNNs remains an important open question.
Acknowledgments and Disclosure of Funding
The authors are grateful to anonymous reviewers for their valuable comments.
This work was supported by JST CREST JPMJCR2015 and JPMJCR1913, JST PRESTO JPMJPR2125, and JST
ACT-X JPMJAX2004.
A Proofs
Additional Notation In the proofs, we use two symbols p ¨ and ¨ 7 for the Fourier transforms in x P X and b P K , respectively.
For example,
[Equation]
With a slight abuse of notation, when σ is a tempered distribution (i.e., σ P S 1 p K q ), then σ 7 is understood as the Fourier transform of distributions.
Namely, σ 7 is another tempered distribution satisfying
[Equation]
for any test function ϕ P
[Equation]
p K q .
For any integer d ą 0 and vector v P R d , | v | denotes the Euclidean norm, and x v y : “ a
[Equation]
| 2 .
For any positive number t ą 0 , △ t { 2 and x △ y t denote fractional differential operators defined as
Fourier multipliers: for any ϕ P S 1 p R d q ,
[Equation]
In particular when t “ 2 , △ t { 2 coincides with the ordinary Laplacian on R d .
A.1 Theorem
Proof.
In the following, we fix a representation T : G Ñ GL p X q of a group G on a (potentially infinite-dimensional)
Hilbert space X over a field K equipped with inner product x¨ , ¨y X , which is a G -invariant vector space: T G r
[Equation]
“ X , and a finite-dimensional closed subspace X m Ă X equipped with the Lebesgue measure λ .
Let X K m be the orthogonal complement of X m in X , i.e., x ‘ y P X m ‘ X K m “ X , and let P : X Ñ X m denote the orthogonal projection onto X m .
Let m : “ dim R X m denotes the real dimension of X m , and let k : “ dim R K denotes the real dimension of K , which is either 1 or 2 .
Without loss of generality, we can assume (A’) that γ p a, ‚q ˚ σ P S p K q for a.e. a P X m , and (A”) that γ 7 σ 7 P L 1 p X m ˆ K q , which will be eventually justified because later in (38), we set γ 7 p a, ω q “ p f p ωa q ρ 7 p ω q .
Step 1 (Fourier expression).
Using an identity: For any function ϕ P S p K q and b P K , ϕ p b q “p 2 π q k ş
K ϕ 7 p ω q e ibω d ω , namely the Fourier inversion formula, we can turn S into a Fourier expression :
[Equation]
By the assumption (A’), the first equation holds at every point b “ p a ˚ T x qp g q , and by the assumption (A”), the Fourier expression is uniformly absolutely convergent:
[Equation]
(36) for all p x, g q P X ˆ G .
Hence, we can change the order of integration freely.
Step 2 (Reconstruction).
By changing the variables as p a, ω q “ p ξ { ω, ω q with d a d ω “ | ω | ´ m d ξ d ω , we have
[Equation]
Hence, using a given f satisfying the assumptions (A1) and (A2), and some function ρ P S p K q , suppose that γ f,ρ satisfies the following separation-of-variables form:
[Equation]
Then,
[Equation]
) where we put pp σ, ρ qq :
[Equation]
Here, the equality ‹ “ holds at every continuous point x c of f for (A2a), in L 2 for (A2b), and in S 1 for (A2c), respectively.
Step 3 (Ridgelet transform).
Since we put
[Equation]
it is calculated as
[Equation]
which is the definition of the ridgelet transform for GCNN
[Equation]
A.2 Theorem
Proof.
Fix arbitrary compact sets K Ă X m and L Ă G , positive number ε ą 0 , and function f P C equi p K ; C p G qq .
An n -term finite p G, T q -CNN is given by
[Equation]
with parameters p a i , b i , c i q P X m ˆ K ˆ C .
Observe that any finite p G, T q -CNN is p G, T q -equivariant, that is,
[Equation]
Put K : “ t T g ´ 1 r x s | x P K, g P L u , which is compact because T is continuous, and put f e p x q : “ f p x qp e q , which is compactly supported, i.e., f e P C p K q Ă C p K q .
By Theorem 3, there exist a finite number N P N and an N -term C -valued fully-connected network F N p x q “ ř Ni “ 1 c i σ px a i , x y X ´ b i q satisfying } F N ´ f e } C p K q ă ε .
Put f N p x qp g q : “ F N p T g
[Equation]
1 r x sq .
Then, it is a p G, T q -CNN because
[Equation]
and it is an ε -neighbour of f because
[Equation]
which concludes the assertion.
Theorem 3 ( cc -universality of scalar-valued finite fully-connected NNs on R m ) .
Suppose that 1.
[Equation]
f P C p X ; C q (not vector-valued C p X ; C G q but scalar-valued), and 3.
there exists k ě 0 and θ ą 0 such that ∆ kθ r σ s P L 8 p R q and Lipschitz continuous.
Then, the finite neural networks of the form f n p x q “ ř ni “ 1 c i σ p a i ¨ x ´ b
i q are cc -universal, that is, for any compact set K Ă R m , positive number ε ą 0 , and continuous function f P C p K q , there exists a finite network f n such that } f ´ f n } C p K q ă ε .
Proof.
Since ř ni “ 1 c i ∆ kθ r σ sp a i ¨ x ´ b i q is rewritten as another finite model ř n 1 i “ 1 c 1 i σ p a 1 i ¨ x ´ b 1 i q , it suffice to consider the case k “ 0 .
In the following, we assume that σ p“ ∆ 0 θ r σ sq is bounded and
Lipschitz continuous.
Step 1 ( f „ f c ).
By the density of C 8 c p R m q in C p K q with respect to the uniform norm, we can take a compactly-supported smooth function f c P C 8 c p R m q satisfying } f
´ f c } C p K q ă ε { 3 .
Since f c is sufficiently smooth and integrable, there exists a compactly-supported smooth function ρ P C 8 c p R q such that
[Equation]
For example, take a compactly-supported smooth function ρ 0 P C 8 c p K q , write k “ dim R K p“ 1 or 2 q , and put ρ p b q : “ △ m { 2 b
[Equation]
|
[Equation]
7 0 p ω q
[Equation]
ib ¨ ω d ω .
Then, pp σ, ρ qq “ p 2 π q m ´ k ş K σ 7 p ω q ρ 7 p ω q| ω | ´ m d ω “ p 2 π q m ´ k ş K σ 7 p ω q ρ 7 0 p ω q d ω “ p 2 π q m ş K σ p b q ρ 0 p b q d b “ x σ, ρ 0 y L 2 p K q , which is an ordinary functional inner product, and it is easy to find a ρ 0 satisfying x σ, ρ 0 y L 2 p K q ‰ 0 .
By normalizing ρ 1 : “ ρ {pp σ, ρ qq , we can find the ρ 1 .
We refer to Sonoda and
Murata [53] and Sonoda et al.
[44] for more details on the scalar product pp σ, ρ qq .
Step 2 ( R r f c ; ρ s ).
To show a discretization f n of the reconstruction formula converges to f c in
C p K q , it is convenient to regard the integration ş
R m ˆ R r¨ ¨ ¨ s d a d b in S as the Bochner integral, and the integrand γ p a , b q σ p a ¨ x ´ b q as a vector-valued function from R m ˆ R to C p K q .
Since f c is C 8 -smooth, R r f c ; ρ sp a , b q is bounded and decays rapidly in a , and thus R r f c ; ρ s σ p a ¨ x ´ b q is Bochner integrable, that is, ż
[Equation]
To see this, the decay property is estimated as follows.
For any positive numbers s, t ą 1 ,
[Equation]
which asserts the integrability as below ż
[Equation]
Step 3 ( f c „ f Q „ f n ).
Next, take a compact domain ( m ` 1 -dimensional hypercube) Q : “ tp a , b q P R m ˆ R | | a i | ď δ { 2 , | b | ď δ { 2 u , and put a band-limited function
[Equation]
so that } f c ´ f Q } C p K q ă ε { 3 (by letting δ sufficiently large).
Then, let Q “ Ů i P I n Q ni be a decomposition of the domain Q into the union of disjoint family of | I n | “ n m ` 1 cubes with volume vol p Q n q “ p δ { n q m ` 1 and the longest diagonal d n “ ? m ` 1 δ { n .
From each cube, take a point p a ni , b ni q P Q ni as a center of gravity, that is, so that c ni “
[Equation]
d b “
R r f c ; ρ sp a ni , b ni q vol p Q n q , and put w ni : “ R r f c ; ρ sp a ni , b ni q , then put a finite network as f n p x q : “
[Equation] [Equation]
Step 4 ( f Q „ f n ).
We show f n Ñ f Q in C p K q .
First, the integrands converge to the limit at almost every p a , b q P Q ni as
[Equation]
Besides, the integrands are uniformly bounded as
[Equation]
Therefore, by the dominated convergence theorem for the Bochner integral, we have
[Equation]
Hence by letting n sufficiently large, we have } f n ´ f Q } C p K q ă ε { 3 .
To sum up, we have shonw the cc -universality:
[Equation]
Notes.
In the proof, we employed a naive discretization based on the regular grids in Q .
However, since we know the closed-form expression of the ridgelet transform, we can discretize it more effectively.
For example, a better discretization scheme is investigated in the so-called Maurey-JonesBarron (MJB) theory and the dimension independent Barron’s bound ([see, e.g., 67]).