Original: Convolutional Neural Networks on Manifolds: From Graphs and Back
Revised: Convolutional Neural Networks on Manifolds: From Graphs and Back

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In recent years, geometric deep learning has gained attraction due to both theneed for machine learning on structured data (e.g., graphs) and the increasingavailability of this type of data.
Revised: In recent years, geometric deep learning has gained attraction due to both the need for machine learning on structured data (e.g., graphs) and the increasing availability of this type of data.

Original: Extensions of deep convolutional architectures tonon-Euclidean domains in particular are a powerful technique in sensor networkapplications — which can be seen as graphs — and 3D model analysis — whichcan be seen as manifolds.
Revised: Extensions of deep convolutional architectures to non-Euclidean domains in particular are a powerful technique in sensor network applications — which can be seen as graphs — and 3D model analysis — which can be seen as manifolds.

Original: While recent works have provided a better theoretical understanding of why convolutional neural network architectures work well on graphsof moderate size, in the large-scale regime that is the setting of most problems ofinterest, their behavior is not as well understood.
Revised: While recent works have provided a better theoretical understanding of why convolutional neural network architectures work well on graphs of moderate size, in the large-scale regime that is the setting of most problems of interest, their behavior is not as well understood.

Original: In this paper, we bridge this gapby modeling large graphs as samples from manifolds and studying manifold neuralnetworks (MNNs).
Revised: In this paper, we bridge this gap by modeling large graphs as samples from manifolds and studying manifold neural networks (MNNs).

Original: Our main contribution is to define a manifold convolution operation which, when “discretized” in both the space and time domains, is consistentwith the practical implementation of a graph convolution.
Revised: Our main contribution is to define a manifold convolution operation which, when “discretized” in both the space and time domains, is consistent with the practical implementation of a graph convolution.

Original: We then show that graphneural networks (GNNs) can be particularized from MNNs, which in turn are thelimits of these GNNs.
Revised: We then show that graph neural networks (GNNs) can be particularized from MNNs, which in turn are the limits of these GNNs.

Original: We conclude with numerical experiments showcasing anapplication of the MNN to point-cloud classification.
Revised: We conclude with numerical experiments showcasing an application of the MNN to point-cloud classification.

Original:  Introduction
Revised:  Introduction

Original: Convolutional neural networks (CNNs) have achieved impressive success in a wide range of applications, including but not limited to natural language processing [33], image denoising [42] andvideo analysis [40].
Revised: Convolutional neural networks (CNNs) have achieved impressive success in a wide range of applications, including but not limited to natural language processing [33], image denoising [43] and video analysis [41].

Original: Due to their remarkable performance, CNNs are recognized as the de facto deeplearning model to process data lying in Euclidean domains.
Revised: Due to their remarkable performance, CNNs are recognized as the de facto deep learning model to process data lying in Euclidean domains.

Original: However, many modern problems ofpractical interest — e.g., detection and recommendation in social networks [1, 9], resource allocationover wireless networks [35] and point cloud analysis for shape segmentation [39, 34] — requireprocessing data that is non-Euclidean.
Revised: However, many modern problems of practical interest — e g., detection and recommendation in social networks [1, 9], resource allocation over wireless networks [35], and point cloud analysis for shape segmentation [40, 34] — require processing data that is non-Euclidean.

Original: In light of this, recent works on geometric deep learning[11, 5, 22, 20] have focused on extending convolutional models to such domains.
Revised: In light of this, recent works on geometric deep learning [11, 5, 22, 20] have focused on extending convolutional models to such domains.

Original: Among these models, graphs are commonly used to construct a discrete model that captures theunderlying geometric structure.
Revised: Among these models, graphs are commonly used to construct a discrete model that captures the underlying geometric structure.

Original: Convolutions can be readily extended to graph convolutions, whichallows defining graph neural networks (GNNs) 
Revised: Convolutions can be readily extended to graph convolutions, which allows defining graph neural networks (GNNs) [13, 11, 44, 30, 7].

Original: In graphs of moderate size, GNNswork well empirically, which is backed by recent theoretical findings on their expressive power[15, 41] and stability [14, 31, 29, 16].
Revised: In graphs of moderate size, GNNs work well empirically, which is backed by recent theoretical findings on their expressive power [15, 42] and stability [14, 31, 29, 16].

Original: However, in the large-scale regime which is the setting ofmost problems of practical interest, their behavior is not as well understood.
Revised: However, in the large-scale regime which is the setting of most problems of practical interest, their behavior is not as well understood.

Original: Even if CNNs onmanifolds, from which graphs can be seen as samples [3, 5], have also been proposed [37, 20, 22],the relationship between CNNs on graphs and CNNs on manifolds remains elusive.
Revised: Even if CNNs on manifolds, from which graphs can be seen as samples [3, 5], have also been proposed [37, 20, 22], the relationship between CNNs on graphs and CNNs on manifolds remains elusive.

Original: In this paper, webridge this gap by defining a manifold convolution which can be particularized on sampled graphsfrom the manifold.
Revised: In this paper, we bridge this gap by defining a manifold convolution which can be particularized on sampled graphs from the manifold.

Original: This  allows seeing GNNs as samples from manifold neural networks (MNNs).
Revised: This makes the convolution operation interpretable both in graph and manifold domains and allows seeing GNNs as samples from manifold neural networks (MNNs).

Original: Because of the convergence of graphs to manifolds [3, 6, 8], we can further show that GNNs builtfrom these convolutions converge to MNNs.
Revised: Because of the convergence of graphs to manifolds [3, 6, 8], we can further show that GNNs built from these convolutions converge to MNNs.

Original: Our definition of manifold convolution is based on the  heat diffusion process.
Revised: Our definition of manifold convolution is based on the linear combination of the heat diffusion process.

Original: By cascading layersconsisting of manifold convolutions and pointwise nonlinearities, we then define manifold neuralnetworks (MNNs).
Revised: By cascading layers consisting of manifold convolutions and pointwise nonlinearities, we then define manifold neural networks (MNNs).

Original: Since we do not have access to the whole manifold — only a discrete set ofits points — we show how the MNN particularizes a GNN on the graph connecting the sampledpoints from the manifold, i.e., how to go from manifolds to graphs.
Revised: Since we do not have access to the whole manifold — only a discrete set of its points — we show how the MNN particularizes a GNN on the graph connecting the sampled points from the manifold, i.e., how to go from manifolds to graphs.

Original: We also show how this GNNconverges to the underlying MNN when the input data is bandlimited, i.e., how to go from graphsto manifolds.
Revised: We also show how this GNN converges to the underlying MNN when the input data is bandlimited, i.e., how to go from graphs to manifolds.

Original: For practical implementation purposes, we further discretize the time axis of the heatdiffusion process, recovering the usual (learnable) parametrization of the spatial graph convolution[13] when the time horizon is finite.
Revised: For practical implementation purposes, we further discretize the time axis of the heat diffusion process, recovering the usual (learnable) parametrization of the spatial graph convolution [13] when the time horizon is finite.

Original: Finally, we verify the performance of our proposed architecturewith numerical experiments on a point cloud dataset.
Revised: Finally, we verify the performance of our proposed architecture with numerical experiments on a point cloud dataset.

Original: Related works include neural networks built on graphons [27, 28], which are limits of sequences ofdense graphs.
Revised: Related works include neural networks built on graphons [27, 28], which are limits of sequences of dense graphs.

Original: Different from manifolds, graphons only represent the limits of graphs with unboundeddegrees [19, 4].
Revised: Different from manifolds, graphons only represent the limits of graphs with unbounded degrees [19, 4].

Original: The convergence of GNNs on random graphs is proposed in [16] while the limit is aconstructed continuous GNN.
Revised: The convergence of GNNs on random graphs is proposed in [16] while the limit is a constructed continuous GNN.

Original: The results in [18] show the convergence and transferability of GNNsif the graphs are sampled with a specific sampling operator from a continuous topological space.
Revised: The results in [18] show the convergence and transferability of GNNs if the graphs are sampled with a specific sampling operator from a continuous topological space.

Original: However, the sampling operator requires the knowledge of the spectrum of the underlying topologicalspace, which in practice is often inaccessible.
Revised: However, the sampling operator requires the knowledge of the spectrum of the underlying topological space, which in practice is often inaccessible.

Original: CNNs on manifolds have been constructed in [20] withgeodesic polar coordinates and in [22] with mixtures of Gaussian kernels.
Revised: CNNs on manifolds have been constructed in [20] with geodesic polar coordinates and in [22] with mixtures of Gaussian kernels.

Original: Both works only focuson spatial approaches.
Revised: Both works only focus on spatial approaches.

Original: Stability of MNNs has also been studied, by considering the perturbations tothe Laplace-Beltrami operator [37, 36] in a spectral approach.
Revised: Stability of MNNs has also been studied, by considering the perturbations to the Laplace-Beltrami operator [37, 36] in a spectral approach.

Original: All these works on MNNs lack theexplicit relationship with GNNs, which are commonly implemented for realizations on manifolds[5, 22].
Revised: All these works on MNNs lack the explicit relationship with GNNs, which are commonly implemented for realizations on manifolds [5, 22].

Original: A general framework for algebraic neural networks has been proposed for architectures withcommutative algebras [25].
Revised: A general framework for algebraic neural networks has been proposed for architectures with commutative algebras [25].

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: We start with some preliminary concepts and definethe manifold convolutions as well as the manifold neural networks in Section 2.
Revised: We start with some preliminary concepts and define the manifold convolutions as well as the manifold neural networks in Section 2.

Original: In Section 3, weimplement the discretization in space and time domains to make the MNNs realizable which alsobring back GNNs and we also prove the convergence of GNNs to MNNs.
Revised: In Section 3, we implement the discretization in space and time domains to make the MNNs realizable which also bring back GNNs and we also prove the convergence of GNNs to MNNs.

Original: Our proposed MNN isverified in a model classification problem in Section 4.
Revised: Our proposed MNN is verified in a model classification problem in Section 4.

Original: The conclusions are presented in Section 5.
Revised: The conclusions are presented in Section 5.

Original:  Manifold Convolutions and Manifold Neural Networks 
Revised:  Manifold Convolutions and Manifold Neural Networks 2.1 Manifold Convolution

Original: 2.1 Manifold Convolution

Original: We consider a compact, smooth, and differentiable d -dimensional submanifold M embedded in R N .
Revised: We consider a compact, smooth, and differentiable d -dimensional submanifold M embedded in R N .

Original: The embedding induces a Riemannian structure [10] on M which endows a measure µ over themanifold.
Revised: The embedding induces a Riemannian structure [10] on M which endows a measure µ over the manifold.

Original: Manifold data supported on M are smooth scalar functions f : M → R , which we referto as manifold signals in the following.
Revised: Manifold data supported on M are smooth scalar functions f : M → R , which we refer to as manifold signals in the following.

Original: We consider f belongs to L 2 ( M ) in which we define theinner product as
Revised: We consider f belongs to L 2 ( M ) in which we define the inner product as

Original: [Equation]
Revised: [Equation]

Original: with the norm defined as ∥ f ∥ 2 L 2 ( M ) = ⟨ f, f ⟩ L 2 ( M ) , representing the energy of manifold signal f .
Revised: with the norm defined as ∥ f ∥ 2 L 2 ( M ) = ⟨ f, f ⟩ L 2 ( M ) , representing the energy of manifold signal f .

Original: We consider manifold signals f with finite energy.
Revised: We consider manifold signals f with finite energy.

Original: The manifold is locally Euclidean and we denote the local Euclidean space around x ∈ M astangent space T x M .
Revised: The manifold is locally Euclidean and we denote the local Euclidean space around x ∈ M as tangent space T x M .

Original: We use T M to represent the disjoint union of all tangent spaces on M .
Revised: We use T M to represent the disjoint union of all tangent spaces on M .

Original: Theintrinsic gradient for differentiation is a local operator [5] and can thus be written as an operator∇ :
Revised: The intrinsic gradient for differentiation is a local operator [5] and can thus be written as an operator ∇ :

Original: L 2 ( M ) → L 2 ( T M ) mapping scalar functions to tangent vector functions on M .
Revised: L 2 ( M ) → L 2 ( T M ) mapping scalar functions to tangent vector functions on M .

Original: The adjointoperator of intrinsic gradient is the intrinsic divergence defined as div : L 2 ( T M )
Revised: The adjoint operator of intrinsic gradient is the intrinsic divergence defined as div : L 2 ( T M )

Original: → L 2 ( M ) .
Revised: → L 2 ( M ) .

Original: Basedon these two differentiation operators, the Laplace-Beltrami (LB) operator L : [Equation]
Revised: Based on these two differentiation operators, the Laplace-Beltrami (LB) operator L : L 2 ( M )

Original: [Equation] → L [Equation] )can be defined as the intrinsic divergence of the intrinsic gradient [26], formally as
Revised: → L 2 ( M ) can be defined as the intrinsic divergence of the intrinsic gradient [26], formally as

Original: [Equation]
Revised: [Equation]

Original: Similar to the Laplacian operator in Euclidean domains or the Laplace matrix in graphs [23], the LBoperator evaluates how much the function value at point x differs from the average function valueof its neighborhood [5].
Revised: Similar to the Laplacian operator in Euclidean domains or the Laplace matrix in graphs [23], the LB operator evaluates how much the function value at point x differs from the average function value of its neighborhood [5].

Original: Since L , like ∇ , is a local operator depending on the tangent space T x M of each point x ∈ M , in the following we make this dependence explicit by writing L = L x and∇ = ∇ x .
Revised: Since L , like ∇ , is a local operator depending on the tangent space T x M of each point x ∈ M , in the following we make this dependence explicit by writing L = L x and ∇ = ∇ x .

Original: The LB operator is essential to capture the heat diffusion over manifolds by the heat equation
Revised: The LB operator is essential to capture the heat diffusion over manifolds by the heat equation

Original: [Equation]
Revised: [Equation]

Original: where u ( x, t ) ∈ L 2 ( M ) measures the temperature at x ∈ M at time t ∈ R
Revised: where u ( x, t ) ∈ L 2 ( M ) measures the temperature at x ∈ M at time t ∈ R

Original: + .
Revised: + .

Original: This can be interpretedto mean that, at point x , the rate at which the manifold “cools down” is proportional to the differencebetween the temperature of x and the local average of the temperature of the points in its neighborhood.
Revised: This can be interpreted to mean that, at point x , the rate at which the manifold “cools down” is proportional to the difference between the temperature of x and the local average of the temperature of the points in its neighborhood.

Original: With initial condition given by u ( x, 0) =
Revised: With initial condition given by u ( x, 0) =

Original: f ( x ) , the solution can be expressed as u ( x, t ) =
Revised: f ( x ) , the solution can be expressed as u ( x, t ) =

Original: e − t L f ( x ) .
Revised: e − t L f ( x ) .

Original: We define the manifold convolution with a filter impulse response function ˜ h and input manifoldsignal f as follows.
Revised: We define the manifold convolution with a filter impulse response function ˜ h and input manifold signal f as follows.

Original: Definition 1 (Manifold convolutional filter) Let ˜ h
Revised: Definition 1 (Manifold convolutional filter) Let ˜ h

Original: : R + → R and let f ∈ L 2 ( M ) be the datasupported on M .
Revised: : R + → R and let f ∈ L 2 ( M ) be the data supported on M .

Original: The manifold filter with impulse response ˜ h , denoted as h , is given by
Revised: The manifold filter with impulse response ˜ h , denoted as h , is given by

Original: [Equation]
Revised: [Equation]

Original: where u ( x, t ) is the solution of the heat equation (3) with u ( x, 0)
Revised: where u ( x, t ) is the solution of the heat equation (3) with u ( x, 0)

Original: =
Revised: =

Original: f
Revised: f

Original: ( x ) .
Revised: ( x ) .

Original: Manifold filters are local spatial operators operating directly on points on the manifold based on the
Revised: Manifold filters are local spatial operators operating directly on points on the manifold based on the

Original: LB operator.
Revised: LB operator.

Original: The exponential term e − t L can be interpreted as a shift operator like the time delay in a
Revised: The exponential term e − t L can be interpreted as a shift operator like the time delay in a

Original: Linear-Time Invariant (LTI) filter [24] and the graph shift in a Linear-Shift Invariant (LSI) graph filter[11].
Revised: Linear-Time Invariant (LTI) filter [24] and the graph shift in a Linear-Shift Invariant (LSI) graph filter [11].

Original: In fact, manifold filters can recover graph filters by discretization, which we discuss thoroughlyin Section 3.
Revised: In fact, manifold filters can recover graph filters by discretization, which we discuss thoroughly in Section 3.

Original: The LB operator L is self-adjoint and positive-semidefinite.
Revised: The LB operator L is self-adjoint and positive-semidefinite.

Original: Considering that manifold M is compactwithout boundary, the operator L possesses a real discrete spectrum { λ i } ∞ i =1 with the eigenvalues λ iand the corresponding eigenfunctions ϕ
Revised: Considering that manifold M is compact without boundary, the operator L possesses a real discrete spectrum { λ i } ∞ i =1 with the eigenvalues λ i and the corresponding eigenfunctions ϕ

Original: i satisfying L ϕ i
Revised: i satisfying L ϕ i

Original: = λ
Revised: = λ

Original: i ϕ i .
Revised: i ϕ i .

Original: Eigenvalue λ i can be interpreted asthe canonical frequency and the eigenfunction ϕ
Revised: Eigenvalue λ i can be interpreted as the canonical frequency and the eigenfunction ϕ

Original: i as the canonical oscillation mode.
Revised: i as the canonical oscillation mode.

Original: By projecting amanifold signal f onto the eigenfunction, we can write the frequency representation ˆ f as
Revised: By projecting a manifold signal f onto the eigenfunction, we can write the frequency representation ˆ f as

Original: [Equation]
Revised: [Equation]

Original: Based on this concept, we can define the bandlimited manifold signal as follows.
Revised: Based on this concept, we can define the bandlimited manifold signal as follows.

Original: Definition 2 (Bandlimited manifold signal)
Revised: Definition 2 (Bandlimited manifold signal)

Original: A manifold signal f ∈ L 2 ( M ) is defined as λ Mbandlimited if [ ˆ f ] i = 0 for all i such that λ i > λ M .
Revised: A manifold signal f ∈ L 2 ( M ) is defined as λ Mbandlimited if [ ˆ f ] i = 0 for all i such that λ i > λ M .

Original: The spectrum and eigenbasis of the LB operator help to understand not only the frequency behavior ofmanifold signal but the manifold filter h ( L ) .
Revised: The spectrum and eigenbasis of the LB operator help to understand not only the frequency behavior of manifold signal but the manifold filter h ( L ) .

Original: The frequency representation of manifold filter output gcan be similarly written as
Revised: The frequency representation of manifold filter output g can be similarly written as

Original: [Equation]
Revised: [Equation]

Original: By substituting e − t L ϕ i
Revised: By substituting e − t L ϕ i

Original: =
Revised: =

Original: e − tλ i ϕ
Revised: e − tλ i ϕ

Original: i , we can get
Revised: i , we can get

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The function solely dependent on λ
Revised: The function solely dependent on λ

Original: i is defined as the frequency response of the filter h ( L ) , whichcan be stated formally as follows.
Revised: i is defined as the frequency response of the filter h ( L ) , which can be stated formally as follows.

Original: Definition 3 (Frequency response of manifold filter)
Revised: Definition 3 (Frequency response of manifold filter)

Original: The frequency response of the filter h ( L ) isgiven by
Revised: The frequency response of the filter h ( L ) is given by

Original: [Equation]
Revised: [Equation]

Original: (8) which leads (7) to [ˆ g ] i
Revised: which leads (7) to [ˆ g ] i

Original: = ˆ h
Revised: = ˆ h

Original: ( λ i )
Revised: ( λ i )

Original: [ ˆ f ] i .
Revised: [ ˆ f ] i .

Original: Definition 3 indicates that the frequency response of manifold filter is point-wise in frequency domain.
Revised: Definition 3 indicates that the frequency response of manifold filter is point-wise in frequency domain.

Original: Combining the frequency representation of ˆ g over the whole spectrum, we can obtain the frequencyrepresentation of manifold filter h as
Revised: Combining the frequency representation of ˆ g over the whole spectrum, we can obtain the frequency representation of manifold filter h as

Original: [Equation]
Revised: [Equation]

Original: If we consider the input manifold signal f as λ M -bandlimited
Revised: If we consider the input manifold signal f as λ M -bandlimited

Original: (Definition 2), the output signal g canbe written as
Revised: (Definition 2), the output signal g can be written as

Original: [Equation]
Revised: [Equation]

Original: where M = # { λ i ≤ λ M } i counts the number of eigenvalues within the bandwidth. 
Revised: where M = # { λ i ≤ λ M } i counts the number of eigenvalues within the bandwidth. Remarks.

Revised: We note that the diffusion equations, e.g. u ( x, t ) =

Revised: e − t L f ( x ) , have spectral representations themselves, which is exactly the motivation for the development of Fourier analysis.

Revised: In the context of GNNs or MNNs, this frequency representation for diffusion equations is pointed out in [5].

Revised: However, this is different from what we have defined as a manifold convolution.

Original:  2.2 Manifold Neural Networks
Revised: We show that convolutions are defined as linear combinations of the elements in the diffusion trajectories [cf. (4)], which are generalizations of convolutions in the time domain [38] and graph convolutions when the manifold is discretized (Section 3). Our defined manifold convolutional filters also have the spectral representation (10) which generalizes the classical Fourier transform and the graph Fourier transform. 2.2 Manifold Neural Networks

Original: Manifold neural networks (MNNs) augment manifold filters as we have defined in Definition 1 witha point-wise nonlinear activation function σ : R → R which is an independent application on eachpoint of the manifold.
Revised: Manifold neural networks (MNNs) augment manifold filters as we have defined in Definition 1 with a point-wise nonlinear activation function σ : R → R which is an independent application on each point of the manifold.

Original: We further make an assumption on its continuity as follows.
Revised: We further make an assumption on its continuity as follows.

Original: Assumption 1 (Normalized Lipschitz activation functions)
Revised: Assumption 1 (Normalized Lipschitz activation functions)

Original: The activation function σ is normalized Lipschitz continuous, i.e., | σ ( a ) − σ ( b ) | ≤ | a − b | , with σ (0)
Revised: The activation function σ is normalized Lipschitz continuous, i.e., | σ ( a ) − σ ( b ) | ≤ | a − b | , with σ (0)

Original: = 0 .
Revised: = 0 .

Original: Note that this assumption is rather reasonable, since most common activation functions (e.g., the
Revised: Note that this assumption is rather reasonable, since most common activation functions (e.g., the

Original: ReLU, the modulus and the sigmoid) are normalized Lipschitz by design.
Revised: ReLU, the modulus and the sigmoid) are normalized Lipschitz by design.

Original: In a single-layer MNN, the manifold signal f is passed through a manifold filter followed by apoint-wise nonlinearity as
Revised: In a single-layer MNN, the manifold signal f is passed through a manifold filter followed by a point-wise nonlinearity as

Original: [Equation]
Revised: [Equation]

Original: (11) which can be seen as a basic nonlinear processing of the input manifold signal.
Revised: [Equation] which can be seen as a basic nonlinear processing of the input manifold signal.

Original: By stacking thisprocedure in layers, a multi-layer MNN can be constructed which can be formally written as afunction composition.
Revised: By stacking this procedure in layers, a multi-layer MNN can be constructed which can be formally written as a function composition.

Original: The output manifold signal of a layer becomes the input signal of the nextlayer.
Revised: The output manifold signal of a layer becomes the input signal of the next layer.

Original: Let l = 1 , 2 , · · · , L stand for the index for the layer and h l ( L ) as the manifold filter on eachlayer.
Revised: Let l = 1 , 2 , · · · , L stand for the index for the layer and h l ( L ) as the manifold filter on each layer.

Original: For a specific layer l , filter h l ( L ) takes the output f l − 1 ( x ) as the input produces the output oflayer l as
Revised: For a specific layer l , filter h l ( L ) takes the output f l − 1 ( x ) as the input produces the output of layer l as

Original: [Equation]
Revised: [Equation]

Original: where f 0 ( x ) = f
Revised: where f 0 ( x ) = f

Original: ( x ) is the given input manifold signal.
Revised: ( x ) is the given input manifold signal.

Original: After a recursive applications through Llayers, we can get the output of the MNN as f L ( x ) .
Revised: After a recursive applications through L layers, we can get the output of the MNN as f L ( x ) .

Original: When considering multiple features in each layer to increase the representation power of MNN, themanifold filters map the input F l − 1 features from layer
Revised: When considering multiple features in each layer to increase the representation power of MNN, the manifold filters map the input F l − 1 features from layer

Original: l − 1 to F l intermediate features in layer lwith a bank of manifold filters, i.e.,
Revised: l − 1 to F l intermediate features in layer l with a bank of manifold filters, i.e.,

Original: [Equation]
Revised: [Equation]

Original:  where h pql ( L ) is the filter mapping the q -th feature from layer
Revised: ) where h pql ( L ) is the filter mapping the q -th feature from layer

Original: l − 1 to the p -th feature of layer l , for1 ≤ q ≤ F l − 1 and 1 ≤ p ≤ F l .
Revised: l − 1 to the p -th feature of layer l , for 1 ≤ q ≤ F l − 1 and 1 ≤ p ≤ F l .

Original: The intermediate features are then processed by the nonlinearity σ as
Revised: The intermediate features are then processed by the nonlinearity σ as

Original: [Equation]
Revised: [Equation]

Original: The output of layer L is the output of MNN with F L features.
Revised: The output of layer L is the output of MNN with F L features.

Original: To represent the MNN more precisely,we gather the impulse responses of all the manifold filters h pql as a function set H and define the
Revised: To represent the MNN more precisely, we gather the impulse responses of all the manifold filters h pql as a function set H and define the

Original: MNN as a map Φ ( H , L , f ) .
Revised: MNN as a map Φ ( H , L , f ) .

Original: This map emphasizes that the neural network is parameterized by boththe filter functions H and the LB operator L .
Revised: This map emphasizes that the neural network is parameterized by both the filter functions H and the LB operator L .

Original:  Discretization in the Space and Time Domains
Revised:  Discretization in the Space and Time Domains

Original: MNNs are built from manifold convolutional filters (Definition 1) operating on a continuous manifoldand over an infinite continuous time horizon.
Revised: MNNs are built from manifold convolutional filters (Definition 1) operating on a continuous manifold and over an infinite continuous time horizon.

Original: This makes it impractical to implement directly theneural network architecture described by (14) in applications.
Revised: This makes it impractical to implement directly the neural network architecture described by (14) in applications.

Original: In this section, we discuss the practicalapplication of MNNs and the connections with GNNs (14) over a set of discrete samples from themanifold in a finite and discrete time frame.
Revised: In this section, we discuss the practical application of MNNs and the connections with GNNs (14) over a set of discrete samples from the manifold in a finite and discrete time frame.

Original: 3.1 Discretization in the Space Domain
Revised: 3.1 Discretization in the Space Domain

Original: In practice, to operate on a continuous topological space, it is natural to access the underlyinggeometric structure by sampling points over the continuous domain.
Revised: In practice, to operate on a continuous topological space, it is natural to access the underlying geometric structure by sampling points over the continuous domain.

Original: Therefore we sample uniformlyon the manifold and connect the sampled points as a graph to approximate the underlying manifold[8, 3].
Revised: Therefore we sample uniformly on the manifold and connect the sampled points as a graph to approximate the underlying manifold [8, 3].

Original: Specifically, we model the set of n sampled points as X = { x 1 , x 2 , . . .
Revised: Specifically, we model the set of n sampled points as X = { x 1 , x 2 , . . .

Original: , x n } which aresampled i.i.d.
Revised: , x n } which are sampled i.i.d.

Original: from measure µ of manifold M ⊂ R N .
Revised: from measure µ of manifold M ⊂ R N .

Original: A complete weighted symmetric graph G ncan be constructed by seeing the sampled points as the vertices of the graph.
Revised: A complete weighted symmetric graph G n can be constructed by seeing the sampled points as the vertices of the graph.

Original: The edge weight w ijconnecting point x i and x j is defined as a graph signal
Revised: The edge weight w ij connecting point x i and x j is defined as a graph signal

Original: [Equation]
Revised: [Equation]

Original: with ∥ x i − x j ∥ representing the Euclidean distance between x i and x j .
Revised: with ∥ x i − x j ∥ representing the Euclidean distance between x i and x j .

Original: Parameter t n controls thechosen Gaussian kernel [3].
Revised: Parameter t n controls the chosen Gaussian kernel [3].

Original: The adjacency matrix of G n is thus defined as [ A n ] ij = w ij for1 ≤ i, j ≤ n with A n ∈ R n × n .
Revised: The adjacency matrix of G n is thus defined as [ A n ] ij = w ij for 1 ≤ i, j ≤ n with A n ∈ R n × n .

Original: The correspondent graph Laplacian matrix L n [21] thus can becalculated as L n = diag ( A n 1 ) − A n .
Revised: The correspondent graph Laplacian matrix L n [21] thus can be calculated as L n = diag ( A n 1 ) − A n .

Original: We define a uniform sampling operator P n : L 2 ( M ) →
Revised: We define a uniform sampling operator P n : L 2 ( M ) →

Original: L 2 ( G n ) to discretize the signal f onmanifold (Definition 2) as the data supported on the graph G n , which is denoted asx n = P n f with [ x n ] i
Revised: L 2 ( G n ) to discretize the signal f on manifold (Definition 2) as the data supported on the graph G n , which is denoted as

Original: [Equation]
Revised: [Equation]

Original: where the i -th entry of the graph signal x n is the manifold signal f evaluated at the sampled point x i .
Revised: where the i -th entry of the graph signal x n is the manifold signal f evaluated at the sampled point x i .

Original: Considering that the discrete points { x 1 , x 2 , . . .
Revised: Considering that the discrete points { x 1 , x 2 , . . .

Original: , x n } are uniformly sampled from manifold M withmeasure µ , the empirical measure associated with d µ can be denoted as p n = 1 n (cid:80) ni
Revised: , x n } are uniformly sampled from manifold M with measure µ , the empirical measure associated with d µ can be denoted as p n = 1 n (cid:80) ni

Original: =

Revised: =

Original: δ x i , where δ x iis the Dirac measure supported on x i .
Revised: δ x i , where δ x i is the Dirac measure supported on x i .

Original: Similar to the inner product defined in the L 2 ( M ) space (1),the inner product on L 2 ( G n ) is denoted as
Revised: Similar to the inner product defined in the L 2 ( M ) space (1), the inner product on L 2 ( G n ) is denoted as

Original: [Equation]
Revised: [Equation]

Original: The norm in L 2 ( G n ) is therefore ∥ u
Revised: The norm in L 2 ( G n ) is therefore ∥ u

Original: ∥ 2 L 2 ( G n ) = ⟨ u, u ⟩ L 2 ( G n ) , with u, v ∈ L 2 ( M ) .
Revised: ∥ 2 L 2 ( G n ) = ⟨ u, u ⟩ L 2 ( G n ) , with u, v ∈ L 2 ( M ) .

Original: For signalsu , v ∈ L 2 ( G n ) , the inner product is therefore ⟨ u , v ⟩ L 2 ( G n ) = 1 n (cid:80) ni
Revised: For signals u , v ∈ L 2 ( G n ) , the inner product is therefore ⟨ u , v ⟩ L 2 ( G n ) = 1 n (cid:80) ni

Original: =1 [ u ] i [ v ] i .
Revised: =1 [ u ] i [ v ] i .

Original: The parametrization of manifold filter (Definition 1) enables us to replace the LB operator with thegraph Laplacian operator L n and turns h to a graph convolutional filter, i.e.,
Revised: The parametrization of manifold filter (Definition 1) enables us to replace the LB operator with the graph Laplacian operator L n and turns h to a graph convolutional filter, i.e.,

Original: [Equation]
Revised: [Equation]

Original: 18) where z n is the output graph signal.
Revised: 18) where z n is the output graph signal.

Original: By cascading the layers containing the graph convolutional filtersas defined in (18) and point-wise nonlinearities, we can define a neural network on this constructedgraph G n as[Equation] 
Revised: By cascading the layers containing the graph convolutional filters as defined in (18) and point-wise nonlinearities, we can define a neural network on this constructed graph G n as

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can represent this neural network as a map Φ ( H , L n , x n ) parametrized by the filterfunctions H and the graph Laplacian L n .
Revised: Similarly, we can represent this neural network as a map Φ ( H , L n , x n ) parametrized by the filter functions H and the graph Laplacian L n .

Original: As the number of sampling points n goes to infinity, thediscrete graph signal x n converges to the manifold signal f while the discrete graph Laplacian matrix
Revised: As the number of sampling points n goes to infinity, the discrete graph signal x n converges to the manifold signal f while the discrete graph Laplacian matrix

Original: L n has also been proved to converge to the LB operator L of the underlying manifold [3, 6, 8].
Revised: L n has also been proved to converge to the LB operator L of the underlying manifold [3, 6, 8].

Original: Byintroducing the definition of Lipschitz continuous manifold filters in Definition 4, we can combinethe convergence of signals ( x n p → f ) and Laplace operators ( L n p→ L ) to derive the convergence ofΦ
Revised: By introducing the definition of Lipschitz continuous manifold filters in Definition 4, we can combine the convergence of signals ( x n p → f ) and Laplace operators ( L n p

Original:  ( H , L n , x n ) p → Φ ( H , L , f ) .
Revised: → L ) to derive the convergence of Φ ( H , L n , x n ) p → Φ ( H , L , f ) .

Original: Definition 4 (Manifold filter with Lipschitz continuity)
Revised: Definition 4 (Manifold filter with Lipschitz continuity)

Original: A manifold filter is C -Lispchitz if its frequency response is Lipschitz continuous with constant C , i.e,
Revised: A manifold filter is C -Lispchitz if its frequency response is Lipschitz continuous with constant C , i.e,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We conclude that the outputs of the neural network on the graph converge to the outputs of thecontinuous MNN as Theorem 1 shows under the mild assumption on the amplitude of the Lipschitzcontinuous manifold filter frequency response (Assumption 2).
Revised: We conclude that the outputs of the neural network on the graph converge to the outputs of the continuous MNN as Theorem 1 shows under the mild assumption on the amplitude of the Lipschitz continuous manifold filter frequency response (Assumption 2).

Original: Assumption 2 (Non-amplifying filters)
Revised: Assumption 2 (Non-amplifying filters)

Original: A manifold filter is non-amplifying if for all λ ∈ (0 , ∞ ) ,its frequency response ˆ h satisfies | ˆ h ( λ ) | ≤ 1 .
Revised: A manifold filter is non-amplifying if for all λ ∈ (0 , ∞ ) , its frequency response ˆ h satisfies | ˆ h ( λ ) | ≤ 1 .

Original: Note that this assumption is reasonable, because the filter function ˆ h ( λ ) can always be normalized.
Revised: Note that this assumption is reasonable, because the filter function ˆ h ( λ ) can always be normalized.

Original: Theorem 1 Let X = { x 1 , x 2 , . . .
Revised: Theorem 1 Let X = { x 1 , x 2 , . . .

Original: , x n } be a set of points sampled i.i.d.
Revised: , x n } be a set of points sampled i.i.d.

Original: from a d -dimensionalmanifold M ⊂ R N .
Revised: from a d -dimensional manifold M ⊂ R N .

Original: Let G n be a graph constructed from X with weight values set as (15) witht n = n − 1 / ( d +2+ α ) and α > 0 .
Revised: Let G n be a graph constructed from X with weight values set as (15) with t n = n − 1 / ( d +2+ α ) and α > 0 .

Original: Let x n be the data supported on G n sampled by operator P n as (16)from a bandlimited manifold signal f .
Revised: Let x n be the data supported on G n sampled by operator P n as (16) from a bandlimited manifold signal f .

Original: Let Φ ( H , · , · ) be the neural network parameterized by the LBoperator L of manifold M (14) or by the graph Laplacian L n of G n .
Revised: Let Φ ( H , · , · ) be the neural network parameterized by the LB operator L of manifold M (14) or by the graph Laplacian L n of G n .

Original: Under the assumption that thefilters in H are Lipschitz continuous and non-amplifying, it holds that
Revised: Under the assumption that the filters in H are Lipschitz continuous and non-amplifying, it holds that

Original: [Equation]
Revised: [Equation]

Original: (21) where the limits are taken in probability.
Revised: where the limits are taken in probability.

Original: Proof.
Revised: Proof.

Original: See Supplementary material.
Revised: See Supplementary material.

Original: Theorem 1 states that, for bandlimited input signals, neural networks on graphs generated from amanifold actually converge to MNNs under certain assumptions as the graph size grows to infinity.
Revised: Theorem 1 states that, for bandlimited input signals, neural networks on graphs generated from a manifold actually converge to MNNs under certain assumptions as the graph size grows to infinity.

Original: Neural networks supported on large graphs have been proved to be stable and transferable empiricallyin many applications [35, 12].
Revised: Neural networks supported on large graphs have been proved to be stable and transferable empirically in many applications [35, 12].

Original: Our result provides a critical theoretical support for understandingthese properties of large graph neural networks regardless of graph sparsity by considering the MNNas their limit.
Revised: Our result provides a critical theoretical support for understanding these properties of large graph neural networks regardless of graph sparsity by considering the MNN as their limit.

Original: We remark that the definition of graph convolutional filter in (18) is a continuous formbased on the integration over the whole time domain.
Revised: We remark that the definition of graph convolutional filter in (18) is a continuous form based on the integration over the whole time domain.

Original: 3.2 Discretization in the Time Domain
Revised: 3.2 Discretization in the Time Domain

Original: Generally, we operate on the neural networks in practice in digital systems with digital filters.
Revised: Generally, we operate on the neural networks in practice in digital systems with digital filters.

Original: Thisindicates that the time horizon on which the convolutional filters are usually defined is not continuous,but discrete.
Revised: This indicates that the time horizon on which the convolutional filters are usually defined is not continuous, but discrete.

Original: Therefore we discretize the impulse response function ˜ h ( t ) of the manifold filter with afixed sampling interval T s = 1 and replace the filter response function with a series of coefficientsh k=
Revised: Therefore we discretize the impulse response function ˜ h ( t ) of the manifold filter with a fixed sampling interval T s = 1 and replace the filter response function with a series of coefficients h k

Original:  ˜
Revised: = ˜

Original: h
Revised: h

Original: ( k ) , k = 0 , 1 , 2 . . . .
Revised: ( k ) , k = 0 , 1 , 2 . . . .

Original: In order to make the digital filters learnable and realizable, we further fixa finite time horizon with K samples and rewrite the manifold filter as (22)[Equation]
Revised: In order to make the digital filters learnable and realizable, we further fix a finite time horizon with K samples and rewrite the manifold filter as (22)

Original: [Equation]
Revised: [Equation]

Original: which corresponds to the form of a finite impulse response (FIR) filter with a shift operator e −L .
Revised: which corresponds to the form of a finite impulse response (FIR) filter with a shift operator e −L .

Original: The manifold convolution in the discrete time domain can therefore be written as a summation over Ktime steps instead of the integration form.
Revised: The manifold convolution in the discrete time domain can therefore be written as a summation over K time steps instead of the integration form.

Original: If we discretize the time domain of the graph convolutionalfilter defined in (18), this leads to a completely practical manifold filter supported on a constructedgraph model and discrete finite time steps, i.e.,
Revised: If we discretize the time domain of the graph convolutional filter defined in (18), this leads to a completely practical manifold filter supported on a constructed graph model and discrete finite time steps, i.e.,

Original: [Equation]
Revised: [Equation]

Original: We observe that (23) recovers the form of graph convolution [11, 31] with e − L n seen as the graphshift operator.
Revised: We observe that (23) recovers the form of graph convolution [11, 31] with e − L n seen as the graph shift operator.

Original: By replacing the filter h pql ( L n ) in (19) with the form of (23), we further recovers
Revised: By replacing the filter h pql ( L n ) in (19) with the form of (23), we further recovers

Original: GNNs with MNNs in this discrete finite time frame.
Revised: GNNs with MNNs in this discrete finite time frame.

Original: Up until now, we have completed our process ofbuilding MNNs from graphs and back, i.e., we relate GNNs with MNNs in a two-way connection.
Revised: Up until now, we have completed our process of building MNNs from graphs and back, i.e., we relate GNNs with MNNs in a two-way connection.

Original: We have shown that neural networks on the graphs sampled from the manifold converge to neuralnetworks built on the manifold when the input manifold signal is bandlimited.
Revised: We have shown that neural networks on the graphs sampled from the manifold converge to neural networks built on the manifold when the input manifold signal is bandlimited.

Original: This provides atheoretical support for analyzing the stability and transferability of large graph neural networkssupported either on dense or sparse graphs.
Revised: This provides a theoretical support for analyzing the stability and transferability of large graph neural networks supported either on dense or sparse graphs.

Original: The discretization over the space and time domainsenables the particularization as GNNs constructed on the sampled points over the manifold and thepractical implementation of MNNs.
Revised: The discretization over the space and time domains enables the particularization as GNNs constructed on the sampled points over the manifold and the practical implementation of MNNs.

Original: We remark that Theorem 1 also indicates that our proposed
Revised: We remark that Theorem 1 also indicates that our proposed

Original: MNN can be well approximated by a large enough GNN sampled from the MNN, which supportsour simulation setup in the following section theoretically.
Revised: MNN can be well approximated by a large enough GNN sampled from the MNN, which supports our simulation setup in the following section theoretically.

Original:  Numerical Experiments
Revised:  Numerical Experiments

Original: We evaluate the performance of our proposed MNN structure with approximated GNNs.
Revised: We evaluate the performance of our proposed MNN structure with approximated GNNs.

Original: We carryout a classification problem with ModelNet10 dataset [38].
Revised: We carry out a classification problem with ModelNet10 dataset [39].

Original: The dataset contains 3,991 meshed CADmodels from 10 categories for training and 908 models for testing.[Figure] [Table]
Revised: The dataset contains 3,991 meshed CAD models from 10 categories for training and 908 models for testing.

Original: [Figure] [Table] [Figure] We sample n points from each meshed model uniformly and construct a dense graph.
Revised: We sample n points from each meshed model uniformly and construct a dense graph.

Original: Explicitly, wesee each point as a node and the edge weights are calculated with (15).
Revised: Explicitly, we see each point as a node and the edge weights are calculated with (15).

Original: The point cloud models areapproximated by the constructed graphs.
Revised: The point cloud models are approximated by the constructed graphs.

Original: The edge weights are calculated according to (15) with t nset as n − 1 / 5 .
Revised: The edge weights are calculated according to (15) with t n set as n − 1 / 5 .

Original: The Laplacian matrix is calculated for each point cloud model.
Revised: The Laplacian matrix is calculated for each point cloud model.

Original: Our goal is to identifythe models for chair from other models as illustrated in Figure 1.
Revised: Our goal is to identify the models for chair from other models as illustrated in Figure 1.

Original: We implement graph filters with1 (GF1Ly) and 2 layers (GF2Ly) along with graph neural networks with 1 (GNN1Ly) and 2 layers(GNN2Ly) respectively.
Revised: We implement graph filters with 1 (GF1Ly) and 2 layers (GF2Ly) along with graph neural networks with 1 (GNN1Ly) and 2 layers [Table] [Figure] [Table]

Original:  The single layer architectures of graph filters and graph neural networkscontain F 0 = 3 input features which are the coordinates of every point in 3d space.
Revised: averaged over 5 data realizations. [Figure] (GNN2Ly) respectively. The single layer architectures of graph filters and graph neural networks contain F 0 = 3 input features which are the coordinates of every point in 3d space.

Original: The outputfeatures are set as F 1 = 64 .
Revised: The output features are set as F 1 = 64 .

Original: The architectures with 2 layers include another layer with F 2 = 32features.
Revised: The architectures with 2 layers include another layer with F 2 =features.

Original: The number of filter taps in each layer are set as K = 5 .
Revised: The number of filter taps in each layer are set as K = 5 .

Original: The nonlinearity is a ReLu function.
Revised: The nonlinearity is a ReLu function.

Original: All the architectures are concluded with a linear readout layer mapping the output features to a binaryscalar to estimate the classifications.
Revised: All the architectures are concluded with a linear readout layer mapping the output features to a binary scalar to estimate the classifications.

Original: We train all the architectures with an ADAM optimizer [17] with the learning rate as 0.005 and decaying factor as 0.9, 0.999 to minimize the entropy loss.
Revised: We train all the architectures with an ADAM optimizer [17] with the learning rate as 0.005 and decaying factor as 0.9, 0.999 to minimize the entropy loss.

Original: Thetraining model set is divided into batches with 10 models over 40 epochs.
Revised: The training model set is divided into batches with 10 models over 40 epochs.

Original: We repeat 5 samplingrealizations for all the architectures and calculate the average classification error rates as well as thestandard deviation.
Revised: We repeat 5 sampling realizations for all the architectures and calculate the average classification error rates as well as the standard deviation.

Original: The classification error rates for n = 300 are shown in Table 1.
Revised: The classification error rates for n = 300 are shown in Table 1.

Original: We observe that GNNs perform betterthan the GFs while architectures with more layers learn more accurate models with more parameterslearned.
Revised: We observe that GNNs perform better than the GFs while architectures with more layers learn more accurate models with more parameters learned.

Original: The averaged error rates for different architectures are shown in Figure 2 for increasingvalue of number of sampling points n .
Revised: The averaged error rates for different architectures are shown in Figure 2 for increasing value of number of sampling points n .

Original: We observe that, for all architectures the error rates decreasewith n .
Revised: We observe that, for all architectures the error rates decrease with n .

Original: This indicates that GNN constructed on L n converges as the graphs G n grow, as expectedfrom Theorem 1.
Revised: This indicates that GNN constructed on L n converges as the graphs G n grow, as expected from Theorem 1.

Original:  Conclusions
Revised:  Conclusions

Original: We have defined a manifold convolution operation with the heat diffusion controlled by the LaplaceBeltrami operator.
Revised: We have defined a manifold convolution operation with the heat diffusion controlled by the LaplaceBeltrami operator.

Original: We have further constructed a manifold neural network architecture by conscadingthe manifold filters and nonlinearities.
Revised: We have further constructed a manifold neural network architecture by conscading the manifold filters and nonlinearities.

Original: To realize the MNN in practice, we have carried out discretization in both space and time domains which recovers the convolution and neural networks ongraphs.
Revised: To realize the MNN in practice, we have carried out discretization in both space and time domains which recovers the convolution and neural networks on graphs.

Original: We have proved the convergence of neural networks on sampled graphs to the MNN when theinput signal is bandlimited.
Revised: We have proved the convergence of neural networks on sampled graphs to the MNN when the input signal is bandlimited.

Original: We have connected GNNs with MNNs by showing that the sequence of
Revised: We have connected GNNs with MNNs by showing that the sequence of

Original: GNNs sampled from the manifold converge to MNN and the discretization of MNN brings back theparticularization on GNNs.
Revised: GNNs sampled from the manifold converge to MNN and the discretization of MNN brings back the particularization on GNNs.

Original: We finally verified the performance of MNN with a model classificationproblem.
Revised: We finally verified the performance of MNN with a model classification problem.

Original: A Supplementary material 
Revised: A Supplementary material A.1 Proof of Theorem

Original: A.1 Proof of Theorem 1

Original: For ease of presentation, we denote the norm ∥ · ∥ L 2 ( G n ) as ∥ · ∥ for short.
Revised: For ease of presentation, we denote the norm ∥ · ∥ L 2 ( G n ) as ∥ · ∥ for short.

Original: We first import the existing results from [2] which indicate the spectral convergence of the constructedgraph Laplacian operator to the LB operator of the underlying manifold.
Revised: We first import the existing results from [2] which indicate the spectral convergence of the constructed graph Laplacian operator to the LB operator of the underlying manifold.

Original: Theorem 2 (Theorem 2.1 [2]) Let X
Revised: Theorem 2 (Theorem 2.1 [2]) Let X

Original: = { x 1 , x 2 , ...x n } be a set of n points sampled i.i.d.
Revised: = { x 1 , x 2 , ...x n } be a set of n points sampled i.i.d.

Original: from ad -dimensional manifold M ⊂ R N .
Revised: from a d -dimensional manifold M ⊂ R N .

Original: Let G n be a graph approximation of M constructed from Xwith weight values set as (15) with t n = n − 1 / ( d +2+ α ) and α > 0 .
Revised: Let G n be a graph approximation of M constructed from X with weight values set as (15) with t n = n − 1 / ( d +2+ α ) and α > 0 .

Original: Let L n be the graph Laplacian of
Revised: Let L n be the graph Laplacian of

Original: G n and L be the Laplace-Beltrami operator of M .
Revised: G n and L be the Laplace-Beltrami operator of M .

Original: Let λ ni be the i -th eigenvalue of L n and ϕ ni bethe corresponding normalized eigenfunction.
Revised: Let λ ni be the i -th eigenvalue of L n and ϕ ni be the corresponding normalized eigenfunction.

Original: Let λ i and ϕ
Revised: Let λ i and ϕ

Original: i be the corresponding eigenvalue andeigenfunction of L respectively.
Revised: i be the corresponding eigenvalue and eigenfunction of L respectively.

Original: Then, it holds that
Revised: Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: where the limits are taken in probability.
Revised: where the limits are taken in probability.

Original: From the definitions of neural networks on the constructed graph G n and manifold M respectively,the output difference can be written as
Revised: From the definitions of neural networks on the constructed graph G n and manifold M respectively, the output difference can be written as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: By inserting the filter definition, we have
Revised: By inserting the filter definition, we have

Original: [Equation]
Revised: [Equation]

Original: with x n, 0 =
Revised: with x n, 0 =

Original: P n f as the input of the first layer.
Revised: P n f as the input of the first layer.

Original: Since the point-wise nonlinearity is normalized
Revised: Since the point-wise nonlinearity is normalized

Original: Lipschitz according to Assumption 1, we have
Revised: Lipschitz according to Assumption 1, we have

Original: [Equation]
Revised: [Equation]

Original: The difference can be further decomposed as
Revised: The difference can be further decomposed as

Original: [Equation]
Revised: [Equation]

Original: The first term can be bounded as ∥ x qn,l − 1 − P n f ql − 1 ∥ with the initial condition ∥ x n, 0 − P n f 0 ∥ = 0 .{+
+}
Revised: The first term can be bounded as ∥ x [Equation] [Equation] − 1 ∥ with the initial condition ∥ x n, 0 − P n f {+
+}

Revised: ∥

Original:  Let us denote the second term as D nl − 1 .
Revised: = 0 . Let us denote the second term as D nl − 1 .

Original: By induction, we have
Revised: By induction, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can focus on the difference term D nl , so we omit the feature and layer index to workon a general form.
Revised: Therefore [Equation] we can focus on the difference term D nl , so we omit the feature and layer index to work on a general form.

Original: Considering that the input manifold signal f is λ M -bandlimited, we can write theconvolution operation as follows.
Revised: Considering that the input manifold signal f is λ M -bandlimited [Equation] we can write the convolution operation as follows.

Original: [Equation]
Revised: [Equation]

Original: The first term in (32) can be bounded by leveraging the C -Lipschitz continuity of the frequencyresponse.
Revised: The first term in (32) can be bounded by leveraging the C -Lipschitz continuity of the frequency response.

Original: From the convergence in probability stated in (24), we can claim that for each eigenvalueλ i ≤ λ M , for all ϵ i > 0 and all δ i > 0 , there exists some N i such that for all n > N i , we have
Revised: From the convergence in probability stated in (24), we can claim that for each eigenvalue λ i ≤ λ M , for all ϵ i > 0 and all δ i > 0 , there exists some N i such that for all n > N i , we have

Original: [Equation]
Revised: [Equation]

Original: Letting ϵ i <
Revised: Letting ϵ i <

Original: ϵ with ϵ > 0 , with probability at least (cid:81) Mi
Revised: ϵ with ϵ > 0 , with probability at least (cid:81) Mi

Original: =1 (1 − δ i ) :
Revised: =1 (1 − δ i ) :

Original: = 1 − δ , the first term is boundedas
Revised: = 1 − δ , the first term is bounded as

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation] (34) ≤ [Equation] (35) for all n > max i N i := N .
Revised: for all n > max i N i := N .

Original: The second term in (32) can be bounded combined with the convergence of eigenfunctions in (37) as(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: The second term in (32) can be bounded combined with the convergence of eigenfunctions in (37) as (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: M (cid:88) i =
Revised: M (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: i
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: From the convergence in probability stated in (24), we can claim that for some fixed eigenfunctionϕ i , for all ϵ i > 0 and all δ i > 0 , there exists some N i such that for all n > N i , we have
Revised: From the convergence in probability stated in (24) [Equation] we can claim that for some fixed eigenfunction ϕ i , for all ϵ i > 0 and all δ i > 0 , there exists some N i such that for all n > N i , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, letting ϵ i < ϵ with ϵ > 0 , with probability at least (cid:81) Mi
Revised: Therefore, letting ϵ i < ϵ with ϵ > 0 , with probability at least (cid:81) Mi

Original: =1 (1 − δ i ) :
Revised: =1 (1 − δ i ) :

Original: = 1 − δ , for alln > max i N i :=
Revised: = 1 − δ , for all n > max i N i :=

Original: N , the first term in (36) can be bounded as
Revised: N , the first term in (36) can be bounded as

Original: [Equation]
Revised: [Equation]

Original: considering that frequency response function is non-amplifying stated in Assumption 2.
Revised: considering that frequency response function is non-amplifying stated in Assumption 2.

Original: The lastequation comes from the definition of norm in L 2 ( G n ) .
Revised: The last equation comes from the definition of norm in L 2 ( G n ) .

Original: The second term in (36) can be written as
Revised: The second term in (36) can be written as

Original: [Equation]
Revised: [Equation]

Original: Because { x 1 , x 2 , · · · , x n } is a set of uniform sampled points from M , based on Theorem 19 in [32]we can claim that
Revised: Because { x 1 , x 2 , · · · , x n } is a set of uniform sampled points from M , based on Theorem 19 in [32] we can claim that

Original: [Equation]
Revised: [Equation]

Original: for all ϵ > 0 and δ > 0 .
Revised: for all ϵ > 0 and δ > 0 .

Original: Taking into consider the boundedness of frequency response | ˆ h ( λ ) | ≤ 1 andthe bounded energy ∥ P n ϕ i ∥ .
Revised: Taking into consider the boundedness of frequency response | ˆ h ( λ ) | ≤ 1 and the bounded energy ∥ P n ϕ i ∥ .

Original: Therefore, we have for all ϵ > 0 and δ > 0 ,
Revised: Therefore, we have for all ϵ > 0 and δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Combining all these results, we can claim that for all ϵ ′ > 0 and δ > 0 , there exists some N , suchthat for all n > N , we have
Revised: Combining all these results, we can claim that for all ϵ ′ > 0 and δ > 0 , there exists some N , such that for all n > N , we have

Original: [Equation]
Revised: [Equation]

Original: With lim n →∞ D nl = 0 in high probability, this concludes the proof.
Revised: With lim n →∞ D nl = 0 in high probability, this concludes the proof.