Original: Signal Processing for Implicit Neural Representationshttps://vita-group.github.io/INSP/
Revised: Signal Processing for Implicit Neural Representations

Original: Abstract
Revised: Abstract

Original: Implicit Neural Representations (INRs) encoding continuous multi-media data via multi-layer perceptrons has shown undebatable promise in various computer vision tasks.
Revised: Implicit Neural Representations (INRs) encoding continuous multi-media data via multi-layer perceptrons has shown undebatable promise in various computer vision tasks.

Original: Despite many successful applications, editing and processing an
Revised: Despite many successful applications, editing and processing an

Original: INR remains intractable as signals are represented by latent parameters of a neural network.
Revised: INR remains intractable as signals are represented by latent parameters of a neural network.

Original: Existing works manipulate such continuous representations via processing on their discretized instance, which breaks down the compactness and continuous nature of INR.
Revised: Existing works manipulate such continuous representations via processing on their discretized instance, which breaks down the compactness and continuous nature of INR.

Original: In this work, we present a pilot study on the question: how to directly modify an INR without explicit decoding?
Revised: In this work, we present a pilot study on the question: how to directly modify an INR without explicit decoding?

Original: We answer this question by proposing an implicit neural signal processing network, dubbed INSP-Net , via differential operators on INR.
Revised: We answer this question by proposing an implicit neural signal processing network, dubbed INSP-Net , via differential operators on INR.

Original: Our key insight is that spatial gradients of neural networks can be computed analytically and are invariant to translation, while mathematically we show that any continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators.
Revised: Our key insight is that spatial gradients of neural networks can be computed analytically and are invariant to translation, while mathematically we show that any continuous convolution ﬁlter can be uniformly approximated by a linear combination of high-order differential operators.

Original: With these two knobs,
Revised: With these two knobs,

Original: INSP-Net instantiates the signal processing operator as a weighted composition of computational graphs corresponding to the high-order derivatives of INRs, where the weighting parameters can be data-driven learned.
Revised: INSP-Net instantiates the signal processing operator as a weighted composition of computational graphs corresponding to the high-order derivatives of INRs, where the weighting parameters can be data-driven learned.

Original: Based on our proposed INSPNet, we further build the first Convolutional Neural Network (CNN) that implicitly runs on INRs, named INSP-ConvNet .
Revised: Based on our proposed INSPNet, we further build the ﬁrst Convolutional Neural Network (CNN) that implicitly runs on INRs, named INSP-ConvNet .

Original: Our experiments validate the expressiveness of INSP-Net and INSP-ConvNet in fitting low-level image and geometry processing kernels (e.g. blurring, deblurring, denoising, inpainting, and smoothening) as well as for high-level tasks on implicit fields such as image classification.
Revised: Our experiments validate the expressiveness of INSP-Net and INSP-ConvNet in ﬁtting low-level image and geometry processing kernels (e.g. blurring, deblurring, denoising, inpainting, and smoothening) as well as for high-level tasks on implicit ﬁelds such as image classiﬁcation.

Original:  Introduction
Revised:  Introduction

Original: The idea that our visual world can be represented continuously has attracted increasing popularity in the field of implicit neural representations (INR).
Revised: The idea that our visual world can be represented continuously has attracted increasing popularity in the ﬁeld of implicit neural representations (INR).

Original: Also known as coordinate-based neural representations, INRs learn to encode a coordinate-to-value mapping for continuous multi-media data.
Revised: Also known as coordinate-based neural representations, INRs learn to encode a coordinate-to-value mapping for continuous multi-media data.

Original: Instead of storing the discrete signal values in a grid of pixels or voxels, INRs represent discrete data as samples of a continuous manifold.
Revised: Instead of storing the discrete signal values in a grid of pixels or voxels, INRs represent discrete data as samples of a continuous manifold.

Original: Using multi-layer perceptrons, INRs bring practical benefits to various computer vision applications, such as image and video compression [1, 2, 3], 3D shape representation [4, 5, 6, 7, 8, 9, 10, 11], inverse problems [12, 2, 13, 14], and generative models [15, 16, 17, 18, 19, 20, 21, 22].
Revised: Using multi-layer perceptrons, INRs bring practical beneﬁts to various computer vision applications, such as image and video compression [1, 2, 3], 3D shape representation [4, 5, 6, 7, 8, 9, 10, 11], inverse problems [12, 2, 13, 14], and generative models [15, 16, 17, 18, 19, 20, 21, 22].

Original: Despite their recent success, INRs are not yet amenable to flexible editing and processing as the standard images could do.
Revised: Despite their recent success, INRs are not yet amenable to ﬂexible editing and processing as the standard images could do.

Original: The encoded coordinate-to-value mapping is too complex to comprehend and the parameters stored in multi-layer perceptrons (MLPs) remains less explored.
Revised: The encoded coordinate-to-value mapping is too complex to comprehend and the parameters stored in multi-layer perceptrons (MLPs) remains less explored.

Original: One direction of existing approaches enables editing on INRs by training them with conditional input.
Revised: One direction of existing approaches enables editing on INRs by training them with conditional input.

Original: For example, 
Revised: For example, ⇤

Original: [Figure]
Revised: [Figure]

Original: [23, 24, 25, 20, 21, 26] utilize conditional codes to indicate different characteristics of the scene including shape and color.
Revised: [23, 24, 25, 20, 21, 26] utilize conditional codes to indicate different characteristics of the scene including shape and color.

Original: Another main direction benefits from existing image editing techniques and operates on discretized instances of continuous INRs such as pixels or voxels.
Revised: Another main direction beneﬁts from existing image editing techniques and operates on discretized instances of continuous INRs such as pixels or voxels.

Original: However, such solutions break down the continuous  of INR due to the prerequisite of decoding and discretizing before editing and processing.
Revised: However, such solutions break down the continuous characteristic of INR due to the prerequisite of decoding and discretizing before editing and processing.

Original: In this paper, we conduct the first pilot study on the question: how to generally modify an INR without explicit decoding?
Revised: In this paper, we conduct the ﬁrst pilot study on the question: how to generally modify an INR without explicit decoding?

Original: The major challenge is that one cannot directly interpret what the parameters in an
Revised: The major challenge is that one cannot directly interpret what the parameters in an

Original: INR stand for, not to mention editing them correctly.
Revised: INR stand for, not to mention editing them correctly.

Original: Our key motivation is that spatial gradients can be served as a favorable tool to tackle this problem as they can be computed analytically, and possess desirable invariant properties.
Revised: Our key motivation is that spatial gradients can be served as a favorable tool to tackle this problem as they can be computed analytically, and possess desirable invariant properties.

Original: Theoretically, we prove that any continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators.
Revised: Theoretically, we prove that any continuous convolution ﬁlter can be uniformly approximated by a linear combination of high-order differential operators.

Original: Based on the above two rationales, we propose an Implicit Neural Signal Processing Network, dubbed INSP-Net , which processes INR utilizing high-order differential operators.
Revised: Based on the above two rationales, we propose an Implicit Neural Signal Processing Network, dubbed INSP-Net , which processes INR utilizing high-order differential operators.

Original: The proposed INSP-Net is composed of an inception fusion block connecting computational graphs corresponding to derivatives of INRs.
Revised: The proposed INSP-Net is composed of an inception fusion block connecting computational graphs corresponding to derivatives of INRs.

Original: The weights in the branchy part are loaded from the INR being processed, while the weights in the fusion block are parameters of the operator, which can be either hand-crafted or learned by the data-driven algorithm.
Revised: The weights in the branchy part are loaded from the INR being processed, while the weights in the fusion block are parameters of the operator, which can be either hand-crafted or learned by the data-driven algorithm.

Original: Even though we are not able to perform surgery on neural network parameters, we can implicitly process them by retrofitting their architecture and reorganizing the spatial gradients.
Revised: Even though we are not able to perform surgery on neural network parameters, we can implicitly process them by retroﬁtting their architecture and reorganizing the spatial gradients.

Original: We further extend our framework to build the first Convolutional Neural Network (CNN) operating directly on INRs, dubbed INSP-ConvNet.
Revised: We further extend our framework to build the ﬁrst Convolutional Neural Network (CNN) operating directly on INRs, dubbed INSP-ConvNet.

Original: Each layer of INSP-ConvNet is constructed by linearly combining the derivative computational graphs of the former layers.
Revised: Each layer of INSP-ConvNet is constructed by linearly combining the derivative computational graphs of the former layers.

Original: Nonlinear activation and normalization are naturally supported as they are element-wise functions.
Revised: Nonlinear activation and normalization are naturally supported as they are element-wise functions.

Original: Data augmentation can be also implemented by augmenting the input coordinates of INRs.
Revised: Data augmentation can be also implemented by augmenting the input coordinates of INRs.

Original: Under this pipeline (shown in Fig. 1), we demonstrate the expressiveness of our INSP-Net framework in fitting low-level image processing kernels including edge detection, blurring, deblurring, denoising, and image inpainting.
Revised: Under this pipeline (shown in Fig. 1), we demonstrate the expressiveness of our INSP-Net framework in ﬁtting low-level image processing kernels including edge detection, blurring, deblurring, denoising, and image inpainting.

Original: We also successfully apply our INSP-ConvNet to high-level tasks on implicit fields such as classification.
Revised: We also successfully apply our INSP-ConvNet to high-level tasks on implicit ﬁelds such as classiﬁcation.

Original: Our main contributions can be summarized as follows:
Revised: Our main contributions can be summarized as follows:

Original: • We propose a novel signal processing framework, dubbed INSP-Net , that operates on INRs analytically and continuously by closed-form high-order differential operators 2 .
Revised: • We propose a novel signal processing framework, dubbed INSP-Net , that operates on INRs analytically and continuously by closed-form high-order differential operators 2 .

Original: Repeatedly cascading the computational paradigm of INSP-Net, we also build a convolutional network, called INSP-ConvNet, which directly runs on implicit fields for high-level tasks.
Revised: Repeatedly cascading the computational paradigm of INSP-Net, we also build a convolutional network, called INSP-ConvNet, which directly runs on implicit ﬁelds for high-level tasks.

Original: • We illustrate the advantage of adopting differential operators by revealing their inherent group invariance.
Revised: • We illustrate the advantage of adopting differential operators by revealing their inherent group invariance.

Original: Furthermore, we rigorously prove that the convolution operator in the continuous regime can be uniformly approximated by a linear combination of the gradients.
Revised: Furthermore, we rigorously prove that the convolution operator in the continuous regime can be uniformly approximated by a linear combination of the gradients.

Original: • Extensive experiments demonstrate the effectiveness of our approach in both low-level processing (e.g. edge detection, blurring, deblurring, denoising, image inpainting, and smoothening) and high-level processing such as image classification.
Revised: • Extensive experiments demonstrate the effectiveness of our approach in both low-level processing (e g. edge detection, blurring, deblurring, denoising, image inpainting, and smoothening) and high-level processing such as image classiﬁcation.

Original: [Figure]
Revised: [Figure]

Revised: Concat

Original: [Figure]
Revised: [Figure]

Original: Inception Fusion Layer
Revised: Inception [Figure] Layer

Original: MLP
Revised: MLP

Original: NL x
Revised: NL x

Original: NL
Revised: NL

Original: NL'
Revised: NL'

Original: NL x
Revised: NL x

Original: NL'
Revised: NL'

Original: NL' x NL x
Revised: NL' x NL x

Original: NL NL
Revised: NL NL

Original: Original INR Network
Revised: Original INR Network

Original: First-order Derivative Network
Revised: First-order Derivative Network

Original:  Preliminaries: Implicit Neural Representation
Revised:  Preliminaries: Implicit Neural Representation

Original: Implicit Neural Representation (INR) parameterizes continuous multi-media signals or vector fields with neural networks.
Revised: Implicit Neural Representation (INR) parameterizes continuous multi-media signals or vector ﬁelds with neural networks.

Original: Formally, we consider an INR as a continuous function Φ : R m → R that maps low-dimension spatial/temporal coordinates to the value space 3 .
Revised: Formally, we consider an INR as a continuous function  : R m ! R that maps low-dimension spatial/temporal coordinates to the value space 3 .

Original: For example, to represent 2D image signals, the domain of Φ is ( x, y ) spatial coordinates, and the range of Φ are the pixel intensities.
Revised: For example, to represent 2D image signals, the domain of  is ( x, y ) spatial coordinates, and the range of  are the pixel intensities.

Original: The typical use of INR is to solve a feasibility problem where Φ is sought to satisfy a set of N constraints {C (Φ , a j | Ω j ) } Nj =1 , where C is a functional that relates function Φ to some observable quantities a j evaluating over a measurable domain Ω j ⊆ R m .
Revised: The typical use of INR is to solve a feasibility problem where  is sought to satisfy a set of N constraints {C (  , a j | ⌦ j ) } Nj =1 , where C is a functional that relates function  to some observable quantities a j evaluating over a measurable domain ⌦ j ✓ R m .

Original: This problem can be cast into an optimization problem that minimizes deviations from each of the constraints:
Revised: This problem can be cast into an optimization problem that minimizes deviations from each of the constraints:

Original: [Equation][Equation]
Revised: [Equation]

Original: For instance, we can let C =
Revised: For instance, we can let C =

Original: Φ( x j ) − a j with Ω j = { x j } , then our objective boils down to a point-to-point supervision which memorizes a signal into Φ [27].
Revised:  x j )  a j with ⌦ j = { x j } , then our objective boils down to a point-to-point supervision which memorizes a signal into  [27].

Original: When functional C is a combination of differential operators taking values in a point set, i.e., C ( a ( x ) , Φ( x ) , ∇ Φ( x ) , · · · ) , ∀ x ∈ Ω j , Eq.
Revised: When functional C is a combination of differential operators taking values in a point set, i.e., C ( a ( x ) ,  ( x ) , r  ( x ) , · · · ) , 8 x 2 ⌦ j , Eq.

Original: 1 is objective to solving a bunch of differential equations [28, 7, 29].
Revised: 1 is objective to solving a bunch of differential equations [28, 7, 29].

Original: Note that in this paper, without particular specification, the gradients are all computed with respect to the input coordinate x .
Revised: Note that in this paper, without particular speciﬁcation, the gradients are all computed with respect to the input coordinate x .

Original: C can also form an integral equation system over some intervals Ω j [12].
Revised: C can also form an integral equation system over some intervals ⌦ j [12].

Original: In practice of computer vision, we reconstruct a signal by capturing sparse observations D = { (Ω j , a j ) } Nj =1 from unknown function Φ , and dynamically sampling a mini-batch from D to minimize Eq.
Revised: In practice of computer vision, we reconstruct a signal by capturing sparse observations D = { ( ⌦ j , a j ) } Nj =1 from unknown function  , and dynamically sampling a mini-batch from D to minimize Eq.

Original: 1 to obtain a feasible Φ .
Revised: 1 to obtain a feasible  .

Original: A handy parameterization of function Φ is a fully-connected neural network, which enables solving
Revised: A handy parameterization of function  is a fully-connected neural network, which enables solving

Original: Eq. 1 via gradient descent through a differentiable C .
Revised: Eq. 1 via gradient descent through a differentiable C .

Original: Common INR networks consist of pure
Revised: Common INR networks consist of pure

Original: Multi-Layer Perceptrons (MLP) with periodic activation functions.
Revised: Multi-Layer Perceptrons (MLP) with periodic activation functions.

Original: Fourier Feature Mapping (FFM) [27] places a sinusoidal transformation before the MLP, while Sinusoidal Representation Network (SIREN)
Revised: Fourier Feature Mapping (FFM) [27] places a sinusoidal transformation before the MLP, while Sinusoidal Representation Network (SIREN)

Original: [28] replaces every piece-wise linear activation with a sinusoidal function.
Revised: [28] replaces every piece-wise linear activation with a sinusoidal function.

Original: Below we give a unified formulation of INR networks:
Revised: Below we give a uniﬁed formulation of INR networks:

Original: [Equation]
Revised: [Equation]

Original: where W i ∈ R

Original: d i − 1 ×

Original:  d i , b i ∈ R d i are the weight matrix and bias of the i -th layer, respectively, n is the number of layers, and σ i ( · ) is an element-wise nonlinear activation function.
Revised: where W i 2 R d i  1 ⇥ d i , b i 2 R d i are the weight matrix and bias of the i -th layer, respectively, n is the number of layers, and  i ( · ) is an element-wise nonlinear activation function.

Original: For FFM architecture, σ i = sin( · ) when i = 1 denotes the positional encoding layer [12, 30] and otherwise σ i = ReLU( · ) .
Revised: For FFM architecture,  i = sin( · ) when i = 1 denotes the positional encoding layer [12, 30] and otherwise  i = ReLU( · ) .

Original: For SIREN, σ i = sin( · ) for every layer i = 1 , · · · , n − 1 .
Revised: For SIREN,  i = sin( · ) for every layer i = 1 , · · · , n  1 .

Original:  Implicit Representation Processing via Differential Operators
Revised:  Implicit Representation Processing via Differential Operators

Original: Digital Signal Processing (DSP) techniques have been widely applied in computer vision tasks, such as image restoration [31], signal enhancement [32] and geometric processing [33].
Revised: Digital Signal Processing (DSP) techniques have been widely applied in computer vision tasks, such as image restoration [31], signal enhancement [32] and geometric processing [33].

Original: Even modern deep learning models are consisting of the most basic signal processing operators.
Revised: Even modern deep learning models are consisting of the most basic signal processing operators.

Original: Suppose we already acquire an Implicit Neural Representation (INR) 
Revised: Suppose we already acquire an Implicit Neural Representation (INR)  : R m !

Original: Φ : R m → R , now we are interested in whether we can run a signal processing program on the implicitly represented signals.
Revised: R , now we are interested in whether we can run a signal processing program on the implicitly represented signals.

Original: One straightforward solution is to rasterize the implicit field with a 2D/3D lattice and run a typical kernel on the pixel/voxel grids.
Revised: One straightforward solution is to rasterize the implicit ﬁeld with a 2D/3D lattice and run a typical kernel on the pixel/voxel grids.

Original: However, this decoding strategy produces a finite resolution and discretizes signals, which is memory inefficient and unfriendly to modeling fine details.
Revised: However, this decoding strategy produces a ﬁnite resolution and discretizes signals, which is memory inefﬁcient and unfriendly to modeling ﬁne details.

Original: In this section, we introduce a computation paradigm that can process an INR analytically with spatial/temporal derivatives.
Revised: In this section, we introduce a computation paradigm that can process an INR analytically with spatial/temporal derivatives.

Original: We show that our proposed method serves as a universal operator that can represent any continuous convolutional kernels.
Revised: We show that our proposed method serves as a universal operator that can represent any continuous convolutional kernels.

Original: 3.1 Computational Paradigm
Revised: 3.1 Computational Paradigm

Original: It has not escaped our notice that spatial/temporal gradients on INRs ∇ k Φ can be computed analytically due to the differentiable characteristics of neural networks.
Revised: It has not escaped our notice that spatial/temporal gradients on INRs r k  can be computed analytically due to the differentiable characteristics of neural networks.

Original: Inspired by this, we propose an
Revised: Inspired by this, we propose an

Original: Implicit Neural Signal Processing (INSP) framework that composes a class of closed-form operators for INRs using functional combinations of high-order derivatives.
Revised: Implicit Neural Signal Processing (INSP) framework that composes a class of closed-form operators for INRs using functional combinations of high-order derivatives.

Original: We denote our proposed signal processing operator by A built upon high-order derivatives.
Revised: We denote our proposed signal processing operator by A built upon high-order derivatives.

Original: Given an acquired INR Φ , we denote the resultant INR processed by operator A asΨ =
Revised: Given an acquired INR  , we denote the resultant INR processed by operator A as =

Original: A Φ : R m → R .
Revised: A  : R m ! R .

Original: To evaluate point x ∈ R m of processed INR, we propose the following computational paradigm:
Revised: To evaluate point x 2 R m of processed INR, we propose the following computational paradigm:

Original: [Equation]
Revised: [Equation]

Original: (3) where Π : R M → R can be arbitrary continuous functions, which can be either handcrafted or learned from data.
Revised: where ⇧ : R M ! R can be arbitrary continuous functions, which can be either handcrafted or learned from data.

Original: To learn an operator A from data, we represent Π by Multi-Layer Perceptrons (MLP) with parameters θ .
Revised: To learn an operator A from data, we represent ⇧ by Multi-Layer Perceptrons (MLP) with parameters ✓ .

Original: Here we can slightly abuse the notation of ∇ k to be a flattened vector of high-order derivatives without multiplicity since differential operators defined over continuous functions form a commutative ring.
Revised: Here we can slightly abuse the notation of r k to be a ﬂattened vector of high-order derivatives without multiplicity since differential operators deﬁned over continuous functions form a commutative ring.

Original: The input dimension of Π depends on the highest order of used derivatives.
Revised: The input dimension of ⇧ depends on the highest order of used derivatives.

Original: Suppose we compute derivatives up to K -th order, then M =(cid:80) Kk =0 (cid:0) k
Revised: Suppose we compute derivatives up to K -th order, then M =

Original:  + m − 1 k (cid:1) = 
Revised: P Kk =  [Equation] + m  1 k  = ( K +1)

Original: ( K +1) (cid:0) K + m K [-+
-]
Revised: K + m K [-+
-]+  /m , where  k + m

Original: (cid:1)
Revised: [Equation]

Original: /m , where (cid:0) k + m − 1 k (cid:1) is the number of distinctive k -th order differential operators 4 .
Revised:  is the number of distinctive k -th order differential operators 4 .

Original: Intuitively, directional derivatives encode (local) neighboring information, which can have similar effects of a convolution.
Revised: Intuitively, directional derivatives encode (local) neighboring information, which can have similar effects of a convolution.

Original: As we will show in Sec. 3.2, Π can construct both shift-invariant and rotation-invariant operators, which introduces favorable inductive bias to images and 3D geometry processing.
Revised: As we will show in Sec. 3.2, ⇧ can construct both shift-invariant and rotation-invariant operators, which introduces favorable inductive bias to images and 3D geometry processing.

Original: More importantly, we rigorously prove that Eq.
Revised: More importantly, we rigorously prove that Eq.

Original: 3 is also a universal approximator of arbitrary convolutional operators.
Revised: 3 is also a universal approximator of arbitrary convolutional operators.

Original: We note that Ψ( x ) as a whole can also be regarded as a neural network.
Revised: We note that ( x ) as a whole can also be regarded as a neural network.

Original: Recall the architecture of Φ( x ) in Eq.
Revised: Recall the architecture of  ( x ) in Eq.

Original: 2, its k -th order derivative is another computational graph parameterized by W i and b i that maps x to ∇ k Φ( x ) .
Revised: 2, its k -th order derivative is another computational graph parameterized by W i and b i that maps x to r k  ( x ) .

Original: For example, the first-order gradient will have the following form:
Revised: For example, the ﬁrst-order gradient will have the following form:

Original: [Equation]
Revised: [Equation]

Original: where ˆ ϕ i ( y ) =
Revised: where ˆ  i ( y ) =

Original: W ⊤ i σ ′ i − 1 ( W i − 1 y + b k − 1 ) , and σ ′{+ 
+} i ( · ) is the first-order derivative of σ i ( · ) .
Revised: W > i  0 i  1 ( W i  1 y + b k  1 ) , and {+ 
+} i ( · ) is the ﬁrst-order derivative of  i ( · ) .

Original: Since ˆ ϕ i shares the weights with ϕ i , ∇ Φ is represented by a closed-form computational network re-using the weights from Φ , which we refer to as the first-order derivative network .
Revised: Since ˆ  i shares the weights with  i , r  is represented by a closed-form computational network re-using the weights from  , which we refer to as the ﬁrst-order derivative network .

Original: The higher-order derivatives should induce the derivative network of similar forms.
Revised: The higher-order derivatives should induce the derivative network of similar forms.

Original: Therefore, the processed INRΨ will have an
Revised: Therefore, the processed INR will have an

Original: Inception-like architecture, namely, a multi-branch structure connecting the original INR network and weight-sharing derivative subnetworks followed by a fusion layer Π .
Revised: Inception-like architecture, namely, a multi-branch structure connecting the original INR network and weight-sharing derivative subnetworks followed by a fusion layer ⇧ .

Original: We call the entire model (Ψ =
Revised: We call the entire model ( =

Original: A Φ or Eq. 3) an Implicit Neural Signal Processing Network or an INSP-Net .
Revised: A  or Eq. 3) an Implicit Neural Signal Processing Network or an INSP-Net .

Original: Note that the only parameters of INSP-Net θ are located at the last fusion layer, and can be trained in an end-to-end manner.
Revised: Note that the only parameters of INSP-Net ✓ are located at the last fusion layer, and can be trained in an end-to-end manner.

Original: We illustrate an INSP-Net in Fig.
Revised: We illustrate an INSP-Net in Fig.

Original: 2 where the color indicates the weight-sharing scheme.
Revised: 2 where the color indicates the weight-sharing scheme.

Original: A similar weight-sharing scheme is also adopted in AutoInt [34].
Revised: A similar weight-sharing scheme is also adopted in AutoInt [34].

Original: In practice, we employ auto-differentiation in PyTorch [35] to automatically create such derivatives networks and reassemble them parallelly to constitute the architecture of an INSP-Net.
Revised: In practice, we employ auto-differentiation in PyTorch [35] to automatically create such derivatives networks and reassemble them parallelly to constitute the architecture of an INSP-Net.

Original: When inputting an INR, we load the weights of the
Revised: When inputting an INR, we load the weights of the

Original: INR to our model following the weight-sharing scheme, and then we obtain an INSP-Net, which implicitly and continuously represents the processed INR Ψ( x ) .
Revised: INR to our model following the weight-sharing scheme, and then we obtain an INSP-Net, which implicitly and continuously represents the processed INR ( x ) .

Original: To effectively express high-order derivatives, we choose SIREN as the base model [28].
Revised: To effectively express high-order derivatives, we choose SIREN as the base model [28].

Original: 3.2 Theoretical Analysis
Revised: 3.2 Theoretical Analysis

Original: In this section, we provide a theoretical justification for the design of our INSP-Net.
Revised: In this section, we provide a theoretical justiﬁcation for the design of our INSP-Net.

Original: We will focus on discussing the latent invariance property and the expressive power of INSP-Net.
Revised: We will focus on discussing the latent invariance property and the expressive power of INSP-Net.

Original: Translation and Rotation Invariance.
Revised: Translation and Rotation Invariance.

Original: Group invariance has been shown to be a favorable inductive bias for image [36], video [37], and geometry processing [38].
Revised: Group invariance has been shown to be a favorable inductive bias for image [36], video [37], and geometry processing [38].

Original: It has also been well-known that group invariance is an intrinsic property of Partial Differential Equations (PDEs) [39, 40].
Revised: It has also been well-known that group invariance is an intrinsic property of Partial Differential Equations (PDEs) [39, 40].

Original: Since our INSP-Net is built using differential operators, we are motivated to reveal its hidden invariance property to demonstrate its advantage in processing visual signals.
Revised: Since our INSP-Net is built using differential operators, we are motivated to reveal its hidden invariance property to demonstrate its advantage in processing visual signals.

Original: In this section, we only consider two transformation groups: translation group T ( m ) and the special orthogonal group SO ( m ) (a.k.a.
Revised: In this section, we only consider two transformation groups: translation group T ( m ) and the special orthogonal group SO ( m ) (a.k.a.

Original: rotation group).
Revised: rotation group).

Original: Elements T v ∈ T ( m ) in translation group shift the function Φ by some offset v ∈ R m .
Revised: Elements T v 2 T ( m ) in translation group shift the function  by some offset v 2 R m .

Original: The shifted function can be denoted as Φ ◦ T v ( x ) =
Revised: The shifted function can be denoted as   T v ( x ) =

Original: Φ( x + v ) .
Revised:  ( x + v ) .

Original: Similarly, elements in rotation group perform a coordinate transformation on function Φ by a rotation matrix R ∈ SO ( m ) .
Revised: Similarly, elements in rotation group perform a coordinate transformation on function  by a rotation matrix R 2 SO ( m ) .

Original: The transformed function can be written as Φ ◦ R ( x ) =
Revised: The transformed function can be written as   R ( x ) =

Original: Φ( Rx ) .
Revised:  ( Rx ) .

Original: Group invariance means deforming the input space of a function first and then processing it via an operator is equivalent to directly applying the transformation to the processed function.
Revised: Group invariance means deforming the input space of a function ﬁrst and then processing it via an operator is equivalent to directly applying the transformation to the processed function.

Original: For a more rigorous argument, A is said to be translation-invariant if ∀ T v ∈ T ( m ) ,Ψ( x + v ) =
Revised: For a more rigorous argument, A is said to be translation-invariant if 8 T v 2 T ( m ) ,

Original: A [Φ ◦ T v ]( x ) .
Revised: [Equation] [Equation]

Original: Likewise,
Revised: Likewise,

Original: A is rotation-invariant if ∀ R ∈ SO ( m ) we have Ψ( Rx ) =
Revised: A is rotation-invariant if 8 R 2 SO ( m ) we have ( Rx ) =

Original: A [Φ ◦ R ]( x ) .
Revised: A [   R ]( x ) .

Original: Below we provide Theorem 1 to characterize the invariance property for our model.
Revised: Below we provide Theorem 1 to characterize the invariance property for our model.

Original: Theorem 1.
Revised: Theorem 1.

Original: Given function Π : R M → R , the composed operator A (Eq. 3) can satisfy:
Revised: Given function ⇧ : R M ! R , the composed operator A (Eq. 3) can satisfy:

Original: shift invariance for every Π . 2.
Revised: shift invariance for every ⇧ . 2.

Original: rotation invariance if Π has the form: Π( y ) =
Revised: rotation invariance if ⇧ has the form: ⇧ ( y ) =

Original: f ( ∥ y ∥ 2 ) for some f : R → R .
Revised: f ( k y k 2 ) for some f : R ! R .

Original: We prove Theorem 1 in Appendix A. Our Theorem 1 implies that operator
Revised: We prove Theorem 1 in Appendix A. Our Theorem 1 implies that operator

Original: A is inherently shiftinvariant.
Revised: A is inherently shiftinvariant.

Original: This is due to the shift-invariant intrinsics of differential operators as we show in the proof.
Revised: This is due to the shift-invariant intrinsics of differential operators as we show in the proof.

Original: Rotation invariance is not guaranteed in general.
Revised: Rotation invariance is not guaranteed in general.

Original: However, if one carefully designs Π , it can also be achieved via our framework.
Revised: However, if one carefully designs ⇧ , it can also be achieved via our framework.

Original: Moreover, we also suggest a feasible solution to constructing a rotation-invariant operator A in Theorem 1.
Revised: Moreover, we also suggest a feasible solution to constructing a rotation-invariant operator A in Theorem 1.

Original: In our construction, Π first isotropically pools over the squares of all directional derivatives, and then maps the summarized information through another scalar function f .
Revised: In our construction, ⇧ ﬁrst isotropically pools over the squares of all directional derivatives, and then maps the summarized information through another scalar function f .

Original: We refer interested readers to [39] for more group invariance in differential forms.
Revised: We refer interested readers to [39] for more group invariance in differential forms.

Original: Universal Approximation.
Revised: Universal Approximation.

Original: Convolution, formally known as the linear shift-invariant operator, has served as one of the most prevalent signal processing tools in the vision domain.
Revised: Convolution, formally known as the linear shift-invariant operator, has served as one of the most prevalent signal processing tools in the vision domain.

Original: Given two (real-valued) signals f and g , we denote their convolution as g ⋆ f = f 
Revised: Given two (real-valued) signals f and g , we denote their convolution as g ? f = f ?

Original: ⋆ g .
Revised: g .

Original: In this section, we examine the expressiveness of our INSP-Net (Eq. 3) by showing it can represent any convolutional filter.
Revised: In this section, we examine the expressiveness of our INSP-Net (Eq. 3) by showing it can represent any convolutional ﬁlter.

Original: To draw this conclusion, we first present an informal version of our main results as follows:
Revised: To draw this conclusion, we ﬁrst present an informal version of our main results as follows:

Original: Theorem 2.
Revised: Theorem 2.

Original: (Informal statement)
Revised: (Informal statement)

Original: For every real-valued function g : R m → R , there exists a polynomial p ( x 1 , · · · , x m ) with real coefficients such that p ( ∇ ) f can uniformly approximate g ⋆ f by arbitrary precision for all real-valued signals f .
Revised: For every real-valued function g : R m ! R , there exists a polynomial p ( x 1 , · · · , x m ) with real coefﬁcients such that p ( r ) f can uniformly approximate g ? f by arbitrary precision for all real-valued signals f .

Original: The formal statement and proof can be found in Appendix B. Theorem 2 involves the notion of polynomials in partial differential operators (see more details in Appendix B).
Revised: The formal statement and proof can be found in Appendix B. Theorem 2 involves the notion of polynomials in partial differential operators (see more details in Appendix B).

Original: p ( ∇ ) f in turn can be written as a linear combination of high-order derivatives of f (a special case of Eq.
Revised: p ( r ) f in turn can be written as a linear combination of high-order derivatives of f (a special case of Eq.

Original: 3 when Π is linear).
Revised: 3 when ⇧ is linear).

Original: The key step to prove Theorem 2 is applying Stone-Weierstrass approximation theorem on the Fourier domain.
Revised: The key step to prove Theorem 2 is applying Stone-Weierstrass approximation theorem on the Fourier domain.

Original: However, we note that functions obtained by the Fourier transform are generally complex functions.
Revised: However, we note that functions obtained by the Fourier transform are generally complex functions.

Original: The prominence of our proof is that we can constrain the range of the polynomial coefficients into the real domain, which makes it implementable via a common deep learning infrastructure.
Revised: The prominence of our proof is that we can constrain the range of the polynomial coefﬁcients into the real domain, which makes it implementable via a common deep learning infrastructure.

Original: The implication of Theorem 2 is that the mapping between convolution and derivative is as simple as a linear transformation.
Revised: The implication of Theorem 2 is that the mapping between convolution and derivative is as simple as a linear transformation.

Original: Recent works [41, 42, 43] show the converse argument that derivatives can be approximated via a linear combination of discrete convolution.
Revised: Recent works [41, 42, 43] show the converse argument that derivatives can be approximated via a linear combination of discrete convolution.

Original: Theoremestablishes the equivalence between differential operator and convolution in the continuous regime.
Revised: Theoremestablishes the equivalence between differential operator and convolution in the continuous regime.

Original: In our proof, k -th order derivatives correspond to k -th order monomial in the spectral domain.
Revised: In our proof, k -th order derivatives correspond to k -th order monomial in the spectral domain.

Original: Fitting convolution using derivatives amounts to approximating spectrum via polynomials.
Revised: Fitting convolution using derivatives amounts to approximating spectrum via polynomials.

Original: This implies higher degree of polynomial induces closer approximation.
Revised: This implies higher degree of polynomial induces closer approximation.

Original: Since p ( ∇ ) is not difficult to be approximated by a neural network Π θ , we can easily derive the next result Corollary 3.
Revised: Since p ( r ) is not difﬁcult to be approximated by a neural network ⇧ ✓ , we can easily derive the next result Corollary 3.

Original: Corollary 3.
Revised: Corollary 3.

Original: For every real-valued function g , there exists a neural network Π θ such thatΨ =
Revised: For every real-valued function g , there exists a neural network ⇧ ✓ such that =

Original: A Φ (Eq. 3) can uniformly approximate g ⋆ Φ by arbitrary precision for every real-valued signals Φ .
Revised: A  (Eq. 3) can uniformly approximate g ?  by arbitrary precision for every real-valued signals  .

Original: As we discussed in Theorem 1, A are constantly shift-invariant.
Revised: As we discussed in Theorem 1, A are constantly shift-invariant.

Original: This means when approximating a convolutional kernel, the trajectory of A is restricted into the shift-invariant space.
Revised: This means when approximating a convolutional kernel, the trajectory of A is restricted into the shift-invariant space.

Original: Moreover, we emphasize that INSP-Net is far more expressive than convolutional kernels since Π θ can also fit any nonlinear continuous functions due to the universal approximation theorem [44, 45, 46].
Revised: Moreover, we emphasize that INSP-Net is far more expressive than convolutional kernels since ⇧ ✓ can also ﬁt any nonlinear continuous functions due to the universal approximation theorem [44, 45, 46].

Original: 3.3 Building CNNs for Implicit Neural Representations
Revised: 3.3 Building CNNs for Implicit Neural Representations

Original: Convolutional Neural Networks (CNN) are capable of extracting informative semantics by only piling up basic signal processing operators.
Revised: Convolutional Neural Networks (CNN) are capable of extracting informative semantics by only piling up basic signal processing operators.

Original: This motivates us to build CNNs based on INSP-Net that can directly run on INRs for high-level downstream tasks.
Revised: This motivates us to build CNNs based on INSP-Net that can directly run on INRs for high-level downstream tasks.

Original: In fact, to simulate exact convolution,
Revised: In fact, to simulate exact convolution,

Original: our
Revised: our

Original: Theorem 2 suggests simplify Π θ to a linear mapping.
Revised: Theorem 2 suggests simplify ⇧ ✓ to a linear mapping.

Original: Then our former computational paradigm Eq.
Revised: Then our former computational paradigm Eq.

Original: 3 is changed to:
Revised: 3 is changed to:

Original: [Equation]
Revised: [Equation]

Original: where θ k ∈ R ( k + m − 1 k  ) are parameters of the operator p ( ∇ ) .
Revised: where ✓ k [Equation] ) are parameters of the operator p ( r ) .

Original: We name this special case of Eq. 3 as
Revised: We name this special case of Eq. 3 as

Original: INSP-Conv.
Revised: INSP-Conv.

Original: One plausible implementation of INSP-Conv is to employ a one-layer MLP to represent Π θ .
Revised: One plausible implementation of INSP-Conv is to employ a one-layer MLP to represent ⇧ ✓ .

Original: When A = p ( ∇ ) , INSP-Conv preserves both linearity and shift-invariance when evolving during the training.
Revised: When A = p ( r ) , INSP-Conv preserves both linearity and shift-invariance when evolving during the training.

Original: We propose to repeatedly apply INSP-Conv with non-linearity to INRs that mimics a CNN-like architecture.
Revised: We propose to repeatedly apply INSP-Conv with non-linearity to INRs that mimics a CNN-like architecture.

Original: We name this class of CNNs composed by multi-layer INSP-Conv (Eq.
Revised: We name this class of CNNs composed by multi-layer INSP-Conv (Eq.

Original: 5) as INSP-ConvNet.
Revised: 5) as INSP-ConvNet.

Original: Previous works [47, 48] extracting semantic features from INR either lack local information by point-wisely mapping INR’s intermediate representation to a semantic space or explicitly rasterize INR into regular grids.
Revised: Previous works [47, 48] extracting semantic features from INR either lack local information by point-wisely mapping INR’s intermediate representation to a semantic space or explicitly rasterize INR into regular grids.

Original: To the best of our knowledge, it is the first time that one can run a CNN directly on an implicit representation thanks to closed-formness of INSP-Net.
Revised: To the best of our knowledge, it is the ﬁrst time that one can run a CNN directly on an implicit representation thanks to closed-formness of INSP-Net.

Original: The overall architecture of INSP-ConvNet can be formulated as:
Revised: The overall architecture of INSP-ConvNet can be formulated as:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where σ is an element-wise non-linear activation, L is the number of INSP-Net layers, and Φ is an input INR.
Revised: where  is an element-wise non-linear activation, L is the number of INSP-Net layers, and  is an input INR.

Original: We use the symbol · to denote operator functioning, and ◦ to denote function composition.
Revised: We use the symbol · to denote operator functioning, and  to denote function composition.

Original: Due to page limit, we defer detailed introduction to INSP-ConvNet to Appendix C.
Revised: Due to page limit, we defer detailed introduction to INSP-ConvNet to Appendix C.

Original:  Related Work 4.1 Implicit Neural Representation
Revised:  Related Work 4.1 Implicit Neural Representation

Original: Implicit Neural Representation (INR) represents signals by continuous functions parameterized by multi-layer perceptrons (MLPs) [28, 27], which is different from traditional discrete representations (e.g., pixel, mesh).
Revised: Implicit Neural Representation (INR) represents signals by continuous functions parameterized by multi-layer perceptrons (MLPs) [28, 27], which is different from traditional discrete representations (e.g., pixel, mesh).

Original: Compared with other representations, the continuous implicit representations are capable of representing signals at infinite resolution and have become prevailing to be applied upon image fitting [28], image compression [1, 49] and video compressing [3].
Revised: Compared with other representations, the continuous implicit representations are capable of representing signals at inﬁnite resolution and have become prevailing to be applied upon image ﬁtting [28], image compression [1, 49] and video compressing [3].

Original: In addition, INR has been applied to more efficient and effective shape representation [4, 5, 6, 7, 8, 9, 10, 11], texture mapping [50, 51], inverse problems [12, 2, 13, 14] and generative models [15, 16, 17, 18, 19, 20, 21, 22].
Revised: In addition, INR has been applied to more efﬁcient and effective shape representation [4, 5, 6, 7, 8, 9, 10, 11], texture mapping [50, 51], inverse problems [12, 2, 13, 14] and generative models [15, 16, 17, 18, 19, 20, 21, 22].

Original: There are also efforts speeding up the fitting of INRs [52] and improving the representation efficiency [53].
Revised: There are also efforts speeding up the ﬁtting of INRs [52] and improving the representation efﬁciency [53].

Original: Nowadays, editing and manipulating multi-media objects gains increasing interest and demand [54].
Revised: Nowadays, editing and manipulating multi-media objects gains increasing interest and demand [54].

Original: Thus, signal processing on implicit neural representation is essentially an important task worth investigating.
Revised: Thus, signal processing on implicit neural representation is essentially an important task worth investigating.

Original: 4.2 Editable Implicit Fields
Revised: 4.2 Editable Implicit Fields

Original: Editing implicit fields has recently attracted much research interest.
Revised: Editing implicit ﬁelds has recently attracted much research interest.

Original: Several methods have been proposed to allow editing the reconstructed 3D scenes by rearranging the objects or manipulating the shape and appearance.
Revised: Several methods have been proposed to allow editing the reconstructed 3D scenes by rearranging the objects or manipulating the shape and appearance.

Original: One line of work alters the structure and color of objects by conditioning latent codes for different characteristics of the scene [25, 20, 21, 26].
Revised: One line of work alters the structure and color of objects by conditioning latent codes for different characteristics of the scene [25, 20, 21, 26].

Original: Another direction involves discretizing the continuous fields.
Revised: Another direction involves discretizing the continuous ﬁelds.

Original: By converting the implicit fields into pixels or voxels, traditional image and voxel editing techniques [55, 56] can be applied effortlessly.
Revised: By converting the implicit ﬁelds into pixels or voxels, traditional image and voxel editing techniques [55, 56] can be applied effortlessly.

Original: These approaches, however, are not capable of directly performing signal processing on continuous INRs.
Revised: These approaches, however, are not capable of directly performing signal processing on continuous INRs.

Original: Functa [54] can use a latent code to control implicit funtions.
Revised: Functa [54] can use a latent code to control implicit funtions.

Original: NID [57] represents neural fields as a linear combination of implicit functional basis, which enables editing by change of sparse coefficients.
Revised: NID [57] represents neural ﬁelds as a linear combination of implicit functional basis, which enables editing by change of sparse coefﬁcients.

Original: However, such editing scheme suffers from limited flexibility.
Revised: However, such editing scheme suffers from limited ﬂexibility.

Original: Recently, NFGP [58] proposes to use neural fields for geometry processing by exploring various geometric regularization.
Revised: Recently, NFGP [58] proposes to use neural ﬁelds for geometry processing by exploring various geometric regularization.

Original: INS [59] distills stylized features into INRs via the neural style transfer framework [60].
Revised: INS [59] distills stylized features into INRs via the neural style transfer framework [60].

Original: Our INSP-Net that makes smart use of closed-form differential operators does not require neither additional per-scene fine-tuning nor discretization to grids.
Revised: Our INSP-Net that makes smart use of closed-form differential operators does not require neither additional per-scene ﬁne-tuning nor discretization to grids.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4.3 PDE based Image Processing
Revised: 4.3 PDE based Image Processing

Original: Partial differential equations (PDEs) have been successfully applied to many tasks in image processing and computer vision, such as image enhancement [61, 62, 63], segmentation [64, 40], image registration [65], saliency detection [66] and optical flow computation [67].
Revised: Partial differential equations (PDEs) have been successfully applied to many tasks in image processing and computer vision, such as image enhancement [61, 62, 63], segmentation [64, 40], image registration [65], saliency detection [66] and optical ﬂow computation [67].

Original: Early traditional PDEs are written directly based on mathematical and physical understanding of the PDEs (e.g., anisotropic diffusion [61], shock filter [62] and curve evolution based equations [68, 69, 70]).
Revised: Early traditional PDEs are written directly based on mathematical and physical understanding of the PDEs (e.g., anisotropic diffusion [61], shock ﬁlter [62] and curve evolution based equations [68, 69, 70]).

Original: Variational design methods [63, 71, 70] start from an energy function describing the desired properties of output images and compute the Euler-Lagrange equation to derive the evolution equations.
Revised: Variational design methods [63, 71, 70] start from an energy function describing the desired properties of output images and compute the Euler-Lagrange equation to derive the evolution equations.

Original: Learning-based attempts [40, 66] build PDEs from image pairs based on the assumption (without proof) that PDEs could be written as linear combinations of fundamental differential invariants.
Revised: Learning-based attempts [40, 66] build PDEs from image pairs based on the assumption (without proof) that PDEs could be written as linear combinations of fundamental differential invariants.

Original: Although it might be feasible to let INRs solve this bunch of signal processing PDEs, one needs to per-case re-fit an INR with an additional temporal axis, which is presumably sampling inefficient.
Revised: Although it might be feasible to let INRs solve this bunch of signal processing PDEs, one needs to per-case re-ﬁt an INR with an additional temporal axis, which is presumably sampling inefﬁcient.

Original: The multi-layer structure appearing in INSP-Net can be viewed as an unfolding network [72, 73] of the Euler method to solve time-variant PDEs [74].
Revised: The multi-layer structure appearing in INSP-Net can be viewed as an unfolding network [72, 73] of the Euler method to solve time-variant PDEs [74].

Original: We elaborate this connection in Appendix D.
Revised: We elaborate this connection in Appendix D.

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate the proposed INSP framework on several challenging tasks, using different combinations of Π .
Revised: In this section, we evaluate the proposed INSP framework on several challenging tasks, using different combinations of ⇧ .

Original: First, we build low-level image processing filters using either hand-crafted or learnable Π .
Revised: First, we build low-level image processing ﬁlters using either hand-crafted or learnable ⇧ .

Original: Then, we construct convolutional neural networks with our INSP-ConvNet framework and validate its performance on image classification.
Revised: Then, we construct convolutional neural networks with our INSP-ConvNet framework and validate its performance on image classiﬁcation.

Original: More results and implementation details are provided in the Appendix E F.
Revised: More results and implementation details are provided in the Appendix E F.

Original: 5.1 Low-Level Vision for Implicit Neural Images
Revised: 5.1 Low-Level Vision for Implicit Neural Images

Original: For low-level image processing, we operate on natural images from Set5 dataset [75], Set dataset [76], and DIV-2k dataset [77].
Revised: For low-level image processing, we operate on natural images from Set5 dataset [75], Set dataset [76], and DIV-2k dataset [77].

Original: Originally designed for super-resolution, the images are diverse in style and content.
Revised: Originally designed for super-resolution, the images are diverse in style and content.

Original: Note that the unprocessed images presented in figures are the images decoded from unprocessed INRs.
Revised: Note that the unprocessed images presented in ﬁgures are the images decoded from unprocessed INRs.

Original: Since our method operates directly on INRs, we firstly fit the images with INRs and then feed the INRs into our framework.
Revised: Since our method operates directly on INRs, we ﬁrstly ﬁt the images with INRs and then feed the INRs into our framework.

Original: The final output is another INR which can be decoded into desired images.
Revised: The ﬁnal output is another INR which can be decoded into desired images.

Original: The training set of our method consists of 90 examples of INRs, where each INR is built on
Revised: The training set of our method consists of 90 examples of INRs, where each INR is built on

Original: SIREN [28] architectures.
Revised: SIREN [28] architectures.

Original: INR Fitted Blur Image
Revised: INR Fitted Blur Image

Original: Wiener Filter
Revised: Wiener Filter

Original: MPRNet
Revised: MPRNet

Original: INSP-Net
Revised: INSP-Net

Original: Target Image
Revised: Target Image

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Revised: PSNR/

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Edge Detection Since the edges correspond to gradients in the images, using gradients of INRs to obtain edges is straightforward.
Revised: Edge Detection Since the edges correspond to gradients in the images, using gradients of INRs to obtain edges is straightforward.

Original: θ 1 is set to 1 while other coefficients are set to 0.
Revised: ✓ 1 is set to 1 while other coefﬁcients are set to 0.

Original: We provide visual comparisons against Sobel filter [78], Canny detector [79] and Prewitt operator [80] in Fig.
Revised: We provide visual comparisons against Sobel ﬁlter [78], Canny detector [79] and Prewitt operator [80] in Fig.

Original: Image Denoising For classical image denoising filters, we compare against the median filter and mean filter.
Revised: Image Denoising For classical image denoising ﬁlters, we compare against the median ﬁlter and mean ﬁlter.

Original: We use MPRNet [81] as a learning-based baseline.
Revised: We use MPRNet [81] as a learning-based baseline.

Original: The input noisy images are synthesized using additive gaussian noise.
Revised: The input noisy images are synthesized using additive gaussian noise.

Original: Visual results are provided in Fig.
Revised: Visual results are provided in Fig.

Original: Image Blurring Image blurring is a low-pass filtering operation.
Revised: Image Blurring Image blurring is a low-pass ﬁltering operation.

Original: We provide a visual comparison against classical filters including 3 × 3 box filter and 3 × 3 gaussian filter.
Revised: We provide a visual comparison against classical ﬁlters including 3 ⇥ 3 box ﬁlter and 3 ⇥ 3 gaussian ﬁlter.

Original: The target images used for training our INSP-Net are the results of the Gaussian filter.
Revised: The target images used for training our INSP-Net are the results of the Gaussian ﬁlter.

Original: Visual results are provided in Fig.
Revised: Visual results are provided in Fig.

Original: Image Deblurring We compare the proposed method with both traditional algorithms (e.g., wiener filter [82]) and learning-based algorithms(e.g.
Revised: Image Deblurring We compare the proposed method with both traditional algorithms (e.g., wiener ﬁlter [82]) and learning-based algorithms(e.g.

Original: , MPRNet [81]).
Revised: , MPRNet [81]).

Original: We synthesize blurry images using
Revised: We synthesize blurry images using

Original: Gaussian filters.
Revised: Gaussian ﬁlters.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, Wiener Filter produce severe artifacts and MPRNet successfully reconstructs clear textures.
Revised: 5, Wiener Filter produce severe artifacts and MPRNet successfully reconstructs clear textures.

Original: INSP-Net is capable of generating competitive results against MPRNet and outperforms the Wiener Filter.
Revised: INSP-Net is capable of generating competitive results against MPRNet and outperforms the Wiener Filter.

Original: Image Inpainting We conduct two kinds of experiments in image inpainting, to inpaint 30% random masked pixels or to remove the texts (“INSP-Net”).
Revised: Image Inpainting We conduct two kinds of experiments in image inpainting, to inpaint 30% random masked pixels or to remove the texts (“INSP-Net”).

Original: Comparison methods include mean filter, median filter, and LaMa [83].
Revised: Comparison methods include mean ﬁlter, median ﬁlter, and LaMa [83].

Original: LaMa is a learning-based method using Fourier convolution for inpainting.
Revised: LaMa is a learning-based method using Fourier convolution for inpainting.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7, mean filter and median filter partially restore the masked pixels, but severely hurt the visual quality of the rest parts.
Revised: 7, mean ﬁlter and median ﬁlter partially restore the masked pixels, but severely hurt the visual quality of the rest parts.

Original: Also, they can not handle the text region.
Revised: Also, they can not handle the text region.

Original: LaMa successfully removes the text and inpaint the masked pixels.
Revised: LaMa successfully removes the text and inpaint the masked pixels.

Original: Our proposed method largely outperforms the filter-based algorithms and performs as well as the LaMa.
Revised: Our proposed method largely outperforms the ﬁlter-based algorithms and performs as well as the LaMa.

Original: 5.2 Geometry Processing on Signed Distance Function
Revised: 5.2 Geometry Processing on Signed Distance Function

Original: We demonstrate that the proposed INSP framework is not only capable of processing images, but also capable of processing geometry.
Revised: We demonstrate that the proposed INSP framework is not only capable of processing images, but also capable of processing geometry.

Original: Signed Distance Function (SDF) [25] is adopted to represent geometries in this section.
Revised: Signed Distance Function (SDF) [25] is adopted to represent geometries in this section.

Original: We first fit an SDF from a point cloud following the training loss proposed in [28, 7].
Revised: We ﬁrst ﬁt an SDF from a point cloud following the training loss proposed in [28, 7].

Original: Then we train our INSP-Net to simulate a Gaussian-like filter similar to image blurring.
Revised: Then we train our INSP-Net to simulate a Gaussian-like ﬁlter similar to image blurring.

Original: Afterwards, we apply the trained INSP-Net to process the specified INR.
Revised: Afterwards, we apply the trained INSP-Net to process the speciﬁed INR.

Original: When visualization, we use marching cube algorithm to extract meshes from SDF.
Revised: When visualization, we use marching cube algorithm to extract meshes from SDF.

Original: We choose Thai Statue, Armadillo, and
Revised: We choose Thai Statue, Armadillo, and

Original: Dragon from Stanford 3D Scanning Repository [84, 85, 86, 87] to demonstrate our results.
Revised: Dragon from Stanford 3D Scanning Repository [84, 85, 86, 87] to demonstrate our results.

Original: Fig.
Revised: Fig.

Original: Input Image
Revised: Input Image

Original: Mean Filter Median Filter
Revised: Mean Filter Median Filter

Original: LaMa
Revised: LaMa

Original: INSP-Net
Revised: INSP-Net

Original: Target Image 12.60/0.
Revised: Target Image 12.60/0.

Original: PSNR/SSIM 26.98/0.
Revised: PSNR/SSIM 26.98/0.

Original: PSNR/SSIM
Revised: PSNR/SSIM

Original: [Figure]
Revised: [Figure]

Original: exhibits our results on Thai Statue.
Revised: exhibits our results on Thai Statue.

Original: Our method is able to smoothen the surface of the geometry and erase high-frequency details acting as if a low-pass filter.
Revised: Our method is able to smoothen the surface of the geometry and erase high-frequency details acting as if a low-pass ﬁlter.

Original: We defer more results to Fig.
Revised: We defer more results to Fig.

Original: 5.3 Classification on Implicit Neural Representations
Revised: 5.3 Classiﬁcation on Implicit Neural Representations

Original: We demonstrate that the proposed INSP framework is not only capable to express low-level image processing filters, but also supports high-level tasks such as image classification.
Revised: We demonstrate that the proposed INSP framework is not only capable to express low-level image processing ﬁlters, but also supports high-level tasks such as image classiﬁcation.

Original: To achieve this goal, we construct a 2-layer INSP-ConvNet.
Revised: To achieve this goal, we construct a 2-layer INSP-ConvNet.

Original: The INSP-ConvNet consists of 2 INSP-Net layers.
Revised: The INSP-ConvNet consists of 2 INSP-Net layers.

Original: Each of them decomposes the INR via the differential operator and combines them with learnable Π .
Revised: Each of them decomposes the INR via the differential operator and combines them with learnable ⇧ .

Original: We build another 2-layer depthwise ConvNets running on pixels as the baseline for a fair comparison, since it has comparable expressiveness to our INSP-ConvNet in theory.
Revised: We build another 2-layer depthwise ConvNets running on pixels as the baseline for a fair comparison, since it has comparable expressiveness to our INSP-ConvNet in theory.

Original: We also build a PCA + SVM method and an MLP classifier that directly classify INRs according to (vectorized) weight matrices.
Revised: We also build a PCA + SVM method and an MLP classiﬁer that directly classify INRs according to (vectorized) weight matrices.

Original: We evaluate the proposed INSP-ConvNet on MNIST ( 28 × 28 resolution) and CIFAR-10 ( 32 ×resolution) datasets, respectively.
Revised: We evaluate the proposed INSP-ConvNet on MNIST ( 28 ⇥ 28 resolution) and CIFAR-10 ( 32 ⇥resolution) datasets, respectively.

Original: For each dataset, we will firstly fit each image into an implicit representation using SIREN [28].
Revised: For each dataset, we will ﬁrstly ﬁt each image into an implicit representation using SIREN [28].

Original: Both experiments take 1000 epochs to optimize with AdamW optimizer [88] and a learning rate of 10 − 4 .
Revised: Both experiments take 1000 epochs to optimize with AdamW optimizer [88] and a learning rate of 10  4 .

Original: Results are shown in Tab. 1.
Revised: Results are shown in Tab. 1.

Original: [Table]
Revised: [Table]

Original: We categorize Depthwise CNN as explicit method, which requires to extract the image grids from
Revised: We categorize Depthwise CNN as explicit method, which requires to extract the image grids from

Original: INRs before classification.
Revised: INRs before classiﬁcation.

Original: PCA + SVM and MLP classifier working on the network parameter space can be regarded as two straightforward implicit baselines.
Revised: PCA + SVM and MLP classiﬁer working on the network parameter space can be regarded as two straightforward implicit baselines.

Original: We find that traditional classifiers can hardly classify INR on weight space due to its high-dimensional unstructured data distribution.
Revised: We ﬁnd that traditional classiﬁers can hardly classify INR on weight space due to its high-dimensional unstructured data distribution.

Original: Our method, however, can effectively leverage the information implicitly encoded in INRs by exploiting their derivatives.
Revised: Our method, however, can effectively leverage the information implicitly encoded in INRs by exploiting their derivatives.

Original: As a consequence, INSP-ConvNet can achieve classification accuracy on-par with
Revised: As a consequence, INSP-ConvNet can achieve classiﬁcation accuracy on-par with

Original: CNN-based explicit method, which validates the learning representation power of INSP-ConvNet.
Revised: CNN-based explicit method, which validates the learning representation power of INSP-ConvNet.

Original:  Conclusion
Revised:  Conclusion

Original: Contribution.
Revised: Contribution.

Original: We present INSP-Net framework, an implicit neural signal processing network that is capable of directly modifying an INR without explicit decoding.
Revised: We present INSP-Net framework, an implicit neural signal processing network that is capable of directly modifying an INR without explicit decoding.

Original: By incorporating differential operators on INR, we can instantiate the INR signal operator as a composition of computational graphs approximating any continuous convolution filter.
Revised: By incorporating differential operators on INR, we can instantiate the INR signal operator as a composition of computational graphs approximating any continuous convolution ﬁlter.

Original: Furthermore, we make the first effort to build a convolutional neural network that implicitly runs on INRs.
Revised: Furthermore, we make the ﬁrst effort to build a convolutional neural network that implicitly runs on INRs.

Original: While all other methods run on discrete grids, our experiment demonstrates our INSP-Net can achieve competitive results with entirely implicit operations.
Revised: While all other methods run on discrete grids, our experiment demonstrates our INSP-Net can achieve competitive results with entirely implicit operations.

Original: Limitations.
Revised: Limitations.

Original: (Theory)
Revised: (Theory)

Original: Our theory only guarantees the expressiveness of convolution by allowing infinite sequence approximation.
Revised: Our theory only guarantees the expressiveness of convolution by allowing inﬁnite sequence approximation.

Original: Construction of more expressive operators and more effective parameterization for convolution remain widely open questions.
Revised: Construction of more expressive operators and more effective parameterization for convolution remain widely open questions.

Original: (Practice) INSP-Net requires the computation of high-order derivatives which is neither memory efficient nor numerically stable.
Revised: (Practice) INSP-Net requires the computation of high-order derivatives which is neither memory efﬁcient nor numerically stable.

Original: This hinders the scalability of our INSP-ConvNet that requires recursive computation of derivatives.
Revised: This hinders the scalability of our INSP-ConvNet that requires recursive computation of derivatives.

Original: Addressing how to reconstruct INRs in a scalable manner is beyond the scope of this paper.
Revised: Addressing how to reconstruct INRs in a scalable manner is beyond the scope of this paper.

Original: All INRs used in our experiments are fitted by per-scene optimization.
Revised: All INRs used in our experiments are ﬁtted by per-scene optimization.

Original: Acknowledgement
Revised: Acknowledgement

Original: Z. Wang is in part supported by an NSF Scale-MoDL grant (award number: 2133861).A Proof of Theorem
Revised: Z. Wang is in part supported by an NSF Scale-MoDL grant (award number: 2133861).

Original: To begin with, we give the formal definitions of translation and rotation group, along with the notion of shift invariance and rotation invariance.

Original: Definition A.1.

Original: (Translation Group & Shift Invariance) Translation group T ( m ) is a transformation group isomorphic to m -dimension Euclidean space, where each group element T v transforms a vector x ∈ R m by T v ( x ) =

Original: x + v .

Original: An operator A is said to be shift-invariant if A (Φ ◦ T v )( x ) =

Original: A Φ( T v ( x ))

Original: =

Original: A Φ( x + v ) .

Original: Definition A.2.

Original: (Rotation Group & Rotation Invariance)

Original: Rotation group SO ( m ) is a transformation group also known as the special orthogonal group, where each group element R ∈

Original: R m × m satisfying

Original: R ⊤ R

Original: = I transforms a vector x ∈ R m by R ( x ) .

Original: An operator A is said to be rotation-invariant if

Original: A (Φ ◦ R )( x ) =

Original: A Φ( Rx ) .

Original: Proof.

Original: (Shift Invariance)

Original: To show the shift invariance of our model Eq.

Original: 3, it is equivalent to show any differential operators are shift-invariant.

Original: For the first-order derivatives (gradients), we consider arbitrary shift operator T v ∈ T , by chain rule we will have:

Original: [Equation]

Original: where the Jacobian matrix of T v ( x ) is an identity matrix.

Original: Eq.

Original: 7 implies that gradient operator is shift-invariant.

Original: By induction, any high-order differential operators must also be shift-invariant:

Original: [Equation]

Original: Therefore, we can conclude Π(Φ , ∇ Φ , ∇ 2 Φ , · · · ) is shift-invariant for any Π combining derivatives in any form.

Original: (Rotation Invariance)

Original: By Lemma A.1, given arbitrary function Φ : R m → R , and for every rotation matrix R ∈ SO ( m ) , we can compute the k -th derivatives as:

Original: [Equation]

Original: Then adopting properties of Kronecker product [89], the norm of ∇ k

Original: [Φ ◦ R ]( x ) can be written as: ∥∇

Original: [Equation]

Original: where Eq.

Original: 11 is due to the fact ( A ⊗ B ) ⊤ =

Original: A ⊤ ⊗ B ⊤ , Eq.

Original: 12 is because of ( A ⊗ B )( C ⊗

Original: D ) =

Original: AC ⊗ BD , and Eq. 13 is yielded by applying the orthogonality of R and repeating step Eq.

Original: 12 to Eq.

Original: Therefore, for every integer k > 0 , operator ∥∇ k Φ( x ) ∥ 22 is rotationinvariant.

Original: Hence, Π = f (cid:0)(cid:13) (cid:13) (cid:2)

Original: Φ( x ) ∇ Φ( x ) ∇ 2 Φ( x ) · · · (cid:3)(cid:13) (cid:13) F (cid:1) =

Original: f (cid:16)(cid:112) (cid:80) k

Original: =0 ∥∇ k Φ( x ) ∥ 2 2 (cid:17) is also rotation-invariant.

Original: Below we supplement the Lemma A.1 used to prove Theorem 1.

Original: Lemma A.1.

Original: Suppose given function f : R m → R and arbitrary linear transformation A ∈ R m × m , then vec (cid:0) ∇ k

Original: [ f ◦ A ]( x ) (cid:1)

Original: =

Original: A ⊤⊗ k vec (cid:0) ∇ k f ( Ax ) (cid:1) for ∀ k ≥ 0 .

Original: Proof.

Original: vec (cid:0) ∇ k

Original: [ f ◦ A ]( x ) (cid:1)

Original: =

Original: A ⊤⊗ k vec (cid:0) ∇ k f ( Ax ) (cid:1) trivially holds for k = 0 , 1 .

Original: Then we prove

Original: Lemma A.1 by induction.

Original: Suppose the ( j − 1) -th case satisfies the equality: vec (cid:0) ∇ j − 1 [ f ◦ A ]( x ) (cid:1) =

Original: A ⊤⊗ j − 1 vec (cid:0) ∇ j − 1 f ( Ax ) (cid:1) , then consider the j -th case:

Original: [Equation]

Original: where the first equality is done by reshaping the m j tensor to be an m × m j − 1 Jacobian matrix, the second equality is due to the induction hypothesis, and the third equality is an adoption of chain rule.

Original: Due to the fact vec( ABC ) =

Original: ( C ⊤ ⊗ A ) vec( B ) , we have vec (cid:0) ∇ j [ f ◦ A ]( x ) (cid:1) =

Original: A ⊤⊗ j vec (cid:0) ∇ j f ( Ax ) (cid:1) .

Original: Then by induction, we can conclude the proof.

Original: B Proof of Theorem

Original: For a sake of clarity, we first introduce few notations in algebra and real analysis.

Original: We use C k

Original: ( X , R ) to denote the k -th differentiable functions defined over domain X , W k,p ( X , R ) to denote the k -th differentiable and L p integrable Sobolev space over domain X .

Original: We use notation A

Original: [ f ] to denote the image of function (say f ) under the transformation of an operator (say A ).

Original: We use symbol ◦ to denote function composition (e.g., f ◦ g ( x ) =

Original: f ( g ( x )) ).

Original: We use dot-product · between two functions (say f and g ) to represent element-wise multiplication of function values (say f · g ( x ) =

Original: f ( x ) · g ( x ) ).

Original: Besides, we list the following definitions and assumptions:

Original: Definition B.1.

Original: (Polynomial) We use R [ x 1 , · · · , x m ] to represent the multivariate polynomials in terms of x 1 , · · · , x m with real coefficients.

Original: We write a (monic) multivariate monomial m ( x 1 , · · · , x m ) =

Original: x n 1 1 x n 2 2 · · · x n m m as m ( x ) =

Original: x n where n = [ n· · · n m ] ∈ N m .

Original: Then we denote a polynomial as p ( x ) =

Original: a 1 x n 1 + · · · + a d x n d ∈ R [ x 1 , · · · , x m ] where x n i denotes the i -th multivariate monomial and a i ∈ R is the corresponding coefficient.

Original: Definition B.2.

Original: (Differential Operator) Suppose a compact set X ⊆ R m .

Original: we denote D n :

Original: C ∞ ( X , R )

Original: → C ∞ ( X , R ) as the high-order differential operator associated with indices n ∈ N m :

Original: [Equation]

Original: Definition B.3.

Original: We define polynomial in gradient operator as: p ( ∇ )

Original: = p (cid:16) ∂∂x

Original: [Equation]

Original: , · · · , ∂∂x m (cid:17) =

Original: a 1 D n 1 + · · · + a d D n d ∈ R

Original: [ x 1 , · · · , x m ] where D n i denotes the n i -th order partial derivative (Definition B.2) and a i ∈ R is the corresponding coefficient.

Original: Remark B.1.

Original: The mapping between p ( x ) and p ( ∇ ) is a ring homomorphism from polynomial ring R [ x 1 , · · · , x m ] to the ring of endomorphism defined over C ∞ ( X , R ) .

Original: Definition B.4.

Original: (Fourier Transform) Given real-valued function f : R m → C that satisfies Dirichlet condition 5 , then Fourier transform F is defined as:

Original: [Equation]

Original: Inverse Fourier transform F − 1 exists and has the form of:

Original: [Equation]

Original: Definition B.5.

Original: (Convolution)

Original: Given two real-valued functions f : R m → R and g : R m → R , convolution between f and g is defined as:

Original: [Equation]

Original: Then we denote f ⋆ g

Original: = T g [ f ] , where T g represents a convolutional operator associated with the function g .

Original: We make the following mild assumptions on the signals and convolutional operators, which are widely satisfied by the common signals and systems.

Original: Assumption B.6.

Original: (Band-limited Signal Space)

Original: Define the signal space S as a Sobolev space

Original: W ∞ , 1 ( R m , R ) of real-valued functions such that for ∀ f ∈ S :

Original: (I) f ∈ C ∞ ( R m , R ) is continuous and smooth over R m .

Original: (II) f satisfies the Dirichlet condition.

Original: [-(III) f has a limited width of spectrum: there exists a compact subset W ⊂ R m such that |F [ f ]( w ) | = 
-]
Revised: [-(III) f has a limited width of spectrum: there exists a compact subset W ⊂ R m such that |F [ f ]( w ) | = 
-]

Original: if w / ∈ W , and (cid:82) W |F [ f ]( w ) | d w < ∞ .

Original: Assumption B.7.

Original: (Convolution Space)

Original: Define a convolutional operator space T such that ∀T g ∈ T :

Original: (IV) g : R m → R is real-valued function.

Original: (V) F [ g ] ∈ C ( R m , R ) has a continuous spectrum.

Original: Before we prove Theorem 2, we enumerate the following results as our key mathematical tools:

Original: First of all, we note the following well-known result without a proof.

Original: Lemma B.1.

Original: (Convolution Theorem)

Original: For every T g ∈ T , it always holds that F ◦ T g [ f ]( w ) =

Original: F [ f ]( w ) · F [ g ]( w ) .

Original: Next, we present Stone-Weierstrass Theorem as our Lemma B.2 as below.

Original: Lemma B.2.

Original: (Stone-Weierstrass Theorem)

Original: Suppose X is a compact metric space.

Original: If A ⊂ C ( X , R ) is a unital sub-algebra which separates points in X .

Original: Then A is dense in C ( X , R ) .

Original: A straightforward corollary of Lemma B.2 is the following Lemma B.3.

Original: Lemma B.3.

Original: Let X ⊂ R m be a compact subset of R m .

Original: For every ϵ > 0 , there exists a polynomial p ( x ) ∈ R [ x 1 , · · · , x m ] such that sup x ∈X | f ( x ) − p ( x ) | < ϵ .

Original: Proof.

Original: Proved by checking polynomials R [ x 1 , · · · , x m ] form a unital sub-algebra separating points in X , and equipping C ( X , R ) with the distance metric d ( f, h ) =

Original: sup x ∈X | f ( x ) − g ( x ) | .

Original: We also provide the following Lemma B.4 to reveal the spectrum-domain symmetry for real-valued signals.

Original: Lemma B.4.

Original: Suppose f is a continuous real-valued function satisfying Dirichlet condition.

Original: Then

Original: F [ f ]( w ) =

Original: F [ f ]( − w ) ∗ , i.e., the spectrum of real-valued function is conjugate symmetric.

Original: Proof.

Original: By the definition of Fourier transform (Definition B.4):

Original: [Equation]

Original: where Eq.

Original: 18 holds because f is a real-valued function.

Original: We present Lemma B.5 as below, which reflects the effect of differential operators on the spectral domain.

Original: Lemma B.5.

Original: Suppose f ∈ C ∞ ( R m , R ) is a smooth real-valued function satisfying Dirichlet condition.

Original: Then F ◦ D n [ f ]( w ) =

Original: (2 πi ) ∥ n ∥ 1 w n · F [ f ]( w ) for every n ∈ N m .

Original: Proof.

Original: We first show the case of first-order partial derivative.

Original: Suppose h ∈ C

Original: ( R m , R ) is L 1 integrable (then Fourier transform exists).

Original: [Equation]

Original: Then we apply the Fourier transform to Eq.

Original: 20, we can obtain:

Original: [Equation]

Original: [Equation]

Original: Note that f ∈ W ∞ , 1 ( R m , R ) ensures all its partial derivatives are differentiable and absolutely integrable.

Original: We can recursively apply ∂∂x i to f for n i times for each i ∈ [ m ] , and use Eq.

Original: 23 above to conclude the proof.

Original: Below is the formal statement of our Theorem 2 and its detailed proof.

Original: Theorem B.6.

Original: For every T g ∈ T and arbitrarily small ϵ > 0 , there exists a polynomial p ( x ) ∈ R [ x 1 , · · · , x m ] such that sup x ∈ R m |T g

Original: [ f ]( x ) − p ( ∇ )

Original: [ f ]( x ) | <

Original: ϵ for all f ∈ S .

Original: Proof.

Original: For every f ∈ S and T g ∈ T , by Lemma B.1, one can rewrite:

Original: [Equation]

Original: where we use ˆ f : R m → C and ˆ g : R m → C to denote the Fourier transform of f and g , respectively.

Original: We can construct an invertible mapping ϕ by letting:

Original: [Equation]

Original: which is also known as the Hartley transform.

Original: By Lemma B.4 (with Assumption (I) (IV)), ˜ f := ϕ

Original: [ ˆ f ] and ˜ g :

Original: = ϕ

Original: [ˆ g ] are both real-valued functions.

Original: Since ˆ f is only supported in W (by Assumption (III)), we only consider ˜ g within the compact subset

Original: W .

Original: By Lemma B.3 (with Assumption (V)), there exists a polynomial ˜ p ( w ) ∈ R [ w 1 , · · · , w m ] =

Original: [-˜ a 
-]
Revised: [-˜ a 
-]

Original: + ˜ a 1 w n 1 + · · · + ˜ a d w n d such that sup w ∈W | ˜ g ( w ) − ˜ p ( w ) | < ϵ/ 2 C for every ϵ > 0 , where d is the number of monomials in ˜ p , ˜ a 0 , ˜ a 1 , · · · , ˜ a d ∈ R are corresponding coefficients, and C > 0 is some constant.

Original: Applying ϕ − 1 to ˜ p , we will obtain a new (complex-valued) polynomial ˆ p :=

Original: ϕ − 1 [˜ p ] ∈ C [ w 1 , · · · , w m ] such that:

Original: [Equation]

Original: We observe that the coefficients of ˆ p satisfy: ˆ a k = ˜

Original: a k if ∥ n k ∥ 1 is even and ˆ a k

Original: = i ˜ a k if ∥ n k ∥ 1 is odd.

Original: Then we bound the difference between ˆ g and ˆ p for every w ∈ W :

Original: [Equation]

Original: In the meanwhile, by Lemma B.5 (with Assumption (I), (II)), F ◦ D n [ f ]( w ) = (2 πi ) ∥ n ∥ 1 w n ·

Original: F [ f ]( w ) for every n ∈ N m .

Original: Define a sequence q n ( w ) =

Original: (2 πi ) ∥ n ∥ 1 w n , then partial derivatives of f in terms of n ∈ N m can be written as:

Original: [Equation]

Original: Next we decompose polynomial ˆ p in terms of q n .

Original: Let a k

Original: = ˆ a k / (2 πi )

Original: ∥ n k ∥ 1 , then ˆ p ( w )

Original: = a 0 + a 1 q n 1 ( w ) + · · · + a d q n d ( w ) .

Original: We note that { a k , ∀ k ∈ [ d ] } must be real numbers since ˆ a k is real/imaginary when ∥ n k ∥ 1 is even/odd, which coincides with (2 πi )

Original: ∥ n k ∥ 1 .

Original: By linearity of inverse Fourier transform and Eq.

Original: 33, element-wisely multiplying (cid:80) dk

Original: [-=
-]
Revised: [-=
-]

Original: a k q n k to ˆ f will lead to a transform on the spatial domain:

Original: [Equation]

Original: ˆ y ( l ) ← (cid:104) y

Original: ( l − 1)

Original: ∂y ( l − 1) ∂ x ⊤ ∂ 2 y ( l − 1) ∂ x 2 ⊤ · · · ∂ K y ( l − 1) ∂ x K ⊤ (cid:105) θ

Original: ( l ) ▷ Convolutional layer l ) where we define polynomial p ( ∇ ) :

Original: = a 0 + a 1 D n 1 + · · · + a d D n d ∈ R (cid:104) ∂∂x 1 , · · · , ∂∂x m (cid:105) over the ring of partial differential operators (Definition B.3).

Original: Now we bound the difference between T g [ f ] and p ( ∇ )[ f ] for every f ∈ S and x ∈ R m : |T g [ f ]( x ) − p ( ∇ )

Original: [ f ]( x ) | = (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:90) W exp(2 πi w ⊤ x ) ˆ f ( w ) (cid:32)

Original: ˆ g ( w ) − d (cid:88) k

Original: = a k q n k ( w ) (cid:33) d w (cid:12) (cid:12)(cid:12)(cid:12)(cid:12) (35) ≤ (cid:90)

Original: W (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: where Eq.

Original: 37 follows from Hölder’s inequality, and Eq.

Original: 38 is obtained by substituting the upper bound of difference | ˆ g ( w ) − (cid:80) dk

Original: [-=
-]
Revised: [-=
-]

Original: a k q n k ( w ) | and letting C equal to the L 1 norm of ˆ f ( w ) (by

Original: Assumption (III)).

Original: C Implementation Details of INSP-ConvNet

Original: We have formulated exact convolution form and INSP-Conv in Sec. 3.3.

Original: We provide a pseudocode to illustrate the forward pass of INSP-ConvNet in Algorithm 1.

Original: Below we elaborate each main component:

Original: Convolutional Layer.

Original: Each A ( l ) represents an implicit convolution layer.

Original: We follow the closedform solution in Eq.

Original: 5 to parameterize A ( l ) with θ ( l ) .

Original: We point out that ConvNet[Φ] also corresponds to a computational graph, which can continuously map coordinates to the output features.

Original: To construct this computational graph, we recursively call for gradient networks of the previous layer until the first layer.

Original: For example, A ( l ) will request the gradient network of A ( l − 1) · σ ◦· · ·◦A (1) · Φ , and then A ( l − 1) will request the gradient network of the rest part.

Original: This procedure will proceed until the first layer, which directly returns the derivative network of Φ .

Original: Kernels in CNNs typically perform multi-channel convolution.

Original: However, this is not memory friendly to gradient computing in our framework.

Original: To this end, we run channel-wise convolution first and then employ a linear layer to mix channels [90].

Original: Nonlinear Activation and Normalization.

Original: Nonlinear activation and normalization are naturally element-wise functions.

Original: They are point-wisely applied to the output of an INSP-Net and participate the computational graph construction process.

Original: This corresponds to the line 6 of Algorithm 1.

Original: Training Recipe.

Original: Given a dataset D

Original: = { (Φ i , y i ) } with a set of pre-trained INRs Φ i and their corresponding labels y

Original: i , our goal is to learn a ConvNet[ · ] that can process each example.

Original: In contrast to standard ConvNets that are designed for grid-based images, the computational graph of INSPConvNet contains parameters of both the input INR Φ i and learnable kernels A ( l ) .

Original: During the training stage, we randomly sample a mini-batch (Φ i , y i ) from D to optimize INSP-ConvNet.

Original: The corresponding loss will be evaluated according to the network output, and then back-propagate the calculated gradients to the learnable parameters in A ( l ) , using the stochastic gradient descent optimization.

Original: Along the whole process, the parameters of Φ i is fixed and only the parameters in A ( l ) is optimized.

Original: Standard data augmentations are included by default, including rotation, zoom in/out, etc.

Original: In practice, we implement these augmentations by using affine transformation on the coordinates of INRs.

Original: D Connection with PDE based Signal Processing

Original: Partial Differential Equation (PDE) has been successfully applied to image processing domain as we discussed in Sec. 4.3.

Original: In this section, we focus on their connection with our INSP-Net.

Original: We summarize the methods of this line of works [61, 40, 66] in the following formulation:

Original: [Equation]

Original: where M t ( · ) is a time-variant function that remaps the direct output and high-order derivatives of function Ψ .

Original: For heat diffusion, M t boils down to be an stationary isotropic combination of secondorder derivatives.

Original: In [61], M t is chosen to be a gradient magnitude aware diffusion operator running on divergence operators.

Original: [40, 66] degenerate M t to a time-dependent linear mapping of pre-defined invariants of the maximal order two.

Original: We note that Eq.

Original: 39 can be naturally solved with INRs, as

Original: INRs are amenable to solving complicated differential equation shown by [28].

Original: One straightforward solution is to parameterize M t by another time-dependent coordinate network [27] and enforce the boundary condition Ψ( x , 0) =

Original: Φ( x ) and minimize the difference between the two hands of the Eq.

Original: However, foreseeable problem falls in sampling inefficiency over the time axis.

Original: [-Suppose we discretize the time axis into small intervals 
-]
Revised: [-Suppose we discretize the time axis into small intervals 
-]

Original: = t 0 < t 1 < · · · < t N , then Eq.

Original: 39 has a closed-form solution given M t by Euler method:

Original: [Equation]

Original: One can see Eq.

Original: 41 can be regarded as a special case of our model Eq.

Original: 3, where we absorb M t n ,

Original: time interval t n +1 − t n , and the residual term Ψ( x , t n ) into one Π .

Original: Considering our multi-layer model

Original: INSP-ConvNet (see Sec. 3.3), we can analogize t n to the layer number, and then solving Eq.

Original: 39 at time t N is approximately equal to forward passing an N -layer INSP-ConvNet.

Original: E More Experiment Details

Original: We implement our INSP framework using PyTorch.

Original: The gradients are obtained directly using the autograd package from PyTorch.

Original: All learnable parameters are trained with AdamW optimizer and a learning rate of 1e-4.

Original: For low-level image processing kernels, images are obtained from Set dataset [75], Set14 dataset [76], and DIV-2k dataset [77].

Original: These datasets are original collected for super-resolution task, so the images are diverse in style and content.

Original: In our experiments, we construct

Original: SIREN [28] on each image.

Original: For efficiency, we resized the images to 256 × 256 .

Original: We use 90 images to construct the INRs used for training, and use the other images for evaluation.

Original: For image classification, we construct a 2-layer INSP-ConvNet framework.

Original: Each INSP layer constructs the derivative computational graphs of the former layers and combines them with learnable Π .

Original: The INSP-layer is capable of approximating a convolution filter.

Original: For a fair comparison, we build another 2-layer depthwise convolutional network running on image pixels as the baseline.

Original: Both our

Original: INSP-ConvNet and the ConvNet running on pixels are trained with the same hyper-parameters.

Original: Both experiments take 1000 training epochs, with a learning rate of 1 e − 4 using AdamW optimizer.

Original: F More Experimental Results

Original: Additional Visualization.

Original: In this section, we provide more experimental results.

Original: Fig.

Original: 9 provides comparisons on edge detection task.

Original: Fig.

Original: 10 shows image denoising results.

Original: Fig.

Original: 11 demonstrates

Original: [Table]

Original: image deblurring results.

Original: Fig.

Original: 12 shows image blurring results.

Original: Fig.

Original: 13 shows image inpainting results.

Original: Fig.

Original: 14 presents additional results on geometry smoothening.

Original: Additional Quantitative Results.

Original: We also provide quantitative comparisons on the test set in

Original: Tab. 2.

Original: The test set consists of 100 INRs fitted from 100 images in DIV-2k dataset [77].

Original: In Tab. 2, their performance is better when the synthetic noise becomes three-channel Gaussian noise.

Original: The synthetic noise is similar to those seen during the training process of MPRNet [81] and MAXIM [91], so they benefit from their much wider training set.

Original: Audio Signal Processing.

Original: We additionally validate the ability of our INSP framework by processing audio signals.

Original: We add synthetic Gaussian noise onto the audio and use it to fit a SIREN.

Original: The noisy audio decoded from the INR is shown in Fig.

Original: 15(b).

Original: Then we use our INSP-Net to implicitly process it to a new INR that can be further decoded into denoised audio.

Original: It’s decoded result is shown in

Original: Fig.

Original: 15(c).

Original: We also provide visualization of the denoising effect in Fig. 15(f).

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Input Image

Original: Mean Filter

Original: LaMa

Original: INSP-Net

Original: Target Image

Original: [Figure]

Original: [Figure]