Approximate Discretization Invariance for Deep Learning on Neural Fields
Abstract
While neural fields have emerged as powerful representations of continuous data, there is a need for neural networks that can perform inference on such data without being sensitive to how the field is sampled, a property called (approximate) discretization invariance.
We develop DI-Net, a framework for learning discretization invariant operators on neural fields of any type.
Whereas current theoretical analyses of discretization invariant networks are restricted to the limit of infinite samples, our analysis does not require infinite samples and establishes upper bounds on the variation in DI-Net outputs given different finite discretizations. Our framework leads to a family of neural networks driven by numerical integration via quasi-Monte Carlo sampling with discretizations of low discrepancy. DINets manifest desirable theoretical properties such as universal approximation of a large class of maps between L 2 functions, and gradients that are also discretization invariant. DINets can also be seen as generalizations of many existing network families as they bridge discrete and continuous network classes, such as convolutional neural networks (CNNs) and neural operators respectively.
Experimentally, DI-Nets derived from CNNs are able to classify and segment visual data represented by neural fields under various discretizations, and sometimes even generalize to new types of discretizations at test time.
1. Introduction
Neural fields (NFs), which encode signals as the parameters of a neural network, have many useful properties.
NFs can efficiently store and stream continuous data (Sitzmann et al., 2020b; Dupont et al., 2022; Takikawa et al., 2022), represent and render detailed 3D scenes at lightning speeds (M¨uller et al., 2022), and integrate data from a wide range of modalities (Gao et al., 2022).
NFs are thus an appealing data representation for many applications.
However, current approaches for training networks on a dataset of NFs have major limitations.
The sampling-based approach converts such data to pixels or voxels as input to discrete networks (Vora et al., 2021), but it incurs interpolation errors and does not leverage the ability to evaluate the NF anywhere on its domain.
The hypernetwork approach trains a model to predict NF parameters (or a lower dimensional “modulation” of such parameters) which can be tailored for downstream tasks (Tancik et al., 2020a; Dupont et al., 2022;
Mehta et al., 2021), but hypernetworks based on the parameter space of one type of NF are incompatible with other types.
Moreover, hypernetworks are unsuitable for important classes of NFs whose parameters extend beyond a neural network. Our work strengthens the sampling-based approach with the notion of approximate discretization invariance : the output of an operator that processes a continuous signal by sampling it at a set of discrete points should be largely independent of how the sample points are chosen, particularly as the number of points becomes large 1 .
In this paper we propose the DI-Net , a discretization invariant neural network for learning and inference on NFs.
DI-Nets leverage numerical integration to yield desirable convergence properties, can be applied to any type of NF, and are universal approximators on a large class of maps.
DI-Nets are a broad class of models that can be applied to classification, segmentation, and many other tasks.
2. Principles of DI-Nets
Let Ω be a bounded measurable subset of a d -dimensional compact metric space.
Let F c denote the space L 2 (Ω , R c )
= { f : Ω → R c : (cid:82) Ω ∥
f ∥ 2 dµ < ∞} .
We refer to all maps f ∈ F c as NFs, although such maps also encompass other types of signals and data formats.
We call d the dimensionality of the NF and c the number of channels.
Definition 1 A DI-Net layer H is a bounded continuous map F n → F m (e.g., linear combinations of channels), or R n → F m (e.g., in a generative model), or F n → R m (e.g., in a classifier), or F n × F m → F c (e.g., in style transfer) for some n, m, c ∈ N . If the map’s domain is F n , it must also be discretization invariant (Definition 3).
A DI-Net is a directed acyclic graph of such layers.
Proposition 2 A DI-Net is invariant to the parameterization of the NF used to train it.
If f, g ∈ F c are equal in the sense that f i
= g
i for all i ∈ { 1 , ..., c } , then any DI-Net or loss function on F c maps f and g to the same output.
This property follows directly from our definition of the DI-Net, and confers interoperability that is missing from hypernetwork or modulation-based learning approaches.
For simplicity we characterize single-channel NFs and DI-Net layers, and in Appendix C we extend our results to the multi-channel case.
The discretization of a neural field f ∈ L 2 (Ω) is a point set
X ⊂ Ω on which the NF is evaluated.
We say that the map H ϕ : L 2 (Ω) → R n is discretizable if it induces a map H ϕ,X :
L 2 (Ω) /I
X → R n for every discretization X , where I X is the ideal of L 2 (Ω) that vanishes at each x ∈ X .
Definition 3 A discretizable map H ϕ : L 2 (Ω) → R n is discretization invariant if for every discretization X and input f , the deviation of H ϕ from the induced map ˆ H Xϕ is bounded by the product of the variation of f and the discrepancy of X .
A map ¯ H ϕ : L 2 (Ω) →
L 2 (Ω) is discretization invariant if it is discretization invariant at each point in its image.
A function’s variation measures how much it fluctuates over its domain, and a discretization’s discrepancy is low for dense, evenly distributed point sets (see Appendix B for precise definitions).
Their product is precisely the upper bound in the celebrated Koksma–Hlawka inequality, which bounds the difference between a function’s integral and its sample mean:
Definition {+
+}
A function f ∈ L 2 (Ω) satisfies a Koksma–Hlawka inequality if for any point set X ⊂ Ω ,
[Equation]
for normalized measure dx , variation V of the function and discrepancy D of the point set. This naturally leads to parameterizations of discretization invariant layers as integrals over Ω. In the L 2 (Ω) → R case:
[Equation]
for a function h which is Fr´echet differentiable at all f ∈ L 2 (Ω), is of bounded variation in Ω for all f , is differentiable w.r.t.
its parameters ϕ , and may depend on weak derivative
D α f = ∂ | α | f ∂x α 11 ...
∂x αnn for multi-index α .
4 Its induced map under discretization X is then:
[Equation]
where w ( x ; µ, X ) are quadrature weights (1 / | X | by default).
We can write the L 2 (Ω) → L 2 (Ω) case similarly.
Such a parameterization opens up a rich toolkit of numerical integration methods, although here we focus on quasi-Monte Carlo (QMC) (Caflisch, 1998), which generates low discrepancy discretizations.
Our definition of discretization invariance immediately yields a class of discretization sequences under which discretized maps converge:
{+Definition 
+}
We call a sequence of discretizations { X N } N ∈ N whose discrepancy tends to 0 as N → ∞ an equidistributed discretization sequence.
Proposition 6 If H ϕ is a discretization invariant map and f ∈ L 2 (Ω) has finite variation, lim N →∞ ˆ H X N ϕ [ f ] = H ϕ [ f ] for any equidistributed discretization sequence { X N } N ∈ N .
Thus a network composed of discretization invariant layers converges in the forward direction. We can also establish convergence of the network’s discretized gradients as follows (proved and expanded on in Appendix C.2): Theorem 7 A DI-Net permits backpropagation of its outputs with respect to its input as well as all its learnable parameters.
Under an equidistributed discretization sequence, the gradients of each layer converge to the appropriate derivative under the measure on Ω .
Recall that discretization invariant maps were defined in terms of an arbitrary discretizable map on a continuous domain. Then Proposition 6 suggests that for every discretizable map there exists some discretization of sufficiently low discrepancy such that the induced discrete map approximates the continuous-domain map to desired precision for particular types of functions. This universal approximation theorem can be formalized as follows:
Theorem 8 Let G , G ′ be subsets on L 2 (Ω) restricted to piecewise smooth functions, with mild conditions on Ω .
For every Lipschitz continuous map R : G → G ′ , there exists a DI-Net T that approximates it to arbitrary accuracy w.r.t.
a finite measure ν on G .
As a corollary, every Lipschitz continuous map G → R n or R n → G can also be approximated by some DI-Net.
3. Design and Implementation of DI-Nets
DI-Nets encompass a very large family of neural networks. On one side of the DI-Net family includes networks such as DeepONets and neural operators (Kovachki et al., 2021b;
Lu et al., 2021), which can learn general maps between function spaces but in practice are designed to solve partial differential equations.
On the other side are networks that are designed entirely on discrete domains such as convolutional neural networks (CNNs), but can be readily extended to the continuous domain. In the same way that neural fields extend signals on point clouds, meshes, grids, and graphs to a compact metric space, DINets extend neural networks that operate on discrete signals by converting every layer to an equivalent discretizable map. We illustrate the case of CNNs: we describe how to extend convolutional layers to DINets here, and discuss other constructions in Appendix D, including normalization, pooling, upsampling, skip connections, tokenization and attention. The resulting convolutional DINets can be initialized directly with the weights of a pre-trained CNN as we investigate in
Appendix G.1.
Convolution
For a measurable S ⊂ Ω and a polynomial basis { p j,ϕ } j ≥ 0 that spans L 2 ( S ), S is the support of a polynomial convolutional kernel K ϕ : Ω × Ω → R defined by: [Equation] for some chosen n ∈ N .
A convolution is the linear map H ϕ :
L 2 (Ω) →
L 2 (Ω) given by:
[Equation]
An MLP convolution is defined similarly except the kernel becomes ˜ K ϕ ( x, x ′ ) =
MLP( x − x ′ ; ϕ ) in the non-zero case.
While MLP kernels are favored over polynomial kernels in many applications due to their expressive power (Wang et al., 2021), polynomial bases can be used to construct filters satisfying desired properties such as group equivariance (Cohen and Welling, 2016a,b), k -Lipschitz continuity, or boundary conditions.
The input and output discretizations of the layer can be chosen independently, allowing for padding or striding (see Appendix D).
The input discretization fully determines which points on S are evaluated.
3.1. Training DI-Nets
As with discrete networks, the pipeline for training an DI-Net varies with the task.
We outline steps for training a classifier and dense prediction model in Algorithms E.2 and F.3.
When training an NF classifier, the discretization may be specified manually or sampled from a low discrepancy sequence to perform QMC integration.
When training networks for dense prediction, the output discretization should be chosen to match the coordinates of the ground truth labels.
The input discretization can be arbitrarily chosen, but in our experiments we set it equal to the output discretization.
At inference time, the network can be evaluated under any desired output discretization, making the output in effect an NF.
Similar pipelines can be used to train DI-Net for other tasks such as scene editing, inverse problems, or representation learning.
4. Experiments
We analyze our model’s performance as a classifier of NFs fit to natural images (ImageNet1k) under different discretizations.
We train DI-Nets with 2 and 4 MLP convolutional layers (DI-Net-2 and DI-Net-4), as well as CNNs with equivalent architectures.
provides experimental details and Appendix F provides an additional experiment on using
DI-Net to segment NFs. DI-Nets somewhat underperform their CNN counterparts (Table E.2), and the gap in performance is larger for the deeper models. However, our discretization invariant model better generalizes to higher-resolution images, where it outperforms CNNs on images at twice the resolution and has a very small drop in performance compared to the resolution it was trained on.
[Table]
Having found that discretization invariance enables generalization to a different resolution at test time, we examine to what extent the network can adapt to an entirely different type of discretization at test time.
We use grid, QMC (randomized Sobol sequence) and shrunk 6 discretizations of 1024 (32 × 32) points. Interestingly, changing discretization type at inference time has varying impact, usually only slightly degrading the performance of DI-Net (Table 1), but resulting in a dramatic drop in performance when shifting from a high discrepancy discretization to a low discrepancy discretization (shrunk → QMC).
This suggests that discretization invariance only provides a weak guarantee on the stability of a model’s behavior in the general case, and points to the importance of training on the right discretizations to attain a network that generalizes well to other discretizations.
Appendix
Appendix A describes related work.
Appendix B provides additional background on quasiMonte Carlo integration and low discrepancy sequences, including details about the conditions under which our theoretical results hold.
Appendix C provides proofs of Theoremsand 8, as well as extensions of these theorems from the single-channel case stated in the main text to multi-channel maps.
Appendix D provides a detailed specification of DI-Net layers that enable DI-Nets to replicate the behavior of CNNs.
Appendix E provides additional details of the classification experiment.
Appendix F provides an additional experiment on NF segmentation.
Appendix G provides several analyses including properties of DI-Nets under different discretizations.
Appendix H describes limitations and directions for future work.
Appendix A. Related Work Neural fields
Multilayer perceptrons (MLPs) can be trained to capture a wide range of continuous data with high fidelity.
The most prominent domains include shapes (Park et al., 2019; Mescheder et al., 2018), objects (Niemeyer et al., 2020; M¨uller et al., 2022), and 3D scenes (Mildenhall et al., 2020; Sitzmann et al., 2021), but previous works also apply NFs to gigapixel images (Martel et al., 2021), volumetric medical images (Corona-Figueroa et al., 2022), acoustic data (Sitzmann et al., 2020b; Gao et al., 2021), tactile data (Gao et al., 2022), depth and segmentation maps (Kundu et al., 2022), and 3D motion (Niemeyer et al., 2019).
Hypernetworks and modulation networks were developed for learning directly with
NFs, and have been demonstrated on tasks including generative modeling, data imputation, novel view synthesis and classification (Sitzmann et al., 2020b, 2021; Tancik et al., 2020a;
Sitzmann et al., 2019, 2020a; Mehta et al., 2021; Chan et al., 2021; Dupont et al., 2021, 2022).
Hypernetworks use meta-learning to learn to produce the MLP weights of desired output NFs, while modulation networks predict modulations that can be used to transform the parameters of an existing NF or generate a new NF.
Another approach for learning maps between NFs in 3D space evaluates an input NF at grid points, produces features at the same points via a U-Net, and passes “raytraced” features through an MLP to produce output values from arbitrary camera angles (Vora et al., 2021).
Discretization invariant networks
Networks that are agnostic to the discretization of the data domain has been explored in several contexts.
Hilbert space PCA, DeepONets and neural operators learn discretization invariant maps between function spaces (Bhattacharya et al., 2020; Lu et al., 2021; Li et al., 2020; Kovachki et al., 2021b), and are tailored to solve partial differential equations efficiently.
On surface meshes, DiffusionNet (Sharp et al., 2022) uses the diffusion operator to achieve convergent behavior under mesh refinement.
At the core of many other discretization invariant approaches is the continuous convolution, which also provides permutation invariance, translation invariance and locality. Its applications include modeling point clouds (Wang et al., 2021; Boulch, 2019), graphs (Fey et al., 2017), fluids (Ummenhofer et al., 2019), and sequential data (Romero et al., 2021b,a).
These previous works define discretization invariance as convergent behavior in the limit of infinite sample points, but do not characterize how different discretizations yield different behaviors in the finite case.
In this work, we choose a stronger definition that bounds the difference between any two finite discretizations, which we show implies the convergence condition.
We formulate discretization invariant networks on general metric spaces, which generalizes DeepONets and neural operators, then we focus on the continuous convolution as a core layer for vision applications.
Approximation capabilities of neural networks
A fundamental result in approximation theory is that the set of single-layer neural networks is dense in a large space of functionals including L p ( R n ) (Hornik, 1991).
Subsequent works designed constructive examples using various non-linear activations (Chen et al., 1995; Chen and Chen, 1993).
While this result is readily extended to multi-dimensional outputs, existing approximation results for the case of infinite dimensional outputs (e.g., L p ( R n ) → L p ( R n )) do not explicitly characterize the contribution of data discretization to the approximation error (Bhattacharya et al., 2020; Lanthaler et al., 2022; Kovachki et al., 2021b,a).
Appendix B. Koksma–Hlawka inequality and low discrepancy sequences
Recall that a function f ∈ L 2 (Ω) satisfies a Koksma–Hlawka inequality if for any point set X ⊂ Ω,
[Equation]
(6) for normalized measure dx , some notion of variation V of the function and some notion of discrepancy D of the point set.
The classical inequality gives a tight error bound for functions of bounded variation in the sense of Hardy-Krause (BVHK), a generalization of bounded variation to multivariate functions on [0 , 1] d which has bounded variation in each variable.
Specifically, the variation is defined as:
[Equation]
1 } d
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation] 7) with { 0 , 1 } d the multi-indices and x α ∈ [0 , 1] d such that x α,j = x j if j ∈ α and x α,j =otherwise.
The classical inequality also uses the star discrepancy of the point set X , given by:
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation] [Equation] where J is the set of d -dimensional intervals that include the origin, and λ the Lebesgue measure.
A point set is called low discrepancy if its discrepancy is on the order of O ((ln N ) d /N ).
Quasi-Monte Carlo calculates the sample mean using a low discrepancy sequence (see Fig.
B.1 for examples in 2D), as opposed to the i.i.d.
point set generated by standard Monte
Carlo, which will generally be high discrepancy.
Because the Koksma–Hlawka inequality is sharp, when estimating the integral of a BVHK function on [0 , 1] d , the error of the QMC approximation decays as O ((ln N ) d /N ), in contrast to the error of the standard Monte Carlo approximation that decays as O ( N − 1 / 2 ) (Caflisch, 1998).
However, BVHK is a rather restrictive class of functions defined on [0 , 1] d that excludes all functions with discontinuities.
Brandolini et al.
(2013) extended the Koksma–Hlawka inequality to two classes of functions defined below:
Piecewise smooth functions
Let f be a smooth function on [0 , 1] d and Ω a Borel subset of [0 , 1] d .
Then f | Ω is a piecewise smooth function with the Koksma–Hlawka inequality given by variation
[Equation]
and discrepancy
[Equation]
W d, 1 functions on manifolds
Let M be a smooth compact d -dimensional manifold with normalized measure dx .
Given local charts { ϕ k } Kk =1 , ϕ k : [0 , 1] d → M , the variation of a function f ∈ W d, 1 ( M ) is characterized as:
[Equation]
∂x [Equation]
[Equation]
with { ψ k } Kk =1 a smooth partition of unity subordinate to the charts, and constant c >that depends on the charts but not on f .
Defining the set of intervals in M as J =
{ U :
U =
ϕ k
( I ) for some k and I ⊆
[0 , 1] d } , with measure µ ( U ) =
λ ( I ), the discrepancy of a point set Y =
{ y j } Ny =1 on M is:
[Equation]
[Equation]
The notion of discrepancy is not limited to the Lebesgue measure.
The existence of low discrepancy point sets has been proven for non-negative, normalized Borel measures on [0 , 1] d due to Aistleitner and Dick (2013).
An extension of our framework to non-uniform measures is a promising direction for future work (see Section H).
Appendix C. Proofs C.1. Proof of Property 8 (Universal Approximator)
Definition 9 Let G , G ′ be subsets of L 2 (Ω) restricted to compactly supported functions of absolute bounded variation.
By this we mean there exists V ∗ such that every f ∈ G ∪ G ′ satisfies a Koksma–Hlawka inequality (6) with V ( | f | ) <
V ∗ .
G , G ′ are bounded in L 1 norm since all their functions are compactly supported and bounded.
Consider a Lipschitz continuous map R : G → G ′ such that d ( R [ f ] , R [ g ])
L 1 ≤ M 0 d ( f, g )
Lfor some constant M 0 and all f, g ∈ G .
Let M = max { M 0 , 1 } .
Select a fixed point set X
= { x j } Nj =1 in Ω with discrepancy D ( X ) =
ϵ 12( M +2)
V ∗ .
By (6) this yields:
[Equation]
for all f ∈ G ∪ G ′ .
Definition
The projection π : f (cid:55)→ f is a quotient map L 2 (Ω) →
L 2 (Ω) / ∼ under the equivalence relation f ∼ g iff f ( x j ) =
g ( x j ) for all x j ∈ X .
L 2 (Ω) / ∼ can be identified with R N , and thus can be given the normalized ℓ 1 norm:
[Equation]
[Equation]
Definition
Denote the preimage of π as π − 1 : f ′ (cid:55)→ { f ′ ∈ G ′ :
πf ′
= f ′ } .
Invoking the axiom of choice, define the inverse projection π − 1 : π G ′
→ G ′ by a choice function over the sets π − 1 ( π G ′ ) .
Remark:
Note that this inverse projection corresponds to some way of interpolating the
N sample points such that the output is in G ′ .
Although our definition implies the existence of such an interpolator, we leave its specification as an open problem.
Since Ω only permits discontinuities along a fixed Borel subset of [0 , 1] d , these boundaries can be specified a priori in the interpolator.
Since all functions in G ′ are bounded and continuous outside this set, the interpolator can be represented by a bounded continuous map, hence it is expressible by a DI-Net layer.
Definition 12 π generates a σ -algebra on G given by A = { π − 1 ( S ) : S ∈ L } , with L the σ -algebra of Lebesgue measurable sets on R N .
Because this σ -algebra depends on ϵ and the
Lipschitz constant of R via the point set discrepancy, we may write it as A ϵ, R .
Remark:
In this formulation, we let the tolerance ϵ and the Lipschitz constant of R dictate what subsets of G are measurable, and thus which measures on G are permitted.
However, if the desired measure ν is more fine-grained than what is permitted by A ϵ, R , then it is ν that should determine the number of QMC samples N , rather than ϵ or R .
We now state the following lemmas which will be used to prove our universal approximation theorem.
Lemma 13 There is a map ˜ R : π G → π G ′ such that
[Equation]
Proof Let g ( x ) =
| f
( x ) | for f ∈ G ∪ G ′ .
Because (13) applies to g ( x ), we have:
[Equation]
Eqn.
(17) also implies that for any f ∈ π G ∪ π G ′ , we have:
[Equation]
Combining (17) and (18), we obtain
[Equation]
By the triangle inequality and applying R :
[Equation]
For any f, g ∈ G such that πf = πg , (17) tells us that d ( f, g ) L 1 is at most ϵ/ 6( M + 2).
Recall M was defined such that d ( R [ f ] , R [ g ])
L 1 ≤ Md ( f, g ) L 1 for any R .
[Equation]
So defining:
[Equation]
[Equation]
we have
[Equation] [Equation]
Then by (20),
[Equation]
Lemma
Consider the extension of ˜ R to R N → R N in which each component of the output has the form:
[Equation]
Then any finite measure ν on the measurable space ( G , A ) induces a finite measure µ on ( R N , L ) , and (cid:82) R N | ˜ R j ( f ) | µ ( d f ) < ∞ for each j .
Proof Since the σ -algebra A on G is generated by π , the measure µ : µ ( πS )
=
ν ( S ) for all S ∈ A is finite and defined w.r.t.
the Lebesgue measurable sets on π G .
Since π G can be identified with a measurable subset of R N , µ can be naturally extended to R N .
Doing so makes it absolutely continuous w.r.t.
the Lebesgue measure on R N .
To show ˜ R j ( f ) is integrable, it is sufficient to show it is bounded and compactly supported.
G is bounded in the L 1 norm.
Thus by (17), π G is bounded in the normalized ℓ 1 norm.
The ℓ 1 norm in R N is strongly equivalent to the uniform norm, so there is some compact set [ − c, c ] N , c > 0 for which the extension of π G to R N vanishes, so supp( ˜ R j ( f )) ⊆
[ − c, c ] N .
Similarly, π G ′ is bounded in the ℓ 1 norm, hence there exists c ′ such that ˜ R j < c ′ for all j .
Lemma
For any finite measure µ absolutely continuous w.r.t.
the Lebesgue measure on
R n , J ∈ L 1 ( µ ) and ϵ > 0 , there is a network K such that: [Equation]
[Equation]
Proof The following construction is adapted from lu2017expressive.
Since J is integrable, there is a cube E =
[ − c, c ] n such that:
[Equation]
Case 1: J is non-negative on all of R n
Define the set under the graph of J | E :
[Equation]
[Equation] G E,J is compact in R n +1 , hence there is a finite cover of open rectangles { R ′ i } satisfying µ ( ∪ i R ′ i ) − µ ( G E,J ) <
ϵ 8 on R n .
Take their closures, and extend the sides of all rectangles indefinitely.
This results in a set of pairwise almost disjoint rectangles { R i } .
Taking only the rectangles R =
{ R i : µ [Equation] R i ∩ G E,J ) > 0 } results in a finite cover satisfying:
[Equation] [Equation]
This implies:
[Equation]
[Equation]
and also,
[Equation]
by the triangle inequality.
For each R i =
[ a i 1 , b i 1 ] × . . .
[ a in , b in ] ×
[ ζ i , ζ i + y i ], let X i be its first n components (i.e., the projection of R i onto R n ).
Then we have
[Equation]
Let Y ( f ) ≜ (cid:80) | R | i =1 y i 1 X
i ( f ).
By the triangle inequality,
[Equation]
by (30) and (36).
So it remains to construct K such that ∥K − Y ∥ 1 < ϵ 4 .
Because 1 X i is discontinuous at the boundary of the rectangle X i , it cannot be produced directly from a DI-Net (recall that all layers are continuous maps).
However, we can approximate it arbitrarily well with a piece-wise linear function that rapidly ramps from 0 to 1 at the boundary.
For fixed rectangle X i and δ ∈ (0 , 0 . 5), consider the inner rectangle X δ ⊂
X
i :
[Equation]
where we omit subscript j for clarity.
Letting b ′ i
=
b i − δ
( b i − a i ), define the function:
[Equation]
[Equation]
where ReLU ( x ) = max( x, 0).
T ( f ) is a piece-wise linear function that ramps from 0 at the boundary of X i to 1 within X δ , and vanishes outside X i .
Note that
[Equation]
[Equation]
if µ is the Lebesgue measure.
δ may need to be smaller under other measures, but this adjustment is independent of the input f
so it can be specified a priori .
Recall that the function we want to approximate is Y ( f )
= (cid:80) | R | i
=
y
i 1 X i ( f ).
We can build
NF-Net layers K : f
(cid:55)→ K ( f ) = (cid:80) | R | i =
y i T i ( f ), since this only involves linear combinations and ReLUs.
Then,
[Equation]
[Equation]
[Equation]
by (33).
And so by choosing:
[Equation]
we have our desired bound ∥K − Y ∥ 1 < [Equation] and thereby ∥ J − K∥ 1 < ϵ 2 .
Case 2: J is negative on some region of R n
Letting J + ( f ) = max(0 , J ( f )) and J − ( f ) = max(0 , − J ( f )), define:
[Equation]
As in (32), construct covers of rectangles R + over G + E,J and R − over G − E,J each with bound ϵ 16 and R n projections X + , X − .
Let:
[Equation]
We can derive an equivalent expression to (36):
[Equation]
Similarly to earlier [Equation] we use (30) and (55) to get:
[Equation]
Choosing T
+ i ( f ) and T − i ( f ) the piece-wise linear functions associated with X + i and X − i [Equation] and:
[Equation]
we have:
[Equation]
applying the triangle inequality,
[Equation]
[Equation]
[Equation] − (cid:13)(cid:13) [Equation] (cid:17)
[Equation]
By choosing: [Equation] and proceeding as before, we arrive at the same bounds ∥K − Y ∥ 1 < ϵ 4 and ∥ J − K∥ 1 < ϵ 2 .
Putting it all together, Algorithm C.1 implements the network logic for producing the function K .
Setup Input: target function J , L 1 tolerance ϵ/
Choose rectangles R + i =
[ a + i 1 , b
+ i 1 ] × . . .
[ a + in , b + in ] ×
[ ζ + i , ζ + i + y + i ] satisfying (32) and R − similarly δ + ← 12 (cid:16) 1 − (1 − ϵ 8 (cid:0) ∥
J + ∥ 1 + ϵ 16 (cid:1) − 1 ) 1 /n (cid:17) δ − ← 12 (cid:16) 1 − (1 − ϵ 8 (cid:0) ∥
J − ∥ 1 + ϵ 16 (cid:1) − 1 ) 1 /n (cid:17)
Inference Input: discretized input f = { f k } nk = x ← (0 , 0 , 1 , 0 , 0) for rectangle R + i ∈ R + do for dimension k ∈ 1 : n do x ← ( f k − b + ik + δ ( b + ik − a + ik ) , f k − a + ik , x 3 , x 4 , x 5 ) x ← ReLU ( x ) x ← ( δ − x 1 , δ − x 2 , x 3 , x 4 , x 5 ) x ← ReLU ( x ) x ← (0 , 0 , x 3 ( x 1 − x 2 ) /δ, x 4 , x 5 ) end x ← (0 , 0 , 1 , y
+ i x 3 + x 4 , x 5 ) end for rectangle R − i ∈ R − do for dimension k ∈ 1 : n do x ← ( f k − b − ik + δ ( b − ik − a − ik ) , f k − a − ik , x 3 , x 4 , x 5 ) . .
end x ← (0 , 0 , 1 , x 4 , y − i x 3 + x 5 ) end
Output: x 4 − x
We can provide x with access to f either through skip connections or by appending channels with the values { c + f k } nk =1 (which will be preserved under ReLU).
Theorem 16 (Maps between Single-Channel NFs)
For any Lipschitz continuous map
R : G → G ′ , any ϵ > 0 , and any finite measure ν w.r.t.
the measurable space ( G , A ϵ, R ) , there exists a DI-Net T that satisfies:
[Equation]
Proof If ν is not normalized, the discrepancy of our point set needs to be further divided by max { ν ( G ) , 1 } .
We assume for the remainder of this section that ν is normalized.
Perform the construction of Lemma 15 N times, each with a tolerance of ϵ/ 2 N .
Choose a partition of unity { ψ j } for which ψ j ( x k ) = δ jk , and output N channels with the values {K j ( f )
ψ j ( · ) } Nj =1 .
By summing these channels we obtain a network ˜ K that fully specifies the desired behavior of ˜ R : R N → R N , with combined error:
[Equation]
Thus, (cid:90)
G
[Equation]
N
[Equation]
By (18) we have:
[Equation]
By Lemma 13 we have:
[Equation]
And thus the network T = π − 1 ◦ ˜
K ◦ π gives us the desired bound:
[Equation]
Corollary 17 (Maps from NFs to vectors)
For any Lipschitz continuous map R :
G → R n , any ϵ > 0 , and any finite measure ν w.r.t.
the measurable space ( G , A ϵ, R ) , there exists a DI-Net T that satisfies:
[Equation]
Proof Let M 0 be the Lipschitz constant of R in the sense that d ( R [ f ] , R [ g ]) ℓ 1 ≤ M 0 d ( f, g ) L 1 .
Let M = max { M 0 , 1 } .
There exists ˜ R : π G → R n such that (cid:13)(cid:13)(cid:13)
[Equation]
[ f
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
As in Lemma 14, consider the extension of ˜ R to R N → R n in which each component of the output has the form:
[Equation]
Then for similar reasoning, ν on G induces a measure µ on R N that is finite and absolutely continuous w.r.t.
the Lebesgue measure, and (cid:82) R N | ˜ R j ( f ) | µ ( d f ) < ∞ for each j .
We construct our R N → R approximation n times with a tolerance of ϵ/ 2 n , such that:
[Equation]
Applying (17), we find that the network T =
˜
K ◦ π gives us the desired bound:
[Equation]
Corollary 18 (Maps from vectors to NFs)
For any Lipschitz continuous map R : R n →
G and any ϵ > 0 , there exists a DI-Net T that satisfies:
[Equation]
Proof
Define the map ˜ R : R n → π G ⊂ R N by ˜ R = π ◦ R .
Since ˜ R is bounded and compactly supported, (cid:82) R N | ˜
R i ( x ) | dx < ∞ for each i .
We construct a R n → R approximation N times each with a tolerance of ϵ/ 2 N , such that:
[Equation]
Applying (18), we find that the network T =
π − 1 ◦ ˜ K gives us the desired bound:
[Equation]
Denote the space of multi-channel NFs as F c = { f : Ω → R c : (cid:82) Ω ∥ f ∥ 1 dx < ∞ , f i ∈ G for each i } .
Denote the norm on this space as:
[Equation]
F 1 is identified with G .
Definition 19 (Concatenation) Concatenation is a map from two NF channels f
i , f j ∈
G to [ f i , f j ] ∈ F 2 .
The concatenation of NFs can be defined inductively to yield F n × F m →
F n + m for any n, m ∈ N .
Remark:
All maps F n × F m → F c can be expressed as a concatenation followed by a map F n + m → F c .
A map R n → F m is also equivalent to m maps R n → G followed by concatenation.
Thus, we need only characterize the maps that take one multi-channel NF as input.
Considering the maps F n → F m , we choose a lower discrepancy point set X on Ω such that the Koksma–Hlawka inequality yields a bound of ϵ/ 12 mn ( M + 2).
Let π project each component of the input to π G , and π − 1 inverts this projection under some choice function.
We take A ′ to be the product σ -algebra generated from this π : A ′ =
{ E 1 × · · · × E c :
E 1 , . .
, E c ∈ A } where A is the σ -algebra on G from Definition 12.
Corollary 20 (Maps between multi-channel NFs)
For any Lipschitz continuous map
R : F n → F m , any ϵ > 0 , and any finite measure ν w.r.t.
the measurable space ( F n , A ′ ϵ, R ) , there exists a DI-Net T that satisfies:
[Equation]
Proof The proof is very similar to that of Theorem 16.
Our network now requires nN maps from R mN → R each with error ϵ/ 2 mnN .
Summing the errors across all input and output channels yields our desired bound.
The multi-channel analogue of Corollary 17 is clear, and we state it here for completeness:
Corollary 21 (Maps from multi-channel NFs to vectors)
For any Lipschitz continuous map R : F n → R m , any ϵ > 0 , and any finite measure ν w.r.t.
the measurable space ( F n , A ′ ϵ, R ) , there exists a DI-Net T that satisfies:
[Equation]
C.2. Convergent Empirical Gradients
Here we establish that backpropagation is sound, by proving the convergence of derivatives through the empirical measure in the following sense:
Theorem 22 (Convergent Empirical Gradients)
A DI-Net permits backpropagation of its outputs with respect to its input as well as all its learnable parameters.
Under a sequence of discretizations tending to 0 discrepancy, the gradients of each layer converge to the appropriate derivative under the measure on Ω .
To establish convergence in the number of QMC samples N , let us suppose we have a family of sequences { X N } N ∈ N : X
N = { x j } Nj =1 on Ω whose discrepancy converges to 0 as
N
→ ∞ .
We note that no convergence statement is needed if the layer does not perform QMC integration.
This includes layers which take R n as input, as well as point-wise transformations.
Then the (sub)derivatives with respect to inputs and parameters need only be well-defined at each point of the output in order to enable backpropagation.
Thus we consider a layer H ϕ which takes an NF f as input.
By (3), it can be written as:
[Equation]
We write the QMC estimate of H ϕ
[ f ] under X N as:
[Equation]
and we call the derivatives of this QMC estimate the empirical derivatives of H ϕ
[ f ].
We are interested in proving the convergence of the empirical gradients of H ϕ
[ f ] with respect to its input f as well as the parameters ϕ of the layer.
As in Appendix C.1, we write πf =
{ f ( x j ) } Nj =1 .
Theorem 23 For any DI-Net layer H ϕ which takes an NF f ∈ F m as input, the empirical gradient of H ϕ
[ f ] w.r.t.
its parameters ϕ is convergent in N :
[Equation]
Additionally, the empirical gradient of H ϕ
[ f ] w.r.t.
its discretized input πf is convergent in N :
[Equation]
Proof
NF to Vector: gradients w.r.t. parameters
Consider the case of a layer H ϕ : F m →
R n .
As in (3), such a layer can be expressed as:
[Equation]
for some h parameterized by ϕ , with weak derivatives up to order | α | taken with respect to each channel.
If ϕ = ( ϕ 1 , . . . , ϕ K ), then denote ϕ + [Equation] [Equation]
. . , ϕ
[Equation] [Equation]
[Equation]
where (87) follows by eqn.
(84) and the Moore-Osgood theorem.
Thus the empirical gradient converges to the Jacobian of H ϕ w.r.t.
each parameter, which is finite by assumption.
NF to NF: gradients w.r.t. parameters
A DI-Net layer ¯ H ϕ ′ :
L 2 (Ω) →
L 2 (Ω) can be expressed as:
[Equation]
We can in fact follow the same steps as the NF to Vector case above, to arrive at:
[Equation]
with equality at each point x ′ ∈ Ω and channel of the output NF.
NF input: gradients w.r.t. inputs
Here we combine the NF to vector and NF to NF cases for brevity.
For fixed ˜ x ∈ Ω, the empirical derivative of H ϕ w.r.t.
f (˜ x ) can be written:
[Equation]
where ψ ˜ x,N is any function in W | α | , 1 (Ω) that is 1 at ˜ x and 0 at every x j ̸
= ˜ x , and whose weak derivatives are 0 at every x j .
As an example, take the bump function which vanishes outside a small neighborhood of ˜ x and smoothly ramps to 1 on a smaller neighborhood of ˜ x , making its weak derivatives 0 at ˜ x .
By (6) we know that the sequences
[Equation]
converge uniformly in N to 0 for any τ > 0, where we can use the ℓ 1 norm for vector outputs or the L 1 norm for NF outputs.
So for any ϵ > 0 and any τ > 0, we can choose N 0 large enough such that for any N > N 0 :
[Equation]
and
[Equation]
Then,
[Equation] [Equation]
by the triangle inequality,
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:16)
[Equation]
(98) hence (cid:13)(cid:13)(cid:13)(cid:16)
ˆ H Nϕ [ f + τψ ˜ x,N ] − ˆ H Nϕ [ f ] (cid:17) − ( H ϕ [
[Equation]
+ τψ ˜ x,N ] − H ϕ
[ f ]) (cid:13)(cid:13)(cid:13) converges uniformly to 0.
Since the distance between two vectors is 0 iff they are the same, we can write:
[Equation] [Equation]
By the Moore-Osgood theorem,
[Equation]
The limit on the right hand side is finite due to Fr´echet differentiability of H ϕ at all f .
Choose a family { Ψ N } N ∈ N of decreasing sequences of bump functions around each x j , Ψ N = { ψ x j ,N } Nj =1 .
An example of such a family is the (appropriately designed) partitions of unity with N elements.
Then the empirical gradient w.r.t.
πf converges to the limit of the Gateaux derivatives of H ϕ w.r.t.
the bump function sequences in this family.
Theorem 24 Chained empirical derivatives converge in N to the chained Gateaux derivatives.
Proof Consider a two-layer DI-Net with NF input f
(cid:55)→ ( H θ ◦ H ϕ )
[ f ].
For the case of derivatives w.r.t.
the input, we would like to show the analogue of (102):
[Equation]
where the bump function ψ ˜ x,N is defined similarly (1 at ˜ x and 0 at each x j ̸ = ˜ x ).
[Equation]
as in (93).
By (6) we know (cid:13) (cid:13)(cid:13)
H θ (cid:104)
ˆ H Nϕ [ f + τψ ˜ x
[Equation]
N ] (cid:105) − ˆ H θ (cid:104)
ˆ H Nϕ [ f + τψ ˜ x
[Equation]
N ] (cid:105)(cid:13)(cid:13)(cid:13) converges to 0 in N for all τ > 0 (where we can use the ℓ 1 norm for vector outputs or the L 1 norm for NF outputs), as does (cid:13)(cid:13)(cid:13)
H θ (cid:104)
ˆ H Nϕ [ f ] (cid:105) − ˆ H θ (cid:104)
ˆ H Nϕ [ f ] (cid:105)(cid:13)(cid:13)(cid:13)
L 1 .
Reasoning as in (94)-(101), we have:
[Equation]
Note that
[Equation]
Then we complete the equality in (103) as follows:
[Equation]
by the chain rule for Gateaux derivatives
[Equation]
The case of derivatives w.r.t.
parameters is straightforward.
In the same way we used (94)-(101) to obtain (108), we have:
[Equation]
Inductively, the chained derivatives extend to an arbitrary number of layers.
Since the properties of DI-Net layers extend to loss functions on DI-Nets [Equation] we can treat a loss function similarly to a layer [Equation]
and write:
[Equation]
where g ′ can be some other input to the loss function such as ground truth labels.
Thus [Equation] we can state the following result.
Corollary
The gradients of a DI-Net’s loss function w.r.t.
its inputs and all its parameters are convergent in the number of QMC sample points.
Appendix D. DI-Net Layers Derived from Grid-Based Networks
We use c in to denote the number of channels of an input NF and c out to denote the number of channels of an output NF.
Convolution
For a measurable S ⊂ Ω and a polynomial basis { p j,ϕ } j ≥ 0 that spans L 2 ( S ),
S is the support of a polynomial convolutional kernel K ϕ : Ω × Ω → R defined by:
[Equation]
for some chosen n ∈ N .
A convolution is the linear map H ϕ :
L 2 (Ω) →
L 2 (Ω) given by:
[Equation]
An MLP convolution is defined similarly except the kernel becomes ˜ K ϕ ( x, x ′ ) =
MLP( x − x ′ ; ϕ ) in the non-zero case.
While MLP kernels are favored over polynomial kernels in many applications due to their expressive power (Wang et al., 2021), polynomial bases can be are amenable to the structures often found in discrete networks DI-Nets can also be initialized directly from pre-trained discrete networks. used to construct filters satisfying desired properties such as group equivariance (Cohen and Welling, 2016a,b), k -Lipschitz continuity, or boundary conditions.
A multi-channel convolution layer aggregates information locally and across channels.
It has c in c out learned filters K ij , which are defined on some support S which may be a ball or orthotope.
The layer also learns scalar biases b j for each output channel:
[Equation]
with ∗ the continuous convolution as in (124).
To transfer weights from a discrete convolutional layer, K can be parameterized as a rectangular B-spline surface that interpolates the weights (Figure D.2 left).
To replicate the behavior of a discrete convolution layer with odd kernel size, S is zero-centered.
For even kernel size, we shift S by half the dimensions of a pixel.
We use a 2nd order B-spline for 3 × 3 filters and 3rd order for larger filters.
We use deBoor’s algorithm to evaluate the spline at intermediate points.
Different padding behaviors from the discrete case are treated differently.
Zero-padding is replicated by scaling H [ f ]( x ) by | ( S + x ) ∩ Ω | S + x where S + x is the kernel support S translated by x .
For reflection padding, the value of the NF at points outside its domain are calculated by reflection.
For no padding, the NF’s domain is reduced accordingly, dropping all sample points that are no longer on the new domain.
Multi-scale architectures
Many discretizations permit multi-scale structures by subsampling the discretization.
Under QMC, downsampling is easily implemented by truncating the list of coordinates in the low-discrepancy sequence to the desired number of terms, as the truncated sequence is itself low-discrepancy.
Similarly, upsampling is implemented by extending the low-discrepancy sequence to the desired number of terms, then perform-
ing nearest neighbor interpolation 7 .
Since downsampling and upsampling are both specified with respect to the same discretization, DI-Nets can use multi-scale structures with residual or skip connections (Figure D.2 right).
Linear combinations of channels
Linear combinations of channels mimic the function of 1 × 1 convolutional layers in conventional networks.
For learned scalar weights W ij and biases b j :
[Equation]
for all x ∈ Ω. These weights and biases can be straightforwardly copied from a 1 ×convolutional layer to obtain the same behavior.
One can also adopt a normalized version, sometimes used in attention-based networks:
[Equation]
Normalization
All forms of layer normalization readily generalize to the continuous setting by estimating the statistics of each channel with QMC integration, then applying point-wise operations.
These layers typically rescale each channel to have some mean m i and standard deviation s i .
[Equation]
where we assume dx is normalized and ϵ > 0 is a small constant.
Just as in the discrete case, µ i and σ 2 i can be a moving average of the means and variances observed over the course of training different NFs, and they can also be averaged over a minibatch of NFs (batch normalization) or calculated per datapoint (instance normalization).
Mean m i and standard deviation s i can be learned directly (batch normalization), conditioned on other data (adaptive instance normalization), or fixed at 0 and 1 respectively (instance normalization).
Max pooling
Max pooling is only a well-defined DI-Net layer if each NF channel is in
L ∞ (Ω).
Assuming this is the case, there are two natural generalizations of the max pooling layer to a collection of points: 1) assigning each point to the maximum of its k nearest neighbors, and 2) taking the maximum value within a fixed-size window around each point.
However, both of these specifications change the output’s behavior as the density of points increases.
In the first case, nearest neighbors become closer together so pooling occurs over smaller regions where there is less total variation in the NF.
In the second case, the empirical maximum increases monotonically as the NF is sampled more finely within each window.
Because we may want to change the number of sampling points on the fly, both of these behaviors are detrimental.
If we consider the role of max pooling as a layer that shuttles gradients through a strong local activation, then it is sufficient to use a fixed-size window with some scaling factor that mitigates the impact of changing the number of sampling points.
Consider the following simplistic model: assume each point in a given patch of an NF channel is an i.i.d.
sample from U ([ − b, b ]).
Then the maximum of N samples { f i ( x j ) } Nj =1 is on average N − 1 N +1 b .
So we can achieve an “unbiased” max pooling layer by taking the maximum value observed in each window and scaling it by N +1 N − 1 (if N = 1 or our empirical maximum is negative then we simply return the maximum), then (optionally) multiplying a constant to match the discrete layer.
To replicate the behavior of a discrete max pooling layer with even kernel size, we shift the window by half the dimensions of a pixel, just as in the case of convolution.
Tokenization
To tokenize an NF, we choose a finite set of non-overlapping regions ω j ⊂ Ω of equal measure such that ∪ j ω j = Ω. We apply the indicator function of each set to each channel f i .
An embedding of each f i | ω j into R n can be obtained by taking its inner product with a polynomial function whose basis spans each L 2 ( ω j ).
To replicate a pre-trained embedding matrix, we interpolate the weights with B-spline surfaces.
Average pooling
An average pooling layer performs a continuous convolution with a box filter, followed by downsampling.
To reproduce a discrete average pooling with even kernels, the box filter is shifted, similarly to max pooling.
An adaptive average pooling layer can be replicated by tokenizing the NF and taking the mean of each token to produce a vector of the desired size.
Parametric functions in L 2 (Ω)
There are several choices of parametric functions in
L 2 (Ω).
Such functions can enable an NF to be modulated by taking its elementwise product with the function, or for global information to be aggregated in a learned manner by taking its inner product with the function.
If Ω is a subset of [ a, b ] d , one can use a separable basis defined by the product of rescaled 1D Legendre polynomials along each dimension.
If Ω is a d -ball, we can use the Zernike polynomial basis.
For any Ω, the function can be represented as an MLP.
Attention layer
There are various ways to replicate the functionality of an attention layer.
Here we present an approach that preserves the domain.
For some d k ∈ N consider a self-attention layer with c in d k parametric functions q ij ∈ L 2 (Ω), c in d k parametric functions k ij ∈
L 2 (Ω), and a convolution with d k output channels, produce the output NF g as:
[Equation]
A cross-attention layer generates queries from a second input NF.
A multihead-attention layer generates several sets of ( Q, K, V ) triplets and takes the softmax of each set separately.
Activation Layer
Pointwise nonlinearities are identical to the discrete case.
We use rectified linear units (ReLU) for all our models.
Data augmentation
Most data augmentation techniques, including spatial transformations, point-wise functions and normalizations, translate naturally to NFs.
Furthermore, spatial transformations are efficient and do not incur the usual cost of interpolating back to the grid.
This suggests that DI-Nets may benefit from new data augmentation methods such as adding Gaussian noise to the coordinates of QMC sample points.
Appendix E. NF Classification
Algorithm E.2 Training for Classification
Input: network T θ , NF/label dataset D , classifier loss L , discretization x
For step s ∈ 1 : N steps do
• NFs f i , labels y
i ← minibatch( D ) • Label estimates ˆ y i ← T θ ( f i ; x ) • Backpropagate L (ˆ y i , y i ) to update θ Output
T θ
[Table]
We perform classification on a dataset of 8,400 NFs fit to a subset of ImageNet1k (Deng et al., 2009), with 700 samples from each of 12 superclasses (Engstrom et al., 2019).
The 12 superclasses (dog, structure/construction, bird, clothing, wheeled vehicle, reptile, carnivore, insect, musical instrument, food, furniture, primate) are based on the big 12 dataset (Engstrom et al., 2019), which is in turn derived from the WordNet hierarchy.
For each class we train on 500 SIRENs (Sitzmann et al., 2020b) and evaluate on
Gaussian Fourier feature networks (Tancik et al., 2020b).
Note that training and testing on different NF types is not possible for hypernetwork or modulation-based approaches.
We fit SIREN (Sitzmann et al., 2020b) to each image in ImageNet using 5 fully connected layers with 256 channels and sine non-linearities, trained for 2000 steps with an Adam optimizer at a learning rate of 10 − 4 .
It takes coordinates on [ − 1 , 1] 2 and produces RGB values in [ − 1 , 1] 3 .
We fit Gaussian Fourier feature (Tancik et al., 2020b) networks using 4 fully connected layers with 256 channels with ReLU activations.
It takes coordinates on [0 , 1] 2 and produces RGB values in [0 , 1] 3 .
The average pixel-wise error of SIREN is 3 · 10 − 4 ± 2 · 10 − 4 , compared to 1 . 6 ·
10 − 2 ± 8 · 10 − 3 for Gaussian Fourier feature networks.
The difference in quality is visible at high resolution, but indistinguishable at low resolution (Fig. E.3).
We train DI-Nets with 2 and 4 MLP convolutional layers (DI-Net-2 and DI-Net-4), as well as convolutional neural networks (CNNs) with equivalent architectures.
Each network is trained for 8K iterations with an AdamW optimizer (Loshchilov and Hutter, 2017) and a learning rate of 10 − 3 .
DI-Net-2 uses strided MLP convolutions, a global average pooling layer, then two fully connected layers.
DI-Net-4 adds a residual block with two MLP convolutions after the strided convolutions.
At training time, the CNNs sample NFs along the 32 × 32 grid while DI-Nets sample 1024 points generated from a scrambled Sobol sequence.
We evaluate each model with top-3 accuracy at the same resolution as well as at higher resolution (64 × 64 or 4096 points).
DI-Nets somewhat underperform their CNN counterparts (Table E.2), and the gap in performance is larger for the deeper models.
However, our discretization invariant model better generalizes to higher-resolution images, where it outperforms CNNs on images at twice the resolution and has a very small drop in performance compared to the resolution it was trained on.
Appendix F. NF Semantic Segmentation
We perform pixel-wise segmentation on SIRENs fit to street view images from Cityscapes (Cordts et al., 2016), grouping the segmentation labels into 7 categories.
SIREN is trained on Cityscapes images for 2500 steps, using the same architecture and settings as ImageNet.
The average pixel-wise error is 3 .
6 · 10 − 4 ± 1 .
· 10 − 4 .
We train on 2975 NFs with coarsely annotated segmentations only, and test on 500 NFs with both coarse and fine annotations (Fig. F.4).
Discrete models sample the 48 × 96 grid, and the DI-Nets sample 4608 points.
We compare the performance of 3 and 5 layer DI-Nets and fully convolutional networks (FCNs).
DI-Net-3 uses three MLP convolutional layers at the same resolution.
DI-Net-5 uses a strided MLP convolution to perform downsampling and nearest neighbor interpolation for upsampling.
There is a residual connection between the higher resolution layers. Networks
Input: network T θ , NF/label dataset D with dense coordinate-label pairs, task-specific loss
L
For step s ∈ 1 : N steps do • NFs f i , coordinates-label pairs ( x ij , y ij ) ← minibatch( D ) • Output NFs g i ← T θ [ f i ] • Point label estimates ˆ y ij ← g i ( x ij ) • Backpropagate L (ˆ y ij , y ij ) to update θ trained network T θ are trained for 10K iterations with a learning rate of 10 − 3 .
We evaluate each model based on mean intersection over union (mIoU) and pixel-wise accuracy (PixAcc)
[Table]
[Table] DI-Net-3 outperforms the equivalent FCN, and avoids confusing features such as shadows and road markings (Fig. F.4). However
[Table] the performance deteriorates when we make the model deeper with a downsampling and upsampling layer (DI-Net-5) [Table] and the FCN surpasses both DI-Nets (Table F.3).
This echoes the difficulty in scaling DI-Nets observed in classification, and suggests that additional innovations in architecture or training methods are needed for deeper DI-Nets.
[Table]
[Table]
Appendix G. Analysis G.1. Initialization with discrete networks
When a DI-Net is initialized with a large pre-trained discrete network, its outputs are similar 8 by construction.
However, the behavior of the pre-trained network is not preserved when the DI-Net switches to QMC – even tiny perturbations from the regular grid are sufficient to change a classifier’s predictions.
Although the effect on the output of a single layer is much lower than the signal, small differences in each layer accumulate to exert a large influence on the final output.
In addition, we find that once grid sampling is abandoned, large DI-Nets cannot easily be fine-tuned to restore the behavior of the discrete network used to initialize it.
This suggests that the optimization landscape of DI-Nets may differ substantially from that of discrete networks, and perhaps the approach for learning maps between L 2 functions requires fundamentally different strategies than those for learning maps between discrete functions, even when the functions represent the same underlying signals.
Figure G.6 illustrates that the behavior of a pre-trained DI-Net is not preserved when its sampling scheme switches to QMC.
For a DI-Net initialized with a truncated EfficientNet, its outputs deviate rapidly as points shift from a regular grid to a low discrepancy sequence.
In Table G.4 and G.5, we illustrate that DI-Net initialized with a large pre-trained discrete network does not match the performance of the original model when fine-tuned with QMC sampling.
We use a truncated version of EfficientNet (Tan and Le, 2019) for classification, and fine-tune for 200 samples per class.
For segmentation we use a truncated version of ConvNexT-UPerNet (Liu et al., 2022), fine-tuning with 1000 samples.
We also find that the output of an NF-Net is less stable under changing sampling resolution with a grid pattern (Fig. G.5).
While the output of a network with QMC sampling converges at high resolution, the grid sampling scheme has unstable outputs until very high resolution.
Only the grids that overlap each other (resolutions in powers of two) produce similar activations.
Our preliminary experience with DI-Nets highlights the need for improved sampling schemes and parameterizations that will allow large continuous-domain neural networks to learn effectively.
Stable, scalable methods are needed to realize DI-Nets’ full potential for continuous data analysis.
G.2. Computational Complexity
The DI-Net’s complexity is similar to that of a discrete model with an equivalent architecture.
In general time and memory both scale linearly with the number of sample points (regardless of the dimensionality of Ω), as well as with network depth and width.
Implemented naively, the computational cost of the continuous convolution is quadratic in the number of sample points, as it must calculate weights separately for each neighboring pair of points.
We can reduce this to a linear cost by specifying a N bin Voronoi partition of the kernel support B , then using the value of the kernel at each seed point for all points in its cell.
Thus the kernel need only be evaluated N bin times regardless of the number of sample points.
Additionally N bin can be modified during training and inference.
DI-Net-4 performs a forward pass on a batch of 48 images in 96 ± 4ms on a single
NVIDIA RTX 2080 Ti GPU.
Appendix H. Future Directions Scaling convolutional DI-Nets
Our initial experiments suggest that convolutional DINets do not scale well in depth. We suspect that within a CNN-like architecture, the gradients of discrete convolutional layers with respect to kernel parameters have much smoother optimization landscapes over large networks relative to continuous convolutional layers parameterized with MLPs or coefficients of a polynomial basis.
It is then no surprise that implementations of neural networks with continuous convolutions do not simply substitute the convolutional layers in a standard CNN architecture, but also make use of a variety of additional techniques (Qi et al., 2017; Wang et al., 2021; Boulch, 2019) which would likely be helpful for scaling convolutional DI-Nets.
Parameterization of output NFs
In this work we assume that a DI-Net that produces an NF specifies the output discretization a priori , but some applications may need the output to be sampled several times at different discretizations. It is inefficient to re-evaluate the entire network in such cases, and we propose two solutions for future work.
One method can store the last few layers of the network alongside the input activation, and adapt the discretizations as needed in these last few layers only.
A second approach can treat the discretized outputs of DI-Net as parameters of the output NF in the manner of Vora et al. (2021), which would maintain interoperability of the entire framework.
Extending DI-Net to high discrepancy sequences
In many applications, there are large regions of the domain that are less informative for the task of interest.
For example, most of the information in 3D scenes is concentrated at object surfaces, so DI-Nets should not need to process a NeRF by densely sampling all 5 dimensions.
Moreover, ground truth labels for dense prediction tasks may only be available along a high discrepancy discretization.
Such a discretization can be handled by quadrature, but more work is required to design efficient quadrature methods within a neural network.
Additional techniques such as learned coordinate transformations or learned discretizations may also be helpful for extending our model to extreme discretizations or highly non-uniform measures. Error propagation When an NF does not faithfully represent the underlying data, it is important to characterize the influence on DI-Net’s output. In the worst case, these deviations are adversarial examples, and robustness techniques for discrete networks can also be applied to DI-Net.
But what can we say about typical deviations of NFs?
Future work should analyze patterns in the mistakes that different types of NFs make, and how to make DI-Nets robust to these.