Original: G ENERALIZED F OURIER F EATURES FOR C OORDINATE -B ASED L EARNING OF F UNCTIONS ON M ANIFOLDS
Revised: G ENERALIZED F OURIER F EATURES FOR C OORDINATE -B ASED L EARNING OF F UNCTIONS ON M ANIFOLDS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, positional encoding of input coordinates has been found crucial to enable learning of high-frequency functions with multilayer perceptrons taking lowdimensional coordinate values.
Revised: Recently, positional encoding of input coordinates has been found crucial to enable learning of high-frequency functions with multilayer perceptrons taking lowdimensional coordinate values.

Original: In this setting, sinusoids are typically used as a basis for the encoding, which is commonly referred to as “Fourier Features”.
Revised: In this setting, sinusoids are typically used as a basis for the encoding, which is commonly referred to as “Fourier Features”.

Original: However, using sinusoids as a basis assumes that the input coordinates lie on Euclidean space.
Revised: However, using sinusoids as a basis assumes that the input coordinates lie on Euclidean space.

Original: In this work, we generalize positional encoding with Fourier features to non-Euclidean manifolds.
Revised: In this work, we generalize positional encoding with Fourier features to non-Euclidean manifolds.

Original: We ﬁnd appropriate bases for positional encoding on manifolds through generalizations of Fourier series.
Revised: We find appropriate bases for positional encoding on manifolds through generalizations of Fourier series.

Original: By ensuring the encodings lie on a hypersphere and that the appropriate shifts on the manifold preserve innerproducts between encodings, our model approximates convolutions on the manifold, according to the neural tangent kernel (NTK) assumptions.
Revised: By ensuring the encodings lie on a hypersphere and that the appropriate shifts on the manifold preserve innerproducts between encodings, our model approximates convolutions on the manifold, according to the neural tangent kernel (NTK) assumptions.

Original: We demonstrate our method on various tasks on different manifolds: 1) learning panoramas on the sphere, 2) learning probability distributions on the rotation manifold, 3) learning neural radiance ﬁelds on the product of cube and sphere, and 4) learning light ﬁelds represented as the product of spheres.
Revised: We demonstrate our method on various tasks on different manifolds: 1) learning panoramas on the sphere, 2) learning probability distributions on the rotation manifold, 3) learning neural radiance fields on the product of cube and sphere, and 4) learning light fields represented as the product of spheres.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent breakthroughs on learning representations of 3D shapes (Mescheder et al., 2019; Park et al., 2019; Sitzmann et al., 2019) or scenes (Mildenhall et al., 2020) employ the so-called “coordinatebased” networks, which take low-dimensional coordinates as inputs and approximate a continuous function.
Revised: Recent breakthroughs on learning representations of 3D shapes (Mescheder et al., 2019; Park et al., 2019; Sitzmann et al., 2019) or scenes (Mildenhall et al., 2020) employ the so-called “coordinatebased” networks, which take low-dimensional coordinates as inputs and approximate a continuous function.

Original: These are sometimes called “implicit models”, when the approximated function implicitly represents the desired output; a typical example is using a signed distance function to represent a 3D shape (Park et al., 2019).
Revised: These are sometimes called “implicit models”, when the approximated function implicitly represents the desired output; a typical example is using a signed distance function to represent a 3D shape (Park et al., 2019).

Original: Perhaps the most important recent advancement in this line of research is NeRF (Mildenhall et al., 2020).
Revised: Perhaps the most important recent advancement in this line of research is NeRF (Mildenhall et al., 2020).

Original: One of the reasons for NeRF’s impressive performance is the positional encoding of input coordinates using sinusoidals of various frequencies, a technique that has been widely adopted (Liu et al., 2020; Schwarz et al., 2020; Yariv et al., 2020) and studied (Tancik et al., 2020; Zheng et al., 2021).
Revised: One of the reasons for NeRF’s impressive performance is the positional encoding of input coordinates using sinusoidals of various frequencies, a technique that has been widely adopted (Liu et al., 2020; Schwarz et al., 2020; Yariv et al., 2020) and studied (Tancik et al., 2020; Zheng et al., 2021).

Original: The sinusoidals typically used for positional encoding are elements of orthonormal bases for functions on Euclidean spaces.
Revised: The sinusoidals typically used for positional encoding are elements of orthonormal bases for functions on Euclidean spaces.

Original: Our key observation is that, to generalize this idea to non-Euclidean manifolds, we should use orthonormal basis functions on the manifold.
Revised: Our key observation is that, to generalize this idea to non-Euclidean manifolds, we should use orthonormal basis functions on the manifold.

Original: Figure 1 (middle) illustrates the evaluation of the Euclidean basis functions on spherical coordinates, which break orthogonality, results in uneven frequency distribution and singularities near the poles.
Revised: Figure 1 (middle) illustrates the evaluation of the Euclidean basis functions on spherical coordinates, which break orthogonality, results in uneven frequency distribution and singularities near the poles.

Original: The spherical harmonics are orthonormal basis functions for the sphere and do not exhibit these undesired properties.
Revised: The spherical harmonics are orthonormal basis functions for the sphere and do not exhibit these undesired properties.

Original: NeRF (Mildenhall et al., 2020) and subsequent works avoid the problem just described by parametrizing the view direction as a unit vector in R 3 instead of two angles.
Revised: NeRF (Mildenhall et al., 2020) and subsequent works avoid the problem just described by parametrizing the view direction as a unit vector in R 3 instead of two angles.

Original: We argue that this overparametrization unnecessarily increases the dimensionality of the problem, since the number of basis functions for a ﬁxed bandwidth b grows exponentially with the number of dimensions d .
Revised: We argue that this overparametrization unnecessarily increases the dimensionality of the problem, since the number of basis functions for a fixed bandwidth b grows exponentially with the number of dimensions d .

Original: Another relevant example is IPDF (Murphy et al., 2021), which takes inputs on the 3-dimensional manifold SO (3) in the form of a ﬂattened rotation matrix (a vector in R 9 ), and applies positional encoding by sparsely sampling the space of
Revised: Another relevant example is IPDF (Murphy et al., 2021), which takes inputs on the 3-dimensional manifold SO (3) in the form of a flattened rotation matrix (a vector in R 9 ), and applies positional encoding by sparsely sampling the space of

Original: O ( b 9 ) basis elements.
Revised: O ( b 9 ) basis elements.

Original: We show that our approach improve results for both NeRF and IPDF.
Revised: We show that our approach improve results for both NeRF and IPDF.

Original: Our basic idea is to ﬁnd an orthonormal basis for the space of functions on the manifold of interest, and introduce principled methods for choosing a subset of the (possibly inﬁnite) basis elements on which the input coordinates are evaluated.
Revised: Our basic idea is to find an orthonormal basis for the space of functions on the manifold of interest, and introduce principled methods for choosing a subset of the (possibly infinite) basis elements on which the input coordinates are evaluated.

Original: We describe mathematical techniques to obtain bases for large classes of manifolds and show experiments on a variety of them.
Revised: We describe mathematical techniques to obtain bases for large classes of manifolds and show experiments on a variety of them.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We introduce a principled way to apply positional encoding for coordinate-based learning of high-frequency functions on manifolds.
Revised: • We introduce a principled way to apply positional encoding for coordinate-based learning of high-frequency functions on manifolds.

Original: • We prove that our approach is shift-invariant under the light of the neural tangent kernel (NTK) theory (Jacot et al., 2018; Tancik et al., 2020) and for the appropriate “shift” on the manifold, which implies a manifold convolutional behavior.
Revised: • We prove that our approach is shift-invariant under the light of the neural tangent kernel (NTK) theory (Jacot et al., 2018; Tancik et al., 2020) and for the appropriate “shift” on the manifold, which implies a manifold convolutional behavior.

Original: • Our experiments show the advantages of the proposed methodology on different applications and manifolds: 1) learning panoramas on the sphere, 2) learning probability distributions on the rotation manifold, 3) learning neural radiance ﬁelds on the product of cube and sphere, and 4) learning light ﬁelds represented as the product of spheres.
Revised: • Our experiments show the advantages of the proposed methodology on different applications and manifolds: 1) learning panoramas on the sphere, 2) learning probability distributions on the rotation manifold, 3) learning neural radiance fields on the product of cube and sphere, and 4) learning light fields represented as the product of spheres.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The idea of positional encoding using a Fourier basis appeared as early as Rahimi & Recht (2008).
Revised: The idea of positional encoding using a Fourier basis appeared as early as Rahimi & Recht (2008).

Original: They introduced the so-called random Fourier features to accelerate training of kernel methods.
Revised: They introduced the so-called random Fourier features to accelerate training of kernel methods.

Original: The idea is to approximate a shift-invariant kernel with random directions sampled from its Fourier transform, on which the inputs are evaluated.
Revised: The idea is to approximate a shift-invariant kernel with random directions sampled from its Fourier transform, on which the inputs are evaluated.

Original: More recently, positional encoding has become popular for sequence modeling in natural language processing and often appears in attention layers and transformers as a way to encode the order of the input tokens.
Revised: More recently, positional encoding has become popular for sequence modeling in natural language processing and often appears in attention layers and transformers as a way to encode the order of the input tokens.

Original: Gehring et al.
Revised: Gehring et al.

Original: (2017) use a learned embedding while Vaswani et al.
Revised: (2017) use a learned embedding while Vaswani et al.

Original: (2017) use the Fourier basis for encoding input token positions.
Revised: (2017) use the Fourier basis for encoding input token positions.

Original: Alternative approaches were introduced by Xu et al.
Revised: Alternative approaches were introduced by Xu et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: Rahaman et al.
Revised: Rahaman et al.

Original: (2019) demonstrated that neural networks tend to learn low frequencies more easily than high frequencies, a property called “spectral bias”.
Revised: (2019) demonstrated that neural networks tend to learn low frequencies more easily than high frequencies, a property called “spectral bias”.

Original: It also shows, for a simple task of approximating an 1D function, that projecting the input coordinate into a basis of sinusoids facilitates learning high frequencies.
Revised: It also shows, for a simple task of approximating an 1D function, that projecting the input coordinate into a basis of sinusoids facilitates learning high frequencies.

Original: NeRF (Mildenhall et al., 2020) reproduced this ﬁnding in the challenging task of photorealistic novel view synthesis from a collection of images.
Revised: NeRF (Mildenhall et al., 2020) reproduced this finding in the challenging task of photorealistic novel view synthesis from a collection of images.

Original: They showed that positional encoding is crucial to achieve photorealism, and signiﬁcantly outperformed the previous state of the art, which included other coordinate-based MLPs (Sitzmann et al., 2019).
Revised: They showed that positional encoding is crucial to achieving photorealism, and significantly outperformed the previous state of the art, which included other coordinate-based MLPs (Sitzmann et al., 2019).

Original: Concurrently, Zhong et al.
Revised: Concurrently, Zhong et al.

Original: (2020) demonstrated the usefulness of Fourier encoding for reconstruction of 3D protein complexes.
Revised: (2020) demonstrated the usefulness of Fourier encoding for reconstruction of 3D protein complexes.

Original: Tancik et al.
Revised: Tancik et al.

Original: (2020) conducted an in-depth study of positional encoding for coordinate-based MLPs.
Revised: (2020) conducted an in-depth study of positional encoding for coordinate-based MLPs.

Original: They show that, under the Neural Tangent Kernel (NTK) theory (Jacot et al., 2018), the positional encoding using a sinusoidal basis results in a stationary kernel, which in turn can be interpreted as a convolutional (shift-invariant) reconstruction ﬁlter, desirable for signals on Euclidean spaces.
Revised: They show that, under the Neural Tangent Kernel (NTK) theory (Jacot et al., 2018), the positional encoding using a sinusoidal basis results in a stationary kernel, which in turn can be interpreted as a convolutional (shift-invariant) reconstruction filter, desirable for signals on Euclidean spaces.

Original: The theory also explains the “spectral bias” (Rahaman et al., 2019) of these models via the rapidly decaying NTK eigenvalues for MLPs.
Revised: The theory also explains the “spectral bias” (Rahaman et al., 2019) of these models via the rapidly decaying NTK eigenvalues for MLPs.

Original: In practice, Tancik et al.
Revised: In practice, Tancik et al.

Original: (2020) propose randomly sampling the frequencies of the Fourier basis elements for the encoding, and show that it outperforms previous coordinate-based MLPs with no positional encoding (Mescheder et al., 2019), or axis-aligned positional encoding (Mildenhall et al., 2020).
Revised: (2020) propose randomly sampling the frequencies of the Fourier basis elements for the encoding, and show that it outperforms previous coordinate-based MLPs with no positional encoding (Mescheder et al., 2019), or axis-aligned positional encoding (Mildenhall et al., 2020).

Original: While groundbreaking results and insightful theoretical developments have been demonstrated, most of the attention so far has been focused on coordinate-based MLPs for functions on Euclidean spaces.
Revised: While groundbreaking results and insightful theoretical developments have been demonstrated, most of the attention so far has been focused on coordinate-based MLPs for functions on Euclidean spaces.

Original: In this paper, we focus on the non-Euclidean case.
Revised: In this paper, we focus on the non-Euclidean case.

Original: One alternative to Fourier features was introduced by Sitzmann et al.
Revised: One alternative to Fourier features was introduced by Sitzmann et al.

Original: (2020), who use the sine function as the MLP nonlinearity.
Revised: (2020), who use the sine function as the MLP nonlinearity.

Original: Major differences are that the sinusoidal is applied on every layer, and the frequencies are deﬁned by the network MLP weights.
Revised: Major differences are that the sinusoidal is applied on every layer, and the frequencies are defined by the network MLP weights.

Original: Zheng et al.
Revised: Zheng et al.

Original: (2021) studied the trade-off between memorization and generalization in positional encoding, showing the former is related to the rank of the embedding matrix and the latter is related to the distance preservation of the embedding.
Revised: (2021) studied the trade-off between memorization and generalization in positional encoding, showing the former is related to the rank of the embedding matrix and the latter is related to the distance preservation of the embedding.

Original: They also propose another alternative to Fourier features that consists of sampling a Gaussian at ﬁxed offsets from the input.
Revised: They also propose another alternative to Fourier features that consists of sampling a Gaussian at fixed offsets from the input.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: NeRF (Mildenhall et al., 2020) recently demonstrated the importance of positional encoding for coordinate-based learning of high frequency functions.
Revised: NeRF (Mildenhall et al., 2020) recently demonstrated the importance of positional encoding for coordinate-based learning of high frequency functions.

Original: For a coordinate x = [ x 1 , x 2 , x 3 ] ∈ R 3 , the following map was applied before the multilayer perceptron (MLP),
Revised: For a coordinate x = [ x 1 , x 2 , x 3 ] ∈ R 3 , the following map was applied before the multilayer perceptron (MLP),

Original: [Equation]
Revised: [Equation]

Original: Recall that { sin( m (cid:62) x ) } ∪ { cos( n (cid:62) x ) } with m, n ∈ Z 3 form an orthonormal basis for functions deﬁned on a compact subset of R 3 .
Revised: Recall that { sin( m ⊤ x ) } ∪ { cos( n ⊤ x ) } with m, n ∈ Z 3 form an orthonormal basis for functions defined on a compact subset of R 3 .

Original: So Eq.
Revised: So Eq.

Original: (1) corresponds to evaluating basis functions of different frequencies at input coordinates; we will refer to this method as “Euclidean encoding”.
Revised: (1) corresponds to evaluating basis functions of different frequencies at input coordinates; we will refer to this method as “Euclidean encoding”.

Original: In the particular case of Eq.
Revised: In the particular case of Eq.

Original: (1), only axis-aligned, powers-of-two frequencies are used, so the basis for bandwidth 2 L is only sparsely sampled.
Revised: (1), only axis-aligned, powers-of-two frequencies are used, so the basis for bandwidth 2 L is only sparsely sampled.

Original: Tancik et al.
Revised: Tancik et al.

Original: (2020) found it better to randomly sample the frequency space, which results in basis elements that are not axis aligned, but it is still a sparse sample of the complete basis.
Revised: (2020) found it better to randomly sample the frequency space, which results in basis elements that are not axis aligned, but it is still a sparse sample of the complete basis.

Original: In NeRF (Mildenhall et al., 2020), a subset of the input encodes the view direction.
Revised: In NeRF (Mildenhall et al., 2020), a subset of the input encodes the view direction.

Original: A direction can be associated to a point on the surface of the unit sphere, a non-Euclidean manifold.
Revised: A direction can be associated with a point on the surface of the unit sphere, a non-Euclidean manifold.

Original: In order to apply the Euclidean encoding to such input, (Mildenhall et al., 2020) represent the direction as a 3D unit vector instead, which increases the input dimensionality from 2D to 3D, and causes the positional encoding to sample from a set of O ( b 3 ) basis functions, instead of the minimal O ( b 2 ) for a bandwidth b .
Revised: In order to apply the Euclidean encoding to such input, (Mildenhall et al., 2020) represent the direction as a 3D unit vector instead, which increases the input dimensionality from 2D to 3D, and causes the positional encoding to sample from a set of O ( b 3 ) basis functions, instead of the minimal O ( b 2 ) for a bandwidth b .

Original: In IPDF (Murphy et al., 2021), the input coordinates are ﬂattened 3 × 3 rotation matrices, which correspond to points on the 3D rotation manifold SO (3) .
Revised: In IPDF (Murphy et al., 2021), the input coordinates are flattened 3 × 3 rotation matrices, which correspond to points on the 3D rotation manifold SO (3) .

Original: The Euclidean encoding then corresponds to sampling basis functions on R 9 .
Revised: The Euclidean encoding then corresponds to sampling basis functions on R 9 .

Original: This results in sampling from a space of O ( b 9 ) basis functions instead of O ( b 3 ) .
Revised: This results in sampling from a space of O ( b 9 ) basis functions instead of O ( b 3 ) .

Original: Our key observation is that positional encoding of input coordinates should reﬂect the geometry of the underlying manifold.
Revised: Our key observation is that positional encoding of input coordinates should reflect the geometry of the underlying manifold.

Original: We propose to use the appropriate orthonormal basis functions on the manifold to implement this idea.
Revised: We propose to use the appropriate orthonormal basis functions on the manifold to implement this idea.

Original: While for many applications the Euclidean manifold is the correct one, for others it is not, as exempliﬁed above.
Revised: While for many applications the Euclidean manifold is the correct one, for others it is not, as exemplified above.

Original: In the following sections we will show how the appropriate basis can be found for different types of manifolds, how to actually choose a subset of the basis to use in the encoding, and demonstrate experimentally the beneﬁts of using them.
Revised: In the following sections we will show how the appropriate basis can be found for different types of manifolds, how to actually choose a subset of the basis to use in the encoding, and demonstrate experimentally the benefits of using them.

Original:  M ETHOD 4. O VERVIEW
Revised:  M ETHOD 4. O VERVIEW

Original: We consider a neural network that takes as inputs points on an n-dimensional manifold M .
Revised: We consider a neural network that takes as inputs points on an n-dimensional manifold M .

Original: We assume there exists an orthonormal basis B = { b i : M → R } for the space of scalar functions on the manifold L 2 ( M ) .
Revised: We assume there exists an orthonormal basis B = { b i : M → R } for the space of scalar functions on the manifold L 2 ( M ) .

Original: Our goal is to apply the map γ : M → R k to positionally encode x ∈ M before feeding it to the network, where k is the encoding dimension and
Revised: Our goal is to apply the map γ : M → R k to positionally encode x ∈ M before feeding it to the network, where k is the encoding dimension and

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst question that arises is how to ﬁnd the basis for a given manifold.
Revised: The first question that arises is how to find the basis for a given manifold.

Original: There is no general solution, but speciﬁc solutions exist for some large classes of manifolds.
Revised: There is no general solution, but specific solutions exist for some large classes of manifolds.

Original: For compact groups (not necessarily abelian), the Peter-Weyl theorem establishes a generalization of the Fourier series that results in a countable set of orthonormal basis functions.
Revised: For compact groups (not necessarily abelian), the Peter-Weyl theorem establishes a generalization of the Fourier series that results in a countable set of orthonormal basis functions.

Original: It gives rise to the Wigner-D matrix elements that are used as a basis for SO (3) in Section 4.3.
Revised: It gives rise to the Wigner-D matrix elements that are used as a basis for SO (3) in Section 4.3.

Original: For locally compact abelian groups, the
Revised: For locally compact abelian groups, the

Original: Pontryagin duality can be used, which is a generalization of the Fourier transform.
Revised: Pontryagin duality can be used, which is a generalization of the Fourier transform.

Original: For Riemannian manifolds, an orthonormal basis can be obtained as eigenfunctions of the Laplace-Beltrami operator.
Revised: For Riemannian manifolds, an orthonormal basis can be obtained as eigenfunctions of the Laplace-Beltrami operator.

Original: For the sphere S 2 , these are the spherical harmonics used in Section 4.2.
Revised: For the sphere S 2 , these are the spherical harmonics used in Section 4.2.

Original: The neural tangent kernel (NTK) theory (Jacot et al., 2018; Arora et al., 2019; Lee et al., 2019) shows that, under certain conditions, an MLP trained for regression f : R n → R , converges to the solution of a kernel regression f ( x ) ≈ (cid:80)
Revised: The neural tangent kernel (NTK) theory (Jacot et al., 2018; Arora et al., 2019; Lee et al., 2019) shows that, under certain conditions, an MLP trained for regression f : R n → R , converges to the solution of a kernel regression f ( x ) ≈ (cid:80)

Original: i w i
Revised: i w i

Original: y i
Revised: y i

Original: k ( x, x i ) , where k is the NTK, w i the kernel regression weights, and the sum is over the whole dataset.
Revised: k ( x, x i ) , where k is the NTK, w i the kernel regression weights, and the sum is over the whole dataset.

Original: Moreover, when the inputs x have constant norm, the kernel depends only on inner products: k ( x, y )
Revised: Moreover, when the inputs x have constant norm, the kernel depends only on inner products: k ( x, y )

Original: =
Revised: =

Original: k (cid:48) ( x (cid:62) y ) .
Revised: k ′ ( x ⊤ y ) .

Original: In other words, it is rotationinvariant.
Revised: In other words, it is rotationinvariant.

Original: Tancik et al.
Revised: Tancik et al.

Original: (2020) leverages this theory to explain the success of Euclidean encoding – it transforms the rotation-invariant kernel in a shift-invariant one, which results in approximating a convolution operation over the whole training set for inputs in a ﬂat space.
Revised: (2020) leverages this theory to explain the success of Euclidean encoding – it transforms the rotation-invariant kernel in a shift-invariant one, which results in approximating a convolution operation over the whole training set for inputs in a flat space.

Original: In this paper, the inputs do not lie on Euclidean space, so the shift-invariance in the sense of translations in Euclidean space is not appropriate.
Revised: In this paper, the inputs do not lie on Euclidean space, so the shift-invariance in the sense of translations in Euclidean space is not appropriate.

Original: For the manifolds considered, we will seek positional encodings that have constant norm (lie on S n ) and such that the natural shift on the manifold also preserves the inner products between positionally encoded inputs.
Revised: For the manifolds considered, we will seek positional encodings that have constant norm (lie on S n ) and such that the natural shift on the manifold also preserves the inner products between positionally encoded inputs.

Original: Then, the NTK should approximate a convolution on the manifold.
Revised: Then, the NTK should approximate a convolution on the manifold.

Original: We’ll show in the following sections that these properties constrain the selection of basis elements.
Revised: We’ll show in the following sections that these properties constrain the selection of basis elements.

Original: 4. T HE SPHERE S
Revised: 4. T HE SPHERE S

Original: The spherical harmonics are eigenfunctions of the Laplace-Beltrami operator on the sphere S 2 and thus form an orthonormal basis for the space of square-integrable functions L 2 ( S 2 ) .
Revised: The spherical harmonics are eigenfunctions of the Laplace-Beltrami operator on the sphere S 2 and thus form an orthonormal basis for the space of square-integrable functions L 2 ( S 2 ) .

Original: The spherical harmonic of degree (cid:96) and order m is given by
Revised: The spherical harmonic of degree ℓ and order m is given by

Original: [Equation]
Revised: [Equation]

Original: where θ, φ are the colatitude and longitude angles, P (cid:96)m is the associate Legendre polynomial of degree (cid:96) and order m , with − (cid:96) ≤ m ≤ (cid:96) .
Revised: where θ, ϕ are the colatitude and longitude angles, P ℓm is the associate Legendre polynomial of degree ℓ and order m , with − ℓ ≤ m ≤ ℓ .

Original: The degree (cid:96) indicates the angular frequency, and α (cid:96)m is a normalization constant.
Revised: The degree ℓ indicates the angular frequency, and α ℓm is a normalization constant.

Original: Under the NTK umbrella, an MLP taking points on the sphere encoded as 3D unit vectors will converge to a kernel regression with the kernel depending only on inner products.
Revised: Under the NTK umbrella, an MLP taking points on the sphere encoded as 3D unit vectors will converge to a kernel regression with the kernel depending only on inner products.

Original: This is convenient because the natural shift on the sphere is a rotation, which preserves inner products.
Revised: This is convenient because the natural shift on the sphere is a rotation, which preserves inner products.

Original: We want to select a subset of the spherical harmonics for positional encoding that maintains these properties: 1) preserves inner-products, and 2) has constant norm.
Revised: We want to select a subset of the spherical harmonics for positional encoding that maintains these properties: 1) preserves inner-products, and 2) has constant norm.

Original: Proposition 1.
Revised: Proposition 1.

Original: For a given (cid:96) , let Y (cid:96) =
Revised: For a given ℓ , let Y ℓ =

Original: [ Y (cid:96) − (cid:96) , Y (cid:96) − (cid:96) +1 · · · , Y (cid:96)(cid:96) ] be a vector concatenating all the 2 (cid:96) + spherical harmonics of degree (cid:96) .
Revised: [ Y ℓ − ℓ , Y ℓ − ℓ +1 · · · , Y ℓℓ ] be a vector concatenating all the 2 ℓ + spherical harmonics of degree ℓ .

Original: Let Rx represent the point x ∈ S 2 rotated by R ∈ SO (3) .
Revised: Let Rx represent the point x ∈ S 2 rotated by R ∈ SO (3) .

Original: Then, for any x, y ∈ S 2 and R ∈ SO (3) , [Equation]
Revised: Then, for any x, y ∈ S 2 and R ∈ SO (3) , ⟨ Y ℓ ( x ) , Y ℓ ( y ) ⟩ = ⟨ Y ℓ ( Rx ) , Y ℓ ( Ry ) ⟩ .

Original: Proof.
Revised: Proof.

Original: See Appendix A.1.
Revised: See Appendix A.1.

Original: Since (cid:13)(cid:13) Y (cid:96) ( x ) (cid:13)(cid:13) =
Revised: Since (cid:13)(cid:13) Y ℓ ( x ) (cid:13)(cid:13) =

Original: (cid:112) (2 (cid:96) + 1) / (4 π ) , the vector Y (cid:96) ( x ) has the same norm for any x , and the NTK depends only on inner products.
Revised: (cid:112) (2 ℓ + 1) / (4 π ) , the vector Y ℓ ( x ) has the same norm for any x , and the NTK depends only on inner products.

Original: These results are easily extended to a concatenation of harmonics of multiple degrees, which suggests a strategy for selecting the basis elements for positional encoding: choose any set of degrees
Revised: These results are easily extended to a concatenation of harmonics of multiple degrees, which suggests a strategy for selecting the basis elements for positional encoding: choose any set of degrees

Original: L ⊂ N and use all harmonics for the chosen degrees: B (cid:48)
Revised: L ⊂ N and use all harmonics for the chosen degrees: B ′ =

Original: = { Y (cid:96) i m | (cid:96) i ∈ L, − (cid:96) i ≤ m ≤ (cid:96) i } .
Revised: { Y ℓ i m | ℓ i ∈ L, − ℓ i ≤ m ≤ ℓ i } .

Original: For simplicity, we typically choose a degree (cid:96) max such that L = { (cid:96) | 1 ≤ (cid:96) ≤ (cid:96) max } .
Revised: For simplicity, we typically choose a degree ℓ max such that L = { ℓ | 1 ≤ ℓ ≤ ℓ max } .

Original: Note that the maximum degree (cid:96) is the only hyperparameter that needs to be selected for both strategies.
Revised: Note that the maximum degree ℓ is the only hyperparameter that needs to be selected for both strategies.

Original: In contrast, the Gaussian encoding from Tancik et al.
Revised: In contrast, the Gaussian encoding from Tancik et al.

Original: (2020) requires both the mapping size and scale as hyperparameters.
Revised: (2020) requires both the encoding size and scale as hyperparameters.

Original: 4. T HE ROTATION MANIFOLD SO(3)
Revised: 4. T HE ROTATION MANIFOLD SO(3)

Original: The rotation manifold SO (3) is non-abelian and compact.
Revised: The rotation manifold SO (3) is non-abelian and compact.

Original: The latter property guarantees a Fourier series for the manifold according to the Peter-Weyl theorem, given by the Wigner-D matrices.
Revised: The latter property guarantees a Fourier series for the manifold according to the Peter-Weyl theorem, given by the Wigner-D matrices.

Original: The
Revised: The

Original: Wigner-D matrices are unitary irreducible representations of SO (3) and its matrix elements form an orthonormal basis for L 2 ( SO (3)) .
Revised: Wigner-D matrices are unitary irreducible representations of SO (3) and its matrix elements form an orthonormal basis for L 2 ( SO (3)) .

Original: The elements for the (2 (cid:96) + 1) × (2 (cid:96) + 1) matrix D (cid:96) of degree (cid:96) are
Revised: The elements for the (2 ℓ + 1) × (2 ℓ + 1) matrix D ℓ of degree ℓ are

Original: [Equation]
Revised: [Equation]

Original: where α, β, γ are ZYZ Euler angles representing the rotation, − (cid:96) ≤ m, n ≤ (cid:96) , and d (cid:96)m,n ( β ) is real and proportional to the Jacobi polynomial P ( a,b ) k (cos β ) , with k, a , b being functions of (cid:96), m, n .
Revised: where α, β, γ are ZYZ Euler angles representing the rotation, − ℓ ≤ m, n ≤ ℓ , and d ℓm,n ( β ) is real and proportional to the Jacobi polynomial P ( a,b ) k (cos β ) , with k, a , b being functions of ℓ, m, n .

Original: Since D (cid:96) is unitary for all (cid:96) , for any set of degrees L ∈ N , an encoding concatenating all matrix elements as follows has constant norm,
Revised: Since D ℓ is unitary for all ℓ , for any set of degrees L ∈ N , an encoding concatenating all matrix elements as follows has constant norm,

Original: [Equation]
Revised: [Equation]

Original: The natural shift on SO (3) is also a rotation (the group acting on itself).
Revised: The natural shift on SO (3) is also a rotation (the group acting on itself).

Original: Therefore, similarly to the discussion in Section 4.2, it is desirable that the inner products between encodings are invariant with respect to rotations of the input.
Revised: Therefore, similarly to the discussion in Section 4.2, it is desirable that the inner products between encodings are invariant with respect to rotations of the input.

Original: Proposition 2.
Revised: Proposition 2.

Original: For a given (cid:96) , let vec ( D (cid:96) ) be the vectorization of the Wigner-D matrix D (cid:96) .
Revised: For a given ℓ , let vec ( D ℓ ) be the vectorization of the Wigner-D matrix D ℓ .

Original: Then, for any R, R 1 , R 2 ∈ SO (3) ,
Revised: Then, for any R, R 1 , R 2 ∈ SO (3) ,

Original: [Equation]
Revised: [Equation]

Original: ) (cid:105) .

Original: Proof.
Revised: Proof.

Original: See Appendix A.
Revised: See Appendix A.

Original: This also holds for a concatenation of multiple vec ( D (cid:96) i ) , so the map in Eq.
Revised: This also holds for a concatenation of multiple vec ( D ℓ i ) , so the map in Eq.

Original: (5), besides constant norm, also has rotation-invariant inner products, as desired.
Revised: (5), besides constant norm, also has rotation-invariant inner products, as desired.

Original: In practice, instead of an arbitrary set of degrees, we choose an (cid:96) max and use all degrees 1 ≤ (cid:96) ≤ (cid:96) max .
Revised: In practice, instead of an arbitrary set of degrees, we choose an ℓ max and use all degrees 1 ≤ ℓ ≤ ℓ max .

Original: 4. P RODUCT OF SPHERES S 2 × S
Revised: 4. P RODUCT OF SPHERES S 2 × S

Original: The coordinates for a point in S 2 × S 2 can be obtained by the concatenation of coordinates on each sphere: ( θ 1 , φ 1 , θ 2 , φ 2 ) .
Revised: The coordinates for a point in S 2 × S 2 can be obtained by the concatenation of coordinates on each sphere: ( θ 1 , ϕ 1 , θ 2 , ϕ 2 ) .

Original: A basis for the product manifold can be obtained by product of the basis functions for the sphere.
Revised: A basis for the product manifold can be obtained by product of the basis functions for the sphere.

Original: Thus, we have the following orthonormal basis functions for L 2 ( S 2 × S 2 ) ,
Revised: Thus, we have the following orthonormal basis functions for L 2 ( S 2 × S 2 ) ,

Original: [Equation]
Revised: [Equation]

Original: with Y (cid:96)m as deﬁned in Eq.
Revised: with Y ℓm as defined in Eq.

Original: We will use the following form interchangeably to reduce notation,
Revised: We will use the following form interchangeably to reduce notation,

Original: Y
Revised: Y

Original: [Equation]
Revised: [Equation]

Original: Y

Original: [Equation] ( x [Equation] ) , where x 1 , x 2 ∈ S 2 .
Revised: where x 1 , x 2 ∈ S 2 .

Original: For a given set of pairs of degrees L ⊂ N × N , we consider the following basis functions, consisting of selecting all possible ( m i , m j ) given ( (cid:96) i , (cid:96) j ) ,
Revised: For a given set of pairs of degrees L ⊂ N × N , we consider the following basis functions, consisting of selecting all possible ( m i , m j ) given ( ℓ i , ℓ j ) ,

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.
Revised: Proposition 3.

Original: For a given pair ( (cid:96) 1 , (cid:96) 2 ) , let Y (cid:96) 1 [-,(cid:96) 
-] = { Y (cid:96) 1 ,(cid:96) 2 m i ,m j } be a vector containing all the (2 (cid:96) 1 + 1)(2 (cid:96) 2 + 1) products of spherical harmonics of degrees [-(cid:96) 
-] and (cid:96) 2 .
Revised: For a given pair ( ℓ 1 , ℓ 2 ) , let Y ℓ 1 [-,(cid:96) 
-],ℓ 2 = { Y ℓ 1 ,ℓ 2 m i ,m j } be a vector containing all the (2 ℓ 1 + 1)(2 ℓ 2 + 1) products of spherical harmonics of degrees [-(cid:96) 
-]ℓ 1 and ℓ 2 .

Original: For any x 1 , x 2 ∈ S 2 , the norm of Y (cid:96) 1 ,(cid:96) 2 is a constant: (cid:13)(cid:13) Y (cid:96) 1 ,(cid:96) 2 ( x 1 , x 2 ) (cid:13)(cid:13) = (cid:112) (2 (cid:96) 1 + [-1)(
 (cid:96)-] 2 + 1) / (4 π ) .
Revised: For any x 1 , x 2 ∈ S 2 , the norm of Y ℓ 1 ,ℓ 2 is a constant: (cid:13)(cid:13) Y ℓ 1 ,ℓ 2 ( x 1 , x 2 ) (cid:13)(cid:13) = (cid:112) (2 ℓ 1 + [-1)(
 (cid:96)-]1)(2 ℓ 2 + 1) / (4 π ) .

Original: Proof.
Revised: Proof.

Original: See Appendix A.3.
Revised: See Appendix A.3.

Original: Since the encodings have constant norm, we are interested in whether the inner product between encodings is preserved under input shift, which would lead to the previously discussed desirable properties that follow from the NTK.
Revised: Since the encodings have constant norm, we are interested in whether the inner product between encodings is preserved under input shift, which would lead to the previously discussed desirable properties that follow from the NTK.

Original: For the case of S 2 × S 2 , an appropriate shift consists in applying the same rotation to both spheres.
Revised: For the case of S 2 × S 2 , an appropriate shift consists in applying the same rotation to both spheres.

Original: For example, in our experiment in Section 5.4, the inputs on S 2 × S 2 represent a pair of points on a sphere, so it makes sense that the appropriate shift should rotate both points.
Revised: For example, in our experiment in Section 5.4, the inputs on S 2 × S 2 represent a pair of points on a sphere, so it makes sense that the appropriate shift should rotate both points.

Original: Proposition 4.
Revised: Proposition 4.

Original: Consider the vector of orthonormal functions Y (cid:96) 1 ,(cid:96) 2 as deﬁned in Proposition 3.
Revised: Consider the vector of orthonormal functions Y ℓ 1 ,ℓ 2 as defined in Proposition 3.

Original: The inner product between encodings is invariant to rotations of the inputs:
Revised: The inner product between encodings is invariant to rotations of the inputs:

Original: [Equation]
Revised: [Equation]

Original: for any pair ( x 1 , x 2 ) ∈ S 2 × S 2 and ( y 1 , y 2 ) ∈
Revised: for any pair ( x 1 , x 2 ) ∈ S 2 × S 2 and ( y 1 , y 2 ) ∈

Original: S 2 × S 2 , and rotation R ∈ SO (3) .
Revised: S 2 × S 2 , and rotation R ∈ SO (3) .

Original: Proof.
Revised: Proof.

Original: See Appendix A.
Revised: See Appendix A.

Original: Therefore, we can construct a basis for positional encoding using Eq.
Revised: Therefore, we can construct a basis for positional encoding using Eq.

Original: In practice, we set a maximum degree (cid:96) max and use all pairs under it: L = { ( (cid:96) i , (cid:96) j ) | 1 ≤ (cid:96) i , (cid:96) j ≤ (cid:96) max } .
Revised: In practice, we set a maximum degree ℓ max and use all pairs under it: L = { ( ℓ i , ℓ j ) | 1 ≤ ℓ i , ℓ j ≤ ℓ max } .

Original:  E XPERIMENTS 5. S PHERICAL PANORAMAS
Revised:  E XPERIMENTS 5. S PHERICAL PANORAMAS

Original: Following Tancik et al.
Revised: Following Tancik et al.

Original: ; Sitzmann et al.
Revised: ; Sitzmann et al.

Original: (2020), we evaluate the learning of high-frequencies on a simple task of training an MLP to regress an image from its input coordinates.
Revised: (2020), we evaluate the learning of high-frequencies on a simple task of training an MLP to regress an image from its input coordinates.

Original: Since we are interested in non-Euclidean manifolds, we evaluate the methods on 10 spherical panoramas from SUN360 (Xiao et al.
Revised: Since we are interested in non-Euclidean manifolds, we evaluate the methods on 10 spherical panoramas from SUN360 (Xiao et al.

Original: (2012)), randomly sampling 1/4 of the pixels as the training set, with probabilities proportional to pixel areas in the input spherical grid.
Revised: (2012)), randomly sampling 1/4 of the pixels as the training set, with probabilities proportional to pixel areas in the input spherical grid.

Original: We follow the image resolution, architecture design, and training schedule of Tancik et al.
Revised: We follow the image resolution, architecture design, and training schedule of Tancik et al.

Original: (2020), and replace their positional encoding with ours, where we use all ( (cid:96) max + 1)( (cid:96) max + 2) / 2 spherical harmonics up to (cid:96) max = 16 .
Revised: (2020), and replace their positional encoding with ours, where we use all ( ℓ max + 1)( ℓ max + 2) / 2 spherical harmonics up to ℓ max = 16 .

Original: Table 1 shows the test peak signal-to-noise ratios (PSNRs).
Revised: Table 1 shows the test peak signal-to-noise ratios (PSNRs).

Original: Figuredepicts the results.
Revised: Figuredepicts the results.

Original: Note the large distortions on the poles with Gaussian encoding; these are the areas with large distortion when assuming an Euclidean manifold.
Revised: Note the large distortions on the poles with Gaussian encoding; these are the areas with large distortion when assuming an Euclidean manifold.

Original: [Table]
Revised: [Table]

Original: 5. P ROBABILITY DISTRIBUTIONS ON SO(3)
Revised: 5. P ROBABILITY DISTRIBUTIONS ON SO(3)

Original: Recently, Murphy et al.
Revised: Recently [Table] Murphy et al.

Original: introduced IPDF, a model that learns to predict probability distributions on SO (3) to represent the object pose from an image.
Revised: introduced IPDF, a model that learns to predict probability distributions on SO (3) to represent the object pose from an image.

Original: The main part of the model is a
Revised: The main part of the model is a

Original: [Table]
Revised: [Table]

Original: coordinate-based MLP taking rotation matrices as inputs.
Revised: coordinate-based MLP taking rotation matrices as inputs.

Original: Euclidean  andshown to
Revised: Euclidean positional encoding is applied and

Original: In this experiment [Table] we show that using Wigner-D matrix elements as the basis for the encoding signiﬁcantly outperforms the Euclidean encoding.
Revised: In this experiment, we show that using Wigner-D matrix elements as the basis for the encoding significantly outperforms the Euclidean encoding.

Original: We encode the input rotation using all basis elements up to degree (cid:96) = 5 by evaluating Eq.
Revised: We encode the input rotation using all basis elements up to degree ℓ = 5 by evaluating Eq.

Original: (4), using the “Spherical Functions” 1 open source package to perform the computations.
Revised: (4), using the “Spherical Functions” 1 open source package to perform the computations.

Original: We use the same architecture of Murphy et al.
Revised: We use the same architecture of Murphy et al.

Original: (2021), and train the model on the SYMSOL I dataset, which was introduced by for pose estimation of symmetric solids.
Revised: (2021), and train the model on the SYMSOL I dataset, which was introduced by for pose estimation of symmetric solids.

Original: We slightly modify the training scheme of IPDF by adding two randomly sampled points near each ground truth and also maximizing the log-likelihood of them both.
Revised: We slightly modify the training scheme of IPDF by adding two randomly sampled points near each ground truth and also maximizing the log-likelihood of them both.

Original: This stabilizes the training and greatly improves the log-likelihood metric.
Revised: This stabilizes the training and greatly improves the log-likelihood metric.

Original: We denote this modiﬁed version “IPDF*”, and also apply it to our approach.
Revised: We denote this modified version “IPDF*”, and also apply it to our approach.

Original: Table 2 shows the results.
Revised: Table 2 shows the results.

Original: The log-likelihood is the average over all equivalent ground truth poses, and the “spread” metric can be seen as the expected angle error to the closest ground truth (Murphy et al., 2021).
Revised: The log-likelihood is the average over all equivalent ground truth poses, and the “spread” metric can be seen as the expected angle error to the closest ground truth (Murphy et al., 2021).

Original: Figure 3 compares predictions from our method and the baseline and shows that our approach produces more accurate outputs.
Revised: Figure 3 compares predictions from our method and the baseline and shows that our approach produces more accurate outputs.

Original: 5. N EURAL RADIANCE FIELDS
Revised: 5. N EURAL RADIANCE FIELDS

Original: NeRF (Mildenhall et al., 2020) has recently shown unprecedented quality on neural rendering and inspired numerous advancements on the ﬁeld.
Revised: NeRF (Mildenhall et al., 2020) has recently shown unprecedented quality on neural rendering and inspired numerous advancements on the field.

Original: Part of the success of NeRF is due to the positional encoding of input coordinates.
Revised: Part of the success of NeRF is due to the positional encoding of input coordinates.

Original: Speciﬁcally, the inputs are a point in Euclidean space x ∈ R 3 and
Revised: Specifically, the inputs are a point in Euclidean space x ∈ R 3 and

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] Ours a direction ( θ, φ ) ∈ S 2 , and the network approximates a density function at x , σ ( x ) and the color c ( x, θ, φ ) of the light ray passing through x at direction ( θ, φ ) .
Revised: Ours a direction ( θ, ϕ ) ∈ S 2 , and the network approximates a density function at x , σ ( x ) and the color c ( x, θ, ϕ ) of the light ray passing through x at direction ( θ, ϕ ) .

Original: The Euclidean positional encoding is applied independently to x and to the 3D unit vector d corresponding to ( θ, φ ) .
Revised: The Euclidean positional encoding is applied independently to x and to the 3D unit vector d corresponding to ( θ, ϕ ) .

Original: In this experiment, we replace NeRF’s Euclidean positional encoding at direction d with an encoding based on the spherical harmonics.
Revised: In this experiment, we replace NeRF’s Euclidean positional encoding at direction d with an encoding based on the spherical harmonics.

Original: We used all the harmonics up to degree (cid:96) max = 4 , Evaluating them on device with the jax.scipy package (Bradbury et al., 2018).
Revised: We used all the harmonics up to degree ℓ max = 4 , Evaluating them on device with the jax.scipy package (Bradbury et al., 2018).

Original: In NeRF, the directional information is intentionally included only at the last layer, which can be interpreted as a strong Lambertian inductive bias.
Revised: In NeRF, the directional information is intentionally included only at the last layer, which can be interpreted as a strong Lambertian inductive bias.

Original: Therefore the effect of the directional information and its positional encoding in the overall metrics is expected to be small.
Revised: Therefore the effect of the directional information and its positional encoding in the overall metrics is expected to be small.

Original: We noticed that inserting the directional information earlier at the 6th layer instead of 8th slightly increases performance.
Revised: We noticed that inserting the directional information earlier at the 6th layer instead of 8th slightly increases performance.

Original: Our experiments (Table 3) show a small, but consistent and statistically signiﬁcant improvement when using the spherical harmonics encoding.
Revised: Our experiments (Table 3) show a small, but consistent and statistically significant improvement when using the spherical harmonics encoding.

Original: We repeat each experiment ﬁve times and report average
Revised: We repeat each experiment five times and report average

Original: PSNRs and standard deviations.
Revised: PSNRs and standard deviations.

Original: 5. S PHERICAL LIGHT FIELDS
Revised: 5. S PHERICAL LIGHT FIELDS

Original: Light ﬁeld rendering (Levoy & Hanrahan, 1996) is a classic approach for novel view synthesis, which consists of interpolating the radiance at unseen rays based on a large database of given rays, without estimating the scene geometry.
Revised: Light field rendering (Levoy & Hanrahan, 1996) is a classic approach for novel view synthesis, which consists of interpolating the radiance at unseen rays based on a large database of given rays, without estimating the scene geometry.

Original: Sitzmann et al.
Revised: Sitzmann et al.

Original: (2021) recently combined learning of light ﬁelds and coordinate-based MLPs.
Revised: (2021) recently combined learning of light fields and coordinate-based MLPs.

Original: By assuming the radiance along a ray is constant, rays can be identiﬁed with a 4D vector of coordinates.
Revised: By assuming the radiance along a ray is constant, rays can be identified with a 4D vector of coordinates.

Original: There are numerous parametrizations in the literature.
Revised: There are numerous parametrizations in the literature.

Original: In this experiment, we consider the
Revised: In this experiment, we consider the

Original: [Table]
Revised: [Table]

Original: one introduced by Camahort et al.
Revised: one introduced by Camahort et al.

Original: (1998), which represents a ray hitting the scene with the two intersections between ray and a sphere bounding the scene.
Revised: (1998), which represents a ray hitting the scene with the two intersections between ray and a sphere bounding the scene.

Original: Therefore, each light ray has coordinates ( θ 1 , φ 1 , θ 2 , φ 2 ) ∈ S 2 × S 2 .
Revised: Therefore, each light ray has coordinates ( θ 1 , ϕ 1 , θ 2 , ϕ 2 ) ∈ S 2 × S 2 .

Original: We introduce a coordinate-based MLP to learn this spherical light ﬁeld, which enables rendering novel views by querying the ray coordinates for a desired viewpoint.
Revised: We introduce a coordinate-based MLP to learn this spherical light field, which enables rendering novel views by querying the ray coordinates for a desired viewpoint.

Original: The underlying manifold is a product of spheres, S 2 × S 2 , and we use the orthonormal basis for this manifold for positional encoding, as described in Section 4.4.
Revised: The underlying manifold is a product of spheres, S 2 × S 2 , and we use the orthonormal basis for this manifold for positional encoding, as described in Section 4.4.

Original: We use all pairs of harmonics up to (cid:96) max = 4 .
Revised: We use all pairs of harmonics up to ℓ max = 4 .

Original: Rendering the light ﬁeld directly with an MLP can be orders of magnitude faster than approaches that use ray tracing (for example, NeRF needs to evaluate the MLP more than 100 times to render a single pixel).
Revised: Rendering the light field directly with an MLP can be orders of magnitude faster than approaches that use ray tracing (for example, NeRF needs to evaluate the MLP more than 100 times to render a single pixel).

Original: However, since light ﬁelds do not model the underlying scene geometry, there is no inherent notion of multi-view consistency in the functions learned.
Revised: However, since light fields do not model the underlying scene geometry, there is no inherent notion of multi-view consistency in the functions learned.

Original: This hampers a light ﬁeld rendering models ability to generalize to novel views from few samples.
Revised: This hampers a light field rendering models ability to generalize to novel views from few samples.

Original: Consequently, akin to classical light ﬁeld methods, we render a large number of views for training.
Revised: Consequently, akin to classical light field methods, we render a large number of views for training.

Original: We sample 1 .
Revised: We sample 1 .

Original: 5 k
Revised: 5 k

Original: HEALPix 2 (G ´ orski et al., 2005; Zonca et al., 2019) points on a hemisphere and render the lego scene from (Mildenhall et al., 2020) with cameras at these points, directed towards the origin.
Revised: HEALPix 2 (G ´ orski et al., 2005; Zonca et al., 2019) points on a hemisphere and render the lego scene from (Mildenhall et al., 2020) with cameras at these points, directed towards the origin.

Original: We train light ﬁeld models on this dataset using different positional encodings.
Revised: We train light field models on this dataset using different positional encodings.

Original: Table 4 shows the results.
Revised: Table 4 shows the results.

Original: We observe an improvement of nearly 1 .
Revised: We observe an improvement of nearly 1 .

Original: 7 dB when using the proposed encoding when compared to axis-aligned encoding.
Revised: 7 dB when using the proposed encoding when compared to axis-aligned encoding.

Original: Figure 4 shows that our method improves considerably the rendering quality, retaining more high-frequency details as compared to baselines.
Revised: Figure 4 shows that our method improves considerably the rendering quality, retaining more high-frequency details as compared to baselines.

Original:  L IMITATIONS
Revised:  L IMITATIONS

Original: Our method is sensitive to the choice of basis elements.
Revised: Our method is sensitive to the choice of basis elements.

Original: Selecting a maximum frequency that is too high leads to too many elements in the encoding and overﬁtting, while low frequencies tend to underﬁt.
Revised: Selecting a maximum frequency that is too high leads to too many elements in the encoding and overfitting, while low frequencies tend to underfit.

Original: Similar limitations were observed by Tancik et al.
Revised: Similar limitations were observed by Tancik et al.

Original: (2020), where the “scale” parameter, which is related to the frequency of the basis elements, needs to be carefully tuned.
Revised: (2020), where the “scale” parameter, which is related to the frequency of the basis elements, needs to be carefully tuned.

Original: Evaluating the special functions that compose the basis used for positional encoding is generally computationally expensive.
Revised: Evaluating the special functions that compose the basis used for positional encoding is generally computationally expensive.

Original: This is amortized when the inputs lie on a ﬁxed grid (as in the spherical panoramas) or can be pre-computed from the dataset (like in the NeRF (Mildenhall et al., 2020) and light ﬁeld experiments).
Revised: This is amortized when the inputs lie on a fixed grid (as in the spherical panoramas) or can be pre-computed from the dataset (like in the NeRF (Mildenhall et al., 2020) and light field experiments).

Original: This, combined with the fast on-device jax.scipy implementation of the spherical harmonics results in almost no overhead for these experiments.
Revised: This, combined with the fast on-device jax.scipy implementation of the spherical harmonics results in almost no overhead for these experiments.

Original: For IPDF (Murphy et al., 2021), however, the training grid is sampled randomly, and there is no on-device implementation, which results is half the training speed.
Revised: For IPDF (Murphy et al., 2021), however, the training grid is sampled randomly, and there is no on-device implementation, which results is half the training speed.

Original: During inference the grid is ﬁxed but very large, requiring a 32 Gb device for fast evaluation.
Revised: During inference the grid is fixed but very large, requiring a 32 Gb device for fast evaluation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper introduced a method to apply positional encoding for learning high-frequency functions on non-Euclidean manifolds.
Revised: This paper introduced a method to apply positional encoding for learning high-frequency functions on non-Euclidean manifolds.

Original: It generalizes the widely used “Fourier Features” by employing orthonormal basis on the manifold for the encoding, as opposed to the sinusoidals that are basis functions for Euclidean spaces.
Revised: It generalizes the widely used “Fourier Features” by employing orthonormal basis on the manifold for the encoding, as opposed to the sinusoidals that are basis functions for Euclidean spaces.

Original: The neural tangent kernel (NTK) theory guides our design; we ensure the encodings lie on a hypersphere and that their inner-products is preserved under the appropriate shift on the manifold.
Revised: The neural tangent kernel (NTK) theory guides our design; we ensure the encodings lie on a hypersphere and that their inner-products is preserved under the appropriate shift on the manifold.

Original: We demonstrated that our approach has advantages with respect to the standard
Revised: We demonstrated that our approach has advantages with respect to the standard

Original: Euclidean encodings in multiple tasks and manifolds.
Revised: Euclidean encodings in multiple tasks and manifolds.

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: We have presented generalized Fourier features for non-Euclidean manifolds.
Revised: We have presented generalized Fourier features for non-Euclidean manifolds.

Original: We have intentionally targeted a broad set of applications, and our focus has been on presenting the core mathematical underpinnings of the technique with the objective of making it accessible to a wide audience.
Revised: We have intentionally targeted a broad set of applications, and our focus has been on presenting the core mathematical underpinnings of the technique with the objective of making it accessible to a wide audience.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: We provide the experimental details necessary for reproducing the results in the appropriate subsections in Section 5 and Appendix B.
Revised: We provide the experimental details necessary for reproducing the results in the appropriate subsections in Section 5 and Appendix B.

Original: We note that the implementations are fairly straightforward as in most cases we are simply making small modiﬁcations to the positional encoding in existing open source experiment code (e g. swapping the positional encoding input in NeRF or IPDF).
Revised: We note that the implementations are fairly straightforward as in most cases we are simply making small modifications to the positional encoding in existing open source experiment code (e g. swapping the positional encoding input in NeRF or IPDF).

Original: We commit to open sourcing the code for our experiments upon publication
Revised: We commit to open sourcing the code for our experiments upon publication

Original: A P ROOFS A. P ROOF OF P ROPOSITION
Revised: A P ROOFS A. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: A formula for the rotation of spherical harmonics is given by
Revised: A formula for the rotation of spherical harmonics is given by

Original: [Equation]
Revised: [Equation]

Original: (8) for all x ∈ S 2 and R ∈ SO (3) where D (cid:96) is a Wigner-D matrix (with elements as deﬁned in Eq.
Revised: for all x ∈ S 2 and R ∈ SO (3) where D ℓ is a Wigner-D matrix (with elements as defined in Eq.

Original: Since D (cid:96) is unitary it preserves inner products.
Revised: Since D ℓ is unitary it preserves inner products.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: (cid:96) ( x ) , Y (cid:96) ( y ) (cid:105) .
Revised: Remark Since Y ℓ − m ( x ) = ( − 1) m Y ℓm ( x ) , we can slightly modify the encodings to be real-valued and use only m ≥ 0 components, [Equation]

Revised: where ℜ and ℑ are the real and imaginary parts, respectively.

Revised: This suggests the following alternative encoding, which also has constant norm and rotationinvariant inner products,

Revised: [Equation]

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: From the representation property of the Wigner-D matrices, we have
Revised: From the representation property of the Wigner-D matrices, we have

Original: [Equation]
Revised: [Equation]

Original: In the following, we use the identity vec ( B ) ∗ vec ( A )
Revised: In the following, we use the identity vec ( B ) ∗ vec ( A )

Original: = tr ( A ∗ B ) ,
Revised: = tr ( A ∗ B ) ,

Original: where x ∗ is the conjugatetranspose of x .
Revised: where x ∗ is the conjugatetranspose of x .

Original: [Equation]
Revised: [Equation]

Revised: Remark Similarly to the case with the spherical harmonics, in practice we use a real-valued encoding using only m ≤ 0 basis elements, by leveraging that D ℓm,n = ( − 1) m − n D ℓ − m, − n , ⟨

Revised: [Equation]

Revised: D ℓ

Revised: [Equation]

Revised: R 1 )) , vec

Revised: [Equation]

Revised: D

Revised: [Equation]

Revised: [Equation]

Revised: Since ℜ ( ab ) =

Revised: ℜ ( a ) ℜ ( b ) + ℑ ( a ) ℑ ( b )

Revised: [Equation]

Revised: we obtain an encoding with constant norm and rotationinvariant inner products as follows

Revised: [Equation]

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: The elements of Y (cid:96) 1 ,(cid:96) 2 ( x 1 , x 2 ) coincide with the matrix elements of Y (cid:96) 1 ( x 1 ) Y (cid:96) 2 ( x 2 ) ∗ , with
Revised: The elements of Y ℓ 1 ,ℓ 2 ( x 1 , x 2 ) coincide with the matrix elements of Y ℓ 1 ( x 1 ) Y ℓ 2 ( x 2 ) ∗ , with

Original: Y (cid:96) as deﬁned in Proposition 
Revised: Y ℓ as defined in Proposition 1.

Original: [Equation] Let (cid:107) X (cid:107) F be the Frobenius norm of X .
Revised: Let ∥ X ∥ F be the Frobenius norm of X .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: We write Y (cid:96) 1 ,(cid:96) 2 as the vectorization of the outer product, with Y (cid:96) as deﬁned in Proposition 1,Y [Equation] [Equation]
Revised: We write Y ℓ 1 ,ℓ 2 as the vectorization of the outer product, with Y ℓ as defined in Proposition 1,

Original: 2 ( x 1 , x 2 )

Original: = vec ( Y (cid:96) 1 ( x 1 ) Y (cid:96) 2 ( x 2 ) ∗ ) .

Original: Then we use the identity (cid:104) vec ( A ) , vec ( B ) (cid:105) =

Original: tr ( B ∗ A ) , the spherical harmonics rotation formula, that the Wigner-D is unitary, and the trace cyclic property, (cid:104) Y (cid:96)

Original: [Equation] ( Rx 1 , Rx 2 ), Y [Equation] [Equation]
Revised: Y ℓ 1 ,ℓ 2 ( x 1 , x 2 )

Original: Ry

Original: [Equation]

Original: Ry

Original: [Equation] = [Equation] Y 
Revised: = vec ( Y ℓ 1 ( x 1 ) Y ℓ 2 ( x 2 ) ∗ ) .

Original: [Equation] Y [Equation] ∗ 
Revised: Then we use the identity ⟨ vec ( A ) , vec ( B ) ⟩ = tr ( B ∗ A ) , the spherical harmonics rotation formula, that the Wigner-D is unitary

Original: [Equation]
Revised: [Equation]

Original: Y

Original: [Equation] Y [Equation] Ry
Revised: and the trace cyclic property,

Original: [Equation]
Revised: [Equation]

Original: ∗

Original: [Equation]

Original: [Equation]

Original: B E XPERIMENTAL DETAILS B. S PHERICAL PANORAMAS
Revised: B E XPERIMENTAL DETAILS B. S PHERICAL PANORAMAS

Original: Spherical PSNR Since pixels sampled on an equiangular grid on the sphere have different areas, we use a simple quadrature rule to evaluate the PSNR.
Revised: Spherical PSNR Since pixels sampled on an equiangular grid on the sphere have different areas, we use a simple quadrature rule to evaluate the PSNR.

Original: For co-latitude θ and longitude φ , the mean squared error (MSE) can be deﬁned as
Revised: For co-latitude θ and longitude ϕ , the mean squared error (MSE) can be defined as

Original: [Equation]
Revised: [Equation]

Original: where I represents an image, K is the noisy approximation of I , and N θ and N φ are the number of sampling points along co-latitude and longitude, respectively.
Revised: where I represents an image, K is the noisy approximation of I , and N θ and N ϕ are the number of sampling points along co-latitude and longitude, respectively.

Original: The PSNR (in dB) can thus be deﬁned as
Revised: The PSNR (in dB) can thus be defined as

Original: [Equation]
Revised: [Equation]

Original: where the maximum possible pixel value of the image I is MAX I =
Revised: where the maximum possible pixel value of the image I is MAX I =

Original: (each pixel value is represented by eight bits).
Revised: (each pixel value is represented by eight bits).

Original: Additional visualizations Figure 5 show additional testing output images with both baseline and the proposed encoding methods.
Revised: Additional visualizations Figure 5 show additional testing output images with both baseline and the proposed encoding methods.

Original: B. S PHERICAL L IGHT F IELDS
Revised: B. S PHERICAL L IGHT F IELDS

Original: For the light ﬁeld rendering task, we train three models using no encoding, axis-aligned encoding and the proposed product of spherical harmonics encoding.
Revised: For the light field rendering task, we train three models using no encoding, axis-aligned encoding and the proposed product of spherical harmonics encoding.

Original: The model is trained to predict the color, given a light ﬁeld representation of ray in space.
Revised: The model is trained to predict the color, given a light field representation of ray in space.

Original: The training signal to train the model is computed as the mean-square error between rendered views and ground truth images.
Revised: The training signal to train the model is computed as the mean-square error between rendered views and ground truth images.

Original: We use an MLP with 12 layers, 1024 channels, ReLU activation for intermediate layers and a sigmoid on the output.
Revised: We use an MLP with 12 layers, 1024 channels, ReLU activation for intermediate layers and a sigmoid on the output.

Original: The models are trained for 250 k iterations with a batch size of 8192 using an Adam optimizer with default settings.
Revised: The models are trained for 250 k iterations with a batch size of 8192 using an Adam optimizer with default settings.

Original: We use a learning rate schedule with warm-up for 2 .
Revised: We use a learning rate schedule with warm-up for 2 .

Original: 5 k iterations followed by an exponential decay.
Revised: 5 k iterations followed by an exponential decay.

Original: The intial and ﬁnal learning rates are set to 2 × 10 − 3 and 2 × 10 − 5 respectively.
Revised: The intial and final learning rates are set to 2 × 10 − 3 and 2 × 10 − 5 respectively.

Original: We set the maximum degree to be 4 for the axis-aligned 
Revised: We set the maximum degree to be 4 for the axis-aligned as well as our encoding.

Revised: C E XTRA EXPERIMENTS

Revised: In this section, we show results on a second spherical image dataset for sensitivity analysis to the most important hyperparameters.

Revised: This dataset is based on the “Text” images of Tancik et al.

Revised: (2020), which consists of characters of different colors and sizes.

Revised: It is ideal for investigate positional encoding methods due to the abundance of sharp edges (high frequency content).

Revised: We use their 32 original images and perform an stereographic projection to the sphere at random orientations to create spherical images.

Revised: We refer to this dataset as “Spherical Text”.

Revised: We also introduce stronger baselines that take 3D points on the sphere, and thus respect the manifold structure, in contrast to the baselines in Section 5.1.

Revised: The following baselines are considered in this section.

Revised: Axis-aligned 2D Similar to the baseline in Section 5.1.

Revised: Input coordinates are Cartesian, treating the spherical image as a planar image.

Revised: For a desired encoding size 2 M , and maximum frequency

Revised: L , each coordinate z

Revised: i is encoded independently as z i (cid:55)→ { sin(2 π 2 j z i ) } ∪ { cos(2 π 2 j z i ) } , where j = kL/ ( M − 1) for 0 ≤ k < M and i ∈ 1 , 2 .

Revised: Follows Mildenhall et al.

Revised: Gaussian 2D Similar to the “Axis-aligned 2D”, but the tuple of coordinates z is encoded jointly as z (cid:55)→ { sin(2 π b ⊤ i z ) } ∪ { cos(2 π b ⊤ i z ) } , where { b i } are vectors randomly sampled from a centered normal distribution of variance σ 2 and 0 ≤ i < M .

Revised: Follows Tancik et al.

Revised: Axis-aligned 3D In the 3D case, instead of treating the spherical image as a planar image, we treat it as a sphere embedded in R 3 , and take the 3D coordinates of points on the sphere as inputs.

Revised: The encoding is the same as the 2D case, with i ∈ 1 , 2 , 3 , and encoding size 3 M .

Revised: This type of encoding is used in the view-direction component in NeRF Mildenhall et al.

Revised: (2020), which is a spherical function.

Revised: Gaussian 3D Similarly to “Axis-aligned 3D”, we apply the Gaussian encoding to 3D points on the sphere.

Revised: The difference with respect to the 2D case is that b i is 3 × 1 instead of 2 × 1 .

Revised: C. B ASELINE SENSITIVITY TO FREQUENCY AND SCALE

Revised: max =the encoding size is slightly lower than 2 M , for ℓ max = 23 it is slightly higher.

Revised: Freq./Scale

Revised: PNSR

Revised: Axis-aligned 3D

Revised: L =25 .

Revised: Axis-aligned 3D

Revised: L =26 .

Revised: Axis-aligned 3D

Revised: L =25 .

Revised: Axis-aligned 2D

Revised: L =25 .

Revised: Axis-aligned 2D

Revised: L =26 .

Revised: Axis-aligned 2D

Revised: L =26 .

Revised: Gaussian 3D σ =25 .

Revised: Gaussian 3D σ =27 .

Revised: Gaussian 3D σ =28 . 17(20)

Revised: Gaussian 3D σ =28 .

Revised: Gaussian 3D σ =27 .

Revised: Gaussian 2D σ =26 .

Revised: Gaussian 2D σ =26 .

Revised: Gaussian 2D σ =26 .

Revised: Gaussian 2D σ =26 .

Revised: Ours ℓ max =28 . 45(5)

Revised: Ours ℓ max =28 .

Revised: One downside of all baselines is that there are two hyperparameters to be tuned: the encoding size 2 M and the scale σ or maximum frequency L .

Revised: First, we evaluate the sensitivity of to σ and L by keeping the encoding size constant and equal to 2 M = 512 .

Revised: Table 5 shows that the baseline methods are highly sensitive to these hyperparameter values, which partially explains our method’s advantages.

Revised: C. S ENSITIVITY TO MAXIMUM FREQUENCY

Revised: The 2D baselines have the major downside of ignoring the manifold topology.

Revised: The 3D baselines remove this limitation, but “Axis-aligned 3D” suffers from another limitation.

Revised: By increasing the dimensions from two to three while keeping the encoding size constant, each dimension must be sampled more coarsely.

Revised: This problem is exacerbated in IPDF (Murphy et al., 2021), which uses a 9D parametrization of a 3D manifold.

Revised: The “Gaussian 3D” baseline mitigates this issue by sampling directions with components on all dimensions.

Revised: As shown in Table 5, it produces the best results among the baselines, and approaches the performance of our spherical harmonics encoding.

Revised: In this section, we thoroughly evaluate the performance of our method against “Gaussian 3D” for different encoding sizes, which determine the maximum frequency of our method.

Revised: For our spherical harmonics encoding, this is the only hyperparameter to be set, while for the Gaussian we also have to pick the scale σ .

Revised: Figure 7 shows that we outperform the baseline for all evaluated encoding sizes.

Revised: [Figure]

Revised: C. I MPORTANCE OF SHIFT - INVARIANCE

Revised: Another major difference between our method and “Gaussian 3D”, is that our encodings have constant norm and rotation-invariant inner products, thus satisfying the NTK conditions for the outputs to approximate a convolution on the manifold (shift-invariance).

Revised: This is appropriate for the tasks we consider, where the relative importance between training inputs (coordinates) should not depend on their absolute position.

Revised: The Gaussian encoding does not satisfy these conditions and results in a translation-invariant encoding, which is not appropriate for points on the sphere.

Revised: In this section, we show the effects of intentionally breaking these two properties in our model.

Revised: We sample a vector, fixed during training, from a standard normal distribution with the same dimensions as the encoding.

Revised: We then replace the positional encoding by its pointwise multiplication with this vector of random factors.

Revised: The resulting encoding does not have constant norm nor rotation-invariant inner products.

Revised: Table 6 shows the performance on the “Spherical Text” dataset, with and without the random factors.

Revised: We notice a consistent drop in the average PSNR when ℓ max > 12 .

Revised: [Table]