Paper: Geographic Location Encoding with Spherical Harmonics and Sinusoidal Representation Networks (ICLR-2024)

Reviewer comment (Novelty):
Positional encoding for implicit neural representations on manifolds, such as the sphere, have been considered previously in Grattarola & Vandergheynst (2022), which is cited, and also in Koestler et al. (2022), which is not cited but should be. Grattarola & Vandergheynst do not specifically consider the sphere and spherical harmonics, instead focusing on graph representations, but consider an embedding based on the eigenfunctions of the graph Laplacian. Koestler et al. consider general manifolds, with an embedding based on the eigenfunctions of the Laplace-Beltrami operator. On the sphere, these eigenfunctions are specifically the spherical harmonics. So an embedding based on spherical harmonics as presented in this article is not new, although in this work the embedding is combined with SIRENs and extensive experiments on the sphere are performed.

Author response:
Thank you for the valuable insights in Koestler et al. (2022). We modified the introduction to acknowledge this work (blue font).

Reviewer comment (Experiments):
One limitation of the proposed approach is that it is limited to very low degrees L on the sphere, typically L of 20 or at most 40. The spherical harmonics are precomputed analytically, which while fine for these very low degrees will not scale to higher degrees.

Author response:
While the computational complexity increases with higher L, as shown in Figure 5, we would like to highlight that the computational runtime is __not excessive even until L=100__, which is the largest L harmonic function we pre-computed. Pre-computing the analytic functions of higher L is easily possible, but we found that such a high L was not necessary for the experiments of this paper.

To illustrate that runtime is not a major limitation, a forward pass of 256 coordinates up to L=100 (in total L$^2$ harmonics) takes only __716 ms ± 17.7 ms__ on a Macbook CPU.
In comparison, a forward pass through a ResNet 50 vision model on a [256 x 3 x 224 x 224] image batch takes 5.33 s ± 225 ms on the same computer. Hence, a SirenNet(SH) location encoder (even with a high L) will likely still contribute less to the runtime than a common vision model.

Reviewer comment (Evaluation):
It is indeed the case that combining the spherical harmonic encoding with SIRENs typically gives the best performance (see Table 1). However, in many cases the improvement is not that substantial, compared to the next-best method. The improvements for the ERA 5 dataset appear to be more marked.

Author response:
While the performance gain of SH may seem comparatively small, it is generally _consistent across all datasets_ and _across neural networks_. We show in Fig 4b) that SHs are most effective at high latitudes and polar areas where the spherical geometry of our planet matters the most. Here, other DFS-based location encoders systematically lose accuracy since their underlying assumptions on the geometry (i.e., the rectangular domain of lon/lat) are not met. Since only a few test points are located at high latitudes or polar areas, the accuracy benefits are numerically small in the tables.

Reviewer comment (Evaluation):
It seems combined spherical harmonic encoding with SIRENs is generally best. However, for the ERA 5 dataset it seems a fully connected network was significantly better than combining with SIREN. Do the authors have any idea why a fully connected network was best here?

Author response:
FcNet is a well-tested network for location encoding. In our opinion, FcNet remains a reasonable choice when it is paired with a suitable positional encoding. SirenNet achieves _similar and often better results_ and, crucially, _SirenNet is less sensitive to the underlying positional encoding_ and _predicts reasonably well even with direct encoding_ of longitude and latitude. In Section 2.3, we have shown that a specifically initialized SirenNet layer becomes equivalent to the Grid DFS-embedding. Out of these reasons, we see SirenNet as an architecturally simpler replacement over FcNet that archives reliably good results irrespectively of the positional embedding.

Reviewer comment (Theory):
Do the authors have any thoughts on how they could extend the method to higher degrees L?

Author response:
One heuristic to enable higher degrees L would be to pre-compute the numerical values of harmonics of explicit training points with a high L (like L=1000). These harmonics can then be loaded directly from the disk.

A more principled solution is likely found in Geodesy, where the Earth's gravity field is stored as weighted spherical harmonics up to several hundred Legendre polynomials (i.e., Linear(SH)). In this field, [Jekeli et al., 2006](https://link.springer.com/article/10.1007/s00190-006-0123-z) and [Balmino et al., 2012](https://link.springer.com/article/10.1007/s00190-011-0533-4) investigated approximations that allow integrating signals with very large L. We plan to explore this direction in future collaborative work.

Reviewer comment (Experiments):
The task on the ERA5 dataset was designed specifically for this work. It is difficult to assess whether the gain in using SH would also be significant in real-life climate science tasks, which is a domain of application that the paper puts forward. Would it be possible to integrate this method other tasks more common with ERA5 such as downscaling?

Author response:
The location encoder is not restricted to gridded data and can predict ERA5 variables continuously. Hence, generating a 1km rather than 30km grid (as in downscaling) is possible on a technical level. Still, adding further (higher-res.) covariates, like detailed topographic features, is likely necessary to improve the effective resolution. The location encoder can be used here as a module for a larger downscaling model.

Reviewer comment (Evaluation):
I am curious to hear the authors' intuition about the combination of SirenNet and some DFS based encodings performing worse than SirenNet alone? (e.g. SphereC+ + SirenNet on INaturalist 2018)

Author response:
Sphere{C,M} keep one lat/lon coordinate at the original (lowest-frequency) scale while scaling up the other. We hypothize that the lowest scale may be too low-resolution, which can be potentially detrimental. Beyond that, we have no further intuition other than that DFS-embeddings are not optimal for the Sphere in general (violated assumption of rectangular lat/lon domain). Regarding the good performance of SirenNet alone (i.e., with Direct), we show in Section 2.3 that a single Siren layer with specifically set weights can produce the same embedding as Grid. Hence, it is not surprising that Siren with direct coordinates performs competitively well.

Reviewer comment (Evaluation):
I am surprised that SphereM performs so poorly in comparison to the other positional embeddings (the difference in performance is not as striking in the Sphere2vec paper on their tasks). What is your take on this poor performance?

Author response:
Thank you for noticing that. 
We double-checked our implementation of SphereM{+}, which corresponds exactly to the code provided by Mai et al., 2023.

To make sure, we also re-ran the land-ocean classification on a different computer (now Macbook Pro on CPU/originally a GPU workstation), which resulted in very similar results to Table 1b:

|PE, NN|Linear|FcNet|Siren|
|-|-|-|-|
|SphereC|$79.9 \pm 0.3$|$94.9 \pm 0.1$|$95.2 \pm 0.1$|
|SphereC+|$84.4 \pm 0.0$|$95.1 \pm 0.1$|$95.4 \pm 0.0$|
|SphereM|$74.0 \pm 0.0$|$89.0 \pm 0.3$|$88.3 \pm 0.4$|
|SphereM+|$81.9 \pm 0.2$|$91.6 \pm 0.2$|$93.7 \pm 0.1$|
|SH (ours)|$94.1 \pm 0.0$|$95.7 \pm 0.1$|$95.8 \pm 0.1$|

Hence, we do not have an explanation for the poor SphereM performance and can only stress that we run all models in the same environment.

Reviewer comment (Evaluation):
When using FCNet or SirenNet (which is not much of a burden in practice), the gains from SH are fairly small (<1%). The gains from SH are only large when using a low-capacity network. Similarly, in Figure 4(a) we see that the difference between $L=10$ and $L=20$ is only significant when using low-capacity networks. To me, this seems to suggest that "expressiveness" is the main issue. From a user's perspective, does it matter whether the expressiveness comes from the input encoding or the network? What's the difference in computational efficiency between SH + SirenNet and X + SirenNet where X is another high-performing input encoding? What's the best argument (combining computational efficiency and performance) that SH is worthwhile to use as a component in a real system compared to the alternatives?

Author response:
> Reviewer:
> When are the performance gains from SH worth it?

The quantitative differences between SH-based encoders and DFS-based encoders may be numerically small but are systematic for a) different neural networks and b) present across different datasets (Tables 1 and 2).

Spherical harmonics are inherently better suited for spheres than DFS-based encodings that assume a rectangular domain between longitude and latitude. As shown in Figure 4b), this assumption does not hold at high latitudes, where DFS-based encoders drop systematically in accuracy, while SH remains accurate.

> Reviewer:
> The gains from SH are only large when using a low-capacity network. [...]. To me, this seems to suggest that "expressiveness" is the main issue.

An expressive high-capacity neural network, such as FcNet or Siren, can compensate for inaccuracies of sub-optimal DFS-based embeddings. 
In our view, the good performance of SH even on the Linear "low-capacity network" highlights that spherical harmonic basis functions alone are inherently better suited for global-scale prediction tasks.

> Reviewer:
> What's the difference in computational efficiency between SH + SirenNet and X + SirenNet where X is another high-performing input encoding?

The table below shows runtimes for the land-ocean dataset (Table 1b) $\pm$ from 5 runs. We reran the models on CPU on a Macbook. The SH embeddings in this used 10 Legendre Polynomials (determined by hyperparameter tuning as detailed in the appendix).

|runtime in sec|Siren(SH) L=10|Siren(SphereC)|Siren(SphereC+)|Siren(SphereM)|Siren(SphereM+)|
|:-|-:|-:|-:|-:|-:|
|train|23.31 $\pm$ 2.14|19.23 $\pm$ 1.37| 12.84 $\pm$ 1.23|14.00 $\pm$ 1.57|43.12 $\pm$ 7.96|
|test|0.45 $\pm$ 0.03|0.80 $\pm$ 0.03|0.45 $\pm$ 0.00|0.44  $\pm$ 0.04|0.70 $\pm$ 0.04|

In practice, we see no disadvantage in terms of the runtime of one over another.

> Reviewer:
> What's the best argument [...] that SH is worthwhile to use as a component in a real system compared to the alternatives?

SH basis functions are effective for all networks and perform well even for simple (i.e., Linear) neural networks (Tables 1,2), they can be faster than DFS encodings until L=20 (Figure 5) and are even up to L=100 not excessive (5.62 sec. run time for 2000 points on Macbook CPU).

Reviewer comment (Experiments):
FCNet and SirenNet are "typically" implemented with a certain number of layers. I might have missed it, but how many layers does SirenNet have in this work? Are FCNet and SirenNet matched in this sense? Do they behave differently as the capacity of the networks increases? What is the effect of training duration / how well the network has fit to the training data? These factors seem very important for comparing the results for these two networks, but I do not see them discussed in the text. This makes it difficult to evaluate key claims in the paper.

Author response:
> Reviewer:
> Isn't network capacity and degree of fit a significant confounder in the experiments? [...] How many layers does SirenNet have in this work? 

We tune each PE-NN combination separately for 100 epochs. The hyperparameter space includes hidden dimensions within the networks, number of layers, learning rate, weight decay, number of frequencies S, $r_{min}$. The exact parameter range is detailed in the appendix.

> Reviewer:
> Are FCNet and SirenNet matched in this sense? 

In the hyperparameter tuning procedure, they are matched in the sense that we take the best-performing model configuration on the validation set.

> Reviewer:
>Do they behave differently as the capacity of the networks increases? 

We see no clear connection between model capacity and classification/interpolation performance. Simpler (lower-capacity) models also interpolate better towards spatially different test points.

> Reviewer:
> What is the effect of training duration / how well the network has fit to the training data?

During training, we monitor the validation loss and perform early stopping with patience of 30 epochs. We do this for all NN-PE combinations consistently, as detailed in the revised appendix.