Paper: SPDER: Semiperiodic Damping-Enabled Object Representation (ICLR-2024)

Reviewer comment (Experiments):
I think that additional experiments could be used as there are other coordinate-based network architectures which claim improvements over positional encoding and SIREN architectures. I'm not sure why Instant-NGP is chosen here, as it is a hybrid implicit-explicit representation and is very different from the other coordinate-based networks. I believe a better fit would be [1] or [2], which show that simple networks with different activations, structures, or positional embeddings can improve on the FF/SIREN architectures.

Author response:
Following the reviewer’s suggestion, we ran additional experiments applying SPDER activations to the models from both of the suggested baselines: GaborNet from MFN [1] and Spline Positional Encoding [2]. Then, we evaluate these new models on the same image dataset as the original MFN paper [1].

**PSNRS**
| Training Steps | 5-layer SPDER | 5-layer GaborNet | Spline | 3-layer SPDER | 3-layer GaborNet |
|----------------|---------------|------------------|--------|---------------|------------------|
| 500            | 75.32         | 49.14            | 29.73  | 33.97         | 46.59            |
| 1000           | 78.01         | 56.07            | 38.87  | 38.62         | 52.78            |
| 5000           | 78.01         | 69.71            | 44.40  | 41.44         | 67.08            |
| 10000          | **108.40**        | 77.65            | 65.86  | 41.44         | 71.98            |

To highlight the difference, SPDER beats the best GaborNet loss in 800 steps and the best Spline loss in 300 steps. SPDER’s representation is lossless because the loss is so small it doesn’t show up on an 8-bit scale (not true for the others, see Appendix A.13). The 5-layer SPDER (which we tested in our paper) has significantly higher PSNRs than the others in very few training steps, meaning it is also very efficient.

We will be sure to include these results in the main paper.

**> I'm not sure why Instant-NGP is chosen here, as it is a hybrid implicit-explicit representation and is very different from the other coordinate-based networks.**

Its paper says it is “state-of-the-art quality on a variety of tasks” related to image representation. We wanted to show our simple MLP can beat this complicated setup which requires a heavy GPU setup.

Also stated in Appendix A.8, our method could also be combined with a hierarchical one like Instant-NGP, as it is more efficient.

Reviewer comment (Experiments):
Following the additional experiments thread, I'm not sure why comparisons on radiance field fitting are not included. While interesting as a toy problem, image signal representation is not very useful in many computer vision tasks, besides maybe super-resolution. However, coordinate-based networks have been very useful in computer graphics, where they can be used in solving inverse problems and reconstructing 3D from 2D. Not including an experiment on radiance field, which is the most impactful field where an improvement in coordinate-based network fitting would be utilized, leads me to believe that the impact of the method is severely limited due to some unknown reason, perhaps speed or memory constraints in the architecture?

Author response:
We observed that SPDER works for signals that can be well approximated in a low dimensional representation over a frequency domain. For example, an image can be well approximated by using the discrete cosine transform (JPEG). Similar methods exist for audio and video. To the extent of our knowledge, domains like SDFs or radiance fields do not exhibit a similar property. We hypothesize that this fact renders SPDER ineffective in these domains (mentioned at the bottom of page 4). This is likely an instance of the “no free lunch” theorem; no single architecture can perform the best on all modalities. 

We anticipate there can be future work on projecting these modalities into a vector space where SPDER’s activation functions can perform effectively. We believe this is reasonable as current methods for SDFs and radiance fields are generally divided into two parts: 1) preprocessing/conditioning 2) MLP. Our current implementation only uses the latter.

The main claim of our paper is that existing models are prone to the inductive bias of not being able to encode all frequencies, which we show our simple MLP overcomes. This is due to the “damping factor” in the nonlinearity (i.e. the square root in $\sin(x) * \sqrt{|x|})$ which allows the network to remember where inputs came from. This is most evident in the image case.

Reviewer comment (Experiments):
Even simpler than radiance field fitting, I don't understand why there is no experiment on overfitting 3D shapes, such as SDFs? This is where coordinate-based networks originated, and could be used as a compressive representation. The paper mentions compressive signal representations in the conclusion, but there is no study on how effective SPDER is for this - no study on memory consumption, or size of networks needed to reach an acceptable level of quality. This is usually done when trying to represent 3D objects with SDFs, so this leads me to believe the method may have some limitations here.

Author response:
We observed that SPDER works for signals that can be well approximated in a low dimensional representation over a frequency domain. For example, an image can be well approximated by using the discrete cosine transform (JPEG). Similar methods exist for audio and video. To the extent of our knowledge, domains like SDFs or radiance fields do not exhibit a similar property. We hypothesize that this fact renders SPDER ineffective in these domains (mentioned at the bottom of page 4). This is likely an instance of the “no free lunch” theorem; no single architecture can perform the best on all modalities. 

We anticipate there can be future work on projecting these modalities into a vector space where SPDER’s activation functions can perform effectively. We believe this is reasonable as current methods for SDFs and radiance fields are generally divided into two parts: 1) preprocessing/conditioning 2) MLP. Our current implementation only uses the latter.

The main claim of our paper is that existing models are prone to the inductive bias of not being able to encode all frequencies, which we show our simple MLP overcomes. This is due to the “damping factor” in the nonlinearity (i.e. the square root in $\sin(x) * \sqrt{|x|})$ which allows the network to remember where inputs came from. This is most evident in the image case.

Reviewer comment (Experiments):
While I appreciate the additional comparisons provided in the author response, and I believe the paper does a convincing job showing that overfitting image signals can be done extremely well with SPDER, I think the paper weakness is justifying *why* this is a relevant problem. The method seems to not work on radiance fields, has not been shown to work on SDFs, nor am I sure it offers compression benefits in overfitting image signals. Thus, I'm not sure what application it would be used in.

Author response:
We observed that SPDER works for signals that can be well approximated in a low dimensional representation over a frequency domain. For example, an image can be well approximated by using the discrete cosine transform (JPEG). Similar methods exist for audio and video. To the extent of our knowledge, domains like SDFs or radiance fields do not exhibit a similar property. We hypothesize that this fact renders SPDER ineffective in these domains (mentioned at the bottom of page 4). This is likely an instance of the “no free lunch” theorem; no single architecture can perform the best on all modalities. 

We anticipate there can be future work on projecting these modalities into a vector space where SPDER’s activation functions can perform effectively. We believe this is reasonable as current methods for SDFs and radiance fields are generally divided into two parts: 1) preprocessing/conditioning 2) MLP. Our current implementation only uses the latter.

The main claim of our paper is that existing models are prone to the inductive bias of not being able to encode all frequencies, which we show our simple MLP overcomes. This is due to the “damping factor” in the nonlinearity (i.e. the square root in $\sin(x) * \sqrt{|x|})$ which allows the network to remember where inputs came from. This is most evident in the image case.

Reviewer comment (Experiments):
I am concerned that this method tends to overfit heavily and easily to a given particular input image/signal, and evidence for generalization was very limited to (as far as I can tell) a single video example. How sensitive is the model to noise in the input, for example? This can be important for real-world applications.

Author response:
**> I am concerned that this method tends to overfit heavily and easily to a given particular input image/signal, and evidence for generalization was very limited to (as far as I can tell) a single video example. How sensitive is the model to noise in the input, for example? This can be important for real-world applications.**

The implicit neural representations should “overfit” to the signal or the object it is being trained on, as the goal is essentially to learn a mapping from pixel position to pixel value for a single object. The desired generalization is *between* the pixels in an image or between frames in a video. Our experiments demonstrate this generalization effect in the image superresolution task (Section 4.3, Appendix A.5), audio interpolation (Section 4.4, Appendix A.6), and video interpolation (Section 4.5, Appendix A.7).

Regarding video examples, we report on two videos (not one). They are “Big Buck Bunny” and “Bikes”, which are from skvideo and are standard in video computer vision research (Appendix A.7).

Reviewer comment (Experiments):
I am not an expert in this space, but one question I have is whether the baselines are strong. SIREN is the most commonly used baseline. Maybe the authors can provide some perspective in this regard.

Author response:
**> I am not an expert in this space, but one question I have is whether the baselines are strong. SIREN is the most commonly used baseline. Maybe the authors can provide some perspective in this regard.**

We compared SPDER to the most popular coordinate-based MLP models we found in the existing literature. Most of these models are simple neural networks. We also compared our work to Instant-NGP (Section 4.2 Image Representation), which uses a much more complicated setup and is state-of-the-art for many INR tasks.

Reviewer comment (Writing):
Lastly, I would recommend the authors, in the spirit of academic inquiry, to tone down the language a little bit. An example is a statement of the form: "We demonstrate how SPDER not only has incredibly low loss", can be restated as "We demonstrate how SPDER not only has significantly lower loss" etc. This is a minor point, admittedly my personal taste.

Author response:
**> Lastly, I would recommend the authors, in the spirit of academic inquiry, to tone down the language a little bit.**

We will change the wording accordingly.

Reviewer comment (Theory):
Do not prove the derivatives of different variants of proposed method.

Author response:
We prove how activation functions of the form $\sin(x) * \delta(x)$ for any sublinear $\delta(x)$ suffice for our task in Appendix A.1 (Proof of Lipschitz Bound).

Reviewer comment (Experiments):
Can you evaluate the proposed method on more complex images?

Author response:
We report our results on the DIV2K dataset (“DIVerse 2K resolution high-quality images”—commonly used for super-resolution and representing complex images) in Table 1. In fact, it is typical for the signal representation literature to evaluate only a few samples [1, 2, 3]. Our sample size is 800, which is far more than the largest one [3] with N=32.

[1] Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions, NeurIPS 2020.  
[2] Fathony et al., Multiplicative Filter Networks, ICLR 2021.  
[3] Tancik et al., Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS 2020

Reviewer comment (Experiments):
Consider to perform formal evaluations on popular benchmarks, such as CelebA, UHDSR4K and Vimeo-90k to demonstrate its superiority.

Author response:
We report our results on the DIV2K dataset (“DIVerse 2K resolution high-quality images”—commonly used for super-resolution and representing complex images) in Table 1. In fact, it is typical for the signal representation literature to evaluate only a few samples [1, 2, 3]. Our sample size is 800, which is far more than the largest one [3] with N=32.

[1] Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions, NeurIPS 2020.  
[2] Fathony et al., Multiplicative Filter Networks, ICLR 2021.  
[3] Tancik et al., Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS 2020