Paper: Accurate Differential Operators for Neural Fields (ICLR-2024)

Reviewer comment (Experiments):
It seems that the paper's claim regarding motivation from empirical observations might be somewhat overstated, particularly when applying it to **all** types of neural fields. The problem of noise overfitting is notably more prevalent in neural fields that heavily emphasize local details but might not be as prominent in other more global architectures like SIREN or Multiplicative Filter Networks. The authors should consider providing additional evidence, particularly in the context of these architectures, to support their claim for all neural fields.

Author response:
We have addressed this in the general response, under “_Experiments on … (SIREN, etc.)_”

We would like to clarify that we only claim to address inaccurate derivatives caused due to high-frequency noise in neural fields. While our approach is not tied to a particular architecture, as discussed, we found such noise to be more prevalent in hybrid neural fields. Non-hybrid architectures like SIREN seem to be plagued mostly by low-frequency errors. We have made edits to the main text to make it clearer and have also added our preliminary analysis on SIREN to Appendix F in the revised paper.

Reviewer comment (Novelty):
Novelty: The proposed method can be applied to any continuous function, but its contribution from a machine learning perspective is limited apart from adding the fine-tuning loss.

Author response:
As rightly noted by __Reviewer LQfg__, the simplicity of our approach is also its strength, and its effectiveness is supported by our quantitative results. Furthermore, to the best of our knowledge, ours is the first work to highlight the issue of high-frequency noise in hybrid neural fields. We have also not come across any other work that claims to address the issue of inaccurate spatial derivatives using autodiff for hybrid neural fields, like our fine-tuning approach. Nor could we find any previous work that uses polynomial-fitting in the context of neural fields to address inaccurate derivatives.

Reviewer comment (Experiments):
It seems that the authors modified the neural field architecture used in Chen et al., (2023a) for the application of PDE simulation (sec. 5.3). Will the claim of avoiding prediction error explosion still hold with the original architecture?

Author response:
We do not claim that the prediction error would also explode for SIREN, i.e., the architecture used in Chen et. al. We also tried Chen et al.’s approach in our PDE simulation experiment and it did not suffer from exploding prediction error (see Appendix C, under "_Comparing PDE simulation with INSR_" and Figure 10 in the supplementary). 

As we discussed in the general response, under “_Experiments on other … (SIREN, etc.)_”, SIREN suffers from a different kind of noise which is mostly low-frequency. While it does still lead to inaccurate derivatives, it does not lead to an error explosion in this case.

We would also like to emphasize that our motivation with the PDE simulation experiment was to merely show that without our approach, hybrid neural fields like Instant NGP lead to exploding errors in applications like PDE simulation, thus limiting their downstream applications. We also observe in Figure 10, that despite minimal additional tuning, using our approach with the hybrid neural field leads to comparable performance w.r.t. INSR, a state-of-the-art approach for this problem.

Reviewer comment (Experiments):
When working with signals that inherently contain high-frequency features, as often encountered in applications like Physics-informed approaches (such as Chen et al., 2023a) for fluid dynamics, it may not be appropriate to employ unified smoothing parameters across the entire domain.

Author response:
It is indeed true that using a unified smoothing parameter is not ideal when the signal itself may have high-frequency components. However, we would like to point out that our operator is capable of accepting a different smoothing parameter for each query point. We agree that this however does not address the challenge of finding the right $\sigma$ for each point of the field. That would depend on the downstream task as well. But as we discuss in Section 6 (under “_Discussion and future works_”), the dependence of $\sigma$ on the downstream task is not limited to our approach, but in fact, is common in other domains where derivatives are computed, like Image Processing.

Reviewer comment (Writing):
Minor issues: Please double-check the citations in your paper. There are some instances of duplicated citations (e.g., "Chen et al., 2022c" and "Chen et al., 2023a", "Müller et al., 2022a" and "Müller et al., 2022b", "Tancik et al., 2022a" and "Tancik et al., 2022b") and incorrect formatting (Sara Fridovich-Keil and Alex Yu et al., 2022).

Author response:
Thank you for pointing out the formatting issues. We have addressed them in the revised version. Also regarding Fig. 7, we had mistakenly plotted the error from the first time step and omitted the error at t=0. We have fixed this in the revised version.

Reviewer comment (Experiments):
Does the prediction error explode using the original architecture with SIREN as in Chen et al., (2023a)?

Author response:
We do not claim that the prediction error would also explode for SIREN, i.e., the architecture used in Chen et. al. We also tried Chen et al.’s approach in our PDE simulation experiment and it did not suffer from exploding prediction error (see Appendix C, under "_Comparing PDE simulation with INSR_" and Figure 10 in the supplementary). 

As we discussed in the general response, under “_Experiments on other … (SIREN, etc.)_”, SIREN suffers from a different kind of noise which is mostly low-frequency. While it does still lead to inaccurate derivatives, it does not lead to an error explosion in this case.

We would also like to emphasize that our motivation with the PDE simulation experiment was to merely show that without our approach, hybrid neural fields like Instant NGP lead to exploding errors in applications like PDE simulation, thus limiting their downstream applications. We also observe in Figure 10, that despite minimal additional tuning, using our approach with the hybrid neural field leads to comparable performance w.r.t. INSR, a state-of-the-art approach for this problem.

Reviewer comment (Experiments):
In Fig. 7, the grid solver gives non-zero loss at the initial time. Is there any issue in the simulator?

Author response:
Thank you for pointing out the formatting issues. We have addressed them in the revised version. Also regarding Fig. 7, we had mistakenly plotted the error from the first time step and omitted the error at t=0. We have fixed this in the revised version.

Reviewer comment (Experiments):
The primary concern lies in the scalability of this method, as most of the presented results pertain to normal-scale setups. While these results are suitable for the paper's scope, questions arise regarding the method's ability to handle gigascale scenarios, especially with the availability of gigascale NeRF implementations.

Author response:
Thank you for the positive review of our work! We are elated that you agree with our problem formulation, and liked our “novel” approach and our experimental results.

- - -

## Scalability
We can break down the scalability question into two parts – post hoc operators and fine-tuning:
- ___Post hoc operators___: Our post hoc operators act on the pre-trained models. Hence, it is possible to apply our operators directly even on gigascale models.
- ___Fine-tuning approach___: Our fine-tuning approach can also be applied to gigascale models. Our current formulation of post hoc operators would indeed require an increased training budget owing to the multiple forward passes required for the smoothed gradient computation for supervision. As we discuss in Section 6 (under “_Discussion and future works_”), caching local neighborhoods between nearby points or designing more efficient sampling patterns could be some approaches to address this. 
- - -

Reviewer comment (Experiments):
It would be beneficial if the authors could provide more insights into the scalability of their method and its potential applicability to gigascale NeRF scenarios.

Author response:
Thank you for the positive review of our work! We are elated that you agree with our problem formulation, and liked our “novel” approach and our experimental results.

- - -

## Scalability
We can break down the scalability question into two parts – post hoc operators and fine-tuning:
- ___Post hoc operators___: Our post hoc operators act on the pre-trained models. Hence, it is possible to apply our operators directly even on gigascale models.
- ___Fine-tuning approach___: Our fine-tuning approach can also be applied to gigascale models. Our current formulation of post hoc operators would indeed require an increased training budget owing to the multiple forward passes required for the smoothed gradient computation for supervision. As we discuss in Section 6 (under “_Discussion and future works_”), caching local neighborhoods between nearby points or designing more efficient sampling patterns could be some approaches to address this. 
- - -

Reviewer comment (Experiments):
All the experiments were conducted with instant-ngp, a particular implementation of a hybrid neural field. Therefore, it is difficult to understand if the problem is systematic across all methods of this class. It would also have been interesting to compare the method with non-grid-based neural fields, such as SIREN, Fourier Features, MFN, etc.

Author response:
## Comparison to other hybrid/non-hybrid architectures

We have addressed this in the general response, under “_Experiments on … (SIREN, etc.)_”. 

Amongst hybrid neural fields, we found that apart from Instant NGP, even “dense” multi-grid architectures that do not utilize hash embedding, as well as tri-plane neural fields also suffer from similar inaccurate derivatives caused by high-frequency noise that our method can address. For more details, please refer to the general response.

As discussed in the general response, we would like to clarify that we only claim to address inaccurate derivatives caused due to high-frequency noise in neural fields. While our approach is not tied to a particular architecture, as discussed, we found such noise more prevalent in hybrid neural fields. Non-hybrid architectures like SIREN seem to be plagued mostly by low-frequency errors (Appendix F in supplementary) which cannot be corrected by our approach

Reviewer comment (Experiments):
Apart from the PDE simulation section - which seems to be unrelated to the rest of the paper - all the experiments were done with SDF data. It would strengthen the paper if the method would also apply to other modalities.

Author response:
## Experiments on other modalities and datasets

Based on your feedback regarding experiments on other modalities, we have also added some results on neural fields for images. See Appendix C, under “_Results on images_”).

In addition, we also show experiments on PDEs, precisely to demonstrate that our approach is beneficial in other applications apart from improving derivatives of SDFs (Section 5.3). In our PDE experiment, we use our accurate differential operators to compute the self-supervised update shown in Eq. 4 of the paper and compare it to the case when the AD derivative is used.

We would also be happy to test our method on any other specific dataset that you suggest.

Reviewer comment (Experiments):
For post hoc operators, how do you find the best value of $\sigma$? What is the criterion to choose the $\sigma$ that yields the best autodiff gradients?

Author response:
## Hyperparameter details

We have provided details for hyperparameter selection in Section 4 (under “_Choosing $\sigma$ and $k$_”), Appendix A, and Section 6 (under “_Discussion and future works_”).

Reviewer comment (Experiments):
Did you try your method on other datasets and with other modalities?

Author response:
## Experiments on other modalities and datasets

Based on your feedback regarding experiments on other modalities, we have also added some results on neural fields for images. See Appendix C, under “_Results on images_”).

In addition, we also show experiments on PDEs, precisely to demonstrate that our approach is beneficial in other applications apart from improving derivatives of SDFs (Section 5.3). In our PDE experiment, we use our accurate differential operators to compute the self-supervised update shown in Eq. 4 of the paper and compare it to the case when the AD derivative is used.

We would also be happy to test our method on any other specific dataset that you suggest.

Reviewer comment (Reproducibility):
What is the training pipeline for fine-tuning? How long does it take?

Author response:
## Computational Cost 

We address this in the general response, under “_Computational Cost_”. For fine-tuning the details of the training pipeline are provided in Appendix B.4.

Reviewer comment (Experiments):
Could you provide more details on the PDE simulation section? Can you compare your result with standard INR?

Author response:
## Clarifications about PDE results

All the details for the PDE simulation experiments are shared in Section 5.3 and Appendix E (under “_PDE Simulation_”). We would be happy to provide any other specific information that is required.

**Comparison to INSR**: (We assume INR in the review is a typo). We have added a plot in the supplementary that compares our approach with the original INSR implementation [A] (Appendix C, under “_Comparing PDE simulations with INSR_”). Our approach allows hybrid neural fields to perform competitively w.r.t. INSR which uses a completely different architecture (SIREN). We would further stress that the primary comparison should be between our approach and AD, because of this difference in representation type from INSR and the reasons noted in “_Experiments on … (SIREN, etc.)_”.
- - -
[A] Implicit Neural Spatial Representations for Time-dependent PDEs. Chen et al. ICML 2023

Reviewer comment (Novelty):
**Lack of rigorous literature review**: It seems that the authors have ignored some of the very relevant papers in this domain. The problem of inaccurate and noisy gradients is a well known problem and several people have proposed alternative solutions: (1) DiGS : Divergence guided shape implicit neural representation for unoriented point clouds (2) Given the high cost of dense point cloud capture, can we use sparse point clouds?​ (3) Use a local region prior to generalize to various unseen local reconstruction targets​ (4) NeuFENet: Neural Finite Element Solutions with Theoretical Bounds for Parametric PDEs (5) mechanoChemML: A software library for machine learning in computational materials physics. These are just a few examples of how people try to use other approaches to compute the derivative than, the approach that authors take. While the authors may and probably will try to defend the paper saying that these papers are not relevant, I would argue that they are. For example: [2] uses the nearest neighbors idea to compute SDF and update the derivative (using AD of course). [4,5] use Lagrange polynomials from Finite Elements to evaluate the gradients for backprop, [1] solves the AD noise problem by adding an additional loss for having zero divergence. Each of them are valid alternative approaches to what the authors of this paper are planning to do. A good paper would try to compare with them and ensure that their approach is truly necessary.

Author response:
We appreciate all the pointers, and agree that the problem of inaccurate derivatives appears in many many settings. We will cite these papers in our related work section. However, our work is primarily focused on hybrid neural fields which (a) have a specific kind of high-frequency noise which is exacerbated by derivative computation, and (b) are used commonly enough that these issues will plague many downstream applications. In particular, 
- (1) assumes a SIREN network, and aspects of the method, like initialization strategies, are specifically designed for SIREN. 

    - As we discussed in the general response (under “_Experiments on … architectures (SIREN, etc.)_”), the issue of high-frequency noise is not present in SIREN (See Appendix F in the modified paper). Hence comparing with a baseline based on SIREN would not be an apples-to-apples comparison.
    - Furthermore, the work does not provide evidence that any specific aspects of their method lead to more accurate AD derivatives. Hence, it is not clear why trying to adapt it for hybrid neural fields would help with our motivation.
    - While they do have a term that minimizes the divergence of autodiff gradients, it is different from our alignment loss. 
- (4) & (5): The methods described in these approaches use significantly different representations from hybrid neural fields. 
     - Our motivation with this work is to improve a well-accepted hybrid neural field like Instant NGP to make it better at providing more accurate derivatives. We could not find previous works that utilize Lagrange polynomials with hybrid neural fields for more accurate derivatives. 
    - Adopting Lagrange polynomials into hybrid neural fields is a really interesting approach, but we feel that this would be a completely different alternative that should be explored in a separate paper. it could be better explored as a future work. This is because it would lead to an altogether different representation with its unique properties. 
- (2) & (3) do not seem to be the right titles of the papers. We would be grateful for a clarification so that we can address them.