Paper: AUTOMATIC NEURAL SPATIAL INTEGRATION (ICLR-2024)

Reviewer comment (Novelty):
Even though this is a nice submission, I recommend rejection because I think that the contributions are, overall, too small.

Author response:
Thanks for recognizing the technical contributions of our paper. As for our main contribution, our goal is to introduce the idea of combining the Neural Field method with the Monte Carlo method using Control Variates. This contribution is very general as it can be applied to any spatial integration setting where the domain is structured. Unlike previous neural control variates methods where the networks are restricted to specific types of architectures, our methods impose minimal restrictions on the choice of network architectures. This allows users to choose more powerful and expressive network architectures, potentially leading to better numerical integration tools in the future. We believe that our contributions add knowledge to the field of numerical integration and can be beneficial to users with various applications in mind.

Reviewer comment (Evaluation):
The simulation results are okay, but the experiments lack a wall-time evaluation. An improvement in error-per-steps compared to plain Monte Carlo and plain AutoInt alone is insufficient because the proposed estimator heavily relies on gradient information, whose computational costs are only unveiled by wall-time measurements.

Author response:
Since our current implementation is based on Jax, which is highly under-optimized for both WoS and Neural networks, wall-time comparison might not be very reliable. It also depends on the hardware where we run the experiments in, as well as what kind of PDE and neural networks we used for the experiment. As a result, we prioritized to present equal sample comparison, as done by many papers (Sawhney et al. 2022).

Per the reviewer’s request, we’ve added the of our wall-time breakdown of our methods in comparison with the baselines, alongside our discussion of wall-time performance in the limitation sessions (Table 1 and L262-L265). Right now our proposed method takes more time compared to WoS to reach a certain MSE threshold in solving the 2D Poisson equation. We believe that the current profiling can be greatly improved if we investigate different ways to make neural network inference faster, such as implementing customized kernels. We also hypothesize that our method’s advantage will be more obvious when applying to solve more complicated PDEs, where WoS baseline will require more time per evaluation of integrands $f$ or per sampling from the domain.

Reviewer comment (Novelty):
The assessment would have been more positive if the contributions were more general or the simulation results were more convincing. As is, the recommendation is rejection.

Author response:
Thanks for recognizing the technical contributions of our paper. As for our main contribution, our goal is to introduce the idea of combining the Neural Field method with the Monte Carlo method using Control Variates. This contribution is very general as it can be applied to any spatial integration setting where the domain is structured. Unlike previous neural control variates methods where the networks are restricted to specific types of architectures, our methods impose minimal restrictions on the choice of network architectures. This allows users to choose more powerful and expressive network architectures, potentially leading to better numerical integration tools in the future. We believe that our contributions add knowledge to the field of numerical integration and can be beneficial to users with various applications in mind.

Reviewer comment (Theory):
It lacks theoretical guarantees for the performance of the proposed method. For example, the variance and bias of the estimator are hard to be assessed since the structure of the neural network used for estimation is undetermined.

Author response:
We would like to clarify the misunderstandings of the variance guarantee. Our method inherits the merits of Monte Carlo methods - unbiased and with predictable variance. As shown in the paper shows that our method’s variance can be proved to decrease in O(1/N)  (Equation 15 and L233-L235) 

Furthermore,  our network is directly optimized to reduce the variance, which we’ve shown in Equation 16. These advantages come with very few restrictions on the network architectures and parameters except for letting the network be n-th time differentiable, which we believe greatly expands the choices of functions for control variates.

Reviewer comment (Experiments):
The numerical experiments in the paper are limited.

Author response:
Per the reviewer's requests, we have provided more numerical experiment results. The additional results are consistent with those we provided within the paper and these results also support our claims that the proposed neural control variates estimator is unbiased and also low-variance. The additional results further suggest that our method can generalize to different 2D/3D domains and boundary conditions.

Reviewer comment (Theory):
The author(s) mentioned that the variance of the Monte Carlo estimator decays at the rate of O(1/N) and thus a large number of independent samples are needed for accurate estimation. What is the rate for the proposed method?

Author response:
The convergence rate of our proposed method is also O(1/N). This is shown in Figure 2 Convergence Plot and Section 5.3 (L233-L235)

Reviewer comment (Experiments):
Given the limited numerical experiments, it is hard to conclude that the proposed method can generally produce more accurate estimation than Monte Carlo. What are the main application scenarios of the proposed method?

Author response:
We have discussed this in the second paragraph of the introduction section(L21- L27). Applications include solving PDEs and physics-based rendering, etc. These applications greatly impact various industries including movie, gaming, and mechanical engineering. Specifically, in this paper, we demonstrate our method’s ability to produce more accurate and efficient PDE solvers for Poisson and Laplace equations, which are among the most fundamental PDEs.

Reviewer comment (Writing):
The principal weakness of this paper is that it is exceptionally unclear. There are many typos, the notations constantly change, some important concepts are not defined, some concepts are introduced twice, and some notations are confusing. This makes the full paper very hard to read. In this current state, this is clearly not fit for publication and requires a major rework before it becomes readable.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
The amount of edits required to make the paper readable is too large to make it acceptable, even if the authors promise to fix all the errors.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Theory):
In terms of methods, the authors do not really discuss the computational complexity of the method. In particular: what if the dimension d is large? For instance, the extension of auto-int requires computing a d-th derivative; the corresponding computational costs should be discussed.

Author response:
Thank you for asking this question. In our original paper, we discussed the computational complexity for different d: “Computing the integration requires evaluating the network approximating the anti-derivative $2^d$ times, with $d$ being the dimension of the space we are integrating in.” ( L271-L274) 

(Please refer to the General Response for details.) 

We've discussed how our method scaled with $d$ naively in the limitation in the paper. Empirically, our method could also be applied to higher dimensional problems with efforts currently: Our PDEs’ path integration domain is infinite dimensions. We demonstrate the applicability of our method by breaking down the infinite-dimensional integration domain into smaller dimensions. In other words, even though our techniques could not naively be applied to higher dimensional problems directly, as long as the problem could be broken down into smaller dimensions, our method would work out. 
In the meantime, we believe extending our method to be naively applicable in higher dimensions is an interesting future work problem. We’ve added this to the discussion in our current paper. (L274)

Reviewer comment (Writing):
In eq.1, f has two variables z, p. Then the authors write f(p) at the end of the paragraph. This confusion is present many times in the paper.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
Vectors are randomly written with either bold font or arrows on top: in eq.1 we have bold, in eq.2 we have arrows.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
At the top of page 2, G is a function of (a, b), then the authors write G'(x). There is also a constant change of notation between G_\theta and F_\theta.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
In the introduction the authors write G as a function of p, while later it will only act on the boxes [l_i, u_i] with variable x.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
The phrase "as takes" appears in the text.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
On page 3, the paragraph "Neural Network Integration Methods" contains many redundant citations.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
In Eq.4, yet another notation is introduced, which is redundant with Eq.1.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Theory):
The mapping $\Phi$ should be one-to-one, not only invertible.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
The expression "a region of R^d" should be replaced with a description of the region as a box constraint.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
$\Phi$ becomes $\Phi^{-1}$ later in the paper. The authors write $\Phi(x)$ in eq.5 but $\Phi(p)$ in, e.g., eq.11 and eq.12.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Theory):
The text "with Jacobian being r." is incorrect; it is the determinant of the Jacobian.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
The phrase "once can see" appears in the text.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
Around eq.8, there are too many different notations—"a, b, T, T_min, T_max"—which are confusing.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
Below eq.8, there is $F'_\theta$.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
The text "(i.e. for all pairs of" is missing the closing parenthesis.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
The notion of the antiderivative should be recalled precisely in the context of multi-dimensional functions.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
The notation $\frac{\partial^d G}{\partial x}$ should be explained clearly.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
The authors write $\Phi(dp)$ in many places; dp should just be p.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
In the big equation above eq.16, $I_\theta$ should be $G_\theta$.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Writing):
$F_\theta$ appears in eq.16.

Author response:
Thank you for pointing out all the typos and unclear notations. We have fixed them in the main paper and marked the change with a cyan color. We hope our revision can improve the mathematical rigorousness and clarity of our method section.

Reviewer comment (Theory):
The authors write "preliminary results show that our proposed method is unbiased": how is it not axiomatic? Can the authors discuss this more?

Author response:
Thank you for pointing out this language issue. Our method is unbiased by construction. The experiments' results further confirm this as our MSE decays at the rate of  O(1/N) where $N$ is the number of samples. We’ve corrected the language issue in the paper in L55-L56.

Reviewer comment (Experiments):
To the non-PDE audience, the experiments on solving a 2D Poisson equation and 3D Laplace equation seem somewhat out of context, so it would be good to explain, even if briefly, why this is an important problem.

Author response:
Thank you for this suggestion! We originally motivated our applications in the introduction section (L21-L27). Per reviewer’s request, we also include a motivation in the result section to improve the flow (L189-L192 in the paper).

Reviewer comment (Novelty):
Have the authors considered how to make the method more scalable? There is some related literature on control variates that may be relevant, e.g., https://arxiv.org/abs/2006.07487, https://arxiv.org/abs/2303.04756.

Author response:
Thank you for providing the relevant papers. As noted in the general response, our experiment is a demonstration that our method can be applied to solve integration problems in higher dimensions. We believe that similar principle might be applicable to other applications where high-dimensional integrations can be broken down into a set of lower-dimensional integrations. We believe that exploring how to apply the principle proposed by our method to different applications will be great future work research. We’ve also included discussion in our revision of the paper. (L271-274)

Reviewer comment (Experiments):
Can other parametric functions be used instead of the neural network?

Author response:
The parametric function can be used. Some specific parametric functions (such as a weighted sum of polynomial kernels) have been studied by prior works (Salaün et al. 2022) (L349- L350).  
In the meantime, our methods open a wider variety of network architectural choices for control variates. We achieve this by proposing a more general way to construct the control variates pairs (i.e. the G and g such that G = \int g), placing minimal restrictions on the network architecture. This allows us to use not only differentiable parametric functions but also other powerful universal approximators such as deep neural networks with nonlinearities.

Reviewer comment (Experiments):
Can the method be applied to other machine learning problems, e.g., problems of Bayesian inference? Have the authors considered this?

Author response:
We agree that extending our methods to Bayesian inference can be a very interesting research direction. One of the key insights of this paper is that when we need to construct a pair of functions for control variates, namely $G$ and $g$ such that $G = \int g$, we can first define a function to approximate $G$ and take its derivative fo find $g$. We believe that key insights like this can be potentially extended to other problems that might benefit from control variates, including Bayesian inference.

Reviewer comment (Experiments):
How sensitive is the estimator to sample size? Have you tried to run the method with different values of N?

Author response:
In the 3D Laplace experiment setting (Fig.3 in the paper and provided above), we have provided an example with different sample sizes for our method. As the sample size became larger, our method results were closer to ground truth. Our error is decreasing in the rate of O(1/N)