Paper: Neural implicit mapping via nested neighborhoods: real-time rendering of neural SDFs with textures (ICLR-2024)

Reviewer comment (Theory):
Both cases of neural attribute mapping are very simple and do not require the unnecessarily complex (pretentious?) formalism of “delta-nested neighborhood”, “integrating along the gradients”, “restriction of ∇f to S and mapping the normal along a path” presented in the paper.

Author response:
The reviewer was right about the text being too formal, compromising understanding. We changed our definitions in Section 3.2 to the minimum necessary. Now instead of a sequence with m neural SDFs, we have a sequence of 3 SDFs {h1, h2, f}, corresponding to the cases in our experiments. In practice, we may choose how to use functions h2 and f to adapt to a specific performance budget. For example, we may choose not to iterate using h2, but directly map the normals of the surface in f to the surface in h1. This setup relies on sphere tracing iterations using the coarse SDF, decreasing
the cost per iteration. Another possible case is to use h2 for a better surface approximation, and map normals of the surface of f as before. We can also use h2 and f for sphere tracing and do not perform
normal mapping. This setup has the best silhouette.
Figure 3 was added to help explaining in practice why we need the definition of nested neighborhoods, why making the deltas equal is too restrictive, and whats happens when they are too big or too small. It shows that the rendering may end up with holes or the silhouette may be lost depending on the values.

Reviewer comment (Experiments):
Case 2 of neural attribute mapping should be compared to a straightforward rendering of the more precise representation f, instead of transferring it. The speed benefits of a transfer of a high resolution field to a coarse surface should be exemplified. The only table reporting a speedup is Tab. 5, and it does not display a quality metric – only speedups.

Author response:
This is now better presented in Figure 10, which presents an perceptual evaluation of ground-truth meshes using rendered images of the meshes and our approach. Table 7 presents the MSE of between those images. All cases present error only in the second decimal digit.

Reviewer comment (Experiments):
The multi-scale sphere tracing algorithm is similar to the extension of marching cubes to multi-LOD presented in BACON. Its speed should be compared to extracting the surface (once) with marching cubes, and then rendering it with a rasterizer.

Author response:
We made a better job at the Introduction to motivate the use of sphere tracing. The alternative to render neural SDFs is to extract a triangle mesh using marching cubes. However, due to the non-real-time nature of marching cubes, its application may be prohibitive in certain contexts involving neural SDFs. For instance, rendering dynamic SDFs requires mesh extraction at each time instant of the animation, posing a memory challenge if performed as preprocessing (extracted meshes use hundreds of MB in high resolutions while neural SDFs use a few KB). The performance is also a concern when surfaces are extracted while the animation is playing. Additionally, the smooth nature of the level sets of neural SDFs, which is a desired property in rendering, may be compromised by discretizing them into triangle meshes. Sphere tracing offers a solution to capture this smoothness. The algorithm is employed for rendering complex shapes, such as fractals and may use recent ray tracing hardware. Advancing neural SDF rendering through sphere tracing is an important topic rooted in classic Computer Graphics.

Reviewer comment (Experiments):
The GEMM implementation of normals computations should be compared to standard backpropagation in pytorch or tensorflow. This is a component-wise computation of the normal vectors. The acronym GEMM is used several times without being defined, nor the concept. NeuS2 (Wang et al., ICCV 2023) also has a fast normal computation algorithm, and it might be worth adding the reference.

Author response:
We agree that this is of great importance and this version presents the comparison in Table 4. Our approach performs 2 times faster.

Reviewer comment (Experiments):
In the experiment section, when reporting training time vs NGLOD and IDF, it is unclear which neural network architecture is used and why it is called “ours.” The whole method and contributions are about rendering neural fields with multiple levels of details, not about a new architecture or a faster training procedure, so the comparison of training times is questionable.

Author response:
The reviewer is right, the previous version did not have enough details about the training. However, we have a specific training procedure now described in more detail in Section 3.3 (Nesting the Neighborhoods). We train h1, h2, f from fine to coarse and use the previously trained SDFs as ground truth for the next. That is, we consider f as ground truth for h2, which in turn is ground truth for h1. During the training of h2, we evaluate |h2 − f |∞, and choose the training step with the smallest norm. We do the same for h1. We note that the resulting norm |h2 − f |∞ is small during most of the training, a consequence of |h2 − f |^2 being a term in our loss function. Additionally, the above procedure provides a nested sequence that gives a robust rendering. Besides the resulting deltas being small (0.02 and 0.006 in Fig. 3), we note that decreasing them results in loss of silhouette (Fig. 3 (c)).

Reviewer comment (Experiments):
The paper does not make the setups of experiments sufficiently clear. The method and experiments section of this paper explains the architecture of the proposed nested SDF representation, however, it is hard to see what are the inputs to the experiments and what are the losses without reading appendix (A.3).

Author response:
We agree that more detail about training was necessary. We now explain our training procedure in more detail in Section 3.3 (Nesting the Neighborhoods). We train h1, h2, f from fine to coarse and use the previously trained SDFs as ground truth for the next. That is, we consider f as ground truth for h2, which in turn is ground truth for h1. During the training of h2, we evaluate |h2 − f |∞, and choose the training step with the smallest norm. We do the same for h1. We note that the resulting norm |h2 − f |∞ is small during most of the training, a consequence of |h2 − f |^2 being a term in our loss function. Additionally, the above procedure provides a nested sequence that gives a robust rendering. Besides the resulting deltas being small (0.02 and 0.006 in Fig. 3), we note that decreasing them results in loss of silhouette (Fig. 3 (c)).

Reviewer comment (Experiments):
The comparison with Instant-NGP (Table 2) seems misleading. If I understand the setting of the experiment correctly, the proposed method is trained with ground-truth SDF and color supervision, while the Instant-NGP is trained with multi-view image supervision. These are drastically different settings.

Author response:
The set of advices regarding Instant-NGP in this review were very important. Trying to find a SOTA method to compare against our approach, we overlooked a few important details. We removed that comparison agreeing that the approaches are different in nature and that we should not have used COLMAP when running Instant-NGP in our case. Since our approach is the first one to compute textures in real-time for neural SDFs, the comparison in the updated paper is against the ground truth textured meshes, presented in Table 7 which shows the MSE between the generated images. The error of our approach is in the second decimal digit for all cases. Figure 10 shows the images used to calculate the MSE.

Reviewer comment (Evaluation):
The comparison with Instant-NGP (Table 2) seems misleading. The images for Instant-NGP are generated with a renderer, thus COLMAP should not be required since it is used for camera pose estimation. Thus the speed comparison seems to be misleading.

Author response:
The set of advices regarding Instant-NGP in this review were very important. Trying to find a SOTA method to compare against our approach, we overlooked a few important details. We removed that comparison agreeing that the approaches are different in nature and that we should not have used COLMAP when running Instant-NGP in our case. Since our approach is the first one to compute textures in real-time for neural SDFs, the comparison in the updated paper is against the ground truth textured meshes, presented in Table 7 which shows the MSE between the generated images. The error of our approach is in the second decimal digit for all cases. Figure 10 shows the images used to calculate the MSE.

Reviewer comment (Experiments):
No performance comparison with the original mesh representation. Ideally, the paper should show the advantages of the proposed representation over triangle meshes in rendering captured objects. However, the training data seem to come from mesh data.

Author response:
The reviewer is right. We use triangle meshes as they provide surface and attribute data, which we need for our framework. However, we made a better job at the Introduction to motivate the use of sphere tracing. The alternative to render neural SDFs is to extract a triangle mesh using marching cubes. However, due to the non-real-time nature of marching cubes, its application may be prohibitive in certain cases involving neural SDFs. For instance, rendering dynamic SDFs requires mesh extraction at each time instant of the animation, posing a memory challenge if performed as preprocessing (extracted meshes use hundreds of MB in high resolutions while neural SDFs use a few KB). Performance is also a concern when surfaces are extracted while the animation is playing. Additionally, the smooth nature of the level sets of neural SDFs, which is a desired property in rendering, may be compromised by discretizing them into triangle meshes. Sphere tracing offers a solution to capture this smoothness. The algorithm is employed for rendering complex shapes, such as fractals and may use recent ray tracing hardware. Advancing neural SDF rendering through sphere tracing is an important topic rooted in classic Computer Graphics.

Reviewer comment (Evaluation):
Even though multiscale sphere tracing is 2x faster than sphere tracing with the original high resolution SDF, ~40 FPS still seems unusable in real applications compared with mesh-based representations.

Author response:
The reviewer is right. We use triangle meshes as they provide surface and attribute data, which we need for our framework. However, we made a better job at the Introduction to motivate the use of sphere tracing. The alternative to render neural SDFs is to extract a triangle mesh using marching cubes. However, due to the non-real-time nature of marching cubes, its application may be prohibitive in certain cases involving neural SDFs. For instance, rendering dynamic SDFs requires mesh extraction at each time instant of the animation, posing a memory challenge if performed as preprocessing (extracted meshes use hundreds of MB in high resolutions while neural SDFs use a few KB). Performance is also a concern when surfaces are extracted while the animation is playing. Additionally, the smooth nature of the level sets of neural SDFs, which is a desired property in rendering, may be compromised by discretizing them into triangle meshes. Sphere tracing offers a solution to capture this smoothness. The algorithm is employed for rendering complex shapes, such as fractals and may use recent ray tracing hardware. Advancing neural SDF rendering through sphere tracing is an important topic rooted in classic Computer Graphics.

Reviewer comment (Evaluation):
Where does the speedup of the proposed GEMM normal computation come from compared to an autodiff framework? Is there improvement in algorithmic complexity, or is it mainly a reduction in overhead (e.g., better use of GPU resources)?

Author response:
That is a core question and proper evaluation was lacking in the submitted version. In the updated version we compare our approach against autodiff in Table 4. Our approach performs 2 times faster. We do not think that this is a result of algorithmic complexity. We believe both approaches may be O(network inference). However, autodiff needs to perform a forward pass and to create and maintain a computational graph before being able to compute the gradient. Our approach just needs the input points and the network weights, calculating the normals in a modified forward pass. It needs to compute a forward pass for each one of the 3 normal coordinates, but those passes are independent and parallelized in Algorithm 2.

Reviewer comment (Theory):
Is the GEMM normal computation differentiable to be used in an optimization framework?

Author response:
That is a great question. We believe it is since we only need matrix multiplications and Hadamard products, and the activation function is smooth (periodic).

Reviewer comment (Theory):
Table 4 suggests that a larger MLP leads to a more accurate zero-level set. It would be beneficial to discuss the theoretical limits or boundaries for achieving perfect results with different MLP sizes.

Author response:
The reviewer is right, we provide now in Table 1 the Hausdorff distances between the ground-truth meshes and the surfaces of the neural SDFs and they suggest that larger MLPs could result in more accurate surfaces. However, our experience with SIREN networks is that they are capable of representing a huge frequency spectrum even with modest sizes. We speculate that the reason is because of the composition of periodic activation functions, which is capable of creating very complex sinusoids really faster. Thus, a very large MLP probably would result in a sinusoid too complex, which could be interpreted as noise. We suspect this is the reason why previous works have been reporting bad results using SIRENs. Those works tend to use networks too large.

Reviewer comment (Experiments):
In general, this algorithm uses more memory (coarse surfaces and finer surfaces) to trade speed. How does the speed compare with using extra spatial data structures, like bounding volume hierarchy, for faster Ray surface intersection?

Author response:
We compare against NGLOD, which uses uses a sparse voxel octree (SVO) to represent the neural SDF and to render its zero-level set using a sparse sphere tracing algorithm. Specifically, the vertices of
the voxels store features. In Figure 5 we show that one implication of that approach is discretization artifacts caused by non-continuous gradients. Our approach is not only continuous when using SIRENs, but it is smooth, since it calcules the attributes directly from the smooth neural SDF (smoothness is guaranteed by SIREN's periodic activation).

Reviewer comment (Theory):
The author does not explicitly bound the sequence of SDF, neither through training regularization nor some explicit normalization. The concept of nested SDF sequence is only mentioned in deriving the algorithm, but in the experiments, it is not guaranteed to be nested. Does it mean that for most of the popular neural SDF with level-of-detail design, without any explicit control of the bound of nested SDF, the proposed algorithm will always converge effectively?

Author response:
We agree that the paper should explain that better. The updated version has several improvements in that direction. We now explain our training procedure in more detail in Section 3.3 (Nesting the Neighborhoods). We train h1, h2, f from fine to coarse and use the previously trained SDFs as ground truth for the next. That is, we consider f as ground truth for h2, which in turn is ground truth for h1. During the training of h2, we evaluate |h2 − f |∞, and choose the training step with the smallest norm. We do the same for h1. We note that the resulting norm |h2 − f |∞ is small during most of the training, a consequence of |h2 − f |^2 being a term in our loss function. Additionally, the above procedure provides a nested sequence that gives a robust rendering. Besides the resulting deltas being small (0.02 and 0.006 in Fig. 3), we note that decreasing them results in loss of silhouette (Fig. 3 (c)).

We have a theorem (Prop. 1 in Section 3.3) which provides tight upper bounds for the deltas. Section 3.4 has an informal proof that the sphere tracing converges when using the deltas computed using Prop. 1.

Reviewer comment (Theory):
The tightness of the bound and how the tightness affects the algorithm is not analyzed and demonstrated.

Author response:
We agree that the paper should explain that better. The updated version has several improvements in that direction. We now explain our training procedure in more detail in Section 3.3 (Nesting the Neighborhoods). We train h1, h2, f from fine to coarse and use the previously trained SDFs as ground truth for the next. That is, we consider f as ground truth for h2, which in turn is ground truth for h1. During the training of h2, we evaluate |h2 − f |∞, and choose the training step with the smallest norm. We do the same for h1. We note that the resulting norm |h2 − f |∞ is small during most of the training, a consequence of |h2 − f |^2 being a term in our loss function. Additionally, the above procedure provides a nested sequence that gives a robust rendering. Besides the resulting deltas being small (0.02 and 0.006 in Fig. 3), we note that decreasing them results in loss of silhouette (Fig. 3 (c)).

We have a theorem (Prop. 1 in Section 3.3) which provides tight upper bounds for the deltas. Section 3.4 has an informal proof that the sphere tracing converges when using the deltas computed using Prop. 1.

Reviewer comment (Experiments):
I don't understand why the GEMM-based implementation will accelerate the normal computation. I thought auto-diff will also use GEMM to compute derivatives. Can you explain it more? Also, measuring the speed of normal computations will make this point more solid.

Author response:
The reviewer is right to assume that auto-diff uses GEMM to compute the derivates, however it also needs to perform a forward pass before that to compute the computational graph, and also has the additional overhead of creating and maintaining that graph. Our approach does not need any additional passes nor graphs, just the input points and the network weights. In the updated paper we compare our approach with the standard autograd normal computation (Table 4). Our approach is consistently 2 times faster.

Reviewer comment (Experiments):
In most of your experiments, you only use two nested SDF. What if you use more? Like, 3 to 5, even to 10 neural SDF?

Author response:
That question is legitimate and now instead of a sequence with m neural SDFs, we our definitions suppose a sequence of 3 SDFs {h1, h2, f}, corresponding to the cases in our experiments. In practice, we may choose how to use functions h2 and f to adapt to a specific performance budget. For example, we may choose not to iterate using h2, but directly map the normals of the surface in f to the surface in h1. This setup relies on sphere tracing iterations using the coarse SDF, decreasing the cost per iteration. Another possible case is to use h2 for a better surface approximation, and map normals of the surface of f as before. We can also use h2 and f for sphere tracing and do not perform normal mapping. This setup has the best silhouette.