Paper: Neural Processing of Tri-Plane Hybrid Neural Fields (ICLR-2024)

Reviewer comment (Writing):
The paper is difficult to follow.

Author response:
Thank you for pointing these weaknesses out. We will be glad to submit an improved version of the paper if you could kindly provide us with more actionable feedback, e.g. pointing us to the unclear paragraphs or describing the parts of the paper that suffer from poor presentation.

Reviewer comment (Presentation):
The presentation has room to improve.

Author response:
Thank you for pointing these weaknesses out. We will be glad to submit an improved version of the paper if you could kindly provide us with more actionable feedback, e.g. pointing us to the unclear paragraphs or describing the parts of the paper that suffer from poor presentation.

Reviewer comment (Evaluation):
The proposed method performs worse than existing point cloud methods as shown in Table 5.

Author response:
Thank you for pointing out a possible source of confusion. The aim of our paper is not to set a new state-of-the-art in point cloud/mesh processing. As done in a few other very recent papers (Dupont et al. ICML22, De Luigi et al. ICLR23, Zhou et al., NeurIPS23a/b, Navon et al. ICML23), we provide an answer to a different research question, i.e. whether and how it is possible to leverage neural architectures to process data encoded as neural fields. Neural fields are functions defined at all spatial coordinates, parameterized by a neural network, which have seen increased adoption in recent years to model a variety of data (images, shapes, radiance fields, etc.) due to their intrinsic continuity and the decoupling of memory requirements from resolution. Hence, the reason to provide an answer to the former question is twofold: (i) it is a novel and challenging research problem that requires to process neural networks with neural networks; (ii) as data are increasingly stored as neural fields, the cited papers conjecture that datasets consisting only of neural fields, i.e. shipped *without* the accompanying discrete data used to create them, will become available, and processing them with deep learning models will be of practical interest. For instance, this is already the case for NeRFs, which are stored and communicated without the original training images and, indeed, do exist only as neural fields.  Our work sets a new state-of-the-art in processing neural fields, outperforming all works addressing the same scenario, as shown in Table 2 and the first two rows of table 5. The aim of Table 3 and the last three rows of Table 5 is to show that by processing a variety of neural fields with the same network architecture we are able to achieve for the first time performance comparable to specialized architectures engineered to process discrete signals like point clouds, meshes, etc.

Reviewer comment (Novelty):
It is not obvious on the advantage of the proposed method over methods working with point clouds and / or meshes.

Author response:
Thank you for pointing out a possible source of confusion. The aim of our paper is not to set a new state-of-the-art in point cloud/mesh processing. As done in a few other very recent papers (Dupont et al. ICML22, De Luigi et al. ICLR23, Zhou et al., NeurIPS23a/b, Navon et al. ICML23), we provide an answer to a different research question, i.e. whether and how it is possible to leverage neural architectures to process data encoded as neural fields. Neural fields are functions defined at all spatial coordinates, parameterized by a neural network, which have seen increased adoption in recent years to model a variety of data (images, shapes, radiance fields, etc.) due to their intrinsic continuity and the decoupling of memory requirements from resolution. Hence, the reason to provide an answer to the former question is twofold: (i) it is a novel and challenging research problem that requires to process neural networks with neural networks; (ii) as data are increasingly stored as neural fields, the cited papers conjecture that datasets consisting only of neural fields, i.e. shipped *without* the accompanying discrete data used to create them, will become available, and processing them with deep learning models will be of practical interest. For instance, this is already the case for NeRFs, which are stored and communicated without the original training images and, indeed, do exist only as neural fields.  Our work sets a new state-of-the-art in processing neural fields, outperforming all works addressing the same scenario, as shown in Table 2 and the first two rows of table 5. The aim of Table 3 and the last three rows of Table 5 is to show that by processing a variety of neural fields with the same network architecture we are able to achieve for the first time performance comparable to specialized architectures engineered to process discrete signals like point clouds, meshes, etc.

Reviewer comment (Experiments):
How well does the proposed method generalize to unseen scenes?

Author response:
Results reported in Table 2 are on an unseen test set of neural fields, which is part of our benchmark and will be publicly released upon acceptance. Our method provides the best generalization.

Reviewer comment (Experiments):
Does the proposed method have a faster run time compared to mesh / point cloud methods?

Author response:
Yes, as shown by the inference times reported in the newly added Table 11 in Appendix F. Indeed, in our scenario data are available only as neural fields and thus running methods designed to process meshes or point clouds requires to reconstruct the discrete signal from the field, a step that makes classification/segmentation slower. It is worth noting that our method not only is faster, but it also enjoys constant run-time regardless of the point cloud resolution. Thanks to this decoupling, if one considers a different scenario where the discrete data structure is readily available, we are comparable to PointNet at low resolution, and again faster when resolution is increased.

Reviewer comment (Novelty):
The idea of using fitted triplanes for downstream tasks like classification is "obvious" in a sense.

Author response:
We agree that the idea of performing tasks on hybrid neural fields by processing their discrete component only is, to some extent, a natural development of the increasing research interest in these new representations. However, we believe that the notion that a tri-plane encodes enough geometric information to classify and segment the underlying signal cannot be taken for granted.  To the best of our knowledge, we are the first to experimentally validate this claim, which has proven to be conducive to a straightforward and effective method for the challenging and novel problem of neural processing of neural fields.

Reviewer comment (Novelty):
There is the general question of what the relevant application of the proposed approach might be. This is a problem for all methods that aim to solve downstream tasks on optimized neural field representations. Usually, data (images or point clouds, etc) was used to obtain the neural field in the first place. Solving the downstream tasks on this input representation instead of the neural field usually leads to better results, which is also confirmed by an experiment in the paper. The gap is reduced a lot though, in comparison to previous works.

Author response:
As described in the introduction, we embrace the vision, first proposed by [De Luigi et al. ICLR23], according to which neural fields will become a standard method to store and communicate 3D information, with repositories hosting digital twins of 3D objects and scenes stored as neural networks becoming commonly available. This will allow to decouple the memory cost of the representation and its spatial resolution (as a surface with arbitrarily fine resolution can be represented with a fixed number of parameters, i.e. the weights of the corresponding neural network), while also holding the potential to provide a unified framework to represent 3D information (instead of differentiating between point clouds, meshes, voxel grids, and scene radiances). In this setting, the original “explicit” data on which the neural field was originally trained might not be available to the user, who accesses the 3D object by downloading the weights of the corresponding neural network.   For instance, this is already the case for NeRFs, which are stored and shipped without the original training images and, indeed, do exist only as neural networks. Therefore, we believe that devising methods that solve tasks by processing neural fields as effectively as those who work on the original data is the key to unlock the benefits of the neural field representation and allow the aforementioned vision to become a reality.

Reviewer comment (Novelty):
The term "universal neural field classifier" (which the authors claim their method to be) is misleading, as the method is not for all neural fields but only for those represented as triplanes.

Author response:
Thank you for pointing this out, we agree that it can be misleading. We replaced it with “universal tri-plane classifier”.

Reviewer comment (Experiments):
The use of instance-specific MLPs as decoders for triplanes makes sense in terms of reconstruction quality. However, this also leads to some information being represented in the MLP, instead of the triplane. There is a question of how the downstream task quality changes when a shared MLP is used. This experiment seems to be missing in the paper, and it would be interesting to see the comparison.

Author response:
We added classification results in the revised version of the paper, in Table 12, Appendix G. Sharing the MLP leads to worse classification performance, which drops by more than 2 points. We also point out that a major limitation of deploying a shared MLP is the significantly worse reconstruction quality, as shown in Table 1 of the paper.  It is also worth highlighting how sharing the MLP requires its availability to create new neural fields, i.e. to create new test data, limiting the deployment scenarios of our methodology, which are instead equivalent to those of discrete data structures when using a (MLP, triplane) pair for each sample.

Reviewer comment (Reproducibility):
The proposed method necessitates per-object optimization to acquire individual tri-plane features. The author conducted a comparative analysis of object reconstruction using this technique against other solutions, which employed a shared network trained on the entire dataset, like Functa (Dupont et al.). They reported the performance and the number of parameters (see Table 1). However, it is worth noting that the required computational resources, particularly in terms of training time, have not been explicitly reported or discussed.

Author response:
Thank you for highlighting a missing dimension in our comparison. We ran additional experiments to measure the difference in fitting time across different ways to implement neural fields. Results are reported in the revised version of the paper in Table 10, Appendix F. Triplanes are the fastest to fit an entire dataset and require roughly half the time of SIRENs and one third the time of Functa.

Reviewer comment (Experiments):
In the ablation study, as presented in Table 6, the primary focus is on investigating various architectural aspects related to the classification of the learned tri-plane representations. While this provides valuable insights, it's important to highlight that certain variations within the proposed method, such as utilizing a shared MLP with distinct tri-planes across data, adjusting spatial resolution, or varying the number of channels within tri-plane structures, have not been subjected to ablation analysis. Addressing these aspects could provide a more comprehensive understanding of the method's performance and potential optimizations.

Author response:
Thank you for suggesting an interesting ablation study we haven’t performed. We tested down-stream classification performance for triplanes with a shared MLP for Manifold40 (SDF), reported in the new Table 12, in Appendix G. Results show that sharing the MLP negatively affects the classification performance, which drops by more than 2 points. This result, combined with the significantly worse reconstruction quality of the shared triplane (chamfer distance  0.18 mm of single versus 1.57 mm for shared MLP configurations) suggests that an MLP for each triplane should be the preferred setup with triplanes, and is indeed the one we have used in all our experiments. 
As for varying the number of channels and the resolution of the triplane, we conducted additional ablation studies following the reviewer's suggestion and reported them in the new Table 13 in Appendix G. Interestingly, the classification accuracy and reconstruction error are quite robust to the choices of the number of channels or the resolution of the triplanes, which therefore are not critical design hyperparameters.

Reviewer comment (Evaluation):
How do you explain the relatively lower performance of model when is trained and tested on the same neural field, like UDF and SDF as shown in Table 4? This seems to be counterintuitive.

Author response:
Thank you very much for being such an attentive reviewer and catching this error. The strange numbers are actually due to a swap of the values between the UDF and SDF columns, which is now fixed. Indeed, due to the swap the old wrong table was not coherent with the result of our method when trained and tested on SDFs (86.8), which was also reported in table 2.

Reviewer comment (Experiments):
As experiments show (e.g. Fig.3 right), the object representation mostly is encoded by the tri-plane parameters, rather than the MLP. This raises a question of whether the MLP can be shared across data samples for efficiency?

Author response:
As discussed above and validated by the results in Table 1 and the additional experiment reported in Table 12 of Appendix G, sharing the MLP significantly lowers the reconstruction quality and also negatively affects the downstream performance. The saving in terms of space are limited, as the MLP only has 15k parameters, while the time efficiency is exactly the same. Moreover, sharing the MLP requires its availability to create new neural fields, i.e. to create new test data, limiting the deployment scenarios of our methodology, which are instead equivalent to those of discrete data structures when using a (MLP, triplane) pair for each sample. Hence, we believe the advantages of sample-specific MLPs to outweigh the limited disadvantages and it is the configuration we recommend.