Paper: INRet: A General Framework for Accurate Retrieval of INRs for Shapes (ICLR-2024)

Reviewer comment (Experiments):
In the baseline where retrieval is based on conversion to explicit representation, the quality of the reconstruction is not quantified, unless I missed it. Therefore it is not possible to know whether these baselines are worse than the proposed approach because the latter is superior or because the reconstruction is not good. Unless I missed something, an alternative may be to use the original meshes for retrieval rather than a reconstruction.

Author response:
> __Reconstruction quality and its impact on retrieval performance__

We provide details on the reconstruction quality achieved by various architectures in this work in Appendix C.1 Table 15. We also provide more visualization in Appendix C.2 Figure 4 to demonstrate the reconstruction quality.

To understand how the retrieval accuracy is impacted by the reconstruction quality, we perform another experiment in Appendix C.1 (Table 16) where we train an iNGP INR with a small number of epochs (2) so that the iNGP INR has a lower reconstruction quality than an MLP-only INR. In summary, we found that despite having lower reconstruction quality than the MLP-only INR, the features in the iNGP hash grid still made retrieval accuracy from a low-reconstruction quality iNGP higher than the MLP-only INR. Therefore, supporting INRs with feature grid in INRet is essential for improving the retrieval accuracy irrespective of the reconstruction quality of the INRs. Please refer to Appendix C.1 Table 16 for more details.

> __Meshes vs Reconstruction__

We would like to note that for INRet, we use the mesh to train INRs and then perform retrieval using the features and weights in these trained INRs. Our goal in this work  is to analyze how retrieval can be done effectively if the shapes are implicitly represented with different kinds of INRs.

If we were to support querying the INR database using meshes, there are two possibilities. Since INRet is designed to store INRs, to query INRs with the mesh, we can either convert the query mesh into an INR and use INRet, or reconstruct the INRs in the database and render into multi-view images and use view-based retrieval methods. As we have demonstrated in the experiments, converting the mesh to INR and use INRet is more accurate compared to multi-view-based methods (Section 5.2 Table 1).

Reviewer comment (Presentation):
Figure 2 could benefit from the addition of symbols used in the text to denote the different elements, as done in Figure 1.

Author response:
> __Figure 2 Labelling__

We thank the reviewer for the suggestion and we have added additional labels/symbols in Figure 2.

Reviewer comment (Experiments):
I am not sure how likely a database containing INRs encoded using various types of INRs is.

Author response:
> __Database containing various types of INRs__

INRs are becoming a very popular representation for shapes, graphics objects, scenes etc. For example, nerfstudio has recently published a tool to export trained NeRF scenes to unreal engine, which would make production use of these INR models much easier [1]. We also see companies starting to use INR to encode many forms of data, such as encoding fonts [2], materials [3] and various forms of 3D assets [4]. We expect to see more tools like this in the future, which means INRs will have an increasing impact in the real world. This would require a database to store these models for efficient retrieval, similar to relevant head searching in existing mesh-based databases [5].

[1] Nerfstudio: Exporting to Unreal Engine [nerfstudio, Unreal Engine [https://docs.nerf.studio/extensions/unreal_engine.html]

[2] Joint Implicit Neural Representation for High-fidelity and Compact Vector Fonts [ICCV2023, Adobe, https://openaccess.thecvf.com/content/ICCV2023/papers/Chen_Joint_Implicit_Neural_Representation_for_High-fidelity_and_Compact_Vector_Fonts_ICCV_2023_paper.pdf]

[3] Real-Time Neural Materials using Block-Compressed Features [preprint, Ubisoft, https://hal.science/hal-04255874v1]

[4] [Luma AI, https://lumalabs.ai/]

[5] [Ubisoft, https://montreal.ubisoft.com/en/using-ml-and-complex-math-to-deconstruct-and-reconstruct-human-faces/]

Reviewer comment (Reproducibility):
Based on my understanding of the approach, I think the encoding of the shapes in the database must be trained with the decoders used to create the INR embeddings (part b and c in Figure 2). Thus it is not clear to me why handling multiple types of representation is necessary. If the database owner knows querying and retrieval is necessary (as part b and c have been trained), why not create an embedding using a single type of representation? Of course, encodings trained before the system is available can be handled by the distillation approach. In experiments 5.2 and 5.3, I was also not able to precisely understand whether the encodings were trained with or without the decoder.

Author response:
> __The need for encoder/decoder after the training process. The need for multiple representations__

One notable feature of INRet is that the user models inserted into the database do not need to be trained along with the decoders that are used to help train the INR encoders. The user INRs can be trained from scratch by the user without having access to the encoder or decoder. To be more precise, the common decoder in Figure 2(c) is not needed for the embedding creation as new INR models are added to the database. The encoder in Figure 2(b) is frozen after the initial database setup, and is required when new INR models are added to the database. In summary, once the encoders are trained, given an INR pretrained by the user, the INR embedding can be created with a single forward pass through the trained encoders, which is much faster than converting the trained INR to other representations such as point cloud.

We leave the option of creating an embedding using different INR representations because different INRs have different strengths. For example, as reviewer “U5A9” has pointed out, a multi-layer car might only be properly represented with a UDF INR. In the meanwhile, one may want to represent a watertight surface and ensure it is watertight so he/she may choose an SDF INR. We design INRet so that the users can feel free to use the best INR implicit function for the most suitable representation without worrying about the incompatibility between them for the retrieval process.

Reviewer comment (Presentation):
For the same figure, I would like to suggest labeling the models that are mesh specific and global for the approach. Though not necessary, it may increase the clarity of the figure.

Author response:
> __Figure 2 mesh specific and global__

INRet is not generally tied to mesh. We use mesh in this work only as a starting point to perform training of INRs. Figure 2(a) involves training of the INRs from mesh, however, specific INRs can also be trained from other 3D representations (UDF from point cloud, Occ from voxel grid). The encoders in Figure 2(b) are used for encoding new INRs put into the database, they are therefore global throughout the training and deployment phases. The decoder in Figure 2(c) is only used for the encoder training, and is not useful once the encoders have been successfully trained.

Reviewer comment (Experiments):
As far as I know, features from different hierarchical levels of a grid-based INR are usually concatenated before being fed to the network. I was surprised to see that in this paper they are summed together (p5, top paragraph). Is there any advantage/downside to summing them?

Author response:
> __Summation and concatenation of the features__

We choose summation or concatenation of the features based on how the features were originally used in the grid-based INR. NGLOD used summation of the features and iNGP used concatenation of the features. One of the reasons for their design choice might be iNGP requires all the features to mitigate the negative effects of hash collision using the MLP, but NGLOD does not have the issue. INRet follows their design choice and uses the same method to combine features as input to the Conv3D encoder.

We ran an additional experiment to test out what would happen if the features from NGLOD are concatenated (instead of summed), and the features from iNGP are summed (instead of concatenated). In summary, the retrieval accuracy significantly decreases, dropping 14.8% for NGLOD and 53.8% for iNGP. Please see further details in Appendix B.7.

Reviewer comment (Theory):
The matching among SDF, UDF and Occ may lead to errors. Both SDF and Occ can only represent watertight shapes, but researchers leverage UDF to represent open-surfaces and multi-layer structures. What if I input an INR in terms of a multi-layer car or a shirt with open surfaces? The matching from the SDF and Occ of watertight mesh of the car or the shirt is wrong for the UDF.

Author response:
One of the main design goals of INRet is to support retrieval across INRs with different implicit functions, especially when the shape might be well represented with a specific kind of implicit function but not the others. While it might be difficult to find a “closed-surface” shirt counterpart to a open-surface shirt, we can easily demonstrate INRet’s robustness using multilayer car and single-layer watertight car.

We demonstrate the robustness of INRet’s cross implicit function retrieval with a multilayer car and watertight car. In Appendix C.2 (Figure 4), we show that using a multilayer car as query (represented by a UDF INR) can successfully retrieve watertight cars (represented by SDF INR).

Reviewer comment (Presentation):
There is no visualizations for comparison or illustrating, which makes the paper lack of qualitative analysis. I also find it hard to get the advantages of INRet compared to other baselines without visualizations.

Author response:
We have added more visualizations in Appendix C.2 to show qualitative comparison between our method and the baseline methods.

Reviewer comment (Experiments):
The authors mentioned that ``the prior work only supports MLP-based architecture, which is not commonly used today.'' I agree with the authors that grid-based approaches are more popular today, however, the MLP-based approaches are also commonly used today, and INRet do not support these methods. Furthermore, there are many other representations for INR today, can INRet also handle these types of INRs (e.g. triplane for TensoRF, point feature-based for Point-NeRF)?

Author response:
We note that INRet trivially supports encoding of MLP-only INRs by simply not encoding the feature grid component if it is lacking. We added more experiments in Appendix B.3 (Table 10) showcasing INRet’s performance when applied to MLP-only INRs. We have also included an evaluation for INRs with triplane feature grids which is supported by INRet (by simply encoding features from triplane grid with Conv3D encoder). For MLP-only INR, using INRet improves retrieval accuracy across different implicit functions drastically.  For triplane INR, using INRet leads to retrieval accuracy similar to NGLOD and iNGP INR. Please see Appendix B.3 (Table 10) for more details.

We would like to note that since Point-NeRF uses points to perform novel view synthesis. The points alone cannot encode a continuous 3D field of SDF/UDF/Occ values and are thus not suitable for implicit shape representation.

Reviewer comment (Novelty):
The paper does not propose novel architectures or representations or even new metrics for similarity between the implicit representations. Instead, it makes use of existing architectures and embeddings (separately for the MLP and the feature grid) as well as L2 losses for the pairwise dissimilarity.

Author response:
> __Novelty in proposing new architectures, representations, and metrics__

Since the goal of our work is to demonstrate the feasibility of creating a common latent space for retrieval of INRs, we selected popular existing INR architectures and implicit functions without further modifications. This obviates the need for the use of new architectures or representations. Our hope is to keep the framework general such that it can easily support additional existing INR representations (such as triplane pointed out by reviewer U5A9) and future INR representations. Our analysis showed that embeddings to encode INRs can be generated using simple architectures such as Conv3D and MLP.

In the updated version, we propose a new metric that addresses shortcomings of previously proposed metrics. We find that existing metrics evaluate the success of retrieval based on the category of the shape, while largely ignoring the differences between shapes within the category. Therefore, we propose a new metric: Category-Chamfer, where the retrieval is only successful if the retrieved shape is the shape that shares the same category while having the least Chamfer Distance with respect to the query shape. This was initially omitted from the paper since it not only applies to INR but to 3D shapes in general. We provide further details about this new metric as well as a way to accelerate retrieval of the shape required by this metric in Section 3.4 and Appendix E.

Reviewer comment (Theory):
The INR embedding space is essentially a product space, and thus a cost function that is used for matching two elements in that space needs to be appropriately weighted. While this type of embedding is interesting to explore (and is one of the strengths of the paper), it also requires proper weighting design.

Author response:
> __Weighting and conversion between UDF/SDF/Occ embedding__

In our original implementation, we used equal weighting for the explicit loss between UDF, SDF and Occ. We ran further experiments that used different weights between different explicit loss pairs. In summary, using different weights for the explicit loss has minimal impact on the final retrieval accuracy. Results and further explanations can be found in Appendix B.5. 

Regarding the potential conversion between UDF/SDF/Occ representations. We would like to note that the features INRet obtained from the feature grids are not actual UDF/SDF/Occ values. They are latent features used by the MLP of the INR for shape reconstruction, or used by the Conv3D encoder of INRet for INR embedding generation. Therefore, unlike the conversion between actual distance values (UDF/SDF/Occ) which is straightforward, there is no simple way to perform conversion between the latent vectors sampled from the feature grid. The difficulty of conversion in the latent space is one of the primary reasons we designed INRet to handle INRs with different implicit functions.

Reviewer comment (Writing):
In Equation (4), it is unclear whether the first loss term is an L1 norm or an L2 norm. This distinction is important because there will be different convergence properties of the encoder/decoder depending on whether it is an L1 norm.

Author response:
> __Q1. L1 vs L2 norm for UDF common decoder in Equation (4)__

The first loss is L1 norm. We chose L1 norm to be consistent with existing work that uses L1 norm for UDF loss (we used a UDF common decoder) [1]. We performed further experiments applying L2 norm to the UDF common decoder. In summary, the choice of L1 and L2 norm does not have a significant impact overall. Please refer to Appendix B.6 for further details.

Reviewer comment (Theory):
In equation (4), since the first term compares the decoder outputs to d_u (a UDF), the loss may be biased towards UDFs. The authors should comment on this potential bias. Furthermore, what would happen if, in the first term, the loss also incorporated pairwise differences between each INR?

Author response:
> __Q2. Potential bias towards the loss used by the common decoder (UDF); Pairwise differences__

We performed further experiments to study the bias with respect to the choice of the common decoder loss. The choice of loss indeed makes the embedding slightly more biased towards the specific type of loss. For example, using the SDF common decoder, the SDF-SDF retrieval accuracy improved from 84.6% to 85.8%, while using the UDF common decoder resulted in the highest UDF-UDF retrieval accuracy. Please refer to Appendix B.4 (Table 11) for more details.

We design a common decoder to output the same type of value regardless of the implicit function of the original INR. Therefore, the common decoder is not designed to take pairwise differences between the INRs. This is performed by the explicit L2 regularization.