Paper: TUVF: Learning Generalizable Texture UV Radiance Fields (ICLR-2024)

Reviewer comment (Experiments):
For rigid shapes like cars and airplanes, one could project the shape onto an enclosing sphere, e.g., via raycasting from the sphere to the shape, to obtain similar mappings—“similar” as in all cars’ front bumpers mostly map to the same location on the canonical sphere. If this simple approach produces similar results, the whole canonical surface autoencoding part becomes invalid.

Author response:
As per the reviewer's suggestion, we develope a new baseline method to establish a correspondence mapping by casting rays from a sphere to shapes. We test the baseline approach on the KeypointNet dataset and evaluate its effectiveness by conducting experiments on the semantic keypoint transfer task. The table below showcases the qualitative results we obtained on Cars. Our findings suggest that this method can generate reasonable results for some samples, but it is prone to errors when it comes to local areas due to sensitivity to scale and offset. Moreover, we have observed noticeable seams while performing texture transfer with synthetic textures, which we show an example [here](https://drive.google.com/file/d/10PhutLoShKmxcoO-Ck-RI0-YyLHdWroq/view?usp=drive_link).

| ShapeNet Car (Image) | ShapeNet Car (Video) |
| -------- | -------- |
|[results](https://drive.google.com/file/d/1xyVqqlm2xI_FSOd1xhwlp1LfSx-M_uUW/view?usp=drive_link)|[results](https://tuvf4iclr.github.io/carkpt)|


To further quantitatively evaluate the method, we measure the correspondence accuracy at different error thresholds, and the results are presented below. Additionally, we experimented with various radius parameters to improve the results. The findings indicate that the radius length significantly influences the correspondence accuracy. This is because only sparse points are cast to the surface when the sphere is large. And if the sphere is small, seams and occlusion will occur more frequently. Despite the variations in radius length, the Canonical Point Auto-encoder outperformed all other variants.

| ShapeNet Airplane | ShapeNet Car | ShapeNet Chair |
| -------- | -------- | -------- |
| [results](https://drive.google.com/file/d/1BMD5FMIgmX6CQy0oAJt_6AalQDrzprpQ/view?usp=drive_link)     | [results](https://drive.google.com/file/d/1rc6oBhffNd-K_Nk7P7W6-p5xke8rouiX/view?usp=drive_link)     | [results](https://drive.google.com/file/d/1RBL5jQ64FJJdEHWekfG7ePWU-JuSfMhu/view?usp=drive_link)     |

Reviewer comment (Novelty):
For non-rigid objects like humans and animals, the authors show mappings learned from parametric models (SMPL or SMAL or similar), for which the canonical space is by definition well established and does not benefit from this work.

Author response:
As mentioned in our main paper, we have also evaluated generic man-made objects, including cars and chairs. Since these are topology-varying objects, it is not easy to fit them with parameterized primitives. As a result, these objects lack a clear, well-defined canonical space. Therefore, our work is particularly useful for these objects.

Reviewer comment (Experiments):
A concrete, simpler alternative to the volumetric approach is to retain the face information for the points on the canonical sphere and use the same faces in the decoded shape, thereby avoiding differentiable surface reconstruction and volume rendering in favor of computationally efficient direct mesh rendering.

Author response:
We follow the reviewer's suggestion and develop a new mesh-based baseline that uses the face information from the canonical sphere to form the reconstructed shape. However, we find that such a design is not feasible. The Canonical Point Auto-encoder allows complex transformations like folding or cutting. Therefore, using the same faces as the points on the canonical sphere would lead to incorrect meshes. Please refer to [General Response (A)](https://openreview.net/forum?id=dN4vpVTvWX&noteId=HLOybPoVgu) for details and results.

Reviewer comment (Experiments):
There is confusion over the adoption of volume rendering when a straightforward mesh rendering approach could achieve the same purpose with much greater efficiency.

Author response:
We chose to use NeRF not because it's trendy but because it's the most intuitive option available. In [General Response (A)](https://openreview.net/forum?id=dN4vpVTvWX&noteId=HLOybPoVgu), we conducted a series of studies exploring alternative approaches with mesh representation. We found that without NeRF, it's not straightforward to maintain correspondence on mesh faces while ensuring good geometry. Therefore, using a continuous representation is a logical and intuitive choice and the best option available. We want to re-emphasize that no steps in our method are unnecessary. We use a point cloud regime because it can better represent the topology-varying geometry and correspondence compared to a mesh. Additionally, we make the points volumetric to maintain surface correspondence for rendering.

Reviewer comment (Novelty):
The rationale for going through the complex pipeline to make the shape volumetric is unclear, and this may be a more severe issue than the mapping question. It would be concerning if the motivation is simply that “volumetric == cool,” without a stronger justification.

Author response:
We chose to use NeRF not because it's trendy but because it's the most intuitive option available. In [General Response (A)](https://openreview.net/forum?id=dN4vpVTvWX&noteId=HLOybPoVgu), we conducted a series of studies exploring alternative approaches with mesh representation. We found that without NeRF, it's not straightforward to maintain correspondence on mesh faces while ensuring good geometry. Therefore, using a continuous representation is a logical and intuitive choice and the best option available. We want to re-emphasize that no steps in our method are unnecessary. We use a point cloud regime because it can better represent the topology-varying geometry and correspondence compared to a mesh. Additionally, we make the points volumetric to maintain surface correspondence for rendering.