Paper: SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation (ICLR-2024)

Reviewer comment (Reproducibility):
In the contrast learning process, a distance of 1cm is set as the threshold to distinguish between similar and dissimilar parts. Can the authors provide clarification on how they precisely define this distance?

Author response:
**Threshold distance choice for the contrastive loss.**

Following the PointContrast paper, let $X_1, X_2$ be two partial point clouds from two views. We sample point pairs from them $(x_{1i}, x_{2i})$. We say that it is a positive pair if $||x_{1i} - x_{2i}||$ < 1cm and a negative pair elsewise. The positive and negative pairs are then plugged into the PointNCE loss defined in Equation 2.

Here we note that, although this threshold is hard, the contrastive loss itself is a soft loss for two reasons: (a) the weighting in the soft PointNCE loss, and (b) the balanced sampling rate between positive and negative pairs.

The specific threshold distance is decided based on the object scale and the overall framework is not too sensitive to this hyperparameter within a reasonable range.  We will add more clarifications to the paper.

Reviewer comment (Theory):
Is distance truly an effective criterion for distinguishing between similarity and dissimilarity?

Author response:
**Is distance truly an effective criterion for distinguishing between similarity and dissimilarity?**

For the feature similarities, we majorly follow prior works on their choices of metrics. Specifically:
- We choose the cosine similarity for contrastive learning, as the InfoNCE loss with theoretical foundations (ref point contrast)
- We choose L1-distance for computing the energy function during optimization, the same as in prior works such as NDF. Here the dominant factor for us to make the choice is not the criteria for similarity, but the optimization curve/landscape. We tried L1-distance, L2-distance, and cosine similarity here and observed that optimizing the hand pose with L1-distance gives the best quality and stability.

Reviewer comment (Writing):
Is there a typographical error in Equation (3)? Why does it contain both an equation symbol and an inequality symbol?

Author response:
**Symbols in Equation (3).**

The # means the cardinality of the set defined in the brackets {...}, and the inequality $||f’_i - f’_j|| < \delta$ defines the region of the set. There are multiple notation systems for sets and we picked the one we’re used to. We apologize for the confusion and will add further clarifications to the paper.

Reviewer comment (Theory):
Additionally, is the pruning process defined in Equation (3) considered reasonable?

Author response:
**Is the pruning process defined in Equation (3) considered reasonable?**

The overall intuition of keeping 80% percent of points is that: (a) we want to discard the points with the worst features (here we recognize “bad” as lower feature consistency within local regions), but (b) we don’t want to discard way too many points to maintain the overall shape of the object/scene.

Reviewer comment (Novelty):
The authors are encouraged to establish connections regarding why distilling DINO features is advantageous and elucidate its applications in downstream tasks.

Author response:
**Elucidating the advantages of distilling DINO features in the downstream tasks.**

The features from large vision models such as DINO provide semantics-aware correspondences across different instances/categories, allowing one-shot/few-shot transfer of manipulation policies. However, the 2D image features are only sufficient for simple end-effectors such as parallel grippers operating on single points in the space. For dexterous hands with complex geometry, we need a 3D feature field to decide the parameters of all the finger joints, and that’s why we need such 2D-to-3D distillation. We will add more explanations and discussions to the paper.

Reviewer comment (Writing):
The definition of **one-shot** should be explained in the manuscript (abstract or introduction).

Author response:
**The definition of one-shot should be explained in the manuscript (abstract or introduction).**

Here the “one-shot” means that given one demonstration on the source object, we seamlessly transfer learned manipulations to novel scenes, accommodating variations in object poses, deformations, scene contexts, and even object categories. We abridge the wording a bit to avoid making the title too long. We will add more clarifications to the paper.

Reviewer comment (Writing):
The input should accurately be described as "multi-view RGB-D scans" rather than "Given a 3D point cloud X", in the method section. And the dimension of the variable should be added.

Author response:
**The input format and the dimension of the variables.**

The 3D point cloud is obtained by merging the depth scans from the 4 views. We will add clarifications to the paper.

Reviewer comment (Theory):
Regarding the motivation of using DINO, while the authors have highlighted, "This field offers semantic understandings for inter-scene correspondences that transcend geometric descriptors", how does it contribute specifically to image matching deep models like LOFTR?

Author response:
**Motivation for using DINO and other possible choices of image-matching deep models like LOFTR.**

Our framework is not specifically curated for DINO features and would be potentially compatible with any image feature extractors. We adopt DINOv2 here as they’ve shown strong cross-instance and cross-category correspondences. LOFTR seems to work mostly on matching two identical objects in different images. We will add the discussions of other image-matching frameworks like LOFTR to the paper.

Reviewer comment (Experiments):
Was depth information also utilized by DFF?

Author response:
**Was depth information also utilized by DFF?**

Yes, depth information is also used in DFF for a fair comparison. We apologize for not clarifying this in the paper.

What we want to address in the paper is that, even with the scanned 3D geometry guiding the feature aggregation, the feature incoherency between different views, if not properly tackled, will lead to a lower-quality 3D feature field, causing a great drop in the success rate.

Reviewer comment (Novelty):
[Novelty Issue] Lack of any exciting factor or in some sense novelty: The difference from prior work in DFF (distill feature field) is that the paper operates in sparse view setting, where simple fusion of features from multiple views doesn’t perform best. To overcome this loss of views from prior work, they refine the point features based on the insight that points close in 3D should have similar features. This seems a very natural and obvious technique of pruning or refining points which have incorrect feature consistency w.r.t 3D consistency.

Author response:
**Novelty. It is natural that feature refinement induces 3D consistency.**

Specifically, we want to address that, 3D consistency itself also exists in vanilla DFF/NeRF without any refinement that directly averages the features from different views (because only the mean is considered and the variance is omitted). However, such view averaging with inconsistent 2D features can result in low-quality 3D features – which is the message we want to convey in this paper.

We agree that feature refinement w.r.t. 3D consistency gives better 3D consistency is not surprising. But what is not obvious is that feature refinement w.r.t. 3D consistency gives better cross-instance correspondences.

Moreover, most contrastive feature learning (such as PointContrast) show their generalization after training on a large dataset of pairwise samples, but here we show that even only overfitted to the 4 views of the training scene with a few minutes of optimization, our refinement network naturally generalizes to novel scenes.

Application-wise, compared to parallel grippers which operate on one single 3D point and can apply infinitely large force, dexterous hands require more accuracy in the end-effector pose to, contact the object surface, wrap the object, and get the forced closure. This makes a lot of methods for parallel grippers not directly applicable to dexterous hands. In such a scenario, fine-grained properties of the feature field become more important.

Reviewer comment (Theory):
[Assumptions on GT depth and camera pose] Secondly, following up on the novelty part, the paper makes the assumption of having GT depth maps and also GT camera poses. Errors in either of them will make the above refinement step tricky.

Author response:
**Noise in depth and camera pose.**

All experiments for our method and the DFF baseline are on real robots, and the depth and camera poses are all from real data with scan errors and calibration errors. Only the UniDexGrasp++ baseline is run in the simulation environment with ground-truth 3D geometry (as we’re running their state-based model).

The sim2real gap is a long-standing problem in robotics, and it is particularly severe for dexterous hands compared to simpler end-effectors such as parallel grippers. In this work, we show our results on real robots to demonstrate our robustness towards this sim2real gap.

Reviewer comment (Novelty):
[Less Relevant] Newer papers like Lseg, Conceptfusion have projected LLM/ vision-LLM features on the 3D scenes, comments on using them as feature backbones will be appreciated.

Author response:
**Other LLM/VLM backbones for feature extractions.**

We choose the latest DINOv2 here as they demonstrated strong cross-instance/category correspondences on images, while some other VLMs are more focused on language queries (such as open-vocab query/segmentation). But we agree that exploring different VLM backbones, especially studying their view consistency and 3D awareness, would be an interesting future work. We will also add more discussions to the paper.

Reviewer comment (Novelty):
[Less Relevant] Papers learning joint hand object poses, leaning visual affordances from images, also seem to be relevant related works, comparison against them in related work would be appreciated (Affordance Diffusion: Synthesizing Hand-Object Interactions, papers on hand-object interaction: HOI etc).

Author response:
**Other related works on learning joint hand-object poses or learning visual affordances from images.**.

Thanks for providing these references. We will add the discussions to our paper.