Paper: CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objects (ICLR-2024)

Reviewer comment (Evaluation):
The paper measures the success of the tasks using a "success rate." However, necessary details, like the specific criteria used to determine successful manipulation, lacks clarity. Moreover, the goal state illustrated in the manipulation videos (or their screenshots) appears to be a snapshot of a future state, which does not accurately represent the actual desired outcome.

Author response:
We describe task success criteria in section 4.1 as the translational offset of the pose within 0.05m distance and angular 0.1 radians. We updated the draft to better elucidate this point in the experiment section.

The goal visualized in the video ***is*** the approximation of the object at the actual desired pose. Obtaining goal images of objects after positioning objects precisely at the goal pose is impractical; we believe this is a reasonable visualization. Below is the actual difference in average pose error between the final object pose and its desired goal location in the real-world experiments.

| Metric | Value |
| --- | --- |
| Translation Error (m) | 0.036 ± 0.010 |
| Orientation Error (rad) | 0.068 ± 0.019 |

Reviewer comment (Evaluation):
Could you elaborate on the "success rate" metric used in your experiments? What specific criteria are used to determine a successful manipulation?

Author response:
We describe task success criteria in section 4.1 as the translational offset of the pose within 0.05m distance and angular 0.1 radians. We updated the draft to better elucidate this point in the experiment section.

Reviewer comment (Novelty):
The novelty of this work is marginal. Basically, it just applies point-cloud-based reinforcement learning to nonprehensile manipulation tasks.

Author response:
> (W1) The novelty of this work is marginal. basically, it just applies point-cloud-based reinforcement learning to non-prehensile manipulation tasks.
- While there have been approaches that consider point-cloud-based RL in robot manipulation, no prior work has solved closed-loop non-prehensile manipulation of general objects. The inherent complexity in contact dynamics between the object, robot and the environment in non-prehensile manipulation renders naive RL application to be prohibitively expensive in our domain without nontrivial design choices, as seen in our results (Figure 7). As such, no prior work has demonstrated the generalization over diverse object geometries with closed-loop feedback control in this domain.

Reviewer comment (Experiments):
The generation of the collision label is questionable with the coverage of the collision states, i.e., if this work truly aims for generalization on unseen objects, I wonder how the collision prediction network can be generalized to the unseen geometry.

Author response:
> (W2) The generation of the collision label is questionable with the coverage of the collision states, i.e., if this work truly aims for generalization on unseen objects, I wonder how the collision prediction network be generalized to the unseen geometry. Besides, if the contact network is trained as a guide for the encoder, how does it guarantee to generate a collision-free policy? The collision decoder itself is not perfect (due to the coverage issue and neural network prediction), and the influence on the encoder is indirect (leads to less perfect), not to mention the policy is distilled from the teacher network to the student network (even less perfect).

> Can collision network generalize to unseen object geometry? How can we ensure that the policy can precisely avoid collision despite errors coming from factors like (indirect guidance, limited coverage and approximation error from distillation)?
- Yes. Our network generalizes to unseen geometry, as evident in our experimental results. All except one of real-world objects are unseen, and the simulation test setting involves novel objects as well. We kindly encourage the reviewer to give a read of our experiment section.

Reviewer comment (Theory):
If the contact network is trained as a guide for the encoder, how does it guarantee to generate a collision-free policy? The collision decoder itself is not perfect (due to the coverage issue and neural network prediction), and the influence on the encoder is indirect (leads to less perfect), not to mention the policy is distilled from the teacher network to the student network (even less perfect).

Author response:
> (W3) Then, though the authors have conducted extensive baseline methods on the encoder side, in my opinion, they should also discuss different decoder schemes to justify the "contact-based" name. For example:
> 
> (a). the encoder is directly trained by the policy network? That is, no contact decoder to pretrain the encoder.
> 
> (b). the decoder is a geometry reconstruction network. In this sense, the encoder can capture the geometry information of the object point cloud and hand. I see no apparent reason why the "action and rewards" cannot be done with such kinds of decoders.

- The baselines you mentioned have already been implemented in the experiment section.
    - In Section 4.2, the End-to-end (E2E) baseline is the one that the encoder is directly trained by the policy network i.e. RL objective. This corresponds to option (a) from your recommendation.
    - In Section 4.2, the Point2Vec (P2V) baseline represents the pretraining scheme where the encoder is trained on shape-reconstruction objectives. Point2Vec learns shape representation by reconstructing the latent shape embeddings (similar to data2vec[1]). This corresponds to option (b) from your recommendation.
    - Ours outperform these both in terms of data and computational efficiency. (Figure 7)
    
[1] Baevski, Alexei, et al. "Data2vec: A general framework for self-supervised learning in speech, vision and language." *International Conference on Machine Learning*. PMLR, 2022

Reviewer comment (Experiments):
Since this method is "contact-based"? In a broad sense, all the manipulation is contact-based, but I assume the word choice here is for the embedding induced by the contact network. Then, though the authors have conducted extensive baseline methods on the encoder side, in my opinion, they should also discuss different decoder schemes to justify the "contact-based" name. For example: (a) the encoder is directly trained by the policy network, i.e., no contact decoder to pretrain the encoder; (b) the decoder is a geometry reconstruction network—in this sense, the encoder can capture the geometry information of the object point cloud and hand. I see no apparent reason why the "action and rewards" cannot be done with such kinds of decoders.

Author response:
First, our goal is not to learn a “collision-free policy” that computes a collision-free motion. As the title of our paper and Figure 1 on page 2 suggest, our problem is that of non-prehensile manipulation, and object manipulation cannot be done without making contact with an object (We suggest the reviewer to checkout the video of our robot performing the task of interest: [https://sites.google.com/view/contact-non-prehensile](https://sites.google.com/view/contact-non-prehensile)). In particular, our problem involves learning where to contact the object and how to apply forces to an object, which involves deliberately colliding with the object. The key aspect of this problem, therefore, is knowing where the robot has contacted the object because that tells us what forces the robot can apply to the object. Our contribution is the design of pretext task and neural network architecture tailored to detect such contacts.

The reviewer raised two concerns based on this misunderstanding. The first was the naming of our method (it has “contact-based” in it). We hope the reasoning above tells you why contact-based representation is important.

Reviewer comment (Writing):
Another possibility of the naming is due to the r_contact in the reward, but the contact potential is not given exactly, since the essential d_h,o is built upon the distance from the CoM of the object to the tip, which does not prevent collision/intersection between the gripper and hand. Therefore, if the contact is not necessarily determinant or explicitly modeled for policy training in both network design and reward shaping, the only sane way to interpret "contact-based" would be in the broadest sense, i.e., all the manipulation is contact-based. In this sense, I would suggest removing the "contact-based" in the title, which would give wrong expectations to the readers, or the paper can present stronger relevance of "contact" to the object representation.

Author response:
> As for the collision-free policy. First, obviously, there will never be an intersection between a gripper and an object in the real world. Thus, the discussion on in-collision/collision-free is based on the training, which takes place in simulation. My question is, how is the collision in simulation avoided when the contact reward is defined on the distance between the object's CoM to the gripper tip, not the object's surface? Will the possible collision make the object move unexpectedly? Could the bad performance on concave objects (irregular place of CoM) and unstable objects (changing CoM) be because of such contact reward design? I assume the interpenetration is handled by the collision detection methods implemented natively in the simulator. If so, the contact reward is just a usable reward, not a meticulous design.

- We want to note that the purpose of the contact-inducing reward $r_{contact}$ is not to prevent collision, but to guide initial exploration. This strategy is commonly adopted in contact-rich manipulation tasks [3,4] to facilitate policy training: during initial exploration, the robot is prone to make meaningless actions (i.e. move in thin air) without making significant progress. To prevent this, the contact-inducing reward encourages the policy to reduce the distance between the gripper and the object, hence increasing the probability of the robot interacting with the object during initial exploration.
- How does the policy learn to avoid spurious collision that might result in undesired motion of the object? Since the policy is primarily driven to maximize task success (which is the dominant reward), and the contact-inducing reward just serves as guidance, the policy is incentivized to avoid collision if it reduces the likelihood of success.
- As for the specific implementation of the contact-inducing reward, we’d like to note that during our early investigation, we also tried the surface-to-surface distance (rather than CoM distance) based contact reward but this did not make a significant difference in policy training. This is only natural, as approximately guiding the robot towards the general region of the object is sufficient to encourage interaction during initial exploration, which is the purpose of the contact-inducing reward.

Reviewer comment (Novelty):
Lack of Unique Design for Complex Operations: The paper emphasizes in the introduction that its approach can execute more complex robotic arm operations than prior grasping work. However, in the method description, there appears to be no clear unique design to directly support this motivation, raising doubts about the novelty and effectiveness of the method.

Author response:
- We’d like to highlight two unique design choices made to address the complexity of the non-prehensile manipulation task: the model architecture and object representation learning.
- The complexity of contact dynamics of non-prehensile manipulation necessitates closed-loop feedback control. However, training such a dense, closed-loop policy is expensive in terms of both time and compute cost. To address this, we first adopt a patch-based transformer architecture, for the first time among works in point-cloud-based robot manipulation. This accelerates the training process and reduces the training time remarkably, as shown in Figure 7.
- Secondly, we observe that non-prehensile manipulation is highly sensitive to the knowledge of contact between the object and the robot. To reflect this, we adopt a novel contact-based pretraining task. Together with our model architecture, we demonstrate significant efficiency gains as a result of adopting our pretraining scheme.

Reviewer comment (Experiments):
Overemphasis on "Contact": While the authors place particular emphasis on the importance of "contact" in nonprehensile operations, it seems that the equally vital role of "contact" in grasping tasks has not been adequately considered. This imbalanced emphasis may impact the comprehensiveness of the method and its practical applicability.

Author response:
- Non-prehensile manipulation requires careful choice of contacts, i.e. sequence of contact mode switches which defines what forces the robot can apply to the object. We strongly encourage the reviewer to check out the video of our method operating in the real world[[https://sites.google.com/view/contact-non-prehensile](https://sites.google.com/view/contact-non-prehensile/)]: this is in stark contrast with grasping with a parallel jaw gripper, which typically only involves approaching an object and closing the gripper.

Reviewer comment (Novelty):
Questioning the Novelty of Point Cloud Processing: The paper utilizes a Transformer-based architecture to process point cloud data, but this approach does not appear groundbreaking, emphasizing the need for a more detailed description of the policy network design.

Author response:
- Most prior works in robot manipulation actually only use PointNet-based architectures. We explore the choice of point cloud processing architecture, and show that patch-based transformer greatly improves computational efficiency in our domain, by about 8x in terms of model inference time and more than 20x in terms of policy training. To our knowledge, we are the first ones to demonstrate the efficacy of patch-based transformer in point-cloud based robot manipulation.

Reviewer comment (Writing):
Insufficient Description of the Policy Network: The paper provides a relatively concise description of the policy network, lacking in-depth details, which might hinder readers from fully understanding the workings of the method and its potential advantages.

Author response:
- We apologize for missing the details, and updated the paper to include additional details. The policy starts with an initial cross-attention layer against the tokens of the current task context(joint state, previous action, goal, and physics parameters) to extract task-relevant information from the patch-level embeddings. The resulting features are concatenated again with the task context, which are then processed by an MLP(multi-layer perception) before being fed to actor and critic networks. Given that our focus is on the point-cloud processing representation model, we placed less emphasis on the policy network architecture when describing our model. We have now added extra details about the policy network to facilitate the reader’s understanding of our model. (Figure 5. & Section 3.1)

Reviewer comment (Experiments):
Inadequate Experimental Setup: The paper does not offer detailed information related to actions and reward settings in the experiments, which is crucial for evaluating the effectiveness of the reinforcement learning portion.

Author response:
- We appreciate the reviewer’s feedback regarding the necessity of additional details on the action-space. We have added a paragraph in the appendix that describes the structure of the action-space for the robot (Appendix A.2). However, we already provide a detailed breakdown of the reward function in section 4.1, and include relevant hyperparameters in Table A.1 (Appendix A.7), and we’d appreciate it if the reviewer could clarify which part of the reward function requires further clarification.

Reviewer comment (Experiments):
The model takes the difference between current object pose and target object pose as input. This can bring significant engineering challenge since object segmentation and pose tracking in the real world can be difficult. However, the authors documented their approach very well in Sec. A.3.

Author response:
We agree that segmentation and pose tracking are significant engineering challenges, and we appreciate the reviewer for recognizing the documentation of our approach. We are also hopeful that recent advances in foundational vision models, such as SAM[1] or POPE[2] can facilitate object pose tracking and segmentation to be more generally accessible for robot manipulation in the future, particularly for the challenges such as segmenting non-singulated objects.

Reviewer comment (Experiments):
The method only works on object well separated from clutter on a tabletop. This is related to the above assumption for the object pose, since having more than one object in close contact will make object segmentation and pose tracking even more challenging.

Author response:
We agree that segmentation and pose tracking are significant engineering challenges, and we appreciate the reviewer for recognizing the documentation of our approach. We are also hopeful that recent advances in foundational vision models, such as SAM[1] or POPE[2] can facilitate object pose tracking and segmentation to be more generally accessible for robot manipulation in the future, particularly for the challenges such as segmenting non-singulated objects.

Reviewer comment (Experiments):
What is the coordinate frame of the hand pose input? Since the hand pose is sampled near the object to ensure good data balance for contact/no contact during pre-training, will the hand pose go out of distribution during policy learning?

Author response:
The hand pose is represented in the world frame. Since the poses are sampled from the workspace during pre-training, it does not actually go out of distribution during policy learning.

Reviewer comment (Presentation):
In Figure 7, is the scale of the success rate the same for the two plots? It seems that the final success rate doesn’t match up for the green, red, purple and brown curves.

Author response:
There was a small bug in the plotting code, which has been now addressed. We updated Figure 7 and made minor adjustments on reported metrics in Section 4.2. Thank you for catching that!

Reviewer comment (Reproducibility):
Which simulator is used? I didn’t find it in the paper.

Author response:
We used Isaac Gym. We added the reference to the simulation in Section 3.1.