Paper: Human-oriented Representation Learning for Robotic Manipulation (ICLR-2024)

Reviewer comment (Experiments):
Recent progress in VLM has showcased vast capabilities in reasoning about state-changes, object detection, and identifying irreversible states. Could these representations or features from a pre-trained VLM be harnessed to refine R3M, MVP, or EgoVLP, rather than the method proposed?

Author response:
We agree that this is a very interesting suggestion! Indeed, the recent advances in VLM have demonstrated a strong representation of learning ability. One potential approach to harnessing the representation from VLM is via distillation where we distill the foundational VLM’s representation extraction ability into smaller models such as R3M. We are very excited about investigating this open problem in our future work!

Reviewer comment (Evaluation):
An observation suggests that tasks across various environments and models tend to underperform in toggle on/off tasks. Is there an underlying reason for this trend?

Author response:
We follow DETR for the finetuning of our network. In the object detection area, the DETR is more fit for large object detection. We hypothesize the toggle is too small, so our representation is not sensitive to that.

Reviewer comment (Evaluation):
The ablation studies indicate that both spatial and temporal tasks, when treated independently, don't yield impressive results. However, their combination surpasses the baseline. Can the authors provide an intuition behind this phenomenon?

Author response:
We hypothesize this is attributed to our task fusion decoder, which explicitly builds feedback paths between tasks and helps each task to learn to query useful information from other tasks, thus making tasks interact with and complement each other, ultimately helping the encoder learn more useful representations compared to learning individual task.

Reviewer comment (Experiments):
The authors might consider leveraging other task or robot demonstration datasets featuring a third-person perspective with actual robot embodiment beyond Ego4D. A few notable examples include:  
- Padalkar et al. "Open X-Embodiment: Robotic learning datasets and RT-X models." arXiv:2310.08864 (2023).  
- Kumar et al. "RoboHive: A Unified Framework for Robot Learning." arXiv:2310.06828 (2023).  
- Duan et al. "AR2-D2: Training a Robot Without a Robot." arXiv:2306.13818 (2023).  
- Chen et al. "Learning generalizable robotic reward functions from 'in-the-wild' human videos." arXiv:2103.16817 (2021).

Author response:
We would like to thank the reviewer for providing these related papers and datasets! We are excited about leveraging these new datasets to test our representation learning approach in our future work!

Reviewer comment (Writing):
The paper does not have a clear definition of the different objectives (object state change classification, point of no return temporal localisation, and state change object detection), and this lack of clarity persists throughout the text. The work would be much clearer if these were defined much earlier (e.g., pg 4). Acronyms or descriptions such as OSCC, PNR, and SCOD are not particularly explanatory and raise more questions for the reader. For example, it is not apparent what the difference is between state change object recognition and object state change classification, or what the point of no return means. These need more crisp definitions and explanations earlier in the paper. Figure 3 is referenced as providing definitions, but it only shows loss functions and model visualisations, without definitions, and the associated descriptions on page 5 are also confusing and raise more questions.

Author response:
**W1. [The unclear for expressions and figures]** \
We appreciate your valuable suggestion for our expressions and figures. We have followed your advice and redrawn the Fig. 2 with a clearer label. We removed the “+” in the new version of Fig. 2, the spatial feature and temporal feature are two separate features fed into the task fusion decoder. Also, we reorganized the part about the three ego4d tasks and moved it to the front paragraph following your advice for a clearer understanding. We have added more explanations for the three tasks in Section 3.2.

Reviewer comment (Presentation):
In Figure 2, while the depiction of the decoder is helpful and linked to equations, it would be more informative if the global and temporal features $(h_*)$ were represented and a functional description of these operations was provided. It is unclear whether the '+' symbol in the figure denotes concatenation, attention, or something else. The appendix suggests a transformer look-up, but the figures should be explicitly mapped to the nomenclature in the text to improve clarity.

Author response:
**W1. [The unclear for expressions and figures]** \
We appreciate your valuable suggestion for our expressions and figures. We have followed your advice and redrawn the Fig. 2 with a clearer label. We removed the “+” in the new version of Fig. 2, the spatial feature and temporal feature are two separate features fed into the task fusion decoder. Also, we reorganized the part about the three ego4d tasks and moved it to the front paragraph following your advice for a clearer understanding. We have added more explanations for the three tasks in Section 3.2.

Reviewer comment (Theory):
The rationale for selecting the particular tasks to be decoded is unclear. It is not explained why these tasks were chosen over other possible options or formulations that could exploit the human data. The task losses appear somewhat arbitrary. Although there is a motivation from a cognitive science perspective (Kirkham et al., 2002), this aspect could be strengthened to make the choice seem less heuristic.

Author response:
**W2. [It is unclear to me why the particular tasks to be decoded were selected, and why this one should be used over various other options or formulations exploiting the human data.]** \
In this paper, we selected three tasks from the ego4d dataset that are most related to human and object interaction and are critical for everyday scenarios as we expect that representations learned from these tasks could benefit the manipulation tasks. We would like to note that our multi-task framework is flexible and can be adapted to other tasks straightforwardly. We are excited about exploring other tasks in our future work.

Reviewer comment (Evaluation):
Training takes three days on 5 A6000 GPUs, which seems to be a very expensive fine-tuning operation for the performance gains observed. While success rate is the conventional metric, it may be worthwhile to evaluate performance as a function of compute resources.

Author response:
**W4. [Training takes three days on 5 A6000 GPUs ]** \
The training time is constrained by the backbone and the dataset. Our method is also compatible with lightweight models that require less data for training. Improving training efficiency would be an exciting future work.

Reviewer comment (Theory):
The decoder outputs information about state changes, but the definition of what constitutes a state change, or what the state is in these tasks, is not provided. This choice seems to explicitly impose structure on the control task being considered.

Author response:
**Q1. [The decoder outputs information about state changes. Could you define what constitutes a state change, or what the state is in these tasks?]** \
The state change label comes from the Ego4D hand-object interaction benchmark label, in the dataset, the state change moment is mostly defined by a contact between hand and object.

Reviewer comment (Writing):
The point of no return is defined as the task to localise the keyframe with the state change in the video clip, but the origin and meaning of the "no-return" terminology are not explained.

Author response:
**Q3. [The point of no return is defined as the task to localize the keyframe with state change in the video clip. Where does the "no-return" terminology come from and what does this mean?]** \
We apologize for the confusion. The no-return concept is also from the Ego4D dataset [A], it means that you now have to continue with what you are doing and it is too late to stop.

[A] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995–19012, 2022.

Reviewer comment (Reproducibility):
In the KL loss function in equation (4), it is unclear what the distributions are. Since there can be multiple state changes in a sequence, these are not categorical. The paper does not clarify how this loss is actually computed.

Author response:
**Q4. [Can you clarify what the distributions are in the KL loss function in (4)? These aren't categorical, since I assume there can be multiple state changes in a sequence... How is this loss actually computed?]** \
Sorry for the confusion. The distribution is state change probability distribution, the decoder outputs a tensor with state change probability in frame dimension. The label is a tensor consisting of 0 and 1, indicating whether the state changes or not in the frame dimension.
We treat the two tensors as distributions and measure the KL distance between them.

Reviewer comment (Writing):
The SCOD task is defined as localising hand-object positions during the interaction process, which applies continuously, but the term "state change object detection" implies it only applies when a state change occurs. This contradiction requires clarification.

Author response:
**Q6. [The SCOD task is defined as the localization of the hand object positions during the interaction process, which seems to apply continuously, but the terminology state change object detection implies this is only when a state change occurs. Which is it?]** \
We apologize for the confusion. The SCOD is the task of detecting the object undergoing a state change from the given video clip.

Reviewer comment (Novelty):
While the task decoding approach is novel, claiming that the paper represents a *paradigm shift* in representation learning may be an overreach, as learning representations by exploiting human data is relatively well established (e.g., Sermanet et al., Time-Contrastive Networks: Self-Supervised Learning from Video, 2018).

Author response:
**Q7. [I appreciate the novelty in the proposed task decoding approach, but is it an overreach to claim that this paper represents a paradigm shift in representation learning? The idea of learning representations by exploiting human data is relatively well established eg. Sermanet et al. Time-Contrastive Networks: Self-Supervised Learning from Video 2018.]** \
Thank you for the suggestion. We have removed the overclaimed expression.

Reviewer comment (Evaluation):
Table 1 lacks detail on how success rates are calculated. It is unclear whether they are averaged over multiple repetitions (and, if so, how many), whether error bars are present, and whether the results are statistically significant.

Author response:
**Q8. [Table 1. Please provide more detail on this - are success rates averaged over multiple repetitions (if so, how many?) Are there error bars, are these results significant?]** \
Currently, our experiment in Table 1, is evaluated 20 times and averages the three best successful rate results for every single environment. Also, for the kitchen and metaworld environments, the result is the average from two camera views, for the adroit environment, the result is the average from three camera views.  Thus, in a total of 12 environments, there are 460 evaluations.

We do have error bars based on different views as the Table below:
| env              |     R3M          | R3M+ours| 
| :--------------------------------------------- | :-----: | :--------: | 
| assembly | 93.67(3.77)|  **98.67**(1.89)  | 
| bin | 44.67(8.01)|  **56.33**(1.89)  | 
| hammer | **92.67**(5.66)|  86.34(3.30)  | 
| pen | 67.73(3.52)|  **70.00**(2.69)  | 
| relocate | 63.33(5.45)|  **66.22**(5.09)  |

Reviewer comment (Reproducibility):
It is unclear whether the dataset shown in Figure 5 will be released for reproducibility.

Author response:
**Q9. [Will the dataset in Fig 5. be released for reproducibility?]** \
Of course. We open-source the dataset at [data_link](https://www.dropbox.com/scl/fi/tw7m6i7cgapefap7eozb3/fanuc_manipulation.zip?rlkey=53lz46vopb5r8r9l2d6d3jegt&dl=0), code at [code_link](https://www.dropbox.com/scl/fi/bdny032hk9q5jmed4doln/human_oriented_code.zip?rlkey=d8g3uvqr8t3s1kyz8edn7s8t1&dl=0).

Reviewer comment (Experiments):
The introduction is inappropriate given the results shown. The claimed message, that the presented fine-tuning represents a method for human alignment without requiring human labels, is not supported by the paper’s results and experiments. What the paper shows is that fine-tuning video models on hand-object interaction tasks improves the representation for robotics, which is not the same as having a scalable proxy for human evaluation. To make that claim, a study comparing representations fine-tuned with human feedback to those fine-tuned with these hand-object interaction tasks would be needed.

Author response:
**W1.[This is absolutely not the same as having a scalable proxy for human evaluation. To make that claim, one would need to run a study comparing representations fine-tuned with human feedback to those fine-tuned with these hand-object interaction tasks.]**\
We apologize for the confusion. We did not intend to make readers interpret the proposed method as a way of achieving alignment without human labels. The key idea we aim to convey is that the proposed approach takes inspiration from cognitive science, which posits that humans learn to extract a generalizable behavior representation from perceptual input by learning about a multitude of simpler perceptual skills that are critical for everyday scenarios and formalize this idea through the lens of multi-task learning where each “task” is a perceptual skill. Not only do each of these perceptual skills define a human prior about what is helpful for manipulation tasks, but they are each associated with large and well-labeled video datasets pioneered by the computer vision community.

Reviewer comment (Experiments):
An additional ablation that separately removes each of the different task heads would be valuable. While there are space vs. time ablations in the current paper, there are three different task heads, and it would be important to see the effect of each one on the downstream performance. Similarly, since the loss uses values of “sigma” for each task contribution, it would be valuable to see what the final learned values of these sigmas are, as they serve as a proxy for how much the model is using the different task outputs for fine-tuning.

Author response:
**Q2. [What about ablating each loss type separately? Please provide ablations for each individual task head]** \
Good question. We have added experiments by each task head, by retraining the model with three single tasks and retraining the policy for different models. We observed that in most situations, adding all three tasks gets the best result.

|env                     | R3M+OSCC | R3M+PNR | R3M+SCOD | R3M+OSCC+PNR+SCOD| 
| :--------------------------------------------- | :-----: | :------------: | :---------: | :---------: | 
|micro | 21.00 |**28.00**|26.00|**28.00**| 
| light  | 82.00 | 69.00 | 70.00 | **83.00** | 
| ldoor| **39.00** |34.00 | 23.00 | 32.00 |
| assembly|84.67|81.33| 83.33| **92.67** |
| relocate| 39.33| **40.00**| **40.00**| 36.67 |

Reviewer comment (Novelty):
The paper is missing a key piece of related work for the main claim that hand-object interactions are an excellent task for fine-tuning models—Bahl et al, CVPR 2023, "Affordances from Human Videos as a Versatile Representation for Robotics." Rather than fine-tuning a video model, Bahl et al. train a model to predict hand-object interaction locations directly and find that this supports many downstream robotics tasks. It would be valuable to explain why one would prefer a fine-tuned video model on these interaction tasks as opposed to directly predicting them from the outset.

Author response:
**W2. [It would be valuable to know why one would prefer a fine-tuned video model on these interaction tasks, as opposed to directly predicting them from the outset.]** \
Good question, directly predicting affordance and visual representation are both major topics in robot learning. Predicting affordance can be more direct and intuitive for policy learning, which is highly correlated with action space. Visual representation learning like R3M, also has the advantage of flexibility, these representations can easily be used in any robot learning tasks with observation space, across the vision-language model, diffusion policy, and so on.
We believe predicting from the outset is a good way for many scenes, but visual representation learning is also crucial, even some scenes like adroit pen task without accurate affordance, or other scenes with too many affordances in multi-task, direct predicting the outset may not cover all the affordance accurately, the hidden visual representation can also benefit.

Reviewer comment (Evaluation):
Why would we expect fine-tuning on the task not to improve the performance on the task? (In relation to section 4.5)

Author response:
**Q1. [Why would we expect fine-tuning on the task not to improve the performance on the task? (In relation to section 4.5)]** \
Sorry for the confusion. For Section 4.5 and Table 4, the first two rows' results are from ego4d pretraining and finetuning on specific results, and our results also pretraining from ego4d, but finetuning on two tasks at the same time with task fusion decoder. The results can, in some sense, show our tasks in the task fusion decoder have information exchange, which can benefit each one.

Reviewer comment (Writing):
The statement “Also, different vision tasks should have information interaction, for the human-like synesthesia” does not add anything. What is human-like synesthesia?

Author response:
**Q3.[“Also, different vision tasks should have information interaction, for the human-like synesthesia.” This does not really add anything. What is human-like synesthesia?]**\
Human-like synesthesia is the way humans perceive spatial, temporal, and other information about the world simultaneously. In a task trajectory, humans gather information at the same time, like object localization, time to happen contact.

Reviewer comment (Experiments):
For the real world experiments: how many test interactions were performed?

Author response:
**Q4. [For the real world experiments: how many test interactions were performed?]** \
We perform 10 interactions for every single model in every single task following r3m evaluation, so in every task, there are 3 baseline models and 3 our models, a total of 240 interactions for Fig 5.

Reviewer comment (Reproducibility):
Can the authors commit to open-sourcing the real-world dataset?

Author response:
**Q5. [Can the authors commit to open-sourcing the real-world dataset?]** \
Sure. We open-source the dataset at [data_link](https://www.dropbox.com/scl/fi/tw7m6i7cgapefap7eozb3/fanuc_manipulation.zip?rlkey=53lz46vopb5r8r9l2d6d3jegt&dl=0), code at [code_link](https://www.dropbox.com/scl/fi/bdny032hk9q5jmed4doln/human_oriented_code.zip?rlkey=d8g3uvqr8t3s1kyz8edn7s8t1&dl=0).

Reviewer comment (Reproducibility):
For the learned sigmas: what are they after training? When you say they are learned, is that per scene, or just differentiable but global?

Author response:
**Q7. [Learned sigmas: what are they after training? When you say they are learned, is that per scene, or just differentiable but global?]** \
We retrained a model to see the change of the sigma, and added one curve figure to Appendix A.5. The sigma for OSCC is about -1.3e-5, for PNR is about -3e-5, for SCOD is about 0.01, while the loss value of SCOD is about 1e3 times over PNR and OSCC. Sigma is just differentiable but global.

Reviewer comment (Experiments):
Please provide ablations for each individual task head.

Author response:
**Q2. [What about ablating each loss type separately? Please provide ablations for each individual task head]** \
Good question. We have added experiments by each task head, by retraining the model with three single tasks and retraining the policy for different models. We observed that in most situations, adding all three tasks gets the best result.

|env                     | R3M+OSCC | R3M+PNR | R3M+SCOD | R3M+OSCC+PNR+SCOD| 
| :--------------------------------------------- | :-----: | :------------: | :---------: | :---------: | 
|micro | 21.00 |**28.00**|26.00|**28.00**| 
| light  | 82.00 | 69.00 | 70.00 | **83.00** | 
| ldoor| **39.00** |34.00 | 23.00 | 32.00 |
| assembly|84.67|81.33| 83.33| **92.67** |
| relocate| 39.33| **40.00**| **40.00**| 36.67 |

Reviewer comment (Experiments):
The evaluation is weak. The TFD has significant computational machinery with $K$ stacked blocks (where $K$ is undocumented) and many design choices, but the ablation study only examines the importance of the "time-related" and "spatial-related" tasks without clearly defining these tasks or explaining Table 3’s numbers. Results are not very convincing: one of the five environments performs worse under the TFD, and another shows no benefit. There is no exploration of the impact of the two attention mechanisms (which are core to this work) or of how many TFD blocks are needed.

Author response:
As shown in the caption of Figure 2, the design for self-attention in each layer is for the interaction of each task, we hypothesize each task should have information change, for cross-attention, it is a connection between vision encoder backbone and TFD, which can decode vision representation from encoder into multi-task with human labels. In our original paper, the choice for the TFD is 2 layers, each layer has one self-attention and one cross-attention. And also, we add more experiments about different choices of K, as shown in the table below.

| env              |      K=2           | K=3 | K=4 | 
| :--------------------------------------------- | :-----: | :--------: | :------------: |
| micro |       **28.00**      | 23.00 |     25.00   | 
| light |       **83.00**       | 67.00 | 69.00 |
| ldoor | 32.00 | **39.00** | 27.00 | 
| assembly | **92.67**|  88.67  |    94.00    | 
| relocate | 36.67 |   **44.67**  |   32.00     | 

After searching the parameter K, we find that the k doesn’t influence the results much, and K=2 with our original setting, maintains a good result in most environments.

Reviewer comment (Presentation):
Figures 2, 3, 9, and 10 are crucial for understanding but are confusing. The connections between various inputs and outputs are not obvious, and notation is inconsistent (e.g., whether $j=10$ in Figure 2). Figures 2, 9, and 10 use different visual styles unnecessarily, making it hard to distinguish important from unimportant differences. Figure 2 could be clearer if it showed just one TFD block instead of the current abstraction with a zoomed view.

Author response:
Thanks for pointing it out! We have deleted the strong words like “paradigm shift” and “consistently”. Fig. 2 is more important for understanding, the Fig. 9 and Fig. 10 are just specific TFD designs for different visual backbones, resnet, vit, and video transformers, which cover the main-trend computer vision backbone. We have a zoom-in for more clarity on the three general tasks. We also change the Fig. 2, making it more readable.