Paper: RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches (ICLR-2024)

Reviewer comment (Novelty):
The paper does not make a strong enough connection between the traditional idea of trajectory tracking and the proposed approach. For example, there is no section in the related work covering the history of this technique and its recent applications to robot learning. As mentioned above, I still see some innovations in the proposed approach (sensorimotor controllers are generally conditioned on a low-dimensional trajectory, not an image of it)—however, this lack of perspective results in overclaiming the system's novelty.

Author response:
We thank the reviewer for pointing out the missing discussion on traditional trajectory tracking for robotic control. We have included a dedicated subsection in the related work to situate our method against the relevant literature on trajectory tracking in control theory and robotics. Namely, we would like to highlight the body of work on adaptive and learning-based control which may be applied to closed-loop control for tracking reference trajectories, which may be provided after an initial trajectory planning stage or be updated dynamically in an online fashion. Most importantly, we emphasize that these model-based trajectory tracking algorithms are generally sensitive to accurate trajectory plans (which must be overspecified in ground truth state space) and are not robust to tracking errors in presence of dynamic or uncertain situations. Our proposed method makes fewer assumptions on the fidelity and accuracy of the coarse trajectory sketch, and aims to leverage the benefits of end-to-end learning to generalize to uncertain or complex scenarios. Nonetheless, we view our method and similar approaches as a hybrid abstraction that aims to leverage some of the benefits showcased in model-based trajectory tracking systems while also exploiting the benefits of end-to-end data-driven policy learning.

Reviewer comment (Experiments):
The comparison against baselines in Sec 4.2 is not apples to apples: while the proposed approach gets as input a trajectory on how to complete the task, the other methods only access a higher level instruction (e.g. language), or the final goal represented as an image. It is, therefore, not surprising that with more and cleaner information, the proposed approach outperforms the baselines. I find the experiments in Table I to be much more informative. They show that when the trajectory is very coarse, e.g. when drawn by a human, the proposed approach can track it better than the off-the-shelf IK. However, these results also show that the approach is, on average, worse than generating a trajectory with a VLM and tracking it with an IK, which somehow is against the main paper's contribution.

Author response:
We acknowledge that the comparison in Sec 4.2 is not apples to apples. The objective of this section is to demonstrate that trajectory-conditioned policies outperform baselines with language or goal image conditioning. It indicates a policy conditioning representation showing how to complete a task can be more amenable to task generalization than common baseline approaches.

---

Our approach performs worse on the task, Open Drawer, since the waypoints (always grasping the drawer handle) generated by LLM on this task are quite different from the motions in our training data (which mainly grasps the drawer edge). As shown in the Appendix (retry behaviors), RT-Trajectory can follow the trajectory in order to grasp the handle first, while finally opening the drawer by grasping the edge. Such retry behaviors are inherently impossible for the IK planner. Besides, we would like to point out that the IK planner requires relatively accurate trajectories of 6D poses while our approach only takes a 2D sketch as input.

Reviewer comment (Experiments):
Each subsection seems to evaluate on different "tasks", and it is unclear how the task selection is done. For example, why change the task from 4.2 to 4.3? And why even change it in the comparisons of Table 1? Note that my point is not that the task were handpicked, but rather that this dynamics makes drawing insights from experiments challenging.

Author response:
Thank you for pointing out this question about task selection. We have done additional experiments to compare RT-Trajectory against the IK planner given trajectories generated from human demonstrations videos on the Open Drawer skill and prompting LLMs on the Fold Towel skill, in order to ensure the same set of skills are evaluated in Table 1. Notably, the IK planner consistently fails to grasp the drawer edge (resulting in a success rate of 0), likely caused by the noisy trajectories generated from human demonstration videos for Open Drawer.

| Method       | Open Drawer (human demo video) | Fold Towel (Prompting LLMs) |
|--------------|--------------------------------|-----------------------------|
| IK Planner   | 0%                             | 25%                         |
| RT-Traj 2D   | 60%                            | 0%                          |
| RT-Traj 2.5D | 90%                            | 25%                         |

The reason why we only evaluate some of the seen or unseen skills in Sec 4.3 rather than all 7 unseen skills in Sec 4.2 is that the proposed unseen skills might be challenging for certain trajectory generation methods. For example, prompting LLMs to achieve swiveling a chair or folding a towel is empirically very difficult and out of the scope.

Reviewer comment (Evaluation):
The paper has as its main claim generalization. Still, its main experiments on generalization over scenes and tasks only occupy a small part of the paper (Sec. 4.5). At the very least, this part should become quantitative since this is what would back up the claims on generalization.

Author response:
In this work, we emphasize task-level generalization, and mainly present quantitative results on 7 unseen skills in Sec 4.2, as well as motion analysis in Sec. 4.4. Sec. 4.5. mainly shows some emergent abilities and case studies qualitatively.

Reviewer comment (Presentation):
The ordering of Fig. 2 seems rather arbitrary. For example, in my subjective opinion, I see trajectory and code to be much closer than what is shown right now. Therefore, either more justification is needed, or the figure should be removed.

Author response:
Thank you for the feedback. We agree that the figure can be subjective. We will remove it in the revision.

Reviewer comment (Theory):
The approach makes strong assumptions that contradict the claim on scalability. Specifically, the two main assumptions are: (1) the camera should be static at training and test time, as well as fully calibrated, and (2) the end effector should always be visible in the camera. This makes data collection and evaluation very challenging and limits the set of tasks that can be done.

Author response:
Thank you for pointing out our assumptions that might affect scalability. 

1) Static and calibrated camera during training and inference: When creating hindsight trajectory labels during training, we extracted segments from demonstrations which only contained stationary manipulation. During inference, one potential way to extend to mobile camera scenarios is to redraw trajectories if the robot moves during the rollout. As a future work, we are exploring automatically re-generating new trajectories on-the-fly during inference. In addition, we would like to emphasize that both our training data and evaluation are collected and conducted across different robots (distributing collection and evaluation on more than 10 robots). Thus, it is not required to have a highly precise camera calibration.

2) “The end effector should always be visible in the camera”: The end-effector is not required to always be visible within one episode. We will snap the projected trajectory to the edge of the image if the end-effector is out of view. It is also reasonable to assume that the end-effector is within the view for most time for a visual servoing system.

Reviewer comment (Presentation):
In Fig. 8, I don't get what "unrelated semantic categories" mean for a low-level skill. At the level of trajectory, why is generalization over semantics a question? Isn't it obvious due to the abstraction of the skill? Or did I misunderstand the meaning?

Author response:
Thank you for the question. We have refined Fig. 8 and removed this description in the revision. For a learning-based policy, the task specification (e.g., semantics) can be highly related to the low-level motion. For example, for a language-conditioned policy, even if the low-level motion is similar, due to large gaps of language conditioning between training and test, the policy can still fail to generalize. We intend to emphasize that our approach can generalize to new tasks with unseen semantics because it is designed to generalize based on motion.

Reviewer comment (Novelty):
Better positioning of the work with respect to previous literature.

Author response:
We thank the reviewer for pointing out the missing discussion on traditional trajectory tracking for robotic control. We have included a dedicated subsection in the related work to situate our method against the relevant literature on trajectory tracking in control theory and robotics. Namely, we would like to highlight the body of work on adaptive and learning-based control which may be applied to closed-loop control for tracking reference trajectories, which may be provided after an initial trajectory planning stage or be updated dynamically in an online fashion. Most importantly, we emphasize that these model-based trajectory tracking algorithms are generally sensitive to accurate trajectory plans (which must be overspecified in ground truth state space) and are not robust to tracking errors in presence of dynamic or uncertain situations. Our proposed method makes fewer assumptions on the fidelity and accuracy of the coarse trajectory sketch, and aims to leverage the benefits of end-to-end learning to generalize to uncertain or complex scenarios. Nonetheless, we view our method and similar approaches as a hybrid abstraction that aims to leverage some of the benefits showcased in model-based trajectory tracking systems while also exploiting the benefits of end-to-end data-driven policy learning.

Reviewer comment (Experiments):
Motivation behind task selection.

Author response:
Thank you for pointing out this question about task selection. We have done additional experiments to compare RT-Trajectory against the IK planner given trajectories generated from human demonstrations videos on the Open Drawer skill and prompting LLMs on the Fold Towel skill, in order to ensure the same set of skills are evaluated in Table 1. Notably, the IK planner consistently fails to grasp the drawer edge (resulting in a success rate of 0), likely caused by the noisy trajectories generated from human demonstration videos for Open Drawer.

| Method       | Open Drawer (human demo video) | Fold Towel (Prompting LLMs) |
|--------------|--------------------------------|-----------------------------|
| IK Planner   | 0%                             | 25%                         |
| RT-Traj 2D   | 60%                            | 0%                          |
| RT-Traj 2.5D | 90%                            | 25%                         |

The reason why we only evaluate some of the seen or unseen skills in Sec 4.3 rather than all 7 unseen skills in Sec 4.2 is that the proposed unseen skills might be challenging for certain trajectory generation methods. For example, prompting LLMs to achieve swiveling a chair or folding a towel is empirically very difficult and out of the scope.

Reviewer comment (Experiments):
Generating the trajectory itself might not be trivial. For example, in the case when a person draws the trajectory, the person needs to understand the trajectory of the end-effector (including any joint limits) and project it to a 2D space. This can become a bottleneck, especially for more difficult tasks (e.g. large-range motion, long-horizon tasks). More analysis of the robustness of the trajectory specification could strengthen the paper.

Author response:
Thank you for the valuable feedback. We acknowledge that trajectory sketches are not as easy to acquire as language instructions, but also note that we show that trajectory-conditioned policies are easier to prompt for generalizing to new robotic tasks compared to language-conditioning. We believe that as foundation models (e.g., image-generating VLMs) rapidly improve, we expect that their trajectory sketch generating capabilities will improve naturally in the future and be usable by RT-Trajectory.

---

> “More analysis of the robustness of the trajectory specification could strengthen the paper.”

Thank you for the valuable suggestion. We have conducted additional experiments to test the robustness of our trajectory specification by adding different types of noise to human-drawn trajectory sketches: 1) adding a global offset (5 pixels) to the entire 2D trajectory, 2) adding independently sampled local offsets (5 pixels) to individual interaction markers. For each noise type, we ran a total of 22 trials on 3 skills: Place Fruit, Fold Towel, Move within Drawer. The results are shown in the table below, where we find that RT-Trajectory is robust to mild noise.

| Method & noise type          | Place Fruit (12) | Fold Towel (4) | Move within Drawer (6) | Total success (22) |
|------------------------------|------------------|----------------|------------------------|--------------------|
| RT-Traj 2D                   |                9 |              4 |                      5 |                 18 |
| RT-Traj 2D + global offset   |                8 |              4 |                      4 |                 16 |
| RT-Traj 2D + marker offset   |                9 |              4 |                      4 |                 17 |
| RT-Traj 2.5D                 |                8 |              3 |                      4 |                 15 |
| RT-Traj 2.5D + global offset |               10 |              3 |                      4 |                 17 |
| RT-Traj 2.5D + marker offset |                9 |              3 |                      4 |                 16 |

Reviewer comment (Theory):
As mentioned in the limitation section already, the method seems to be limited to a fixed-view camera with calibration, limiting the applicability of the method.

Author response:
We agree with this limitation of our current implementation. Nonetheless, we are optimistic that this limitation can potentially be addressed by training on cross-embodiment data with different camera configurations, which we leave to future work.

Reviewer comment (Presentation):
Figure 8 is confusing to me. From the left visualization, the query trajectories are well captured by the training trajectories, suggesting no extrapolation and mostly interpolation. On the other hand, the right part is difficult to understand. Can the authors clarify or change the claim accordingly?

Author response:
Thank you for pointing out the unclarity of Fig. 8. We have updated Fig. 8 in the revision, by separating it into multiple figures and improving captions for more clarity. 

The left side of Fig. 8 shows the 10 most similar trajectories in the training data. Note that we also show additional visualizations from other viewpoints in Fig. 13 (new), where the most similar training trajectories might look more different from query ones.

We find that the trajectories of some unseen tasks can be surprisingly similar to those of seen tasks. Thus, we analyze statistics of the most similar training samples over a set of queries for unseen skills on the right side of Fig. 8. In Fig. 12 (new), query trajectories of unseen skills in general see larger Frechet distances to the most similar training trajectories, compared to query trajectories from training skills. It indicates that many proposed unseen tasks show significant differences from seen tasks w.r.t. motion.

Besides, in Fig. 10 (new), we show the distribution of semantics skills of the most similar training trajectories to rollout trajectories. The unseen task placing fruit into bowl is similar to a seen task move X near Y w.r.t. motion; however, their semantics are quite different, which explains why language-conditioned policy struggles to generalize in this case. Moreover, instead of “extrapolating”, we expect RT-Trajectory to stitch seen motion segments to tackle new tasks. As a good example, RT-Trajectory manages to combine motions from Open top drawer and Place object into top drawer to tackle Move within Drawer.

Furthermore, we would like to point out that some criteria are missing in the current version of our proposed motion similarity metric, e.g., the end-effector orientation and when the gripper is closed or opened. For example, RT-Trajectory can respect the trajectory conditioning for Swivel Chair and does not close the gripper during execution, while its most similar training trajectories all require at least one gripper interaction. Such a difference is not captured yet. And we leave it to future work.