Paper: End-Effector-Elbow: A New Action Space for Robot Learning (ICLR-2024)

Reviewer comment (Novelty):
My biggest concern is, the proposed method is more like a small engineering trick, rather than a rigorous scientific approach. It is specific to 7-DoF arm, which presents opportunities for such method due to the one additional redundant joint. The method won't work for 6-DoF arms, which although doesn't come with redundancy, but will also present multiple disconnected IK solutions given one end-effector pose, and only a subset of them would satisfy the task constraints. It also won't work for arms with DoFs greater than 7: which joint should we choose then? Or do we need to control 2 joints?

Author response:
**The work can be seen as an engineering trick**: Contrary to viewing the 7 DoF as an inherent opportunity for E3, our new action space is a simple strategic response to this overactuation problem, commonly encountered in robot arms with greater than 6 DoF. The E3 action space applies classical robotics control techniques to Robot Learning (see "Redundant manipulators control" part of the "Related Work" section of the original paper), aiming to replace Joint and EE action spaces currently used in the robot learning literature.

While the E3 action space offers a straightforward solution, its simplicity is its strength: it can be seamlessly applied to many existing and future robot learning works. Additionally, it is clear from our findings that this simple change in action space for robot learning dramatically improves performance on tasks where whole-arm control is imperative. As we confront increasingly complex challenges, such as navigating real-world scenes involving cupboards and other intricate, hard-to-navigate areas, it becomes evident that conventional action spaces fall short in providing the necessary control over the arm's configuration whilst maintaining data-efficient learning.

Previous contributions, which also mostly focus on re-engineering the action space, e.g. by adopting primitives [1] or next-best pose [2], has had a significant impact on the field, leading to higher performance, through only minor modifications of the learning algorithm.



**Generalization to $\neq$ 7 DoF arms**: In the case of a 6 DoF arm, such as the UR robot family, where the "elbow" has two solutions for end-effector (EE) poses (elbow up or elbow down), E3J method would utilize pose information along with a binary flag indicating elbow orientation. This contrasts with the conventional approach in robot control for UR robots, where the elbow joint is typically constrained to either the up or down position.

Additionally, 6 DoF arms face limitations due to their morphology. Unlike overactuated arms, they may struggle with tasks demanding flexibility. For instance, consider our real-world setup with a cabinet except replacing the Franka Emika Panda arm with a UR5. The UR robot's vertical elbow position would hinder solving the task from this position where the elbow needs to be rotated to the horizontal place in order to reach inside the cupboard. Visual example in simulation available at https://imgur.com/a/n4L5UZH : while the cup position would be reachable by the UR5, the presence of the obstacle and the stiffness of the UR5's elbow do prevent the robot from reaching inside the cabinet.

Reviewer comment (Writing):
Some explanations are not clear: It says `elbow`, but in the method it says empirically it uses the base joint.

Author response:
**Why the method talks about the elbow?** in most 7 DoF arms, the redundancy of the robot allows control over one more dimension, which can be identified in the position of the elbow. As we mention in the Introduction, E3A allows direct control of the elbow by controlling its angle, while E3J allows indirect control of the elbow by constraining one of the joints to fix the elbow position. Thus, both the methods presented allow control over the elbow redundancy (see Figures 2 and 3 from the original submission for further illustration).

Reviewer comment (Experiments):
I don't understand why choosing any joint in between would make problems for IK solver: it's just removing 1 DoF and the solver should work just fine. There's also no experiments to empirically validate which joint to choose.

Author response:
**Experiments to validate the choice of the joint**: in Section 4.4, we motivated our choice for mainly considering E3J-base and E3J-wrist in the first iteration of the manuscript. In order to further validate our choice, we have now completed the ablation study, where we attempt to constrain other joints of the robot for E3J (see Section A4 and Figure 13 in Appendix). The results show that our intuitions generally proved correct, but there are tasks where constraining other joints than the base or the wrist improves sample efficiency. We also believe E3A may constitute a more general strategy than E3J, but in practice we found that the agent struggles to directly control the elbow angle more than indirectly controlling the elbow through one constrained joint. As finding out what's the best action space for robot learning, in terms of full-body control and efficiency, is one of the main goals of this work, we have chosen E3J as the idea that best fits this definition according to empirical evidence.

Reviewer comment (Experiments):
The overall approach is highly engineered for particular scenarios and robots and is not general. For instance, in the E3J method (the main method), the paper proposes to constrain the base joint, but that is an arbitrary choice. Certain parts of a task may benefit more from constraining the wrist joint, and there can be tasks that use configurations where constraining the middle joint is useful. For example, in a scenario where the Franka arm has been rotated such that it lies flat and the middle joint acts as an elbow, the proposed approach would be infeasible.

Author response:
**The approach is not general**: as pointed out by the reviewer, the idea of constraining one specific joint for E3J is an arbitrary choice and may not always be the most effective choice. In the original submission, we discussed this in Section 4.4, in the Joint Selection and Practical Implementation Choices paragraphs, where we motivated our choice for mainly considering E3J-base and E3J-wrist. In order to further validate our choice, we have now completed the ablation study, where we attempt to constrain other joints of the robot for E3J (find results in the updated manuscript's Appendix). The results show that our intuitions generally proved correct, but there are tasks where constraining other joints than the base or the wrist improves sample efficiency. We also believe E3A could be a more general strategy than E3J, but in practice we found that the agent struggles to directly control the elbow angle more than indirectly controlling the elbow through one constrained joint. As studying what's the best action space for robot learning, in terms of full-body control and efficiency, is one of the main goals of this work, we have selected E3J as the idea that best fits this definition according to empirical evidence.

Reviewer comment (Novelty):
The E3J idea can be viewed through the lens of a hierarchical policy (where first a constrained joint is learned and then the EE action is determined). Similar ideas of hierarchical control and learning in hierarchical action spaces, especially with respect to null-spaces, have been considered in prior work (Sharma et al.), but this prior work is not cited or discussed in the paper.

Author response:
**Relation to existing work**: we found the work of Sharma et al, which presents a hierarchical framework to combine task-space constraints, to be relevant to our method, and thus we added a discussion about it in the revised manuscript. Overall, as pointed out by the reviewer, the work presents several differences from our approach, in the settings and the goal pursued: (i) it presents a hierarchical method for selecting actions, while our method focuses on learning flat policies; (ii) it applies task-space constraints, while we combine task-space control with joint constraints, (iii) the goal of Sharma et al's work is to present a modular framework that improves sample efficiency and generalization, while we explicitly focus on studying the most convenient action spaces that enable reinforcement learning or imitation learning methods to solve full-body tasks, whilst retaining the benefits of controlling the agent in task space coordinates. We discuss the work in the Related Work section of the revised manuscript (highlighted in green), along with citing the relevant literature about null-space projections and task-space constraints that have been recommended.  

Furthermore, the reviewer made a comment that an interesting direction could be to combine our constraints for controlling the elbow (controlling the elbow angle or one of the robot's joints) with a hierarchical controller (as in Sharma et al). This indeed sounds promising for future research, and so we added it in the Conclusion (see the revised manuscript, in green text).

Reviewer comment (Novelty):
The idea of hierarchical composition in Sharma et al. is more general, as it composes an arbitrary number of controllers in null-spaces. This automatically leads to the E3J approach, where first a joint is selected and then IK is used under the constraint of this joint to reach the desired target (thus automatically acting in the null-space). A difference between Sharma et al. and the current work is that Sharma et al. only consider task-space constraints and use task-space impedance control for the robot, whereas the current work uses a joint constraint and a task-constraint/goal with IK for control. However, there are also prior works (as referenced in Sharma et al.) that do controller composition directly in the joint space.

Author response:
**Hierarchical control**

While the idea of hierarchical control would definitely be interesting to explore for future work, and we discussed this in the paper, as the reviewer acknowledged, **the current work is about a new action space, involving no hierarchy**.  Hierarchical control requires the agent to reason about actions in different stages while our work requires the agent to output a hybrid set of coordinates, in task and joint spaces, in one flat stage. Hierarchical control mostly employs actions that are temporally-consistent and last for a longer time, while our work focuses on atomic actions that should be provided at high frequency for closed-loop control.

For example, in the work mentioned by the reviewer [1], the agent should reason about which controllers should be combined, by selecting multiple object-axis controllers in a prioritized order, which are composed together using null-space projections. In contrast, in our work, there is no priority as the action's dimensions have all the same priority, and thus the policy is "flat".

Another example is primitives [2], where the agent should reason, first, about which primitive to use, and then, on the parameters for applying the primitive to the environment. Our approach jointly controls the robot's configuration by providing targets in task and joint coordinates, rather than prioritizing any of them.

Extending our work to flexibly choose which joint to directly control with E3J is described as a future work direction in the Conclusion, and we believe our work, including all the assets we are releasing (new E3A and E3J action spaces, new full-body tasks to evaluate, new robot models to experiment with), will provide **an useful starting point and a solid baseline for future research in this area**.

[1] Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation, *Sharma et al*

[2] Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives, *Dalal et al*

Reviewer comment (Novelty):
It may be useful to combine the current paper with Sharma et al. and consider more general hierarchical policies. Additionally, there are other related works that focus on learning constraints from demonstrations, which should also be cited. Examples include:
- Sharma et al. *Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation*
- Lin et al. *Learning Null Space Projections*
- Howard et al. *A novel method for learning policies from variable constraint data*

Author response:
**Relation to existing work**: we found the work of Sharma et al, which presents a hierarchical framework to combine task-space constraints, to be relevant to our method, and thus we added a discussion about it in the revised manuscript. Overall, as pointed out by the reviewer, the work presents several differences from our approach, in the settings and the goal pursued: (i) it presents a hierarchical method for selecting actions, while our method focuses on learning flat policies; (ii) it applies task-space constraints, while we combine task-space control with joint constraints, (iii) the goal of Sharma et al's work is to present a modular framework that improves sample efficiency and generalization, while we explicitly focus on studying the most convenient action spaces that enable reinforcement learning or imitation learning methods to solve full-body tasks, whilst retaining the benefits of controlling the agent in task space coordinates. We discuss the work in the Related Work section of the revised manuscript (highlighted in green), along with citing the relevant literature about null-space projections and task-space constraints that have been recommended.  

Furthermore, the reviewer made a comment that an interesting direction could be to combine our constraints for controlling the elbow (controlling the elbow angle or one of the robot's joints) with a hierarchical controller (as in Sharma et al). This indeed sounds promising for future research, and so we added it in the Conclusion (see the revised manuscript, in green text).

Reviewer comment (Evaluation):
Regarding the generality of the approach, it is important to discuss the challenges of using this approach in very different settings. For example, consider the UR arms where the task only requires one degree of freedom for successful execution. The presented approach would not consider this scenario, but the benefits of a constrained action space for learning could still be useful.

Author response:
**Benefits of applying the approach in tasks that require less DoFs**: in order to compare the learning efficiency of our approach compared to EE on tasks that do not require full-body control, we presented results on 8 RLBench popular tasks, providing aggregated performance in the main paper and detailed results in the Appendix. We did not find any significant difference in performance between EE and E3 in these settings. However, we would like to highlight that 2 out of the 6 full-body tasks are also standard RLBench tasks (`meat off grill` and `take cup out cabinet`), requiring in theory <= 6 DoFs. However, the presence of obstacles in reaching the task's main object ends up favouring our method in terms of learning efficiency, as E3 provides the control necessary for consistent obstacle avoidance.

Reviewer comment (Presentation):
It would be useful to show images or videos of simulation tasks that are being considered as full-body tasks.

Author response:
**Images/videos of full-body tasks**: we added the requested visualizations, with a Figure in the Appendix, and with videos on the [project website](https://doubleblind-repos.github.io/).

Reviewer comment (Reproducibility):
As briefly mentioned in the paper, the authors should clarify whether they used constrained IK (with the joint constraint) to solve for the end-effector target pose. They should also discuss whether there were any issues or challenges related to solving this optimization problem. For example, what if the IK failed because the joint constraint prevented the robot from reaching the desired EE pose? Such scenarios seem likely, but the paper provides very few details on these implementation issues.

Author response:
**Use of IK:** we provided additional details in the Appendix on which IK solver we employed in simulation and real-world tasks (see updated text in green). We thank the reviewer for the suggestion of comparing the running time of different action spaces. For the delta action space experiments, similarly to OSC controllers, we employ a Jacobian-based solver for computing the IK, which takes around 35% more time than directly providing joint coordinates to the simulator. This important detail has also been added to the manuscript (see new green text in Appendix A.5).

As we use a Jacobian-based solver the main issue with the IK failing comes from the inability of satisfying the delta coordinates provided by the agent as a target at each step. As we show in Figure 6b, the action's invalid rate is similar from the start for EE and E3J using the base joint, as opposed to E3A and E3J using the wrist joint. This led us to the hypothesis that the agent failing to provide IK-valid actions more consistently for E3J-WRIST and E3A might be the cause of underperformance, compared to E3J-BASE (Figure 6a).

As OSC controllers generally employ Jacobian-based methods, similar issues might arise if we switched to these controllers for our action space. Nonetheless, it would be worth verifying such a hypothesis in future work, i.e. studying the effects of applying E3 under different underlying controllers.

Reviewer comment (Experiments):
Another issue with an IK-based approach is that it can be expensive to solve at every RL step, especially when performing delta end-effector actions. In such cases, most works use the Jacobian-based controller (OSC). The paper should address how the proposed approach would work with such a choice.

Author response:
**Use of IK:** we provided additional details in the Appendix on which IK solver we employed in simulation and real-world tasks (see updated text in green). We thank the reviewer for the suggestion of comparing the running time of different action spaces. For the delta action space experiments, similarly to OSC controllers, we employ a Jacobian-based solver for computing the IK, which takes around 35% more time than directly providing joint coordinates to the simulator. This important detail has also been added to the manuscript (see new green text in Appendix A.5).

As we use a Jacobian-based solver the main issue with the IK failing comes from the inability of satisfying the delta coordinates provided by the agent as a target at each step. As we show in Figure 6b, the action's invalid rate is similar from the start for EE and E3J using the base joint, as opposed to E3A and E3J using the wrist joint. This led us to the hypothesis that the agent failing to provide IK-valid actions more consistently for E3J-WRIST and E3A might be the cause of underperformance, compared to E3J-BASE (Figure 6a).

As OSC controllers generally employ Jacobian-based methods, similar issues might arise if we switched to these controllers for our action space. Nonetheless, it would be worth verifying such a hypothesis in future work, i.e. studying the effects of applying E3 under different underlying controllers.

Reviewer comment (Theory):
The E3 action space does not consider the dynamics or contact forces of the robot and the environment, which may affect the performance and stability of the robot learning algorithms.

Author response:
**The evaluation does not consider dynamics/contact forces**: for all the approaches we tested, we control the robot by setting target joint positions derived from the agent's output action space: either directly with joint space or through Inverse Kinematics (IK) for EE and E3 action spaces. In regard to the observations/inputs, the agent receives as inputs two RGB images, from the environment and wrist cameras, and a set of proprioceptive states. We did not account for contact forces, however in the RL experiments the agent can infer contacts, by visual or proprioceptive feedback (e.g. the end-effector is blocked) or through the rewards (e.g. the agent is rewarded to stay in contact with some object to complete the task). Nonetheless, we understand this would be useful to know for future readers and thus we added some additional comments on this, when discussing the agent's inputs from the environment, in the revised manuscript (see green text in Appendix A.5).

Reviewer comment (Experiments):
The E3 action space is only tested on 7 DoF robot arms, and may not generaliz to other types of manipulators with different degrees of freedom or kinematics.

Author response:
Dear reviewer,

we would like to thank you for answering to our rebuttal.

In addition, we would like to report that the following experimental studies have been added to the revised manuscript, to corroborate the general applicability of our method:
*  **evaluation on different robotic platforms**: using the Kuka iiwa 7 and xArm. In this setup, we showed that E3J can be applied in other platforms than the Panda
 * **evaluation on a robot with more than 7 joints**: given the absence of common 8 DOFs, we designed a custom Panda, with additional joints, where the agent can control 8 joints. In this setup, we showed that E3J can be applied on robotic arms with more than 1 redundant joint to control.
The results can be found in the Appendix of the revised manuscript. 

We hope the additional evaluations address the concerns that have been raised about evaluating only on a single 7 DOFs arm and extending the work to arms with higher DoFs.

Reviewer comment (Experiments):
Only evaluated on a 7 DoF arm. The framework may not extend well to arms with higher DoFs and 6 DoFs. How does this method extend to higher DoFs? Is there is a generalized version of this framework?

Author response:
**Extension to higher DoFs**: arms with more than 7 DoFs are less common than 6 and 7 DoFs arms, and they are often designed for specific applications, which makes them hard to evaluate in common environments. We discussed this choice of focussing mostly on 7 DoF arms in Section 4.2 (highlighted in blue from the original submission) and, in order to provide a more general idea on how to extend our framework, we provided additional comments (in green) in Section 4.4, discussing how E3J could be adapted for higher DoFs.

Reviewer comment (Novelty):
I think there is a lack of discussion of other action spaces (primitives, OSC etc)

Author response:
**Lack of discussion/comparison to other action spaces**: the reviewer mentioned primitives and OSC as potential action spaces to compare with E3. We believe that, while both ideas are very relevant to the robot learning setting we work on, they are also orthogonal to our approach. Primitives, as in [1,2], provide a more abstract level for controlling the agent, through temporally consistent behaviors (often parameterized in task space coordinates). OSC controllers provide an alternative to IK solvers for controlling the robot in task space coordinates. In both cases, when working with a 7 DoF robot, additional control over the robot configuration needs to be provided to the learning agent for solving tasks in constrained spaces. 

In our work, we discuss and implement two ways (E3J and E3A) to address such limitations, and we test them in RL and IL settings, in a temporally fine-grained setting and using IK solvers. Nonetheless, we agree with the reviewer that testing our approach using OSC controllers, or defining primitives to control the remaining degrees of freedom of the agent, i.e. the elbow of the robot, are relevant ideas to mention for additional evaluations/future work and thus we discuss them in our revised manuscript (highlighted blue content in the Related Work section about primitives and OSC, new green content about OSC controller comparison in Section 5.1/Analysis).

Reviewer comment (Experiments):
There is a lack of comparison to other action spaces as well

Author response:
**Lack of discussion/comparison to other action spaces**: the reviewer mentioned primitives and OSC as potential action spaces to compare with E3. We believe that, while both ideas are very relevant to the robot learning setting we work on, they are also orthogonal to our approach. Primitives, as in [1,2], provide a more abstract level for controlling the agent, through temporally consistent behaviors (often parameterized in task space coordinates). OSC controllers provide an alternative to IK solvers for controlling the robot in task space coordinates. In both cases, when working with a 7 DoF robot, additional control over the robot configuration needs to be provided to the learning agent for solving tasks in constrained spaces. 

In our work, we discuss and implement two ways (E3J and E3A) to address such limitations, and we test them in RL and IL settings, in a temporally fine-grained setting and using IK solvers. Nonetheless, we agree with the reviewer that testing our approach using OSC controllers, or defining primitives to control the remaining degrees of freedom of the agent, i.e. the elbow of the robot, are relevant ideas to mention for additional evaluations/future work and thus we discuss them in our revised manuscript (highlighted blue content in the Related Work section about primitives and OSC, new green content about OSC controller comparison in Section 5.1/Analysis).

Reviewer comment (Novelty):
I don't fully believe this approach is novel - it can be seen in some ways as a special case of [1]. I believe the authors should discuss this further.

Author response:
**Is the work a special case of Riemannian Motion Policies (RMP; Ratliff et al)?** We acknowledge that RMP represents one more way of controlling the elbow of a 7 DoF robot, and thus we added it to the "Redundant manipulators control" paragraph in the related work section, where we discuss other ideas to control the elbow. However, the nature of our work is very different from RMP: we aim to specifically discuss the issue of controlling the full-body pose for 7 DoFs robots, using data-driven learning techniques such as reinforcement and imitation learning. We discussed and empirically validated two implementations, and we acknowledge in the Discussion section that there are many opportunities to extend our study, along which we now also mention RMPs (see Related Work and Conclusion in the updated manuscript).

Reviewer comment (Experiments):
It would be good to see this applied to other robots than Franka (including those with higher Dof).

Author response:
**Testing on other robots than Franka**: we understand that an evaluation on multiple robot platforms could be more insightful and we have started experimentation with two other 7 DoFs robots: XArm 7 with XArm gripper and Kuka iiwa 7 with Robotiq gripper, and we aim to update the manuscript with partial results before the end of the rebuttal period. We will post additional comments when the results are ready. 

Dear reviewer,

we would like to report that the following additional experimental studies have been added to the revised manuscript, to corroborate the general applicability of our method:
*  **evaluation on different robotic platforms**: using the Kuka iiwa 7 and xArm. In this setup, we showed that E3J can be applied in other platforms than the Panda
 * **evaluation on a robot with more than 7 joints**: given the absence of common 8 DOFs, we designed a custom Panda, with additional joints, where the agent can control 8 joints. In this setup, we showed that E3J can be applied on robotic arms with more than 1 redundant joint to control.
The results can be found in the Appendix of the revised manuscript. 

We hope the additional evaluations address the concerns that have been raised about testing on other robots than the classic Franka Panda (including those with higher DOFs).

Reviewer comment (Evaluation):
There are also concerns about how general this approach can be - it is only applicable to overactuated manipulation scenarios. It would be good to discuss this more in the paper.

Author response:
**Specify the method is designed for overactuated manipulation scenarios**: The initial submission made this statement in the Abstract, Introduction, Methods and Conclusion sections (see parts highlighted in blue in the manuscript). To further make it clear, we specified this additional times along the manuscript (in green, see the mentions to "overactuacted arms").