Repository: simpler-env/SimplerEnv
Issue: Question about rotation representation in action space

Hi, 

Thanks for developing this great evaluation benchmark in simulation! I'm a bit confused about how rotation is represented in the action space, and hope you can help with the following questions. 

1. Based on my understanding, SIMPLER represents the rotation in axis angle format for both Google robot and WidowX environments right?
2. Based on your [inference wrapper code](https://github.com/simpler-env/SimplerEnv/blob/d55e19162be86794875839725fd484b768e25873/simpler_env/policies/rt1/rt1_model.py#L188), it seems that RT models directly predict axis angle for Google robot tasks, and roll pitch yaw for WidowX tasks, while Octo predict roll pitch yaw for both types of tasks? I was wondering why it works in this way? As I checked the Open X-Embodiment dataset and it seems that both Google robot and Bridge datasets label the rotation action dimensions by delta of roll pich yaw? So shouldn't RT models also be trained to predict roll pitch yaw instead of axis angle for Google robot tasks?

Thanks a lot for your help!

1. This follows the rotation convention in ManiSkill2, which take xyz translation and axis-angle rotation as input. The full control mode is https://github.com/simpler-env/SimplerEnv/blob/d55e19162be86794875839725fd484b768e25873/simpler_env/utils/env/env_builder.py#L20 . Controller details are in the ManiSkill2_real2sim repo and explanations of them are in the appendix of the paper.
2. This follows the released RT policies. If you have a new policy to eval you'll need to modify the inference code to match its rotation convention.

Hi Xuanlin, 

Thanks a lot for your help! Sorry that I'm not quite familar with low-level controller in robotics, so have some follow-up questions that hope you can help with. 

For the first point, I was wondering that why Google robot uses delta pose control while WidowX uses *target* delta pose control as shown in your code? Is it designed in this way to align with how the rotation action is represented in the Open X-Embodiment dataset?

For the second point, could you help point me to where in the RT code they do the rotation transformation from euler angle to axis angle? I checked their code but didn't find it. 

Thanks a lot!

1. They are designed to align with real robot controller, not the dataset
2. I think it's some internal code from google

Thanks! I see. I still have some follow-up questions: 
1. Just to confirm, Simpler uses *delta* of both translation and rotation as action input right? Are they both defined w.r.t. to the body frame (the end effector) instead of the root frame?
2. Is the action representation used in the Open X-Embodiment dataset aligned with Simpler? E.g., for Google robot dataset, the metadata says the rotation action is defined as "rpy commanded orientation displacement, in base-relative frame", which seems different from the body frame used by Simpler?

1. Delta w/ decoupled rotation and translation, both wrt robot base frame
2. They don't align with the axis-angle rotation action representation used in ManiSkill. Thus there is a translation from RT-1 / Octo output rotation actions to ManiSkill rotation actions.

Thanks! Sorry that I'm not quite familiar with rotation translations so I have a further question: 

Based on my current understanding, OXE dataset uses delta of euler angle w.r.t. the base frame in its action, while Simpler uses delta of axis angle w.r.t. the base frame for action (based on your reply), so we need to translate the rotation action dimensions from delta euler to delta axis angle.  
For euler angle, we have `euler_current + euler_delta = euler_target`, where `euler_delta` is the policy output. For axis angle, we have `axangle_current + axangle_delta = axangle_target`, where `axangle_delta` is the input to Simpler.  
In Simpler code, we do `axangle_delta = euler2axangle(euler_delta)`.  
But another way is to compute `axangle_delta = euler2axangle(euler_target) - euler2axangle(euler_current)`. As `euler2axangle` is a nonlinear transformation, I think these two ways should give different results and the second way makes more sense?

Or are the equations `euler_current + euler_delta = euler_target` and `axangle_current + axangle_delta = axangle_target` wrong in the first place? As we are actually computing with rotation matrices and can not directly add terms together in either euler or axis angle space?

I'm not sure about the math here, and hope you can help clarify which of these ideas are correct. Thanks!

Rotation in euler angle / axangle cannot be added / subtracted. Precisely, R_policy_output * R_current = R_target where R is a SO(3) (rotation) matrix   (and xyz_policy_output + xyz_current = xyz_target). Both rotation and translation are with respect to the robot base frame for Google Robot and WidowX. Conversion between different rotation formats can be done via `transforms3d` package. Internally, ManiSkill uses `Sapien.core.Pose(q=some_quaternions) * Sapien.core.Pose(q=other_quaternions)` to perform rotation multiplications. 

I see. Everything makes sense to me now. Thanks a lot for your detailed reply!