Repository: simpler-env/SimplerEnv
Issue: Poor performance of OpenVLA on Bridge
Root cause: Environment Incompatibility

Hi, 

I am investigating the reason for poor performance of OpenVLA on Bridge tasks, and I have two questions. Specifically, I am looking at the Put Carrot on Plate task. On real, we have around 65% success for this task on a toy kitchen setup with OpenVLA. But in sim, the success rate drops to around 10%. Certainly, sub-goals such as just grasping have higher success rates. I am trying to understand why the overall success rate drops so low on Bridge tasks for the WidowX, but the drop is not as drastic for Google Robot.

First, consider a running example with action commands of just $\Delta roll$ of the end-effector; there is no change in x,y,z, pitch, and yaw. I conduct a demo in real, and replay the  actions in simulation. The videos of the real and sim demos are shown below. Qualitatively, Simpler seems to be showing the correct robot behavior: rolling the gripper with respect to the x-axis of the global base frame, but the real hardware seems to just yaw. I double-checked the hardware implementation for correctness. Conversely, when we command yaw-only, the real robot behavior shows a similar qualitative performance. I have opened a separate issue on this with [Interbotix](https://github.com/Interbotix/interbotix_ros_manipulators/issues/247#issue-2903359181).

https://github.com/user-attachments/assets/32ce8b7a-dc76-4593-8e2a-d4a7a653a48e
https://github.com/user-attachments/assets/07fd9a8d-d04e-4aa1-bfb2-27ff19aac0d1

Quantitatively, the robot joint angles and end-effector poses are given below:

[joint_waist_sim_traj.pdf](https://github.com/user-attachments/files/19195759/joint_waist_sim_traj.pdf)
[joint_elbow_sim_traj.pdf](https://github.com/user-attachments/files/19195762/joint_elbow_sim_traj.pdf)
[joint_shoulder_sim_traj.pdf](https://github.com/user-attachments/files/19195761/joint_shoulder_sim_traj.pdf)
[joint_wrist rotate_sim_traj.pdf](https://github.com/user-attachments/files/19195760/joint_wrist.rotate_sim_traj.pdf)
[joint_forearm_sim_traj.pdf](https://github.com/user-attachments/files/19195764/joint_forearm_sim_traj.pdf)
[joint_wrist angle_sim_traj.pdf](https://github.com/user-attachments/files/19195763/joint_wrist.angle_sim_traj.pdf)

[ee_x_sim_ee_traj.pdf](https://github.com/user-attachments/files/19195778/ee_x_sim_ee_traj.pdf)
[ee_y_sim_ee_traj.pdf](https://github.com/user-attachments/files/19195774/ee_y_sim_ee_traj.pdf)
[ee_z_sim_ee_traj.pdf](https://github.com/user-attachments/files/19195775/ee_z_sim_ee_traj.pdf)
[ee_roll_sim_ee_traj.pdf](https://github.com/user-attachments/files/19195777/ee_roll_sim_ee_traj.pdf)
[ee_pitch_sim_ee_traj.pdf](https://github.com/user-attachments/files/19195779/ee_pitch_sim_ee_traj.pdf)
[ee_yaw_sim_ee_traj.pdf](https://github.com/user-attachments/files/19195776/ee_yaw_sim_ee_traj.pdf)

In the ee_yaw plot, the sim is showing a considerable yaw though it is not qualitatively visible in the video. Conversely, the real is showing zero yaw, which does not match qualitatively with the real video. This brings me to the first question.

1. What is the best way to compute the end-effector pose in Simpler? I am using the following: 
```
def get_tcp_pose_at_robot_base():
        tcp_pose_at_robot_base = env.agent.robot.pose.inv() * env.tcp.pose
        xyz = tcp_pose_at_robot_base.p.tolist()
        rot_quat = tcp_pose_at_robot_base.q.tolist()
        rot_euler = quat2euler(rot_quat)
        tcp_pose = [*xyz, *rot_euler]
        return tcp_pose
```
I have also tried directly reading the poses of the ee_gripper_link, but the matching is worse. 

3.  In a different pull request, OpenVLA performance in Simpler with visual matching worked quite well for Move Near tasks on the Google Robot. However, this performance drastically drops on the WidowX. 

Could the mismatch in dynamics with the real WidowX be a reason for this? Also, Simpler provides several different control modes (the default is given here in [env_builder](https://github.com/simpler-env/SimplerEnv/blob/052eaea54dc7c37447cf40c499134b55534bbde4/simpler_env/utils/env/env_builder.py#L26)). Do you think a different control mode might improve performance? Also, I was assuming that the provided config for WidowX is after sysID conducted in the Simpler paper.

Thanks very much!

As a rule of thumb, if you replace the Simpler visual with real visual and rollout the policy in sim, and the video show similar behaviors in the real world (you can write a script like https://github.com/simpler-env/SimplerEnv/blob/main/simpler_env/utils/debug/octo_inference_real_video.py), it is unlikely that the action conversion is wrong in sim (using the original Simpler code before your custom conversion), and becomes more of a visual robustness problem for the policy itself. More recent works like open pi-0 and SpatialVLA have demonstrated much better performance on Bridge in Simpler compared to OpenVLA, so it seems to be a unique problem of OpenVLA.

Control params in Simpler are already sysIDed using Bridge dataset in OXE. Control mode should be correct except that real eval is non-blocking, but this is largely compensated by sysID (real control is slightly stiffer).

Thanks, just wanted to clarify whether the custom script I pasted above is for reading the end-effector pose is good. In particular the following line: 

`tcp_pose_at_robot_base = env.agent.robot.pose.inv() * env.tcp.pose`

I only use the above function get_tcp_pose_at_robot_base () for plotting purposes.

Yes it's correct. It's the end-effector tcp pose with respect to the robot base.