Repository: simpler-env/SimplerEnv
Issue: Adding new environments
Root cause: Documentation Deficiency

Hi, 

I am trying to setup a new environment in Simpler for the WidowX robot. 

I have a couple of questions regarding the WidowX setup:
1. Did you perform camera calibration with respect to the WidowX robot to set the CameraConfig in [mani_skill2_real2sim/agents/configs/defaults.py](https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/agents/configs/widowx/defaults.py)? If yes, did you perform april tag calibration? If not, would you just recommend estimating the camera pose relative to the robot base in the real and setting the terms p and q accordingly?

Without changing the camera pose, but adjusting for the correct intrinsics, I get the following:

![irom_realsense_intrinsics](https://github.com/user-attachments/assets/ae1a2057-680a-49ce-8821-bf4d1c27a7e3)

The edges are close but not entirely lined up. In addition to the camera extrinsics, perhaps the issue is also with placing the robot correctly with respect to the table. I'm trying to ensure that on both fronts, I am setting up the sim environment correctly.

2. Questions on correctly placing the robot base and the tabletop to match the real-world setup. For this, I'm trying to understand some of the scene setup parameters in [base_env.py](https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/envs/custom_scenes/base_env.py). 
             a. What is the scene_offset parameter? For the WidowX, line [160](https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/envs/custom_scenes/base_env.py#L160) says this is the default offset corresponding to the default offset of bridge_table_1_v1.glb. If I modify the .glb file, for example by changing the table dimensions, how should I derive this offset?
             b. What is the scene_pose and how is it decided? It seems hardcoded to scene_pose = sapien.Pose(q=[0.707, 0.707, 0, 0]).
             c. What are the [_register_render_cameras](https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/envs/custom_scenes/base_env.py#L333) and [_setup_viewer](https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/envs/custom_scenes/base_env.py#L339) functions meant for? I am confused because they seem to be changing the camera extrinsics. I am relying on the demo script to check if the camera observations from the sim and the real line up.

Thank you very much! 


1. I took images from the Bridge dataset and used fSpy with 3 vanishing lines to get the intrinsics for the camera. See https://github.com/simpler-env/SimplerEnv/blob/main/ADDING_NEW_ENVS_ROBOTS.md -> "Notes on camera calibration"
2. (a) The glb file doesn't separate the table and the scene. However, you can set the origin of the glb file to be any location using e.g., Blender.
(b) The scene rotation is due to the habitat's transformation
(c) That's just for human camera used for debugging, not for policy evaluation input.