zhiminy/EvalEng huggingface dataset:
Repository: simpler-env/SimplerEnv
Issue: TypeError: array_validator() got an unexpected keyword argument 'labels'
Root cause: External Dependency Breakage

I executed the code as below.

```python
import simpler_env
from simpler_env.utils.env.observation_utils import get_image_from_maniskill2_obs_dict
import mediapy
import sapien.core as sapien
import numpy as np

from mbodied.types.motion.control import HandControl
from mbodied.robots import Robot
from mbodied.types.sense.vision import Image

class SimplerENVRobot(Robot):
  def __init__(self):
    self.task_name = "google_robot_pick_coke_can"  # @param ["google_robot_pick_coke_can", "google_robot_move_near", "google_robot_open_drawer", "google_robot_close_drawer", "widowx_spoon_on_towel", "widowx_carrot_on_plate", "widowx_stack_cube", "widowx_put_eggplant_in_basket"]
    self.env = simpler_env.make(self.task_name)
    # Note: we turned off the denoiser as the colab kernel will crash if it's turned on
    # To use the denoiser, please git clone our SIMPLER environments
    # and perform evaluations locally.
    sapien.render_config.rt_use_denoiser = False
    self.obs, self.reset_info = self.env.reset()
    print("Reset info", self.reset_info)

  def do(self, action: HandControl):
    # Map the motion to the robot.
    print("[ROBOT ACTION]:", action)
    obs, reward, done, truncated, info = self.env.step(np.array(action.flatten('list')))
    self.obs = obs

  def capture(self) -> Image:
    # Capture observation from the robot.
    return Image(get_image_from_maniskill2_obs_dict(self.env, self.obs))

robot = SimplerENVRobot()
robot.capture()
```

The following error occurs.

```
TypeError: array_validator() got an unexpected keyword argument 'labels'
```

How can I solve it?

We didn't test our integration with `mbodied`. You can use debuggers in e.g., vscode, and backtrace to find the errors.
Repository: simpler-env/SimplerEnv
Issue: New object metadata
Root cause: Documentation Deficiency

Hi,

I am setting up a custom environment following instructions given here: [https://github.com/simpler-env/SimplerEnv/blob/main/ADDING_NEW_ENVS_ROBOTS.md](https://github.com/simpler-env/SimplerEnv/blob/main/ADDING_NEW_ENVS_ROBOTS.md).

On bullet 7 of adding new environments, it says: 

"Our environments load metadata json files for the object assets (in ManiSkill2_real2sim/data/custom/info_*.json). Based on your environment implementation, fill in the metadata for each new object asset in existing json files or create new json files."

An example of these info.json files has object descriptions of their bounding box, scale, and density: 

`"bridge_spoon_generated_modified": {
    "bbox": {
      "min": [
        -0.07,
        -0.0173,
        -0.013021
      ],
      "max": [
        0.07,
        0.0173,
        0.013021
      ]
    },
    "scales": [
      1.0
    ],
    "density": 1200
  },`
  
  If I should add a new object, where do I get the bbox characteristics from? Is there a tool that can easily provide this?
  
  Thanks!
 


Hi, you can measure them using blender (press N to toggle object dimension properties), or get the bbox using trimesh. Pay careful attention to the object center.
Repository: simpler-env/SimplerEnv
Issue:  if there are any resources available for migrating SIMPLER to Isaac Sim?
Root cause: Documentation Deficiency

Hello, Dear author. I would like to ask if there are any reference works for migrating SIMPLER to Isaac Sim. I saw in the papers that SIMPLER can be reproduced in Isaac Sim, and I was wondering if there are more detailed guides or tutorials available to help me with the migration. thank you so much!

Hi, you can contact Kyle Hsu for more details https://www.kylehsu.org/
Repository: simpler-env/SimplerEnv
Issue: ImportError: cannot import name 'sa' from 'simulated_annealing'
Root cause: Documentation Deficiency

I encountered an issue when trying to import sa from the simulated_annealing module in `tools/sysid/sysid.py.` The import statement fails so i  installed the module using pip:
```
pip install simulated-annealing
```
the following error arises:
```
cannot import name 'sa' from 'simulated_annealing' (/home/test/miniconda3/envs/simpler_env/lib/python3.10/site-packages/simulated_annealing/__init__.py) 
  File "/home/test/Code/git_repo/SimplerEnv/tools/sysid/sysid.py", line 11, in <module>
    from simulated_annealing import sa
ImportError: cannot import name 'sa' from 'simulated_annealing' (/home/test/miniconda3/envs/simpler_env/lib/python3.10/site-packages/simulated_annealing/__init__.py)
```
I am using version 0.4.0 of simulated-annealing, Upon inspecting the __init__.py file in the installed module directory, I noticed there is no reference to sa, nor any indication of its availability. If sa is not part of the package, it would be very helpful if the documentation could clarify its usage or provide a module of sa to guide me on how to achieve similar functionality.

Thank you for your help and guidance!



When I modified the code as follows:
```
import simulated_annealing as sa
```

I encountered an error indicating that the sa module does not have the minimize() method. How can I correctly import this module and perform the corresponding system identification?

![Screenshot from 2024-12-13 17-41-10](https://github.com/user-attachments/assets/29b9188e-2cf2-458e-b1b0-081d574ff18c)


Could you please advise on the appropriate steps to resolve this issue?

Did you `pip install git+https://github.com/nathanrooy/simulated-annealing`, not `pip install simulated-annealing`?

Thank you for your response. Your solution indeed resolved my issue. During the process of identifying the system, I used my own dataset instead of utilizing RT-1 and OCTO, thus skipping some crucial related steps.
Repository: simpler-env/SimplerEnv
Issue: Object placement in the scene
Root cause: Configuration Error

Hi,

I'm having an issue with placing an object correctly onto the table surface. The object seems to "sink" into the surface after it settles (see image below). It could be that while developing code, I might have changed code that is now resulting in this. Where does ManiSkill decide the "surface" of the table to be? I assumed that this would be encoded in the .glb file. I even downloaded a fresh version of the bridge_table_1_v1.glb to rule out the possibility that the mistake is in the .glb file. However, this issue still persists.

Thanks,
Apurva

![Screenshot from 2024-12-14 18-08-45](https://github.com/user-attachments/assets/f356df99-7ff0-44bb-acef-6130bb0782eb)


This means the object is not modelled perfectly probably. Is this your own object or is this the original one from this repo? And are you using the batched version (maniskill3) of simpler or the original cpu based one (maniskill2)

I was using ManiSkill2. The issue was with the scene_pose; I accidentally modfified the default setting in base_env.py. Is Simpler now compatible with ManiSkill3? 

4 of Simpler environments (bridge dataset) are officially supported in the faster ManiSkill3. See the ManiSkill3 branch: https://github.com/simpler-env/SimplerEnv/tree/maniskill3

It is eventually planned to support the other 4 environments (using google robot) but the timeline for that is uncertain at the moment.
Repository: simpler-env/SimplerEnv
Issue: Google robot end-effector pose

Hi! This is probably not the right place to ask this, but do you happen to know, which frame of the Google Robot is the EE pose (`base_pose_tool_reached`) data collected in the fractal20220817_data dataset w.r.t?

I see [here](https://github.com/simpler-env/ManiSkill2_real2sim/blob/87dc84508520310e61c972ece399a0f034095e42/mani_skill2_real2sim/agents/configs/google_robot/defaults.py#L141-L142) the end-effector is defined as `link_gripper_tcp`, but I am not sure if the fractal dataset treats that as the end-effector. I am trying to get the correct EE pose from Simpler that matches the fractal data.

Thanks!

The simpler policy output EE (output of rt1/octo) should match the fractal EE. Specifically, next_rot = action_rot * cur_rot (wrt base frame), next_transl = action_transl + cur_transl (wrt base frame). `link_gripper_tcp` is the end-effector.

(Internally in ManiSkill env, action are stepped using the "ee-align" mode for google robot, see https://github.com/simpler-env/SimplerEnv/blob/main/simpler_env/utils/env/env_builder.py#L21  https://github.com/simpler-env/ManiSkill2_real2sim/blob/main/mani_skill2_real2sim/agents/controllers/pd_ee_pose.py#L202)

I see, thanks very much Simon! The ee-align part makes sense.

My question was actually more about the **proprioception** EE pose saved in the fractal data. But I suppose, since Simpler uses delta EE including the rotation for the controller, and rt1 works in Simpler, the EE frame defined for action in fractal should be the same as Simpler, and presumably that EE frame for action in fractal is the same as that for proprioception.

Oh actually this is not true any more with "ee-aligning"? suppose there is an actual EE frame in fractal that has a fixed translation from `link_gripper_tcp`, and ee_aligning still works for `link_gripper_tcp` even though the policy is trained with delta actions collected on the actual EE frame. But the propriocetion may have a fixed translation offset now between fractal and simpler

"ee-align" is just a name for decoupling rotation and translation actions (next_rot = action_rot * cur_rot (wrt base frame), next_transl = action_transl + cur_transl (wrt base frame)), instead of multiplying 2 SE(3) matrices (in which case new_pose = action_se3 @ cur_pose_se3). "ee-align" doesn't change the location of ee frame. `link_gripper_tcp` is set exactly at the fractal dataset's ee frame (since it is defined in urdf)

The name "ee-align" follows from the original ManiSkill2, which is not a good name itself. (ManiSkill3 uses "root_translation:root_aligned_body_rotation" to avoid confusion regarding the name.)

Got it! If `link_gripper_tcp` is set exactly at the fractal dataset's ee frame, then there should be no issue. Thanks for the detailed explanation!
Repository: simpler-env/SimplerEnv
Issue: System Identification

I have been trying to run the system identification scripts that have been provided. They do run good but simulated annealing takes too much time for me to complete. Currently, the code is set for 2000 iterations. Did you run it for 2000 iterations? how much time did it take to complete? what were the host system specifications that you ran it on. 

I typically run it for 400 iterations and initialize 3 rounds of sysid using the best result from the prior iteration. 2000 is just for illustrative purposes in case you want to let it run overnight and wait till the next day to see the result

I use a machine w/ 32-cpu thread. 3 rounds in total (or 1200 iterations) take probably 6-7 hours (it only uses 18 threads on the machine as initializing >18 envs on a single 11G GPU will go OOM for ManiSkill2). But you only need to do it once per robot. I was thinking about implementing finite difference-based method which would be faster but didn't have a chance to do it.

Hi @xuanlinli17  I am running the direct example code. The code takes about 6 hours just for a 200 Iterations. I have been trying to do this multiple times.  I am running this on a 48 core AMD processor with 80GB A100. Are you sure this is the latest code you are using ? 



This is quite strange since multiprocessing is already automatically handled; but since you have A100 you can def use larger pool than 18: https://github.com/simpler-env/SimplerEnv/blob/main/tools/sysid/sysid.py#L151

Also how many trajectories do you have? If there are too many trajectories it will be proportionally slower. I used 18 trajectories .

I think this is a vulkan issue ? Maybe my processing is defaulting to CPU? 

Is your gpu utilization >0? you can try "time.time()" to benchmark time elapsed for stepping the environment. The simulated annealing is on cpu though.

Hi @xuanlinli17, the default WidowX configuration that you have released with Simpler is after performing sys ID correct?

> Hi [@xuanlinli17](https://github.com/xuanlinli17), the default WidowX configuration that you have released with Simpler is after performing sys ID correct?

Yes
Repository: simpler-env/SimplerEnv
Issue: maniskill3 environment problem
Root cause: Documentation Deficiency

Hello, I'm new to the robotics domain and have explored SimplerEnv. I find it is an really interesting and  impressive work for robotic development! 

Last month, I used SimplerEnv with maniskill2 version. But I found it's too slow to evaluate octo/openvla models. So I want to switch to maniskill3 version to accelerate evaluation process. However when I install this new version, some problems happened.

***First***, there maybe some mistakes in you maniskill3 readme file.

<div style="text-align: center;">
  <img src="https://github.com/user-attachments/assets/d6a71e64-89a3-4a2d-8964-000d74c0ac2e" alt="image" width="500">
</div>


***Second***, when I followed installation and Full Installation part. There are still some errors.
To be specific, when I run `bash scripts/octo_bridge.sh`, `ModuleNotFoundError` happened.

```
  File "/mnt/hdd1/xingyouguang/projects/robotics/simpler-env3/simpler_env/main_inference.py", line 6, in <module>
    from simpler_env.evaluation.argparse import get_args
  File "/mnt/hdd1/xingyouguang/projects/robotics/simpler-env3/simpler_env/__init__.py", line 2, in <module>
    import mani_skill2_real2sim.envs
ModuleNotFoundError: No module named 'mani_skill2_real2sim'
```
I think, there are some reasons caused this problem:

1. I just run `git clone https://github.com/simpler-env/SimplerEnv`, but also need to git checkout some version. So i git checkout maniskill3. However, I do not find any info about this maniskill3 version in ReadMe file.
2. I switch to maniskill3, but why this error report mani_skill2 ??? I can run `.gitmodules` to clone https://github.com/simpler-env/ManiSkill2_real2sim. But i think this is not correct, because this may convert to maniskill2 environment.
3. I checked `git+https://github.com/haosulab/ManiSkill.git` rep. However, I want to convert it to maniskill2_sim2real. But I do not see their corelations. Because the overlay file does not exists in ManiSkill3 rep.

Could you please help me to solve these problems? Thank you again for your contributions to the community!

Looking forward to your reply!!






@StoneT2000 

fixed

Hi @Lucky-Light-Sun. Could you talk about how to solve this? Really appreciate it!

> Hi [@Lucky-Light-Sun](https://github.com/Lucky-Light-Sun). Could you talk about how to solve this? Really appreciate it!

Checkout the branch, and then build the environment as README. 
The most important thing is that `scripts/*.sh` should not be used any more. We should `simpler_env/real2sim_eval_maniskill3.py` as mentioned in README.

@Lucky-Light-Sun were you able to setup openvla with maniskill3?

> [@Lucky-Light-Sun](https://github.com/Lucky-Light-Sun) were you able to setup openvla with maniskill3?

I didn't have time to modify it last month. But I will do this in the next two weeks. 


@Lucky-Light-Sun Any updates?
Repository: simpler-env/SimplerEnv
Issue: how does the env judge success?
Root cause: Documentation Deficiency

I'm confused of the principle to judge whether a trial is success? often the success rate is zero, maybe it's too strict? how it worked

See "evaluate()" functions in the environments https://github.com/simpler-env/ManiSkill2_real2sim/tree/main/mani_skill2_real2sim/envs/custom_scenes ; the success criterion in sim matches the real eval criterion.

thanks!!
Repository: simpler-env/SimplerEnv
Issue: How to get the gripper's state
Root cause: Documentation Deficiency

Thanks so much for this amazing work.
I'm wondering if it's possible to get the gripper's state information that is aligned with the dataset.
I checked `env.agent.get_gripper_closedness()`, but it seems not to be similar to the real gripper state recorded `episode_steps[i]['observation']['state'][6]`


In this case you need `env.agent.robot.get_qpos()` and retrieve the gripper qpos and remap to the dataset range. Gripper qpos ranges can be obtained from `env.agent.robot.get_qlimits()`.

You can also visualize the robot urdfs in https://github.com/simpler-env/SimplerEnv/tree/main/tools/robot_object_visualization to understand the meaning of each joint's qpos.

Thank you so much for the clarification. 
I have a side question regarding the camera poses.
To get good real-to-sim visual matching, we do need the camera extrinsic w.r.t the robot base to be consistent across real rollout and simulated environments. As far as I know, neither Bridge nor RT-1 data provided camera extrinsic. 
I noticed says [`here`](https://github.com/simpler-env/SimplerEnv/blob/d55e19162be86794875839725fd484b768e25873/ADDING_NEW_ENVS_ROBOTS.md?plain=1#L73), that you can manually tune the camera pose to make it better align with real demonstrations. Is this also how you get the camera extrinsic for the Bridge and RT-1 settings?

I use fSpy to get the camera extrinsics for Bridge. Google Robot's camera extrinsics are already known.

Thank you so much for patiently answering my questions.
Congratulations again on this amazing work!
Repository: simpler-env/SimplerEnv
Issue: `pip install -r requirements_full_install.txt` got error "pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000"
Root cause: Configuration Error

When I try `pip install -r requirements_full_install.txt`. I got following error:
`Collecting transformers (from -r requirements_full_install.txt (line 22))
  Using cached transformers-4.28.1-py3-none-any.whl.metadata (109 kB)
  Using cached transformers-4.28.0-py3-none-any.whl.metadata (109 kB)
  Using cached transformers-4.27.4-py3-none-any.whl.metadata (106 kB)
  Using cached transformers-4.27.3-py3-none-any.whl.metadata (106 kB)
  Using cached transformers-4.27.2-py3-none-any.whl.metadata (106 kB)
  Using cached transformers-4.27.1-py3-none-any.whl.metadata (106 kB)
  Downloading transformers-4.27.0-py3-none-any.whl.metadata (106 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/home/leoliu/anaconda3/envs/simpler_env/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 106, in _run_wrapper
    status = _inner_run()
  File "/home/leoliu/anaconda3/envs/simpler_env/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 97, in _inner_run
    return self.run(options, args)
  File "/home/leoliu/anaconda3/envs/simpler_env/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "/home/leoliu/anaconda3/envs/simpler_env/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 386, in run
    requirement_set = resolver.resolve(
  File "/home/leoliu/anaconda3/envs/simpler_env/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/home/leoliu/anaconda3/envs/simpler_env/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/leoliu/anaconda3/envs/simpler_env/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve
    raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000`

I create conda env with python=3.10, numpy=1.24.4 on a Ubuntu 22.04.5 server with RTX3090. Any idea hot to fix this?

NVM, I fix this issue by install pages in the file line by line.
Repository: simpler-env/SimplerEnv
Issue: Failed to build maniskill2_environment
Root cause: Environment Incompatibility

I attempted to run the CogACT evaluation script within the SimplerENV environment, but my script appeared to hang during the creation of the ManiSkill2 environment. I attempted to log the program’s execution flow and ultimately discovered that it stalls at the following line:

```python
env = build_maniskill2_env(
    env_name,
    **additional_env_build_kwargs,
    **kwargs,
)
```

Below is the relevant output during runtime:
```
[DEBUG] Will call gym.make() with env_name=GraspSingleOpenedCokeCanInScene-v0, kwargs={'lr_switch': True, 'urdf_version': 'None', 'obs_mode': 'rgbd', 'robot': 'google_robot_static', 'sim_freq': 513, 'control_mode': 'arm_pd_ee_delta_pose_align_interpolate_by_planner_gripper_pd_joint_target_delta_pos_interpolate_by_planner', 'control_freq': 3, 'max_episode_steps': 80, 'scene_name': 'google_pick_coke_can_1_v4', 'camera_cfgs': {'add_segmentation': True}, 'rgb_overlay_path': './ManiSkill2_real2sim/data/real_inpainting/google_coke_can_real_eval_1.png', 'rgb_overlay_cameras': ['overhead_camera']}
[2025-02-23 16:31:31.571] [svulkan2] [error] GLFW error: X11: Failed to open display 
[2025-02-23 16:31:31.572] [svulkan2] [warning] Continue without GLFW.
```

After this point, the program remains stuck without producing any additional output.

I would greatly appreciate any insights you can provide, and I am thankful for your contributions!


Does the troubleshooting section of readme solve it?

I re-installed the virtual environment and the problem is solved. Thanks for your reply!

> I attempted to run the CogACT evaluation script within the SimplerENV environment, but my script appeared to hang during the creation of the ManiSkill2 environment. I attempted to log the program’s execution flow and ultimately discovered that it stalls at the following line:
> 
> env = build_maniskill2_env(
>     env_name,
>     **additional_env_build_kwargs,
>     **kwargs,
> )
> Below is the relevant output during runtime:
> 
> ```
> [DEBUG] Will call gym.make() with env_name=GraspSingleOpenedCokeCanInScene-v0, kwargs={'lr_switch': True, 'urdf_version': 'None', 'obs_mode': 'rgbd', 'robot': 'google_robot_static', 'sim_freq': 513, 'control_mode': 'arm_pd_ee_delta_pose_align_interpolate_by_planner_gripper_pd_joint_target_delta_pos_interpolate_by_planner', 'control_freq': 3, 'max_episode_steps': 80, 'scene_name': 'google_pick_coke_can_1_v4', 'camera_cfgs': {'add_segmentation': True}, 'rgb_overlay_path': './ManiSkill2_real2sim/data/real_inpainting/google_coke_can_real_eval_1.png', 'rgb_overlay_cameras': ['overhead_camera']}
> [2025-02-23 16:31:31.571] [svulkan2] [error] GLFW error: X11: Failed to open display 
> [2025-02-23 16:31:31.572] [svulkan2] [warning] Continue without GLFW.
> ```
> 
> After this point, the program remains stuck without producing any additional output.
> 
> I would greatly appreciate any insights you can provide, and I am thankful for your contributions!

Same error,  reinstall virtual env but still  cannot work . 
Did you have vulkan-utils? I think I install vulkan-utils and then I cannot use it.