zhiminy/EvalEng huggingface dataset:
Repository: simpler-env/SimplerEnv
Issue: RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed on A100 (when using other gpus except CUDA_VISIBLE_DEVICES=0)
Root cause: Configuration Error

Hi authors,

Thanks for your wonderful work. But I met a runtime error when I run
```bash
CUDA_VISIBLE_DEVICES=4 python simpler_env/simple_inference_visual_matching_prepackaged_envs.py --policy rt1 \
        --ckpt-path ./checkpoints/rt_1_tf_trained_for_000400120  --task google_robot_pick_coke_can  \
        --logging-root ./results_simple_eval/  --n-trajs 10
```
It only works when I run it on `CUDA_VISIBLE_DEVICES=0`. How can I fix it? Thank you.

The `vulkaninfo` is shown below.
![Image](https://github.com/user-attachments/assets/479e7aca-4b83-4df1-aad3-f9fc5c542148)

Try `DISPLAY="" CUDA_VISIBLE_DEVICES=x python xxx`

> Hi authors,
> 
> Thanks for your wonderful work. But I met a runtime error when I run
> 
> CUDA_VISIBLE_DEVICES=4 python simpler_env/simple_inference_visual_matching_prepackaged_envs.py --policy rt1 \
>         --ckpt-path ./checkpoints/rt_1_tf_trained_for_000400120  --task google_robot_pick_coke_can  \
>         --logging-root ./results_simple_eval/  --n-trajs 10
> It only works when I run it on `CUDA_VISIBLE_DEVICES=0`. How can I fix it? Thank you.
> 
> The `vulkaninfo` is shown below. ![Image](https://github.com/user-attachments/assets/479e7aca-4b83-4df1-aad3-f9fc5c542148)

Did you find the solution?
Repository: simpler-env/SimplerEnv
Issue: Are the tasks in SimplerEnv based on I.I.D. expeirment?

Hello, thanks for your impressive work.
In recent days, I wanna to do some out of distribution tasks for google robot and widowx robot. But I see in simpler paper that "The tasks are chosen to be representative of those in the corresponding training datasets". So I wanna to ask 

1. Is every thing including objects, distractors, backgrounds and instructions already present in training dataset ? Which means that the model training on OXE dataset has already seen these things.
2. Is combination of objects, scenes and so on same in the training datset(maybe just follow some trajectories in training dataset i.i.d).


Looking forward to you reply.


1. Instructions are in-distribution. Objects being manipulated are in-distribution. Some distractors are in-distribution and others are ood. For variant aggregation evaluation, backgrounds, table textures, lighting conditions are out-of-distribution. For visual matching evaluation, these are as in-distribution as possible (but still slightly ood).
2. Not necessarily; but our eval is iid. The difference is our eval's iid does not guarantee in-distribution with the training data.

> 1. Instructions are in-distribution. Objects being manipulated are in-distribution. Some distractors are in-distribution and others are ood. For variant aggregation evaluation, backgrounds, table textures, lighting conditions are out-of-distribution. For visual matching evaluation, these are as in-distribution as possible (but still slightly ood).
> 2. Not necessarily; but our eval is iid. The difference is our eval's iid does not guarantee in-distribution with the training data.

Thanks for your timely reply. I have some other questions:

1. the object position is set by you or following the training dataset? I see that bridge/widowx evaluation use the `obj_episode_range` as function call paramter.
2. In TABLE IV and TABLE V, the real world evaluation test is evaluated by you, or evaluated in the original paper(however i do not find), or the auther team evaluated it again as SimplerEnv setting?

1. (Range) set by me. They are not necessarily in training dataset.
2.  Evaluated by Google for Google Robot and Berkeley & Stanford for Bridge. The real eval object poses roughly match the ones in sim (due to evaluator variances, they are not exact match).
Repository: simpler-env/SimplerEnv
Issue: Camera Calibration for Open VLA in Isaac Sim
Root cause: Documentation Deficiency

Hello everyone,

I'm currently working on a simulation project with the following setup:

-Operating System: Ubuntu 22.04
-ROS 2 Distribution: Humble
-Motion Planning Framework: MoveIt 2
-Simulation Environment: NVIDIA Isaac Sim 4.2.0
-Programming Language: Python

I'm utilizing Isaac Sim's built-in camera and require assistance with the following aspects of camera calibration:

1. Intrinsic Parameters:
-Determining the camera matrix
-Identifying focal length

2. Extrinsic Parameters:
- Establishing the camera's pose within the simulation environment

3. Calibration Process:
- Best practices for calibrating the simulated camera to ensure accurate data representation

I aim to adjust the camera settings appropriately to enhance the fidelity of my simulation.

I have referred to the official NVIDIA documentation on ROS 2 Cameras in Isaac Sim 
https://docs.omniverse.nvidia.com/isaacsim/latest/ros2_tutorials/tutorial_ros2_camera.html

and the Monocular Camera Calibration guide 
https://nvidia-isaac-ros.github.io/getting_started/hardware_setup/sensors/camera_calibration.html

. While these resources provide foundational information, I am seeking community insights or experiences that could offer additional guidance or practical tips.

Any assistance or pointers to relevant resources would be greatly appreciated.

Thank you!

You can copy the Google robot and widowx camera parameters from SimplerEnv. If you have real setup you can use ChArUco boards for calibration. If only offline image datasets are are availble you can use fSpy for intrinsics calibration as in documentation.

I am doing simulation on isaac sim, and I need to know about camera position and camera calibration. I could not find any document talk about that.

In fact you don't do camera calibration in simulation; instead you do it in the real world / from real datasets and copy the params to the sim you use
The camera calibration process is identical regardless of any simulator you use; it's just the simulation environment that's different.
Repository: simpler-env/SimplerEnv
Issue: Segmentation Error when load model to device.
Root cause: External Dependency Breakage

`https://github.com/DelinQu/SimplerEnv-OpenVLA/`

I am using numpy 1.26

```
~/SimplerEnv-OpenVLA$ bash scripts/run_openvla.sh 


> self.vla = self.vla.to("cuda:0", dtype=torch.bfloat16)
(Pdb) n
scripts/bridge.sh: line 51: 761268 Segmentation fault      (core dumped) CUDA_VISIBLE_DEVICES=${gpu_id} python simpler_env/main_inference.py --policy-model ${policy_model} --ckpt-path ${ckpt_path} --action-ensemble-temp ${action_ensemble_temp} --logging-dir ${logging_dir} --robot ${robot} --policy-setup widowx_bridge --control-freq 5 --sim-freq 500 --max-episode-steps 120 --env-name PutEggplantInBasketScene-v0 --scene-name ${scene_name} --rgb-overlay-path ${rgb_overlay_path} --robot-init-x ${robot_init_x} ${robot_init_x} 1 --robot-init-y ${robot_init_y} ${robot_init_y} 1 --obj-variation-mode episode --obj-episode-range 0 24 --robot-init-rot-quat-center 0 0 0 1 --robot-init-rot-rpy-range 0 0 1 0 0 1 0 0 1
```

If I don't move to device, I have another problem
```
[2025-03-08 18:08:18.317] [svulkan2] [error] GLFW error: X11: Failed to open display 
[2025-03-08 18:08:18.317] [svulkan2] [warning] Continue without GLFW.
Traceback (most recent call last):
  File "/homSimplerEnv-OpenVLA/simpler_env/evaluation/maniskill2_evaluator.py", line 89, in run_maniskill2_eval_single_episode
    is_final_subtask = env.is_final_subtask()
AttributeError: 'TimeLimit' object has no attribute 'is_final_subtask'
```

`env.unwrapped.is_final_subtask()` seems can solve this problem. 

Try `DISPLAY="" CUDA_VISIBLE_DEVICES=x python xxx`
Also what's your gymnasium version?

> `https://github.com/DelinQu/SimplerEnv-OpenVLA/`
> 
> I am using numpy 1.26
> 
> ```
> ~/SimplerEnv-OpenVLA$ bash scripts/run_openvla.sh 
> 
> 
> > self.vla = self.vla.to("cuda:0", dtype=torch.bfloat16)
> (Pdb) n
> scripts/bridge.sh: line 51: 761268 Segmentation fault      (core dumped) CUDA_VISIBLE_DEVICES=${gpu_id} python simpler_env/main_inference.py --policy-model ${policy_model} --ckpt-path ${ckpt_path} --action-ensemble-temp ${action_ensemble_temp} --logging-dir ${logging_dir} --robot ${robot} --policy-setup widowx_bridge --control-freq 5 --sim-freq 500 --max-episode-steps 120 --env-name PutEggplantInBasketScene-v0 --scene-name ${scene_name} --rgb-overlay-path ${rgb_overlay_path} --robot-init-x ${robot_init_x} ${robot_init_x} 1 --robot-init-y ${robot_init_y} ${robot_init_y} 1 --obj-variation-mode episode --obj-episode-range 0 24 --robot-init-rot-quat-center 0 0 0 1 --robot-init-rot-rpy-range 0 0 1 0 0 1 0 0 1
> ```

It happens again and I don't know why. My gymnasium is 1.1.1
1.   install gymnasium==0.28.1, cannot work. 
2. try `DISPLAY="" CUDA_VISIBLE_DEVICES=0 python  ...` cannot work. `export DISPLAY=""
export CUDA_VISIBLE_DEVICES=${gpu_id}` cannot work.  Try https://github.com/simpler-env/SimplerEnv/issues/17#issuecomment-2248106758 cannot work. 
3. degrade numpy from 1.26.4 to 1.24, it works .   https://github.com/simpler-env/SimplerEnv/issues/12#issuecomment-2869122912
Repository: simpler-env/SimplerEnv
Issue: Source of Real Evaluation Results

Hi,

I am wondering where are the [real evaluation results](https://github.com/simpler-env/SimplerEnv/blob/052eaea54dc7c37447cf40c499134b55534bbde4/simpler_env/utils/metrics.py#L7) from `simpler_env/utils/metrics.py` are coming from? 

I want to add more environments and would like to test the real to sim gap in a similar way as the paper. 

Thanks! 


Google robot eval by Google people; Bridge eval by berkeley and stanford.

Thanks for the quick response! Just to clarify, did they conduct real evaluations only for the environments/tasks already in the repository, or are there additional resources with real evaluation results for other environments/tasks?

Tasks are the same as in simulation (this repository)
Repository: simpler-env/SimplerEnv
Issue: Sim-and-real trial results
Root cause: Documentation Deficiency

Hi,

I'm wondering if you can make available the sim-and-real results for each experiment trial? If this data already available, could you please point me to it?

So far, I have found the real and simulated success rates for each task in the file [metrics.py](https://github.com/simpler-env/SimplerEnv/blob/main/simpler_env/utils/metrics.py). However, this also does not seem to have the success rates for the grasping tasks (shown in Fig. 7). Where can this be found?

For the variants experiments, real success rates for variations of a task (e.g., pick_coke_can when placed horizontally/vertically/standing) is given in the file [calc_metrics_evaluation_videos.py](https://github.com/simpler-env/SimplerEnv/blob/main/tools/calc_metrics_evaluation_videos.py). 

You can find the simulation evaluation's per trial results at https://huggingface.co/datasets/xuanlinli17/simpler-env-eval-example-videos/tree/main . The real eval's per-trial results are split across different sources from different people who performed the experiments. If you really want them, you can send me an email at xul012@ucsd.edu (unfortunately I don't have full per trial videos from Google Robot operators, only example videos and numbers). However, note that SIMPLER averages success over different policy seeds (e.g., Octo) and different tuned robot arm colors (e.g., for google robot) to reduce the variance of evaluation, so it is not very meaningful to draw conclusion if you want to compare real and sim eval trial-by-trial (instead of comparing their average success). This is because policies often exhibit different error patterns (in both real and sim) under slight visual changes or seed changes even when robots & objects placed in the same position & orientation. On the other hand, the average successes are much more meaningful as they have much lower variance.

The "grasping" tasks in the Bridge evaluation are the defined as the "partial success" of the full Bridge tasks (e.g., for "put eggplant in yellow basket", the "grasping" task is successful as long as the robot grasps the eggplant, regardless of whether the robot has put the eggplant in the yellow basket). With that said, the trajectories for the "grasping" tasks are the same as the trajectories for the full Bridge tasks. Their raw success rates are in https://github.com/simpler-env/SimplerEnv/blob/main/tools/calc_metrics_evaluation_videos.py. You can also find the raw success rates in Appendix Table 4 and 5.
Repository: simpler-env/SimplerEnv
Issue: Render problem of visiualization error
Root cause: Environment Incompatibility

Hi! Did anyone face this problem when testing the task "Google robot close drawer", as I checked the video, the images are not matched 
It seems to have no effect on the execution of policy(as I can tell success by varible), why could this happen?
![Image](https://github.com/user-attachments/assets/0ec0285f-dda2-4908-8f36-846c67e65295)

Thank you so much!

Are the assets you downloaded complete and repo is fully cloned? (not due to internet issues etc)
Repository: simpler-env/SimplerEnv
Issue: How to use my own dataset in evaluating a finetuned octo model
Root cause: Documentation Deficiency

Thanks for your great work!

When I wanted to use simpler env to evaluate my own finetuned octo model with my data, I found some missing parameters: robot_init_x, robot_init_y, object_init_x, and object_init_y. Are these parameters determined by my own data? Or could you please tell me how can I define them? 

Moreover, how should I define the dataset_id parameter in my own way? And it seems that there is no explicit place for me to modify the code to load my own dataset? 

Looking forward to your reply!

Thank you very much.

The robot and object poses (parameters) should be the same regardless of the model you evaluate. You can do custom eval by modifying the robot and object poses in eval script. See scripts in https://github.com/simpler-env/SimplerEnv/blob/main/scripts/
Repository: simpler-env/SimplerEnv
Issue: `main_inference.py: error: argument --enable-raytracing: expected one argument`
Root cause: Validation Gap

When I try to run `bash scripts/run_openvla.sh`. 
is it should be true or false?



See https://github.com/simpler-env/SimplerEnv/blob/main/scripts/rt1_drawer_visual_matching.sh
It should be `--enable-raytracing` as it stores true. I don't have `run_openvla.sh` in my repo, so it's a fork.

> See https://github.com/simpler-env/SimplerEnv/blob/main/scripts/rt1_drawer_visual_matching.sh It should be `--enable-raytracing` as it stores true. I don't have `run_openvla.sh` in my repo, so it's a fork.

Yes I am runnning `  drawer_variant_agg.sh`  and show this error.

What is the effect/benefit of enable-raytracing?   It will make the A100 GPU evaluate slower. I saw `enable raytracing, slow for non RTX gpus`

Unfortunately for drawer envs (non-table-top envs) you have to use ray tracing, otherwise there is a large visual gap; A100 is not RTX GPU so it's a lot slower than RTX GPUs.
Repository: simpler-env/SimplerEnv
Issue: Will you release more environments?

Dear researchers,
Thanks so much for your hard work. I can imagine the huge engineering efforts required here.
May I ask if there are any plans to replicate more environments?

Currently there are some efforts to build more environments like the Track 2 of ICML-MFM-EAI challenge (https://icml-mfm-eai.github.io/challenges/#TRACK2) which will use real-to-sim environments based on ManiSkill3 for evaluation; for our environments @StoneT2000 is migrating them to ManiSkill3 first so it will take a while.

That's good to know. Thanks. I am closing the issue.
Repository: simpler-env/SimplerEnv
Issue: Choosing camera sizes
Root cause: Documentation Deficiency

How to properly specify the image dimensions that should be returned from the simulation? E.g. Octo requires an image dimension of `(256, 256)` and performs manual resizing in `OctoInference` using `tf.image.resize()` with `method="lanczos3"`. 

Since the environments will by default return images of size `(512, 640)`, doesn't the distortion effect (implied by the aspect ratio change to `(256, 256)`) negatively impact methods' performances?

What is the intended way to use your `env.make()` while specifying image dimensions?

I recommend do this outside of env. See rt1 / octo inference code for examples. As long as the resize method is consistent w/ the original method for model training it won't affect performance.
Repository: simpler-env/SimplerEnv
Issue: Can simpler continue from previous results
Root cause: Unimplemented Feature Gap

since evaluate google robot takes 16 hours for me , I wonder whether it could continue from previous saved mp4 instead of generate all mp4 from scratch. 

If you used the scripts here https://github.com/simpler-env/SimplerEnv/tree/main/scripts  you can continue evaluating the rest of the scripts. I didn't implement autocontinue.
Repository: simpler-env/SimplerEnv
Issue: Coordinate Representations: SimplerEnv vs. Libero
Root cause: Documentation Deficiency

If I use the postprocessing from this repo to run the Libero benchmark, the performance is much worse. I get that visuals are much more realistic here compared to Libero and that there is a difference between the low-level controllers.

However, I am still confused why all publications report successes around 80% and I barely get any observation right. 

My Question:
Does Simpler use the same coordinate representations as LIBERO?
1. Angles are initially returned in Euler angles and then postprocessed to `axangle`. Is this the same convention that Libero uses?
2. Do they use the same reference frames (i.e. the robot base frame)?
3. Do both frameworks use a delta for xyz/angles or does libero use some absolute positioning?

I appreciate the work you put into this project!

This repo does not use any part from, nor does it have any relationship with, LIBERO. This repo uses maniskill envs and controller name for each robot is at https://github.com/simpler-env/SimplerEnv/blob/main/simpler_env/utils/env/env_builder.py#L20  . Controller for each robot can be found in https://github.com/simpler-env/ManiSkill2_real2sim/tree/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/agents/configs .
Repository: simpler-env/SimplerEnv
Issue: Could you add support for folks without a NVIDIA GPU? e.g. MuJoCo on macOS
Root cause: Unimplemented Feature Gap

Hi, I'm looking to use SimplerEnv in [my fork of PickAgent](https://github.com/tjadams/PickAgent) so I can run OpenVLA in simulation, but I don't have a NVIDIA GPU. Is it possible to add support for MuJoCo? Then I can run SimplerEnv with apple silicon.

The environments are based on public SAPIEN 2 which is based on physx. Unfortunately NVIDIA GPU is required. We don't plan to add mujoco support (due to time constraints).
Repository: simpler-env/SimplerEnv
Issue: optimization when lack of CPU
Root cause: Configuration Error

```
top - 15:41:43 up 2 days,  6:18,  8 users,  load average: 148.76, 159.89, 162.12
Tasks: 1130 total,   2 running, 1128 sleeping,   0 stopped,   0 zombie
%Cpu(s): 98.3 us,  1.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 257602.6 total,  95593.6 free,  27249.9 used, 134759.2 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used. 228307.0 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                          
3439911      20   0   39.7g   3.9g   1.0g R  1302   1.6  23:52.89 pt_main_thread                                                                                                                   
3412216     20   0   40.0g   4.0g   1.0g S  1072   1.6 131:49.12 pt_main_thread                                                                                                                   
3372667     20   0   40.6g   4.7g   1.0g S 162.7   1.9 290:17.24 pt_main_thread                                                                                                                   
3444674       20   0 9892.2m 134088  37632 S 121.9   0.1   0:03.73 ffmpeg             
```

I launched 4 Google robot evaluations using run_openvla.sh to evaluate 4 checkpoints simultaneously.
However, my 96-core machine is running out of CPU resources.
Do you have any suggestions for improving performance? Thank you!

Looks like your model is running on cpu instead of gpu. If it's using cpu, it will consume many cpu resources (and the env itself is slow). Pay attention to the tensorflow / jax info printed out in command line.
Repository: simpler-env/SimplerEnv
Issue: Interface for Default Frame-Wise Patch Annotations in SImplerEnv

Dear Authors,

Hello! Could you please let me know if SImplerEnv provides an interface that, for each frame, automatically annotates which patches contain the target object and the robotic gripper? I look forward to your reply!

Could you elaborate what "patches" is?

@xuanlinli17 Sorry, I apologize for not explaining myself clearly. What I mean is: in the full image, will there be a bounding box or boundary around the regions containing the target object and the robot that can be directly viewed or accessed?

Then you need to render the segmentation mask in the camera. You need to modify the camera config in robot agent to include segmentation https://github.com/haosulab/ManiSkill/blob/493be36121a9dd06071a57172274babe617b789f/mani_skill2/sensors/camera.py#L118

thank you for patiently answering my questions!
Repository: simpler-env/SimplerEnv
Issue: About end-effector position API

Hi, i am new to simulation environment, can i ask you that: to get the eef_pos of robot, do `obs["agent"]["eef_pos"]` (and `env.unwrapped.agent.robot.get_qpos()` return the same outputs? 
Thank you so much!

No they don't;    end-effector 6d pose (xyz, quat) can be obtained via `env.unwrapped.agent.robot.tcp.pose`;  `get_qpos` is robot's joint qpos (dim = # robot joints) instead of end-effector 6d pose.

Thank you for your quick response, 
I have tried `env.unwrapped.agent.robot.tcp.pose` but i receive an error: `AttributeError: 'sapien.core.pysapien.Articulation' object has no attribute 'tcp'`. Can you suggest what is wrong here?

Sorry; see https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/examples/demo_manual_control_custom_envs.py#L487

`env.tcp.pose`

Thank you, i will try it