zhiminy/EvalEng huggingface dataset:
Repository: simpler-env/SimplerEnv
Issue: Strange view from env.reset()
Root cause: Algorithmic Error

Hi,

When I create an environment simply with `env = simpler_env.make("google_robot_pick_coke_can", render_mode="rgb_array")`, I get a normal view from above the table like the first image with

```python
obs, _ = env.reset()
obs = get_image_from_maniskill2_obs_dict(env, obs)
```

but with `env.render()`, I get the second strange view.

Is that an expected behavior, and how can I render from the same angle as `env.reset()`?

Thanks.

<img width="640" height="512" alt="Image" src="https://github.com/user-attachments/assets/0c5edb1c-024a-4d35-9452-31e854d87a01" />

<img width="512" height="512" alt="Image" src="https://github.com/user-attachments/assets/8a9e4534-2aec-4cbd-953a-bb1773f06679" />

This is unexpected as the camera should look at the table even if w/o background overlay; so idk what's happening. You can checkout the sapien viewer.

Thanks for the reply.

By sapien viewer you mean the interactive one by setting `render_mode="human"`, correct?
But what about when I want to just simply log evaluation videos during training? Am I wrong that we need `render_mode="rgb_array"` for that purpose.

Trying out other environments but having the same issue.

As before, the code to reproduce is simply:
```python
env = simpler_env.make("widowx_spoon_on_towel", render_mode="rgb_array")
obs, reset_info = env.reset()
mediapy.show_image(env.render())
```

If nobody can reproduce this, I think I will redo my installation. Or might do some investigations into the source code...
Thanks.

<img width="637" height="478" alt="Image" src="https://github.com/user-attachments/assets/97e21b8f-5438-4acf-9666-7efeb9b38d45" />

<img width="511" height="510" alt="Image" src="https://github.com/user-attachments/assets/bb762e1d-1a9d-4c74-9846-ab8f8a55485b" />
Repository: simpler-env/SimplerEnv
Issue: Compatibility Issue on Jetson: SAPIEN 2.2.2 Unsupported and Errors with SAPIEN 3.x
Root cause: External Dependency Breakage

Hello,

First of all, thank you for the great work on this project.

I’m currently trying to run the SIMPLER benchmark on a Jetson environment.
Most of the required libraries work fine, **but there is a problem with SAPIEN on Jetson, only SAPIEN versions 3.0 and above are supported**, 
so I cannot install the 2.2.2 version required by the current SIMPLER benchmark.

If I proceed with `SAPIEN==3.0.0`, I encounter the following error:

```
AttributeError: module 'sapien.core' has no attribute 'Actor'
```


I would like to kindly ask if there is any recommended way to resolve this issue or any workaround you could suggest.

Hi,

This repo is only compatible with sapien 2 and doesn't support SAPIEN 3.x. Please visit ManiSkill3 for the sapien3 replica of Simpler.

Thank you for your fast reply :)

However, it seems like Maniskill3 doesn't have the Google Robot evaluations 😢 (It only has evaluations from Bridgedata v2)
I think I should find another way to solve this problem...
Repository: simpler-env/SimplerEnv
Issue: Different results of configuration evaluation during simulation
Root cause: Documentation Deficiency

Hello! 
Thank you very much for your work.

I run the script rt1x_bridge.sh and my results don't match yours. 
My results: 
-     1 experiment: 
1.         PutCarrotOnPlateInScene-v0; Average success: 0.08333333333333333
2.         StackGreenCubeOnYellowCubeBakedTexInScene-v0; Average success: 0.0
3.         PutSpoonOnTableClothInScene-v0; Average success: 0.041666666666666664
4.         PutEggplantInBasketScene-v0; Average success: 0.0
-     2 experiment:
1.         PutCarrotOnPlateInScene-v0; Average success: 0.041666666666666664
2.         StackGreenCubeOnYellowCubeBakedTexInScene-v0; Average success: 0.0
3.         PutSpoonOnTableClothInScene-v0; Average success: 0.041666666666666664
4.         PutEggplantInBasketScene-v0; Average success: 0.0

Your results(from TABLE V of your paper):
        Put Carrot on Plate; Success: 0.042
        Stack Green Block on Yellow Block; Success: 0.000
        Put Spoon on Towel; Success: 0.000
        Put Eggplant in Yellow Basket; Success: 0.000


Questions:
1) Why might my results differ from yours?
2) Why do the names of the indicators differ? ( Put Spoon on Towel / PutSpoonOnTable)



There are some variance. Typically on the Bridge envs one needs to evaluate for at least 75 trials (repeat the evaluation 3 times) and take the average (but rt-1x's output is deterministic so idk why, probably because simulation itself is not deterministic). Default instructions in Simplerenv match the corresponding evals in the real world.

ok, thanks for your answer
Repository: simpler-env/SimplerEnv
Issue: Pick coke can variants.
Root cause: Documentation Deficiency

I saw in your eval scripts for pick coke can: [`declare -a coke_can_options_arr=("lr_switch=True" "upright=True" "laid_vertically=True")`](https://github.com/simpler-env/SimplerEnv/blob/bcf62960dc9d95501cf36dc8c64d2128e9b10e09/scripts/rt1_pick_coke_can_visual_matching.sh#L11).

In Table IV in your paper, there's `Horizontal Laying`, `Vertical Laying`, `Standing` three variants. Does `lr_switch=True` correspond to the `Horizontal Laying` variant? If so, does that mean you didn't evaluate the default setting where `"lr_switch=False, upright=False, laid_vertically=False"`?

"lr_switch" is coke can oriented by 180 degrees to match real eval's "horizontally laying" setting.

If you set "lr_switch=False, upright=False, laid_vertically=False", then the coke can will be oriented randomly. There is no "default setting"; you can evaluate policies however you want. The purpose of our paper's experiments is to draw correlation with real-world evaluations under a similar setup, and 3 default orientations (horizontal / vertical laying, standing) with a "grid sweep" over coke can positions already covers many variations.

Thanks for the clarification!

Hello @xuanlinli17, I wonder is it possible to change the position of the Coke can in the simulator?

For xy positions, yes (see example scripts)

Thank you! I'll have a look!
Repository: simpler-env/SimplerEnv
Issue: Eval code for RT-2-X?
Root cause: Unimplemented Feature Gap

Hi, I saw that you have RT-2-X results in the paper. Do you plan on releasing the code to evaluate RT-2-X on SimplerEnv?

RT-2-X has a very similar evaluation code as RT-1(-X) but requires an API key to access the model. Thus, until Google publicly releases the RT-2-X API, I can't release the code yet.

Got it!

Hi,how did you get RT-2-X API key?I want to use RT-2-X,but I can't find any model in internet.

Unfortunately I don't think Google decides to publicly release it yet...
Repository: simpler-env/SimplerEnv
Issue: Octo behaves strange
Root cause: Documentation Deficiency

Hi, thanks for the great work! 

I'm using your work to try to reproduce and analyze octo and openvla.

I'm not fully familiar with the code yet, and I just ran through the provided octo test script. openvla can behave normally, although the success rate is not very high. But the octo performance feels like an error in the output mapping to the simulation engine. Noted that I have not made any changes to the code yet.

I would like to ask where to start checking this problem?

Thanks.


https://github.com/user-attachments/assets/cb437665-8c62-4281-a951-5c7ad86a4ea1



Octo behavior is normal. It staggers very frequently under (unseen) simulation scene backgrounds. If you put a real background that matches the training, then its behavior is better

I tested different backgrounds and different settings and found that Octo's performance was very different. Thanks for your guidance.

@Udntknwme How do you test different backgrounds?
Repository: simpler-env/SimplerEnv
Issue: RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed
Root cause: Environment Incompatibility

```
[2024-08-02 11:36:11.730] [svulkan2] [error] GLFW error: X11: Failed to open display 
[2024-08-02 11:36:11.731] [svulkan2] [warning] Continue without GLFW.
Traceback (most recent call last):
  File "/home/user/SimplerEnv/simpler_env/main_inference.py", line 60, in <module>
    success_arr = maniskill2_evaluator(model, args)
  File "/home/user/SimplerEnv/simpler_env/evaluation/maniskill2_evaluator.py", line 215, in maniskill2_evaluator
    success_arr.append(run_maniskill2_eval_single_episode(obj_episode_id=obj_episode_id, **kwargs))
  File "/home/user/SimplerEnv/simpler_env/evaluation/maniskill2_evaluator.py", line 60, in run_maniskill2_eval_single_episode
    env = build_maniskill2_env(
  File "/home/user/SimplerEnv/simpler_env/utils/env/env_builder.py", line 16, in build_maniskill2_env
    env = gym.make(env_name, **kwargs)
  File "/home/user/cache/conda/envs/simpler_env/lib/python3.10/site-packages/gymnasium/envs/registration.py", line 802, in make
    env = env_creator(**env_spec_kwargs)
  File "/home/user/SimplerEnv/ManiSkill2_real2sim/mani_skill2_real2sim/utils/registration.py", line 92, in make
    env = env_spec.make(**kwargs)
  File "/home/user/SimplerEnv/ManiSkill2_real2sim/mani_skill2_real2sim/utils/registration.py", line 34, in make
    return self.cls(**_kwargs)
  File "/home/user/SimplerEnv/ManiSkill2_real2sim/mani_skill2_real2sim/envs/custom_scenes/put_on_in_scene.py", line 310, in __init__
    super().__init__(
  File "/home/user/SimplerEnv/ManiSkill2_real2sim/mani_skill2_real2sim/envs/custom_scenes/put_on_in_scene.py", line 162, in __init__
    super().__init__(**kwargs)
  File "/home/user/SimplerEnv/ManiSkill2_real2sim/mani_skill2_real2sim/envs/custom_scenes/move_near_in_scene.py", line 58, in __init__
    super().__init__(**kwargs)
  File "/home/user/SimplerEnv/ManiSkill2_real2sim/mani_skill2_real2sim/envs/custom_scenes/base_env.py", line 134, in __init__
    super().__init__(**kwargs)
  File "/home/user/SimplerEnv/ManiSkill2_real2sim/mani_skill2_real2sim/envs/sapien_env.py", line 107, in __init__
    self._renderer = sapien.SapienRenderer(**renderer_kwargs)
RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed
```

I am meeting this problem when trying to run octo_bridge.sh script. How can I resolve this problem? 

Is a display required?

I am using H100. Is it not supported?

Try the link in troubleshooting section and ensure 3 json files exist. Ensure numpy < 2.0. Display is not required.

If above doesn't work, you can create a fake display using `sudo X :0 &; export DISPLAY=:0` or sth like `xvfb-run -a {script}`.
Repository: simpler-env/SimplerEnv
Issue: What's the difference between simplerenv and maniskill?


SIMPLER environments use the SAPIEN simulator and the ManiSkill2 base environment. The individual environments inherit the ManiSkill2 base environment (and thus follow ManiSkill2's convention of controller, robot setup, etc). The SIMPLER environments (and their assets) are not contained in the original ManiSkill2 suite of tasks.

@StoneT2000 is migrating the environment to SAPIEN 3 & ManiSkill3. The new environments will support GPU-parallelization.

@xuanlinli17 is spot on here. 

There is a larger project in ManiSkill3 that is ongoing and a bit delayed atm to add more digital twins for evaluation and/or training. The main use case for ManiSkill 3 and SIMPLER integration is faster evaluation as ManiSkill 3 generates visual data far faster in parallel compared to ManiSkill 2 / other simulators.
Repository: simpler-env/SimplerEnv
Issue: what are the differences between the different urdfs?

https://github.com/simpler-env/SimplerEnv/blob/2a0389226175853baeaa76d53ca0e3a48eaf238d/scripts/octo_move_near_visual_matching.sh#L17

They are differently colored (using gnu image manipulation bucket paint tool) based on different real-world images from different tasks to reflect the variations in arm color during manipulation.
Repository: simpler-env/SimplerEnv
Issue: Does SimplerEnv support fine-tuning evaluation?
Root cause: Documentation Deficiency

Many real-world evaluations of generalist robotic models require fine-tuning on data from target settings due to differences in action space, coordinate frames, and other contextual factors. Without such fine-tuning, models often fail when no data from similar settings are present. Evaluating models in this manner is crucial as it demonstrates the effectiveness of the pretrained knowledge they have acquired. Does SimplerEnv support fine-tuning on data from specific target settings for such evaluations?

SIMPLER is agnostic to the model it evaluates (you can evaluate any policy you want to). Though, our current project is only about policy evaluation and not about policy finetuning.

How can I fine-tune for SimpleEnv? How to collect BC data from the simulation environment? Any guide? 

I suspect fine-tuning on simulated environment will likely give very good result as simulated environment should be much easier to model, thus rendering such evaluation meaningless. Do you think there is still any benefit of doing such evaluation?

Simpler env serves as “test set” for robot manipulation  akin to supervised learning. You shouldn’t train on test set or the test set will be meaningless.

Hmmm, I am new to this field, but based on this https://github.com/openvla/openvla/issues/19 and this https://github.com/openvla/openvla/issues/17, model will not work out of the box if similar data are not part of the training mix. What am I missing here? 

OpenVLA seems to be performing badly on SimplerEnv right now.

I think Google Robot and Bridge are in their training set, so you can evaluate OpenVLA immediately on Simpler (though I'm not sure if current OpenVLA's Bridge implementation is correct).

If you want to finetune OpenVLA, use rollouts from the Open-X-Embodiment dataset to train your model; don't use rollouts from Simpler as they are meant for "test set".
Repository: simpler-env/SimplerEnv
Issue: [svulkan2] [error] GLFW error: X11: Failed to open display
Root cause: Documentation Deficiency

hi,I met this error when I test :
python simpler_env/simple_inference_visual_matching_prepackaged_envs.py
Maybe I can't see the operation window because of this ,how can I solve this ,and also the warning:
[svulkan2] [warning] Continue without GLFW.

I solve this error to set  os.environ["DISPLAY"] = ":1" in the py file,but I still can't see the whole progress in a display window,is it normal or is there something else I don't set correctly.


This is normal. You can ignore the warning.

> This is normal. You can ignore the warning.

I mean that after running the program, a window should appear to display the robot's running process,but I don't see any window.Is it normal or the program is just like this with no window displaying

The script does not display a window. Instead, evaluation videos are saved. Please see the script for more details.
Repository: simpler-env/SimplerEnv
Issue: How to adjust the camera angle in the output video to show more of the robotic arm?
Root cause: Documentation Deficiency

Hi,
I want to camera angle in the output video to show more of the robotic arm, but dont want to change the input of inference model, how to do it?

For visual matching evaluation, unfortunately this is not possible to do (since we overlay the real background into simulated observations)

For variant aggregation evaluation, you can render images from a 3rd camera view. You can adjust the "human" camera config in the code.

> For visual matching evaluation, unfortunately this is not possible to do (since we overlay the real background into simulated observations)
> 
> For variant aggregation evaluation, you can render images from a 3rd camera view. You can adjust the "human" camera config in the code.

thank you for your reply xuanlin, 

I think what you mean is script like `scripts/octo_pick_coke_can_variant_agg.sh` ?
I didn't find option to adjust the camera config, so this camera config is hard code in some file?

Yes.

You can add cameras to the robot config file like https://github.com/simpler-env/ManiSkill2_real2sim/blob/cd45dd27dc6bb26d048cb6570cdab4e3f935cc37/mani_skill2_real2sim/agents/configs/google_robot/defaults.py

> Yes.
> 
> You can add cameras to the robot config file like https://github.com/simpler-env/ManiSkill2_real2sim/blob/cd45dd27dc6bb26d048cb6570cdab4e3f935cc37/mani_skill2_real2sim/agents/configs/google_robot/defaults.py

ok, I will try this. thank you
Repository: simpler-env/SimplerEnv
Issue: Installation
Root cause: Documentation Deficiency

Hello,
The [full installation instructions](https://github.com/simpler-env/SimplerEnv#full-installation-rt-1-and-octo-inference-env-building) have the following lines:

"""
pip install tensorflow==2.15.0
pip install -r requirements_full_install.txt
pip install tensorflow[and-cuda]==2.15.1 # tensorflow gpu support
"""

I don't understand why first installing tensorflow 2.15.0 and then 2.15.1 with GPU support, is the last line not sufficient? Could someone please explain this, thank you!

It's a hack to get around the weird tensorflow install dependencies; some requirements in `requirements_full_install.txt` requires an existing installed tensorflow. You also can't directly install `tensorflow[and-cuda]==2.15.1` before installing from `requirements_full_install.txt` as some dependenicies for tensorflow will be broken.

As Xuanlinli said, The reason is `dlimp 0.0.1 requires tensorflow==2.15.0, but you have tensorflow 2.15.1 which is incompatible.`
Repository: simpler-env/SimplerEnv
Issue: Issue with 'RuntimeWarning: Mean of empty slice.'
Root cause: Documentation Deficiency

Hello guys, 

I encountered a strange problem when I tried to run **calc_metrics_evaluation_videos.py** to see the results of the evaluation. When I run this script I always get this message '_RuntimeWarning: Mean of empty slice._', followed by a lot of warning, like the one behind the arrow ==> **WARNING: avg_sim_success is nan** and balabala. 
My models are all downloaded followed by instructions and in the correct path, but I get these warnings regardless of which model, and the result is like 'vertical visual matching sim success {'rt-2-x': nan, 'rt-1-converged': nan, 'rt-1-15pct': nan, 'rt-1-x': nan, 'rt-1-begin': nan, 'octo-base': nan}', all empty value. 
Has anyone experienced a similar problem? If it's convenient, could you share your solution?

In the end, thanks for the amazing tools and the wisdom of the authors.

I think it's because the videos aren't placed in the correct path? please check the script for more details.

> I think it's because the videos aren't placed in the correct path? please check the script for more details.

Hi Xuanlin,

Thank you very much for your answer, it makes me clear a lot! 
I now have the video generated by **simple_inference_visual_matching_prepackaged_envs.py** and correct path of the video. But **calc_metrics_evaluation_videos.py** still has the same problem, giving me a _WARNING: avg_sim_success is nan_ . 
I'm a little confused about what is the correct input for **calc_metrics_evaluation_video.py**?

Oh, for `simple_inference_visual_matching_prepackaged_envs.py`, you can directly get the results from the reported success rates in the terminal. The `calc_metrics_evaluation_videos.py` is, on the other hand, used to reproduce the paper's evaluation results (i.e., correlation between sim & real using identical task, object pose, and robot pose setup) after you running all of `scripts/*` (or if you download all evaluation videos from huggingface and unzip them). It doesn't apply to the results of `simple_inference_visual_matching_prepackaged_envs.py`

Hi Xuanlin, 

Thank you very much for your clear explanation! 
And thank you for your patience and excellent work.
Repository: simpler-env/SimplerEnv
Issue: something about parameters

The command is like:
python simpler_env/main_inference.py --policy-model ${policy_model} --ckpt-path ${ckpt_path} \
  --robot ${robot} --policy-setup widowx_bridge \
  --control-freq 5 --sim-freq 500 --max-episode-steps 60 \
  --env-name PutCarrotOnPlateInScene-v0 --scene-name ${scene_name} \
  --rgb-overlay-path ${rgb_overlay_path} \
  --robot-init-x ${robot_init_x} ${robot_init_x} 1 --robot-init-y ${robot_init_y} ${robot_init_y} 1 --obj-variation-mode episode --obj-episode-range 0 24 \
  --robot-init-rot-quat-center 0 0 0 1 --robot-init-rot-rpy-range 0 0 1 0 0 1 0 0 1;

I want to know what does "robot-init-x,robot-init-y,obj-variation-mode,obj-episode-range,robot-init-rot-quat-center,robot-init-rot-rpy-range"    means

`robot-init-x,robot-init-y,robot-init-rot-quat-center,robot-init-rot-rpy-range`

These are used to initiate a gridded sweep of robot poses.

`obj-variation-mode,obj-episode-range`

"obj-variation-mode" can be `xy` or `episode`; `xy` means sweeping evaluations using object xy locations (for single object task like pick coke can; not in the script you posted). `episode` means sweeping evaluations using episode ids, each episode id corresponds to some predefined object configurations (this applies when a task has >1 objects, e.g., move near)
Repository: simpler-env/SimplerEnv
Issue: NameError: name 'OctoInference' is not defined.
Root cause: External Dependency Breakage

I have download the octo files according to Octo Inference Setup,why still no octo model found

by the way I am running command:
python simpler_env/main_inference.py --policy-model octo-small --ckpt-path None --robot widowx --policy-setup widowx_bridge   --control-freq 5 --sim-freq 500 --max-episode-steps 60   --env-name StackGreenCubeOnYellowCubeBakedTexInScene-v0 --scene-name bridge_table_1_v1 --rgb-overlay-path ManiSkill2_real2sim/data/real_inpainting/bridge_real_eval_1.png  --robot-init-x 0.147 0.147 1 --robot-init-y 0.028 0.028 1 --obj-variation-mode episode --obj-episode-range 0 24   --robot-init-rot-quat-center 0 0 0 1 --robot-init-rot-rpy-range 0 0 1 0 0 1 0 0 1 --additional-env-save-tags octo_init_rng_0


And the folder is like:
SimplerEnv-main-octo-docs
SimplerEnv-main-octo-examples
SimplerEnv-main-simpler_env-main_inference.py
SimplerEnv-main-octo-octo-model-octo_model.py,this is the py file quoted by the "octo_model" py file which is quoted by the main_inference.py:
try:
    from simpler_env.policies.octo.octo_model import OctoInference
except ImportError as e:
    print("Octo is not correctly imported.")
    print(e)


I think you forgot to `cd octo/; pip install -e .`

Please follow the instruciton in the Octo inference setup in README (it also requires installing jax)

> I think you forgot to `cd octo/; pip install -e .`
> 
> Please follow the instruciton in the Octo inference setup in README (it also requires installing jax)

I am sure I have done this all,the "pip install -e ." result is:
Successfully built octo
Installing collected packages: octo
Successfully installed octo-0.0.0

but still the error


Did you solve the issue already? I am facing the same issue.

It is because the jax version is too new. 

Installing the jax[cuda11_pip]==0.4.20 works.

pip install --upgrade "jax[cuda11_pip]==0.4.20" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html