zhiminy/EvalEng huggingface dataset:
Repository: simpler-env/SimplerEnv
Issue: Any way to get expert demos for each task?
Root cause: Documentation Deficiency

Hi, 

Thanks for open-sourcing this brilliant benchmark! I was wondering that is there any quick way to get expert demos for each task in Simpler? Like a script oracle policy, or a RL policy trained on each env?

I didn't find anything like these in the codebase. But I was wondering if the authors may have run some relevant experiments before and could help share some insights on this? Like how good can a RL policy perform in each env? Thanks!

The "expert demos" are from the open-x-embodiment dataset.

Sorry for the confusion. I mean expert demos for each Simpler env in simulation. I guess directly running a open-x-embodiment action sequence in simpler would just fail due to dynamics mismatch?

Open-loop unrolling demo trajectories from open-x-embodiment will most likely be successful, since simpler-env already performs sysid to match the dynamics gap between sim & real.

Thanks for your reply! If I want to run a OXE expert trajectory in a open-loop way in the corresponding `simpler` environment, how should I setup and reset the environment to match with those in the expert trajectory? Thanks!

For existing environments in Simpler, you just need to take the trajectories with the same language instruction from the demo dataset and adjust the initial object pose to match the demo's first frame. You can then open-loop unroll the demo trajectory.

For environments not already present in this repo, you need to create new environment, which means you need the assets, a proxy simulation env, and a frame from demos to overlay the sim observation (with robot arm removed); you can generate assets via https://www.sudo.ai/ 

I see. Thanks! I wanted to run demos for existing `simpler` environments. I'll try with the first approach you suggest. Thanks!

Hi @xuanlinli17 , thanks a lot for your help!

About "adjust the initial object pose to match the demo's first frame" in your previous reply, I was wondering that how can I achieve this adjustment? I checked the OXE data and there seems to be no information about the demo's initial pose. Or should I do this manually by checking their visual appearance by myself? Could you help provide a simple example code of how to do the adjustment here? Thanks!

You have to manually tune the object pose to match demo's first frame. You can use the nvdiffrast utilities in https://github.com/Jiayuan-Gu/GeTex (except baking texture).

Hi! @MasterXiong , I wonder if you ever succeeded at creating a dataset like what you described, and if so, could you share it?
Repository: simpler-env/SimplerEnv
Issue: Support for Droid dataset and evaluation
Root cause: Unimplemented Feature Gap

Hi,

I am wondering whether you can integrate the robotic setup (Franka panda), and camera setup (2 overhead and 1 wrist camera) for the Droid dataset, since it is currently the largest one open-sourced robotic dataset?

Unfortunately I don't have bandwidth for adding it; the visual matching setup also has the limitation of applying only to fixed camera setups (instead of dynamics camera setups like wrist cameras), though variant aggregation evaluations do not suffer from this limitation.

Got It. 

> Unfortunately I don't have bandwidth for adding it; the visual matching setup also has the limitation of applying only to fixed camera setups (instead of dynamics camera setups like wrist cameras), though variant aggregation evaluations do not suffer from this limitation.

I wonder whether the  bridge env setup also has the limitation of applying only to fixed camera setups (instead of dynamics camera setups like wrist cameras) ? 

Yes, it's currently fixed camera setup.

> Yes, it's currently fixed camera setup.

Thank you. What is the reason? Is it because physical setting or software? 
Is it possible for me to open a PR to add **wrist cameras** in bridge ?  If it is possible, how many coding time it may need. 

The wrist camera moves which breaks visual matching setup. The variant aggregation setup is fine w/ wrist camera.
You can modify the robot agent. It should be fast.
Repository: simpler-env/SimplerEnv
Issue: [Errno 2] No such file or directory
Root cause: Environment Incompatibility

when I run 'bash openvla_drawer_variant_agg.sh',it met error:
![Screenshot from 2024-08-24 11-14-44](https://github.com/user-attachments/assets/8df2b37f-7407-4c3b-babf-22b1893d207c)


Please run the script in the root directory of this repository.
Repository: simpler-env/SimplerEnv
Issue: RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed
Root cause: Environment Incompatibility

I run  python simpler_env/simple_inference_visual_matching_prepackaged_envs.py --policy openvla/openvla-7b \
        --ckpt-path None --task google_robot_move_near_v1  --logging-root ./results_simple_eval/  --n-trajs 10
I checked the troubleshooting section and followed the link's instructions,but it still didn't work.
Meet the error : RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed
GPU : RTX3090
Ubuntu22.04

I added a josn file, this error is gone, but I do not see the robot simulation screen

No screen should appear. Evaluation videos should be automatically recorded. You can find them at `results/`


    self._renderer = sapien.SapienRenderer(**renderer_kwargs)
RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed

hi， I meet same problem, how to solve ,then the program can run normally? IF the program can't run normally, how can i get evalution videos,thank you!

Solved it by following this [doc](https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/installation.html)
Repository: simpler-env/SimplerEnv
Issue: Migrate example environments to ManiSkill 3
Root cause: Unimplemented Feature Gap

Intention is to replace the currently git submoduled enviroments with the new GPU accelerated ManiSkill 3 package and some better packaged code designed to be easily extended for user's custom real2sim environments.

Hi @StoneT2000 @xuanlinli17 is there an update on this? I'm trying to use visual matching with ManiSkill 3, and I was wondering if you had any examples?


See branch https://github.com/simpler-env/SimplerEnv/tree/maniskill3 for how to evaluate some baselines on the gpu parallelized rendering envs. We only support 4 of the envs at the moment on the GPU version.

Migrating every functionality of simpler will take a while so we only migrate the base environments / default settings for reporting success rates.

Migrating the google robot environments currently is on hold due to the work required to replicate the controller design for the GPU sim used in the CPU sim for the google robot specifically.
Repository: simpler-env/SimplerEnv
Issue: How to run the environment on CPU
Root cause: Documentation Deficiency

Hi, 

I was wondering that how can I run the environment on CPU? As the default config seems to be running on GPU, but I found it hard to parallelize the environments with GPU ([link](https://github.com/simpler-env/SimplerEnv/issues/36#issue-2502007201)). Not sure if this issue can be solved by simulating on CPU instead? Thanks!

Simpler is based on ManiSkill2 and rendering requires a GPU. 
However, ManiSkill3 will support both GPU parallelization and CPU rendering. @StoneT2000  is currently migrating the environments.

Thanks! Just to confirm, so there is no way to run multiple environments with the current version based on ManiSkill2 right?

You can run multiple envs, but it's a bit tricky; however you can always split your eval script into multiple scripts (but you need to load the networks multiple times in this case).

Thanks! Can you help give some more details on how to implement this tricky way to run multiple envs? I'm trying to accelerate my evaluation pipeline by feeding a large batch of observations to a model insteading of processing only one env observation at a time. 

run many different bash scripts, each like seed 0-1000, seed 1000-2000, etc
Repository: simpler-env/SimplerEnv
Issue: How to specify which objects to manipulate in pick and move tasks
Root cause: Documentation Deficiency

Hi, 

I was wondering how can I specify which object(s) to manipulate in the `pick objects` and `move A to B` tasks? Based on my understanding of the code, the candidate set of possible objects are defined like [here](https://github.com/simpler-env/ManiSkill2_real2sim/blob/cd45dd27dc6bb26d048cb6570cdab4e3f935cc37/mani_skill2_real2sim/envs/custom_scenes/move_near_in_scene.py#L611). Then I should change the `options` input to the `reset` function to specify which object(s) to use in the environment. Is my understanding correct? Thanks!

The two envs have different initialization mechanisms. `move A to B` uses episode-based randomization, so you can do so by passing in `episode_id` like https://github.com/simpler-env/SimplerEnv/blob/0b13b698138576c449f8853c99b04202e3c04952/simpler_env/evaluation/maniskill2_evaluator.py#L82 . For `pick objects` , you can do so by passing in different seeds in the options

Thanks a lot! A further question is that it seems that only a subset of objects or triplets of objects are used for the pick and move tasks (like [here](https://github.com/simpler-env/ManiSkill2_real2sim/blob/cd45dd27dc6bb26d048cb6570cdab4e3f935cc37/mani_skill2_real2sim/envs/custom_scenes/grasp_single_in_scene.py#L594) and [here](https://github.com/simpler-env/ManiSkill2_real2sim/blob/cd45dd27dc6bb26d048cb6570cdab4e3f935cc37/mani_skill2_real2sim/envs/custom_scenes/move_near_in_scene.py#L613)). I was wondering that how are these task configurations chosen? Are they chosen to reflect the objects that are seen in the OXE demos? If that's the case, can I add more configurations to test out-of-distribution generalization to unseen objects or triplets in OXE? Thanks!

These are not necessarily seen by OXE demos; they are just randomly chosen.

Thanks! Where can I find a full list of all the possible objects supported in `simpler` for picking and moving tasks then?

Checkout these json files at https://github.com/simpler-env/ManiSkill2_real2sim/tree/cd45dd27dc6bb26d048cb6570cdab4e3f935cc37/data/custom which are loaded in the envs
Repository: simpler-env/SimplerEnv
Issue: Problems with environment rendering, and speed
Root cause: Documentation Deficiency

Hello, thank you for providing this project! 

I have been using this environment to train SAC and PPO agents and have noticed some odd interactions with the environment like the one attached below. 

![simpler_bug](https://github.com/simpler-env/SimplerEnv/assets/70121156/568e9716-8556-4294-81aa-dfc993498819)

I have also noticed that in my case, the environment runs at about 12 FPS whereas maniskill can achieve roughly 350 FPS. Is this normal? Looking forward to your reply!

Hi,

The "odd interactions" are due to the robot arm bumping into the robot camera link and causes the camera link's pose to change. Since we overlay a **fixed** real-world background image onto the simulation observation, i.e., only the foreground robot and objects are rendered, and the background image will remain the same no matter what the robot's camera pose is, this causes "weird observations" in the video.

For the environment speed, we use 500hz simulation frequency (i.e., # simulation steps / second) and 3hz robot control (action) frequency for the google robot tasks. If I understand correctly, the 12fps you refer to is the number of robot actions / second. Note that the 3hz control frequency is a lot lower than the commonly-used ones e.g., 20hz when benchmarking simulation speed. Also, the rendered image's resolution is much higher (640x512 for google robot) compared to simulation speed tests that render 128x128 image. Thus simulation "slow down" is normal.

Though, once the ManiSkill3 environment is ready, the simulation speed will be much faster and also with GPU parallelizations.
Repository: simpler-env/SimplerEnv
Issue: How to convert RT-1-X action to Octo action
Root cause: Documentation Deficiency

Hi, 

I wanted to use successful episodes collected by RT-1-X in the google robot envs as demos to fine-tune an Octo model. I was wondering that how could I convert the raw action generated by RT-1-X to the corresponding raw action of Octo? As I note that the inference function for the two models are different in `simpler` code, which may indicate that they do not use the same action space and some coordinates transformation is required here? Thanks!

Just compare the 2 inference code. I think only the gripper is different. 

Thanks! But I noticed that for the actions dimensions of orientations, RT-1 uses axis angle representation for google robot tasks, and roll, pitch, yaw representation for widowx tasks ([link](https://github.com/simpler-env/SimplerEnv/blob/0b13b698138576c449f8853c99b04202e3c04952/simpler_env/policies/rt1/rt1_model.py#L45)). And Octo seems to use roll, pitch, yaw for both kinds of robots. So should I transform axangle to euler for google robot?

I think so.
Repository: simpler-env/SimplerEnv
Issue: Number of videos per trial

Hi there, 

  May I kindly ask for which part of code that regulate the number of videos saved is always 25?

The number of videos saved is not 25 and varies between different tasks (e.g., "pick coke can" has 75 videos per policy and "move near" has 60 videos per policy)

Please see `scripts` for details; we save video for every trial evaluated and we perform grid-based evaluation.
Repository: simpler-env/SimplerEnv
Issue: AttributeError: 'TimeLimit' object has no attribute 'get_language_instruction'
Root cause: External Dependency Breakage

https://colab.research.google.com/github/simpler-env/SimplerEnv/blob/main/example.ipynb no longer works.

The error is:
AttributeError: 'TimeLimit' object has no attribute 'get_language_instruction'

<img width="747" alt="image" src="https://github.com/user-attachments/assets/acbb59cb-fae1-4c06-bbd5-27b4c0e01c7e">


@branyang02 This is due to Gymnasium 1.0; use `env.unwrapped.get_language_instruction()` instead, or downgrade Gymnasium.

Forced gymnasium < 1.0 in latest commit

Just for a note for others like me . https://github.com/simpler-env/SimplerEnv/commit/f3035e0135d102587f2acd25e5cada3189c06d0b    


You  can just run `pip install gymnasium==0.28.1`
Repository: simpler-env/SimplerEnv
Issue: In multiple GPUs: RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed #20
Root cause: Environment Incompatibility

Hi 👋, I caught the error "RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed #20" when starting the environment in a non-zero device. The simulator works well when using the default CUDA device 0, but it failed in any other devices in `1,2,3,4,5,6,7`:

I tried the solution at https://github.com/simpler-env/SimplerEnv/issues/20 and https://github.com/haosulab/ManiSkill/issues/73, but it doesn't work for me.
<img width="1266" alt="image" src="https://github.com/user-attachments/assets/132b4154-defe-4323-ab6a-30c56cdc8b64">



You can try ManiSkill3 version of the Bridge envs. @StoneT2000 will migrate the Google Robot envs later.

I think a fix for ManiSkill2 is to set `DISPLAY="" CUDA_VISIBLE_DEVICES=x python {}`

See e.g.., https://github.com/haosulab/ManiSkill/issues/79 (from old ManiSkill2)

for maniskill 3 the fix should work proposed by xuanlin should work

Thanks for your replies, the DISPLAY has already been unset in https://github.com/simpler-env/SimplerEnv/blob/d55e19162be86794875839725fd484b768e25873/simpler_env/main_inference.py#L21C2-L21C31, I have no idea why it doesn't work for me. So I will migrate the environment to maniskiill3, does it cause any difference to the evaluation results, compared with mainiskill2? I must make the evaluation fair.

It should be very similar.

Weird. Simpler start-up successfully if I set the `CUDA_VISIBLE_DEVICES=x,0 python {}`. The Memory and utils of `CUDA:0` are almost zero, but it's critical for setup maniskill2 environments:

<img width="2314" alt="image" src="https://github.com/user-attachments/assets/9c363939-02b8-49ca-aec8-f236f1272d1c">


If you only need to measure success / partial success rate in the main study (not the texture randomization ablations, which have not been ported over) I'd just recommend moving to ManiSkill3, it has better support and less bugs related to rendering/gpus.

> You can try ManiSkill3 version of the Bridge envs. [@StoneT2000](https://github.com/StoneT2000) will migrate the Google Robot envs later.
> 
> I think a fix for ManiSkill2 is to set `DISPLAY="" CUDA_VISIBLE_DEVICES=x python {}`

Hi, thanks so much for your great efforts on this project. Could you kindly let me know when the Google Robot migration is expected to be completed?

Might be a while sorry. I can try taking a look again next week perhaps but my initial avoidance was trying to avoid diving too deep into the ruckig package.

Alternatively maybe i can setup one of our existing controllers and if the sr and mmrv values are good then maybe we can skip ruckig despite not modeling the real world controller perfectly
Repository: simpler-env/SimplerEnv
Issue: Success rate about grasping objects in bridge
Root cause: Documentation Deficiency

Hi, thanks for providing detailed evaluation scripts.

I have a small question, when we evaluate models on bridge environment, how can we calculate the success rate of grasping objects like spoon and carrot? (which as shown in Figure 7)
I think maybe we need to change the maniskill_evaluator? I'm not sure

Thank you for your help!

Best,

This is already reported in the names of the saved evaluation videos, if you run `scripts/*`; the results are also extracted via https://github.com/simpler-env/SimplerEnv/blob/main/tools/calc_metrics_evaluation_videos.py#L930
Repository: simpler-env/SimplerEnv
Issue: Different ways to handle inference latency in simulation and on real robot?
Root cause: Documentation Deficiency

Hi, 

Thanks for open-sourcing this great benchmark! I wanted to ask a general question about how is model inference latency handled in simulation and on real robot? For example, the model takes observation at time t as its input to predict the action, and due to model inference latency, it may get the predicted action at time t'. In simulation (Simpler), I think we are "freezing" the env and wait until the model finished inference right? So that the action will be executed on the same observation that the model uses as its input. But on real robot, the robot still moves when the model does inference, so the action will be executed on an observation that is different from the one used as model input. Will these two different modes cause domain gap between simulation and real world, especially considering the high inference latency of RT-1 and Octo? Thanks!

We only step simulation after the model has finished inference, so SIMPLER uses a fixed control frequency (3hz for google robot and 5hz for Bridge). Action will be executed on the same observation that the model uses as its input.

In the real world the control frequency can indeed vary due to latency (and Google Robot and WidowX use non-blocking controller I believe). To handle the latency gap, we perform system identification for simulation (fix-frequency blocking control) controller PD parameters such that it matches real-world (non-blocking) controller behaviors, so it doesn't cause much gap in sim & real behaviors.
Repository: simpler-env/SimplerEnv
Issue: Visual Texture Randomization for Maniskill3 branch
Root cause: Documentation Deficiency

Hi,  
I would like to know how to alter the visual configuration of the env/task (eg: set custom textures or randomize textures between episodes, randomize lighting, camera fov etc).  

The documentation provided [here](https://maniskill.readthedocs.io/en/latest/user_guide/tutorials/custom_tasks/advanced.html#texture-modification) does not show an example.   
  
Any pointer is appreciated!

I managed to change the `base_color` but I cannot randomize or change the texture, any pointer is appreciated.   
```python
base_color_texture = RenderTexture2D(
    "/home/user/Downloads/cliff_side_4k.blend/textures/cliff_side_diff_4k.jpg"
)
for actor_name in self.base_env.unwrapped.scene.actors.keys():
    for part in self.base_env.unwrapped.scene.actors[actor_name]._objs:
        for triangle in (
            part.find_component_by_type(sapien.render.RenderBodyComponent)
            .render_shapes[0]
            .parts
        ):
            # triangle.material.set_base_color([0.8, 0.1, 0.1, 1.0])
            triangle.material.set_base_color_texture(base_color_texture)

obs_dict, _ = self.base_env.reset()
```

@StoneT2000 

Hi @AlexandreBrown. I will address the issue on the ManiSkill repo that you raised, thanks for raising it.
Repository: simpler-env/SimplerEnv
Issue: how to make camera rotate around some objects in scene

Hello, I'm new to the robotics domain and have just started exploring SimplerEnv. I find it is an really impressive work! Currently, I'm doing some experiments using SimplerEnv.

To be specific, I wanna to rotate the camera around some points in SimplerEnv. These points can be some objects in scene or some meaningless points. So I can test Octo/OpenVLA models' generalizability to view point. For example, I run scripts/octo_pick_coke_can_variant_agg.sh code, and get the scene shown below:


![image](https://github.com/user-attachments/assets/c9c3abca-19e2-4e7b-9f5e-25e879ee8b39)


In this scene, 
Q1.how can I get the coke can coordinate relative to the world/robot coordinate system? I may use it as rotation center.
Q2.how can I get the camera coordination frame(position, rotation) relative to the world/robot coordinate system.

So with these coordinations information, I can make camera rotate around some points.

What'more, in Q2, should I use forward kinematics to calculate the relative coordinate? But I have no idea how to get the relative frame rotation for each robot joints. 

Could you please help me to solve these problems? Thank you again for your contributions to the community!

Looking forward to your reply!!


Q1: You can checkout the environment source code at e.g., https://github.com/simpler-env/ManiSkill2_real2sim/blob/main/mani_skill2_real2sim/envs/custom_scenes/grasp_single_in_scene.py  and you can get the object pose wrt world by `env.obj.pose` 
Q2: The camera pose wrt world frame can be directly obtained through e.g., https://github.com/simpler-env/ManiSkill2_real2sim/blob/main/mani_skill2_real2sim/examples/demo_manual_control_custom_envs.py#L253C13-L253C66 . You don't need to perform forward kinematics.

> Q1: You can checkout the environment source code at e.g., https://github.com/simpler-env/ManiSkill2_real2sim/blob/main/mani_skill2_real2sim/envs/custom_scenes/grasp_single_in_scene.py and you can get the object pose wrt world by `env.obj.pose` Q2: The camera pose wrt world frame can be directly obtained through e.g., https://github.com/simpler-env/ManiSkill2_real2sim/blob/main/mani_skill2_real2sim/examples/demo_manual_control_custom_envs.py#L253C13-L253C66 . You don't need to perform forward kinematics.

Thanks for your timely reply!!! I'll try it.