zhiminy/EvalEng huggingface dataset:
Repository: ray-project/llmperf
Issue: Cannot build with `ray` in Docker
Root cause: Environment Incompatibility

In a bog standard Docker build:
```
FROM python

WORKDIR /app

ADD ./ ./

RUN pip install -e .
```


Getting:
```
2.227 Obtaining file:///app
2.228   Installing build dependencies: started
5.611   Installing build dependencies: finished with status 'done'
5.611   Checking if build backend supports build_editable: started
5.843   Checking if build backend supports build_editable: finished with status 'done'
5.844   Getting requirements to build editable: started
6.087   Getting requirements to build editable: finished with status 'done'
6.089   Preparing editable metadata (pyproject.toml): started
6.322   Preparing editable metadata (pyproject.toml): finished with status 'done'
6.731 Collecting pydantic<2.5 (from LLMPerf==0.1.0)
6.827   Downloading pydantic-2.4.2-py3-none-any.whl.metadata (158 kB)
7.090 INFO: pip is looking at multiple versions of llmperf to determine which version is compatible with other requirements. This could take a while.
7.093 ERROR: Could not find a version that satisfies the requirement ray (from llmperf) (from versions: none)
7.364
7.364 [notice] A new release of pip is available: 24.3.1 -> 25.0.1
7.364 [notice] To update, run: pip install --upgrade pip
7.365 ERROR: No matching distribution found for ray
------
Dockerfile:7
--------------------
   5 |     ADD ./ ./
   6 |
   7 | >>> RUN pip install -e .
   8 |
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install -e ." did not complete successfully: exit code: 1
```

Looks like this has less to do with Docker and more to do with the dependency selection of Ray doesn't work with Python 3.11. Had to downgrade to 3.10.

https://github.com/ray-project/ray/issues/33524
Repository: ray-project/llmperf
Issue: Using multiples CPUs
Root cause: Architectural Constraint

Is there any reason why both these lines:

https://github.com/ray-project/llmperf/blob/f1d6bed47e4501b0e371082b41601b59ab55269f/token_benchmark_ray.py#L94
https://github.com/ray-project/llmperf/blob/f1d6bed47e4501b0e371082b41601b59ab55269f/token_benchmark_ray.py#L148

have fixed `num_clients=1`. Couldn't them be changed to:

```python
import multiprocessing
num_cores = multiprocessing.cpu_count()  # Get total available CPU cores
clients = construct_clients(llm_api=llm_api, num_clients=num_cores) 
```

As I've seen in the docs, the parallelism in ray can be done 2 ways:

```python
clients = [OpenAIChatCompletionsClient.remote() for _ in range(8)]  # multiple actors 

# OR

@ray.remote(num_cpus=2)  # Each actor uses 2 CPUs
class OpenAIChatCompletionsClient(LLMClient):
    pass
````	

And in this case I would prefer the first way... Am I missing anything or the way the code is written does not use all the available CPUs??

This one appears to be related: https://github.com/ray-project/llmperf/issues/56
Repository: ray-project/llmperf
Issue: Feature Request: Support multimodal models in correctness test
Root cause: Unimplemented Feature Gap

Hi team, thanks for the great work on `llmperf` — it's been super helpful for benchmarking LLM APIs.

I’m wondering if there are any plans to extend the **correctness test** framework to support **multimodal models**, e.g., models that accept both text and image inputs (like Qwen2-VL-7B-Instruct, glm-4v-9b, etc.).

This would be especially useful for evaluating models on tasks like OCR, image-to-text, or visual question answering.

Would love to hear your thoughts!

Thanks 🙏
Repository: ray-project/llmperf
Issue: Are benchmark results released somewhere
Root cause: Documentation Deficiency

Thanks for this work!

Can we have access to some of your benchmarks without having to run them again? Is there some sort of leaderboard data available somewhere?

Same question to @waleedkadous. If not then I'd be happy to contribute to creating/hosting a public and open-source one.

hey! We will be publishing this soon :) 

Hey @taivop and @ogencoglu , we've published a leaderboard here! 

https://github.com/ray-project/llmperf-leaderboard

Feel free to contribute or comment.

Awesome!

The following would also be interesting to add:

- [Inferless](https://www.inferless.com/)
- [banana.dev](https://www.banana.dev/)
- [beam](https://www.beam.cloud/)
- [mystic.ai](https://mystic.ai/)
- [RunPod](https://www.runpod.io/)
- [fal.ai](https://www.fal.ai/)
- [DataCrunch](https://datacrunch.io/) 
- [DeepInfra](https://deepinfra.com/)
- [Vast.AI](https://vast.ai/)
Repository: ray-project/llmperf
Issue: Support --tokenizer-path to load custom tokenizer (local or remote)
Root cause: Unimplemented Feature Gap

Currently, the tokenizer path is hardcoded and cannot be overridden, which causes issues when evaluating local or private models.

I suggest adding a --tokenizer-path argument to support loading tokenizers from either a local directory or a remote Hugging Face model.

This would require changes in /token_benchmark_ray.py and /src/llmperf/utils.py.

I’m happy to submit a PR for this if it sounds good.
Repository: ray-project/llmperf
Issue: Vimgolf, Cybergod and CTF environments integration, WebUI computer operation recorder integration
Root cause: Others

I have been working with an AI benchmark solving Vimgolf challenges.

It is available here:

https://github.com/James4Ever0/vimgolf-gym

With my library you are able to perform any action in the terminal and get a real time text dump of the entire terminal screen, get the location of the cursor, and take a screenshot of the terminal.

Also there is a primitive CTF library, which is very extensible and environment independent. It can be used for reinforcement learning of terminal and GUI agents.

View it here: https://github.com/James4Ever0/agi_computer_control/blob/master/gym-primitives%2Fctf%2FREADME.md

If you want your agents to earn real money, you can check out Cybergod:

https://github.com/James4Ever0/agi_computer_control/blob/master/gym-primitives%2Fcybergod%2FREADME.md

There is also a WebUI based, local/remote terminal/GUI operation recorder:

https://github.com/James4Ever0/agi_computer_control/blob/master/web_gui_terminal_recorder%2FREADME.md

Your work is inspiring and I will integrate it into my project. Hope my work will do the same for you.
Repository: simpler-env/SimplerEnv
Issue: robot_init_options: Confusion about head tilt
Root cause: Documentation Deficiency

I was playing around with the initial scene configuration. I have set fixed positions for the objects to be placed. Now, I am adjusting some values in the `qpos` array that can be passed as `robot_init_options` but has some provided default values in [base_env](https://github.com/simpler-env/ManiSkill2_real2sim/blob/main/mani_skill2_real2sim/envs/custom_scenes/base_env.py)

The default values and ranges look as follows:
```python
qpos = np.array(
                [-0.2639457174606611, # joint_torso [-4.49, 1.35]
                0.0831913360274175, # joint_shoulder [-2.66, 3.18]
                0.5017611504652179, # joint_bicep [-2.13, 3.71]
                1.156859026208673, # joint_elbow [-2.05, 3.79]
                0.028583671314766423, # joint_forearm [-2.92, 2.92]
                1.592598203487462, # joint_wrist [-1.79, 1.79]
                -1.080652960128774, # joint_gripper (arm joint) [-4.49, 1.35]
                0, 0, # joint_finger_right / joint_finger_left [-0.0001, 1.3]
                -0.00285961, # joint_head_pan [-3.79, 2.22]
                0.7851361 # joint_head_tilt [-1.17, 1.17]
                ]
)
```

This results in the following scene:
![Image](https://github.com/user-attachments/assets/714c9835-1184-4135-a29c-5d7c1b24d092)

When changing the value for the joint_head_pan, I would expect the robot to rotate its head horizontally, resulting in a different crop of the image.
```python
qpos = np.array(
                [-0.2639457174606611,
                0.0831913360274175,
                0.5017611504652179,
                1.156859026208673,
                0.028583671314766423,
                1.592598203487462,
                -1.080652960128774,
                0, 0,
                -3.5, # only value that changed
                0.7851361
                ]
)
```

However, the scene doesn't change at all. The robot arm and objects disappear. Why is that the case?

![Image](https://github.com/user-attachments/assets/86550d59-e637-4f4f-8cb3-10ff7b938b18)

It's because you are using visual matching. Visual matching always overlays a fixed background png into the robot observation (the table in this case). To see the effect of adjusting robot qpos, try the variant aggregation setup or https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/examples/demo_manual_control_custom_envs.py  (remove rgb_overlay_path)
Repository: simpler-env/SimplerEnv
Issue: Evaluate whether the task is completed
Root cause: Documentation Deficiency

Thanks for your great work! I want to speed up the evaluation progress, can I add this code after this line [maniskill2_evaluator.py#L137](https://github.com/simpler-env/SimplerEnv/blob/main/simpler_env/evaluation/maniskill2_evaluator.py#L137)?
```python
if done:
    break
```

Not recommended. Standard evaluation uses a fixed episode length and requires the robot to succeed at the last time step for the episode to be successful, in order to match the real-world criterion.
Repository: simpler-env/SimplerEnv
Issue: About target

Hello! I’d like to ask how to query the Target at each step in SImplerenv. Or, if I want to train my model on SImplerenv, are there any tutorials I could study?Thank you!

Which target? Controller target?

If so you can read the controller code in ManiSkill2_real2sim.

SimplerEnv is not designed for training; it's designed for zero-shot evaluation.

@xuanlinli17 
Thank you for your reply! Currently, we can only obtain the sparse environment rewards and the basic state information (robot position, TCP coordinates, etc.). The reward sequence we’re getting looks like this:

```
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]
```

Over the course of the 15-step execution, the reward is 0.0 for the first 14 steps and then suddenly jumps to 1.0 only when the task is successfully completed.

Our experimental procedure is as follows:

```python
env = simpler_env.make("google_robot_pick_coke_can")
obs, reward, success, truncated, info = env.step(action_array)
```

and we retrieve the reward via the `reward` variable. We’re not sure what’s going wrong—could you please advise us further? Thank you!


SimplerEnv is not designed for RL training, so we don't provide dense reward.

Understood—thank you for your explanation! I have one final question: if we don’t use reinforcement learning but instead use imitation learning, does SImplerenv provide any expert trajectories?

SimplerEnv is not designed for imitation learning either. It is only designed for zero-shot evaluation.

Thank you for patiently answering my questions!
Repository: simpler-env/SimplerEnv
Issue: How to build digital twins in simulation

Hi, I'm studing your excellent work! I want to know that how did you build those scenes (which are from the real dataset, e.g. bridgev2), in maniskill simulator? Since I checked the bridge data, it only provides the end-effector poses of robot girppers, no any other information about the camera, the base linke of the robot,  and all other properties...

Hope any advice from you! Thanks in advance!

Hi, please see https://github.com/simpler-env/SimplerEnv?tab=readme-ov-file#adding-new-real-to-sim-evaluation-environments-and-robots in readme.

Thanks! It means we can only add each scene manually? Not automatically?

Yes, manually. You need to at least get new object meshes using whatever method (even if it is 2D image to 3D mesh).
Repository: simpler-env/SimplerEnv
Issue: Missing GroundingTruth entries for selected google_robot_* tasks in Fractal dataset
Root cause: Documentation Deficiency

Hello,We have selected the following tasks in `SImplerenv`:

```python
[
    "google_robot_pick_coke_can",
    "google_robot_pick_horizontal_coke_can", 
    "google_robot_pick_vertical_coke_can",
    "google_robot_pick_standing_coke_can",
    "google_robot_pick_object",
    "google_robot_move_near_v0",
    "google_robot_move_near_v1",
    "google_robot_move_near"
]
```

We’d like to locate the corresponding episodes (and GroundingTruth annotations) for each of these tasks in the Fractal dataset. However, after searching through the first 15,000 episodes, we only found a GroundingTruth entry for `google_robot_pick_coke_can`. Is this expected? Have we overlooked something?

Our extraction method is as follows:

```python
# Assign episode_id and extract
episode_data = self._extract_episode_data(episode, matched_task, task_counters[matched_task])

# In _extract_episode_data we assemble the actual action trajectory:
action_trajectory = np.array([
    np.concatenate([
        step['action']['world_vector'].numpy(),
        step['action']['rotation_delta'].numpy(),
        step['action']['gripper_closedness_action'].numpy()
    ])
    for step in episode['steps']
])
```

Any pointers on whether the other tasks should appear under different names, or if there’s an alternative way to locate their GroundingTruth, would be greatly appreciated. Thank you!


Could you elaborate what "ground truth" means? The names of the mp4 files saved for visualization have the success conditions.

@xuanlinli17 Sorry to bother!I means the samples in SImplerenv could be easily mapped to a trajectory in open‑x‑embodiment?


SimplerEnv is not designed to map trajectories to open-x-embodiment; it's designed for zero-shot evaluation only. Tasks can be in or out-of-distribution.

thank you！
Repository: simpler-env/SimplerEnv
Issue: Unstable success rates for evaluating OpenVLA
Root cause: Documentation Deficiency

I tried an OpenVLA model finetuned using Simpler-env, and the exact model file is here:
https://huggingface.co/ZijianZhang/OpenVLA-7B-SFT-Simpler/tree/main
And I found that for the same task like PutCarrotOnPlateInScene the success rates are very unstable, For example, for the same model files, SR may differ from 16% to 40%, which brings me difficulty on evaluating the performance of of different models.  I wonder whether it is a model issue or the benchmark issue. Thanks you so much.

You should run it for at least 75 trials; 25 trials is too noisy; paper's results were based on 75 trials.

Thanks a lot. My simulation results are based on 50 trials. Maybe I will try more.
Repository: simpler-env/SimplerEnv
Issue: How to configure new scenes in SImplerenv?
Root cause: Documentation Deficiency

Hello!I’m exploring SImplerenv and would like to define custom scenarios. Could you please clarify:

1、Configurable parameters: Can I change object positions, quantities, types, colors, robot‑arm poses, etc.?
2、Scene definition: How do I create and load a new scene? Is there a specific config format or API?

References: Are there example files or documentation you recommend?

Thanks for your guidance!

(1) Yes, see example scripts folder.
(2) Please follow the readme https://github.com/simpler-env/SimplerEnv?tab=readme-ov-file#adding-new-real-to-sim-evaluation-environments-and-robots
Repository: simpler-env/SimplerEnv
Issue: Map segmentation indices to object labels in SimpleEnv
Root cause: Documentation Deficiency

Hello!Sorry to bother.After running segmentation in SimpleEnv, I get a list of index Object Id. How can I translate these indices into the corresponding object names? Any pointers or examples would be appreciated.Thanks!

![Image](https://github.com/user-attachments/assets/992bcb21-0f7a-452c-ad01-67eba495f79c)


SimplerEnv main branch is based on maniskill2, you can try 
```
env.get_actors() # [Actor(name="ground", id="14"), Actor(name="{name}", id="15")]
{name}_seg = (obs['image'][{camera_name}]['Segmentation'][..., 1] == {id})
```

Thank you for your answer! I’ve followed your guidance to solve this problem.
Repository: simpler-env/SimplerEnv
Issue: Object Placement Instability in SimplerEnv Multi-Object Scenes
Root cause: Documentation Deficiency

Dear SimplerEnv Authors,

We are encountering object placement instability when creating custom multi-object scenes using sapien.Pose. Objects placed at identical coordinates show inconsistent stability across episodes. We would appreciate guidance on proper multi-object placement design and improvement suggestions.

## Problem Summary

**Issue**: Identical objects at same positions sometimes remain stable, sometimes fall through table

## Reproduction Steps

```python
def _initialize_actors(self):
    """Improved stable object placement algorithm - resolving object dropping issues"""
    
    # Predefined safe positions - manually specified known safe positions  
    safe_positions = [
        (-0.30, 0.10), (-0.25, 0.10), (-0.20, 0.10),
        (-0.30, 0.20), (-0.25, 0.20), (-0.20, 0.20),
        # ... additional positions
    ]
    
    # Stage 1: Locked placement with motion constraints
    for i, obj in enumerate(all_objects):
        x, y = safe_positions[i]
        z = self.scene_table_height + object_specific_height
        
        obj.set_pose(sapien.Pose([x, y, z], euler2quat(0, 0, 0)))
        obj.lock_motion(0, 0, 0, 1, 1, 0)  # Lock x,y rotation only
        obj.set_damping(object_specific_damping, object_specific_damping)
```

## Questions

1. Are there recommended physics parameters per object type for stable placement?
2. What is the proper settling time after object placement?
3. Should we use different placement strategies for different object shapes?
4. Is there a deterministic way to ensure consistent placement across episodes?


Thank you for your excellent work and any guidance you can provide.

Best regards 

<img width="1009" height="912" alt="Image" src="https://github.com/user-attachments/assets/92639125-26c1-46c3-970d-b0036361a279" />

<img width="1009" height="912" alt="Image" src="https://github.com/user-attachments/assets/87b80eae-49ad-435f-a0d6-cde1c28745a1" />

All "strange" object movements are definitely caused by penetration due to initialization mistakes, especially if something falls through the table. Remove visual matching (i.e., render raw sim backgrounds) by passing in `rgb_overlay_path=None` in env creation to see what happened.
Repository: simpler-env/SimplerEnv
Issue: An error occurred on the headless server
Root cause: Environment Incompatibility

Hi, 
First of all, thank you for your contribution to this article on spatialVLA.
I encountered an error when configuring simpler_env on the headless server:

                 sapien.SapienRenderer(offscreen_only=True,**renderer_kwargs)
                 RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorExtensionNotPresent

I've tried many solutions, but none of them work. I would like to know if you have any experience running a successful application on a headless server?

Hi, have you tried the troubleshooting section in readme?

I encountered the same error,have you solved it？

Try `CUDA_VISIBLE_DEVICES=xx DISPLAY="" python xxx`?

This error does not appear on other environments but appears only on `move_near_variant_agg`. And for this env, it occurs only after some episodes.
Repository: simpler-env/SimplerEnv
Issue: Overly Eager Success Criteria in Manipulation Tasks
Root cause: Algorithmic Error

Some experiments judge the end condition too early.
For example, in the stack cube task, it is marked as successful when the gripper merely hovers above the target. It should instead wait until the gripper opens and the cube is stably stacked without falling.

In the put eggplant task, it is marked as successful when the gripper touches the edge of the sink dish rack. It should instead wait until the gripper moves to the center of the rack.

Eggplant shouldn't be the case? See https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/envs/custom_scenes/put_on_in_scene.py for success criterions. Object A needs to be on object B based on bbox. I didn't check gripper open though.
Repository: simpler-env/SimplerEnv
Issue: RuntimeError: 'gymnasium.wrappers.common.TimeLimit' object has no attribute 'get_language_instruction'
Root cause: External Dependency Breakage

Hi, when I evaluate RT-1 on your benchmark, I have this issue:
```bash
RuntimeError: 'gymnasium.wrappers.common.TimeLimit' object has no attribute 'get_language_instruction'
```
I figured it out by downgrade the version of gymnasium to `0.29.1`.

You did not specify this version in your codebase however it can download the latest gymnasium (e.g. `1.2.1` for now), which is incompatitable with the simplerenv.

thanks will fix!

https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/requirements.txt
it already enforces <1.0 here?

Sorry, I mixed up your repo and https://github.com/allenzren/ManiSkill2_real2sim/blob/cd45dd27dc6bb26d048cb6570cdab4e3f935cc37/requirements.txt#L4
Repository: simpler-env/SimplerEnv
Issue: Segmentation fault when computing IK
Root cause: External Dependency Breakage

System: UBUNTU 20.04
Python: 3.10.14

All codes are set up with the introduction in README. _vulkaninfo_ and _python -m sapien.example.offscreen_ works fine, seems that there are no rendering issues.

Segmentation fault is located at:
ManiSkill2_real2sim/mani_skill2_real2sim/agents/controllers/pd_ee_pose.py line 61:
result, success, error = self.pmodel.compute_inverse_kinematics(
            self.ee_link_idx,
            target_pose,
            initial_qpos=self.articulation.get_qpos(),
            active_qmask=self.qmask,
            max_iterations=max_iterations,
        )

How can I solve this problem?

Another person #6 has pointed out that pinocchio encounters issue when numpy>=2.0. 
Installing numpy==1.24.4         (numpy<2.0) will resolve this issue.
I've updated the readme.

> Another person [#6](https://github.com/simpler-env/SimplerEnv/issues/6) has pointed out that pinocchio encounters issue when numpy>=2.0. Installing numpy==1.24.4 (numpy<2.0) will resolve this issue. I've updated the readme.

Thank you . It works! numpy 1.26.4  leads to Segmentation fault
Repository: simpler-env/SimplerEnv
Issue: Moving ManiSkill2_real2sim tasks to Maniskill3
Root cause: Unimplemented Feature Gap

Hi,

First of all, congratulation for your incredible work, the environment is growing so fast, it's amazing.

I was trying to upgrade my code by moving from ManiSkill2 to ManiSkill3, however, I found that the same tasks that I have in my benchmark are not available for ManiSkill3, is there any way to move all the Custom_tasks (the ones in the ManiSkill2_real2sim repository) to ManiSkill3?

I tried to copy and paste but it didn't work to me, I feel that I'm missing something. I would appreciate if you could provide me some guidance.

Thank you in advance,
Pablo.

Bridge tasks are already done; google robot pending (due to controller migration which is tricky, as the environment uses cpu-based ruckig, which takes time to parallelize in gpu)