zhiminy/EvalEng huggingface dataset:
Repository: Farama-Foundation/Metaworld
Issue: Two questions about image rendering

Hi, 

Thanks for open-sourcing this great benchmark! I have two questions regarding image rendering and hope you can help with. Thanks!
1. I think it has been mentioned that the rendered image is up side down for some systems. However, do I also need to flip the left and right side of the image? Because I notice that for some tasks, the expert trajectory involves rotation that is opposite to the language discription of the task. E.g., for the task `faucet-open-v2`, Appendix A of the paper says that this task "rotate the faucet counter clockwise". However, the rendered video rotates the faucet clockwise as shown below. So I want to check that if left-right flipping is also needed? Or is it an issue with the language instruction itself?

https://github.com/user-attachments/assets/21144221-1435-49cb-afb8-1fd3192121e5


2. I notice that there are some small spheres with different colors and sizes in the rendered image. Intuitively they seem like some reference points for things like goal positions? I was wondering that what rules are followed to generate these spheres? And will their colors change with parametric variations inside each task?

Thanks for your help!

Hi! 1. It be just code drift through the years that switched the close and open descriptions. You shouldn’t need to do anything else to the video, other than flip it 
2. Yes those *should* change with the parametric variations!
Repository: Farama-Foundation/Metaworld
Issue: consult about action scale in diffierent environment

I'm a RL learner and wanna to use meta-world to deal with multi-task problem. But I found the action in different task maybe has different scale. Is there some illustrations? Such as, [0.1, 0.1, 0.1, 0] in push is equal [0.1, 0.1, 0.2, 0]？
I'll very appreciate it if someone could reply me!

you can get the action space with `env.action_space`

you may be confusing the action (what the agents does for a single time step) and the action space (the set of all possible actions the agent may take)

> you can get the action space with `env.action_space`

I may not have described my problem clearly. I want to know, whether the distances, which the robotic arm actually moved in different tasks, would be different if the actions is same, such as [0.1, 0.1, 0.1, 0].

Assuming the obstacles, such a wall or the objects which are meant to interacted, are not touched and that the state is the same then yes you will have the same movement

Note: the initialization is not deterministic, you may want to use the same seed in order to have the same initial state


Hope this helps

Thanks very much! I think maybe my question is related to obstacles!
Repository: Farama-Foundation/Metaworld
Issue: The depth and action distance in Metaworld
Root cause: Documentation Deficiency

Hello,

I have a couple of questions regarding the data obtained from MetaWorld:

When retrieving depth images from MetaWorld, what real-world values do these depth values correspond to? Specifically, what are the units?

Regarding the actions in MetaWorld, what real-world values do they correspond to? Could you provide details on the units or scaling used for the actions within the environment?
Repository: Farama-Foundation/Metaworld
Issue: Questions about rendering

![Image](https://github.com/user-attachments/assets/2259c685-ec6b-48bf-8b4c-632de6a23652)
![Image](https://github.com/user-attachments/assets/f2110f85-817b-4f61-ac98-128453ad51a1)
Does this have any impact and how should this be addressed?

Is this after the env.step is called? There are similar errors if you don't call `env.close()` due to the way garbage collection happens after your code is run.
Repository: Farama-Foundation/Metaworld
Issue: How to set the frequency of the environment to serve the inference of low-frequency systems (such as VLA)
Root cause: Documentation Deficiency

### Question

I noticed some variables like:

```python
self.metadata["render_fps"]
...
self.model.opt.timestep
...
self.frame_skip
...
```

How can I set this up for seamless integration of low frequency systems (5-10Hz). 🤔
Repository: Farama-Foundation/Metaworld
Issue: [Question] an unexpected issue with the 2.0.0 release
Root cause: External Dependency Breakage

### Question

Hello Metaworld team,

I’d like to report an unexpected issue with the 2.0.0 release:

**1. API compatibility broken**
In previous installs, `metaworld==2.0.0` correctly followed the Gymnasium 1.0.0 API and provided the expected “v2” task environments. However, when I created a fresh conda environment and ran:

`pip install metaworld==2.0.0`

the resulting package unexpectedly contains the “v3” task definitions instead.

**2. Impact on research reproducibility**
This sudden change in the contents of a fixed version tag disrupts ongoing experiments and makes it impossible to reproduce earlier results. I kindly ask that you:

- Restore the original 2.0.0 functionality (i.e. the Gymnasium‑compatible “v2” tasks)
- 
- Ensure that published version tags remain immutable and maintain backward compatibility
- 
- Provide a patch or guidance on how to revert to the intended 2.0.0 behavior

Thank you for your attention and for all your work on Metaworld! Any help or clarification you can offer would be greatly appreciated.

Hi! Thanks for using Meta-World. It looks like when I created the v3 release and pushed to pypi it used 2.0.0 as the version instead of 3.0.0. I will look into that asap. 

The v2 functionality is available in the v3 envs by default -- there's really only the change from v2->v3 naming. But if you want to install the version of Meta-World you were using before you probably want commit `304d79db1fd56401aff00a92181db0c977a04663` so I think the command would be:
`pip install pip install git+https://github.com/Farama-Foundation/Metaworld.git@304d79db1fd56401aff00a92181db0c977a04663`
Repository: Farama-Foundation/Metaworld
Issue: Parametric variations for multi-task training (MT-x)
Root cause: Unimplemented Feature Gap

Hi all. Thanks for putting the benchmark together!

One comment however on multi-task training: why are the MT-x benchmarks constrained to fixed goal locations? Including parametric variations would increase the size of the training distribution and increase the potential for transfer across tasks. As is, the optimal policies for MT-10 and MT-50 would be a discrete mixture of 10 or 50 policies, which might preclude behavior sharing across tasks. If parametric variations were to be included, then goal positions (in addition to task ID) would be provided as input to the policy -- in contrast to the meta-learning benchmarks. Thanks for taking this into consideration!

@gdesjardins our revised API now allows you to use up to 50 variations on each MT10/MT50 environment using the `set_task` API. See the README for details.
Repository: Farama-Foundation/Metaworld
Issue: [Proposal] Please fix readme
Root cause: Documentation Deficiency

### Proposal

Hi,

The readme has pretty simple errors and mistakes in its quickstart instructions, namely the gym.make command. The website documentation has the correct versioning but the metaworld website points here and not there. I love the project and think this type of simple change would be much appreciated by other newcomers looking to get started!


Thanks

### Motivation

readme has simple mistakes

### Pitch

_No response_

### Alternatives

_No response_

### Additional context

_No response_

### Checklist

- [x] I have checked that there is no similar [issue](https://github.com/Farama-Foundation/Gymnasium/issues) in the repo


Hi @delara38 , thank you for noticing this. I am only seeing the use of 'V3' instead of 'v3.' Is there anything else that needs to be updated?
Repository: Farama-Foundation/Metaworld
Issue: [Question] Failed to use MT50
Root cause: Environment Incompatibility

### Question

Exception: Failed to load XML file: /home/user/Zhaoy1s/planseqlearn/metaworld/metaworld/envs/assets_v2/sawyer_xyz/sawyer_table_with_hole.xml. mj_loadXML error: b"Error: PNG file load error 'empty input or file doesn't exist' in texture id = 1\nObject name = T_table, id = 1, line = 9, column = 6"
the code is 

![Image](https://github.com/user-attachments/assets/0d398982-6032-46d3-ab07-94cdd2ae1eb4)
I checked that all the files were intact, but neither using absolute nor relative paths worked.
Repository: Farama-Foundation/Metaworld
Issue: 'ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE' is missing
Root cause: External Dependency Breakage

### Question

I'm tring to import  lift3d.envs from MetaWorldEnv. However, the issue occurred. 



(lift3d) [issacb25@SPGL-1-1 LIFT3D]$ python -m lift3d.tools.sample_obs_metaworld
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/grp01/cs_hszhao/ale/LIFT3D/lift3d/tools/sample_obs_metaworld.py", line 8, in <module>
    from lift3d.envs import MetaWorldEnv
  File "/grp01/cs_hszhao/ale/LIFT3D/lift3d/envs/__init__.py", line 2, in <module>
    from .metaworld_env import (
  File "/grp01/cs_hszhao/ale/LIFT3D/lift3d/envs/metaworld_env.py", line 10, in <module>
    from metaworld.envs import ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE
ImportError: cannot import name 'ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE' from 'metaworld.envs' (/home/issacb25/.conda/envs/lift3d/lib/python3.11/site-packages/metaworld/envs/__init__.py)

I've checked metaworld.env and there is no any variable named 'ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE' . 


You won't find a variable like that because the current version of MetaWorld is V3.
Repository: Farama-Foundation/Metaworld
Issue: Dynamic Task Loading
Root cause: Documentation Deficiency

### Question

I now hope to achieve dynamic loading tasks while creating multiple environments for parallel computing. For example, at the beginning of training, I only created one task with 48 environments. When the task training ended, the number of tasks became 2 and the number of environments was assumed to be 48. This process continued. Please note that I need to sample and obtain these tasks.

Hi, I'm not sure what you're asking. Can you provide some more information? Or even some pseudo code? 

# Define task pool
ALL_METAWORLD_TASKS = sorted(list(metaworld.MT.available_tasks))

TASK_POOL_A = ALL_METAWORLD_TASKS[:10] # for example：['button-press-v2', ..., 'reach-v2']


#Main loop: Process a pair of A and B tasks in each iteration
    for i in range(len(TASK_POOL_A)-1):
        current_task_A_set = TASK_POOL_A[:i+1] #Task set A: From the first to the current A task
        current_task_B = TASK_POOL_A[i+1]       #Task B: Each iteration is a new task in the B pool


        #Create a multi tasking environment for Current_taskA_set
        try:
            env_A = metaworld.MT.from_task_names(current_task_A_set, seed=seed + i * 100) 
            env_A_wrapped = normalize(GymEnv(env_A, max_episode_length=env_A.spec.max_episode_length))
        except Exception as e:
            print(f"Error creating MT environment for Task A subset {current_task_A_set}: {e}")
            continue 
	
	# Reinforcement Learning Algorithm Update
         Training

        env_A.close() 

       #Create a single task B environment
        try:
            mt1_B = metaworld.ML1(current_task_B, seed=seed + i * 200)             env_cls_B = mt1_B.env_cls
            env_B = env_cls_B()
            env_B_wrapped = normalize(GymEnv(env_B, max_episode_length=env_B.spec.max_episode_length))
        except Exception as e:
            print(f"Error creating ML1 environment for Task B {current_task_B}: {e}")
            continue 
	
	# Reinforcement Learning Algorithm Update
        Training
        
        env_B.close() 


```import gymnasium as gym
import metaworld

from metaworld.env_dict import ALL_V3_ENVIRONMENTS

mw_tasks = sorted(list(ALL_V3_ENVIRONMENTS.keys()))

task_pool_a = mw_tasks[:10]

for i in range(len(task_pool_a)-1):
    current_task_A_set = task_pool_a[:i+1] 
    envs_a = gym.make('Meta-World/custom-mt-envs', envs_list=current_task_A_set, seed=seed + i * 100, vector_strategy='async') 
    current_task_B = task_pool_a[i+1]
    envs_b = gym.make('Meta-World/custom-mt-envs', envs_list=current_task_B, seed=seed + i * 100, vector_strategy='async') # there is also custom-ml-envs if you want an meta-learning environment
```

Okay！Thanks！ I have already solved！
Repository: Farama-Foundation/Metaworld
Issue: Object and goal consistency with random_init=False
Root cause: Validation Gap

Not all tasks seem compatible with the `random_init=False` mode of the ML1 benchmark. For example, some tasks require some degree of consistency between the object location and the goal. For example, in [sawyer_drawer_close](https://github.com/rlworkgroup/metaworld/blob/master/metaworld/envs/mujoco/sawyer_xyz/sawyer_drawer_close.py#L220) `random_init=True` first samples a random object location, and then configures the goal location as the same location with an offset of 0.2 (in one of the xyz coordinates). No such consistency is enforce when `random_init=False` and goals are configured by hand via  `random_init=False`. See code example below.

```python
from metaworld.benchmarks.ml1 import ML1

def diagnose(env):
  img = env.sim.render(height=480, width=640)
  plt.figure(); plt.imshow(img[::-1])
  print('Goal location:', env._state_goal)
  print('Object location: ', env.obj_init_pos)
  print('Reward:', reward)

env = ML1.get_train_tasks('drawer-close-v1', sample_all=True)
env.active_env.random_init = False
env.active_env.initialize_camera(top_down_camera)
task_a, task_b = env.sample_tasks(2)  

env.set_task(task_a)
obs = env.reset()  # Reset environment
diagnose(env.active_env)

env.set_task(task_b)
obs = env.reset()  # Reset environment
diagnose(env.active_env)

env.set_task(task_a)
obs = env.reset()  # Reset environment
diagnose(env.active_env)
```

which clearly shows the discrepancy between goal locations a/b and the static object:

    Goal location: [0.2470843  0.7214825  0.38136098]
    Object location:  [-0.08980742  0.89999998  0.04      ]
    Reward: -0.11522688447982872
    Goal location: [-0.14852686  0.57654315  0.45528063]
    Object location:  [-0.08980742  0.89999998  0.04      ]
    Reward: -0.11522688447982872
    Goal location: [0.2470843  0.7214825  0.38136098]
    Object location:  [-0.08980742  0.89999998  0.04      ]
    Reward: -0.11522688447982872



@gdesjardins The revised API fixes this issue by making goal changes in your control using the `set_task` API. Each environment comes with 50 train configurations and (in the case of meta-RL) 50 test configurations.

Please give it a shot and let us know what you think!
Repository: Farama-Foundation/Metaworld
Issue: Meta-World/MT50 registered but entry_point is not specified
Root cause: Documentation Deficiency

### Question

I attempted to install metaworld on ubuntu22.04 using python 3.10 or Python 3.11, and tried to use the code in the readme document.：
```
import gymnasium as gym
import metaworld

seed = 42

envs = gym.make('Meta-World/MT50', vector_strategy='sync', seed=seed) # this returns a Synchronous Vector Environment with 50 environments

obs, info = envs.reset() # reset all 50 environments

a = envs.action_space.sample() # sample an action for each environment

obs, reward, truncate, terminate, info = envs.step(a) # step all 50 environments
```
However, when testing the MT50 dataset, I encountered the following issues:
```
Traceback (most recent call last):
  File "/home/rm/wxz/EmbodiedAI/vla_infer_2/vla_infer/metaworld/test_meta50.py", line 18, in <module>
    envs = gym.make('Meta-World/MT50', env_name="assembly-v3",vector_strategy='sync', seed=seed) # this returns a Synchronous Vector Environment with 50 environments
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rm/miniconda3/envs/py3.11_meta/lib/python3.11/site-packages/gymnasium/envs/registration.py", line 691, in make
    raise error.Error(f"{env_spec.id} registered but entry_point is not specified")
gymnasium.error.Error: Meta-World/MT50 registered but entry_point is not specified
```

Yep sorry, it should be gym.make_vec instead of make
Repository: Farama-Foundation/Metaworld
Issue: [Proposal] Proposal title
Root cause: Unimplemented Feature Gap

### Proposal

The observation for the RL environment is insufficient, which leads to poor stability in RL training algorithms. An important part of the problem is that the object's dimension and shape information are completely absent from the observation. Position and orientation alone are not always sufficient.

### Motivation

_No response_

### Pitch

_No response_

### Alternatives

The observation for the RL environment is insufficient, which leads to poor stability in RL training algorithms. An important part of the problem is that the object's dimension and shape information are completely absent from the observation. Position and orientation alone are not always sufficient.

### Additional context

_No response_

### Checklist

- [x] I have checked that there is no similar [issue](https://github.com/Farama-Foundation/Gymnasium/issues) in the repo


`observation for the RL environment is insufficient... Position and orientation alone are not always sufficient.`
This is quite the claim. Can you provide some evidence for this somehow?

`object's dimension and shape information`
The reward function **should** encompass this information, shouldn't it?

Please fill out the information a bit more thoroughly, as you haven't explicitly shown 
(a) that this is a problem
(b) where the problem comes from
(c) what we can do to overcome this issue
Repository: Farama-Foundation/Metaworld
Issue: How to fix the scene position when using env.reset
Root cause: Documentation Deficiency

### Question

Thank you very much for providing the dataset. However, when I tested the VLA model using your MT50 data, even though I fixed the random seed using:
```
envs = gym.make_vec(
    'Meta-World/MT50', 
    render_mode='rgb_array', 
    camera_id=4, 
    seed=self.cfg['seed']
)  
```
the scene position still changes every time ```env.reset() ```is called. How can I resolve this?

Well that's the thing, it should be changing every time :)

If you don't want to use multiple goal/object locations, then you should be able to use only a single initialization as follows:

```
envs = gym.make_vec(
    'Meta-World/MT50', 
    render_mode='rgb_array', 
    camera_id=4, 
    seed=self.cfg['seed'],
    num_goals=1,
)  
```
Repository: Farama-Foundation/Metaworld
Issue: [Question] Issue with RL Training Results in MT1
GitHub labels: question

### Question

Hello,
Thank you very much for providing the Metaworld platform—it has been very helpful for RL training. However, I’ve encountered some confusion when using the previous V2 version and the new V3 version.

In V2, the interaction speed was quite slow, which was limiting for researchers with constrained computational resources. Therefore, I switched to the V3 version, which is much faster in terms of interaction.

However, when I trained under the same environment and random seed, I found that the results in V3 seem to be worse than those in V2. Could the developers please share any insights or experiences regarding this issue?

I look forward to your reply.

Can you provide which tasks you are using and the code you are using in v2 and v3 to create your environments? As far as I know, there wasn't any degradation in performance. 

Thank you for your answer. I have identified the issue, I was using the camera position adjustment method from the V2 version, which is not applicable in V3. As a result, the observations differ between the two versions, leading to different outcomes.
Repository: Farama-Foundation/Metaworld
Issue: [Question] How to Adjust Camera Position in MT1 V3 Version
Root cause: Documentation Deficiency

### Question

Hello,

Thank you very much for providing the Metaworld platform, it has been extremely helpful for RL training. However, I’m a bit confused about the differences between the previous V2 version and the new V3 version.

In V2, I was able to change the camera position using the following code:
---------------------------------------------
if self._camera == "corner2":
    self._env.model.cam_pos[2][:] = [0.75, 0.075, 0.7]
------------------------------------------------

However, I couldn’t find an effective way to modify the camera position in V3. Could you please provide guidance or methods on how to adjust the camera position in the V3 version? It would be very helpful for my research. I look forward to any insights or experience you could share regarding this issue.


Something like this should work.

```
import gymnasium as gym
import metaworld
import numpy as np

env = gym.make("Meta-World/MT1", env_name='assembly-v3', seed=42, camera_name='corner2', render_mode='rgb_array')

env.reset()
env.render()

env.unwrapped.model.camera('corner2').pos = np.array([0.75, 0.075, 0.7])
```

https://github.com/user-attachments/assets/a51f086f-8f2b-42b3-8879-95fc7d9de04c

Thank you for your reply. I conducted some tests and saved the evaluation videos. I found that the images are upside down.

Yep, I'm not sure why they're rotated like that. You can wrap the rendered RGB array in two numpy rot90 calls like:
`
img = np.rot90(np.rot90(env.render()))
`
To get the image flipped the correct way

Thank you for your reply. I conducted some tests and saved the evaluation videos. I found that using method `np.rot90` results in a right-side view, and strangely, some RL methods fail to learn good policies in this case. However, when using method `np.flipud` results in a left-side view, I can reproduce the results reported in the RL papers.

`np.rot90(np.rot90(env.render()))`

https://github.com/user-attachments/assets/9a90a093-6ee6-459e-9f4a-4095c38c73f2
-----------------------------------------------------------------------------
`np.flipud(env.render())`

https://github.com/user-attachments/assets/01c04452-51f5-43df-b189-6bc2b127c236
-----------------------------------------------------------------------------------
Repository: Farama-Foundation/Metaworld
Issue: [Action space] Lerobot dataset
Root cause: Documentation Deficiency

### Question

Dear Reggie,I have a problem，the action space in lerobot/metaworld_mt50 is not [-1,1],but in the eval simulation the action space is [-1,1]，when the policy inference the action is not correctly.please help me thanks.

According to the official MetaWorld documentation (https://metaworld.farama.org/benchmark/action_space/), the action space is defined as Box(-1.0, 1.0, (4,), float32), representing normalized Cartesian displacements (dx, dy, dz) and gripper control.
However, when I checked the dataset statistics at:
https://huggingface.co/datasets/lerobot/metaworld_mt50/resolve/main/meta/stats.json
I found that the action ranges are:

min: [-11.47, -14.93, -17.28, -1.0]
max: [19.46, 12.47, 24.63, 1.0]

The first three dimensions (dx, dy, dz) appear to be well outside the expected [-1, 1] range, while the gripper dimension remains within [-1, 1].
What do the action values in the dataset represent?
How should these actions be converted when deploying a trained policy back to the MetaWorld environment?

Hi @afengleafs, unfortunately I was not a part of collecting the data so I am not sure I can be much help. There are ways to generate this data yourself in [our docs](https://metaworld.farama.org/benchmark/expert_trajectories/) which I can help with if you describe what you are trying to do. :)