zhiminy/EvalEng huggingface dataset:
Repository: Farama-Foundation/Metaworld
Issue: Making Environments Serializable
Root cause: Resource Mishandling

This is an issue we encountered while trying to duplicate multiple `ML1` environments per worker, and hopefully someone can help us resolve this bug because it’s a blocker with our codebase.

In a meta-learning setup, each meta-batch makes use of several workers in parallel, each of which rolls out episodes from a sampled task (=setting of environment parameters). In our codebase, we use `pickle` to serialize environments for each worker (in order to ensure that environment/task parameters are constant for a worker).

`set_task` in `meta-world` takes a task index and obtains the goal via indexing into `self.discrete_goals`. After some debugging, it turns out that after pickling environments, `self.discrete_goals`, which is a list of 50 goal positions, is different from the value from before pickling. This is with `self.random_init=False`.

We are wondering if there is a recommended way to make `self.discrete_goals` deterministic before and after pickling an `ML1` environment. (Relatedly, we would benefit from clarification on issue [#24](https://github.com/rlworkgroup/metaworld/issues/24), which details what constitutes a task in `ML1`.) Your help is greatly appreciated! 

As a working example:
```envs_list = [] 
env = ML1.get_train_tasks('pick-place-v1')
envs_list.append(env) 
env_pickle = pickle.dumps(env) 
while len(envs_list) < self.num_envs_per_worker:
    envs_list.append(pickle.loads(env_pickle))
print(envs_list[0].active_env.discrete_goals)
print(envs_list[1].active_env.discrete_goals)
```
Env[0] Discrete Goals: ``` [array([0.05635804, 0.8268249 , 0.26080596], dtype=float32), array([-0.08220328,  0.8992955 ,  0.27001566], dtype=float32), array([0.08398727, 0.8188896 , 0.05937913], dtype=float32), array([-0.03422436,  0.82531315,  0.08296145], ... ] ```

Env[1] Discrete Goals: ``` [array([0.04696276, 0.8596079 , 0.12688547], dtype=float32), array([-0.05456738,  0.8163504 ,  0.24694112], dtype=float32), array([-0.09329244,  0.85606927,  0.22053242], dtype=float32), array([-0.00348601,  0.81342274,  0.28464478], dtype=float32), ... ] ```



See https://github.com/rlworkgroup/metaworld/pull/40

@michaelzhiluo does #40 solve your problem?

Thank you @ryanjulian  and @naeioi for the quick fix! This is good 💯
Repository: Farama-Foundation/Metaworld
Issue: Feature request: Online updating leaderboard
Root cause: Unimplemented Feature Gap

I think this would be really cool, and good to track the progress of the community. A possible service that already exists for hosting this  kind of thing: https://eval.ai/web/challenges/list

If any user is inclined to create a leader board that would be great, but as the maintainers (me and @krzentner) we don't have the time to do this.

As Meta-World ownership has been transferred from the RLWorkGroup to [Farama Foundation](https://farama.org/), I am closing this issue. If there are any questions or requests for features please join our [Discord](https://discord.gg/PfR7a79FpQ)
Repository: Farama-Foundation/Metaworld
Issue: TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
Root cause: External Dependency Breakage

Hello

Right after installation of the metaworld, when importing into python (for instance running 'import metaworld'), the following error appears:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/victor/metaw/lib/python3.9/site-packages/metaworld/__init__.py", line 7, in <module>
    import metaworld.envs.mujoco.env_dict as _env_dict
  File "/home/victor/metaw/lib/python3.9/site-packages/metaworld/envs/__init__.py", line 1, in <module>
    from metaworld.envs.mujoco.env_dict import (ALL_V2_ENVIRONMENTS_GOAL_HIDDEN,
  File "/home/victor/metaw/lib/python3.9/site-packages/metaworld/envs/mujoco/env_dict.py", line 6, in <module>
    from metaworld.envs.mujoco.sawyer_xyz.v1 import (
  File "/home/victor/metaw/lib/python3.9/site-packages/metaworld/envs/mujoco/sawyer_xyz/v1/__init__.py", line 1, in <module>
    from metaworld.envs.mujoco.sawyer_xyz.v1.sawyer_assembly_peg import SawyerNutAssemblyEnv
  File "/home/victor/metaw/lib/python3.9/site-packages/metaworld/envs/mujoco/sawyer_xyz/v1/sawyer_assembly_peg.py", line 5, in <module>
    from metaworld.envs.mujoco.sawyer_xyz.sawyer_xyz_env import SawyerXYZEnv, _assert_task_is_set
  File "/home/victor/metaw/lib/python3.9/site-packages/metaworld/envs/mujoco/sawyer_xyz/sawyer_xyz_env.py", line 11, in <module>
    from metaworld.envs.mujoco.mujoco_env import MujocoEnv, _assert_task_is_set
  File "/home/victor/metaw/lib/python3.9/site-packages/metaworld/envs/mujoco/mujoco_env.py", line 31, in <module>
    class MujocoEnv(gym.Env, abc.ABC):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

using python 3.9

(note that a similar issue happened in the mtenv repository and the issue was solved somehow in the last commit)
for the discussion: https://github.com/facebookresearch/mtenv/issues/14
for the fix : https://github.com/facebookresearch/mtenv/compare/main...v2.0.0


Do you have an idea of what is happening?
Thanks for your help

This should be fixed in upstream: https://github.com/openai/gym/pull/3001. You can upgrade your `gym` to 0.25.2.
note only 0.25.0 and 0.25.1 has that problem.

I'm hesitant to update gym versions right now because of the changes in the base gym api. If you'd like to submit a pr that would be great :) 

Actually there is no need to update gym macro or minor version, only need to skip two micro version. i.e. 0.25.0 and 0.25.1
and currently metaworld doesnt pin gym version, so just let user update their gym is enough.
for mtenv, their pinnes gym version has no problem.

Thanks for clarifying!
Repository: Farama-Foundation/Metaworld
Issue: Panda model in Meta-World environment
Root cause: Architectural Constraint

Hello, I'd like to conduct real-world experiments with Franka Panda robot to verify my policy, which is trained on MetaWorld benchmark. Thus, I try o adaptPanda model to MetaWorld environment, where the Panda model is cloned from [franka_robot](https://github.com/SicongShen97/RL-MPC/blob/main/envs_real/assets/franka/franka_robot.xml). However, the Panda model in the MetaWorld environment is deformed. I guess that the body "hand" in the sawyer_xyz_base.xml causes the problem. I tried to adjust the quat of the "hand" body, but it didn't work. The video shows the problem [panda in MetaWorld](https://stealth-hovercraft-8b0.notion.site/October-16f06377fdcd4b73a18e9c7f55d86537). Has anyone ever had the same problem?

I solved this issue by reducing the range of some joints, such as joint3, joint4 and joint5. The modified xml files can be accessed in [panda_model](https://github.com/yaoxt3/kuka_metaworld/blob/master/metaworld/envs/assets_v1/sawyer_xyz/), where **sawyer_xyz_base_panda_v3.xml** and **shared_config_panda.xml** are the modified files. However, I narrow the range of the goal, object and hand, if you want to adapt the panda model to the original setting of MT environment, you have to adjust the range of joint4 in the **sawyer_xyz_base_panda_v3.xml**. BTW, the original range of joint4 is [-3.0718 -0.0698], while the current range in the modified file is [-2.6 -0.55].
Repository: Farama-Foundation/Metaworld
Issue: Button press reward functions also reward pulling on button
Root cause: Algorithmic Error

This rarely matters, but `button-push` family of tasks also reward pulling on the button. This makes it possible to get very high reward without ever succeeding at the task, which should probably be fixed.

Fortunately this rarely matters in practice, since most RL algorithms never attempt to pull on the button.

@krzentner any ideas on ways to fix this?

I saw this happening in one of my experiments, but it's not obvious looking at the code what is wrong. The first step to fixing it would probably be reproducing the issue (perhaps by making a modified scripted policy that triggers the bug). The relevant line of the reward function is [here](https://github.com/Farama-Foundation/Metaworld/blob/master/metaworld/envs/mujoco/sawyer_xyz/v2/sawyer_button_press_v2.py#L122).
Repository: Farama-Foundation/Metaworld
Issue: Add API / wrappers for life long learning RL
Root cause: Unimplemented Feature Gap

In discussions with a PhD college, they were explaining that life long learning in RL doesn't have a set of environments, API, wrappers, etc for researchers to use. 
Therefore, I would propose that we add support for this as it is related to meta and multi-task RL. 
In addition, we could look at transfer learning as an additional type of RL which I don't know the status of.



@pseudo-rnd-thoughts we're going to be adding the wrappers from [ContinualWorld](https://github.com/awarelab/continual_world) soon

Amazing, let me know if you want me to review any code
Repository: Farama-Foundation/Metaworld
Issue: About environments with version v1
Root cause: Unimplemented Feature Gap

Hello, I'd like to ask how to use environments with the version v1 like hammer-v1. The following code will cause an error

import metaworld
MT50 = metaworld.MT50()
env = MT50.train_classes\["hammer-v2"\]()

KeyError: 'hammer-v1'

Thanks a lot!
Repository: Farama-Foundation/Metaworld
Issue: Difference between the version 1 and version 2

I have been working on implementing the multi-task SAC and multi-head multi-task SAC algorithms in Meta-world v2, and I have observed that the success rates are higher compared to the previous results obtained from Meta-world v1. I was wondering if these improvements in performance are due to updates in the reward function for each task.

Could you please provide more information on the changes made to the reward function and how they have impacted the performance of these algorithms in Meta-world v2?

Thank you for looking this.

@krzentner would know more about the exact changes, but the one of the bigger changes is the change in observation space from v1 -> v2. In v1, observations were non-markovian which made solving tasks more difficult. More information can be found in #312, #226, and this comment [here](https://github.com/Farama-Foundation/Metaworld/issues/337#issuecomment-1312644809) about what the exact changes were.
Repository: Farama-Foundation/Metaworld
Issue: Users are confused about goal conditioning
Root cause: Documentation Deficiency

Meta-World was designed to be both a Meta-RL and a Multi-Task RL benchmark.
One of the awkward consequences of that is that the way goal conditioning is handled is very complicated in Meta-World.
**Specifically, all environments in Meta-World are goal conditioned, in every benchmark.**
However, goals are hidden in Meta-RL, and visible in Multi-Task RL.
This is intended to make "goal inference" part of the Meta-RL objective.
This allows ML1 to be used in a very similar way to older Meta-RL benchmark tasks (like `HalfCheetahVelEnv` or Ant Direction).
However, Meta-RL requires that each task be a fully-observable MDP. This requires each "goal" to be considered a different task, and the API reflects this (a `ML1` benchmark object contains 50 train task objects, `ML10` contains 500 train task objects).

However, Meta-World uses the same API for both Meta-RL and Multi-Task RL. Consequently, using the `Benchmark` API, the goal is changed by passing one of the `task` objects to the `set_task` function.
In particular, many users don't use the `Benchmark` API, and don't set the `seeded_rand_vec` flag either (which randomizes the goals on `reset` using the seed passed to the environment on init).
This leads users to believe the environments are not goal conditioned, even though they definitely are supposed to be (50 goals per task, set by the seed).
I don't know how many inconsistent results have been published because of this confusion, but at least a few.

TL;DR: Meta-RL requires ML10 to have 500 tasks, Multi-Task RL wants MT10 to have 10 tasks with 50 goals. This confuses users. 

We should make the documentation and API more clear and harder to mis-use.
A good first start would be renaming the `seeded_rand_vec` flag, and setting it to True by default in all of the environment constructors when not using `Benchmark` API. Unfortunately, this is a breaking change, and we haven't published any versioned package, so we should definitely make sure we have published at least one version of the package before we do this.

Given this confusion between Multi-task and Meta-task RL environments where metaworld treats them as identical and requires the user to differentiate (if I understand @krzentner right). 
Could we not separate these out into two env classes? I understand that users might wish to explore using multi-task environments for meta-task rl but we could create converter classes to support this
Thoughts? @reginald-mclean @krzentner 

The individual environment classes are fine. Having a flag which controls goal visibility really is simpler than adding another wrapper. The `Benchmark` API, however, is a poor fit for Multi-Task RL. We should probably just change the documentation to recommend using the `metaworld.envs.ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE` API for Multi-Task RL.

Having said that, the `Benchmark` API is basically necessary for Meta-RL to work at all, so we need to keep that too.

Example of a paper that assumes that MT10 isn't goal conditioned: https://arxiv.org/abs/2003.13661
Repository: Farama-Foundation/Metaworld
Issue: The code for creating "single-task goal conditioned" environments is confusing
Root cause: Architectural Constraint

The code used to create `metaworld.envs.ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE` and `metaworld.envs.ALL_V2_ENVIRONMENTS_GOAL_HIDDEN` is highly non-obvious and among other things involves running a regex on the environment name. It should be possible to implement this with a simple for loop and a small wrapper class. The current implementation is [here](https://github.com/Farama-Foundation/Metaworld/blob/master/metaworld/envs/mujoco/env_dict.py#L579). Fortunately, there are test [here](https://github.com/Farama-Foundation/Metaworld/blob/18118a28c06893da0f363786696cc792457b062b/tests/integration/test_single_goal_envs.py), so this change should be fairly easy to make and low-risk.
Repository: Farama-Foundation/Metaworld
Issue: Bug in SawyerNutDisassembleEnvV2 obj_low is greater than obj_high
Root cause: Validation Gap

Hi,

the class SawyerNutDisassembleEnvV2 gets a small bug and thus cannot be initialized correctly. 

The obj_low is greater than obj_high in x-axis.

See line 15 and line 16 in:
https://github.com/Farama-Foundation/Metaworld/blob/master/metaworld/envs/mujoco/sawyer_xyz/v2/sawyer_disassemble_peg_v2.py

Fixed with most recent commit!
Repository: Farama-Foundation/Metaworld
Issue: Access to RGB images
Root cause: Documentation Deficiency

Could someone provide a tutorial to take the RGB observation maybe from different positions from these environments, try to find it on the internet but not found a single one. Thanks very much, it will help a lot.

Here's a snippet from how I was able to get the RGB image when evaluating on-policy (boilerplate code credit to the [CORL](https://github.com/tinkoff-ai/CORL/tree/main/algorithms) github). However, this approach is a bit buggy in the single-environment setting, as I frequently get segmentation faults when calling `env_cls()`. However, if I don't call `env_cls()`, then I get a "max timesteps exceeded" error. Somehow just `env.reset()` isn't cutting it. Hope this helps a little, and would be great if anyone has ideas about the aforementioned issues.

```@torch.no_grad()
def eval_actor(
    env_cls,
    actor: nn.Module,
    device: str,
    n_episodes: int,
) -> np.ndarray:
    actor.eval()
    num_success = 0
    episode_rewards = []
    seeds = np.arange(n_episodes)
    for i in range(n_episodes):
        env = env_cls()
        env.seed(int(seeds[i]))
        state, done = env.reset(), False
        episode_reward = 0.0
        for _ in range(env.max_path_length):
            state = env.sim.render(camera_name="corner2", width=64, height=64)
            action = actor.act(state, device)
            state, reward, done, info = env.step(action)
            episode_reward += reward
            if done or info["success"]:
                break
        if info["success"]:
            episode_rewards.append(episode_reward)
            num_success += 1
        else:
            episode_rewards.append(0)

    actor.train()
    return np.asarray(episode_rewards), num_success
```

@tattrongvu you should examine the render function in [mujoco_env.py](https://github.com/Farama-Foundation/Metaworld/blob/b2a4cbb98e20081412cb4cc7ae3d4afc456a732a/metaworld/envs/mujoco/mujoco_env.py#LL125C4-L125C4)



@jren44 I am not sure why you are getting a seg fault. What commit of Meta-World are you using? 
You are getting the error "max timesteps exceeded" because there is a maximum number of steps that you can take in the environment. If you call env.reset(), it should reset the step counter. There was a commit that didn't reset that counter, but it has been fixed in the most recent commit. 

Thanks for the heads up @reginald-mclean . I pulled the latest commits, but for some reason am still getting a seg fault on the line `state = env.sim.render(camera_name="corner2", width=64, height=64)`. I've tried using the `faulthandler` package from python, but the stack trace does not go further up than this call to `env.sim.render()`. Perhaps I'm using the render function incorrectly?


You should be able to do env.render(camera_name="corner2", width=64, height=64)

@jren44 did you resolve this? 


Yes! We talked it out on Discord 😄. Thanks!

@jren44 didn't realize that was you! No problem
Repository: Farama-Foundation/Metaworld
Issue: Missing Environments 
Root cause: Unimplemented Feature Gap

If you try running scripts/demo_sawyer.py, many of the imports don't work because of missing environment such as 
`from metaworld.envs.mujoco.sawyer_xyz.sawyer_stack import SawyerStackEnv`

Hi @jcoreyes, thanks for reporting, those are deprecated and we will update them soon.

@zhanpenghe the 'hello world' env of this suite - SawyerReach - is also missing from the master branch

Hi @MishaLaskin, we use `SawyerReachPushPickPlaceEnv` for reaching, pushing and pickplace tasks.

To use it in a single task way, just do `env = SawyerReachPushPickPlaceEnv(task_type='reach')`.

ah got it thanks @zhanpenghe! Looking forward to reading the accompanying paper when it comes out 

Probably worth updating the gym env registration to reflect this since it currently uses the old sawyer reach that does not exist

@zhanpenghe is this fixed?

Dupe of #70 

Hi @zhanpenghe, I wonder if SawyerStackEnv is available now?

@xiaofei-w I think the current set of environments you see in the repo are everything we plan on having, at least in the forseeable future.
Repository: Farama-Foundation/Metaworld
Issue: About the specific meaning of each parameter in the state？

The state is a vector of length 39, but what does each parameter represent specifically? Where can I find this information?

What environment in particular are you talking about?


> What environment in particular are you talking about?

Is there a significant difference in the meaning of parameters in different environments? For example, I want to know the meaning of the parameters for Door-open and Window-open. Is there a document that describes the meaning of state parameters in all environments?

@reginald-mclean Will know the answer but is on holiday until the end of the week I believe

> @reginald-mclean Will know the answer but is on holiday until the end of the week I believe

Okay, thank you for your response.

The state is a vector of size 36 with the first 18 elements being the current observation and the next 18 elements being the previous observation. The first observation after doing an env.reset() will have the same vector for both of the 18 elements.

Each group of 18 elements can be broken down into the following items: gripper xyz position, single element that describes how open/closed the gripper is, object 1 xyz position, object 1 quaternion, object 2 xyz position, and object 2 quaternion. If a field isn't used, then it is zeroed out (ie only one object, then the object 2 fields are zeroes). In summary [(gripper x, y, z), (open/close), (obj1 x, y, z), (obj 1 quaternion a, b, c, d), (obj 2 x, y, z), (obj2 quaternion a, b, c, d)]. 

If you are doing multi-task learning than this is a vector of size 39 with the addition of the goal as the last 3 elements of the vector. 

> The state is a vector of size 36 with the first 18 elements being the current observation and the next 18 elements being the previous observation. The first observation after doing an env.reset() will have the same vector for both of the 18 elements.
> 
> Each group of 18 elements can be broken down into the following items: gripper xyz position, single element that describes how open/closed the gripper is, object 1 xyz position, object 1 quaternion, object 2 xyz position, and object 2 quaternion. If a field isn't used, then it is zeroed out (ie only one object, then the object 2 fields are zeroes). In summary [(gripper x, y, z), (open/close), (obj1 x, y, z), (obj 1 quaternion a, b, c, d), (obj 2 x, y, z), (obj2 quaternion a, b, c, d)].
> 
> If you are doing multi-task learning than this is a vector of size 39 with the addition of the goal as the last 3 elements of the vector.

Thank you very much for your response.We also want to know the meaning of each parameter in the action.

If you look at the [step function](https://github.com/Farama-Foundation/Metaworld/blob/e5804e300db0ca4235aeecfd7557c93abfec8622/metaworld/envs/mujoco/sawyer_xyz/sawyer_xyz_env.py#LL409C28-L409C28) the first three elements are used to calculate the position delta, and then the last element is used as a control force sent to the actuators in the simulation.
Repository: Farama-Foundation/Metaworld
Issue: Adopt Semantic Versioning and create new releases

Hey there! Seems like recent commits might changed some of the behaviour of some environments on `reset` (Another issue will be made about this soon).

Would it be possible for this project to adopt a semantic versioning scheme, and to publish new releases at some point?

Thanks a lot!

The issue is that `setup.py` has `gym >= 0.15` and `gym` has had a number of API changes since `0.15`. I would suggest reverting back to gym < `0.21`

> The issue is that `setup.py` has `gym >= 0.15` and `gym` has had a number of API changes since `0.15`. I would suggest reverting back to gym < `0.21`

Wouldnt it make more sense to instead change setup.py to restrict it to the compatible gym versions then?