zhiminy/EvalEng huggingface dataset:
Repository: Farama-Foundation/Metaworld
Issue: Baseline results on v2 benchmarks
Root cause: Unimplemented Feature Gap

Hi, 

Since the benchmarks are moving to v2 environments, I would imagine this would change the baseline results too. Have you tried running the baselines on v2 benchmarks, and if so could you share the results? This would make it much easier to run and compare algorithms on the new version. 

We definitely intend to share results for v2 as soon as we've finished running and aggregating them. @avnishn has been working on that, and I expect we will release those results within the next couple of weeks.

> We definitely intend to share results for v2 as soon as we've finished running and aggregating them. @avnishn has been working on that, and I expect we will release those results within the next couple of weeks.

Great!!!

Waiting for an update

Hi @avnishn, Do you know if there's an update on v2 benchmark results? 

We've had baselines available in our sister repo, garage, for a while now. 

I'm updating them with what I ran to get the v2 baseline results. 

You can track my progress here:

https://github.com/rlworkgroup/garage/pull/2287

The updates are stable, I'm just adding documentation and tests in order to pass our code merge-ability checks. 



Hello @avnishn,

Thanks for providing us your PR. Quick question: is it possible to also share the "progress.csv" files of the baseline runs? It would save a lot of time and compute resources for those who want to have the baseline curves (as we have for Figures 12~18). 

Thanks!

I'll have to get back to you on that @luckeciano. Although I'd recommend re-running your results for any benchmarks that you decide to use, because there might be algorithmic differences in the implementations of these algorithms that you use vs. the ones that I wrote. 

As Meta-World ownership has been transferred from the RLWorkGroup to [Farama Foundation](https://farama.org/), I am closing this issue. If there are any questions or requests for features please join our [Discord](https://discord.gg/PfR7a79FpQ)
Repository: Farama-Foundation/Metaworld
Issue: Question on the design choice of Sawyer arm's workspace

Thanks for sharing this valuable environment that can be a seminal baseline for reinforcement learning for manipulation
research. 

I have a question on the workspace of the Sawyer robot arm defined in the Cartesian space.
It seems in the implementation of the SawyerXYZEnv, the upper/lower bound of the Cartesian workspace is confined, and the action of the agent is nullified if the target action is out of the boundary [as here](https://github.com/rlworkgroup/metaworld/blob/9e3863d3a3b7d3f0ab6f1b759bb156abbbb27a1a/metaworld/envs/mujoco/sawyer_xyz/sawyer_xyz_env.py#L175).

What I am curious about is, does the agent get to learn a valid behavior that avoids exploring around the boundary with this setting? 

In my intuition, if the action is nullified, the reward term does not change (since most reward terms in meta-world are designed based on the distance between the end-effector and target object & distance between the target object and the goal), and the agent will explore other state-action space that outputs the change in the reward term. In this sense, this design choice of nullifying the action out-of-boundary seems valid.

As an alternative that I have been working on in my custom environment, I defined out-of-boundary (oob) penalty that penalizes the agent exploring outside the desired workspace. Once the end-effector of the agent is oob, the episode is reset with a significant amount of penalty added to the reward function. However, this technique does not seem effective as I expected.

Could you give me some insight on 
1) why penalizing the oob behavior is not working,
2) and the reason for the design choice of nullifying oob exploration?

Thanks in advance!


I would personally expect that setting a large penalty for reaching outside of the boundary would work. However, it's possible that observation normalization would have issues if the boundaries were changed or ignored, resulting in the policy network being poorly conditioned for that part of the observation space.

In general, debugging issues with RL agents during training like this is quite involved, and unfortunately I can't provide significant assistance. I would recommend starting with a working configuration and slowly modifying it. We decided to simply prevent the gripper from leaving the workspace because it is a simple solution for ensuring that the observations are usually bounded and that random exploration is likely to reach the correct part of the workspace.
Repository: Farama-Foundation/Metaworld
Issue: Discrepancy between paper and code in `sawyer_push_v2` reward.
Root cause: Documentation Deficiency

Hi, 

In the  `sawyer_push_v2`  environment, the reward for each step is computed as 
written [here](https://github.com/rlworkgroup/metaworld/blob/9e3863d3a3b7d3f0ab6f1b759bb156abbbb27a1a/metaworld/envs/mujoco/sawyer_xyz/v2/sawyer_push_v2.py#L165).

 What I am curious about is these lines of code,

`reward = 2 * object_grasped`
`if tcp_to_obj < 0.02 and tcp_opened > 0:`
`    reward += 1. + reward + 5. * in_place`

which seems to add  `object_grasped` twice.

Is this intentional or a typo? If intentional, what is the reason for this design choice?

Thanks in advance!


Typically, our reward functions are made up of logical terms computed using the `tolerance` function, then combined together using the Hamacher product, before being combined by a linear weighting of terms.

The code here is somewhat confusing because it gives the `object_grasped` term a weight of `3` in a surprising way.  The coefficients in this linear combination are found empirically, so if they work, we're not changing them.

Having said that, this does look like a small discrepancy between the code and the reward function listed in the paper. We will update the paper to have the correct coefficient. Thanks for finding it!

Thank you for the clarification!
Repository: Farama-Foundation/Metaworld
Issue: How meta-world (or Mujoco) allows the random explorative actions in Cartesian space?

Hi, thanks for sharing the awesome benchmark for this challenging domain!

I have a question about the action space defined for the meta-world.

AFAIK, the action space of the meta-world is Cartesian XYZ translation, which requires solving inverse kinematics afterward.
After some research, I found that Mujoco has the concept of 'mocap' for setting the Cartesian pose of the articulated robot arm
and meta-world is using that feature.

I am currently using [RLBench](https://github.com/stepjam/RLBench) API that is built upon [Coppeliasim](https://www.coppeliarobotics.com/), and it often times raise IK error if I set the action space as Cartesian space translation and do the random exploration of the robot arm.

So, I am curious about how meta-world can avoid IK error in the early stage of training that would yield numerous invalid target configurations for solving IK during the control period.

Thank you in advance!

Mujoco's IK works completely differently from Coppeliasim's, so this difference isn't so surprising. Unfortunately, I can't really help you more than that.
Repository: Farama-Foundation/Metaworld
Issue: May I ask a slightly silly question

Hello！I am new to Mujoco and may ask a slightly silly question.
I followed your steps step by step, ran the code you provided, and found no results at last. I don't know where I did wrong. Could you please give me some advice?
May I ask if the source code you provide is just the underlying library code, similar to NUMPY, and we have to write the framework code of the final reinforcement learning?
![image](https://user-images.githubusercontent.com/75004709/128330387-c503e8b8-ae2a-4b66-acb2-e3ba10dfa847.png)


This library only contains the environments (similar to OpenAI's Gym). If you want RL code as well, please take a look at [garage](https://github.com/rlworkgroup/garage), which we also maintain and has examples for Meta-World (the Meta-World results reported in the paper were run using [this branch](https://github.com/rlworkgroup/garage/tree/merge-mw-v2-examples)).

Hello！I ran the program in Garage, but there was no visual code in it.

I changed the program here：

But the program reported an error

Can you tell me how to visualize the interface like in the paper?








------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "rlworkgroup/metaworld"                                                                                    ***@***.***&gt;;
发送时间:&nbsp;2021年8月7日(星期六) 凌晨3:14
***@***.***&gt;;
***@***.******@***.***&gt;;
主题:&nbsp;Re: [rlworkgroup/metaworld] May I ask a slightly silly question (#349)





 
This library only contains the environments (similar to OpenAI's Gym). If you want RL code as well, please take a look at garage, which we also maintain and has examples for Meta-World (the Meta-World results reported in the paper were run using this branch).
 
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
Repository: Farama-Foundation/Metaworld
Issue: Mismatch of render() interface between MetaWorld and Gym
Root cause: Interface Contract Mismatch

Currently the render function of MujocoEnv does not take a `mode` argument for the `render()` function and instead exposes the `offscreen` flag. 
https://github.com/rlworkgroup/metaworld/blob/a0009ed9a208ff9864a5c1368c04c273bb20dd06/metaworld/envs/mujoco/mujoco_env.py#L121-L133
This is an issue when using the environments e.g. with any form of gym wrappers. As seen here https://github.com/openai/gym/blob/2853ce4797a4dadb1f8702c8e35143d9fbd43ea6/gym/core.py#L253-L254 the mode is passed as argument to the underlying environment per default. If render is not explicitly called as `render(False)`, we have `offscreen='human'` and the check for `not offscreen` will fail.

From what I can see the `offscreen` flag seems to be the same as `mode=human` and `mode=rgb_array` in the base gym interface. Is there any specific reason the `offscreen` flag is introduced over the default behavior?
If not, I think the `render()` function should not be overloaded this way to remain compatible with existing gym functionality. For the remaining kwargs I do not see any issues, as they fit the interface. 

Thanks for looking into this. 

This could be fixed by changing all environments to use mujoco (and the gymnasium MujocoEnv)


Is there any reason in general why the mujoco base env is re-impelmented instead of inherited from mujoco_py?

@csimo005 it is a heavily modified version of the mujoco base env. In a future release we are going to be using the mujoco base env from Gymnasium.
Repository: Farama-Foundation/Metaworld
Issue: Which commits to run to reproduce paper results?
Root cause: Documentation Deficiency

Hi, I've been trying to reproduce the paper's results for the MT10 setting with MTSAC (from Garage - running [mtsac_metaworld_mt10.py](https://github.com/rlworkgroup/garage/blob/master/src/garage/examples/torch/mtsac_metaworld_mt10.py)) and I'm yet to stably get the algorithm to exceed a 40% success rate on either v1 or v2. Assuming I'm interpreting the data correctly, per the revised paper, I believe we should expect >60% success with MTSAC on v2.

I noticed in Issue #344 that you'd been working on updating the garage codebase to include the baselines and that was a few months ago and both codebases have been updated since then, so I was just wondering which versions of both codebases I should be using in order to reproduce the results from the paper.

Edit: I've tried both the current masters as well as code with commit dates closest to the v2 paper release.

I'm having similar issues. I've been running my own version of MTSAC w/ similar hyperparameters as the one suggested in the paper's appendix and Garage code. I can't get the performance up to >60%. 

Interestingly, out of 8 seeds, 3-4 runs won't learn anything and ofc this as a great impact on the aggregated performance.

I'm curious if @sidmysore sees the same behaviour.

Also, the shaded areas in the Appendix, are those standard deviations of standard errors?
 

Hi @optimass and @sidmysore,

Thanks for checking out Meta-World!

1) This is the exact branch that I used to run the paper results:
https://github.com/rlworkgroup/garage/pull/2287

https://github.com/rlworkgroup/garage/tree/08492007d6e2d9ead9beb83a8a4247e52019ac7d/metaworld_examples

2) The shaded areas are 95% confidence intervals.

I actually graduated, and no longer maintain Garage. The pull request/branch that I linked is what I used to run the experiments for the paper. These examples haven't been updated on master/merged onto master. You'd have to ask the current garage team about that. 

Thanks,
@avnishn 

cc
@krzentner
Repository: Farama-Foundation/Metaworld
Issue: Reconfirm the environment & task & seed & Successs Rate
Root cause: Documentation Deficiency

@avnishn 

Noticing that there are lots of issues about the environment & task, I am sorry to open this type of question again. Reading almost of them, I want to confirm whether my understanding is correct.

1) In my understanding, in meta-RL settings, such as ML10 and ML45.
You set 10 or 45 meta-training environments and 5 meta-test environments, each with 50 tasks.
Is it True?

2) In the Figure 4-8, "seed (N) = 10" is also used. What's is the seed?
Does it mean that in one environment, like reach-v2, you set 50 different variants/tasks (with different initial object and goal positions) and each variant/task you repeat 10 times? 

3) Based on the above understanding of seeds, I think that 500 (50 tasks * 10 seeds) total epsisodes/trajecories/rollouts are reached for each environment. Therefore the Success Rate  = # success episidoes/ 500. Is it true?


As Meta-World ownership has been transferred from the RLWorkGroup to [Farama Foundation](https://farama.org/), I am closing this issue. If there are any questions or requests for features please join our [Discord](https://discord.gg/PfR7a79FpQ)
Repository: Farama-Foundation/Metaworld
Issue: Running with mujoco210 vs 200
Root cause: External Dependency Breakage

Hi there,

is seems the default mujoco-py is now version 210. However metaworld seems to expect mujoco200, an older version. Is there a simple tweak that can be made to run this environment in the most updated mujoco env or do I need to wrestle with a 200 install instead?

Thanks

Hey Neeraj,

I can spend a little time this week updating the mujoco version. 

I think the short answer to your problem is that there is a version of mujoco-py that supports the newest mujoco, and you'll need to install that. Can you try that and get back to me? 

Thanks! 
@avnishn 

Hi @avnishn ,

I have downloaded the newest version of mujoco which works with the newest version of mojoco-py, following these instructions [here](https://docs.google.com/document/d/1eBvfKoczKmImUgoGMbqypODBXmI1bD91/edit). 

In a sandbox, the examples for mujoco-py work fine. When I get to the step in metaworld's README:

`pip install -e .`

I get the following:

![image](https://user-images.githubusercontent.com/9519733/150835622-232cd81b-6e45-422b-9937-bae4b4824458.png)

I have tried downloading mujoco200 to match this but it's really been quite a headache. Is there a quick fix to make metaworld compatible with mujoco210?



The reason why it's failing is because the version of mujoco-py that is installed by metaworld looks for mujoco200. If you update that version of mujoco-py in the setup.py, then run pip install -e . , then you'll be able to run with mujoco 210. 

Yeah...looking back that really was a simple fix. Apologies...

Fixed by subbing out:

`mujoco-py<2.2,>=2.1`

In the setup.py

Hey no worries, happy to be of help. Going to leave this issue open do visibility! 

I have a fix for this on a branch. Since it's clearly affecting other users I'll push a change and also update the version number to `0.1.0` soon.

As Meta-World ownership has been transferred from the RLWorkGroup to [Farama Foundation](https://farama.org/), I am closing this issue. If there are any questions or requests for features please join our [Discord](https://discord.gg/PfR7a79FpQ)
Repository: Farama-Foundation/Metaworld
Issue: [BUG] ML10 Tasks incorrectly specified in env_dict.py
Root cause: Validation Gap

in the ML10_V2 specification in env_dict.py I believe the dictionary value corresponding to 'button-press-topdown-v2' should be SawyerButtonPressTopdownEnvV2 instead of SawyerButtonPressEnvV2. This way the name would match the environment and ML10 would parallel MT10.

See here:
https://github.com/rlworkgroup/metaworld/blob/master/metaworld/envs/mujoco/env_dict.py#L391


That certainly would make sense. Unfortunately, I believe this discrepancy is intentional. I'll ask around to make sure.

As Meta-World ownership has been transferred from the RLWorkGroup to [Farama Foundation](https://farama.org/), I am closing this issue but it will be fixed in a future version of Meta-World. If there are any questions or requests for features please join our [Discord](https://discord.gg/PfR7a79FpQ)
Repository: Farama-Foundation/Metaworld
Issue: Bug(or not) about drawer-close-v1 environment.
Root cause: Algorithmic Error

When I'm training my multi-task policy for Multi-Task 10 benchmark, I find that drawer-close-v1 task always succeeds at the first timestep of the episode. 

I checked the implementation of the drawer-close and drawer-open environments, and I noticed that they use the same goal by default and they used the same condition to judge whether the task succeeded. I'm not sure whether this is correct, would you mind taking a look at it?


Drawer-close init function 
```python
        self.init_config = {
            'obj_init_angle': np.array([0.3, ], dtype=np.float32),
            'obj_init_pos': np.array([0., 0.9, 0.04], dtype=np.float32),
            'hand_init_pos': np.array([0, 0.6, 0.2], dtype=np.float32),
        }
        self.goal = np.array([0., 0.55, 0.04])
        self.obj_init_pos = self.init_config['obj_init_pos']
        self.obj_init_angle = self.init_config['obj_init_angle']
        self.hand_init_pos = self.init_config['hand_init_pos']
```
success condition: (in step and compute_reward function)
```
objPos = obs[3:6]

rightFinger, leftFinger = self.get_site_pos('rightEndEffector'), self.get_site_pos('leftEndEffector')
fingerCOM  =  (rightFinger + leftFinger)/2

pullGoal = self._state_goal[1]

reachDist = np.linalg.norm(objPos - fingerCOM)

pullDist = np.abs(objPos[1] - pullGoal)
```
```python
info = {'reachDist': reachDist, 'goalDist': pullDist, 'epRew' : reward, 'pickRew':None, 'success': float(pullDist <= 0.06)}
```

Drawer-open init function:
```python

        self.init_config = {
            'obj_init_angle': np.array([0.3, ], dtype=np.float32),
            'obj_init_pos': np.array([0., 0.9, 0.04], dtype=np.float32),
            'hand_init_pos': np.array([0, 0.6, 0.2], dtype=np.float32),
        }

        self.goal = np.array([0., 0.55, 0.04])

        self.obj_init_pos = self.init_config['obj_init_pos']
        self.obj_init_angle = self.init_config['obj_init_angle']
        self.hand_init_pos = self.init_config['hand_init_pos']

```
success condition: (in step and compute reward function)
```python

objPos = obs[3:6]

rightFinger, leftFinger = self.get_site_pos('rightEndEffector'), self.get_site_pos('leftEndEffector')
fingerCOM  =  (rightFinger + leftFinger)/2

pullGoal = self._state_goal

pullDist = np.abs(objPos[1] - pullGoal[1])
```
```python
info = {'reachDist': reachDist, 'goalDist': pullDist, 'epRew' : reward, 'pickRew':None, 'success': float(pullDist <= 0.08)}
```

@tianheyu927 @zhanpenghe 

Thanks! That's indeed a bug. I just submitted a PR to fix it.
Repository: Farama-Foundation/Metaworld
Issue: Pick-Place-Wall-V2 and Shelf-Place-V2's rewards seem sparse
Root cause: Documentation Deficiency

Hi, I am experimenting with the latest version of Meta-World on the Master branch. 

I found that while in other environments, the agent always get some reward every timestep, in MT50's Pick-Place-Wall-V2 and Shelf-Place-V2, random policy always gets 0 reward. I wonder if this is a bug, as random policy in Pick-Place-V2 gains non-zero rewards.

How to reproduce:
```

import metaworld
import random


mt1 = metaworld.MT1('pick-place-v2') # Construct the benchmark, sampling tasks

env = mt1.train_classes['pick-place-v2']()  # Create an environment with task `pick_place`
task = random.choice(mt1.train_tasks)
env.set_task(task)  # Set task

obs = env.reset()  # Reset environment
a = env.action_space.sample()  # Sample an action
obs, reward, done, info = env.step(a)
reward
```



It gives 0.00014508224409682773. 

```
import metaworld
import random

mt1 = metaworld.MT1('pick-place-wall-v2') # Construct the benchmark, sampling tasks

env = mt1.train_classes['pick-place-wall-v2']()  # Create an environment with task `pick_place`
task = random.choice(mt1.train_tasks)
env.set_task(task)  # Set task

obs = env.reset()  # Reset environment
a = env.action_space.sample()  # Sample an action
obs, reward, done, info = env.step(a)
reward
```

It gives 0.0.

Looking forward to some help! Thank you!

They aren't sparse rewards. You can read the reward functions in their respective files, or in the appendix of the Meta-World v2 paper. 

They are somewhat difficult reward functions to optimize on, because they involve multiple fuzzy components to guide them through a sequence of actions.