zhiminy/EvalEng huggingface dataset:
Repository: Farama-Foundation/Metaworld
Issue: Atari benchmark?

Hi, this is some great work! I was wondering if there's work in progress to benchmark MetaRL algorithms on atari game environments? Or would there be any advice adapting this work to work with atari games? Thanks!

@snowbabyjia thank you for the question! Metaworld's focus is squarely on robotics, so I don't think you'll ever see Atari-style gameplay benchmarks in this repository.

For adaptation in Atari-like domains, I recommend you take a look at [Procgen](https://openai.com/blog/procgen-benchmark/) or other ALE derivates, e.g. https://arxiv.org/abs/1709.06009
Repository: Farama-Foundation/Metaworld
Issue: Gym versions
Root cause: External Dependency Breakage

Is there a reason gym version is pinned to precisely `0.12.1`?

In general, what gym versions are compatible?

It's pinned at `0.12.1` because that is the version we are sure will work. gym has a habit of introducing breaking API changes without notice, and this is a measure which helps us insulate us from that.

I recommend you run metaworld in it's own virtual environment, which in most cases means the precise version of gym you use won't be of much concern. If you have a need for a later version (e.g. you are working with software which relies on it), we'll happily accept a pull request updating the version to a later one (but we will keep it pinned).
Repository: Farama-Foundation/Metaworld
Issue: How to check whether a task is successfully solved
Root cause: Documentation Deficiency

What is the command to use for checking the task success/not success condition? Is it using the 'done' variable? But I tried to run SAC on one of the task, e.g. window open, and the done var keeps returning False even though the agent successfully opened the window

Thank you for your information!

@aswinnahrendra thank you for trying metaworld!

This is encoded by the `'success'` key of `env_info` and is computed using OR on a per-episode basis.

See https://github.com/rlworkgroup/garage/blob/master/src/garage/_functions.py#L147 for an example
Repository: Farama-Foundation/Metaworld
Issue: AttributeError: 'MjViewer' object has no attribute 'finish'
Root cause: External Dependency Breakage

The rendering works fine until we call `.close()` on the environment.

Minimal code to reproduce the error

```python
from metaworld.envs.mujoco.sawyer_xyz.sawyer_reach_push_pick_place import SawyerReachPushPickPlaceEnv

if __name__ == "__main__":
  env = SawyerReachPushPickPlaceEnv(task_type='push', random_init=True)

  obs = env.reset()
  done = False
  while not done:
    env.render()
    next_obs, reward, done, info = env.step(env.action_space.sample())

  env.close()
```

Stack trace
```
Creating window glfw
Traceback (most recent call last):
  File "try.py", line 13, in <module>
    env.close()
  File "/miniconda3/lib/python3.7/site-packages/metaworld/envs/mujoco/mujoco_env.py", line 135, in close
    self.viewer.finish()
AttributeError: 'MjViewer' object has no attribute 'finish'
```

Gym Version: 0.12.1
Mujoco-py Version: 2.0.2.8

@activatedgeek thanks for the bug report. This is a long-standing bug in `mujoco-py` for which we need to write a workaround.

You can see we use a special helper routine for this in our test suite:
https://github.com/rlworkgroup/metaworld/blob/master/tests/helpers.py#L16

Thanks for putting in the fix!
Repository: Farama-Foundation/Metaworld
Issue: Why is len(ml1.test_tasks) = 50 and not 10?
GitHub labels: question

This might be a trivial question (pardon me if so), but I thought each ML1 environment had 50 training and 10 test tasks?

@avnishn 

Hi @Shushman,
This is a good question thank you for asking it!
ML1 is a benchmark where we train and test algorithms on a single environment with 50 different goal variations. For that reason, any environment that is tested in the ml1 benchmark will have the following properties, which you might have found:

`len(ml1.train_tasks) == 50`

`len(ml1.test_tasks) == 50`

Did this answer your question? Please let me know and feel free to @ me (avnishn) :)

Thanks @avnishn  I do understand your stated motivation of both `train_tasks` and `test_tasks` being 50. My query came from the paper (Pg 6; para 1)
" ML1 uses single Meta-World Tasks, with the meta-training “tasks” corresponding to 50 random initial object and goal positions, and meta-testing on 10 held- out positions".
That made me think `length(ml1.test_tasks)` should be 10, but perhaps I misunderstood?

@Shushman , I spoke with some of the authors on the paper, and they confirmed that ML1.test_tasks should only be 10. 

I'll go ahead and open a pr to address this, thank you so much for catching this!

closed via #312
Repository: Farama-Foundation/Metaworld
Issue: Peg-insert produces obs outside of obs space inside ML/MTx wrappers
Root cause: Interface Contract Mismatch

When being called via the ML1 wrapper, peg-insert-side still produces observations outside of its observation space, but all the other envs are now fixed. I have no idea why peg-insert-side is special, and it's fine if instantiated on its own (as far as I can tell).

closed via #312
Repository: Farama-Foundation/Metaworld
Issue: Feature request: meta-learning baselines
Root cause: Unimplemented Feature Gap

Hi! Thank you for your work!

It would be great to have some baselines that allow for a fast start with Metaworld (e.g. code used for the paper). Or at least to mention some repositories that use Metaworld.

@Guitaricet thanks for the FR. I agree that this would be an excellent contribution to the meta-RL community, but it's definitely out-of-scope for this repo (which intentionally includes the environments only).

We believe it's essential to separate benchmarks from implementations. If you'd like strong baseline implementations, I recommend you take a look at our sister project [garage](https://github.com/rlworkgroup/garage).

Since it's out-of-scope for this repo, I'm going to close this issue. Please don't hesitate to open a new one with a different problem.
Repository: Farama-Foundation/Metaworld
Issue: mujoco131 compatible with metaworld?
Root cause: Environment Incompatibility

My mujoco version is 131, mujoco-py version is 0.5.7. When I try to install metaworld, it shows 
`You appear to be missing MuJoCo.  We expected to find the file here: /home/xxx/.mujoco/mujoco200`, 
`ERROR: Could not build wheels for mujoco-py which use PEP 517 and cannot be installed directly`.
So I cannot use metaworld with mujoco131?
Thank you!

As you can see in the `setup.py` file, `mujoco-py<2.1,>=2.0` is required, together with `gym>=0.15.4`. 
Also, be sure to have installed mujoco in the right way, following the instructions in this page: https://github.com/openai/mujoco-py#install-mujoco.
Repository: Farama-Foundation/Metaworld
Issue: All the tasks are non-Markovian
Root cause: Architectural Constraint

The original MetaWorld paper evaluates the benchmarks with standard memoryless agents like SAC, TRPO, and PPO, none of which have memory in order to solve non-Markovian tasks. However, most (if not all) of the environments are non-Markovian since they only include position information of the end effector, object, and the goal. Is this intentional? This seems to unnecessarily complicate the benchmarks and indeed with a bit more information (e.g. hand/object velocities and object orientations) the environments become solvable much more reliably.

Hi @hartikainen,

Thanks for bringing this up.

This is certainly a valid feedback and I addressed your point in #226.

While Github is a valid place to be having these discussions, if you're looking for more direct communication with the original authors of the paper, please join our slack community, as many of them are more active on there. You can find the link to sign up on our README.

Lastly I'm curious to know what all information did you add to the observation space in the process of doing your own rewrite of the reward functions?

Best,
Avnish Narayan

Thanks @avnishn, this was useful.

> Lastly I'm curious to know what all information did you add to the observation space in the process of doing your own rewrite of the reward functions?

I simply added end effector and object velocities and the object orientation as detailed here: https://github.com/hartikainen/metaworld/blob/reward-tweaks-rebase/metaworld/envs/mujoco/sawyer_xyz/sawyer_xyz_env.py#L304-L319. I think it might be useful to also include some acceleration or contact information but I did not test things that far.

closed via #312
Repository: Farama-Foundation/Metaworld
Issue: How to create two indentical benchmark instances?
Root cause: Unimplemented Feature Gap

It seems like the MT10/ML10 benchmarks (maybe others as well) initialize the goal positions internally every time a benchmark is called. It would be nice if there was a way to create two identical benchmark instances, for example by passing a seed to the environment as an argument.

Currently, I get:
```python
>>> import pickle
>>> import metaworld
>>> mt10_1, mt10_2 = metaworld.MT10(), metaworld.MT10()
>>> pickle.loads(mt10_1.train_tasks[0].data)['rand_vec']
array([-0.05115513,  0.63916668,  0.02      ,  0.07276333,  0.86412829,
        0.12647354])
pickle.loads(mt10_2.train_tasks[0].data)['rand_vec']
>>> pickle.loads(mt10_2.train_tasks[0].data)['rand_vec']
array([ 0.09087166,  0.69323066,  0.02      , -0.0617444 ,  0.81741752,
        0.14243739])
```

Would be useful if something like the following was possible:
```python
import pickle

import numpy as np
import metaworld

seed = 10
mt10_1, mt10_2 = metaworld.MT10(seed=seed), metaworld.MT10(seed=seed)
rand_vec_1 = pickle.loads(mt10_1.train_tasks[0].data)['rand_vec']
rand_vec_2 = pickle.loads(mt10_2.train_tasks[0].data)['rand_vec']
np.testing.assert_equal(rand_vec_1, rand_vec_2)
```

Hey @hartikainen,

Thanks for bringing this feature request to our attention.

This is actually a similar but not quite duplicate request to ~#217~ #147. LMK if you disagree, but we are working on this. 

Granted we are more than happy to accept any new features through pull requests if you'd like to contribute one!

I'm going to leave this open for now, as to make sure it is included in the conversation w.r.t to ~#217~ #147. 

Thanks!
Avnish

Thanks for quick the reply! #217 is so massive that it's not obvious how that relates to this feature request. Would you mind pointing me to the part of the code that helps towards this?

Oh sorry about that I linked the wrong issue. I meant #147 my b. 



@hartikainen does pickling/unpicking and/or copy.deepcopy achieve this right now? It won't get you cross-experiment uniformity but it is a step. 

@avnishn is this FR tracked in an issue?

@ryanjulian the specific feature where we add this as a part of the constructor isn't tracked, although the general feature where we give the option to set the seed is tracked in #147.
Repository: Farama-Foundation/Metaworld
Issue: random_init in the constructor of handle press side
Root cause: Validation Gap

we must have missed this environment or have accidentally let this slip in during a merged pr.

https://github.com/rlworkgroup/metaworld/blob/cb2b475734185cd41f064633c524028ddbc8f9de/metaworld/envs/mujoco/sawyer_xyz/v2/sawyer_handle_press_side_v2.py#L20
Repository: Farama-Foundation/Metaworld
Issue: Success rates for v2 modifications.
Root cause: Unimplemented Feature Gap

I was wondering if there is some place (code) we can use to see out of box performance for the tasks in the envs here. I tried the garage implementation (using PPO) on 3 different envs (`reach-v2`, `pick-place-v2` and `button-press-topdown-v2`, in a MT1 style setting). In addition to garage I also tried running stable-baselines3 (again PPO) on each of these envs again in a MT1 style setting. Out of all of these envs, both the given garage code and stable-baselines3 were only able to solve the reacher task while the success rate for both other envs were 0.  

I don't really think there is any bug in the PPO codes, so I have been wondering how hard are these envs to solve. I saw some other threads which talk about the non-markovian nature of some of these tasks #224, but I am wondering how big of an issue is it? 

Also, is there any place where the performance on each of these envs has been documented? I saw the original arxiv paper but that is seemingly with an older API and I am not really sure if the results hold.

Additionally, is there some place which documents the appropriate hyper-parameters that have been used for these tasks?  I did try hyper-param finetuning (although lightly), so I am wondering if I should try to do that more extensively. However, I am a bit skeptical about it because at least pick-place should not be super complicated (IMO) ? 

Hi @mohitsharma0690, thanks for checking out Metaworld!

The v2 environments are a work in progress. Currently we've upgraded things like the observation space and the meshes of the environments. 

I'm actively in the process of changing their reward functions. I should have them ready by mid October. 

These should not be used in the benchmark right now at all. They are a work in progress. Please do not publish any results using them. Please do not modify the public API when running your benchmarks. 

For now stick the v1 environments. They do have issues with seed sensitivity, however they are definitely solvable.

To answer some of your other questions, there is no place where the hyperparameters for the single task solvability figures in the original paper are published. Also I am currently in the process of running ablation studies to determine how much of an effect #224 has on single task solvability. 

Lastly some environments are generally harder to solve with PPO than algorithms like SAC. Perhaps you'll have some luck using those.

Did I answer your questions?
Please feel free to @ me (@avnishn) if you have any follow up questions. You can also join our slack community (sign up in the README), as the slack community is probably a better venue for questions like this. Given that this isn't a bug I'm going to close this for now.
Repository: Farama-Foundation/Metaworld
Issue: Done signal not correct in the environments
Root cause: Documentation Deficiency

For some environments, the done signal is not given even when the task should terminate already and when I try to step another step it is giving the following error: "ValueError: Maximum path length allowed by the benchmark has been exceeded"

Hi @xinleipan thanks for checking out Metaworld!

This is a purposeful design choice. Metaworld environments don't return `done=True` on termination because these aren't true environment termination states. We've also uncovered that if you return `done=True` on time limit termination, this negatively affects the performance of off policy algorithms that use Q-functions, because the Q-functions are taught that timesteps that are terminated due to environment termination are terminal states. Because agents have no concept of their current time step, this makes their mdps actually pomdps, which are traditionally hard for off policy rl algorithms to learn.

In order to handle the error that you're getting, your sampler will need to reset the environment after env.max_path_length timesteps.

I hope this helps,
@avnishn
Repository: Farama-Foundation/Metaworld
Issue: Correctly set task for V2 Env
Root cause: Unimplemented Feature Gap

How can I successfully find task for v2 environments? All I found in ML1, ML45 etc are tasks with env_name ending with v1

@avnishn 

@buoyancy99 thanks for filing this issue. I'm glad you've noticed there are changes afoot in the repo!

In short, we don't yet support constructing the benchmarks with "v2" environments because those environment versions are a work in progress [1]. We hope to finish them up and release them in the next month or two. We may add an experimental flag shortly which constructs the benchmarks with the "v2" environments, depending on when @avnishn gets around to it.

For your own experimental purposes, you are free to modify the `Benchmark` classes to use the v2 environments, but please don't report results from those environments as corresponding to the metaworld benchmark until we officially release them and update the benchmark. We have no plans to modify the `set_task` API for "v2" so your code probably won't change when we switch to "v2."

[1] The goal of the "v2" environments is to make metaworld more robustly trainable by many different algorithms, and to provide a clearer picture of relative performance. These are the same tasks as "v1", but may include different observation spaces, dynamics, and even reward functions, so you can't compare results from one algorithms "v1" to results from a different algorithm on "v2."
Repository: Farama-Foundation/Metaworld
Issue: Unstable simulation while using Metaworld
Root cause: Validation Gap

Hi, I am testing my algorithm on metaworld and I am frequently getting this exception

File "/home/lthpc/anaconda3/envs/MAML/lib/python3.6/site-packages/mujoco_py/builder.py", line 359, in user_warning_raise_exception
    raise MujocoException('Got MuJoCo Warning: {}'.format(warn))
mujoco_py.builder.MujocoException: Got MuJoCo Warning: Nan, Inf or huge value in QACC at DOF 0. The simulation is unstable. Time = 0.1250.

I met this problem on ML1  'reaching' and 'pushing' envs, and my codes use the same policy structure as PEARL, so I think my codes give out legal actions.

I would greatly appreciate it if you can provide some insight about what's going wrong. Thank you.

@NagisaZj These errors can occur occasionally with some MuJoCo simulations, but you should not be seeing them with regularity. If you are, I recommend you ensure that your policy is not producing actions which exceed the `action_space` of each environment.

If your actions are all compliant with the action space, I think we will only be able to help you if you can provide us with some code which reproduces the problem exactly.

Please re-open this issue if your problem persists (preferably with some code to reproduce your issue).



Is this still an on-going issue? I've encountered this myself with the "window-open-v2" task. I create a list of three hand-picked tasks and doing a sequential continual learning on those tasks. The issue I had was during training on the third task, i.e. "window-open-v2". The error message says the following repeateldy:

`WARNING: Nan, Inf or huge value in CTRL at ACTUATOR 0. The simulation is unstable. Time = 0.0125.`

Then the simulation malfunctions and my loss functions get `NaN` as well. I've checked that agent's action is clipped to the range of (-1, 1). 

Thanks

@George614 can you provide more information about the actions you are using? I am quite confident this is not an issue with Meta-World but I could be wrong. As the above information mentions, it's generally an issue with inputs to the Mujoco simulator so the actions you are applying to the environment
Repository: Farama-Foundation/Metaworld
Issue: Non-Markov behavior in pushing environment
Root cause: Validation Gap

In the SawyerReachPushPickPlaceEnv, it seems like the 3D rotation of the puck is not being reset at the start of each episode (the XYZ position is being correctly). This means that if the puck ends up on its edge, it'll start the next episode on its edge.

If this is not the intended behavior, one easy fix is to define an UP quaternion and set qpos[12:16] = UP.copy().

thanks for pointing this out @ben-eysenbach!

We've actually addressed this in our set of V2 environments, which should be ready for full usage in mid-november :) . Stay on the lookout by join our slack community and/or join our mailing list if you haven't already, that way you can be notified when those are ready.

Best,
@avnishn
Repository: Farama-Foundation/Metaworld
Issue: sweep reward function
Root cause: Algorithmic Error

Hi,
it seems like the reward function in sweep-v1 is not working.  It always returns zero reward if reachDist > 0.05. So if it has solved the task and the block has fallen down from the table the agent doesn't get any reward. So the agent never learns to drop the object.

hi @AlexanderKoch-Koch,

Apologies for the late reply :(. I also apologize if any of the v1 environments have caused you any frustration. 

The Metaworld team has been in the process of making large scale improvements to the benchmark, and we're making various updates to all of the environments. One of those updates is that we have rewritten each environment's reward function. We plan on releasing our updates in the next week or so.

Best,
@avnishn
Repository: Farama-Foundation/Metaworld
Issue: can you manipulate the environment configurations?
Root cause: Unimplemented Feature Gap

Does the package provide access to change the environment configurations? For example, change the object shape or size, change environment dynamics parameters, or change the color or texture of objects? 

@xinleipan no, this package does not provide those capabilities. We would welcome PRs with well-designed features for this.

Note that the way MuJoCo is written makes this fairly difficult to do efficiently on a per-rollout basis (it requires re-loading model XML and assets, which is fairly slow). If you are very motivated to try this, I recommend you take a look a PyMJCF, which inside the [`dm_control`](https://github.com/deepmind/dm_control/tree/master/dm_control/mjcf) library from DeepMind.
Repository: Farama-Foundation/Metaworld
Issue: env.render save images
Root cause: Documentation Deficiency

Hi,
thanks for sharing this amazing project!

I'm trying to use a single task to do pixel-based RL.
But the env.render(mode = 'rgb_array') returns a top-down view only.
I'm wondering how to change the camera viewpoint? say, left-to-right.

Thanks!

@haydenshively knows a lot about this!

thanks @ryanjulian , I solved the issues. I looked into mujoco_env.py 	def viewer_setup(self):

@haoyu-x Glad you figured it out! If you need additional camera views, let me know and I can try to add new cameras to the XML files. Also, I've only tested the existing camera views on some of the V2 environments, so let me know if you experience any issues

> @haoyu-x Glad you figured it out! If you need additional camera views, let me know and I can try to add new cameras to the XML files. Also, I've only tested the existing camera views on some of the V2 environments, so let me know if you experience any issues


Can you please share how did you get images from env.render()? Can you share the code changes that you did? (for V2)
Thanks!