zhiminy/EvalEng huggingface dataset:
Repository: Farama-Foundation/Metaworld
Issue: Reproducing PEARL results on ML10/45
Root cause: Documentation Deficiency

I am trying to reproduce the ML10 results for Pearl from figure 8 of the Metaworld paper. Here is the learning curve I reproduced: 

<img width="917" alt="pearl_ml10_learning_curve" src="https://user-images.githubusercontent.com/13114311/86184201-7c5efb80-bae8-11ea-8b38-01d41549a92e.png">


My results are different from the paper's. The average success rate on training tasks are frequently zero and does not resemble a "curve". Can you give me some pointers on how to reproduce the paper results? 

Basically, I used Kate Rakelly's implementation of Pearl ([https://github.com/katerakelly/oyster](https://github.com/katerakelly/oyster)). I adapted the code to log the success rate. I'm using the same hyperparameters listed in the Metaworld paper. For meta-testing, I set the number of exploration trajectories to 10, and I take the average success rate over 10 test rollouts (set num_evals=10 in Pearl's hyperparameters). 

It would be great if you can provide some insight on how to reproduce Pearl results on ML10/45 and provide some feedback on hyperparameters and evaluation protocol. 

@bernwang please retry with the new API, which allows you to use `set_task` to change when an environment's configuration changes. Managing these configuration changes is key to training and evaluating PEARL and was previously quite implicit/difficult under the new API.

I have reproduced the PEARL results using metaworld and find them quite consistent.

Sorry to bother U! I'd like to reproduce ML10/45, but I am confused about the setting or meta-train tasks and meta-test tasks. Is the total env num of ML10 10? If so, what is the total task num of different task configurations? Or meta-train tasks comprise 10 different envs and every envs comprise 5 different task configurations. The same question also arises in the meta-test task.

@avnishn 

Hi @jp18813100494, 

Let me try to respond to your questions in order:
> Is the total env num of ML10 10? If so, what is the total task num of different task configurations?

Yes, the total number of training environments in ML10 is 10 environments. There are 5 testing environments. You would evaluation your pearl on these 5 environments. Each of these environments has 50 different task configurations. A task configuration consists of a unique combination of (object_initial_position, goal_position). 

This means that for ML10, there are 10 * 50 = 500 unique task configurations for the ML10 training environments, and 5 * 50 = 250 unique task configurations for the ML10 testing environments.

Did this answer your question? Feel free to @ me (avnishn) if you have any more questions, or if this didn't answer your question!

Best,
Avnish
Repository: Farama-Foundation/Metaworld
Issue: random_init flag for ML benchmarks
Root cause: Documentation Deficiency

As far as I know, I could not find document or Github issues properly explaining this, so I apologize if this question is redundant. 

From the table in [https://github.com/rlworkgroup/metaworld/issues/50#issuecomment-618081913](https://github.com/rlworkgroup/metaworld/issues/50#issuecomment-618081913), I understand that the starting state is fixed for ML1 and random for ML10 and ML45. This is reflected in the random_init flag which is False for ML1 and True for ML10 and ML45, e.g. [https://github.com/rlworkgroup/metaworld/blob/6f1a94f7b540045e81cf8d2fb758230b7dad5bdc/metaworld/benchmarks/ml10.py#L23](https://github.com/rlworkgroup/metaworld/blob/6f1a94f7b540045e81cf8d2fb758230b7dad5bdc/metaworld/benchmarks/ml10.py#L23). 

But from what I can make sense of it, when random_init=True, the state goal is randomly sampled instead of the starting state. Is this intentional? What is the reasoning for this?

Thank you!

@bernwang `random_init` has been removed throughout the codebase. You now can control when an environment's configuration changes using the `set_task` API. Take a look at the README for more info.

Visiting many different initial conditions of an environment is essential for allowing meta-RL algorithms to properly generalize. Given only a single starting configuration, there's no way for the meta-RL algorithm to infer which features of the environment are salient to configuration changes it might encounter during adaptation, and which ones are irrelevant to a adaptation. Put in a more terse way, your training environment set needs to contain enough variation information to support the test-time task distribution.
Repository: Farama-Foundation/Metaworld
Issue: Joint angle
Root cause: Documentation Deficiency

How can i get joint angle or rotation of 7DoF Sawyer robot for each step?

Hi! Thanks for expressing interest in MetaWorld! All of the environments in the `sawyer_xyz` folder inherit from `SawyerMocapBase`. This class has a function called `get_env_state` which should return a tuple containing (1) the joint state and (2) the motion capture ("mocap") state. That mocap state is itself a tuple containing position and an orientation quaternion.

That said, the orientation of the end effector (the tip/fingers of the Sawyer) should never change in the MetaWorld environments. It's held constant with the quaternion being `[1, 0, 1, 0]`. It's also never passed as part of an environment's observation, so the `get_env_state` function is more of a debugging tool than anything else.

@haydenshively Thank you very much !

I have tried to get parameters as follows.
```
env = ML1.get_train_tasks('reach-v1')
joint_state, mocap_state = env.active_env.get_env_state()

MjSimState(time=0.12500000000000006, qpos=array([ 2.08394164e+00, -5.70318419e-01, -1.19094847e+00,  1.77256338e+00,
        9.66836328e-01,  1.15366423e+00,  2.40350109e+00, -1.36425008e-04,
        1.72924933e-04, -7.63206202e-02,  6.96439282e-01,  1.99999996e-02,
        1.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00]), qvel=array([-5.36537248e+00, -7.20529153e-02,  6.36763108e+00, -3.74349469e+00,
        1.87573374e+00, -3.64890076e+00, -1.54358735e+00, -1.24176395e-03,
       -2.47893757e-05,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00]), act=None, udd_state={})
(array([[0. , 0.6, 0.2]]), array([[0.70710678, 0.        , 0.70710678, 0.        ]]))
```
I am clear about mocap state, but I can't understand joint_state well.
Are there any document about joint_state?
or I would appreciate it if you could tell me more detail.
Repository: Farama-Foundation/Metaworld
Issue: Make everything that's not part of the gym.Env API private

The only public members of envs should be `step()`, `reset()`, `render()`, `close()`, `action_space`, `observation_space`, and `set_task()`

As Meta-World ownership has been transferred from the RLWorkGroup to [Farama Foundation](https://farama.org/), I am closing this issue. If there are any questions or requests for features please join our [Discord](https://discord.gg/PfR7a79FpQ)
Repository: Farama-Foundation/Metaworld
Issue: Difference between `goal` and `_state_goal`
GitHub labels: question

Hello :)

Thank for open-sourcing the code for MetaWorld. Could you please tell me the difference between `goal` and `_state_goal` attrs of the `env` object? Since `_state_goal` is returned as part of `obs` and is used for computing the rewards, I am assuming that it is the goal that the learning agent is trying to reach. I could not understand what is the significance of the `goal` attribute. Since it is returned as part of `info`, I am assuming it has some significance :)

Hi Shagun,

Thanks for using metaworld. The relationship between `goal` and `_state_goal` is different in different environments. Generally speaking, the `goal` field cannot be relied upon to mean anything specific, even though it is returned in `state_info`. In older (pre-publication) versions of metaworld, the `goal` field was what `_state_goal` is now. However, it does not vary across tasks set in the same environment (even though the true goal location / reward function does).

The field `goal` field will probably be removed in the future, because its presence is confusing.

For multi-task benchmarks (MT10 and MT50), the `_state_goal` field is part of the observation, as you've noted. However, for meta-learning benchmarks (ML10 and ML45), the meta-learner is expected to adapt to the goal location without direct access to it. In other words, reading the `_state_goal` field violates the benchmark, and results that use it should not be presented as results on the ML10 or ML45 benchmark.

Hopefully that answers your question. Feel free to leave further questions if you have them.

Thank you so much for explaining this :)
Repository: Farama-Foundation/Metaworld
Issue: Enable randomization of configuration for hammer, sweep into, and bin picking
Root cause: Unimplemented Feature Gap

Currently for the hammer, sweep into, and bin picking environments, the object initial position is randomized, but not the goal position. This is the incorrect behavior. The goal state should be randomized in addition to the object initial position.

@avnishn didn't we conclude that these environments already have randomization?

I had filed this issue before we had our conversation. I'll close it now.
Repository: Farama-Foundation/Metaworld
Issue: Apply `state_rand_vec` logic to v2 envs
Root cause: Unimplemented Feature Gap

When we did the massive refactor/new api implementation, `state_rand_vec` was only put into v1 environments. v2 envs still have the old logic in `reset()`

Please make this high-priority since it will block our ability to run v1 vs v2 benchmarks
Repository: Farama-Foundation/Metaworld
Issue: get_env_state function

_Originally posted by @haydenshively in https://github.com/rlworkgroup/metaworld/issues/137#issuecomment-655071547_

I have tried to get parameters as follows.
```
env = ML1.get_train_tasks('reach-v1')
joint_state, mocap_state = env.active_env.get_env_state()

MjSimState(time=0.12500000000000006, qpos=array([ 2.08394164e+00, -5.70318419e-01, -1.19094847e+00,  1.77256338e+00,
        9.66836328e-01,  1.15366423e+00,  2.40350109e+00, -1.36425008e-04,
        1.72924933e-04, -7.63206202e-02,  6.96439282e-01,  1.99999996e-02,
        1.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00]), qvel=array([-5.36537248e+00, -7.20529153e-02,  6.36763108e+00, -3.74349469e+00,
        1.87573374e+00, -3.64890076e+00, -1.54358735e+00, -1.24176395e-03,
       -2.47893757e-05,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00]), act=None, udd_state={})
(array([[0. , 0.6, 0.2]]), array([[0.70710678, 0.        , 0.70710678, 0.        ]]))
```
I am clear about mocap state, but I can't understand joint_state well.
Are there any document about joint_state?
or I would appreciate it if you could tell me more detail.

@emuemuJP `get_env_state()` is not a public API in metaworld, which means we don't make great effort to document it and it can change arbitrarily at any time. You should expect it to be made private or disappear in the near future, as we rewrite the API to be clear about what's public (see #141 ).

The public API for observations is exposed via `gym.Env`, i.e. the return values from `reset()` and `step()`.

The contents of `get_env_state()` is the raw state of the underlying MuJoCo simulation, and its elements can be quite different between different environments. If you would like to reverse-engineer the exact assignment from array elements to simulation joints and bodies, I recommend you read the underlying MuJoCo XML model associated with that environment and cross-reference with the [MuJoCo XML documentation](http://mujoco.org/book/XMLreference.html). 

Thank you for your advice despite the private API！
Repository: Farama-Foundation/Metaworld
Issue: Max timestep limit
GitHub labels: question


@HeegerGao thanks for bringing up this feedback!

The `max_path_length` limit is intended to ensure that results generated with metaworld are reproducible across research papers. The performance of two algorithms can be drastically different using two different `max_path_length` numbers, given the same number of environment samples. This is especially true in the case of meta-learning, which typically uses trajectories 

I'll take this issue as feedback for future revisions of the benchmark. It may be artifically-constraining to limit the max_path_length, when we can already measure the total environment steps taken during training regardless of max_path_length (and in the case of meta-RL, the total environment steps prior to adaptation).
Repository: Farama-Foundation/Metaworld
Issue: Is max timestep limit useful?
Root cause: Documentation Deficiency

Hi, I am new to metaworld. I find that you set a `max_path_length` in `metaworld/envs/mujoco/mujoco_env.py`. I find that you mentioned this in [#39](https://github.com/rlworkgroup/metaworld/issues/39), where you said it is because `Mujoco environments have a max_path_length of 150`. 

But I find that in other mujoco envs (like openai gym), there is no such timestep limit. So I just delete line 101 and 102 in `metaworld/envs/mujoco/mujoco_env.py` to unrestrict the limit. Then I just run an env from your benchmark envs for 1000 timesteps and I don't encounter any problem. So I just wonder if this timestep limit is useful.

Reposting my response from #157 

My short answer is "yes," and you can read below for why it exists.

@HeegerGao thanks for bringing up this feedback!

The `max_path_length` limit is intended to ensure that results generated with metaworld are reproducible across research papers. The performance of two algorithms can be drastically different using two different `max_path_length` numbers, given the same number of environment samples. This is especially true in the case of meta-learning, which typically uses trajectories during the adaptation step. It also functions to give users a reasonable idea of how long an environment takes to solve.

I'll take this issue as feedback for future revisions of the benchmark. It may be artificially-constraining to limit the max_path_length, when we can already measure the total environment steps taken during training regardless of max_path_length (and in the case of meta-RL, the total environment steps prior to adaptation).

If you plan on writing a research paper, please note: regardless of your perspective on design decisions, it's never valid to report results in a research paper as "metaworld" using a modified version of the benchmark code. If a limitation is particularly-cumbersome and you need to change it, we ask that you present the results from both the modified and unmodified version of the benchmark in all figures/tables/plots, and make clear in the main text the exact modifications you made.
Repository: Farama-Foundation/Metaworld
Issue: Fix _state_goal value in a single-task environment
Root cause: Documentation Deficiency

Hello, I appreciate you providing great Meta-RL environments! I'm recently working on this environment for further implementation. However, it may sound silly, but I'm stuck in finding out how to fix the position of goal-marker or env._state_goal. 

For example, for 'sawyer_reach_push_pick_place', I want to solve a task 'reach-v1' where the start and goal positions are fixed using any RL algorithm like PPO. Yet, currently, the goal position is changed whenever 'env.reset()' is called. Because of this issue, I can't train my model. I'm not sure but, it seems this is related to 'reset_model' or 'random_init' in the 'envs/mujoco/sawyer_xyz/sawyer_reach_push_pick_place.py' from line 126.

Am I confused about something or is there any approach I can fix the goal position so that I solve a typical single task? 

Thank you in advance!

@HyeongYeolRyu please see the latest API changes on the `master` branch, as they are quite extensive and address your issue.

Using the new API, you can ensure that the goal position never changes by only calling `set_task()` once.

You still need to ensure that you use the MT-x version of the benchmark, which is fully-observable. The ML-x versions of the benchmark are partially-observable.

Please also note that members like `_state_goal` are private and not part of the public API. They can, and will, change or be removed arbitrarily and at any time.
Repository: Farama-Foundation/Metaworld
Issue: Remove useless codes

I noticed that there are more codes that need to be removed. For example this [environment](https://github.com/rlworkgroup/metaworld/blob/master/metaworld/envs/mujoco/sawyer_xyz/sawyer_pick_and_place.py) was not used in our benchmarking. Also, some scripts under the scripts folder are outdated.

We need to do another round of audit for this.

The codecov report is a good guide to which code is actually used: https://codecov.io/gh/rlworkgroup/metaworld/list/master/metaworld
Repository: Farama-Foundation/Metaworld
Issue: Wrong goal state in SawyerReachEnvV2
Root cause: Documentation Deficiency

Hi, I am using metaworld.ML1 for studying meta reinforcement learning. I find that in the SawyerReachEnvV2 environment, when I run 
`obs, reward, done, info = env.step(a)`,
 the `info['goal']` is not the correct goal, and it seems fixed to `[-0.1  0.8  0.2]`. 

I read your source code and I find that `env._state_goal` returns the true goal, and I think you should add `self.goal = self._state_goal` in the `step` function of SawyerReachEnvV2. I am not sure if this should be done in other envs.

By the way, I think for goal-conditioned envs, the `obs` returned from `step` function should be like this: (which is the same with openai gym)

```
{
'observation': robot joint state, end_effector pos, obj_pos, vel……,
'deseired_goal': deseired_goal,
'achieved_goal': achieved_goal
}
```

And I am also quite interested in how you trained these benchmarks. Can you release the training code and some results for these environments?

Hello Chongkai,

Thank you for using MetaWorld. To be clear, ML1 is *not* a goal conditioned environment. Accessing the goal in ML1, ML10, and ML45 benchmarks is intentionally impossible, because the meta-learning algorithm must adapt to the hidden goal. Please do not publish results on ML1, ML10, or ML45 that use goal information, because it would be disingenuous. The goal is accessible as the last three dimensions of the MT10 and MT50 (and soon MT1: #165 ) benchmarks, but we also don't generally consider those to be goal conditioned, and thus do not returned the `achieved_goal`.

Unfortunately we're unable to release the training code originally used for these benchmarks, as it comes from multiple different sources, and uses a version of MetaWorld that has never been published. We have been able to reproduce the multi-task and meta-RL results using the [Garage](https://github.com/rlworkgroup/garage) RL library, so please consider that library if you are interested. Results after the recent redesign (and for V2 environments) are not currently available, but providing them is a priority in our group.

Hopefully that answer your question, so I'm going to close this issue, but feel free to re-open if you have further questions.

Best, K.R.

Thanks! It helps me to better understand the mechanism of meta learning.
Repository: Farama-Foundation/Metaworld
Issue: Delete info['goal'], rename self.goal, goal_space, and self._state_goal

People keep expecting these sources to be useful for goal conditioning, but MLxx are not goal conditioned benchmarks. (See, most recently, #168)

To what do you want these renamed?

I think `info['task_context']`, `self.__configuration_space` would be good names. `self._goal` should be private anyway but it can be `self.__configuration`. 

Is there any way to get rid of `self._state_goal`? Perhaps you could rename it `self.__target_state`?

Given how much trouble we've had with people trying to use these, I suggest you use double-underscore (which makes them even harder to access).
Repository: Farama-Foundation/Metaworld
Issue: Find better place for stats on V2 policies in V1 envs

See https://github.com/rlworkgroup/metaworld/pull/154#discussion_r459213695

As Meta-World ownership has been transferred from the RLWorkGroup to [Farama Foundation](https://farama.org/), I am closing this issue. If there are any questions or requests for features please join our [Discord](https://discord.gg/PfR7a79FpQ)
Repository: Farama-Foundation/Metaworld
Issue: Success definition
Root cause: Documentation Deficiency

Hi!
1) How do you define a "success" in an environment? Does the success metric have to be true for a single time step (i.e. at least once env_infos["success"] > 0) or be true at the end of the episode (after 150 steps in most environments) or something different?

2) How do you define the "success rate"? As something like "test_tasks_with_success / all_test_tasks"?

Sorry if this is mentioned somewhere, but could not find a clear definition in code or paper.

Thanks!

`success` is a property of an episode. An episode is successful iff at least time one step in the episode satisfied `env_info['success'] == True`.

The success signal provided in env_infos doesn't latch-up like this -- it is computed per-step.

See https://github.com/rlworkgroup/garage/blob/master/src/garage/_functions.py#L143 for sample evaluation code
Repository: Farama-Foundation/Metaworld
Issue: Update all environments to new XML
Root cause: Unimplemented Feature Gap

If either the `sawyer_foo_bar.py` environment file or the `sawyer_foo_bar_vX_policy.py` policy file require changes (other than new getters for renamed XML properties), we should copy them to a v2 file, commit the copy, and then make changes. If v2 already exists, just use that (no need to create v3).

Expect to have to increase steps in `_reset_hand` from 10 to 50 (to ensure Sawyer starts in the correct orientation). This requirement probably means that *every* environment will end up getting a v2 version.

When finished, if both v1 and v2 exist, keep both in the `test_scripted_policies.py` script. Success rates should not decrease.

- [x] SawyerNutAssemblyEnv
- [x] SawyerBasketballEnv
- [x] SawyerBinPickingEnvV2
- [x] SawyerBoxCloseEnv
- [x] SawyerButtonPressEnv
- [x] SawyerButtonPressTopdownEnv
- [x] SawyerButtonPressTopdownWallEnv
- [x] SawyerButtonPressWallEnv
- [x] SawyerCoffeeButtonEnv @haydenshively 
- [x] SawyerCoffeePullEnv @haydenshively 
- [x] SawyerCoffeePushEnv @haydenshively 
- [x] SawyerDialTurnEnv
- [x] SawyerNutDisassembleEnv
- [x] SawyerDoorEnv
- [x] SawyerDoorCloseEnv
- [x] SawyerDoorLockEnv
- [x] SawyerDoorUnlockEnv
- [x] SawyerDrawerCloseEnv
- [x] SawyerDrawerOpenEnv
- [x] SawyerFaucetCloseEnv
- [x] SawyerFaucetOpenEnv
- [x] SawyerHammerEnv
- [x] SawyerHandInsertEnv
- [x] SawyerHandlePressEnv
- [x] SawyerHandlePressSideEnvV2
- [x] SawyerHandlePullEnv 
- [x] SawyerHandlePullSideEnv
- [x] SawyerLeverPullEnvV2
- [x] SawyerPegInsertionSideEnvV2
- [x] SawyerPegUnplugSideEnv
- [x] SawyerPickOutOfHoleEnv
- [x] SawyerPickPlaceEnvV2
- [x] SawyerPickPlaceWallEnvV2
- [x] SawyerPlateSlideEnv
- [x] SawyerPlateSlideBackEnv
- [x] SawyerPlateSlideBackSideEnvV2
- [x] SawyerPlateSlideSideEnv
- [x] SawyerPushBackEnv
- [x] SawyerPushEnvV2
- [x] SawyerPushWallEnvV2
- [x] SawyerReachEnvV2
- [x] SawyerReachWallEnvV2
- [x] SawyerShelfPlaceEnv
- [x] SawyerSoccerEnv
- [x] SawyerStickPullEnv 
- [x] SawyerStickPushEnv
- [x] SawyerSweepEnv
- [x] SawyerSweepIntoGoalEnv
- [x] SawyerWindowCloseEnvV2
- [x] SawyerWindowOpenEnvV2