zhiminy/EvalEng huggingface dataset:
Repository: Farama-Foundation/Metaworld
Issue: off screen render
Root cause: Unimplemented Feature Gap

Hi,
thanks for sharing this amazing project!

I'm trying to use env.render(mode = 'rgb_array') to return images on a server.
[File "/h/haoyux/metaworld/metaworld/envs/mujoco/mujoco_env.py", line 131, in _get_viewer
    self.viewer = mujoco_py.MjViewer(self.sim)
  File "/scratch/gobi1/haoyux/gaifo_ae/lib/python3.6/site-packages/mujoco_py/mjviewer.py", line 137, in __init__
    super().__init__(sim)
  File "/scratch/gobi1/haoyux/gaifo_ae/lib/python3.6/site-packages/mujoco_py/mjviewer.py", line 28, in __init__
    super().__init__(sim)
  File "mujoco_py/mjrendercontext.pyx", line 311, in mujoco_py.cymj.MjRenderContextWindow.__init__
  File "mujoco_py/mjrendercontext.pyx", line 46, in mujoco_py.cymj.MjRenderContext.__init__
  File "mujoco_py/mjrendercontext.pyx", line 102, in mujoco_py.cymj.MjRenderContext._setup_opengl_context
  File "mujoco_py/opengl_context.pyx", line 44, in mujoco_py.cymj.GlfwContext.__init__
  File "mujoco_py/opengl_context.pyx", line 64, in mujoco_py.cymj.GlfwContext._init_glfw
mujoco_py.cymj.GlfwError: Failed to initialize GLFW](url)

I'm wondering how to do off-screen render.
(e.g., robosuite has an option of offscreen render, so you don't need to open a glfw window to call render)

Thanks!

@haydenshively may be able to help with this. 

It looks like the robosuite implementation dynamically replaces the render with an off-screen instance here: https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/environments/base.py#L222

Note that I think you can render images using the default render if you run metaworld using a fake display server, e.g. Xvfb.

You can see an example here in the garage Dockerfile: https://github.com/rlworkgroup/garage/blob/master/docker/entrypoint-headless.sh#L18

I'm not 100% sure whether this will work on a headless machine (though I suspect it will), but we use an offscreen rendering mode in `scripts/scripted_policy_movies.ipynb` -- the line looks like this:

```
yield r, done, info, env.sim.render(*res, mode='offscreen', camera_name=camera)
```

I think you should be able to adapt this to your needs. If not, let us know and we'll be happy to help more

thanks for your reply!
I solved this by:

[	def _get_viewer(self, mode):
		self.viewer = self._viewers.get(mode)
		if self.viewer is None:
			if mode == 'human':
				self.viewer = mujoco_py.MjViewer(self.sim)
			elif 'rgb_array' in mode:
				self.viewer = mujoco_py.MjRenderContextOffscreen(self.sim, device_id=-1)
			self.viewer_setup()
			self._viewers[mode] = self.viewer
		# if mode == 'rgb_array_y':
		#     self.viewer_setup(view_angle='y')
		# else:
		#     self.viewer_setup(view_angle='x')
		#self.viewer_setup()
		return self.viewer](url)


in /metaworld/metaworld/mujoco_env.py
Repository: Farama-Foundation/Metaworld
Issue: How many 'shots' per task

Hi there

I wondered if you could provide me with some details about the dataset:
1. How many 'shots' per task are used when adapting the model?
2. How many total samples/observations are in the dataset?
3. How many samples/observations are there per manipulation task in the dataset?

Thanks

@avnishn 

Hi @danielamassiceti thanks for checking out Metaworld!

Though Metaworld could be thought of as a dataset, it is really actually a set of environments that an agent will interact with during its training process.

Let me try to answer your original questions to the best of my ability:

> How many 'shots' per task are used when adapting the model?

This is entirely up to you, there is no specific requirement by the simulator. You can use any amount of trajectories when doing meta-adaptation.

> How many total samples/observations are in the dataset?

Technically infinite. Since we're using a set of simulators, you can step the simulators an infinite number of times, although with Metaworld-v1, you'll need to train for about 20m timesteps for ML10 and MT10 and 100m timesteps for ML45 and MT50.

> How many samples/observations are there per manipulation task in the dataset?

Technically infinite. This is also up to the user, since when using a simulator, the amount of available data is dependent on how much you'd like to step the environment. The only limitation here is that each environment has a different max-path-length [Max path length means the maximum amount of times that a simulator can be stepped before it needs to be reset to its initial position, by calling env.reset()].

For each manipulation task, there are however 50 environment configurations (initial object and goal positions), which are found under MLXX.[train/test]_tasks, and while users aren't required to use all of them during training, many should be used during meta-adapt to display sensitivity of an agent's meta-adaptation to different environment configurations. These can be changed by making a call to env.set_task() using a corresponding task from MLXX.[train/test]_tasks.

Did this answer your questions? Please let me know/ask as many questions as you would like, here by @ing me, or on our slack community, which you can join via the sign-up link in our README.

Best,
@avnishn 



I should note that though these are all true of the MetaWorld environments, the relevant hyperparameters (e.g. how many trajectories uses for adaptation) used for the results appearing in the MetaWorld publication are all in the appendix.

I'm going to go ahead and close this for now, but feel free to reopen if you have any additional questions @danielamassiceti!
Repository: Farama-Foundation/Metaworld
Issue: Pre-trained policies for each env
Root cause: Unimplemented Feature Gap

I emailed this, but also seems appropriate to post here. We want to do some research that builds on top of your excellent code. I am guessing [this gif figure](https://meta-world.github.io/figures/ml45-1080p.gif) was made using trained policies for each env. Any chance you could provide the trained policies for each individual env to aid with research that uses them? The code to generate gifs of a rollout for a would also be nice, if you can share it.


hi @andreykurenkov -- sorry I didn't respond over email due to the holiday week in the US.

In short, we don't have ready-trained policies available for each env yet. The GIF in the paper was generated using policies trained with https://github.com/rail-berkeley/softlearning.

We don't have code for autogenerating the composite GIF (it was composed by hand), but if you make some that would be a cool addition in a PR.

I'll leave this issue open, since I think it would be a worthy addition from a contributor.

Got it, thanks! If we do end up training policies we'll try to contribute :)

@andreykurenkov we don't have pre-trained policies for each env, but we do have (soon-to-be-complete) scripted policies for each environment. Do those a satisfy your use case?

That would be quite useful (though I no longer have a need for this, at least for now -- i'll go ahead and close the issue)
Repository: Farama-Foundation/Metaworld
Issue: Gripper status not reflected in observation space?
Root cause: Documentation Deficiency

In my experiments with v1 and v2 tasks, as far as I can tell, the degree of open/closedness of the gripper isn't reflected in the 12-dimensional observation space. I imagine this makes many of the tasks significantly more challenging (e.g. trying to grip an object without observing your own gripper). I'd love to understand the intention behind this design choice.

Hi @mhauskn, thanks for checking out Metaworld!

Your observation is correct about the v1 environments, however, in the v2 environments, we updated them to have observation spaces that include a measurement of how open/closed the gripper is. 

We haven't updated any of the benchmarks to use the newest v2 environments, but we'll be doing that soon (in the next week or so) as apart of a large overhaul that we're completing with Metaworld!

Some things that we are going to update include:

- updated observation spaces to include measurements relevant to making all environments markovian.
- updated meshes for a new updated look
- updated reward functions that should be much easier to solve.

For now, if you want to toy with the v2 environments that we'll be releasing, you'll have to use them directly on the branch titled `new-reward-functions`. Keep in mind that we haven't merged all of the new reward functions as it's a WIP, but all of the updated ML10 and MT10 environments are on there.

Feel free to @me (@avnishn) if you have any more questions!


Thank @avnishn - I'll give the ```new-reward-functions``` branch a try. Out of curiosity is there an ETA for v2 release?

@mhauskn expect it to be out in the next 2 weeks or so. We're very close to finishing it.
Repository: Farama-Foundation/Metaworld
Issue: How to save image of a camera with the scene rendered?
Root cause: Documentation Deficiency

Hi, I am using keyboard control to collect demonstrations for imitation learning. I want to know if there is anyway to save image when I am rendering the scene (because I need to see the environment to perform keyboard control). 

I only find this method in the repo:
`env.sim.render(*resolution, mode='offscreen', camera_name=camera)[:,:,::-1]`

It works well when I am not rendering the scene. But when I use `env.render()` at the same time with this method, it will output an error:
```
  File "mjsim.pyx", line 156, in mujoco_py.cymj.MjSim.render
  File "mjsim.pyx", line 158, in mujoco_py.cymj.MjSim.render
  File "mjrendercontext.pyx", line 45, in mujoco_py.cymj.MjRenderContext.__init__
  File "mjrendercontext.pyx", line 113, in mujoco_py.cymj.MjRenderContext._setup_opengl_context
  File "opengl_context.pyx", line 130, in mujoco_py.cymj.OffscreenOpenGLContext.__init__
RuntimeError: Failed to initialize OpenGL
```

If I change the mode arguments to 'window', it outputs another error:
```
TypeError: 'NoneType' object is not subscriptable
```

Can you give me some advice? Thank you!

Hi @HeegerGao,

Please try replacing the `render` function of `mujoco_env.py` with the following:
```
    def render(self, mode='human'):
        if mode == 'human':
            self._get_viewer(mode).render()
            return self.sim.render(
                *self._rgb_array_res,
                mode='offscreen',
            )[:, :, ::-1]
        elif mode == 'rgb_array':
            return self.sim.render(
                *self._rgb_array_res,
                mode='offscreen',
                camera_name='topview'
            )[:, :, ::-1]
        else:
            raise ValueError("mode can only be either 'human' or 'rgb_array'")
```

Then use `mode='human'` when you call `env.render`. This will cause mujoco to switch back and forth between the viewer (in which you should have keyboard control) and the rgb_array renderer. The function will return the pixels which you can then save using `cv2`. This works on my machine, but it seems to be pretty non-standard behaviour, so let me know if you have any trouble. Good luck!

I'm going to go ahead and close this for now @HeegerGao but please feel free to reopen it!
Repository: Farama-Foundation/Metaworld
Issue: Mujoco 2.0.0
Root cause: External Dependency Breakage

Does `metaworld` require mujoco 2.0.0? Or is there a way I can get it to work with the 1.5.x versions?

The code has been run with MuJoCo 1.5.0 before and that may still be possible, but we can't guarantee it will work.

MuJoCo 2.0 added details to the friction model that MuJoCo 1.5.0 does not have, and those might affect task success. So any benchmark results using MuJoCo 1.5.0 should not be compared with results using MuJoCo 2.0.
Repository: Farama-Foundation/Metaworld
Issue: Cannot use gym.make with external code without error
Root cause: Validation Gap

I am trying to get this package integrated with my own. I figured I could just add "from metaworld.envs.mujoco import *" at the top of a file and do gym.make to create a metaworld env with a particular name, since the __init__ of that package registers all these envs, but that results in this error:

> ...
>     self.env = gym.make(self.config.robot.task.name)
>   File "/cvgl2/u/andreyk/projects/manipxonomy/env/lib/python3.5/site-packages/gym/envs/registration.py", line 183, in make
>     return registry.make(id, **kwargs)
>   File "/cvgl2/u/andreyk/projects/manipxonomy/env/lib/python3.5/site-packages/gym/envs/registration.py", line 125, in make
>     env = spec.make(**kwargs)
>   File "/cvgl2/u/andreyk/projects/manipxonomy/env/lib/python3.5/site-packages/gym/envs/registration.py", line 89, in make
>     env = cls(**_kwargs)
>   File "/cvgl2/u/andreyk/projects/manipxonomy/metaworld/metaworld/envs/mujoco/sawyer_xyz/sawyer_pick_and_place.py", line 43, in __init__
>     **kwargs
> TypeError: __init__() got multiple values for keyword argument 'hand_high'
> 

That actually makes total sense, since in registrations the kwargs are specified as such:


```
    register(
        id='SawyerPickupEnv-v0',
        entry_point='metaworld.envs.mujoco.sawyer_xyz'
                    '.sawyer_pick_and_place:SawyerPickAndPlaceEnv',
        tags={
            'git-commit-hash': '30f23f7',
            'author': 'steven',
        },
        kwargs=dict(
            hand_low=(-0.1, 0.55, 0.05),
            hand_high=(0.0, 0.65, 0.2),
            action_scale=0.02,
            hide_goal_markers=True,
            num_goals_presampled=1000,
        )
)
```

And then these parameters are also set in the constructor:

```
class SawyerPickAndPlaceEnv(SawyerXYZEnv):
    def __init__(
            self,
            obj_low=None,
            obj_high=None,
            random_init=False,
            tasks = [{'goal': np.array([0.1, 0.8, 0.2]),  'obj_init_pos':np.array([0, 0.6, 0.02]), 'obj_init_angle': 0.3}], 
            goal_low=None,
            goal_high=None,
            hand_init_pos = (0, 0.6, 0.2),
            liftThresh = 0.04,
            rewMode = 'orig',
            rotMode='rotz',#'fixed',
            **kwargs
    ):
        self.quick_init(locals())
        hand_low=(-0.5, 0.40, 0.05)
        hand_high=(0.5, 1, 0.5)
        obj_low=(-0.5, 0.40, 0.05)
        obj_high=(0.5, 1, 0.5)
        SawyerXYZEnv.__init__(
            self,
            frame_skip=5,
            action_scale=1./100,
            hand_low=hand_low,
            hand_high=hand_high,
            model_name=self.model_name,
            **kwargs
)
```

It appears the kwargs are just outdated, should they just be removed? Happy to submit a quick PR with that change, it seems to fix the gym.make.

@andreykurenkov the only supported way for constructing metaworld envs is through the benchmarks interface in the README. You're right -- these registrations should be removed.
Repository: Farama-Foundation/Metaworld
Issue: Number of Experiments and Questions to Success Rate / Rewards

Dear all,

First of all, thank you very much for providing us with a nice benchmark like meta-world @tianheyu927 @zhanpenghe @ryanjulian!

I have a question about your experiment design:
1. How many experiments have you taken for ML1, ML10, and ML45?

Another thing is regarding the effect of Reward / Success Rate. The results you show are exciting.
For example, in the pick&place task, MAML gets a success rate of 0 but the highest reward.
2. I am wondering why that is the case? Can lead bad behavior to a high reward, but the success rate is low? If so, wouldn't we need to shape the reward function differently?

You described that you tried to make the reward function structurally fitting. But this seems odd to me. 
3. How would an agent learn the best policy if the success rate isn't somehow refracted in the reward function?

Any answer would be very appreciated!

![grafik](https://user-images.githubusercontent.com/1223186/104732642-d306ec80-573d-11eb-9d93-f5fec7a229a2.png)
![grafik](https://user-images.githubusercontent.com/1223186/104732708-e87c1680-573d-11eb-9da5-d1b7cfb6e907.png)


@avnishn 

Hi @janbolle,

Thanks for checking out metaworld! I am one of its maintainers and I am happy to answer your questions. 

1) meta rl and multi task rl algorithms are highly seed sensitive. You should run more than 1 seed of the same experiment on any of these benchmarks. I run 3 when I'm testing out ideas, and 5-10 when I run results. 

2) The success rates that you're seeing in the plots are reflective of our v1 environments. We have been working on v2 environments that include completely revamped reward functions. Specifically for pick-place, the new reward function we reduced the ability of the agent to reward hack (bad behavior leading to good rewards, like you said). 

You can check out those environments on our branch new-reward-functions. 

3) the reward function and success rates are based on physical quantities in the environment (such as position of the puck in pick and place). The new environments are written in such a way that the maximum reward is only achieved when we enter the successful state. Further more an environment can't observe its success rate. The success rate of an agent is computed by taking multiple episodes from that agent in an environment, and checking what fraction of those episodes entered a successful state. For pick and place it would mean how many episodes an agent successfully pick up the puck and place it at the goal location. If in 7 out of 10 episodes an agent was successful, then its success rate would be 70 percent. 

I hope this explanation explains why the success rate metric can't be directly incorporated into the reward. 

Have I answered all your questions? Please @ me, @avnishn , if you want to follow up, or join our slack community (sign up link in th e read me) for faster responses! 

Thanks!
@avnishn 

I'm going to go ahead and close this for now, but please feel free to reopen @janbolle
Repository: Farama-Foundation/Metaworld
Issue: A small mistake(or not) in the observation space for multi-class-multi-task environment.
Root cause: Validation Gap

It looks like that the observation-space definition of the multi-class-multi-task environment is kind of confusing (I am not sure whether it's a mistake or not).

For observation space with the obs_type is **with_goal_id** or **with_goal_and_id**, current code is useing **goal_id_low** for high in Box, and **goal_id_high** for low in Box.

```python
elif self._obs_type == 'with_goal_id' and self._fully_discretized:
    goal_id_low = np.zeros(shape=(self._n_discrete_goals,))
    goal_id_high = np.ones(shape=(self._n_discrete_goals,))
    return Box(
        high=np.concatenate([plain_high, goal_id_low,]),
        low=np.concatenate([plain_low, goal_id_high,]))
elif self._obs_type == 'with_goal_and_id' and self._fully_discretized:
    goal_id_low = np.zeros(shape=(self._n_discrete_goals,))
    goal_id_high = np.ones(shape=(self._n_discrete_goals,))
    return Box(
        high=np.concatenate([plain_high, goal_id_low, goal_high]),
        low=np.concatenate([plain_low, goal_id_high, goal_low]))
```

Besides, It looks like that in **observation_space** function the goal id is before the goal, which is not the case when the observation is augmented in **_augment_observation**

```python
if self._obs_type == 'with_goal_id' or self._obs_type == 'with_goal_and_id':
    if self._obs_type == 'with_goal_and_id':
        obs = np.concatenate([obs, self.active_env._state_goal])
    task_id = self._env_discrete_index[self._task_names[self.active_task]] + (self.active_env.active_discrete_goal or 0)
    task_onehot = np.zeros(shape=(self._n_discrete_goals,), dtype=np.float32)
    task_onehot[task_id] = 1.
    obs = np.concatenate([obs, task_onehot])
```

Thanks. This is indeed a bug. We will submit a fix soon.

@zhanpenghe did you open a PR for this?

@ryanjulian, @adibellathur has been working on this issue

@avnishn isn't this obviated by recent changes?

hi @RchalYang, since we last reviewed this issue, we have released a new API for Metaworld that we believe has fixed this issue. I'm going to close this for now, but feel free to reopen and comment if you have any more questions.
Repository: Farama-Foundation/Metaworld
Issue: Parallel Environments
Root cause: Unimplemented Feature Gap

Is there a straightforward way to run multiple environments in parallel or do you have to hack it using the wrapper from gym?

@rmrafailov we don't consider parallel sampling to be in-scope for this repository, since it is highly algorithm-dependent.

I have never tried the wrappers which gym recently started including, but you might take a look at https://github.com/rlworkgroup/garage/tree/master/src/garage/sampler for several well-tested parallel samplers.

Most samplers don't yet support sampling from multiple tasks at once, and will require a little bit of hacking to use with meta-RL.

I'm going to close this issue since I feel we've adequately addressed it. Please open a new one if you have further question, concerns, or improvements.
Repository: Farama-Foundation/Metaworld
Issue: How to judge whether the task has been successfully completed?
Root cause: Documentation Deficiency

There is no actual terminate in meta-world, so how to judge whether the task has been successfully completed? Can info['success'] be the basis for judging the success of the task?

Yes. If you OR together `info['success']` across all time steps of an episode, the result is the success/failure of the episode.

@zengrui324 you evaluate it by or-ing together info['succes'] for all timesteps in an episode/trajectory.
Repository: Farama-Foundation/Metaworld
Issue: Experiment parameters to reproduce PEARL results
Root cause: Documentation Deficiency

Hello all,

I am trying to reproduce the PEARL results from your paper.
Can you guide me to the place where you configured the experiments?

I try to figure out the experiment parameters...
How many different task instances per task did you use for ML10 and ML50?

I really appreciate any help you can provide.

@ryanjulian, @avnishn maybe you could help me here?
In PEARL the latent_variable size is really important. Which value did you set here?
I also struggle with '# of samples / # of train steps per iteration | 22500 env steps / 4,000 training steps'
How do you get the # of samples? In PEARL there are multiple parameters as num_initial_steps, num_steps_prior, etc.
If you used the PEARL implementation from Kate Rakelly* I really would appreciate it if you could make the config file available.

* https://github.com/katerakelly/oyster

hi @janbolle 

I have been trying to tune a pearl agent for use on benchmarking metaworld-v2 as well.
I haven't had a great amount of success in trying.

Here is my launcher file for ML10:
https://github.com/rlworkgroup/garage/blob/avnish-new-metaworld-results-mt1/metaworld_launchers/ml10/pearl_metaworld_ml10.py

I use the PEARL implementation that we wrote in our sister repo, https://github.com/rlworkgroup/garage.

I've modified it on the branch I linked to include lots of training and evaluation statistics.

The training of the encoder is what really matters with PEARL. You'll want to pay attention to the batch size for your encoder (larger is generally better), latent size (not to small, not to big), and the KL loss, which is used for training the encoder (I log this quantity in my modified PEARL implementation). 

Again, I really haven't had much success in tuning our PEARL implementation to succeed on the ML benchmarks. If you want to take a crack at it that would be great.

Thanks!
@avnishn 

@avnishn Thank you very much for your fast reply!
What does "haven't had a great amount of success in trying" exactly mean?
Do you have some charts you can share?

I definitely will share my parameters, if I can get something out of it.
Unfortunately, I wasn't aware of the PEARL garage implementation.
I added meta world to Kate's code and I'm planning to parallelize sampling of the environment to boost training time.
Maybe this is something which is also interesting for you.


please check out our V2 release on arxiv for the hyperparameters we officially used