Repository: Farama-Foundation/Metaworld
Issue: Allow setting the seed when sampling configurations
Root cause: Unimplemented Feature Gap

This is useful for reproducibility, but might also place a larger responsibility on us to ensure the goal sampling process doesn't change across versions.

I think we should also ensure that MuJoCo is properly seeded when we do this.

Could you please give a demo about how to use seed? Like in MT10, `task = random.choice([task for task in ml.train_tasks if task.env_name == name])` contains random, I suppose each task in `training_envs` also needs a seed. I have no idea how to set seed for them.
Thank you!

Hi @Banmahhhh! Thanks for checking out Metaworld.

We don't have a feature in Metaworld that actively allows you to do this at a moment, but here's a potential solution: In a nutshell you could seed the different random number generators used in Metaworld. In this case you'd want to seed the numpy random generator.

Here's an example where we do that in our sister directory, Garage:
https://github.com/rlworkgroup/garage/blob/fc3ddc6bb43aa98e6c092871d8e3aa9f77215955/src/garage/experiment/deterministic.py#L24

Hope this was helpful!
Avnish


Hey @avnishn just to make sure I did not misunderstand anything -- we can not set the seed for the benchmark currently and we should rely on seeing numpy/torch/python?

@shagunsodhani that is correct. This issue has been opened so that eventually we add the feature.

An extra point about this: If I may ask, what is the rationale behind randomly generating the goals each time? Do we lose something by having a fixed list of goals for each environment that don't need to be generated at runtime? This might sound pedantic, but I actually think it is important. Right now I'm trying to train agents for MT1, MT10, and MT50, and according to the benchmark specifications this means that the goals/object configurations should be fixed between episodes. However, since the goals are generated internally when the benchmarks are instantiated (in `_make_tasks()` from metaworld/\_\_init\_\_.py and `SawerXYZEnv._get_state_rand_vec()`), this creates problems when I want to do parallel training over multiple processes with the same goals. It would be nice to be able to seed processes differently (this is necessary to have separate processes sample different sequences of tasks for multi-task benchmarks), but if each process is seeded differently here, then each process has a different goal, which breaks the requirements of the MTX benchmarks.

I think I can get around this by setting the same numpy seeds over each process and seeding torch differently, so that the actions sampled from the distribution produced by the policy are different over each process and this will produce different trajectories (which is a critical part of multi-process training). This is a bit gross though and it relies on the fact that RNG for the model behavior (torch) can be seed differently than goal sampling (numpy), which isn't a guarantee in all situations. This also relies on the fact that other training behavior which should be different between processes (such as sampling a task for training) doesn't use np.random. If it does, then that behavior will be identical between processes.

Having a fixed list of goals for each environment would definitely allow us to decouple the goal values from other RNG and it would create consistency to allow for a more apples-to-apples comparison between training runs. There may be a downside but I can't really think of one. Any thoughts?

@mtcrawshaw we want randomly generated goals to generate as much goal diversity as possible across different seeds.

We think it allows for an overall better evaluation of multi-task and meta-rl algorithms than if we had a set of 50 unique tasks, which could potentially be skewed to have easier/harder goal/object_initial positions. 

But I hear you. It is definitely messy to have to seed numpy on your own, and then seed torch separately, which probably seeds numpy itself. It makes it confusing to keep track of which random seed that you're actually using.

So right now there is a PR up for seeding benchmark instances, #305, and how it works is that it seeds the numpy random number generator that is used to generate tasks in metaworld.

An issue that I forsee is that in seeding the numpy random number generator, we actually overwrite the random seed that a user was using in another part of their experiment. 

What we could potentially do is determine what the random seed that was being used prior to initializing the benchmark, and then reset the state of the numpy random number generator with that previous state, but after we have generated the metaworld task distribution.

It is a bit round a bout but it satisfies the need for the maximum amount of task diversity across different random seeds, while still preserving the prior seed, so that you don't have to go through the gross process of seeding torch and numpy differently. 

Instead upon creating an instance of the benchmark, you would do something like `benchmark(seed)` and you are free to always use the same seed in a single experiment. 

In terms of evaluation though, diversity in the sets of tasks is a feature that we would like to had, and I'm inclined to merge #305 after getting some more feedback from you.

Thanks!
@avnishn