Repository: Farama-Foundation/Metaworld
Issue: Missing one-hot vector
Root cause: Documentation Deficiency

Hello :)

The documentation mentions that `MT10 and MT50 augment environment observations with a one-hot vector which identifies the task.`. When I create a MT10 instance (using the code below) I do not get the task id. Could you please explain what am I missing.


```
import metaworld
import random

mt10 = metaworld.MT10() # Construct the benchmark, sampling tasks

training_envs = []
for name, env_cls in mt10.train_classes.items():
  env = env_cls()
  task = random.choice([task for task in mt10.train_tasks
                        if task.env_name == name])
  env.set_task(task)
  training_envs.append(env)

for env in training_envs:
  obs = env.reset()  # Reset environment
  a = env.action_space.sample()  # Sample an action
  obs, reward, done, info = env.step(a)  # Step the environoment with the sampled random action
```

The shape of `obs` is `(12,)` and not `(13,)`

@avnishn 

Hi @shagunsodhani,

Thanks for checking out Metaworld! We recently updated Metaworld to a **new API that by default does not append task one-hot ids to the observations of environments.** Even though we had appended task one-hot ids to observations when we originally ran metaworld, we received the feedback that this design constricted researchers when they went to implement their own multi-task algorithms. Researchers were looking for ways to use their own task-ids.

That being said, I think there should some out of box solution for appending one-hot ids to observations from Multi-task environments. **We (the Metaworld maintainers) will discuss whether we should have some tooling in the Metaworld package that allows to users to append task one hot ids to the observations of environments.**

Lastly,

I noticed that you said

> The shape of obs is `(12,)` and not `(13,)`

This behavior is correct, however if you were to append a one-hot to the observation, its length would never be `(13,)`.

One hots have the shape of `(num_environments,)` where `num_environments` is the number of environments in a multi task benchmark. They take the form `[0,0,0,0,....,1, 0, 0] where there is a single 1 in the place in the one-hot to represent the task number, and 0s in all other positions. This would mean in MT10, if we were using task 1, where task 1 corresponds to some environment in MT10, the one-hot could be [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]. 

For `MT10` and `MT50`, `num_environments` would be 10 and 50 respectively, making the shape of observations from the those MT benchmarks `(22,)` and `(62,)`

**While we're developing a solution to your problem, feel free to use [this gym wrapper](https://github.com/rlworkgroup/garage/blob/master/src/garage/envs/multi_env_wrapper.py) from our sister package, Garage.**

It contains enough functionality to append one-hots to environments.

Please, let me know if you have any more questions!
best,
Avnish

Thanks for the answer. Could you please point me to the commit where the code is aligned with the benchmark proposed in the paper. While it is useful to have an ever-evolving benchmark, it makes it quite hard to replicate other's results and to build on them.

And if there is a more recent, stable, version of the benchmark, please point me to that as well :)

@shagunsodhani the current code on master is the most faithful to the original paper. Much of the improvements in the past year have been around enhancing reproducibility of the original results.

@ryanjulian Maybe I am missing something obvious but I see some clear differences:

* ` The observation space is represented as a 3-tuple of either the 3D Cartesian
positions of the end-effector, the object, and the goal, or the 3D Cartesian positions of the endeffector, the first object, and the second object, and is always 9 dimensional.` - I think the obs space is 12 dim on the master version.

* `The positions of objects and goal positions are fixed in all tasks in this evaluation, so
as to focus on acquiring the distinct skills, rather than generalization and robustness.` - This does not hold in the current version imo.

Please do not take this as some sort of criticism. I am thankful that your team open-sourced the benchmark. I have been trying to reproduce the results for a few weeks now and some of these differences mean I can not just reuse the params mentioned in the paper. 

I think improving the benchmark is an imp thing but knowing which version of the benchmark corresponds with the paper will be helpful as well.



Hey guys - could you please provide some help here :)

Hi @shagunsodhani sorry for not answering your follow up questions earlier. I am not an original author, however I am now a maintainer and I'll do my best to answer your 3 inquiries.

1) the observation space is in fact of the shape (12, ) now. And it technically always was, allow me to explain: 

The observation space before was (xyz-sawyer, xyz-object, xyz-goal) for most environments. The XYZ goal would be 0ed out for the metarl benchmark. In practice, we did however always pad the observations of these environments to match environments of the form (xyz-sawyer, xyz-object1, xyz-object2, xyz-goal), and would 0 out the entries of the observation belonging to xyz-object2 if there was no 2nd object in those environments.

2) the statement that goal and object states are held to be the exact same is true when one uses the Multitask-RL benchmarks, but not the Meta-RL benchmarks. In those benchmarks Meta-RL benchmarks every time set task is called on an environment it's initialization will change (goal and initial object position)

3) the current version of the benchmark is the most stable and most true to the original paper. We don't have access to the version that was originally used to run for the paper under this repo, and are planning to update the results on the publicly available Metaworld.



Hi @shagunsodhani I'm going to close this issue for now because I believe that I have answered your question. Please do @ me and reopen this issue if you have more questions :)

Thanks for all the replies @avnishn . I will get back if I have any other questions.