Repository: Farama-Foundation/Metaworld
Issue: Some question about metaworld environment.
GitHub labels: question

I'm interested in your MetaWorld.
It will be good benchmark about Meta-RL.
I have some questions about this benchmark.

1. When start at environment, sawyer arm move to specific pose during first K steps.(K is about 10)
It seems like 'init' function called by mujoco.
I think this will cause problems when agent do reinforcement learning.
Is this intended?

2. Reward gap is big between some environment.
When reach env get almost 100 at first time, push env get small reward(between 0 and 1).
Is this intended?





Thank you for your hard work.
Let me ask you two more question.

3. 
The environment codes(sawyer_drawer_close.py, etc.) contain the function 'compute_rewards(self, actions, obsBatch)' and has the commnet 'Required by HER-TD3'.
As far as I know, the HER algorithm has set another goal using the last state of the trajectory.
However, setting another goal does not exist in this function.
Instead, this function computes rewards using self._state_goal. (It is not different from normal reward computing function which using in interaction between agent and environment)
Is it intended? If that be so how can I use this function for HER algorithm?

4.
It seems like state does not contain the gripper state which is open or close.
I think agent need to know this information because he doesn't know whether he's caught or not.
Is it intended?

@zhanpenghe 

Hi @pigranya1218 

1. We haven't found this to cause any problems when training single-task or multi-task policies. Can you be more specific about the problem you are describing? Perhaps you can link to some code?

2. We are aware that the reward functions have huge changes in magnitude within the same task, and large differences between tasks. We didn't do this to make the benchmark artificially hard, we just engineered reward functions until each environment can train with a single-task RL algorithm. We believe this is a realistic simulation of a real multi-task learning problem, in which it is very difficult (and sometimes intractable) to hand-engineer dense reward functions which satisfy inter and intra-task normalization. Note that you are free to add a normalization technique to your algorithm to deal with this problem. In fact, if you find a general one, I believe that's a notable research contribution to MTRL. You are also free to ignore the reward functions entirely and instead using the sparse `'success'` signal provided in `env_info` instead;.

3. Metaworld is not a goal-conditioned benchmark, and we don't support accessing, mutating, etc. the task-specific goals at this time. In the future, we may add a goal-conditioned version. Some tasks may keep an internal state called a "goal," "state_goal," etc. but sometimes this goal is not actually the final state of a successful task, sometimes it contains more information (or less) than needed to complete the task, etc. You could still apply HER to metaworld by somehow extracting goal positions from regular state observations, but there is intentionally no interface for asking the environment what a successful state (goal) is.

4. Can you point to a code snippet reproducing this?

Thank you for your apply. 

1. This question was just curiosity. thanks.

2. It is same above.

3. I understand. Thank you.

-----------------------------------
4.

As far as I know, MetaWorld action means (end-effector pos move(3), gripper move(1))

my test code.

```
from metaworld.benchmarks import ML1

env = ML1.get_train_tasks('pick-place-v1')
tasks = env.sample_tasks(1)
env.set_task(tasks[0])

s = env.reset()
step = 0
while True:
    print(f'step {step} : {["%.10f"%st for st in s]}')
    step += 1
    env.render()
    s, _, done, _ = env.step([0, 0, 0, 1] if step > 120 else [0, 0, 0, 0])
    if done:
        break
```

and state printed in terminal
```
...
step 113 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 114 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 115 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 116 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 117 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 118 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 119 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 120 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 121 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 122 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 123 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 124 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 125 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 126 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 127 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 128 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 129 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 130 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 131 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 132 : ['0.0000000000', '0.6000270231', '0.1995718118', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 133 : ['0.0000000127', '0.6000268332', '0.1995718804', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 134 : ['0.0000000169', '0.6000261844', '0.1995721087', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 135 : ['-0.0000000238', '0.6000259190', '0.1995721756', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 136 : ['-0.0000000811', '0.6000266110', '0.1995718869', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 137 : ['-0.0000000588', '0.6000272799', '0.1995716341', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 138 : ['-0.0000000049', '0.6000275432', '0.1995715666', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 139 : ['0.0000000328', '0.6000275468', '0.1995716084', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 140 : ['0.0000000449', '0.6000274461', '0.1995716811', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 141 : ['0.0000000399', '0.6000273261', '0.1995717447', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 142 : ['0.0000000279', '0.6000272227', '0.1995717872', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 143 : ['0.0000000155', '0.6000271464', '0.1995718101', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 144 : ['0.0000000060', '0.6000270956', '0.1995718194', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 145 : ['0.0000000000', '0.6000270641', '0.1995718210', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 146 : ['-0.0000000030', '0.6000270454', '0.1995718193', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 147 : ['-0.0000000041', '0.6000270348', '0.1995718167', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 148 : ['-0.0000000040', '0.6000270289', '0.1995718143', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
step 149 : ['-0.0000000033', '0.6000270258', '0.1995718125', '0.0399984124', '0.6762223672', '0.0149963212', '-0.0813703599', '0.8250478189', '0.1446661658']
```
State is 9-dimension because I modified the code.(obs_type='with_goal').
It looks like there are no state difference over 120 step though grip action acted.
Very small difference is found since 132 step~, but I think it is just changed by X pos which made by gripper closing, whether gripper is closed or open.
I think that agent doesn't know using state the object is gripped. (fortunately the object is moved by gripped then agent can know, but there are some unfortunate case)

-----------------------------------------

+5) one more question!
A few weeks ago, I received an e-mail saying the MTRL code used in experiments will be provided in a few weeks.
When will it be provided? I think this code will be good baseline code to learn MTRL.

Thanks. 

@pigranya1218 are you saying (1) there's nothing in the state telling the agent that the object is gripped, or (2) nothing in the state telling the agent the state of the gripper is open/closed?

Please email me at <first_initial><last_name><at>usc.edu for more info on MTRL code used for experiments.

Thanks for reply!
I sent email to you.

----
I said (2).



@tianheyu927 can you comment on the gripper state?

@pigranya1218 the gripper open/close state is not present in the observation space, but we have not found it required to solve the environments. For proof, you can take a look at the scripted policies in `metaworld.policies`

For instance if you have an object position and gripper position, and need to know if you've grasped the object, you can infer that based on the distance between gripper and object.