Repository: Farama-Foundation/Metaworld
Issue: Evaluation Protocol
Root cause: Documentation Deficiency

Hello,

I have some questions about the evaluation presented in the paper. Although this is not the best place for questions related to the paper, I think it could help to reproduce some experiments.

1) How many samples are used to train individual policies to reproduce Figure 6 ("Single Task Success Rate")? If this is different for each task, could you provide ate least an average number of samples?

2) Could you please provide the number of samples used to train the policies in Figure 7 and 8, in order to obtain the presented asymptotic performance?

3) Do you have the code used to those evaluations released? If so, could you provide them? I am referring to the methods themselves, such as SAC/PPO and MAML/PEARL/RL2.

Thank you so much for the support and for providing us this benchmark!



Hi,

1. We train each individual policy for 2M samples. However, for easy tasks, it can be learned fairly quickly, e.g. <200K samples.

2. For Figure 7, maybe @zhanpenghe can answer? For Figure 8, the asymptotic performances of MT10, ML10, and ML45 have already been reached as shown in the plots while the asymptotic performance of MT50 is reached with 45M samples.

3. We use SAC code in https://github.com/rail-berkeley/softlearning, PPO code in https://github.com/rlworkgroup/garage, MAML and RL2 code in https://github.com/jonasrothfuss/ProMP/tree/full_code, and PEARL code in https://github.com/katerakelly/oyster.

Hope this helps!

@luckeciano We used the versions mentioned by @tianheyu927 , but have not released the exact code used to train these codebases with metaworld. I can probably offer some help with this if you email me at <first_initial><last_name><at>usc.edu

I'm going to close this issue for now. Please feel free to reopen (or open a new one) if you have more questions!

Hey, 

Thanks for the replies. It has cleared my questions.

I have two new questions about ML10/45 and paper results.

1) So, in these benchmarks, is there goal variation across episodes?

As an example, lets say I sample reach-v1 task in ML10. In the official benchmark, should I expect to have constant goal during all episodes OR the goal can change after each reset?

2) In the paper, specifically in Figure 6 ("Single Task Success Rate"), is the goal constant for each task? Or is there goal variation during learning as well? 

@luckeciano You can see the goal and starting-state variation configurations in the ML*/MT* files in the repository: https://github.com/rlworkgroup/metaworld/tree/master/metaworld/benchmarks

In summary
| Benchmark | Starting State | Goal  |
|---|---|---|
| ML1 | Fixed | Random (50 unique chosen once) |
| ML10 | Random | Random (1 per-task chosen once) |
| MT10 | Random| Fixed |
| ML45 | Random | Random (1 per-task chosen once) |
| MT50 | Random | Fixed |

Hello!

I have a small question regarding how you count the number of samples you train on in the case of MAML. Do you count as samples 1) only the samples used for the outer loop update (_D'i_ ) or 2) also the samples generated in inner loop for the fast adaptation ( _D + D'i_ ) ?

![image](https://user-images.githubusercontent.com/9952015/81549725-7e75cb00-937f-11ea-982d-18ecdf367bc1.png)

I am trying to reproduce the results of your paper and it seems MAML requires much more samples than 2M. The following figure counts samples using the 2) case with the default hyper-parameter values you mention in your paper and `outer learning rate=0.1` . Could you also mention what value you used for the outer_lr? I couldn't find it in your paper.

![image](https://user-images.githubusercontent.com/9952015/81550390-69e60280-9380-11ea-9463-1b92f62d8fd9.png)

Congratulations for setting up this environment! It is a great contribution to the Meta-RL research! 

If you want me to open a new issue about this or send an email let me know and I can remove this comment.

hi @Kostis-S-Z -- sorry for my delay in replying.

The proper way to count TotalEnvSteps is to include all calls to `step()` generated by the algorithm during training, so outer+inner loop.

I think there's an error in paper plots, and the x-axis is off by at least an order of magnitude. The plot you show here is in line with my experience. Most of the meta-RL algorithms take 40-50M steps to converge on ML1 (assuming random_init=True)