Repository: Farama-Foundation/Metaworld
Issue: Reproducing single-task PPO results
Root cause: Documentation Deficiency

Hello,

First of all, thanks for working on this cool benchmark!

I've been trying to reproduce single-task results with PPO, on 10 single tasks from MT10 benchmark. However I haven't been able to reproduce the results and also there's been a lot of issues with stability of the training.

I took the code from https://github.com/rlworkgroup/garage/blob/master/examples/torch/mtppo_metaworld_mt10.py , and modified it so that it runs single tasks. As for hyperparameters, I tried several options:
- default setting from the script;
- setting from the paper, Appendix E.2,
- my own setting, derived from experimentation. The clear changes comparing to setting from the paper are: very big batch size (32000), and turned-off entropy regularization (set to 0.0)

The paper reports (Figure 6) that PPO solves (success rate close to 1) 9 out of 10 of these environments. However, with my best settings, I managed to:
- solve 4/10 envs (success rate reaches ~1 and stays there)
- pseudo-solve 4/10 envs (success rate reaches ~1 but then drops or oscillates); example for the task `button-press-topdown-v1` below:
![Screenshot 2020-06-23 at 16 24 38](https://user-images.githubusercontent.com/4266489/85415818-0d285b00-b56e-11ea-8acb-387f6b77a0be.png)
- for 2 tasks I did not manage to get close to 1.
The results I report here come from my own settings of hyperparameters, but for the settings from the paper the results were not better.
All runs were trained for > 6M environment steps.

So could you please give me any hints on reproducing PPO results?
Also, did you see such instabilities in training for some of the environments? (in single task setting)
Also, how the results in Figure 6 in the Appendix C were produced? Was there 1 training run for each environment or more? How were they aggregated?

Thanks!

Hi @zajaczajac, thanks for checking out metaworld!

I want to address a few things first:
1) MTPPO is relatively unstable
2) I think that if an algorithm is able to achieve a success rate of 1, then its not guaranteed that PPO is able to train the policy in later iterations to keep the success rate at 1. This is partially because MTPPO is an unstable. I should also say that most Multi-Task RL algorithms are relatively unstable. After all, if I've trained a policy to approach a success rate of 1, and its performance later drops due to instabilities in the algorithm that I'm using for training, then why not keep the policy from the training itr that had the high success rate.
3) We definitely experience instabilities in the current set of environments when training using single task algorithms (such as SAC and PPO), *depending on seed*. A single seed isn't enough to report the stability of an algorithm on an environment. Try 3-10 seeds, and see if you experience the same results.
4) To answer your question of how results were produced in Figure 6 of Appendix, this kinda goes back to what I was saying in 2. The single task algorithms were trained on the individual environments, and then the max avg success rate observed was reported.

I hope this answers your questions :)

Since this isn't necessarily an issue, I'm going to close this for now. Feel free to continue communicating over here however!