Repository: Farama-Foundation/Metaworld
Issue: Some questions regarding to the paper

First thank you for the awesome simulation environment! And here are some of my questions:

1. Can I know the your computational resources to run the experiments, and the corresponding running time for each of the tasks.

2.  For the bottom three tasks of Figure 6, i.e., "push with wall", "disassemble nut" and "turn dial", the success rates are low. Are these tasks too hard to solve?

3.  Can I know the standard definition of "structrual similarities"? In order word, how do you measure the similarities between tasks?

4. Can you elaborate the reason why multi-head SAC performs so much better that SAC with single head?

5. It is surprising that PPO performs so much worse that TRPO in MT50, which seems to be contradict with the resuls from model-free RL. Do you have a good explaination for this?

6. The results in Table 1 and Figure 8 seem to be inconsistent. For example, in ML45, Table 1 says PEARL performs the best on meta-test. However, it is the worst according to Figure 8.

7. Can you provide some evidence for the claims "special care needs to be taken to ensure that the task cannot be inferred directly from the image, else meta-learning algorithms will memorize the training tasks rather than learning to adapt"?

When the authors have the time, would you mind answering the following question so that we can get an accurate understanding of the performance of current algorithms?

6. The results in Table 1 and Figure 8 seem to be inconsistent. For example, in ML45, Table 1 says PEARL performs the best on meta-test. However, it is the worst according to Figure 8.



HI there -- We usually use GitHub for questions/issues about software, not the paper.

This is a lot to answer, which is why you haven't gotten much of a response yet.

Are either of you attending the NeurIPS conference in Vancouver this week? That could be an opportunity to meet up. 

I can provide some partial answers to the original questions, as fast as I can. I apologize for the terseness of the response. Please use email or meetings for conversations like this.

1. Running times "for each of" of the tasks is highly dependent on the algorithm, implementation, benchmark, and machine. We used a range of compute resources to get the results for the paper, but most training and evaluation for the ML45/MT50 benchmarks used virtual compute instances with 8-16 CPUs, 40-150GB of memory, and 1 or 2 Tesla K80 or Titan Xp GPUs. PPO, TRPO, RL2, and MAML required instances with around 30 CPUs and 1500GB of memory. We recently fixed some bugs in the benchmark which should allow ML45/MT50 to be run with much less memory, probably around 150GB. The ML9/MT10 and smaller benchmarks require far fewer resources. Most training and evaluations of ML45/MT50 took 18-36 hours of wall-clock time.

2. As noted in the paper, all tasks are individually solved by SAC with standard hyperparameters in few million time steps or less.

3. There is no standard definition of structural similarities. This is a qualitative description, not a measurement. We intend it to mean "tasks which require similar motion (sub-)skills to solve," even if those (sub-)skills are reordered (e.g. reaching, then grasping), reversed in time (e.g. open door vs close door), etc. Devising a meaningful quantitative metric for this aspect of MTRL would be an interesting research question...

4. Again, this is a research question in MTRL. I think it is intuitive based on the success of multi-headed architectures for (for instance) image classification and language tasks. Multiple heads allows the Q-network to learn a shared representation for all tasks (the body) and a small specialization for each task (the heads).

5. The beauty of a good benchmark is that you don't need to explain the benchmark results for the results to be valid. Often, explaining counter-intuitive benchmark results is what creates new research. I would suggest that TRPO outperforming PPO does not contradict the results from model-free RL. TRPO imposes a direct constraint on the KL-divergence between policy updates whereas PPO uses an amortized version of that constraint (i.e. the constraint is probably-approximately satisfied in the limit). PPO is designed for massively-parallel sampling and is probably more popular recently because it is (1) easier TRPO to implement correctly than TRPO and (2) has faster wall-clock run-time on some architectures because it is easier to data-parallelize. I would challenge the assertion that there's strong evidence that PPO achieves higher asymptotic performance than TRPO for the same number of samples.

    If you want a plausible theory for why PPO and PPO-based algorithms (e.g. MAML-PPO and RL2-PPO) perform poorly on multi-task benchmarks compared to TRPO, I would suggest that amortized constraint approximations (such as those used by PPO) perform poorly on multi-modal problems like multi-task learning, in ways which TRPO does not suffer from, because TRPO constrains KL-divergence directly.

6. We'll have to take a look and get back to you on that one.

7. Memorization-not-generalization is a well-known problem in meta-RL (and, really, deep learning). I think a long discussion on this is well outside the scope of this GitHub issue, but you might consult [this recent paper](https://arxiv.org/abs/1912.03820) for some exposition on the problem in general. The particular sentence you quote refers to the difficulty of creating task distributions for image-based RL which force meta-RL algorithms to learn reusable shared representations which generalize to unseen tasks during pre-training, rather than just a simple inference model which identifies which member of the pre-training set the meta-RL algorithm is currently solving. It's very easy to accidentally reveal task identity using an image -- for instance, if the scene for door opening has a door but no cup, and the scene for cup-grasping has a cup but no door, then a meta-RL algorithm can easily overfit during pre-training to identifying whether they are in the cup world or the door world and use that to maximize returns, rather than learning features which are useful for both cup-grabbing and door-opening. This is a form of overfitting. For a very nice exploration on the relationship between gradient-based meta-RL and inference, I recommend you refer to [this paper](https://arxiv.org/abs/1909.09157). Essentially, if the inference a meta-RL algorithm learns during pre-training is only useful for inferring the identity of training tasks, then the meta-RL algorithm will not generalize well.

Hi Julian,

Thank you for your detailed response. That was very helpful.

Have you got the chance to look closer at question 6?

@quanvuong Sorry the long wait for a response. Figure 8 is learning curves for meta-train, not meta-test. AFAICT Figure 8 is consistent with the meta-train performance reported in Table 1.

I'm going to close this issue for now. Please feel free to reopen (or open another one) if you have more questions.