Repository: Farama-Foundation/Metaworld
Issue: Unnecessarily complicated rewards
Root cause: Unimplemented Feature Gap

The original MetaWorld paper discusses the reward decisions as follows:
> Designing reward functions for Meta-World requires two major considerations. First, to guarantee
that our tasks are within the reach of current single-task reinforcement learning algorithms, which
is a prerequisite for evaluating multi-task and meta-RL algorithms, we design well-shaped reward
functions for each task that make each of the tasks at least individually solvable. More importantly,
the reward functions must exhibit shared structure across tasks. Critically, even if the reward function admits the same optimal policy for multiple tasks, varying reward scales or structures can make
the tasks appear completely distinct for the learning algorithm, masking their shared structure and
leading to preferences for tasks with high-magnitude rewards [8].

Looking at the MT10/ML10 benchmarks, it seems to me that the current reward structure is unnecessarily complicated to achieve the above-mentioned properties. First, the rewards/returns for nearly all these tasks are extremely large, making it difficult for the standard RL algorithms (e.g. SAC) to solve the tasks in single-task setting. For example, in `reach-v1`, the maximum reward is in the order of thousands and the maximum return in the order of hundreds of thousands. Obviously, this can be overcome by tweaking the learning rates and other algorithm hyperparameters, but that seems overly complicated given that the goal of these tasks is to provide a unified benchmark for meta/multi-task learning. Secondly, the rewards/returns across tasks are wildly different. With the scripted policies, for example, the returns for `window-{open,close}-v1` environments are less than 100, compared to the hundreds of thousands for `reach-v1`.

I ran a subset of MT10/ML10 tasks with a simple negative-distance rewards that are scaled between [0, 1] and SAC solves 8/10 tasks with 100% success rate within ~100k steps, and for example `reach-v1` only takes 15k steps to solve. Even the two harder tasks I ran, `pick-place-v1` and `peg-insert-side-v1`, could be solved with 100% success rate in just a couple of hundred steps with SAC with default hyperparameters. These improvements are not fully due to changes in the reward structure though, since I also had some additional information in the observation to make the environment Markovian (see #224). However, even without the augmented observations, for example, `reach-v1` can be solved an order of magnitude faster with the linear reward. I can report more detailed results here once I've managed to decouple the effect of the full observability and reward structure. In the meanwhile, I'd be curious to hear if you all observed some different behavior that led to such complicated rewards? Or maybe there's something I'm completely missing here? If so, apologies for the noise.

Hi @hartikainen thanks for checking out Metaworld and providing us with such detailed feedback!

I agree that the reward functions contain the bulk of the issue that you have described. In the MTSAC implementation that I wrote in our sister repo, Garage, we found that just by normalizing rewards to fall between 0 and 1 (not using negative distance) vastly improved our agents performance. 

I am actually in the process of rewriting many of our reward functions to address this and #224 to apply them to our v2 environments, which are apart of the public codebase, but are not included as a part of any of the benchmarks yet. We plan to release them soon, sometime later this year. 

Perhaps @ryanjulian can comment further. 

Thanks!
Avnish



@hartikainen via #312 We are going to close this issue.