Repository: stepjam/RLBench
Issue: Questions about the roadmap for RLBench

Hello, 

I find the RLBench framework very interesting and thanks for open-sourcing it. Since RLBench is still evolving, I had some questions regarding the roadmap for RLBench. This would help me make an informed choice about using it.

- I could not find any benchmark numbers on RLBench. Can you please redirect me there exist some? If not, are there any plans to release any baselines methods for RLBench?

- This is somewhat related to my last question. In your opinion, how "solvable" is RLBench at this point in time. I am asking this as no one has reported any success in using an off-the-shelf method for even a single task. Does it have something to do with the environment itself? A similar environment for manipulation, particularly surreal (https://surreal.stanford.edu/img/surreal-corl2018.pdf) have had some success using off-the-shelf algorithms. Have you tried training an off-the-shelf algorithm long enough on just a single task? If so, how successful was it? 

- Since RLBench has had relatively quick releases, are there any features that are in pipeline or any major refactoring planned? This would help me know if RLBench is stable enough.

Looking forward to your response. Thanks!


Hi @imankgoyal 

Thanks fo the questions.
There are no baseline methods for RLBench as it can be used for many different applications (RL, imitation learning, few-shot learning, planning, etc), and so having official benchmark numbers don't really make sense. Having said that, I imagine most people will use this for RL. The reason why there are no 'official' baseline RL numbers is because 'vanilla'  RL methods cannot solve even the easiest of tasks, so the results would just be a bunch of flat curves. This is because the sparseness and lengths of the tasks. This is the whole point of RLBench after all, to really challenge the RL community and ask the question: are RL methods capable of solving such a challenging benchmarks, or do we have to reply on more explicit model-based methods.

Regarding how **solvable** RLBench is: each task is **guaranteed to be solvable**. This is because every task comes with demonstrations that have been generated from motion planners. When a new configuration is spawned, that configuration is first 'checked' behind the scenes to make sure its solvable before allowing the user to interact. Whether an RL algorithm can then _solve it_, is up to the method 😃 . As I already mentioned, I'm unaware of any existing method that would be able to solve many of these tasks 😄 

No major refactoring planned. Only bug fixes and new tasks.


Hello,

I've been trying to get something out of RLBench and RLzoo, without any success. I'm feeling somewhat lost here.
You state that RLBench is a learning environment, but we have no clue on how to effectively use it and if someone has had any success in training an RL algorithm.
- Do you have any suggestion on which algorithm to use?

Right now I'm trying on the most (I think) basic task: Reach Target. The reward is solely based on the gripper reaching the target, so the total reward of an episode is 1 if the gripper has reached the target, 0 otherwise.
- Is this kind of reward sufficient for an algorithm to learn?
- Are we supposed to modify your code to implement a reward shaping method? (Like the one proposed in [this paper](https://arxiv.org/pdf/1909.11730.pdf))

Thanks for your work!
Mirko

Hi,
RLBench is challenging by design; emulating the challenges that we face in robotics (dealing with demonstrations, sparse rewards, long horizon planning, etc). Because of that, naïve methods/algorithms won't work, i.e. you can't expect to just run vanilla PPO/TD3/SAC etc on these tasks and expect them to work. To solve these tasks, new methods will need to be developed that use demonstrations, use some built in prior knowledge along with model building, etc.

Having said that, you are free to alter RLBench however you like to fit your research needs. I.e. if you are doing an undergraduate project and have an assignment to run PPO/TD3 on one of the tasks, then yes, you will need to override the task class and add a shaping reward.