Repository: stepjam/RLBench
Issue: Problems with parallellization.
Root cause: Architectural Constraint

Hi StepJam, it's interesting that you are discussing headless. I am facing similar issues. 
Many things actually work: single threaded with CoppeliaSim, headless.
However, our program is too slow (that may be owing to our algorithm, but it seems that it is in fact the sampler), so we are trying to get a parallel version going, using RLPyt, but it hangs. I don't know if this has to do with the environment or the runner. I have looked at YARR, which is a lot simpler than other runners, but I hope RLPyt is familiar to you, is it?

Running just starts CoppeliaSim (I can do it with headless=False over vnc).
in rlpyt/rlpy/samplers/parallel/base.py:
```
for w in self.workers:
    w.start()
```
this sets off ```sampling_process``` in rlpyt/rlpyt/samplers/parallel/worker.py
and it hangs on 
```
envs = [c.EnvCls(**c.env_kwargs) for _ in range(w.n_envs)]
```
I have actually changed
```
if subprocess:
```
in rlpyt/rlpyt/samplers/buffer.py to
```
if False
```
because it hung on ```mgr = mp.Manager()```

In sum, I have cobbled together a main.py script, hoping to run it with RLBench.
It is here on the parallel branch:
[https://github.com/deepdad/SharedWorldModels/blob/parallel/dreamer-pytorch/main_parallel.py](https://github.com/deepdad/SharedWorldModels/blob/parallel/dreamer-pytorch/main_parallel.py) 

Most of it agent isn't yours obviously, just the environment. It would be nice if you could log in with me over VNC to the GCloud VM where PyCharm is running.
The problem seems to be that the parallel (RLPyt) sampler can't get any samples.

```
import numpy as np
from rlpyt.envs.base import Env, EnvStep
from rlpyt.spaces.int_box import IntBox
from rlpyt.spaces.float_box import FloatBox
from rlbench.environment import Environment
from rlbench.action_modes import ArmActionMode, ActionMode
from rlbench.observation_config import ObservationConfig, CameraConfig
from rlbench.tasks import ReachTarget

from dreamer.envs.env import EnvInfo


class RLBench(Env):

    def __init__(self, config):
        self.config = config
        self._env, self._task = self._initialize()

    def _initialize(self):
        # we just choose one camera
        obs = self.config.get("obs_config", ObservationConfig(CameraConfig(image_size=(2, 2)),
                                                              CameraConfig(image_size=(2, 2)),
                                                              CameraConfig(image_size=(2, 2)),
                                                              CameraConfig(image_size=(2, 2)),
                                                              CameraConfig(image_size=(64, 64))))
        print("OBS0", vars(obs)
        obs.front_camera.set_all(True)
        action_mode = self.config.get("action_mode", ActionMode(ArmActionMode.ABS_JOINT_VELOCITY))
        headless = self.config.get("headless", True)
        env = Environment(action_mode, obs_config=obs, headless=headless,
                          robot_configuration="panda" #panda', 'jaco', 'mico', 'sawyer', 'ur5'
                         )
        env.launch()
        task = env.get_task(self.config.get("task", ReachTarget))
        return env, task

    @property
    def observation_space(self):
        return IntBox(low=0, high=255, shape=(3,) + self.config.get("size", (64, 64)),
                      dtype="uint8")

    @property
    def action_space(self):
        return FloatBox(low=-1.0, high=1.0, shape=(self._env.action_size,))

    def step(self, action):
        obs, reward, done = self._task.step(action)
        obs = np.transpose(obs.front_rgb, (2, 0, 1))
        info = EnvInfo(None, None, done)
        return EnvStep(obs, reward, done, info)

    def reset(self):
        descriptions, obs = self._task.reset()
        obs = np.transpose(obs.front_rgb, (2, 0, 1))
        del descriptions  # Not used.
        return obs

    def render(self, *args, **kwargs):
        pass

    @property
    def horizon(self):
        raise NotImplementedError
```


I assume it hangs because CoppeliaSim (and by extension, RLBench) must be launched in separate processes if you want to run multiple environments. See here: https://github.com/stepjam/PyRep#running-multiple-pyrep-instances

All the best,
Stephen