Repository: stepjam/RLBench
Issue: RLBench not working with stable-baseline3.
Root cause: Resource Mishandling

I give you a minimal example,

```
import gym
import rlbench.gym
import stable_baselines3.common.env_checker
from stable_baselines3 import PPO


env = gym.make('reach_target-state-v0')

print(stable_baselines3.common.env_checker.check_env(env))
model = PPO('MlpPolicy', env, verbose=1)
model.learn(total_timesteps=10000)

obs = env.reset()
for i in range(1000):
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, done, info = env.step(action)
    print(obs.shape)
    print(reward)
    if done:
      obs = env.reset()

env.close()
```

If I execute the code above, I get the following error

```
Traceback (most recent call last):
  File "/home/samuele/Projects/LAMPO/try.py", line 12, in <module>
    model.learn(total_timesteps=10000)
  File "/home/samuele/miniconda3/envs/mini-rl/lib/python3.7/site-packages/stable_baselines3/ppo/ppo.py", line 264, in learn
    reset_num_timesteps=reset_num_timesteps,
  File "/home/samuele/miniconda3/envs/mini-rl/lib/python3.7/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 240, in learn
    logger.dump(step=self.num_timesteps)
  File "/home/samuele/miniconda3/envs/mini-rl/lib/python3.7/site-packages/stable_baselines3/common/logger.py", line 378, in dump
    Logger.CURRENT.dump(step)
  File "/home/samuele/miniconda3/envs/mini-rl/lib/python3.7/site-packages/stable_baselines3/common/logger.py", line 543, in dump
    _format.write(self.name_to_value, self.name_to_excluded, step)
  File "/home/samuele/miniconda3/envs/mini-rl/lib/python3.7/site-packages/stable_baselines3/common/logger.py", line 142, in write
    self.file.write("\n".join(lines) + "\n")
ValueError: I/O operation on closed file.
QMutex: destroying locked mutex
```

Interestingly, if I debug the same code, I get a different error,

```
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/samuele/miniconda3/envs/mini-rl/lib/python3.7/site-packages/rlbench/gym/__init__.py", line 12, in <module>
    task_class = name_to_task_class(task_name)
  File "/home/samuele/miniconda3/envs/mini-rl/lib/python3.7/site-packages/rlbench/utils.py", line 35, in name_to_task_class
    % (class_name, name)) from e
rlbench.utils.InvalidTaskName: Cannot find the class name 'OpenDrawer' in the file 'open_drawer'.
```

Stable-baselines3, works well with many different environments that I've tested so far.



I get that same error, with stable-baselines3, and stable-baseline2 as well.

@Atlis I actually fixed the bad behavior with the debugger by reinstalling everything with miniconda and using PyQt5. 
But the problem mentioned of the  "I/O operation on closed file" persists (both in execution and debugging mode).

@Atlis Trivially, the problem disappears with 

```
model = PPO('MlpPolicy', env, verbose=0)
```

However, I do not know how to use the EvalCallback, as it needs one environment for evaluation and another for learning, and that is not possible with RLBench.

Did you solve that?


Hi,
I'm afraid I've not used stable-baselines, so not sure how much help I can be. 
However, RLBench can run multiple environments simultaneously (this is how I usually train), but they have to be in seperate processes (using the python multiprocessing module, see: https://github.com/stepjam/PyRep#running-multiple-pyrep-instances).

Thank you @stepjam! I really like RLBench, it has been really useful to me. It would be really nice if it'd work with stable-baselines (I think many people in our community are using it). I tried to debug myself, but it is really hard to understand what is happening. I am not sure that the multiprocess solution will work in our case (seems that the code must be synchronous), but I try it immediately and I'll let you know.


Excellent! In the meantime, I've managed to make it work with RLzoo. But, I may give a shot at stable-baselines again later considering the multiprocessing stuff. Thank you both!

Thanks, glad you find it useful.
To be honest, the reason why I don't use things like stable-baselines, rllib, etc, are because they make certain assumptions about observations and they are not very customisable once you want to start having multi sensor inputs which you want to treat in specific ways (e.g. wrist depth, wrist rgb, overhead rgb, proprio, etc). I've found that they lead to a lot of headaches when you want to start doing non-standard stuff.  This is why I essentially built my own robot-rl training framework, which I use internally, and makes customising these sorts of things much easier. Perhaps I should look at releasing this.

@Atlis @stepjam I momentarily solved it by modifying the stable baseline code at ```stable_baseline3/common/logger.py line 142```:

```
        if self.file.closed:
            self.file = open("/dev/stdout", "w")
            self.file.write("\n".join(lines) + "\n")
            self.file.flush()
        else:
            self.file.write("\n".join(lines) + "\n")
            # Flush the output to the file
            self.file.flush()
```

Of course, this is a very provisory solution and does not solve the real problem.

I just looked at the issue you opened on stabe-baselines.
Indeed, PyRep does do some stuff for redirecting std-in/out; the reason being that the verbosity level in the ompl pluging was not working, and so it would spit out a bunch of logs to the terminal.

But to see if this is actually the problem, you could change [this flag](https://github.com/stepjam/PyRep/blob/7057e19c6f2dfb72d2ab101705fb534bf98aa722/pyrep/__init__.py#L4) to False, and then reinstall PyRep.

@Atlis @SamuelePolimi Did my comment above fix the issue for you? Can we close this issue?

@stepjam @SamuelePolimi Thank you both! Setting `suppress_std_out_err=False` works for me.

Yes @stepjam, you can close the issue! Thanks a lot!

@stepjam, we are looking to run something across from rllib on RLBench but reading [https://github.com/stepjam/RLBench/issues/103#issuecomment-758026135)] doesn't bode well.
Can you pinpoint these issues, elaborate?