Repository: pytorch
Issue: Bugs: Score Function approach in REINFORCE for PONG
Labels: todo,module: crash,module: loss,module: cuda,module: memory usage,triaged

I adapted the reinforce and actor-critic code for Cartpole to PONG. The original Cartpole code is located in:

https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py

In the Cartpole example, the input to the policy NN is the state outputted by OpenAI, which is an array of 4 float. In PONG, the input is a downsampled image vector of 1600 (40x40) or 6400 (80x80). When using the reinforce.py as the guideline and using the score function approach described in Pytorch.org documentation (http://pytorch.org/docs/master/distributions.html), I encountered 2 bugs:

(1) GPU Memory Error
The exact error is:
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THC/generic/THCStorage.cu:58

The REINFORCE implementation requires the policy NN to run forward pass 1000-1500 times depending on how many game steps it takes for 21 points to be reached. Running "nvidia-smi l 1", I discover that GPU memory is not released on the Nvidia card even after the program exit. 

I have to do 2 things to get the code to run:
- Add torch.cuda.empty_cache() at the end of each episode
- Limit the forward passes to a fixed batch size (e.g. 800-1200) instead of letting the code run until a flag (done) is True

The typical workflow for policy-based RL is 1000-10000 forward pass, and then 1 single backward pass utilizing stacked intermediary data such as hidden units, states, rewards, log_prob and so on. Either Torch or the underlying CUDA has memory management problem with this kind of workflow. 

(2) Underperformance of the trained RL agent

I have run the REINFORCE agent to 30K episodes and it is not learning as well as the original REINFORCE implemented by Andrej Karpathy using numpy. 

![image](https://user-images.githubusercontent.com/10457709/36557214-83f34f6c-17bc-11e8-9e8c-754da4573f90.png)

Details of my experimentation is detailed in:
https://github.com/LUKELIEM/deep_rl/blob/master/gym/reinforce.ipynb

There is 1 major difference between my code and Karpathy's py-pong.py:

- Karpathy implements the backprop directly 
   - His code accumulates the intermediary states (epx), hidden states (eph), rewards (epr) and log_prob (eplogd) during the forward pass
   - At the single backpass after 1000-1500 forward passes, the backprop is implemented by matrix multiplication of these stacked intermediary values to produce a gradient 
   - The code then uses RMSprop to update the weights directly.
  
- Pytorch's code implements backprop using policy_loss.backward() through this code fragment:

    probs = policy(Variable(state))
    m = Categorical(probs)
    action = m.sample()
    policy.saved_log_probs.append(m.log_prob(action))

I have tried various update rules and they all seem incapable to learning beyond running reward of 0.

![image](https://user-images.githubusercontent.com/10457709/36679195-7a01bada-1ac7-11e8-9122-5c330d3fe9c3.png)


- OS: Ubuntu 16.04
- PyTorch version:  0.3.0
- How you installed PyTorch (conda, pip, source): conda
- Python version: 3.6.4
- CUDA/cuDNN version: 
- GPU models and configuration: ASUS ROG STRIX GeForce GTX 1080 TI 11GB VR Ready 5K HD Gaming Graphics Card 
- GCC version (if compiling from source):