Repository: pytorch
Issue: Memory not being deallocated in backward()
Labels: module: autograd,module: memory usage,triaged,quansight-nack

## 🐛 Bug

I've recently discovered an issue with memory not being freed after the first iteration of training. It's not a leak, as memory usage stays consistent after the second pass through the loop. It appears on both CPU and GPU, however it is much more significant when running on CPU.

The issue seems to come from the either backward or optimizer.step(), as removing their calls provides stable memory usage.

I ran into this while attempting to train a rather large model that uses pretty much all of my available GPU memory. It will complete the first iteration successfully, then OOM during the second.

## To Reproduce

Steps to reproduce the behavior:

I have compiled a minimal CPU and GPU gist that should reproduce this issue:

[CPU](https://gist.github.com/mdlockyer/3ff43f00ad7b7e2c2a3a7f33469658da)
[GPU](https://gist.github.com/mdlockyer/1b728751113067c47ef104a5ecf1691d)

The CPU gist uses the memory-profile package, so that will need to be installed with pip

## Expected behavior

The memory usage should be relatively the same in the first pass through the training loop, and all following loops.

## Environment

PyTorch version: 1.0.1.post2
Is debug build: No
CUDA used to build PyTorch: None

OS: Mac OSX 10.13.6
GCC version: Could not collect
CMake version: version 3.9.4

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.16.2
[pip3] torch==1.0.1.post2
[pip3] torchvision==0.2.2.post3
[conda] torch                     1.0.1.post2               <pip>
[conda] torchsummary              1.5.1                     <pip>
[conda] torchvision               0.2.1                     <pip>

## Additional context

I ran some profiles on the CPU memory usage that highlight the issue:

### With backward pass and update:

![with_backward_4](https://user-images.githubusercontent.com/30844227/55268463-cc9d3d00-5257-11e9-9ff0-cc1e82c08081.png)

#### Iteration 1
    Line #    Mem usage    Increment   Line Contents
    ================================================
        23    360.0 MiB    360.0 MiB   @profile
        24                             def train(model, criterion, optim):
        25    360.1 MiB      0.0 MiB       x = torch.rand(1, 3, 8, 8)
        26    360.1 MiB      0.0 MiB       y = torch.ones(1, 1, 8, 8)
        27                             
        28    402.7 MiB     42.6 MiB       out = model(x)
        29    402.7 MiB      0.1 MiB       loss = criterion(out, y)
        30                             
        31    402.7 MiB      0.0 MiB       optim.zero_grad()
        32    663.8 MiB    261.1 MiB       loss.backward()
        33    664.0 MiB      0.1 MiB       optim.step()
        34    664.0 MiB      0.0 MiB       optim.zero_grad()
        35    664.0 MiB      0.0 MiB       del x, y, out, loss
        36    664.0 MiB      0.0 MiB       gc.collect()

#### Iteration 2
    Line #    Mem usage    Increment   Line Contents
    ================================================
        23    664.0 MiB    664.0 MiB   @profile
        24                             def train(model, criterion, optim):
        25    664.0 MiB      0.0 MiB       x = torch.rand(1, 3, 8, 8)
        26    664.0 MiB      0.0 MiB       y = torch.ones(1, 1, 8, 8)
        27                             
        28    701.7 MiB     37.7 MiB       out = model(x)
        29    701.7 MiB      0.0 MiB       loss = criterion(out, y)
        30                             
        31    701.7 MiB      0.0 MiB       optim.zero_grad()
        32    671.7 MiB      0.0 MiB       loss.backward()
        33    671.7 MiB      0.0 MiB       optim.step()
        34    671.7 MiB      0.0 MiB       optim.zero_grad()
        35    671.7 MiB      0.0 MiB       del x, y, out, loss
        36    671.7 MiB      0.0 MiB       gc.collect()

### Without backward pass and update:

![without_backward_2](https://user-images.githubusercontent.com/30844227/55268515-42090d80-5258-11e9-8a54-81e2afeadf85.png)

#### Iteration 1
    Line #    Mem usage    Increment   Line Contents
    ================================================
        23    351.2 MiB    351.2 MiB   @profile
        24                             def train(model, criterion, optim):
        25    351.2 MiB      0.0 MiB       x = torch.rand(1, 3, 8, 8)
        26    351.3 MiB      0.0 MiB       y = torch.ones(1, 1, 8, 8)
        27                             
        28    392.4 MiB     41.1 MiB       out = model(x)
        29    392.5 MiB      0.1 MiB       loss = criterion(out, y)
        30                             
        31    392.5 MiB      0.0 MiB       optim.zero_grad()
        32                                 #loss.backward()
        33                                 #optim.step()
        34    392.5 MiB      0.0 MiB       optim.zero_grad()
        35    361.7 MiB      0.0 MiB       del x, y, out, loss
        36    361.7 MiB      0.0 MiB       gc.collect()

#### Iteration 2
    Line #    Mem usage    Increment   Line Contents
    ================================================
        23    361.7 MiB    361.7 MiB   @profile
        24                             def train(model, criterion, optim):
        25    361.7 MiB      0.0 MiB       x = torch.rand(1, 3, 8, 8)
        26    361.7 MiB      0.0 MiB       y = torch.ones(1, 1, 8, 8)
        27                             
        28    392.0 MiB     30.3 MiB       out = model(x)
        29    392.0 MiB      0.0 MiB       loss = criterion(out, y)
        30                             
        31    392.0 MiB      0.0 MiB       optim.zero_grad()
        32                                 #loss.backward()
        33                                 #optim.step()
        34    392.0 MiB      0.0 MiB       optim.zero_grad()
        35    361.7 MiB      0.0 MiB       del x, y, out, loss
        36    361.7 MiB      0.0 MiB       gc.collect()

cc @ezyang @gchanan @zou3519 @SsnL @albanD @gqchen