Repository: pytorch
Issue: Autograd profiler memory leak when use_cuda=True
Labels: needs reproduction,module: autograd,triaged

## 🐛 Bug

When using the autograd profiler with `use_cuda=True`, memory usage continuously increases. This behaviour does not occur when `use_cuda=False`.

## To Reproduce

```python
import torch
import torchvision
import psutil

model = torchvision.models.alexnet(pretrained=False).cuda()
x = torch.rand([1, 3, 224, 224]).cuda()

for i in range(10000):
    with torch.autograd.profiler.profile(use_cuda=True):
        model(x)
    if i % 1000 == 0:
        print("Iteration: {}, memory: {}".format(i, psutil.virtual_memory()))
```
```text
Iteration: 0, memory: svmem(total=135112507392, available=130796793856, percent=3.2, used=3056013312, free=122434646016, active=7324913664, inactive=3956400128, buffers=608055296, cached=9013792768, shared=9961472, slab=975155200)
Iteration: 1000, memory: svmem(total=135112507392, available=130689802240, percent=3.3, used=3162726400, free=122327621632, active=7371665408, inactive=3956678656, buffers=608055296, cached=9014104064, shared=10240000, slab=978624512)
Iteration: 2000, memory: svmem(total=135112507392, available=130581217280, percent=3.4, used=3271311360, free=122219036672, active=7422038016, inactive=3956678656, buffers=608055296, cached=9014104064, shared=10240000, slab=981524480)
Iteration: 3000, memory: svmem(total=135112507392, available=130470469632, percent=3.4, used=3382059008, free=122108289024, active=7472951296, inactive=3956678656, buffers=608055296, cached=9014104064, shared=10240000, slab=984805376)
Iteration: 4000, memory: svmem(total=135112507392, available=130361843712, percent=3.5, used=3490684928, free=121999663104, active=7523426304, inactive=3956678656, buffers=608055296, cached=9014104064, shared=10240000, slab=988176384)
Iteration: 5000, memory: svmem(total=135112507392, available=130251255808, percent=3.6, used=3601272832, free=121889075200, active=7574122496, inactive=3956678656, buffers=608055296, cached=9014104064, shared=10240000, slab=991485952)
Iteration: 6000, memory: svmem(total=135112507392, available=130140499968, percent=3.7, used=3712028672, free=121778319360, active=7625084928, inactive=3956678656, buffers=608055296, cached=9014104064, shared=10240000, slab=994746368)
Iteration: 7000, memory: svmem(total=135112507392, available=130031747072, percent=3.8, used=3820781568, free=121669558272, active=7675588608, inactive=3956678656, buffers=608055296, cached=9014112256, shared=10240000, slab=998010880)
Iteration: 8000, memory: svmem(total=135112507392, available=129920745472, percent=3.8, used=3931783168, free=121558556672, active=7726501888, inactive=3956678656, buffers=608055296, cached=9014112256, shared=10240000, slab=1001361408)
Iteration: 9000, memory: svmem(total=135112507392, available=129810280448, percent=3.9, used=4042248192, free=121448091648, active=7776731136, inactive=3956678656, buffers=608055296, cached=9014112256, shared=10240000, slab=1004154880)
```

## Expected behavior
System memory usage should stay roughly constant. See output of the code, only no `cuda` tensors and `use_cuda` kwarg is not set.

```python
import torch
import torchvision
import psutil

model = torchvision.models.alexnet(pretrained=False)
x = torch.rand([1, 3, 224, 224])

for i in range(10000):
    with torch.autograd.profiler.profile():
        model(x)
    if i % 1000 == 0:
        print("Iteration: {}, memory: {}".format(i, psutil.virtual_memory()))
```

```text
Iteration: 0, memory: svmem(total=135112507392, available=132605038592, percent=1.9, used=1255895040, free=124242784256, active=5647179776, inactive=3943378944, buffers=608104448, cached=9005723648, shared=1847296, slab=968876032)
Iteration: 1000, memory: svmem(total=135112507392, available=132597575680, percent=1.9, used=1263345664, free=124235264000, active=5654331392, inactive=3943436288, buffers=608116736, cached=9005780992, shared=1847296, slab=968986624)
Iteration: 2000, memory: svmem(total=135112507392, available=132597575680, percent=1.9, used=1263353856, free=124235132928, active=5654401024, inactive=3943436288, buffers=608116736, cached=9005903872, shared=1847296, slab=969125888)
Iteration: 3000, memory: svmem(total=135112507392, available=132595933184, percent=1.9, used=1264959488, free=124233482240, active=5655736320, inactive=3943436288, buffers=608120832, cached=9005944832, shared=1847296, slab=969125888)
Iteration: 4000, memory: svmem(total=135112507392, available=132596064256, percent=1.9, used=1264828416, free=124233613312, active=5655764992, inactive=3943436288, buffers=608120832, cached=9005944832, shared=1847296, slab=968994816)
Iteration: 5000, memory: svmem(total=135112507392, available=132591153152, percent=1.9, used=1269780480, free=124228661248, active=5660770304, inactive=3943477248, buffers=608120832, cached=9005944832, shared=1847296, slab=968994816)
Iteration: 6000, memory: svmem(total=135112507392, available=132590772224, percent=1.9, used=1270120448, free=124228272128, active=5660790784, inactive=3943477248, buffers=608120832, cached=9005993984, shared=1847296, slab=969003008)
Iteration: 7000, memory: svmem(total=135112507392, available=132590772224, percent=1.9, used=1270120448, free=124228272128, active=5660852224, inactive=3943477248, buffers=608120832, cached=9005993984, shared=1847296, slab=969003008)
Iteration: 8000, memory: svmem(total=135112507392, available=132589510656, percent=1.9, used=1271422976, free=124226961408, active=5660696576, inactive=3943526400, buffers=608120832, cached=9006002176, shared=1847296, slab=969048064)
Iteration: 9000, memory: svmem(total=135112507392, available=132587188224, percent=1.9, used=1273704448, free=124224638976, active=5664169984, inactive=3943526400, buffers=608124928, cached=9006039040, shared=1847296, slab=969211904)
```

## Environment

PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: TITAN Xp
Nvidia driver version: 410.48
cuDNN version: /usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudnn.so.7

Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==1.1.0
[pip3] torchprof==0.1.0
[pip3] torchvision==0.3.0
[conda] Could not collect

## Additional context

<!-- Add any other context about the problem here. -->


cc @ezyang @SsnL @albanD @zou3519 @gqchen