Repository: pytorch
Issue: Traced resnet101 leaks memory during `forward`
Labels: module: memory usage,triaged

## 🐛 Bug
Memory consumption increases when running inference with torch script module.
## To Reproduce

code samples:
```
import os

import psutil
import torch
import torchvision

traced_script_module = torch.jit.trace(torchvision.models.resnet101(pretrained=True), torch.rand(1, 3, 224, 224))

process = psutil.Process(os.getpid())
print(process.memory_info().rss / 1e9)

for i in range(10):
    samples = torch.rand(32, 3, 224, 224)
    predictions = traced_script_module(samples)
    print(predictions)
    process = psutil.Process(os.getpid())
    print(process.memory_info().rss / 1e9)
```

output 
```
0.595202048
tensor([[-1.5747,  0.6994, -0.7375,  ...,  0.8081,  1.2429, -0.5155],
        [ 1.2828, -1.1425, -0.2093,  ..., -1.0598,  1.4675,  0.2916],
        [-2.8575,  0.1437, -0.9690,  ..., -1.2187,  2.2080,  1.3188],
        ...,
        [ 0.2986,  0.6763,  1.1985,  ...,  1.7018,  0.4733,  0.8496],
        [-0.8345, -1.2992, -0.7214,  ..., -0.6559,  1.0582, -0.1334],
        [-0.7516, -0.8837,  0.1562,  ..., -0.5259, -0.7300,  0.5229]],
       grad_fn=<DifferentiableGraphBackward>)
5.99044096
tensor([[-1.5746, -1.8324, -1.2751,  ..., -1.7721,  0.2030,  1.1744],
        [ 1.7051, -0.3812, -2.5524,  ..., -1.3434,  1.6557,  0.2199],
        [ 0.0429,  0.2007,  1.2892,  ..., -1.7945, -0.8777,  0.0374],
        ...,
        [-0.6182, -0.0681,  0.4620,  ...,  0.0140,  0.1492,  0.3290],
        [-0.0735,  0.3230, -0.7785,  ..., -0.8295,  0.1701,  0.3623],
        [-1.1048, -0.9883,  0.0532,  ..., -1.6516, -0.7215,  0.1046]],
       grad_fn=<DifferentiableGraphBackward>)
8.926531584
tensor([[-3.3263, -0.3973, -2.1202,  ..., -0.9237,  1.4103,  1.6226],
        [-1.8091, -1.1055, -2.0259,  ..., -0.3219,  0.2350,  0.8425],
        [-1.7041, -0.3857, -1.3425,  ...,  2.0590,  1.9119,  0.3337],
        ...,
        [-0.2145, -0.0698, -1.4803,  ...,  2.4058,  1.6146,  1.4400],
        [ 0.7031,  0.0917,  1.5251,  ...,  1.4080,  1.0682,  0.9095],
        [-1.8758,  0.2686, -0.0282,  ...,  0.2069,  0.5055,  0.7498]],
       grad_fn=<DifferentiableGraphBackward>)
9.022865408
tensor([[-1.5912, -1.1433, -2.2109,  ..., -1.6980,  3.1191, -0.4465],
        [-1.2363, -0.8868, -0.2385,  ..., -1.9240,  0.8099,  0.6127],
        [-2.8276, -0.5101,  0.5070,  ..., -2.0766, -1.3493,  1.4947],
        ...,
        [-1.4246, -1.2391, -1.4561,  ..., -1.8967,  1.7254,  0.9091],
        [-0.5911, -0.2010, -1.3293,  ...,  1.2857,  0.5298, -0.3506],
        [-2.2977, -0.2686, -1.9806,  ..., -0.6035,  1.0328,  1.3565]],
       grad_fn=<DifferentiableGraphBackward>)
10.608922624
tensor([[ 0.9014,  1.0406, -0.0942,  ...,  0.7062, -0.9783, -1.1471],
        [-1.0258, -0.0566, -1.1202,  ..., -0.5074, -0.1088, -0.1099],
        [-0.2558,  1.1689,  1.3797,  ..., -2.4711,  0.0562,  1.8224],
        ...,
        [ 0.2017, -0.3057, -0.5456,  ...,  0.9477,  1.2708,  0.4418],
        [-1.5674,  0.1624, -1.1115,  ...,  1.0431,  1.0180, -0.0383],
        [-2.1499, -0.7444, -1.4897,  ...,  0.6625,  0.8116,  2.2398]],
       grad_fn=<DifferentiableGraphBackward>)
10.608922624
tensor([[-0.7774,  0.4201, -3.0641,  ..., -2.9160,  2.4548, -0.6954],
        [-1.5430, -0.3068, -2.7503,  ..., -2.2520,  0.5887,  0.9933],
        [-0.6780,  0.1065,  1.2061,  ..., -2.1829,  0.6028, -0.0989],
        ...,
        [-1.1435, -0.7770, -1.7809,  ..., -1.6382, -0.0364,  0.1657],
        [ 0.0038, -0.7613, -1.7814,  ..., -0.9964,  1.4763,  1.1532],
        [-0.1706, -0.0197,  0.8526,  ..., -1.4628,  2.2953,  1.3528]],
       grad_fn=<DifferentiableGraphBackward>)
10.608922624

```

## Expected behavior

memory consumption stays constant as the first iteration.
## Environment

(or fill out the checklist below manually).
Collecting environment information...
PyTorch version: 1.3.0
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: Ubuntu 16.04.2 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.13.3
[pip3] numpydoc==0.6.0
[pip3] torch==0.1.12.post1
[pip3] torchvision==0.1.8
[conda] torch                     1.3.0                     <pip>
[conda] torchvision               0.4.1                     <pip>


cc @ezyang @gchanan @zou3519 @jerryzh168 @suo