Repository: pytorch
Issue: Iterative Global Pruning Cause GPU Memory Leak
Labels: high priority,module: nn,module: memory usage,triaged

### 🐛 Describe the bug

# Bug Description

According to the official [tutorial](https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#iterative-pruning), pruning should work if applied for multiple times. 

This works just fine for per-module pruning, as demonstrated in the tutorial. 
**However, it doesn't work for global pruning, in that it will cause GPU memory leak.** 

Specifically, the CUDA memory footprint of the process (as observed through `torch.cuda.memory_allocated()`) will steadily increase until a `RuntimeError` of `Cuda out of memory` is thrown.

The increase in size is about the same as the total parameter size of the model. For example, an FP32 model with 37.5M parameters will leak memory in increments of `37.5M * 4B = 150MB`.

I tried `gc.collect` and `torch.cuda.empty_cache()`, neither helped.

# Reproducing Example
Here's a minimal example that reproduces this problem:

```python
import torch
import torch.nn as nn
import torch.nn.utils.prune as prune
import gc


def prune_once(model, prune_amount):
    prune_params = []
    for module in model.modules():
        if isinstance(module, nn.Conv2d):
            prune_params.append((module, "weight"))
            prune_params.append((module, "bias"))
    
    prune.global_unstructured(
        prune_params,
        pruning_method=prune.L1Unstructured,
        amount=prune_amount,
    )

model=nn.Sequential(nn.Conv2d(1024, 2048, 3), nn.Conv2d(2048, 1024, 3))
model.to(torch.device('cuda:0'))

for i in range(300):
    prune_once(model, 0.07)
    print("GPU Memory Used:", torch.cuda.memory_allocated() / 1e9)

    # None of these help
    gc.collect()
    torch.cuda.empty_cache()
```

## Expected output
```
GPU Memory Used: 0.604028928
GPU Memory Used: 0.604028928
...(Repeat 300 times)
```

## Actual output
```
GPU Memory Used: 0.604028928
GPU Memory Used: 0.75503616
GPU Memory Used: 0.906043392
GPU Memory Used: 1.057050624
GPU Memory Used: 1.208057856
GPU Memory Used: 1.359065088
...(Stedily increase by about 150MB per iteration)
GPU Memory Used: 32.919576576
GPU Memory Used: 33.070583808
GPU Memory Used: 33.22159104
GPU Memory Used: 33.372598272

# Then throws an exception:
# RuntimeError: CUDA out of memory. Tried to allocate 72.00 MiB (GPU 0; 39.59 GiB total capacity; 31.50 GiB already allocated; 5.85 GiB free; 31.93 GiB reserved in total by PyTorch)
```


### Versions

Collecting environment information...
PyTorch version: 1.9.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] torch==1.9.1+cu111
[pip3] torchaudio==0.9.1
[pip3] torchvision==0.10.1+cu111
[conda] _tflow_select             2.3.0                       mkl  
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py38h7f8727e_0  
[conda] mkl_fft                   1.3.1            py38hd3c417c_0  
[conda] mkl_random                1.2.2            py38h51133e4_0  
[conda] numpy                     1.23.1           py38h6c91a56_0  
[conda] numpy-base                1.23.1           py38ha15fc14_0  
[conda] tensorflow                2.8.2           mkl_py38hb41d75a_0  
[conda] tensorflow-base           2.8.2           mkl_py38hf890080_0  
[conda] torch                     1.9.1+cu111              pypi_0    pypi
[conda] torchaudio                0.9.1                    pypi_0    pypi
[conda] torchvision               0.10.1+cu111             pypi_0    pypi

cc @ezyang @gchanan @zou3519 @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are