Repository: pytorch
Issue: Possible issue with memory allocation.
Labels: module: memory usage,triaged,module: numpy

### 🐛 Describe the bug

Allocating small tensors and keeping a reference to them can severely degrade performance. This seems to be a less of an issue for larger tensors, suggesting that it may have something to do with memory fragmentation. Is this something I should expect or is this a bug?

The script below illustrates the issue. It allocates memory for tensors of different sizes and, depending on a flag, keeps a reference to the tensor or not. For comparison, memory can also be allocated using numpy. The results can vary significantly across repeated runs.

```bash
# Allocate 1e6 tensors of 10 elements and repeat five times without keeping a reference.
$ python alloc.py 5 1000000 --array_size=10
#1: 0.636 seconds
#2: 0.634 seconds
#3: 0.646 seconds
#4: 0.642 seconds
#5: 0.648 seconds
```

```bash
# Run the same script, except let's keep a reference to the small tensors. This takes more 
# than order of magnitude longer.
python alloc.py 5 1000000 --array_size=10 --save
#1: 14.480 seconds
#2: 27.128 seconds
#3: 38.828 seconds
#4: 18.525 seconds
#5: 29.354 seconds
```

```bash
# Using a larger array of 100 elements is *much* faster even though we're allocating more 
# memory.
python alloc.py 5 1000000 --array_size=100 --save
#1: 1.001 seconds
#2: 0.970 seconds
#3: 1.018 seconds
#4: 0.914 seconds
#5: 0.999 seconds
```

```bash
# We only see a marginal slowdown compared with discarding the arrays immediately.
python alloc.py 5 1000000 --array_size=100       
#1: 0.700 seconds
#2: 0.712 seconds
#3: 0.717 seconds
#4: 0.711 seconds
#5: 0.701 seconds
```

```bash
# Rerunning the small-tensor experiments using numpy shows a negligible difference.
python alloc.py 5 1000000 --array_size=10 --numpy
#1: 0.400 seconds
#2: 0.406 seconds
#3: 0.401 seconds
#4: 0.401 seconds
#5: 0.403 seconds

python alloc.py 5 1000000 --array_size=10 --numpy --save
#1: 0.447 seconds
#2: 0.445 seconds
#3: 0.440 seconds
#4: 0.444 seconds
#5: 0.457 seconds
```

```bash
# Rerunning the small-tensor results with the --tqdm flag for a progress bar shows that
# the script generally makes good progress but occasionally gets "stuck" (is memory being 
# reallocated?).
python alloc.py 5 1000000 --array_size=10 --save --tqdm
...
```

# Script for reproducing results.

```python
import argparse
import numpy as np
import time
import torch as th
from tqdm import tqdm


def __main__():
    parser = argparse.ArgumentParser()
    parser.add_argument('num_batches', type=int, help='number of batches to run')
    parser.add_argument('batch_size', type=int, help='number of arrays per batch')
    parser.add_argument('--array_size', type=int, default=10, help='size of each array')
    parser.add_argument('--save', action='store_true', help='store the array in a list')
    parser.add_argument('--numpy', action='store_true', help='use numpy for allocation')
    parser.add_argument('--tqdm', action='store_true', help='show progress bar for each batch')
    args = parser.parse_args()

    if args.numpy:
        alloc = lambda: np.empty(args.array_size)  # noqa: E731
    else:
        alloc = lambda: th.empty(args.array_size)  # noqa: E731

    durations = []
    for i in range(args.num_batches):
        batch = []
        start = time.time()
        steps = range(args.batch_size)
        if args.tqdm:
            steps = tqdm(steps)
        for _ in steps:
            x = alloc()
            if args.save:
                batch.append(x)
            else:
                batch.append(None)
        duration = time.time() - start
        durations.append(duration)
        print(f'#{i + 1}: {duration:.3f} seconds')


if __name__ == '__main__':
    __main__()
```

### Versions

PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.2.1 (arm64)
GCC version: Could not collect
Clang version: 12.0.5 (clang-1205.0.22.9)
CMake version: version 3.22.1
Libc version: N/A

Python version: 3.9.9 (main, Jan 15 2022, 13:38:04)  [Clang 12.0.5 (clang-1205.0.22.9)] (64-bit runtime)
Python platform: macOS-12.2.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.1
[pip3] torch==1.10.1
[conda] Could not collect

cc @mruberry @rgommers