Repository: pytorch
Issue: Small model occupies too much GPU in CUDA11.1 + Torch1.8.1 but is normal in Torch 1.6 + CUDA10.1
Labels: module: performance,module: cuda,module: memory usage,triaged

## 🐛 Bug

At the forward step, pass a small ResNet model occupies me more than 15G GPU with torch1.8. Torch1.6 does not have this issue using the same code.

## To Reproduce

Steps to reproduce the behavior:

1. I reimplement the code https://github.com/zlai0/MAST on DAVIS dataset and their pretrained model.
2. I first tested the code on the first environment with Torch1.6 + CUDA10.1 + 2080Ti, with the --ref 2 as said in the README.md of the repo, the model only requires about **6G GPU** memory.
3. Then I tested the code on 3090, Torch1.8 + CUDA11.1, with the --ref 2, the model requires about **20G GPU** memory. There is a large memory gap in the two environments.
4. I tracked the memory changes and found that the memory cost (more than 15G) was located at the backbone (ResNet18).
5. Memory significantly changes code place (https://github.com/zlai0/MAST -> .\models\mast.py -> 32 row: feats_r = [self.post_convolution(self.feature_extraction(rgb)) for rgb in rgb_r]. ( This step contains a ResNet18 with a small convolution layer)
6. I test the code 32 row in CPU environment using torchstat stat(self.feature_extraction, (3, 480, 910)) and found that ResNet only occupies 1241.54MB in CPU.

Thanks!

## Expected behavior
How to solve the issue that GPU memory cost significantly increase when passing the small model and what happened there. My 3090 does not support the low version PyTorch, so I have to solve this issue.

## Environment

PyTorch version: 1.8.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.2.142
GPU models and configuration: GPU 0: GeForce RTX 3090
Nvidia driver version: 460.67
cuDNN version: Probably one of the following:
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.8.1+cu111
[pip3] torchstat==0.0.7
[pip3] torchvision==0.9.1+cu111
[conda] blas                      1.0                         mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl                       2017.0.3                      0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] torch                     1.8.1+cu111              pypi_0    pypi
[conda] torchstat                 0.0.7                    pypi_0    pypi
[conda] torchvision               0.9.1+cu111              pypi_0    pypi

```

 - PyTorch Version (e.g., 1.0): 1.8.1
 - OS (e.g., Linux): Linux
 - How you installed PyTorch (`conda`, `pip`, source): pip
 - Build command you used (if compiling from source):
 - Python version: 3.6.2
 - CUDA/cuDNN version: 11.1
 - GPU models and configuration: code at https://github.com/zlai0/MAST, inference on DAVIS dataset with their pretrained model.
 - Any other relevant information:

Thanks!



cc @ngimel @VitalyFedyunin