Repository: pytorch
Issue: `torch.load(..., map_location="cuda:0")` allocates memory on both CPU and GPU
Labels: module: serialization,triaged,module: python frontend

### 🐛 Describe the bug

As per title, using `torch.load(..., map_location="cuda:0")` allocates memory on CPU while I think it should not.

Reproduce:

```
wget https://huggingface.co/openai/whisper-medium/resolve/main/pytorch_model.bin
```

Then:
```python
import os, psutil
import gc
import torch

process = psutil.Process(os.getpid())
print(f"Before torch.load: {process.memory_info().rss / 1e6:.2f} MB")

res = torch.load("/path/to/pytorch_model.bin", map_location=torch.device("cuda:0"))
gc.collect()
print(f"After torch.load: {process.memory_info().rss / 1e6:.2f} MB")
```

prints:
```
Before torch.load: 290.95 MB
After torch.load: 3355.66 MB
```

and you can check with `nvidia-smi` that memory is reserved as well on GPU.

### Versions

PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 14.0.0-1ubuntu1
CMake version: version 3.25.0
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-57-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU
Nvidia driver version: 515.86.01
cuDNN version: Probably one of the following:
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.7.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.7.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.4
[pip3] torch==1.13.1
[pip3] torch-model-archiver==0.6.1
[pip3] torch-tb-profiler==0.4.0
[pip3] torch-workflow-archiver==0.2.5
[pip3] torchaudio==0.13.0
[pip3] torchinfo==1.7.0
[pip3] torchserve==0.6.1
[pip3] torchtriton==2.0.0+0d7e753227
[pip3] torchvision==0.14.0
[conda] cudatoolkit               11.3.1               h2bc3f7f_2    anaconda
[conda] numpy                     1.22.4                   pypi_0    pypi
[conda] torch                     1.13.1                   pypi_0    pypi
[conda] torch-model-archiver      0.6.1                    pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                     dev_0    <develop>
[conda] torch-workflow-archiver   0.2.5                    pypi_0    pypi
[conda] torchaudio                0.13.0                   pypi_0    pypi
[conda] torchinfo                 1.7.0                    pypi_0    pypi
[conda] torchserve                0.6.1                    pypi_0    pypi
[conda] torchtriton               2.0.0+0d7e753227          pypi_0    pypi
[conda] torchvision               0.14.0                   pypi_0    pypi


cc @mruberry @albanD