Repository: pytorch
Issue: Pytorch 2.1 / 2.2 uses 4x more memory during data loading than Pytorch 2.4 or 1.13
Labels: needs reproduction,module: dataloader,module: memory usage,triaged,module: regression

### 🐛 Describe the bug

In the process of upgrading from torch 1.13.1 we are hitting a memory leak during data loading.

Using mem top I started observing many entries which seem to be leaking tensors.


```
bytes:
47775816         (broken __repr__, type=<class 'torch.storage.UntypedStorage'>)
...
```


After some time I also saw segfaults and when compiled latest main with debug the following message:


```
RuntimeError: owning_weak_ptr == NullType::singleton() || owning_weak_ptr->weakcount_.load() > 1 || (owning_weak_ptr->refcount_.load() == 0 && owning_weak_ptr->weakcount_.load() > 0) INTERNAL ASSERT FAILED at "/home/piotr/pytorch/c10/util/intrusive_ptr.h":910, please report a bug to PyTorch. weak_intrusive_ptr: Can only weak_intrusive_ptr::reclaim() owning pointers that were created using weak_intrusive_ptr::release().
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/home/piotr/pytorch/torch/multiprocessing/reductions.py", line 574, in reduce_storage
    shared_cache[cache_key] = StorageWeakRef(storage)
  File "/home/piotr/pytorch/torch/multiprocessing/reductions.py", line 84, in __setitem__
    self.free_dead_references()
  File "/home/piotr/pytorch/torch/multiprocessing/reductions.py", line 89, in free_dead_references
    if storage_ref.expired():
  File "/home/piotr/pytorch/torch/multiprocessing/reductions.py", line 46, in expired
    return torch.Storage._expired(self.cdata)  # type: ignore[attr-defined]
  File "/home/piotr/pytorch/torch/storage.py", line 1133, in _expired
    return UntypedStorage._expired(*args, **kwargs)
RuntimeError: owning_weak_ptr == NullType::singleton() || owning_weak_ptr->weakcount_.load() > 1 || (owning_weak_ptr->refcount_.load() == 0 && owning_weak_ptr->weakcount_.load() > 0) INTERNAL ASSERT FAILED at "/home/piotr/pytorch/c10/util/intrusive_ptr.h":910, please report a bug to PyTorch. weak_intrusive_ptr: Can only weak_intrusive_ptr::reclaim() owning pointers that were created using weak_intrusive_ptr::release().
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/home/piotr/pytorch/torch/multiprocessing/reductions.py", line 574, in reduce_storage
    shared_cache[cache_key] = StorageWeakRef(storage)
  File "/home/piotr/pytorch/torch/multiprocessing/reductions.py", line 84, in __setitem__
    self.free_dead_references()
  File "/home/piotr/pytorch/torch/multiprocessing/reductions.py", line 89, in free_dead_references
    if storage_ref.expired():
  File "/home/piotr/pytorch/torch/multiprocessing/reductions.py", line 46, in expired
    return torch.Storage._expired(self.cdata)  # type: ignore[attr-defined]
  File "/home/piotr/pytorch/torch/storage.py", line 1133, in _expired
    return UntypedStorage._expired(*args, **kwargs)
RuntimeError: owning_weak_ptr == NullType::singleton() || owning_weak_ptr->weakcount_.load() > 1 || (owning_weak_ptr->refcount_.load() == 0 && owning_weak_ptr->weakcount_.load() > 0) INTERNAL ASSERT FAILED at "/home/piotr/pytorch/c10/util/intrusive_ptr.h":910, please report a bug to PyTorch. weak_intrusive_ptr: Can only weak_intrusive_ptr::reclaim() owning pointers that were created using weak_intrusive_ptr::release().
```

So far I haven't been able to isolate a minimal reproduction. In my implementation the data loader does quite a bit of work and is not trivial.

### Versions

Collecting environment information...
PyTorch version: 2.4.0a0+git3beb9d8
Is debug build: True
CUDA used to build PyTorch: 12.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-1037-aws-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 545.23.08
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          32
On-line CPU(s) list:             0-31
Vendor ID:                       AuthenticAMD
Model name:                      AMD EPYC 7R32
CPU family:                      23
Model:                           49
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       1
Stepping:                        0
BogoMIPS:                        5599.99
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       512 KiB (16 instances)
L1i cache:                       512 KiB (16 instances)
L2 cache:                        8 MiB (16 instances)
L3 cache:                        64 MiB (4 instances)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-31
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] flake8==7.0.0
[pip3] msgpack-numpy==0.4.8
[pip3] mypy==1.9.0
[pip3] mypy-boto3-ec2==1.34.73
[pip3] mypy-boto3-s3==1.28.55
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.4
[pip3] onnx==1.14.0
[pip3] onnx-graphsurgeon==0.3.27
[pip3] onnxsim==0.4.36
[pip3] optree==0.11.0
[pip3] pytorch-lightning==2.1.3
[pip3] pytorch-quantization==2.1.2
[pip3] torch==2.4.0a0+git3beb9d8
[pip3] torchmetrics==0.10.0
[pip3] torchpack==0.3.1
[pip3] torchvision==0.19.0a0+2c4665f
[conda] Could not collect

cc @SsnL @VitalyFedyunin @ejguan @dzhulgakov