Repository: pytorch
Issue: RuntimeError: free_upper_bound + pytorch_used_bytes[device]
Labels: module: cuda,triaged,module: CUDACachingAllocator

### 🐛 Describe the bug

I wasn't sure if this belonged here, but since the error message specify to create a bug report for Pytorch I will

Traceback
```
Traceback (most recent call last):
  File "C:\AI Stuff\ComfyUI\ComfyUI\execution.py", line 152, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\execution.py", line 82, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\execution.py", line 75, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Bringing-Old-Photos-Back-to-Life\nodes.py", line 300, in run
    return RestoreOldPhotos.restore(image, bopbtl_models, scratch_mask)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Bringing-Old-Photos-Back-to-Life\nodes.py", line 291, in restore
    restored_image = model.inference(transformed_image, transformed_mask)[0]
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Bringing-Old-Photos-Back-to-Life\Global\models\mapping_model.py", line 344, in inference
    label_feat_map=self.mapping_net.inference_forward(label_feat.detach(),inst_data)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Bringing-Old-Photos-Back-to-Life\Global\models\NonLocal_feature_mapping_model.py", line 195, in inference_forward
    x2 = self.NL_scale_1.inference_forward(x1,mask)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Bringing-Old-Photos-Back-to-Life\Global\models\networks.py", line 773, in inference_forward
    concat_1=self.F_Combine(concat_1)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\NEWvenv\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\NEWvenv\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\NEWvenv\Lib\site-packages\torch\nn\modules\conv.py", line 458, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\AI Stuff\ComfyUI\ComfyUI\NEWvenv\Lib\site-packages\torch\nn\modules\conv.py", line 454, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: free_upper_bound + pytorch_used_bytes[device] <= device_total INTERNAL ASSERT FAILED at "C:\\actions-runner\\_work\\pytorch\\pytorch\\builder\\windows\\pytorch\\c10\\cuda\\CUDAMallocAsyncAllocator.cpp":542, please report a bug to PyTorch.
```


### Versions


Versions
```
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Home
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.30.2
Libc version: N/A

Python version: 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22635-SP0
Is CUDA available: True
CUDA runtime version: 12.6.20
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060
Nvidia driver version: 560.81
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3600
DeviceID=CPU0
Family=107
L2CacheSize=3072
L2CacheSpeed=
Manufacturer=AuthenticAMD
MaxClockSpeed=3600
Name=AMD Ryzen 5 5500
ProcessorType=3
Revision=20480

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] onnx==1.16.2
[pip3] onnxruntime==1.18.1
[pip3] onnxruntime-gpu==1.18.1
[pip3] open_clip_torch==2.26.1
[pip3] pytorch-lightning==2.4.0
[pip3] torch==2.4.0+cu121
[pip3] torchaudio==2.4.0+cu121
[pip3] torchmetrics==1.4.1
[pip3] torchsde==0.2.6
[pip3] torchvision==0.19.0+cu121
```

cc @ptrblck @msaroufim