Repository: pytorch
Issue: [ROCm] [APU] Incorrect call of HIP mem outage
Labels: module: rocm,triaged

```
HSA_OVERRIDE_GFX_VERSION=9.0.0 \
        CL_DEVICE_GLOBAL_FREE_MEMORY_AMD=24396768 \
        CL_DEVICE_GLOBAL_MEM_SIZE=25189056512 \
        CL_DEVICE_MAX_MEM_ALLOC_SIZE=21410698032 \
        PYTORCH_HIP_MEM_ALLOC=strict PYTORCH_NO_HIP_MEMORY_CACHING=1 AMD_SERIALIZE_KERNEL=3 TORCH_USE_HIP_DSA=1 \
        HSA_ENABLE_SDMA=0 HSA_ENABLE_INTERRUPT=0 AMD_LOG_LEVEL=1 AMD_LOG_MASK=0x1 HIP_LAUNCH_BLOCKING=0 GPU_DUMP_CODE_OBJECT=0 AMD_SERIALIZE_COPY=3 AMD_KERNEL_DISPATCH=1 GPU_MAX_HW_QUEUES=0 \
        HIP_HIDDEN_FREE_MEM=16384 GPU_STREAMOPS_CP_WAIT=1 PAL_ALWAYS_RESIDENT=1 REMOTE_ALLOC=1 HIP_HOST_COHERENT=1 HIP_MEM_POOL_SUPPORT=1 GPU_MAX_REMOTE_MEM_SIZE=8192 HIP_VMEM_MANAGE_SUPPORT=0 \
        PYTORCH_HIP_ALLOC_CONF=garbage_collection_threshold:0.25,max_split_size_mb:2560,expandable_segments:True \
        python3.11 main.py --highvram --disable-smart-memory --disable-cuda-malloc --listen 127.0.0.1 --auto-launch \
        --verbose ERROR --port 4096 --preview-method latent2rgb \
        --output-directory /opt/local/.8A/terminal/ComfyAI/Downloads/04112025
```
```
## System Information
- **ComfyUI Version:** unknown
- **Arguments:** main.py --highvram --disable-smart-memory --disable-cuda-malloc --listen 127.0.0.1 --auto-launch --verbose ERROR --port 4096 --preview-method latent2rgb --output-directory /opt/local/.8A/terminal/ComfyAI/Downloads/04112025
- **OS:** posix
- **Python Version:** 3.11.9 (main, Jul  9 2024, 00:31:01) [GCC 14.1.1 20240522]
- **Embedded Python:** false
- **PyTorch Version:** 2.5.1+rocm6.3.0
## Devices

- **Name:** cuda:0 AMD Radeon Graphics : native
  - **Type:** cuda
  - **VRAM Total:** 25189052416
  - **VRAM Free:** 4532178944
  - **Torch VRAM Total:** 0
  - **Torch VRAM Free:** 0
```
### ROCmInfo
```
ROCk module is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.14
Runtime Ext Version:     1.6
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    AMD Ryzen 7 PRO 4750G with Radeon Graphics
  Uuid:                    CPU-XX                             
  Marketing Name:          AMD Ryzen 7 PRO 4750G with Radeon Graphics
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   3600                               
  BDFID:                   0                                  
  Internal Node ID:        0                                  
  Compute Unit:            16                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Memory Properties:       
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: FINE GRAINED        
      Size:                    49197372(0x2eeb13c) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    49197372(0x2eeb13c) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 3                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    49197372(0x2eeb13c) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 4                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    49197372(0x2eeb13c) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
*******                  
Agent 2                  
*******                  
  Name:                    gfx90c                             
  Uuid:                    GPU-XX                             
  Marketing Name:          AMD Radeon Graphics                
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
    L2:                      1024(0x400) KB                     
  Chip ID:                 5686(0x1636)                       
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   2100                               
  BDFID:                   1024                               
  Internal Node ID:        1                                  
  Compute Unit:            8                                  
  SIMDs per CU:            4                                  
  Shader Engines:          1                                  
  Shader Arrs. per Eng.:   1                                  
  WatchPts on Addr. Ranges:4                                  
  Coherent Host Access:    FALSE                              
  Memory Properties:       APU
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                               
  Wavefront Size:          64(0x40)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        40(0x28)                           
  Max Work-item Per CU:    2560(0xa00)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Packet Processor uCode:: 472                                
  SDMA engine uCode::      40                                 
  IOMMU Support::          None                               
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    24598684(0x177589c) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    24598684(0x177589c) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 3                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Recommended Granule:0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx90c:xnack-   
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*** Done ***
```
### ImageOnlyCheckpointSave
```
!!! Exception during processing !!! HIP error: out of memory
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

Traceback (most recent call last):
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/execution.py", line 328, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/execution.py", line 203, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/execution.py", line 174, in _map_node_over_list
    process_inputs(input_dict, i)
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/execution.py", line 163, in process_inputs
    results.append(getattr(obj, func)(**inputs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy_extras/nodes_video_model.py", line 121, in save
    comfy_extras.nodes_model_merging.save_checkpoint(model, clip_vision=clip_vision, vae=vae, filename_prefix=filename_prefix, output_dir=self.output_dir, prompt=prompt, extra_pnginfo=extra_pnginfo)
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy_extras/nodes_model_merging.py", line 222, in save_checkpoint
    comfy.sd.save_checkpoint(output_checkpoint, model, clip, vae, clip_vision, metadata=metadata, extra_keys=extra_keys)
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy/sd.py", line 960, in save_checkpoint
    model_management.load_models_gpu(load_models, force_patch_weights=True)
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy/model_management.py", line 526, in load_models_gpu
    loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights)
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy/model_management.py", line 342, in model_load
    self.model_use_more_vram(use_more_vram, force_patch_weights=force_patch_weights)
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy/model_management.py", line 371, in model_use_more_vram
    return self.model.partially_load(self.device, extra_memory, force_patch_weights=force_patch_weights)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy/model_patcher.py", line 759, in partially_load
    raise e
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy/model_patcher.py", line 756, in partially_load
    self.load(device_to, lowvram_model_memory=current_used + extra_memory, force_patch_weights=force_patch_weights, full_load=full_load)
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy/model_patcher.py", line 601, in load
    self.patch_weight_to_device("{}.{}".format(n, param), device_to=device_to)
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy/model_patcher.py", line 513, in patch_weight_to_device
    out_weight = comfy.lora.calculate_weight(self.patches[key], temp_weight, key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/.8A/terminal/ComfyAI/ComfyUI/comfy/lora.py", line 497, in calculate_weight
    weight += function(strength * comfy.model_management.cast_to_device(diff, weight.device, weight.dtype))
                       ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: HIP error: out of memory
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.


fatal: No names found, cannot describe anything.
```
### Reproductions,
(1) Download & run this workflow for Comfy on your AMD APU HW system. https://pastebin.com/qfKUrJc9, with downloading Shutt1eMix and Shutt1e3D Official SFT ckpt model images (CivitAI) onto `$COMFY_ROOT/models/unet`, and Comfy’s ClipVisionG (HF) onto `$COMFY_ROOT/models/clip_vision`,
(2) Look at checkpoints saving phase and also with Plasma system monitre,
(3) Workflow previews and demonstraitions of errors in pink and the not so outaged HW memories but still HIP mem outage somewhy,
![spectac1e 120802 122324](https://github.com/user-attachments/assets/9f699eb1-0b10-44d9-9f93-63d0f96457c7)
![spectac1e 131901 122324](https://github.com/user-attachments/assets/6f743323-1c4a-4b49-ad08-ba3bcbf27f27)
### HW
AMD Zen2 R4750G APU (HSA 9.0.0) × 14.9GiB UniVRAM’s with OpenCL-ROCm 6.3.0            

cc @ROCm @ComfyAnonymous @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd