Repository: pytorch
Issue: OOM Message is truncated
Labels: needs reproduction,module: cuda,module: error checking,triaged

### 🐛 Describe the bug

On my system, the OutOfMemory message does not show information about the OOM event such as how much free memory was present and how much was reserved.

Reproduction:

```
import torch
torch.zeros(1000000000000, device='cuda')
```

Result:

```
OutOfMemoryError: CUDA out of memory. Tried to allocate 3725.29 GiB. GPU
```

Expected something similar to:

```
OutOfMemoryError: CUDA out of memory. Tried to allocate 29.07 GiB. GPU 3 has a total capacity of 39.43 GiB of which 25.15 GiB is free. Including non-PyTorch memory, this process has 14.27 GiB memory in use. Of the allocated memory 11.74 GiB is allocated by PyTorch, and 932.02 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
```

### Versions

Collecting environment information...                                                                                                                                                    
PyTorch version: 2.3.0                                                                                                                                                                   
Is debug build: False                                                                                                                                                                    
CUDA used to build PyTorch: 11.8                                                                                                                                                         
ROCM used to build PyTorch: N/A                                                                                                                                                          
                                                                                                                                                                                         
OS: Ubuntu 22.04.3 LTS (x86_64)                                                                                                                                                          
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0                                                                                                                                       
Clang version: Could not collect                                                                                                                                                         
CMake version: Could not collect                                                                                                                                                         
Libc version: glibc-2.35                                                                                                                                                                 
                                                                                                                                                                                         
Python version: 3.10.11 | packaged by conda-forge | (main, May 10 2023, 18:58:44) [GCC 11.3.0] (64-bit runtime)                                                                          
Python platform: Linux-6.5.0-14-generic-x86_64-with-glibc2.35                                                                                                                            
Is CUDA available: True                                                                                                                                                                  
CUDA runtime version: 11.8.89                                                                                                                                                            
CUDA_MODULE_LOADING set to: LAZY                                                                                                                                                         
GPU models and configuration:                                                                                                                                                            
GPU 0: NVIDIA A100-SXM4-40GB                                                                                                                                                             
GPU 1: NVIDIA A100-SXM4-40GB                                                                                                                                                             
GPU 2: NVIDIA A100-SXM4-40GB                                                                                                                                                             
GPU 3: NVIDIA A100-SXM4-40GB                                                                                                                                                             
                                                                                                                                                                                         
Nvidia driver version: 525.147.05                                                                                                                                                        
cuDNN version: Could not collect                                                                                                                                                         
HIP runtime version: N/A                                                                                                                                                                 
MIOpen runtime version: N/A                                                                                                                                                              
Is XNNPACK available: True                                                                                                                                                               
                                                                                                                                                                                         
CPU:                                                                                                                                                                                     
Architecture:                       x86_64                                                                                                                                               
CPU op-mode(s):                     32-bit, 64-bit                                                                                                                                       
Address sizes:                      48 bits physical, 48 bits virtual                                                                                                                    
Byte Order:                         Little Endian                                                                                                                                        
CPU(s):                             128                                                                                                                                                  
On-line CPU(s) list:                0-127                                                                                                                                                
Vendor ID:                          AuthenticAMD                                                                                                                                         
Model name:                         AMD EPYC 7543 32-Core Processor                                                                                                                      
CPU family:                         25                                                                                                                                                   
Model:                              1                                                                                                                                                    
Thread(s) per core:                 2                                                                                                                                                    
Core(s) per socket:                 32                                                                                                                                                   
Socket(s):                          2                                                                                                                                                    
Stepping:                           1                                                                                                                                                    
Frequency boost:                    enabled                                                                                                                                              
CPU max MHz:                        3737.8899                                                                                                                                            
CPU min MHz:                        1500.0000                                                                                                                                            
BogoMIPS:                           5589.56
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm    constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm          cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single  hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc    cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin brs arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca                                                                                         
Virtualization:                     AMD-V                                                                                                                                                
L1d cache:                          2 MiB (64 instances)                                                                                                                                 
L1i cache:                          2 MiB (64 instances)                                                                                                                                 
L2 cache:                           32 MiB (64 instances)                                                                                                                                
L3 cache:                           512 MiB (16 instances)                                                                                                                               
NUMA node(s):                       2                                                                                                                                                    
NUMA node0 CPU(s):                  0-31,64-95                                                                                                                                           
NUMA node1 CPU(s):                  32-63,96-127                                                                                                                                         
Vulnerability Gather data sampling: Not affected                                                                                                                                         
Vulnerability Itlb multihit:        Not affected                                                                                                                                         
Vulnerability L1tf:                 Not affected                                                                                                                                         
Vulnerability Mds:                  Not affected                                                                                                                                         
Vulnerability Meltdown:             Not affected                                                                                                                                         
Vulnerability Mmio stale data:      Not affected                                                                                                                                         
Vulnerability Retbleed:             Not affected                                                                                                                                         
Vulnerability Spec rstack overflow: Mitigation; safe RET                                                                                                                                 
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl                                                                                              
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization                                                                                 
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected                                            
Vulnerability Srbds:                Not affected                                                                                                                                         
Vulnerability Tsx async abort:      Not affected                                                                                                                                         
                                                                                                                                                                                         
Versions of relevant libraries:                                                                                                                                                          
[pip3] functorch==0.2.1                                                                                                                                                                  
[pip3] numpy==1.26.4                                                                                                                                                                     
[pip3] pytorch-lightning==2.1.0                                                                                                                                                          
[pip3] pytorch_optimizer==2.12.0                                                                                                                                                         
[pip3] torch==2.3.0                                                                                                                                                                      
[pip3] torchaudio==2.3.0                                                                                                                                                                 
[pip3] torchinfo==1.8.0                                                                                                                                                                  
[pip3] torchmetrics==1.2.0                                                                                                                                                               
[pip3] torchvision==0.18.0                                                                                                                                                               
[pip3] triton==2.3.0                                                                                                                                                                     
[conda] blas                      2.116                       mkl    conda-forge                                                                                                         
[conda] blas-devel                3.9.0            16_linux64_mkl    conda-forge                                                                                                         
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch                                                                                                             
[conda] functorch                 0.2.1                    pypi_0    pypi                                                                                                                
[conda] libblas                   3.9.0            16_linux64_mkl    conda-forge                                                                                                         
[conda] libcblas                  3.9.0            16_linux64_mkl    conda-forge                                                                                                         
[conda] libjpeg-turbo             2.0.0                h9bf148f_0    pytorch                                                                                                             
[conda] liblapack                 3.9.0            16_linux64_mkl    conda-forge                                                                                                         
[conda] liblapacke                3.9.0            16_linux64_mkl    conda-forge
[conda] pytorch-mutex             1.0                        cuda    pytorch                                                                                                             
[conda] pytorch-optimizer         2.10.1                   pypi_0    pypi                                                                                                                
[conda] torchaudio                2.3.0               py310_cu118    pytorch                                                                                                             
[conda] torchmetrics              1.2.0              pyhd8ed1ab_0    conda-forge                                                                                                         
[conda] torchtriton               2.3.0                     py310    pytorch                                                                                                             
[conda] torchvision               0.18.0              py310_cu118    pytorch

cc @ptrblck @msaroufim @malfet