Repository: pytorch
Issue: Python sometimes crashes inexplicably, with "/var/log/kernel. log" displaying index errors
Labels: needs reproduction,module: crash,module: cuda,triaged

### 🐛 Describe the bug

I opened two python programs  on a linux machine . using the same gpu . However sometimes python programes crash , even when i just open a program , the `/var/log/kern.log` immediately display an error message . Here are some information below 

```
2024-05-15T10:09:25.118507+08:00 trana6b-Default-string kernel: [  826.919949] UBSAN: array-index-out-of-bounds in /tmp/selfgz2789/NVIDIA-Linux-x86_64-535.154.05/kernel/nvidia-uvm/uvm_pmm_gpu.c:857:39
2024-05-15T10:09:25.118508+08:00 trana6b-Default-string kernel: [  826.919950] index 0 is out of range for type 'uvm_gpu_chunk_t *[*]'
2024-05-15T10:09:25.118508+08:00 trana6b-Default-string kernel: [  826.919951] CPU: 11 PID: 5286 Comm: python Tainted: P           OE      6.5.0-9-generic #9-Ubuntu
2024-05-15T10:09:25.118508+08:00 trana6b-Default-string kernel: [  826.919952] Hardware name: Gigabyte Technology Co., Ltd. Z790 AORUS ELITE AX/Z790 AORUS ELITE AX, BIOS FHe 12/08/2023
2024-05-15T10:09:25.118508+08:00 trana6b-Default-string kernel: [  826.919953] Call Trace:
2024-05-15T10:09:25.118508+08:00 trana6b-Default-string kernel: [  826.919953]  <TASK>
2024-05-15T10:09:25.118509+08:00 trana6b-Default-string kernel: [  826.919954]  dump_stack_lvl+0x48/0x70
2024-05-15T10:09:25.118509+08:00 trana6b-Default-string kernel: [  826.919956]  dump_stack+0x10/0x20
2024-05-15T10:09:25.118509+08:00 trana6b-Default-string kernel: [  826.919957]  __ubsan_handle_out_of_bounds+0xc6/0x110
2024-05-15T10:09:25.118509+08:00 trana6b-Default-string kernel: [  826.919959]  merge_gpu_chunk+0xc6/0x1d0 [nvidia_uvm]
2024-05-15T10:09:25.118510+08:00 trana6b-Default-string kernel: [  826.919987]  free_chunk_with_merges+0x13d/0x180 [nvidia_uvm]
2024-05-15T10:09:25.118510+08:00 trana6b-Default-string kernel: [  826.920013]  free_chunk+0xa4/0xd0 [nvidia_uvm]
2024-05-15T10:09:25.118510+08:00 trana6b-Default-string kernel: [  826.920039]  uvm_pmm_gpu_free+0xbf/0xf0 [nvidia_uvm]
2024-05-15T10:09:25.118510+08:00 trana6b-Default-string kernel: [  826.920064]  phys_mem_deallocate+0x33/0xd0 [nvidia_uvm]
2024-05-15T10:09:25.118511+08:00 trana6b-Default-string kernel: [  826.920093]  uvm_page_tree_put_ptes_async+0x4d5/0x580 [nvidia_uvm]
2024-05-15T10:09:25.118511+08:00 trana6b-Default-string kernel: [  826.920123]  uvm_page_table_range_vec_deinit+0x3e/0xd0 [nvidia_uvm]
2024-05-15T10:09:25.118511+08:00 trana6b-Default-string kernel: [  826.920151]  uvm_ext_gpu_map_destroy+0xd7/0x1f0 [nvidia_uvm]
2024-05-15T10:09:25.118511+08:00 trana6b-Default-string kernel: [  826.920176]  uvm_va_range_destroy+0x324/0x590 [nvidia_uvm]
2024-05-15T10:09:25.118511+08:00 trana6b-Default-string kernel: [  826.920203]  ? _nv025923rm+0x2b/0xf0 [nvidia]
2024-05-15T10:09:25.118512+08:00 trana6b-Default-string kernel: [  826.920401]  ? _nv043203rm+0xe9/0x1c0 [nvidia]
2024-05-15T10:09:25.118512+08:00 trana6b-Default-string kernel: [  826.920648]  uvm_api_free+0x188/0x320 [nvidia_uvm]
2024-05-15T10:09:25.118512+08:00 trana6b-Default-string kernel: [  826.920667]  uvm_ioctl+0xf6e/0x1cd0 [nvidia_uvm]
2024-05-15T10:09:25.118512+08:00 trana6b-Default-string kernel: [  826.920683]  ? _raw_spin_lock_irqsave+0xe/0x20
2024-05-15T10:09:25.118513+08:00 trana6b-Default-string kernel: [  826.920684]  ? os_acquire_spinlock+0x12/0x30 [nvidia]
2024-05-15T10:09:25.118513+08:00 trana6b-Default-string kernel: [  826.920828]  ? os_release_spinlock+0x1a/0x30 [nvidia]
2024-05-15T10:09:25.118513+08:00 trana6b-Default-string kernel: [  826.920970]  ? _nv047682rm+0xed/0x1d0 [nvidia]
2024-05-15T10:09:25.118513+08:00 trana6b-Default-string kernel: [  826.921113]  ? _nv043407rm+0x77/0xd0 [nvidia]
2024-05-15T10:09:25.118513+08:00 trana6b-Default-string kernel: [  826.921263]  ? _nv011756rm+0x86/0xa0 [nvidia]
2024-05-15T10:09:25.118514+08:00 trana6b-Default-string kernel: [  826.921413]  ? _raw_spin_lock_irqsave+0xe/0x20
2024-05-15T10:09:25.118514+08:00 trana6b-Default-string kernel: [  826.921414]  ? _raw_spin_lock_irqsave+0xe/0x20
2024-05-15T10:09:25.118514+08:00 trana6b-Default-string kernel: [  826.921415]  ? thread_context_non_interrupt_add+0x13a/0x2c0 [nvidia_uvm]
2024-05-15T10:09:25.118514+08:00 trana6b-Default-string kernel: [  826.921439]  uvm_unlocked_ioctl_entry.part.0+0x7b/0xf0 [nvidia_uvm]
2024-05-15T10:09:25.118515+08:00 trana6b-Default-string kernel: [  826.921455]  ? nvidia_ioctl+0x369/0x8a0 [nvidia]
2024-05-15T10:09:25.118515+08:00 trana6b-Default-string kernel: [  826.921595]  ? kfree+0x78/0x120
2024-05-15T10:09:25.118515+08:00 trana6b-Default-string kernel: [  826.921596]  ? nvidia_ioctl+0x369/0x8a0 [nvidia]
2024-05-15T10:09:25.118515+08:00 trana6b-Default-string kernel: [  826.921736]  uvm_unlocked_ioctl_entry+0x6b/0x90 [nvidia_uvm]
2024-05-15T10:09:25.118516+08:00 trana6b-Default-string kernel: [  826.921752]  __x64_sys_ioctl+0xa0/0xf0
2024-05-15T10:09:25.118516+08:00 trana6b-Default-string kernel: [  826.921753]  do_syscall_64+0x59/0x90
2024-05-15T10:09:25.118516+08:00 trana6b-Default-string kernel: [  826.921754]  ? syscall_exit_to_user_mode+0x37/0x60
2024-05-15T10:09:25.118516+08:00 trana6b-Default-string kernel: [  826.921756]  ? do_syscall_64+0x68/0x90
2024-05-15T10:09:25.118516+08:00 trana6b-Default-string kernel: [  826.921757]  ? rcu_core_si+0xe/0x20
2024-05-15T10:09:25.118517+08:00 trana6b-Default-string kernel: [  826.921757]  ? __do_softirq+0xd6/0x346
2024-05-15T10:09:25.118517+08:00 trana6b-Default-string kernel: [  826.921759]  ? hrtimer_interrupt+0x11f/0x250
2024-05-15T10:09:25.118517+08:00 trana6b-Default-string kernel: [  826.921759]  ? exit_to_user_mode_prepare+0x30/0xb0
2024-05-15T10:09:25.118517+08:00 trana6b-Default-string kernel: [  826.921761]  ? irqentry_exit_to_user_mode+0x17/0x20
2024-05-15T10:09:25.118519+08:00 trana6b-Default-string kernel: [  826.921762]  ? irqentry_exit+0x43/0x50
2024-05-15T10:09:25.118520+08:00 trana6b-Default-string kernel: [  826.921763]  ? sysvec_apic_timer_interrupt+0x4b/0xd0
2024-05-15T10:09:25.118520+08:00 trana6b-Default-string kernel: [  826.921764]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
2024-05-15T10:09:25.118520+08:00 trana6b-Default-string kernel: [  826.921765] RIP: 0033:0x7fd25c72396f
2024-05-15T10:09:25.118520+08:00 trana6b-Default-string kernel: [  826.921769] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 18 48 8b 44 24 18 64 48 2b 04 25 28 00 00
2024-05-15T10:09:25.118521+08:00 trana6b-Default-string kernel: [  826.921770] RSP: 002b:00007fd2111db6d0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
2024-05-15T10:09:25.118521+08:00 trana6b-Default-string kernel: [  826.921770] RAX: ffffffffffffffda RBX: 00007fd048ef01c0 RCX: 00007fd25c72396f
2024-05-15T10:09:25.118521+08:00 trana6b-Default-string kernel: [  826.921771] RDX: 00007fd2111db730 RSI: 0000000000000022 RDI: 0000000000000005
2024-05-15T10:09:25.118530+08:00 trana6b-Default-string kernel: [  826.921771] RBP: 00007fd2111db780 R08: 0000000000000000 R09: 0000000000000000
2024-05-15T10:09:25.118530+08:00 trana6b-Default-string kernel: [  826.921772] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
2024-05-15T10:09:25.118530+08:00 trana6b-Default-string kernel: [  826.921772] R13: 00007fd048ef01c0 R14: 00007fd2111db730 R15: 0000000000000005
2024-05-15T10:09:25.118530+08:00 trana6b-Default-string kernel: [  826.921773]  </TASK>
2024-05-15T10:09:25.118530+08:00 trana6b-Default-string kernel: [  826.921775] ================================================================================

```


I have sought help from the NVIDIA forum, but they have asked me to seek Torch's help

here's the link https://forums.developer.nvidia.com/t/linux-always-crash-when-run-the-python-program-using-gpu/292905 

Looking forward to a reply 

### Versions

torch version is different 
- one is `2.1.1`
- the  another is `2.0.0`

nvidia-smi  driver version : 
```
Driver Version: 535.154.05   CUDA Version: 12.2 
```
nvcc --version 
```
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
```

cc @ptrblck @msaroufim