Repository: pytorch
Issue: RuntimeError: CUDA unknown error even after sudo modprobe -r nvidia_uvm && sudo modprobe nvidia_uvm
Labels: module: cuda,triaged

### 🐛 Describe the bug

# The bug
## Minimal reproductive example

```python
>>> import torch
>>> torch.cuda.is_available()
$HOME/.conda/envs/my_env/lib/python3.10/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
False
```

## Related topics and tested solutions
The [following *closed* issue](https://github.com/pytorch/pytorch/issues/49081) and any other related issue, which will offer the following solution :

    sudo modprobe -r nvidia_uvm && sudo modprobe nvidia_uvm
    
 The problem is, on my machine, this command works only so rarely and the result is not constant over time. I also tried the 
  [following solution](https://discuss.pytorch.org/t/cuda-driver-initialization-failed-you-might-not-have-a-cuda-gpu/191316/2) which suggested disabling MIG but running the command they give yields :
  ```bash
  Unable to disable MIG Mode for GPU 00000000:01:00.0: Not Supported
Treating as warning and moving on.
All done.
  ```
 # My configuration
 ## Local
 Neofetch:
 ```bash
 OS: Fedora Linux 38 (Thirty Eight) x86_64 
    .:cccccccccccccccccccccccccc:.       Host: XPS 15 9520 
  .;ccccccccccccc;.:dddl:.;ccccccc;.     Kernel: 6.2.15-703.inttf.fc38.x86_64 
 .:ccccccccccccc;OWMKOOXMWd;ccccccc:.    Uptime: 30 mins 
.:ccccccccccccc;KMMc;cc;xMMc:ccccccc:.   Packages: 7322 (rpm), 22 (flatpak) 
,cccccccccccccc;MMM.;cc;;WW::cccccccc,   Shell: bash 5.2.21 
:cccccccccccccc;MMM.;cccccccccccccccc:   Resolution: 1920x1200 
:ccccccc;oxOOOo;MMM0OOk.;cccccccccccc:   WM: i3 
cccccc:0MMKxdd:;MMMkddc.;cccccccccccc;   Theme: Adwaita [GTK2/3] 
ccccc:XM0';cccc;MMM.;cccccccccccccccc'   Icons: Adwaita [GTK2/3] 
ccccc;MMo;ccccc;MMW.;ccccccccccccccc;    Terminal: terminator 
ccccc;0MNc.ccc.xMMd:ccccccccccccccc;     CPU: 12th Gen Intel i7-12700H (20) @ 4.600GHz 
cccccc;dNMWXXXWM0::cccccccccccccc:,      GPU: NVIDIA GeForce RTX 3050 Ti Mobile 
cccccccc;.:odl:.;cccccccccccccc:,.       GPU: Intel Alder Lake-P GT2 [Iris Xe Graphics] 
:cccccccccccccccccccccccccccc:'.         Memory: 4610MiB / 15649MiB
```

nvidia-smi :
```bash
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3050 ...    Off | 00000000:01:00.0 Off |                  N/A |
| N/A   50C    P8               5W /  35W |      8MiB /  4096MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1556      G   /usr/libexec/Xorg                             4MiB |
+---------------------------------------------------------------------------------------+
```
## Conda env

pip list (relevant packages) :
```bash
torch                    2.1.1
torchaudio               2.1.1
torchvision              0.16.1
nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu12   12.1.105
nvidia-cuda-nvrtc-cu12   12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12        8.9.2.26
nvidia-cufft-cu12        11.0.2.54
nvidia-curand-cu12       10.3.2.106
nvidia-cusolver-cu12     11.4.5.107
nvidia-cusparse-cu12     12.1.0.106
nvidia-nccl-cu12         2.18.1
nvidia-nvjitlink-cu12    12.3.101
nvidia-nvtx-cu12         12.1.105
```
python --version :
```bash
Python 3.10.13
```


I am out of option with the researches that I made. It worked at some point with older drivers but before I resort to installing older drivers (which aren't compatible to run X), I would like to know if anyone managed to run Pytorch with about the same architecture as I have ?

### Versions

```bash
Collecting environment information...
/home/grybouilli/.conda/envs/my_env/lib/python3.10/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
PyTorch version: 2.1.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Fedora Linux 38 (Thirty Eight) (x86_64)
GCC version: (GCC) 13.2.1 20231011 (Red Hat 13.2.1-4)
Clang version: 16.0.6 (Fedora 16.0.6-3.fc38)
CMake version: version 3.27.7
Libc version: glibc-2.37

Python version: 3.10.13 (main, Sep 11 2023, 13:21:10) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.2.15-703.inttf.fc38.x86_64-x86_64-with-glibc2.37
Is CUDA available: False
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3050 Ti Laptop GPU
Nvidia driver version: 535.129.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture :                          x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Tailles des adresses:                   39 bits physical, 48 bits virtual
Boutisme :                              Little Endian
Processeur(s) :                         20
Liste de processeur(s) en ligne :       0-19
Identifiant constructeur :              GenuineIntel
Nom de modèle :                         12th Gen Intel(R) Core(TM) i7-12700H
Famille de processeur :                 6
Modèle :                                154
Thread(s) par cœur :                    2
Cœur(s) par socket :                    14
Socket(s) :                             1
Révision :                              3
multiplication des MHz du/des CPU(s) :  59%
Vitesse maximale du processeur en MHz : 4700,0000
Vitesse minimale du processeur en MHz : 400,0000
BogoMIPS :                              5376,00
Drapeaux :                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
Virtualisation :                        VT-x
Cache L1d :                             544 KiB (14 instances)
Cache L1i :                             704 KiB (14 instances)
Cache L2 :                              11,5 MiB (8 instances)
Cache L3 :                              24 MiB (1 instance)
Nœud(s) NUMA :                          1
Nœud NUMA 0 de processeur(s) :          0-19
Vulnérabilité Itlb multihit :           Not affected
Vulnérabilité L1tf :                    Not affected
Vulnérabilité Mds :                     Not affected
Vulnérabilité Meltdown :                Not affected
Vulnérabilité Mmio stale data :         Not affected
Vulnérabilité Retbleed :                Not affected
Vulnérabilité Spec store bypass :       Mitigation; Speculative Store Bypass disabled via prctl
Vulnérabilité Spectre v1 :              Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnérabilité Spectre v2 :              Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnérabilité Srbds :                   Not affected
Vulnérabilité Tsx async abort :         Not affected

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==2.1.1
[pip3] torchaudio==2.1.1
[pip3] torchvision==0.16.1
[pip3] triton==2.1.0
[conda] numpy                     1.22.3          py310h4ef5377_2    conda-forge
[conda] torch                     2.1.1                    pypi_0    pypi
[conda] torchaudio                2.1.1                    pypi_0    pypi
[conda] torchvision               0.16.1                   pypi_0    pypi
[conda] triton                    2.1.0                    pypi_0    pypi
```

cc @ptrblck