Repository: simpler-env/SimplerEnv
Issue: Vulkan enumeratePhysicalDevices: ErrorInitializationFailed on Headless A100 Server
Root cause: Environment Incompatibility

**Describe the bug：**

I am unable to initialize a ManiSkill environment on a headless server equipped with an NVIDIA A100 GPU. The process fails with a Vulkan error: RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed.

This appears to be related to the NVIDIA driver and Vulkan ICD (Installable Client Driver) configuration, as system-level checks for NVIDIA's graphics libraries are also failing.

**Troubleshooting Steps Taken：**

Followed Headless Rendering Guide: I have followed the official documentation for Vulkan setup on headless servers: [https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/installation.html#vulkan](https://www.google.com/search?q=https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/installation.html%23vulkan)
I created three JSON files, but it still does not work.



I am encountering the same bug with H200 cards, with the following error message: `RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed.` However I suspect this is a problem with ManiSkill & Vulkan, so scrapping through the issues section at that repo may be a better place to find answers. If you have solved the problem, can you provide the solutions here? Would be very helpful.

I did not solve it yet....

As in https://github.com/simpler-env/SimplerEnv/issues/68#issuecomment-3252775532, try
```
DISPLAY="" CUDA_VISIBLE_DEVICES=0 VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json python xxx
(use other icd filenames path if different)
```

Thanks for the great work @xuanlinli17 , as well as the prompt reply. The solution I got working on a 8-card headless H200 server was reinstalling the CUDA driver to a newer version (in my case, I used `CUDA Version: 12.7, Driver Version: 565.57.01`), accompanied by following the instructions of https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/installation.html#vulkan. Using `numpy==1.24.4`, even without any display (either virtual or physical), I am able to run the environment. 

I believe such errors, as well as errors of `Segmentation Fault` during initialization of scene's cameras (which I also encountered), is mostly due to the inability of `vulkan` to find or utilize GPU devices. One can run `vulkaninfo` to check whether vulkan can find the GPUs. A successful snippet of running the command should contain sth similiar to this:
```bash
VK_LAYER_INTEL_nullhw (INTEL NULL HW) Vulkan version 1.1.73, layer version 1:
	Layer Extensions: count = 0
	Devices: count = 9
		GPU id = 0 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 1 (llvmpipe (LLVM 15.0.7, 256 bits))
		Layer-Device Extensions: count = 0

		GPU id = 2 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 3 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 4 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 5 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 6 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 7 (NVIDIA H200)
		Layer-Device Extensions: count = 0

		GPU id = 8 (NVIDIA H200)
		Layer-Device Extensions: count = 0
```