Repository: EleutherAI/lm-evaluation-harness
Issue: Generate using vllm and ray and not gpu detected 
Root cause: Configuration Error

I am trying to use vllm to do the generation but encountered following issue. 

Error: 
```
ValueError: Current node has no GPU available. current_node_resource={'memory': 803011488768.0, 'bundle_group_2_5999116f8a6e3cc0472c77e709e801000000': 1000.0, 'GPU_group_0_5999116f8a6e3cc0472c77e709e801000000' : 1.0, 'node:__internal_head__': 1.0, 'node:131.179.168.116_group_5999116f8a6e3cc0472c77e709e801000000': 0.001, 'accelerator_type:A100': 1.0, 'node:131.179.168.116_group_0_5999116f8a6e3cc0472c77e709e801000000'
: 0.001, 'object_store_memory': 199995880792.0, 'bundle_group_3_5999116f8a6e3cc0472c77e709e801000000': 1000.0, 'GPU_group_2_5999116f8a6e3cc0472c77e709e801000000': 1.0, 'bundle_group_5999116f8a6e3cc0472c77e709e
801000000': 4000.0, 'bundle_group_1_5999116f8a6e3cc0472c77e709e801000000': 1000.0, 'GPU_group_1_5999116f8a6e3cc0472c77e709e801000000': 1.0, 'GPU_group_3_5999116f8a6e3cc0472c77e709e801000000': 1.0, 'node:131.17
9.168.116': 0.999, 'GPU_group_5999116f8a6e3cc0472c77e709e801000000': 4.0, 'bundle_group_0_5999116f8a6e3cc0472c77e709e801000000': 1000.0, 'CPU': 92.0}. vLLM engine cannot start without GPU. Make sure you have a
t least 1 GPU available in a node current_node_id='da16a20c6521a2e0e888d6115f7e5b6935a719ce5ab2d2b99f1c301a' current_ip='131.179.168.116'.
```

The command I am using is ```lm_eval --model vllm     --model_args pretrained=mistralai/Mistral-7B-v0.1,tensor_parallel_size=4,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=1    --tasks gsm8k     --batch_size autoray status```

I checked the torch.cuda.is_available() and returned true. 

The ray status also shows the gpu information: 
```
======== Autoscaler status: 2024-10-19 21:02:30.697323 ========                                                                                                                                                  
Node status                                                                                                                                                                                                      
---------------------------------------------------------------                                                                                                                                                  
Active:                                                                                                                                                                                                          
 1 node_da16a20c6521a2e0e888d6115f7e5b6935a719ce5ab2d2b99f1c301a                                                                                                                                                 
Pending:                                                                                                                                                                                                         
 (no pending nodes)                                                                                                                                                                                              
Recent failures:                                                                                                                                                                                                 
 (no failures)                                                                                                                                                                                                   
                                                                                                                                                                                                                 
Resources                                                                                                                                                                                                        
---------------------------------------------------------------                                                                                                                                                  
Usage:                                                                                                                                                                                                           
 0.0/96.0 CPU                                                                                                                                                                                                    
 1.0/4.0 GPU (0 used of 0.0 reserved in placement groups)                                                                                                                                                        
 0B/747.86GiB memory                                                                                                                                                                                             
 0B/186.26GiB object_store_memory  
```

Any one has similar issue? 

Hi! are you able to run it using the Offline Batched Inference guide [here](https://docs.vllm.ai/en/latest/getting_started/quickstart.html)

Hey, thanks for your response. I figured it out and it is because I made the allocation incorrectly. 

Thank you!

I got the same issue when data_parallel_size>1: `RuntimeError: No CUDA GPUs are available`, and I have set CUDA_VISIBLE_DEVICES. 
![image](https://github.com/user-attachments/assets/ee7867f0-4cfc-48f8-8ef7-53fd07a87456)



> Hey, thanks for your response. I figured it out and it is because I made the allocation incorrectly.
> 
> Thank you!

@mandyyyyii  How do you fix it? Can you show the details?