Repository: EleutherAI/lm-evaluation-harness
Issue: vLLM causing GPU memory leak with data_parallel_size=3
Root cause: Resource Mishandling

Not too sure if I should be raising this issue here or at vllm, but running
```
CUDA_VISIBLE_DEVICES=1,2,3 lm_eval --model vllm \
    --model_args pretrained=/hdd2/model/path/,tensor_parallel_size=1,dtype=half,gpu_memory_utilization=0.8,data_parallel_size=3,max_model_len=4096 \
    --tasks mmlu,gsm8k,winogrande \
    --batch_size auto \
    --output_path /home/path/to.output
```
results in vllm not deallocating memory after evaluation job is completed.

here is my nvidia-smi:

```+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA TITAN RTX               On  | 00000000:06:00.0 Off |                  N/A |
| 41%   26C    P8              11W / 280W |  11963MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA TITAN RTX               On  | 00000000:0A:00.0 Off |                  N/A |
| 41%   23C    P8               3W / 280W |  23591MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA TITAN RTX               On  | 00000000:84:00.0 Off |                  N/A |
| 41%   25C    P8              10W / 280W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA TITAN RTX               On  | 00000000:88:00.0 Off |                  N/A |
| 41%   25C    P8              14W / 280W |      1MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A   2614703      C   text-embeddings-router                      872MiB |
|    0   N/A  N/A   2615343      C   /opt/conda/bin/python                      5180MiB |
|    0   N/A  N/A   3537139      C   ...jin/anaconda3/envs/envt1/bin/python      190MiB |
|    0   N/A  N/A   4125721      C   ...jin/anaconda3/envs/envt1/bin/python     5718MiB |
|    1   N/A  N/A   2615081      C   python                                    23580MiB |
+---------------------------------------------------------------------------------------+

```

specifically, there is a process called 'python' that isn't killed and takes up the entirety of the VRAM with the process owner as root which requires sudo to kill. (GPU 0 is being used by someone else.)

is there a part of the config that i'm running wrongly?

this has happened twice now.


Hi @johnwee1 !

To try to determine whether this is a VLLM issue or one on our end, could you try to pull the vllm calls + model initialization out into a minimal script and run that, then see if you observe the same memory leak?

i can't seem to repro this issue of the mem leak (it could be conflicting stuff on my end) but i am still getting OOM issue in 1923 so i think i'll close this one here