Repository: EleutherAI/lm-evaluation-harness
Issue: Uneven GPU utilization with Data Parallelism (DP) in vLLM due to context-agnostic request distribution.
Root cause: Algorithmic Error

Description:

While utilizing Data Parallelism (DP) on top of vLLM for processing multiple requests across several GPUs, we observed that the current request distribution logic is overly simplistic. It does not take into account the varying lengths of request contexts. This results in an uneven distribution of requests across GPUs, leading to some GPUs finishing their assigned tasks significantly earlier and subsequently becoming idle.

https://github.com/EleutherAI/lm-evaluation-harness/blob/f78e2da45f034a23b1b13cde3235105b0f55d830/lm_eval/models/vllm_causallms.py#L179

Thanks for reporting this! Looks like our typical approach of sorting requests by length high-to-low is backfiring here since the 8th worker will (I think) receive all the shortest eighth of the requests.

I've made #1477 to address this--could you let me know if it fixes your issue?


Tested on the `arc_challenge` task with 4687 responses on 2 GPUs.

main: after 00:14 second, one GPU finishes the job and becomes idle.
```
Processed prompts:   0%|                                                                          | 0/2344 [00:00<?, ?it/s]
Processed prompts:   0%|                                                                          | 0/2343 [00:00<?, ?it/s]
Processed prompts:   0%|                                                                | 1/2343 [00:01<1:13:47,  1.89s/it]
Processed prompts:   0%|                                                                | 1/2344 [00:03<2:01:58,  3.12s/it]
Processed prompts:  11%|███████                                                         | 257/2343 [00:03<00:26, 77.72it/s]
Processed prompts:   9%|█████▍                                                          | 201/2344 [00:06<00:57, 37.00it/s]
Processed prompts:  22%|█████████████▊                                                 | 513/2343 [00:05<00:16, 107.95it/s]
Processed prompts:  33%|████████████████████▋                                          | 769/2343 [00:07<00:12, 124.65it/s]
Processed prompts:  19%|████████████▍                                                   | 457/2344 [00:09<00:31, 59.88it/s]
Processed prompts:  44%|███████████████████████████                                   | 1025/2343 [00:09<00:09, 132.58it/s]
Processed prompts:  55%|█████████████████████████████████▉                            | 1281/2343 [00:10<00:07, 141.45it/s]
Processed prompts:  30%|███████████████████▍                                            | 713/2344 [00:11<00:22, 72.93it/s]
Processed prompts:  66%|████████████████████████████████████████▋                     | 1537/2343 [00:12<00:05, 152.32it/s]
Processed prompts:  41%|██████████████████████████▍                                     | 969/2344 [00:14<00:16, 83.36it/s]
Processed prompts:  77%|███████████████████████████████████████████████▍              | 1793/2343 [00:13<00:03, 165.60it/s]
Processed prompts:  87%|██████████████████████████████████████████████████████▏       | 2049/2343 [00:14<00:01, 179.79it/s]
Processed prompts: 100%|██████████████████████████████████████████████████████████████| 2343/2343 [00:14<00:00, 160.21it/s]
Processed prompts:  52%|████████████████████████████████▉                              | 1225/2344 [00:16<00:12, 91.49it/s]
Processed prompts:  63%|███████████████████████████████████████▊                       | 1481/2344 [00:18<00:08, 98.98it/s]
Processed prompts:  74%|█████████████████████████████████████████████▉                | 1737/2344 [00:20<00:05, 105.69it/s]
Processed prompts:  85%|████████████████████████████████████████████████████▋         | 1993/2344 [00:22<00:03, 110.57it/s]
Processed prompts: 100%|███████████████████████████████████████████████████████████████| 2344/2344 [00:23<00:00, 98.70it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 4687/4687 [01:01<00:00, 76.39it/s]
```

your branch has a more balanced job distribution! Nice work. Thanks

```
Processed prompts:   0%|                                                                          | 0/2343 [00:00<?, ?it/s]
Processed prompts:   0%|                                                                          | 0/2344 [00:00<?, ?it/s]
Processed prompts:   0%|                                                                | 1/2343 [00:03<2:02:48,  3.15s/it]
Processed prompts:   0%|                                                                | 1/2344 [00:03<2:03:11,  3.15s/it]
Processed prompts:  10%|██████▎                                                         | 231/2343 [00:05<00:45, 46.77it/s]
Processed prompts:  10%|██████▎                                                         | 231/2344 [00:06<00:46, 45.47it/s]
Processed prompts:  21%|█████████████▎                                                  | 487/2343 [00:08<00:25, 72.23it/s]
Processed prompts:  21%|█████████████▎                                                  | 487/2344 [00:08<00:26, 70.93it/s]
Processed prompts:  32%|████████████████████▎                                           | 743/2343 [00:10<00:18, 87.82it/s]
Processed prompts:  32%|████████████████████▎                                           | 743/2344 [00:10<00:18, 85.62it/s]
Processed prompts:  43%|██████████████████████████▊                                    | 999/2343 [00:12<00:13, 100.14it/s]
Processed prompts:  43%|███████████████████████████▎                                    | 999/2344 [00:12<00:13, 99.00it/s]
Processed prompts:  54%|█████████████████████████████████▏                            | 1255/2343 [00:14<00:09, 112.34it/s]
Processed prompts:  54%|█████████████████████████████████▏                            | 1255/2344 [00:14<00:09, 111.42it/s]
Processed prompts:  64%|███████████████████████████████████████▉                      | 1511/2343 [00:15<00:06, 124.17it/s]
Processed prompts:  64%|███████████████████████████████████████▉                      | 1511/2344 [00:16<00:06, 123.73it/s]
Processed prompts:  75%|██████████████████████████████████████████████▊               | 1767/2343 [00:17<00:04, 134.08it/s]
Processed prompts:  75%|██████████████████████████████████████████████▋               | 1767/2344 [00:17<00:04, 132.40it/s]
Processed prompts:  86%|█████████████████████████████████████████████████████▌        | 2023/2343 [00:18<00:02, 150.28it/s]
Processed prompts: 100%|██████████████████████████████████████████████████████████████| 2343/2343 [00:18<00:00, 123.40it/s]
Processed prompts:  86%|█████████████████████████████████████████████████████▌        | 2023/2344 [00:18<00:02, 149.17it/s]
Processed prompts: 100%|██████████████████████████████████████████████████████████████| 2344/2344 [00:19<00:00, 121.93it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 4687/4687 [01:07<00:00, 69.54it/s]
```


sweet! glad it helped!