Repository: EleutherAI/lm-evaluation-harness
Issue: Low GPU Utilization During Multi-GPU evaluation - Efficiency Optimization
Root cause: Documentation Deficiency

Hello,

I want to express my gratitude for your outstanding work. The powerful lm-evaluation-harness and your continuous maintenance have made LLM-evaluation much more convenient. 

However, I have recently encountered an issue regarding multi-GPU testing for my hundreds ckpt. I am noticing low GPU utilization, and I am uncertain whether the setup is fully leveraging the hardware potential. I am also curious if there are any parts, such as repeated downloading, that could be optimized?

I set my environment as follows based on error messages:
```
cd /path/lm-evaluation-harness/
export HF_HUB_OFFLINE="1"
export HF_DATASETS_OFFLINE="0"
export HF_DATASETS_TRUST_REMOTE_CODE=1
export HF_ENDPOINT=https://hf-mirror.com
export TRUST_REMOTE_CODE=True
```

Here is the command I am using to launch the process:

```
python3 -m lm_eval --model vllm     --model_args pretrained=[Model 1], tensor_parallel_size=1, dtype=auto, gpu_memory_utilization=0.9, data_parallel_size=6, trust_remote_code=True     --tasks mmlu,ceval-valid,arc_easy,arc_challenge,hellaswag,openbookqa,piqa,winogrande     --batch_size auto     --output_path "/path/mmlu-ceval-valid-arc_easy-arc_challenge-hellaswag-openbookqa-piqa-winogrande"     --save_results "/path.log"     --cache_requests true
python3 -m lm_eval --model vllm     --model_args pretrained=[Model 2], tensor_parallel_size=1, dtype=auto, gpu_memory_utilization=0.9, data_parallel_size=6, trust_remote_code=True     --tasks mmlu,ceval-valid,arc_easy,arc_challenge,hellaswag,openbookqa,piqa,winogrande     --batch_size auto     --output_path "/path/mmlu-ceval-valid-arc_easy-arc_challenge-hellaswag-openbookqa-piqa-winogrande"     --save_results "/path.log"     --cache_requests true
python3 -m lm_eval --model vllm     --model_args pretrained=[Model 3], tensor_parallel_size=1, dtype=auto, gpu_memory_utilization=0.9, data_parallel_size=6, trust_remote_code=True     --tasks mmlu,ceval-valid,arc_easy,arc_challenge,hellaswag,openbookqa,piqa,winogrande     --batch_size auto     --output_path "/path/mmlu-ceval-valid-arc_easy-arc_challenge-hellaswag-openbookqa-piqa-winogrande"     --save_results "/path.log"     --cache_requests true
```

I use the [vllm] and `data_parallel_size=6, trust_remote_code=True and  --cache_requests true` for efficiency.

Command outputs as follows:

```
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 52/52 [00:00<00:00, 208512.24it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 52/52 [00:00<00:00, 140802.97it/s]
(skip 50x repeated lines)

/usr/local/python/lib/python3.8/site-packages/datasets/load.py:930: FutureWarning: The repository for hellaswag contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at /path/data/hellaswag/hellaswag.py
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
/usr/local/python/lib/python3.8/site-packages/datasets/load.py:930: FutureWarning: The repository for mmlu_no_train contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at /path/data/mmlu_no_train/mmlu_no_train.py
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
/usr/local/python/lib/python3.8/site-packages/datasets/load.py:930: FutureWarning: The repository for piqa contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at /path/data/piqa/piqa.py
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
/usr/local/python/lib/python3.8/site-packages/datasets/load.py:930: FutureWarning: The repository for winogrande contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at /path/data/winogrande/winogrande.py
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
    Running loglikelihood requests:   0%|                                                                                                                               | 0/124118 [00:00<?, ?it/s]

id=105521) INFO 09-14 14:13:41 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='/path/results/dir', speculative_config=None, tokenizer='/path/results/dir', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=1234, served_model_name=/path/results/dir, use_v2_block_manager=False, enable_prefix_caching=False)
(run_inference_one_model pid=105521) /usr/local/python/lib/python3.8/site-packages/vllm/executor/ray_gpu_executor.py:115: UserWarning: Failed to get the IP address, using 0.0.0.0 by default.The value can be set by the environment variable VLLM_HOST_IP or HOST_IP.
(run_inference_one_model pid=105521)   driver_ip = get_ip()
(RayWorkerWrapper pid=112853)   return get_ip()
(run_inference_one_model pid=105551) INFO 09-14 14:13:47 model_runner.py:680] Starting to load model /path/results/dir...
(run_inference_one_model pid=105434) INFO 09-14 14:13:42 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='/path/results/dir', speculative_config=None, tokenizer='/path/results/dir', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=1234, served_model_name=/path/results/dir, use_v2_block_manager=False, enable_prefix_caching=False) [repeated 5x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(RayWorkerWrapper pid=112934) /usr/local/python/lib/python3.8/site-packages/vllm/executor/ray_utils.py:27: UserWarning: Failed to get the IP address, using 0.0.0.0 by default.The value can be set by the environment variable VLLM_HOST_IP or HOST_IP. [repeated 11x across cluster]
(run_inference_one_model pid=105434)   driver_ip = get_ip() [repeated 5x across cluster]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:14<00:00, 14.27s/it]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:14<00:00, 14.27s/it]
(run_inference_one_model pid=105537) 
(RayWorkerWrapper pid=112934)   return get_ip() [repeated 5x across cluster]
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s] [repeated 5x across cluster]
(run_inference_one_model pid=105521) INFO 09-14 14:14:03 model_runner.py:692] Loading model weights took 2.0512 GB
(run_inference_one_model pid=105525) INFO 09-14 14:13:48 model_runner.py:680] Starting to load model /path/results/dir... [repeated 5x across cluster]
(run_inference_one_model pid=105537) INFO 09-14 14:14:03 distributed_gpu_executor.py:56] # GPU blocks: 98616, # CPU blocks: 11915
(run_inference_one_model pid=105566) INFO 09-14 14:14:35 model_runner.py:1181] Graph capturing finished in 12 secs.
(run_inference_one_model pid=105525) INFO 09-14 14:14:23 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. [repeated 5x across cluster]
(run_inference_one_model pid=105525) INFO 09-14 14:14:23 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. [repeated 5x across cluster]
Processed prompts:   0%|          | 0/20686 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:14<00:00, 14.39s/it] [repeated 10x across cluster]
(run_inference_one_model pid=105525)  [repeated 5x across cluster]
Processed prompts:   0%|          | 1/20686 [00:00<44:21,  7.77it/s, est. speed input: 9375.66 toks/s, output: 7.77 toks/s]
Processed prompts:   0%|          | 2/20686 [00:00<43:22,  7.95it/s, est. speed input: 9475.98 toks/s, output: 7.92 toks/s]
(run_inference_one_model pid=105434) INFO 09-14 14:14:40 model_runner.py:1181] Graph capturing finished in 17 secs. [repeated 4x across cluster]
(Progress bar for 50x lines)
Processed prompts:  78%|███████▊  | 16171/20687 [02:36<00:13, 343.93it/s, est. speed input: 13132.71 toks/s, output: 103.04 toks/s] [repeated 213x across cluster]
Processed prompts: 100%|██████████| 20687/20687 [02:44<00:00, 125.69it/s, est. speed input: 13386.94 toks/s, output: 125.69 toks/s]
Running loglikelihood requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 124118/124118 [06:29<00:00, 318.77it/s]
vllm (pretrained=/path/model,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=6,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto
|                         Tasks                         |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------------------------------------------------|-------|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge                                          |      1|none  |     0|acc     |↑  |0.2944|±  |0.0133|
|                                                       |       |none  |     0|acc_norm|↑  |0.3217|±  |0.0137|
|arc_easy                                               |      1|none  |     0|acc     |↑  |0.6124|±  |0.0100|
|                                                       |       |none  |     0|acc_norm|↑  |0.5577|±  |0.0102|
|ceval-valid                                            |N/A    |none  |     0|acc     |↑  |0.2526|±  |0.0118|
```
It takes 15 mins and then repeat for next. I know it's pretty fast for evaluation but **only 20% peak usage GPU-utilization** with full memory usage. I am wondering if I am misconfiguring something, or if there are any additional steps I could take to further accelerate the process.

Thank you so much for your time and assistance!

Best regards,

Rummy

Hi Rummy! The command looks right. I think it's because all the tasks you are using are loglikelihood ones — where it's just one forward pass per sequence and calculating the logprobs of the prefill. You could try setting `gpu_memory_utilization` really low (it's used to pre-allocate the kv-cache as well as model weights, so depends on the model). Conversely, for tasks with the same fewshot context like mmlu, you could also try `enable_prefix_caching`. I think testing HF as a baseline would also be helpful.

As most of the optimizations in vllm are kv-cache related, generation tasks such as `gsm8k` or `hendrycks_math` will provide better utilization (and is much faster than HF).