Repository: EleutherAI/lm-evaluation-harness
Issue: Using fine-tuning with LoRA and evaluation 
Root cause: Documentation Deficiency

I have fine-tuned the model with LoRA implementation, and after model saving, I and only the adapter_config.json file. 

Is this correct way to evaluate the fine-tuned model? 
I got some result but I'm not sure what should I put to [pretrained model before fine-tuning]


 python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=[pretrained model before fine-tuning],peft=[fine-tuned model with LoRA] \
    --tasks hellaswag,anli_r1,anli_r2,anli_r3\
    --device cpu

The PEFT checkpoint only saves the adapter weights, so in order to use it you need to combine it with the base model. `[pretrained model before fine-tuning]` should be the name of the model that you are finetuning with LoRA. It will be loaded from the Hub, and is probably a pretrained model like GPT-J or LLaMA.

Closing because the method you've described is correct to invoke a model + its LoRA, if I'm understanding what you're asking correctly.

How can I do this with vllm? 

I have something like this, but I am not able to get it working. Where do I add the path to the saved lora modules? 

```
    lm-eval --tasks gsm8k --model vllm \
        --model_args pretrained={base_model_path},tensor_parallel_size=4,enable_lora=True \
        --log_samples --num_fewshot 0 --batch_size auto:N \
        --gen_kwargs temperature=0.4,top_p=0.95,repetition_penalty=1.1
```



Hi @mohit-rag , passing `local_lora_path=/path/to/lora` (https://github.com/EleutherAI/lm-evaluation-harness/blob/feff1b55c57993c4d42c8f913a22eeec395cd690/lm_eval/models/vllm_causallms.py#L62C9-L62C37) should do this.

Hi @haileyschoelkopf 
While running above command, I am getting error
lm-eval --tasks gsm8k --model vllm \
    --model_args pretrained="meta-llama/Meta-Llama-3.1-8B-Instruct", enable_lora=True, lora_local_path="/datadrive/llama_finetune/data/llama_3_1_8b_v3" \
    --device cpu \
    --limit 5
    
  Error
  
![image](https://github.com/user-attachments/assets/0d657285-4c33-4b70-aa8e-1eca19c9199d)

lm-eval: error: unrecognized arguments: enable_lora=True, lora_local_path=/datadrive/llama_finetune/data/llama_3_1_8b_v3

Can you help me here?
Thank you

Hello, were you able to resolve the above issue?

yes, I am able to solve the issue.

> yes, I am able to solve the issue.

how?

@chintanshrinath 

> I have fine-tuned the model with LoRA implementation, and after model saving, I and only the adapter_config.json file.
> 
> Is this correct way to evaluate the fine-tuned model? I got some result but I'm not sure what should I put to [pretrained model before fine-tuning]
> 
> python main.py --model hf-causal-experimental --model_args pretrained=[pretrained model before fine-tuning],peft=[fine-tuned model with LoRA] --tasks hellaswag,anli_r1,anli_r2,anli_r3 --device cpu

with HF this worked for me:

```
command='lm_eval --model hf         --model_args pretrained=Qwen/Qwen2-1.5B,peft=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594         --tasks mmlu_college_mathematics         --trust_remote_code         --batch_size auto:4         --device cuda         --output_path /data/miranebr-sandbox/lm_harness/eval_results/qwen2_1_5b_ld_pf_dsm/mmlu_college_mathematics         --log_samples '
```
out
```
2024-10-07:17:21:50,094 WARNING  [huggingface.py:1344] Failed to get model SHA for /data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594 at revision main. Error: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594'. Use `repo_type` argument if needed.
2024-10-07:17:21:59,270 INFO     [evaluation_tracker.py:206] Saving results aggregated
2024-10-07:17:21:59,285 INFO     [evaluation_tracker.py:287] Saving per-sample results for: mmlu_college_mathematics
hf (pretrained=Qwen/Qwen2-1.5B,peft=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:4 (64,64,64,64)
|       Tasks       |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|-------------------|------:|------|-----:|------|---|----:|---|-----:|
|college_mathematics|      1|none  |     0|acc   |↑  | 0.33|±  |0.0473|

Time taken: 87.57 seconds, or 1.46 minutes, or 0.02 hours.

real    1m27.800s
user    0m42.524s
sys     0m14.372s
(AI4Lean_lm_harness
```

> Hi @haileyschoelkopf While running above command, I am getting error lm-eval --tasks gsm8k --model vllm --model_args pretrained="meta-llama/Meta-Llama-3.1-8B-Instruct", enable_lora=True, lora_local_path="/datadrive/llama_finetune/data/llama_3_1_8b_v3" --device cpu --limit 5
> 
> Error
> 
> ![image](https://private-user-images.githubusercontent.com/7765864/358978416-0d657285-4c33-4b70-aa8e-1eca19c9199d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjgzMjIyOTIsIm5iZiI6MTcyODMyMTk5MiwicGF0aCI6Ii83NzY1ODY0LzM1ODk3ODQxNi0wZDY1NzI4NS00YzMzLTRiNzAtYWE4ZS0xZWNhMTljOTE5OWQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MTAwNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDEwMDdUMTcyNjMyWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZTIxMTMxZWQzMzkyYTc0NWI1NDc5ODQxY2Q2OGNmMDBjOWFjNzc4ZjNhNjhjYzZlOWUyOGI5Y2VjODQ3ZDk2YSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.5nbg6M9xrGJXlr6KL0P-Fzd445qBbcXr1seR0shEGpE)
> 
> lm-eval: error: unrecognized arguments: enable_lora=True, lora_local_path=/datadrive/llama_finetune/data/llama_3_1_8b_v3

I got this error:

```
pre_model_args='pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594' type(pre_model_args)=<class 'str'>
Path to save results for model and task/benchmark: model_task_output_path='/data/miranebr-sandbox/lm_harness/eval_results/qwen2_1_5b_ld_pf_dsm/mmlu_college_mathematics'
lm-harness model arguments: model_args='pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594'
--- Run the lm_eval command
command='lm_eval --model vllm         --model_args pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594         --tasks mmlu_college_mathematics         --trust_remote_code         --batch_size auto:4         --device cuda         --output_path /data/miranebr-sandbox/lm_harness/eval_results/qwen2_1_5b_ld_pf_dsm/mmlu_college_mathematics         --log_samples '
2024-10-07:17:31:40,978 INFO     [__main__.py:279] Verbosity set to INFO
2024-10-07:17:31:42,537 INFO     [__init__.py:491] `group` and `group_alias` keys in TaskConfigs are deprecated and will be removed in v0.4.5 of lm_eval. The new `tag` field will be used to allow for a shortcut to a group of tasks one does not wish to aggregate metrics across. `group`s which aggregate across subtasks must be only defined in a separate group config file, which will be the official way to create groups that support cross-task aggregation as in `mmlu`. Please see the v0.4.4 patch notes and our documentation: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#advanced-group-configs for more information.
2024-10-07:17:31:57,523 INFO     [__main__.py:364] Passed `--trust_remote_code`, setting environment variable `HF_DATASETS_TRUST_REMOTE_CODE=true`
2024-10-07:17:31:57,523 INFO     [__main__.py:376] Selected Tasks: ['mmlu_college_mathematics']
2024-10-07:17:31:57,541 INFO     [evaluator.py:161] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2024-10-07:17:31:57,541 INFO     [evaluator.py:198] Initializing vllm model, with arguments: {'pretrained': 'Qwen/Qwen2-1.5B', 'enable_lora': True, 'lora_local_path': '/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594', 'trust_remote_code': True}
WARNING 10-07 17:31:57 arg_utils.py:940] The model has a long context length (131072). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
INFO 10-07 17:31:57 llm_engine.py:226] Initializing an LLM engine (v0.6.1.dev238+ge2c6e0a82) with config: model='Qwen/Qwen2-1.5B', speculative_config=None, tokenizer='Qwen/Qwen2-1.5B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=1234, served_model_name=Qwen/Qwen2-1.5B, use_v2_block_manager=False, num_scheduler_steps=1, multi_step_stream_outputs=False, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
INFO 10-07 17:32:00 model_runner.py:1014] Starting to load model Qwen/Qwen2-1.5B...
INFO 10-07 17:32:00 weight_utils.py:242] Using model weights format ['*.safetensors']
INFO 10-07 17:32:00 weight_utils.py:287] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.86s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.86s/it]

INFO 10-07 17:32:02 model_runner.py:1025] Loading model weights took 2.9104 GB
INFO 10-07 17:32:08 gpu_executor.py:122] # GPU blocks: 139212, # CPU blocks: 9362
INFO 10-07 17:32:11 model_runner.py:1329] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 10-07 17:32:11 model_runner.py:1333] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 10-07 17:32:35 model_runner.py:1456] Graph capturing finished in 25 secs.
2024-10-07:17:32:37,948 INFO     [evaluator.py:279] Setting fewshot random generator seed to 1234
2024-10-07:17:32:37,949 WARNING  [model.py:422] model.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.
2024-10-07:17:32:37,949 INFO     [task.py:423] Building contexts for mmlu_college_mathematics on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 868.91it/s]
2024-10-07:17:32:38,068 INFO     [evaluator.py:465] Running loglikelihood requests
Running loglikelihood requests:   0%|                                                                                                                             | 0/400 [00:00<?, ?it/sINFO 10-07 17:32:39 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241007-173239.pkl..., est. speed input: 0.00 toks/s, output: 0.00 toks/s]
WARNING 10-07 17:32:39 model_runner_base.py:143] Failed to pickle inputs of failed execution: Can't pickle local object 'weak_bind.<locals>.weak_bound'
[rank0]: Traceback (most recent call last):
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1633, in execute_model
[rank0]:     output: SamplerOutput = self.model.sample(
[rank0]:                             ^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 401, in sample
[rank0]:     next_tokens = self.sampler(logits, sampling_metadata)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/model_executor/layers/sampler.py", line 257, in forward
[rank0]:     logits = logits.to(torch.float)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.27 GiB. GPU 0 has a total capacity of 79.15 GiB of which 5.73 GiB is free. Including non-PyTorch memory, this process has 0 bytes memory in use. Of the allocated memory 70.72 GiB is allocated by PyTorch, with 23.60 MiB allocated in private pools (e.g., CUDA Graphs), and 158.20 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[rank0]: The above exception was the direct cause of the following exception:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/bin/lm_eval", line 8, in <module>
[rank0]:     sys.exit(cli_evaluate())
[rank0]:              ^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/__main__.py", line 382, in cli_evaluate
[rank0]:     results = evaluator.simple_evaluate(
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/utils.py", line 397, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/evaluator.py", line 301, in simple_evaluate
[rank0]:     results = evaluate(
[rank0]:               ^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/utils.py", line 397, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/evaluator.py", line 476, in evaluate
[rank0]:     resps = getattr(lm, reqtype)(cloned_reqs)
[rank0]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/api/model.py", line 378, in loglikelihood
[rank0]:     return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/models/vllm_causallms.py", line 442, in _loglikelihood_tokens
[rank0]:     outputs = self._model_generate(requests=inputs, generate=False)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/models/vllm_causallms.py", line 261, in _model_generate
[rank0]:     outputs = self.model.generate(
[rank0]:               ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/utils.py", line 1047, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 388, in generate
[rank0]:     outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 877, in _run_engine
[rank0]:     step_outputs = self.llm_engine.step()
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 1264, in step
[rank0]:     outputs = self.model_executor.execute_model(
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/executor/gpu_executor.py", line 130, in execute_model
[rank0]:     output = self.driver_worker.execute_model(execute_model_req)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/worker/worker_base.py", line 327, in execute_model
[rank0]:     output = self.model_runner.execute_model(
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 146, in _wrapper
[rank0]:     raise type(err)(f"Error in model execution: "
[rank0]: torch.OutOfMemoryError: Error in model execution: CUDA out of memory. Tried to allocate 16.27 GiB. GPU 0 has a total capacity of 79.15 GiB of which 5.73 GiB is free. Including non-PyTorch memory, this process has 0 bytes memory in use. Of the allocated memory 70.72 GiB is allocated by PyTorch, with 23.60 MiB allocated in private pools (e.g., CUDA Graphs), and 158.20 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Processed prompts:   0%|                                                                                      | 0/400 [00:01<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Running loglikelihood requests:   0%|                                                                                                                             | 0/400 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/data/miranebr-sandbox/AI4Lean/py_src/ailean_evals/run_lm_harness_eval.py", line 54, in <module>
    fire.Fire(main)
  File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/data/miranebr-sandbox/AI4Lean/py_src/ailean_evals/run_lm_harness_eval.py", line 48, in main
    subprocess.run(command, shell=True, check=True)
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'lm_eval --model vllm         --model_args pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594         --tasks mmlu_college_mathematics         --trust_remote_code         --batch_size auto:4         --device cuda         --output_path /data/miranebr-sandbox/lm_harness/eval_results/qwen2_1_5b_ld_pf_dsm/mmlu_college_mathematics         --log_samples ' returned non-zero exit status 1.

real    1m19.466s
user    1m9.009s
sys     0m14.190s
(AI4Lean_lm_harness_vllm) root@miranebr-math-p4de-math-aif-sft:~# 
(AI4Lean_lm_harness_vllm) root@miranebr-math-p4de-math-aif-sft:~# nvidia-smi
Mon Oct  7 17:32:58 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  | 00000000:10:1C.0 Off |                    0 |
| N/A   27C    P0              62W / 400W |      0MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-SXM4-80GB          On  | 00000000:10:1D.0 Off |                    0 |
| N/A   25C    P0              60W / 400W |      0MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA A100-SXM4-80GB          On  | 00000000:20:1C.0 Off |                    0 |
| N/A   27C    P0              62W / 400W |      0MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA A100-SXM4-80GB          On  | 00000000:20:1D.0 Off |                    0 |
| N/A   25C    P0              60W / 400W |      0MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   4  NVIDIA A100-SXM4-80GB          On  | 00000000:90:1C.0 Off |                    0 |
| N/A   27C    P0              60W / 400W |      0MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   5  NVIDIA A100-SXM4-80GB          On  | 00000000:90:1D.0 Off |                    0 |
| N/A   26C    P0              63W / 400W |      0MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   6  NVIDIA A100-SXM4-80GB          On  | 00000000:A0:1C.0 Off |                    0 |
| N/A   28C    P0              64W / 400W |      0MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   7  NVIDIA A100-SXM4-80GB          On  | 00000000:A0:1D.0 Off |                    0 |
| N/A   26C    P0              62W / 400W |      0MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```

Command
```
lm_eval --model vllm         --model_args pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594         --tasks mmlu_college_mathematics         --trust_remote_code         --batch_size auto:4         --device cuda         --output_path /data/miranebr-sandbox/lm_harness/eval_results/qwen2_1_5b_ld_pf_dsm/mmlu_college_mathematics         --log_samples 
```

@haileyschoelkopf have you ran a local lora ckpt with vllm + lm-harness before? I'm surprised that I get out of memory issue.

Code:
```python

import os
import subprocess

from pdb import set_trace as st

def main(
    model = 'hf',
    pre_model_args = 'Qwen/Qwen2-1.5B',
    model_nickname = 'qwen2_1_5b',
    cuda_visible_devices = '0',
    benchmark_and_optional_task = 'mmlu_college_mathematics',
    root_eval_results = '~/lm_harness/eval_results',
    # root_use_cache_path = '~/lm_harness/cache' Takes a string of format /path/to/sqlite_cache_ in order to create a cache db at /path/to/sqlite_cache_rank{i}.db
    # --limit 500
):
    print(f'{pre_model_args=} {type(pre_model_args)=}')
    # - Path to save results for model and task/benchmark
    model_task_output_path = os.path.expanduser(f'{root_eval_results}/{model_nickname}/{benchmark_and_optional_task}')
    os.makedirs(model_task_output_path, exist_ok=True)
    print(f'Path to save results for model and task/benchmark: {model_task_output_path=}')

    # cache_path = os.path.expanduser(f'{root_use_cache_path}/{model_nickname}/{benchmark_and_optional_task}')
    # os.makedirs(cache_path, exist_ok=True)
    # print(f'{cache_path=}')

    # model_args = f'pretrained={pre_model_args}'
    model_args = pre_model_args
    print(f'lm-harness model arguments: {model_args=}')

    # Set the environment variables
    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
    os.environ["CUDA_VISIBLE_DEVICES"] = str(cuda_visible_devices)

    # Run the lm_eval command
    print('--- Run the lm_eval command')
    # --model_args pretrained={model_name},tensor_parallel_size={GPUs_per_model},dtype=auto,gpu_memory_utilization=0.8,data_parallel_size={model_replicas} \
    command = f'lm_eval --model {model} \
        --model_args {model_args} \
        --tasks {benchmark_and_optional_task} \
        --trust_remote_code \
        --batch_size auto:4 \
        --device cuda \
        --output_path {model_task_output_path} \
        --log_samples '
        # --use_cache {cache_path}'
    print(f'{command=}')
    subprocess.run(command, shell=True, check=True)

if __name__ == '__main__':
    import fire
    import time
    start_time = time.time()
    fire.Fire(main)
    print(f"Time taken: {time.time() - start_time:.2f} seconds, or {(time.time() - start_time) / 60:.2f} minutes, or {(time.time() - start_time) / 3600:.2f} hours.\a")
```
    
    args
    
```python
# - vllm lora
source ~/.virtualenvs/AI4Lean_lm_harness_vllm/bin/activate
export model=vllm
# export model=hf
# export pre_model_args=Qwen/Qwen2-1.5B
export pre_model_args=Qwen/Qwen2-1.5B
export lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594
export pre_model_args='pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594'
echo ${pre_model_args}
export model_nickname=qwen2_1_5b_ld_pf_dsm
export CUDA_VISIBLE_DEVICES=0
export benchmark_and_optional_task=mmlu_college_mathematics
# export benchmark_and_optional_task=minerva_math
# time python ~/AI4Lean/py_src/ailean_evals/run_lm_harness_eval.py --model ${model} --pre_model_args ${pre_model_args} --model_nickname ${model_nickname} --cuda_visible_devices ${CUDA_VISIBLE_DEVICES} --benchmark_and_optional_task ${benchmark_and_optional_task}
time python ~/AI4Lean/py_src/ailean_evals/run_lm_harness_eval.py --model ${model} --pre_model_args ${pre_model_args} --model_nickname ${model_nickname} --cuda_visible_devices ${CUDA_VISIBLE_DEVICES} --benchmark_and_optional_task ${benchmark_and_optional_task}
```

> Hi @mohit-rag , passing `local_lora_path=/path/to/lora` (https://github.com/EleutherAI/lm-evaluation-harness/blob/feff1b55c57993c4d42c8f913a22eeec395cd690/lm_eval/models/vllm_causallms.py#L62C9-L62C37) should do this.

Thanks, it works!

but maybe **lora_local_path** is the right name :P

>         lora_local_path: str = None,

> > Hi @haileyschoelkopf While running above command, I am getting error lm-eval --tasks gsm8k --model vllm --model_args pretrained="meta-llama/Meta-Llama-3.1-8B-Instruct", enable_lora=True, lora_local_path="/datadrive/llama_finetune/data/llama_3_1_8b_v3" --device cpu --limit 5
> > Error
> > ![image](https://private-user-images.githubusercontent.com/7765864/358978416-0d657285-4c33-4b70-aa8e-1eca19c9199d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjgzMjIyOTIsIm5iZiI6MTcyODMyMTk5MiwicGF0aCI6Ii83NzY1ODY0LzM1ODk3ODQxNi0wZDY1NzI4NS00YzMzLTRiNzAtYWE4ZS0xZWNhMTljOTE5OWQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MTAwNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDEwMDdUMTcyNjMyWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZTIxMTMxZWQzMzkyYTc0NWI1NDc5ODQxY2Q2OGNmMDBjOWFjNzc4ZjNhNjhjYzZlOWUyOGI5Y2VjODQ3ZDk2YSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.5nbg6M9xrGJXlr6KL0P-Fzd445qBbcXr1seR0shEGpE)
> > lm-eval: error: unrecognized arguments: enable_lora=True, lora_local_path=/datadrive/llama_finetune/data/llama_3_1_8b_v3
> 
> I got this error:
> 
> ```
> pre_model_args='pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594' type(pre_model_args)=<class 'str'>
> Path to save results for model and task/benchmark: model_task_output_path='/data/miranebr-sandbox/lm_harness/eval_results/qwen2_1_5b_ld_pf_dsm/mmlu_college_mathematics'
> lm-harness model arguments: model_args='pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594'
> --- Run the lm_eval command
> command='lm_eval --model vllm         --model_args pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594         --tasks mmlu_college_mathematics         --trust_remote_code         --batch_size auto:4         --device cuda         --output_path /data/miranebr-sandbox/lm_harness/eval_results/qwen2_1_5b_ld_pf_dsm/mmlu_college_mathematics         --log_samples '
> 2024-10-07:17:31:40,978 INFO     [__main__.py:279] Verbosity set to INFO
> 2024-10-07:17:31:42,537 INFO     [__init__.py:491] `group` and `group_alias` keys in TaskConfigs are deprecated and will be removed in v0.4.5 of lm_eval. The new `tag` field will be used to allow for a shortcut to a group of tasks one does not wish to aggregate metrics across. `group`s which aggregate across subtasks must be only defined in a separate group config file, which will be the official way to create groups that support cross-task aggregation as in `mmlu`. Please see the v0.4.4 patch notes and our documentation: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#advanced-group-configs for more information.
> 2024-10-07:17:31:57,523 INFO     [__main__.py:364] Passed `--trust_remote_code`, setting environment variable `HF_DATASETS_TRUST_REMOTE_CODE=true`
> 2024-10-07:17:31:57,523 INFO     [__main__.py:376] Selected Tasks: ['mmlu_college_mathematics']
> 2024-10-07:17:31:57,541 INFO     [evaluator.py:161] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
> 2024-10-07:17:31:57,541 INFO     [evaluator.py:198] Initializing vllm model, with arguments: {'pretrained': 'Qwen/Qwen2-1.5B', 'enable_lora': True, 'lora_local_path': '/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594', 'trust_remote_code': True}
> WARNING 10-07 17:31:57 arg_utils.py:940] The model has a long context length (131072). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
> INFO 10-07 17:31:57 llm_engine.py:226] Initializing an LLM engine (v0.6.1.dev238+ge2c6e0a82) with config: model='Qwen/Qwen2-1.5B', speculative_config=None, tokenizer='Qwen/Qwen2-1.5B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=1234, served_model_name=Qwen/Qwen2-1.5B, use_v2_block_manager=False, num_scheduler_steps=1, multi_step_stream_outputs=False, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
> INFO 10-07 17:32:00 model_runner.py:1014] Starting to load model Qwen/Qwen2-1.5B...
> INFO 10-07 17:32:00 weight_utils.py:242] Using model weights format ['*.safetensors']
> INFO 10-07 17:32:00 weight_utils.py:287] No model.safetensors.index.json found in remote.
> Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
> Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.86s/it]
> Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.86s/it]
> 
> INFO 10-07 17:32:02 model_runner.py:1025] Loading model weights took 2.9104 GB
> INFO 10-07 17:32:08 gpu_executor.py:122] # GPU blocks: 139212, # CPU blocks: 9362
> INFO 10-07 17:32:11 model_runner.py:1329] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
> INFO 10-07 17:32:11 model_runner.py:1333] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
> INFO 10-07 17:32:35 model_runner.py:1456] Graph capturing finished in 25 secs.
> 2024-10-07:17:32:37,948 INFO     [evaluator.py:279] Setting fewshot random generator seed to 1234
> 2024-10-07:17:32:37,949 WARNING  [model.py:422] model.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.
> 2024-10-07:17:32:37,949 INFO     [task.py:423] Building contexts for mmlu_college_mathematics on rank 0...
> 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 868.91it/s]
> 2024-10-07:17:32:38,068 INFO     [evaluator.py:465] Running loglikelihood requests
> Running loglikelihood requests:   0%|                                                                                                                             | 0/400 [00:00<?, ?it/sINFO 10-07 17:32:39 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241007-173239.pkl..., est. speed input: 0.00 toks/s, output: 0.00 toks/s]
> WARNING 10-07 17:32:39 model_runner_base.py:143] Failed to pickle inputs of failed execution: Can't pickle local object 'weak_bind.<locals>.weak_bound'
> [rank0]: Traceback (most recent call last):
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
> [rank0]:     return func(*args, **kwargs)
> [rank0]:            ^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1633, in execute_model
> [rank0]:     output: SamplerOutput = self.model.sample(
> [rank0]:                             ^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 401, in sample
> [rank0]:     next_tokens = self.sampler(logits, sampling_metadata)
> [rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
> [rank0]:     return self._call_impl(*args, **kwargs)
> [rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
> [rank0]:     return forward_call(*args, **kwargs)
> [rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/model_executor/layers/sampler.py", line 257, in forward
> [rank0]:     logits = logits.to(torch.float)
> [rank0]:              ^^^^^^^^^^^^^^^^^^^^^^
> [rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.27 GiB. GPU 0 has a total capacity of 79.15 GiB of which 5.73 GiB is free. Including non-PyTorch memory, this process has 0 bytes memory in use. Of the allocated memory 70.72 GiB is allocated by PyTorch, with 23.60 MiB allocated in private pools (e.g., CUDA Graphs), and 158.20 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
> 
> [rank0]: The above exception was the direct cause of the following exception:
> 
> [rank0]: Traceback (most recent call last):
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/bin/lm_eval", line 8, in <module>
> [rank0]:     sys.exit(cli_evaluate())
> [rank0]:              ^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/__main__.py", line 382, in cli_evaluate
> [rank0]:     results = evaluator.simple_evaluate(
> [rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/utils.py", line 397, in _wrapper
> [rank0]:     return fn(*args, **kwargs)
> [rank0]:            ^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/evaluator.py", line 301, in simple_evaluate
> [rank0]:     results = evaluate(
> [rank0]:               ^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/utils.py", line 397, in _wrapper
> [rank0]:     return fn(*args, **kwargs)
> [rank0]:            ^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/evaluator.py", line 476, in evaluate
> [rank0]:     resps = getattr(lm, reqtype)(cloned_reqs)
> [rank0]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/api/model.py", line 378, in loglikelihood
> [rank0]:     return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
> [rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/models/vllm_causallms.py", line 442, in _loglikelihood_tokens
> [rank0]:     outputs = self._model_generate(requests=inputs, generate=False)
> [rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/lm_eval/models/vllm_causallms.py", line 261, in _model_generate
> [rank0]:     outputs = self.model.generate(
> [rank0]:               ^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/utils.py", line 1047, in inner
> [rank0]:     return fn(*args, **kwargs)
> [rank0]:            ^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 388, in generate
> [rank0]:     outputs = self._run_engine(use_tqdm=use_tqdm)
> [rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 877, in _run_engine
> [rank0]:     step_outputs = self.llm_engine.step()
> [rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 1264, in step
> [rank0]:     outputs = self.model_executor.execute_model(
> [rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/executor/gpu_executor.py", line 130, in execute_model
> [rank0]:     output = self.driver_worker.execute_model(execute_model_req)
> [rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/worker/worker_base.py", line 327, in execute_model
> [rank0]:     output = self.model_runner.execute_model(
> [rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
> [rank0]:     return func(*args, **kwargs)
> [rank0]:            ^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 146, in _wrapper
> [rank0]:     raise type(err)(f"Error in model execution: "
> [rank0]: torch.OutOfMemoryError: Error in model execution: CUDA out of memory. Tried to allocate 16.27 GiB. GPU 0 has a total capacity of 79.15 GiB of which 5.73 GiB is free. Including non-PyTorch memory, this process has 0 bytes memory in use. Of the allocated memory 70.72 GiB is allocated by PyTorch, with 23.60 MiB allocated in private pools (e.g., CUDA Graphs), and 158.20 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
> Processed prompts:   0%|                                                                                      | 0/400 [00:01<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
> Running loglikelihood requests:   0%|                                                                                                                             | 0/400 [00:01<?, ?it/s]
> Traceback (most recent call last):
>   File "/data/miranebr-sandbox/AI4Lean/py_src/ailean_evals/run_lm_harness_eval.py", line 54, in <module>
>     fire.Fire(main)
>   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
>     component_trace = _Fire(component, args, parsed_flag_args, context, name)
>                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
>     component, remaining_args = _CallAndUpdateTrace(
>                                 ^^^^^^^^^^^^^^^^^^^^
>   File "/data/miranebr-sandbox/.virtualenvs/AI4Lean_lm_harness_vllm/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
>     component = fn(*varargs, **kwargs)
>                 ^^^^^^^^^^^^^^^^^^^^^^
>   File "/data/miranebr-sandbox/AI4Lean/py_src/ailean_evals/run_lm_harness_eval.py", line 48, in main
>     subprocess.run(command, shell=True, check=True)
>   File "/usr/lib/python3.11/subprocess.py", line 571, in run
>     raise CalledProcessError(retcode, process.args,
> subprocess.CalledProcessError: Command 'lm_eval --model vllm         --model_args pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594         --tasks mmlu_college_mathematics         --trust_remote_code         --batch_size auto:4         --device cuda         --output_path /data/miranebr-sandbox/lm_harness/eval_results/qwen2_1_5b_ld_pf_dsm/mmlu_college_mathematics         --log_samples ' returned non-zero exit status 1.
> 
> real    1m19.466s
> user    1m9.009s
> sys     0m14.190s
> (AI4Lean_lm_harness_vllm) root@miranebr-math-p4de-math-aif-sft:~# 
> (AI4Lean_lm_harness_vllm) root@miranebr-math-p4de-math-aif-sft:~# nvidia-smi
> Mon Oct  7 17:32:58 2024       
> +---------------------------------------------------------------------------------------+
> | NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
> |-----------------------------------------+----------------------+----------------------+
> | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
> | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
> |                                         |                      |               MIG M. |
> |=========================================+======================+======================|
> |   0  NVIDIA A100-SXM4-80GB          On  | 00000000:10:1C.0 Off |                    0 |
> | N/A   27C    P0              62W / 400W |      0MiB / 81920MiB |      0%      Default |
> |                                         |                      |             Disabled |
> +-----------------------------------------+----------------------+----------------------+
> |   1  NVIDIA A100-SXM4-80GB          On  | 00000000:10:1D.0 Off |                    0 |
> | N/A   25C    P0              60W / 400W |      0MiB / 81920MiB |      0%      Default |
> |                                         |                      |             Disabled |
> +-----------------------------------------+----------------------+----------------------+
> |   2  NVIDIA A100-SXM4-80GB          On  | 00000000:20:1C.0 Off |                    0 |
> | N/A   27C    P0              62W / 400W |      0MiB / 81920MiB |      0%      Default |
> |                                         |                      |             Disabled |
> +-----------------------------------------+----------------------+----------------------+
> |   3  NVIDIA A100-SXM4-80GB          On  | 00000000:20:1D.0 Off |                    0 |
> | N/A   25C    P0              60W / 400W |      0MiB / 81920MiB |      0%      Default |
> |                                         |                      |             Disabled |
> +-----------------------------------------+----------------------+----------------------+
> |   4  NVIDIA A100-SXM4-80GB          On  | 00000000:90:1C.0 Off |                    0 |
> | N/A   27C    P0              60W / 400W |      0MiB / 81920MiB |      0%      Default |
> |                                         |                      |             Disabled |
> +-----------------------------------------+----------------------+----------------------+
> |   5  NVIDIA A100-SXM4-80GB          On  | 00000000:90:1D.0 Off |                    0 |
> | N/A   26C    P0              63W / 400W |      0MiB / 81920MiB |      0%      Default |
> |                                         |                      |             Disabled |
> +-----------------------------------------+----------------------+----------------------+
> |   6  NVIDIA A100-SXM4-80GB          On  | 00000000:A0:1C.0 Off |                    0 |
> | N/A   28C    P0              64W / 400W |      0MiB / 81920MiB |      0%      Default |
> |                                         |                      |             Disabled |
> +-----------------------------------------+----------------------+----------------------+
> |   7  NVIDIA A100-SXM4-80GB          On  | 00000000:A0:1D.0 Off |                    0 |
> | N/A   26C    P0              62W / 400W |      0MiB / 81920MiB |      0%      Default |
> |                                         |                      |             Disabled |
> +-----------------------------------------+----------------------+----------------------+
>                                                                                          
> +---------------------------------------------------------------------------------------+
> | Processes:                                                                            |
> |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
> |        ID   ID                                                             Usage      |
> |=======================================================================================|
> |  No running processes found                                                           |
> +---------------------------------------------------------------------------------------+
> ```
> 
> Command
> 
> ```
> lm_eval --model vllm         --model_args pretrained=Qwen/Qwen2-1.5B,enable_lora=True,lora_local_path=/data/miranebr-sandbox/data/runs/09302024_11h37m55s_run/train/checkpoint-2594         --tasks mmlu_college_mathematics         --trust_remote_code         --batch_size auto:4         --device cuda         --output_path /data/miranebr-sandbox/lm_harness/eval_results/qwen2_1_5b_ld_pf_dsm/mmlu_college_mathematics         --log_samples 
> ```
> 
> @haileyschoelkopf have you ran a local lora ckpt with vllm + lm-harness before? I'm surprised that I get out of memory issue.

Generally speaking, it's out of memery.

Maybe you can use all your 8 GPUS instead of "export CUDA_VISIBLE_DEVICES=0"

e.g. passing "gpu_memory_utilization=0.9,data_parallel_size=8" for your scripts.

If your base model is too large, adding "tensor_parallel_size=8" to parallel your model but for Qwen2-1.5B tensor_parallel_size=1 is enough.

Here is my available script for your information :)

> python3 -m lm_eval --model vllm     --model_args pretrained=/yourbasemodel,lora_local_path=/yourcheckpoint,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=6,trust_remote_code=True   --batch_size auto