Repository: EleutherAI/lm-evaluation-harness
Issue: Inconsistencies with Model Performance when using LoRA Requests & vLLM
Root cause: Algorithmic Error

Im curious about other folks experiences with using `vllm` and `lm-eval` -- when using a command like the following:

```
export TASK="custom-function-calling-task"
export lora_adapter_dir="/some/path/where/an/adapter_config/lives"

lm_eval --model vllm    \
    --model_args  pretrained=meta-llama/Llama-3.1-8B-Instruct,enable_lora=True,lora_local_path=$lora_adapter_dir,dtype=auto     \
    --task $TASK    \
    --limit 30    \
    --batch_size auto                \
    --log_samples                    \
    --output_path out
```

Which I believe is calling into the api correctly ([see discussion here](https://github.com/EleutherAI/lm-evaluation-harness/issues/672)). The results of using this command to evaluate an adapter/model are as if there was no adapter included in the evaluation. As in running an evaluation with a fine-tuned adapter and vllm results in the exact same (greedy decoded) results as running the evaluation with just the base model. I have confirmed that the adapter does get loaded, and that at runtime we have a valid value for `self.lora_request` inside of `lm_eval/models/vllm_causallms.py`. To verify that the adapter weights weren't "borked" in my case, I merged it with the base model and re-ran the evaluation with the merged model -- the results were quite different from the other greedy results.


Here are some relevant values of `model_args` I used and screenshots of corresponding eval results:
- lora loaded in via api: `--model_args  pretrained=meta-llama/Llama-3.1-8B-Instruct,enable_lora=True,lora_local_path=$lora_adapter_dir`
  <img width="1159" alt="Screenshot 2024-10-28 at 11 38 49 AM" src="https://github.com/user-attachments/assets/f588dc61-90b1-4fab-aef2-cba93af4b88b">

- base model: `--model_args  pretrained=meta-llama/Llama-3.1-8B-Instruct`
  <img width="1372" alt="Screenshot 2024-10-28 at 11 39 15 AM" src="https://github.com/user-attachments/assets/ebcd6610-21ea-486d-b624-3fa6d2b9bc21">

- lora merged into base model: `--model_args  pretrained=/models/merged_model_output`
  <img width="1504" alt="Screenshot 2024-10-28 at 11 41 34 AM" src="https://github.com/user-attachments/assets/adb02bd9-70d1-444d-8c9d-df0f217b6c29">


Has anyone else came across this sort of issue? Might there be discontinuity between the way LoRA requests are issued in `lm-eval` and the integrated `vLLM`?



As an aside, would someone know exactly where `enable_lora` get used? It presumably is parsed as a `kwarg` in [lm_eval/models/vllm_causallms.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/feff1b55c57993c4d42c8f913a22eeec395cd690/lm_eval/models/vllm_causallms.py#L62C9-L62C37) but then I don't think it gets applied/referenced anywhere in the `LLM` init, reference- [vllm/entrypoints/llm.py](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py).

- Answer: this is used for the [EngineArgs](https://github.com/vllm-project/vllm/blob/5f8d8075f957d5376b2f1cc451e35a2a757e95a5/vllm/engine/arg_utils.py#L139)

Mmmm, the issue [could be here](https://github.com/EleutherAI/lm-evaluation-harness/blob/feff1b55c57993c4d42c8f913a22eeec395cd690/lm_eval/models/vllm_causallms.py#L240-L272): 
```
# commit: 67a990e in lm_eval/models/vllm_causallms.py
        if self.data_parallel_size > 1:
            # vLLM hangs if tensor_parallel > 1 and resources are set in ray.remote
            # also seems to only work with decorator and not with ray.remote() fn
            # see https://github.com/vllm-project/vllm/issues/973
            # note: this has changed on 0.3.3, and it only works now if num_gpus are set.
            # but then tensor_parallel breaks
            @ray.remote
            def run_inference_one_model(
                model_args: dict, sampling_params, requests: List[List[int]]
            ):
                llm = LLM(**model_args)
                return llm.generate(
                    prompt_token_ids=requests, sampling_params=sampling_params
                )

            # dispatch requests to all self.data_parallel_size workers, in interleaved fashion
            # interleaved important to balance context lengths across workers
            requests = [list(x) for x in distribute(self.data_parallel_size, requests)]
            inputs = ((self.model_args, sampling_params, req) for req in requests)
            object_refs = [run_inference_one_model.remote(*x) for x in inputs]
            results = ray.get(object_refs)
            # Invoke ray.shutdown() to prevent hang-ups if subsequent calls required.
            ray.shutdown()
            # flatten results
            return undistribute(results)

        if self.lora_request is not None:
            outputs = self.model.generate(
                prompt_token_ids=requests,
                sampling_params=sampling_params,
                use_tqdm=True if self.batch_size == "auto" else False,
                lora_request=self.lora_request,
            )
```


when using data parallelism and vllm, it appears that lora requests are not distributed to all of the workers, the contents of `run_inference_one_model` distributes base model requests.