Repository: EleutherAI/lm-evaluation-harness
Issue: Keep getting error: 'VLLM' object has no attribute 'AUTO_MODEL_CLASS'
Root cause: Validation Gap

Hello, i tried to run eval on local with vllm, but the same error occurs.

version
- evaluation-harness==0.4.2
- vllm==0.3.2

```
export HF_DATASETS_OFFLINE=1;
lm-eval \
--tasks mmlu \
--model vllm \
--model_args pretrained=/model/path/Meta-Llama-3-8B,tokenizer=/model/path/Meta-Llama-3-8B,tensor_parallel_size=1,dtype=bfloat16,data_parallel_size=4,gpu_memory_utilization=0.8 \
--batch_size "auto" \
--num_fewshot 5 \
--write_out \
--output_path ../results/
```
```
2024-06-10:18:05:24,757 INFO     [evaluator.py:362] Running loglikelihood requests
Traceback (most recent call last):
  File "/usr/local/bin/lm-eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/jovyan/workspace/llm-data/lm-evaluation-harness-0.4.2/lm_eval/__main__.py", line 342, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/jovyan/workspace/llm-data/lm-evaluation-harness-0.4.2/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/home/jovyan/workspace/llm-data/lm-evaluation-harness-0.4.2/lm_eval/evaluator.py", line 234, in simple_evaluate
    results = evaluate(
  File "/home/jovyan/workspace/llm-data/lm-evaluation-harness-0.4.2/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/home/jovyan/workspace/llm-data/lm-evaluation-harness-0.4.2/lm_eval/evaluator.py", line 373, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/home/jovyan/workspace/llm-data/lm-evaluation-harness-0.4.2/lm_eval/api/model.py", line 325, in loglikelihood
    context_enc, continuation_enc = self._encode_pair(context, continuation)
  File "/home/jovyan/workspace/llm-data/lm-evaluation-harness-0.4.2/lm_eval/api/model.py", line 300, in _encode_pair
    if self.AUTO_MODEL_CLASS == transformers.AutoModelForCausalLM:
AttributeError: 'VLLM' object has no attribute 'AUTO_MODEL_CLASS'
```

As implied, the error occured in the [model initialization](https://github.com/EleutherAI/lm-evaluation-harness/blob/b3e4c49a1271887c5721bc5f92e77deaef76ed5d/lm_eval/api/model.py#L340), where [VLLM template](https://github.com/EleutherAI/lm-evaluation-harness/blob/b3e4c49a1271887c5721bc5f92e77deaef76ed5d/lm_eval/models/vllm_causallms.py#L34) has no such attribute 'AUTO_MODEL_CLASS'.

Do I need to manually put it as argument? 

you can try if it is fixed in the latest state of main branch. there were many fixes, changes and improvements since release of 0.4.2

Hi, This is fixed on the `main` branch! We will soon be putting out a new version release for v0.4.3 .

Hi, Could you provide me with steps to solve this? I have tried updating the repo a couple of times (i am in main) but i keep getting this. @haileyschoelkopf 

as a temporary solution you can skip the conditon causing the error by choosing your type of model in the lm_eval/api/model.py in    def _encode_pair(self, context, continuation): (in line 299) and setting the "AUTO_MODEL_CLASS" manually. so in my case for "llama3 70b" i changed     if self.AUTO_MODEL_CLASS == transformers.AutoModelForCausalLM: to if True and it worked. 

@malhajar17 have you uninstalled and reinstalled after pulling the latest commits from main?

having the same issue here. When is the 0.43 version scheduled for a release?

> @malhajar17 have you uninstalled and reinstalled after pulling the latest commits from main?

Yeah!, I also fall this hole.  Yes, of course.  Where is 0.4.3? Extremely anxious

v0.4.3 is released on PyPI!