Repository: EleutherAI/lm-evaluation-harness
Issue: vllm backend faild
Root cause: Environment Incompatibility

hi, i tried to eval :
`
export CUDA_VISIBLE_DEVICES="2,3"
accelerate launch -m lm_eval --model vllm \
	--model_args pretrained="THUDM/glm-4-9b",dtype=bfloat16 \
        --tasks mmlu \
        --device cuda \
	--batch_size 2 \
        --trust_remote_code \
        --cache_requests true \
        --num_fewshot 5 \
`
for accelerate, but failed in A100 (vllm=0.5.0.post1,  vllm-flash-attn  =2.5.9, torch2.3.0+cu121)： 
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/17668109/2f55f7fe-ed1b-4e36-b73e-9411e3ebaab6)




Hi, in order to help solve your issue we'd need more information. Namely: 

- can you provide the *full* traceback, not just this snippet from it?
- can you provide the library version that you are running with?

full traceback , it's too long,  abstract key lines:
vllm/vllm/engine/llm_engine.py", line 230,
vllm/vllm/executor/executor_base.py", line 41
/vllm/vllm/executor/gpu_executor.py"
vllm/vllm/distributed/parallel_state.py", line 771,

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/17668109/f2b49a27-598f-4560-9ed1-b67934171ae8)

library version:

vllm=0.5.0.post1,  vllm-flash-attn  =2.5.9, torch2.3.0+cu121) , lm-eval 0.4.2

I see the problem-- `accelerate launch` should not be used with VLLM. instead use just `lm_eval --model vllm --model_args data_parallel_size=NUMGPUS`

thank for ur comment, another error, test:
`
export CUDA_VISIBLE_DEVICES="2,7"
#accelerate launch -m 
lm_eval --model vllm \
        --model_args pretrained="THUDM/glm-4-9b",dtype=bfloat16,data_parallel_size=2 \
        --tasks mmlu \
        --device cuda \
        --batch_size 2 \
        --trust_remote_code \
        --cache_requests true \
        --num_fewshot 5
`
/*/python3.10/site-packages/lm_eval/api/model.py", line 300, in _encode_pair
    if self.AUTO_MODEL_CLASS == transformers.AutoModelForCausalLM:
AttributeError: 'VLLM' object has no attribute 'AUTO_MODEL_CLASS'

and when pip install lm_eval[vllm],  and fix refer https://github.com/EleutherAI/lm-evaluation-harness/issues/1953
but, always OOM, no matter i use 4 card A100 or 2 card A100

Hi, could you try the following:

- setting `enforce_eager=True` as described in #1923 
- setting `gpu_memory_utilization=0.8` or lower
- using `lm_eval==0.4.3` from PyPI ; or the most recent commit from `main`

pip install lm_eval[vllm] .  error report:

lm-evaluation-harness/lm_eval/evaluator.py", line 15, in <module>
    from lm_eval.caching.cache import delete_cache
ModuleNotFoundError: No module named 'lm_eval.caching.cache'

but i can import in python client:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/17668109/886a4619-0a1c-4509-8a2f-28ccbb803fab)

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/17668109/4e7284c9-cdf7-4bf0-aee3-e834e1c51a64)
 
 

@chunniunai220ml are you by chance running your PyPI-installed `lm_eval` in the same folder as an older git clone of lm-evaluation-harness?

at first , pip install lm_eval[vllm],  error as reported.  then git pull the latest code, pip install -e . , same error. 
i tried to create __init__.py in lm_eval/cache,  doesn't work untill now
@haileyschoelkopf 

I got the same `No module named 'lm_eval.caching.cache'` after installing dependencies for ifeval -- creating an __init__.py in lm_eval/caching and rebuilding worked