Repository: open-compass/VLMEvalKit
Issue: 4090上使用vllm评估会oom
Root cause: Configuration Error

CUDA_VISIBLE_DEVICES=1 python run.py \
  --data MMBench_TEST_EN_V11 MMBench_TEST_CN_V11 CCBench \
  --model MIS-Qwen2.5-VL-7B \
  --use-vllm \
  --api-nproc 1 \
  --verbose


好像是max-token的问题，这个在哪设置？

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 260.00 MiB. GPU 0 has a total capacity of 23.64 GiB of which 77.12 MiB is free. Process 250179 has 19.19 GiB memory in use. 
Process 259318 has 4.36 GiB memory in use. Of the allocated memory 3.82 GiB is allocated by PyTorch, and 102.20 MiB is reserved by PyTorch but unallocated. If reserved but unallocated me
mory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.h
tml#environment-variables)                                                                                                                                                                
[rank0]:[W530 13:04:44.772806306 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please 
see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())                                                                                                       
[2025-05-30 13:04:45] ERROR - RUN - run.py: main - 483: Model MIS-Qwen2.5-VL-7B x Dataset CCBench combination failed: Engine core initialization failed. See root cause above., skipping t
his combination.  

评估MMStar的时候是正常的，但是评估MMBench_Test_EN_V11就会OOM

INFO 05-30 13:04:05 [config.py:717] This model supports multiple tasks: {'reward', 'classify', 'embed', 'generate', 'score'}. Defaulting to 'generate'.                                   
INFO 05-30 13:04:05 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=8192.                                                                                         
[2025-05-30 13:04:11,359] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)                                                                   
/usr/local/lib/python3.11/site-packages/deepspeed/runtime/zero/linear.py:47: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., 
device_type='cuda')` instead.                                                                                                                                                             
  @autocast_custom_fwd                                                                                                                                                                    
/usr/local/lib/python3.11/site-packages/deepspeed/runtime/zero/linear.py:66: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., 
device_type='cuda')` instead.                                                                                                                                                             
  @autocast_custom_bwd                                                                                                                                                                    
[2025-05-30 13:04:12] ERROR - misc.py: load_env - 219: Did not detect the .env file at /home/zhoupc/safe_alignment/VLMEvalKit/.env, failed to load.                                       
INFO 05-30 13:04:12 [importing.py:53] Triton module has been replaced with a placeholder.                                                                                                 
INFO 05-30 13:04:13 [__init__.py:239] Automatically detected platform cuda.                                                                                                               
INFO 05-30 13:04:15 [core.py:58] Initializing a V1 LLM engine (v0.8.5.post1) with config: model='/home/zhoupc/safe_alignment/ms-swift/checkpoint/MIS-sft-Qwen2.5-VL-7B', speculative_confi
g=None, tokenizer='/home/zhoupc/safe_alignment/ms-swift/checkpoint/MIS-sft-Qwen2.5-VL-7B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tok
enizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, 
disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasonin
g_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=
None, served_model_name=/home/zhoupc/safe_alignment/ms-swift/checkpoint/MIS-sft-Qwen2.5-VL-7B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_
prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"
splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture
_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,1
60,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512} 

不设置--use-vllm的时候是不会oom的


> CUDA_VISIBLE_DEVICES=1 python run.py --data MMBench_TEST_EN_V11 MMBench_TEST_CN_V11 CCBench --model MIS-Qwen2.5-VL-7B --use-vllm --api-nproc 1 --verbose
> 
> 好像是max-token的问题，这个在哪设置？

max_new_tokens目前没有作为参数传递，你可以在模型加载时更改这个属性，建议参考[aria model](https://github.com/open-compass/VLMEvalKit/blob/57ca395fd8f4e56fae90c8fad9dbef6e38fe408f/vlmeval/vlm/aria.py)