Repository: open-compass/opencompass
Issue: [Bug] The performance of VLLM opencompass is significantly lower than expected
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda-12.1',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-SXM4-80GB',
 'MMEngine': '0.10.4',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 12.1, V12.1.105',
 'OpenCV': '4.10.0',
 'PyTorch': '2.3.0+cu121',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.3.6 (Git Hash '
                              '86e6af5974177e513fd3fee58425e1063e7f1361)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX512\n'
                              '  - CUDA Runtime 12.1\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 8.9.2\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=12.1, '
                              'CUDNN_VERSION=8.9.2, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DUSE_FBGEMM '
                              '-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-unused-function -Wno-unused-result '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=pedantic '
                              '-Wno-error=old-style-cast -Wno-missing-braces '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, TORCH_VERSION=2.3.0, '
                              'USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, '
                              'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, '
                              'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0]',
 'TorchVision': '0.18.0+cu121',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.2.6+a62c613',
 'sys.platform': 'linux'}

### Reproduces the problem - code/configuration sample

# Normal llama 7B inference. 
# configs/llama7b.py
from mmengine.config import read_base

from opencompass.partitioners import SizePartitioner
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask

with read_base():
    # 直接从预设数据集配置中读取需要的数据集配置
    from .datasets.mmlu.mmlu_gen_4d595a import mmlu_datasets
    # from .datasets.cmmlu.cmmlu_gen_c13365 import cmmlu_datasets
    # from .datasets.ceval.ceval_gen_5f30c7 import ceval_datasets
    from .datasets.GaokaoBench.GaokaoBench_no_subjective_gen_4c31db import GaokaoBench_datasets
    from .datasets.triviaqa.triviaqa_wiki_1shot_gen_bc5f21 import triviaqa_datasets
    # # from .datasets.nq.nq_open_1shot_gen_2e45e5 import nq_datasets
    # from .datasets.race.race_gen_69ee4f import race_datasets
    # from .datasets.winogrande.winogrande_5shot_gen_b36770 import winogrande_datasets
    from .datasets.hellaswag.hellaswag_10shot_gen_e42710 import hellaswag_datasets
    # from .datasets.bbh.bbh_gen_2879b0 import bbh_datasets
    from .datasets.gsm8k.gsm8k_gen_1d7fe4 import gsm8k_datasets
    # from .datasets.math.math_0shot_gen_393424 import math_datasets
    # # from .datasets.TheoremQA.TheoremQA_5shot_gen_6f0af8 import TheoremQA_datasets
    from .datasets.humaneval.humaneval_gen_8e312c import humaneval_datasets
    from .datasets.mbpp.sanitized_mbpp_gen_830460 import sanitized_mbpp_datasets
    # # from .datasets.gpqa.gpqa_gen_4baadb import gpqa_datasets
    # # from .datasets.IFEval.IFEval_gen_3321a3 import ifeval_datasets
    from .summarizers.chat_OC15 import summarizer

# 将需要评测的数据集拼接成 datasets 字段
# datasets = [*mmlu_datasets]
datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), [])

# 使用 HuggingFaceCausalLM 评测 HuggingFace 中 AutoModelForCausalLM 支持的模型
from opencompass.models import HuggingFaceBaseModel

models = [
    dict(
        type=HuggingFaceBaseModel,
        # 以下参数为 `HuggingFaceCausalLM` 的初始化参数
        path='llama-7B',
        tokenizer_path='llama-7B',
        tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),
        max_seq_len=2048,
        batch_padding=False,
        # 以下参数为各类模型都有的参数，非 `HuggingFaceCausalLM` 的初始化参数
        abbr='llama7B',            # 模型简称，用于结果展示
        max_out_len=100,            # 最长生成 token 数
        batch_size=8,              # 批次大小
        run_cfg=dict(num_gpus=2),   # 运行配置，用于指定资源需求
    )
]

infer = dict(
    partitioner=dict(type=SizePartitioner),
    runner=dict(
        type=LocalRunner,
        max_num_workers=8,
        task=dict(type=OpenICLInferTask)),
)

### Reproduces the problem - command or script

`python run.py <path to config file> -a vllm`

### Reproduces the problem - error message

The performance of vllm is significantly lowered comparing to normal NVIDIA vllm inference. It is extremely slow. We observed that the utilization of GPU is highly lower than 100%, while in the normal situation, the utilizition of GPU is exectly 100%. We highly suspect that the vllm method has a serious bug on vllm interface. 
Also, during inference, GPU OUT OF MEMORY bug has occurred several time. Thus we want to ask if you have forgot to free the allocated memory after one iteration. 

### Other information

_No response_

你这里使用的是HuggingFaceBaseModel吧，不是VLLM

哦哦，原来如此，在config文件后面加-a vllm还是没用的，问题解决了，谢谢！