Repository: open-compass/opencompass
Issue: [Bug] 使用vllm进行ppl推理时显存占用时高时低甚至溢出(llama3.1-8B)
Root cause: Resource Mishandling

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```
{'CUDA available': True,
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'MMEngine': '0.10.6',
 'MUSA available': False,
 'OpenCV': '4.11.0',
 'PyTorch': '2.1.0',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 10.2\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) MKL-DNN v3.1.1 (Git Hash '
                              '64f6bcbcbab628e96f33a62c3e975f8535a7bde4)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: NO AVX\n'
                              '  - Build settings: BLAS_INFO=open, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER '
                              '-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-unused-function -Wno-unused-result '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=old-style-cast '
                              '-Wno-invalid-partial-specialization '
                              '-Wno-unused-private-field '
                              '-Wno-aligned-allocation-unavailable '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=open, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.1.0, USE_CUDA=OFF, '
                              'USE_CUDNN=OFF, USE_EIGEN_FOR_BLAS=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.10.2 (main, Jan  8 2025, 06:13:36) [GCC 9.4.0]',
 'TorchVision': '0.16.0',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.9+',
 'sys.platform': 'linux',
 'transformers': '4.46.3'}
```

### Reproduces the problem - code/configuration sample

当我在使用vllm运行llama3.1-8b进行ppl推理模型时(所用数据集均为官方configs/目录下所选取的数据集)，发现显存占用时高时低，甚至会导致溢出错误
```
python run.py --models vllm_llama3_1_8b --datasets base_Custom --summarizer base_Custom --work-dir outputs/Llama3_1-8B-vllm --max-num-workers 8
```

### Reproduces the problem - command or script

vllm_llama3_1_8b.py代码如下:
```
from opencompass.models import VLLM


models = [
    dict(
        type=VLLM,
        abbr='llama-3.1-8b-vllm',
        path='/home/ma-user/work/s00851266/Llama-3.1-8B',
        model_kwargs=dict(tensor_parallel_size=1, gpu_memory_utilization=0.7, max_model_len=4096),  # 由于发生过几次显存溢出, 于是我添加了gpu_参数设置为0.7--没用. 添加了max_model_len使用4k，但也在溢出的边缘(好像此参数影响也不大), 并且不清楚4k是否满足数据集长度
        max_out_len=512,
        batch_size=8,
        generation_kwargs=dict(temperature=0),
        run_cfg=dict(num_gpus=1),
    )
]
```

### Reproduces the problem - error message

同一个任务不同运行时显存占用:

![Image](https://github.com/user-attachments/assets/8280afff-4eb4-4e59-9193-0071fc8b1263)

![Image](https://github.com/user-attachments/assets/fd7de303-a1d2-47e6-aa47-7a8554745eec)
[P.S.]值得注意的是, instruct模型完全没有类似问题. 那么感觉在ppl代码可能有问题吗?

### Other information

_No response_

另外补充一条, 使用vllm评测出来的结果差hf的特别多, base模型ppl的话, 只是在脚本添加-a vllm以后结果差20%以上(例如mmlu_ppl_ac766d). instruct模型gen评测同样没有这个问题

Thanks for the report. We will check the accuracy issue. For the memory issue, can you try LMDeploy?