Repository: open-compass/opencompass
Issue: [Bug] Base model infer OOM
Root cause: Resource Mishandling

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

{'CUDA available': True,
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'MMEngine': '0.10.6',
 'MUSA available': False,
 'OpenCV': '4.11.0',
 'PyTorch': '2.5.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 10.2\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: NO AVX\n'
                              '  - Build settings: BLAS_INFO=open, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, '
                              'CXX_FLAGS=-ffunction-sections -fdata-sections '
                              '-D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=open, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=OFF, '
                              'USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.2 (main, Jan  8 2025, 06:13:36) [GCC 9.4.0]',
 'TorchVision': '0.20.1',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.9+',
 'sys.platform': 'linux',
 'transformers': '4.48.2'}

### Reproduces the problem - code/configuration sample

my configs are as follows:
for hf
```
from opencompass.models import HuggingFaceBaseModel

models = [
    dict(
        type=HuggingFaceBaseModel,
        abbr='qwen2.5-7b-hf',
        path='//Qwen-Series/Qwen2.5-7B',
        max_out_len=4096,
        batch_size=8,
        run_cfg=dict(num_gpus=1),
    )
]
```
for vllm
```
from opencompass.models import VLLM


models = [
    dict(
        type=VLLM,
        abbr='qwen2.5-7b-vllm',
        path='/Qwen-Series/Qwen2.5-7B',
        model_kwargs=dict(tensor_parallel_size=1, gpu_memory_utilization=0.9, max_model_len=4096),
        max_out_len=512,
        batch_size=8,
        generation_kwargs=dict(temperature=0),
        run_cfg=dict(num_gpus=1),
    )
]
```

### Reproduces the problem - command or script

```
python run.py --models vllm_qwen2_5_7b --datasets mmlu_ppl  --max-num-workers 8
python run.py --models hf_qwen2_5_7b --datasets mmlu_ppl  --max-num-workers 8
```

### Reproduces the problem - error message

when i run vllm mode it is always oom but i set max_out_len=512 and max_model_len=4096.
hf mode uses 60G for a single card with config as above.
I don't think setting it up as above (at least for vllm) will result in oom, a single card with 64 gigs of video memory. It's strange, when I opened the infer log I found the following picture.

![Image](https://github.com/user-attachments/assets/cac75875-8fdd-44f9-b7e3-d380fe504e6a)
This is supposed to mean batch size, but I've set bs=8. But the image shows bs changing with the dataset

### Other information

_No response_