Repository: open-compass/opencompass
Issue: [Bug] Evaluating an OpenAI Compatible Server hang
Root cause: Resource Mishandling

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```
{'CUDA available': False,
 'GCC': 'Apple clang version 15.0.0 (clang-1500.3.9.4)',
 'MMEngine': '0.10.5',
 'MUSA available': False,
 'OpenCV': '4.10.0',
 'PyTorch': '2.4.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 4.2\n'
                              '  - C++ Version: 201703\n'
                              '  - clang 14.0.3\n'
                              '  - OpenMP 201811\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: NO AVX\n'
                              '  - Build settings: BLAS_INFO=accelerate, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=/Applications/Xcode_14.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++, '
                              'CXX_FLAGS= -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER '
                              '-DUSE_PYTORCH_QNNPACK '
                              '-DAT_BUILD_ARM_VEC256_WITH_SLEEF -DUSE_XNNPACK '
                              '-DUSE_PYTORCH_METAL_EXPORT '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-DUSE_COREML_DELEGATE -O2 -fPIC -Wall -Wextra '
                              '-Werror=return-type -Werror=non-virtual-dtor '
                              '-Werror=braced-scalar-init '
                              '-Werror=range-loop-construct '
                              '-Werror=bool-operation -Wnarrowing '
                              '-Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-unused-function -Wno-unused-result '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wvla-extension -Wsuggest-override '
                              '-Wnewline-eof -Winconsistent-missing-override '
                              '-Winconsistent-missing-destructor-override '
                              '-Wno-pass-failed -Wno-error=pedantic '
                              '-Wno-error=old-style-cast '
                              '-Wno-error=inconsistent-missing-override '
                              '-Wno-error=inconsistent-missing-destructor-override '
                              '-Wconstant-conversion '
                              '-Wno-invalid-partial-specialization '
                              '-Wno-missing-braces -Qunused-arguments '
                              '-fcolor-diagnostics -faligned-new '
                              '-Wno-unused-but-set-variable -fno-math-errno '
                              '-fno-trapping-math -Werror=format -DUSE_MPS '
                              '-Wno-unused-private-field -Wno-missing-braces, '
                              'LAPACK_INFO=accelerate, TORCH_VERSION=2.4.1, '
                              'USE_CUDA=OFF, USE_CUDNN=OFF, '
                              'USE_CUSPARSELT=OFF, USE_EIGEN_FOR_BLAS=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=OFF, '
                              'USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, '
                              'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.14 (main, May  6 2024, 14:42:37) [Clang 14.0.6 ]',
 'TorchVision': '0.19.1',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.2.post1+070a92b',
 'sys.platform': 'darwin',
 'transformers': '4.45.1'}
```

### Reproduces the problem - code/configuration sample


./configs/eval_api_demo.py
```python
from opencompass.models import OpenAI
from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.demo.demo_gsm8k_chat_gen import gsm8k_datasets
    from opencompass.configs.datasets.demo.demo_math_chat_gen import math_datasets
    from opencompass.configs.models.openai.gpt_4o_2024_05_13 import models as gpt4

datasets = gsm8k_datasets + math_datasets
api_meta_template = dict(round=[
    dict(role='HUMAN', api_role='HUMAN'),
    dict(role='BOT', api_role='BOT', generate=True),
], )
models = [
    dict(
        abbr='glm4-9b-chat',
        type=OpenAI,
        path='glm4-9b-chat',
        key='ENV',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
        meta_template=api_meta_template,
        query_per_second=1,
        max_out_len=2048,
        max_seq_len=4096,
        batch_size=8
    )
]

```

### Reproduces the problem - command or script

export OPENAI_BASE_URL="http://test-host:18080/v1"
export OPENAI_API_KEY="EMPTY"
opencompass  ./configs/eval_api_demo.py --debug

### Reproduces the problem - error message

Hang for long time

09/28 17:12:34 - OpenCompass - INFO - Current exp folder: outputs/default/20240928_171234
09/28 17:12:34 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
09/28 17:12:34 - OpenCompass - INFO - Partitioned into 1 tasks.
09/28 17:12:34 - OpenCompass - INFO - Task [glm4-9b-chat/demo_gsm8k,glm4-9b-chat/demo_math]
09/28 17:12:34 - OpenCompass - INFO - Start inferencing [glm4-9b-chat/demo_gsm8k]
09/28 17:12:34 - OpenCompass - WARNING - 'Could not automatically map glm4-9b-chat to a tokeniser. Please use `tiktoken.get_encoding` to explicitly get the tokeniser you expect.', tiktoken encoding cannot load glm4-9b-chat

### Other information

_No response_

Finally completed

> Finally completed

I encountered the same issue. Could you please share how it was resolved in the end?


> > Finally completed
> 
> I encountered the same issue. Could you please share how it was resolved in the end?

Do nothing, you need to wait for more than ten minutes, the program will continue to run.