Repository: open-compass/opencompass
Issue: [Bug] humaneval dataset infer success, but eval failed
Root cause: Environment Incompatibility

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```python
{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)',
 'GPU 0': 'Quadro P4000',
 'MMEngine': '0.8.4',
 'NVCC': 'Cuda compilation tools, release 12.2, V12.2.91',
 'OpenCV': '4.8.0',
 'PyTorch': '2.0.1+cu118',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2022.2-Product Build 20220804 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v2.7.3 (Git Hash '
                              '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 11.8\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 8.7\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
                              'CUDNN_VERSION=8.7.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -Wno-deprecated '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_QNNPACK '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wunused-local-typedefs '
                              '-Wno-unused-parameter -Wno-unused-function '
                              '-Wno-unused-result -Wno-strict-overflow '
                              '-Wno-strict-aliasing '
                              '-Wno-error=deprecated-declarations '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=redundant-decls '
                              '-Wno-error=old-style-cast '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.10.10 (main, Jul 28 2023, 09:11:36) [GCC 8.5.0 20210514 (Red Hat '
           '8.5.0-20)]',
 'TorchVision': '0.15.2+cu118',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.1.2+339a9cc',
 'sys.platform': 'linux'}
```

### Reproduces the problem - code/configuration sample

```python
from mmengine import read_base

with read_base():
    from ..Xsum.Xsum_gen import Xsum_datasets
    
    from ..CLUE_cmnli.CLUE_cmnli_gen import cmnli_datasets
    
    from ..crowspairs.crowspairs_gen import crowspairs_datasets
    from ..humaneval.humaneval_gen import humaneval_datasets
datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), [])
```

### Reproduces the problem - command or script

```
python run.py configs/eval_llama2_70b_api.py -w outputs/llama2_70b_api_multi_dataset --reuse
```

### Reproduces the problem - error message

> logs/infer/llama2-70b-api/openai_humaneval.out:
```shell
100%|██████████| 164/164 [18:43<00:00,  4.65s/it]
100%|██████████| 164/164 [18:43<00:00,  6.85s/it]
/root/code/opencompass/.venv/lib/python3.10/site-packages/mmengine/utils/manager.py:113: UserWarning: <class 'mmengine.logging.logger.MMLogger'> instance named of OpenCompass has been created, the method `get_instance` should not accept any other arguments
  warnings.warn(
09/05 17:30:35 - OpenCompass - INFO - time elapsed: 1525.58s
```

> logs/eval/llama2-70b-api/openai_humaneval.out
```shell
/root/code/opencompass/.venv/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
  warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/openai_humaneval/2955cebd73602e828fa8c0a424c594e5fab4ec863b316ca98f3d8fdb6a626e75 (last modified on Fri Sep  1 15:27:25 2023) since it couldn't be found locally at openai_humaneval., or remotely on the Hugging Face Hub.
Reading samples...

0it [00:00, ?it/s]
164it [00:00, 32110.25it/s]
Running test suites...

  0%|          | 0/164 [00:00<?, ?it/s]Process SyncManager-1:
Traceback (most recent call last):
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/managers.py", line 591, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/managers.py", line 156, in __init__
    self.listener = Listener(address=address, backlog=16)
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/connection.py", line 448, in __init__
    self._listener = SocketListener(address, family, backlog)
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/connection.py", line 592, in __init__
    self._socket.bind(address)
PermissionError: [Errno 1] Operation not permitted
...
Process SyncManager-164:
Traceback (most recent call last):
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/managers.py", line 591, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/managers.py", line 156, in __init__
    self.listener = Listener(address=address, backlog=16)
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/connection.py", line 448, in __init__
    self._listener = SocketListener(address, family, backlog)
  File "/usr/local/python3.10/lib/python3.10/multiprocessing/connection.py", line 592, in __init__
    self._socket.bind(address)
PermissionError: [Errno 1] Operation not permitted

```

### Other information

1. other datasets get score, only human-eval dataset failed in evaling
2. I used root, shouldn't have permission problem

Not sure where the problem is，you might check following things
1. Is there any other humaneval eval task not being killed？
2. Your `multiprocess` package is not in the same env as your `/root/code/opencompass/.venv/lib/python3.10` env, might be the problem
3. Try use sudo to run the script

> Not sure where the problem is，you might check following things
> 
> 1. Is there any other humaneval eval task not being killed？
> 2. Your `multiprocess` package is not in the same env as your `/root/code/opencompass/.venv/lib/python3.10` env, might be the problem
> 3. Try use sudo to run the script

multiprocess is built-in module for python3, It's seems right for venv not same as my opencompass venv.

finally, I fixed it by rebuild python and reinstall opencompass venv