Repository: open-compass/opencompass
Issue: 评测结果内容为空
Root cause: Environment Incompatibility

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

{'CUDA available': True,
 'CUDA_HOME': '/usr',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'Tesla P40',
 'MMEngine': '0.8.4',
 'NVCC': 'Cuda compilation tools, release 10.1, V10.1.24',
 'OpenCV': '4.8.0',
 'PyTorch': '2.0.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v2.7.3 (Git Hash '
                              '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 11.8\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
                              '  - CuDNN 8.7\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
                              'CUDNN_VERSION=8.7.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -Wno-deprecated '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_QNNPACK '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wunused-local-typedefs '
                              '-Wno-unused-parameter -Wno-unused-function '
                              '-Wno-unused-result -Wno-strict-overflow '
                              '-Wno-strict-aliasing '
                              '-Wno-error=deprecated-declarations '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=redundant-decls '
                              '-Wno-error=old-style-cast '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]',
 'TorchVision': '0.15.2',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.1.4+3e980a9',
 'sys.platform': 'linux'}


### 重现问题 - 代码/配置示例

from opencompass.models import HuggingFaceCausalLM

models = [
dict(
type=HuggingFaceCausalLM,
abbr='vicuna-33b-v1.3-hf',
path="/data/models/vicuna-33b-v1.3",
tokenizer_path="/data/models/vicuna-33b-v1.3",
tokenizer_kwargs=dict(
padding_side='left',
truncation_side='left',
use_fast=False,
),
max_out_len=100,
max_seq_len=2048,
batch_size=8,
model_kwargs=dict(device_map='auto'),
batch_padding=False, # if false, inference with for-loop without batch padding
run_cfg=dict(num_gpus=4, num_procs=1)
)
]

### 重现问题 - 命令或脚本

python run.py --models hf_vicuna_33b_v13 --datasets ceval_gen

### 重现问题 - 错误信息

![image](https://github.com/open-compass/opencompass/assets/131145127/b6efd114-1fb1-4a5a-823b-a5f86fa8e3ca)

### 其他信息

_No response_

There was something wrong happening in your evaluation. There must be some "ERROR"s logged on the terminal before, and you can find out the reason in those `.out` files, which are all stored in the experiment directory.

Traceback (most recent call last):
&nbsp; File "/data/project/opencompass/opencompass/tasks/openicl_infer.py", line 7, in <module&gt;
&nbsp; &nbsp; from mmengine.config import Config, ConfigDict
ModuleNotFoundError: No module named 'mmengine'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 606843) of binary: /usr/bin/python3
Traceback (most recent call last):
&nbsp; File "/home/fastchat/.local/bin/torchrun", line 8, in <module&gt;
&nbsp; &nbsp; sys.exit(main())
&nbsp; File "/home/fastchat/.local/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
&nbsp; &nbsp; return f(*args, **kwargs)
&nbsp; File "/home/fastchat/.local/lib/python3.8/site-packages/torch/distributed/run.py", line 794, in main
&nbsp; &nbsp; run(args)
&nbsp; File "/home/fastchat/.local/lib/python3.8/site-packages/torch/distributed/run.py", line 785, in run
&nbsp; &nbsp; elastic_launch(
&nbsp; File "/home/fastchat/.local/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
&nbsp; &nbsp; return launch_agent(self._config, self._entrypoint, list(args))
&nbsp; File "/home/fastchat/.local/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
&nbsp; &nbsp; raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:&nbsp;
============================================================
/data/project/opencompass/opencompass/tasks/openicl_infer.py FAILED
------------------------------------------------------------
Failures:
&nbsp; <NO_OTHER_FAILURES&gt;
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
&nbsp; time&nbsp; &nbsp; &nbsp; : 2023-09-26_14:00:25
&nbsp; host&nbsp; &nbsp; &nbsp; : supermicro-G4A60
&nbsp; rank&nbsp; &nbsp; &nbsp; : 0 (local_rank: 0)
&nbsp; exitcode&nbsp; : 1 (pid: 606843)
&nbsp; error_file: <N/A&gt;
&nbsp; traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html









还有很多 ERROR不清楚原因
ERROR - /data/project/opencompass/opencompass/tasks/openicl_eval.py - _score - 163 - Task [vicuna-33b-v1.3-hf/ceval-operating_system]: No predictions found.

ERROR - /data/project/opencompass/opencompass/tasks/openicl_eval.py - _score - 163 - Task [vicuna-33b-v1.3-hf/ceval-probability_and_statistics]: No predictions found.







&nbsp;


十&nbsp;指&nbsp;扣、
***@***.***



&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人: "Tong ***@***.***&gt;; 
发送时间: 2023年9月26日(星期二) 下午2:14
收件人: ***@***.***&gt;; 
抄送: "十 指 ***@***.***&gt;; ***@***.***&gt;; 
主题: Re: [open-compass/opencompass] 评测结果内容为空 (Issue #436)






你的评估有问题。之前一定有一些“错误”记录在终端上，你可以在这些错误中找到原因。.在外文件，这些文件都存储在实验目录下。

-
直接回复此邮件，在GitHub上查看，或取消订阅.
***@***.***与&gt;.

> ModuleNotFoundError: No module named 'mmengine'
> ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 606843) of binary: /usr/bin/python3

Looks like your environment does not link `python3` correctly, but still maps `python3` to `/usr/bin/python3` and leads to such dependency missing error. You can fix it by reinstalling python3 in your environment and retry.

我也遇到了类似的问题，想请教下你已经解决了吗？


没有
&nbsp;


十&nbsp;指&nbsp;扣、
***@***.***



&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人: ***@***.***&gt;; 
发送时间: 2023年10月7日(星期六) 晚上11:44
收件人: ***@***.***&gt;; 
抄送: "十 指 ***@***.***&gt;; ***@***.***&gt;; 
主题: Re: [open-compass/opencompass] 评测结果内容为空 (Issue #436)





 
我也遇到了类似的问题，想请教下你已经解决了吗？
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

同样遇到了这个问题

我遇到同样的问题，我在相同目录下新建test.py,并且直接执行并不会报错
![image](https://github.com/open-compass/opencompass/assets/76529076/4e74ca99-b859-4ca8-bb69-b59d8c3a1c49)

直接进入python import这个库也不会报错


> 我遇到同样的问题，我在相同目录下新建test.py,并且直接执行并不会报错 ![image](https://user-images.githubusercontent.com/76529076/274793226-4e74ca99-b859-4ca8-bb69-b59d8c3a1c49.png)
> 
> 直接进入python import这个库也不会报错
在进行Debug run.py时候 脚本在这里报错
![image](https://github.com/open-compass/opencompass/assets/76529076/87267545-efaf-4b25-bc63-4b8751081e76)
继续单步调试 发现文件在
![image](https://github.com/open-compass/opencompass/assets/76529076/44738741-db4d-468d-8ab4-7c99f2075ccc)
这里报错，这里。最后还是没有找到具体报错原因


Please try the latest version. If this error still exist, please let us know, thanks.

Feel free to re-open if needed.

> Please try the latest version. If this error still exist, please let us know, thanks.

我现在使用最新版按照教程配置，又出现了这个问题，目前执行不了任何评测任务。我尝试了官方文档给的例子：`python run.py --models hf_opt_125m hf_opt_350m --datasets siqa_gen winograd_ppl`，就会出现和 https://github.com/open-compass/opencompass/issues/436#issuecomment-1734895676 一样的错误

@Magnetic2014 我也遇到了相似的问题，请问解决了吗


> @Magnetic2014 我也遇到了相似的问题，请问解决了吗

没有……我有一个几个月前的老版本，先退回去了

Check this out, [我解决了torchrun环境异常情况下的No predictions found问题. ](https://github.com/open-compass/opencompass/discussions/1370#discussion-6984033)