Repository: open-compass/opencompass
Issue: 部分数据的子集提示报错，没有经过测试
Root cause: Validation Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda-12.1',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100 80GB PCIe',
 'MMEngine': '0.10.3',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 12.1, V12.1.66',
 'OpenCV': '4.9.0',
 'PyTorch': '2.2.1+cu121',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2022.2-Product Build 20220804 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.3.2 (Git Hash '
                              '2dc95a2ad0841e29db8b22fbccaf3e5da7992b01)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX512\n'
                              '  - CUDA Runtime 12.1\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 8.9.2\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=12.1, '
                              'CUDNN_VERSION=8.9.2, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DUSE_FBGEMM '
                              '-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-unused-function -Wno-unused-result '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=pedantic '
                              '-Wno-error=old-style-cast -Wno-missing-braces '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, TORCH_VERSION=2.2.1, '
                              'USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, '
                              'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, '
                              'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, '
                              'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]',
 'TorchVision': '0.17.1',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.2.3+',
 'sys.platform': 'linux'}

### 重现问题 - 代码/配置示例

![image](https://github.com/open-compass/opencompass/assets/11584869/c17f46de-0896-46ca-9d65-acc6569696dc)
只选择了部分数据进行测试
datasets = [*ceval_datasets,*cmmlu_datasets,*GaokaoBench_datasets]

### 重现问题 - 命令或脚本

python run.py configs/eval_qwen_7b_chat.py

### 重现问题 - 错误信息

cmmlu-conceptual_physics: {'accuracy': 59.863945578231295}
cmmlu-construction_project_management: {'accuracy': 46.76258992805755}
cmmlu-education: {'accuracy': 61.963190184049076}
cmmlu-electrical_engineering: {'accuracy': 49.41860465116279}
cmmlu-elementary_chinese: {'accuracy': 53.96825396825397}
cmmlu-elementary_information_and_technology: {'accuracy': 77.73109243697479}
cmmlu-food_science: {'accuracy': 58.04195804195804}
cmmlu-genetics: {'accuracy': 48.29545454545455}
cmmlu-high_school_mathematics: {'accuracy': 34.756097560975604}
cmmlu-high_school_politics: {'accuracy': 60.83916083916085}
cmmlu-jurisprudence: {'accuracy': 58.63746958637469}
cmmlu-logical: {'accuracy': 47.15447154471545}
cmmlu-nutrition: {'accuracy': 55.172413793103445}
cmmlu-professional_accounting: {'accuracy': 70.85714285714285}
cmmlu-professional_law: {'accuracy': 43.60189573459716}
cmmlu-professional_medicine: {'accuracy': 41.755319148936174}
cmmlu-professional_psychology: {'accuracy': 71.55172413793103}
cmmlu-public_relations: {'accuracy': 59.195402298850574}
cmmlu-sports_science: {'accuracy': 57.57575757575758}
cmmlu-world_history: {'accuracy': 68.32298136645963}
GaokaoBench_2010-2022_Math_I_MCQs: {'score': 34.112149532710276}
GaokaoBench_2010-2022_Physics_MCQs: {'score': 28.90625}
GaokaoBench_2010-2022_Chinese_Lang_and_Usage_MCQs: {'score': 32.5}
GaokaoBench_2010-2022_Math_I_Fill-in-the-Blank: {'score': 0}
GaokaoBench_2010-2022_Chemistry_Open-ended_Questions: {'score': 0}
GaokaoBench_2010-2022_Physics_Open-ended_Questions: {'score': 0}
GaokaoBench_2010-2022_Political_Science_Open-ended_Questions: {'score': 0}
GaokaoBench_2010-2022_Chinese_Language_Language_and_Writing_Skills_Open-ended_Questions: {'score': 0}
cmmlu-humanities: {'error': "missing metrics: ['cmmlu-arts', 'cmmlu-chinese_literature', 'cmmlu-college_law', 'cmmlu-global_facts', 'cmmlu-international_law', 'cmmlu-marxist_theory', 'cmmlu-philosophy', 'cmmlu-world_religions']"}
cmmlu-stem: {'error': "missing metrics: ['cmmlu-anatomy', 'cmmlu-astronomy', 'cmmlu-college_actuarial_science', 'cmmlu-college_engineering_hydrology', 'cmmlu-college_mathematics', 'cmmlu-college_medical_statistics', 'cmmlu-computer_science', 'cmmlu-elementary_mathematics', 'cmmlu-high_school_biology', 'cmmlu-high_school_chemistry', 'cmmlu-high_school_physics', 'cmmlu-machine_learning', 'cmmlu-virology']"}
cmmlu-social-science: {'error': "missing metrics: ['cmmlu-business_ethics', 'cmmlu-chinese_civil_service_exam', 'cmmlu-chinese_food_culture', 'cmmlu-chinese_foreign_policy', 'cmmlu-college_education', 'cmmlu-economics', 'cmmlu-ethnology', 'cmmlu-high_school_geography', 'cmmlu-journalism', 'cmmlu-management', 'cmmlu-marketing', 'cmmlu-modern_chinese', 'cmmlu-security_study', 'cmmlu-sociology']"}
cmmlu-other: {'error': "missing metrics: ['cmmlu-agronomy', 'cmmlu-chinese_driving_rule', 'cmmlu-college_medicine', 'cmmlu-computer_security', 'cmmlu-elementary_commonsense', 'cmmlu-human_sexuality', 'cmmlu-legal_and_moral_basis', 'cmmlu-traditional_chinese_medicine']"}
cmmlu-china-specific: {'error': "missing metrics: ['cmmlu-chinese_civil_service_exam', 'cmmlu-chinese_driving_rule', 'cmmlu-chinese_food_culture', 'cmmlu-chinese_foreign_policy', 'cmmlu-chinese_literature', 'cmmlu-elementary_commonsense', 'cmmlu-ethnology', 'cmmlu-modern_chinese', 'cmmlu-traditional_chinese_medicine']"}
cmmlu: {'error': "missing metrics: ['cmmlu-agronomy', 'cmmlu-anatomy', 'cmmlu-arts', 'cmmlu-astronomy', 'cmmlu-business_ethics', 'cmmlu-chinese_civil_service_exam', 'cmmlu-chinese_driving_rule', 'cmmlu-chinese_food_culture', 'cmmlu-chinese_foreign_policy', 'cmmlu-chinese_literature', 'cmmlu-college_actuarial_science', 'cmmlu-college_education', 'cmmlu-college_engineering_hydrology', 'cmmlu-college_law', 'cmmlu-college_mathematics', 'cmmlu-college_medical_statistics', 'cmmlu-college_medicine', 'cmmlu-computer_science', 'cmmlu-computer_security', 'cmmlu-economics', 'cmmlu-elementary_commonsense', 'cmmlu-elementary_mathematics', 'cmmlu-ethnology', 'cmmlu-global_facts', 'cmmlu-high_school_biology', 'cmmlu-high_school_chemistry', 'cmmlu-high_school_geography', 'cmmlu-high_school_physics', 'cmmlu-human_sexuality', 'cmmlu-international_law', 'cmmlu-journalism', 'cmmlu-legal_and_moral_basis', 'cmmlu-machine_learning', 'cmmlu-management', 'cmmlu-marketing', 'cmmlu-marxist_theory', 'cmmlu-modern_chinese', 'cmmlu-philosophy', 'cmmlu-security_study', 'cmmlu-sociology', 'cmmlu-traditional_chinese_medicine', 'cmmlu-virology', 'cmmlu-world_religions']"}
ceval-stem: {'error': "missing metrics: ['ceval-computer_network', 'ceval-operating_system', 'ceval-computer_architecture', 'ceval-college_programming', 'ceval-college_physics', 'ceval-college_chemistry', 'ceval-advanced_mathematics', 'ceval-probability_and_statistics', 'ceval-electrical_engineer', 'ceval-metrology_engineer', 'ceval-high_school_physics', 'ceval-high_school_chemistry', 'ceval-high_school_biology', 'ceval-middle_school_mathematics', 'ceval-middle_school_biology', 'ceval-middle_school_physics', 'ceval-middle_school_chemistry', 'ceval-veterinary_medicine']"}
ceval-social-science: {'error': "missing metrics: ['ceval-business_administration', 'ceval-marxism', 'ceval-mao_zedong_thought', 'ceval-education_science', 'ceval-high_school_politics', 'ceval-high_school_geography', 'ceval-middle_school_politics']"}
ceval-other: {'error': "missing metrics: ['ceval-sports_science', 'ceval-plant_protection', 'ceval-basic_medicine', 'ceval-clinical_medicine', 'ceval-fire_engineer', 'ceval-environmental_impact_assessment_engineer']"}
ceval-hard: {'error': "missing metrics: ['ceval-advanced_mathematics', 'ceval-probability_and_statistics', 'ceval-college_chemistry', 'ceval-college_physics', 'ceval-high_school_chemistry', 'ceval-high_school_physics']"}
ceval: {'error': "missing metrics: ['ceval-computer_network', 'ceval-operating_system', 'ceval-computer_architecture', 'ceval-college_programming', 'ceval-college_physics', 'ceval-college_chemistry', 'ceval-advanced_mathematics', 'ceval-probability_and_statistics', 'ceval-electrical_engineer', 'ceval-metrology_engineer', 'ceval-high_school_physics', 'ceval-high_school_chemistry', 'ceval-high_school_biology', 'ceval-middle_school_mathematics', 'ceval-middle_school_biology', 'ceval-middle_school_physics', 'ceval-middle_school_chemistry', 'ceval-veterinary_medicine', 'ceval-business_administration', 'ceval-marxism', 'ceval-mao_zedong_thought', 'ceval-education_science', 'ceval-high_school_politics', 'ceval-high_school_geography', 'ceval-middle_school_politics', 'ceval-modern_chinese_history', 'ceval-ideological_and_moral_cultivation', 'ceval-logic', 'ceval-law', 'ceval-chinese_language_and_literature', 'ceval-art_studies', 'ceval-professional_tour_guide', 'ceval-legal_professional', 'ceval-high_school_chinese', 'ceval-high_school_history', 'ceval-middle_school_history', 'ceval-sports_science', 'ceval-plant_protection', 'ceval-basic_medicine', 'ceval-clinical_medicine', 'ceval-fire_engineer', 'ceval-environmental_impact_assessment_engineer']"}
GaokaoBench: {'error': "missing metrics: ['GaokaoBench_2010-2022_Math_II_MCQs', 'GaokaoBench_2010-2022_History_MCQs', 'GaokaoBench_2010-2022_Biology_MCQs', 'GaokaoBench_2010-2022_Political_Science_MCQs', 'GaokaoBench_2010-2022_Chemistry_MCQs', 'GaokaoBench_2010-2013_English_MCQs', 'GaokaoBench_2010-2022_Chinese_Modern_Lit', 'GaokaoBench_2010-2022_English_Fill_in_Blanks', 'GaokaoBench_2012-2022_English_Cloze_Test', 'GaokaoBench_2010-2022_Geography_MCQs', 'GaokaoBench_2010-2022_English_Reading_Comp']"}
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

### 其他信息

_No response_

上面结果是保存在summary-**.txt文件下

麻烦看一下 logs/infer 路径下的内容，尤其是报错的对应的子集


[2024-03-23 17:17:04,808] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 964785 closing signal SIGHUP
Traceback (most recent call last):
  File "/home/miniconda3/envs/opencompass/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    result = agent.run()
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 727, in run
    result = self._invoke_run(role)
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 868, in _invoke_run
    time.sleep(monitor_interval)
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 62, in _terminate_process_handler
    raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 964588 got signal: 1

难道是多进程执行的问题？我在执行代码的时候CUDA_VISIBLE_DEVICES=1,2,3,4,5,6 python run.py configs/eval_qwen_7b_chat.py，同时修改了run.py中的--max-workers-per-gpu参数为3




应该是多进程执行有问题，尝试 `python run.py -r latest .....` 重试吧

> 我在执行代码的时候CUDA_VISIBLE_DEVICES=1,2,3,4,5,6 python run.py configs/eval_qwen_7b_chat.py，同时修改了run.py中的--max-workers-per-gpu参数为3

重新执行代码的时候python run.py configs/eval_qwen_7b_chat.py，同时修改了run.py中的--max-workers-per-gpu参数为3，报同样的错误，不知道是不是--max-workers-per-gpu参数为3影响了，继续进行测试实验
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 727, in run
    result = self._invoke_run(role)
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 868, in _invoke_run
    time.sleep(monitor_interval)
  File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 62, in _terminate_process_handler
    raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 993678 got signal: 1

> > 我在执行代码的时候CUDA_VISIBLE_DEVICES=1,2,3,4,5,6 python run.py configs/eval_qwen_7b_chat.py，同时修改了run.py中的--max-workers-per-gpu参数为3
> 
> 重新执行代码的时候python run.py configs/eval_qwen_7b_chat.py，同时修改了run.py中的--max-workers-per-gpu参数为3，报同样的错误，不知道是不是--max-workers-per-gpu参数为3影响了，继续进行测试实验 File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 727, in run result = self._invoke_run(role) File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 868, in _invoke_run time.sleep(monitor_interval) File "/home/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 62, in _terminate_process_handler raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval) torch.distributed.elastic.multiprocessing.api.SignalException: Process 993678 got signal: 1

重新执行代码的时候python run.py configs/eval_qwen_7b_chat.py，同时修改了run.py中的--max-workers-per-gpu参数为1没有报错了

Feel free to re-open if needed.

这个问题依然存在，我是用简单的推理脚本:
```bash
#! /bin/bash

MKL_SERVICE_FORCE_INTEL=1 python run.py --dataset cmmlu_ppl \
        --hf-path /root/my-internlm2-7b \
        --tokenizer-path /root/my-internlm2-7b/ \
        --tokenizer-kwargs trust_remote_code=True \
        --model-kwargs trust_remote_code=True device_map='auto' \
	--max-seq-len 2048 \
        --debug \
	--num-gpus 2 \
	--batch-size 96 
```
我花费了很多时间，重复试验了多次，发现每一次丢失的子集prediction都是一样的。解决方法就是修改batch-size，降低batch-size到64就可以正常运行了。

---

The issue persists with the following inference script commands:
```bash
#! /bin/bash

MKL_SERVICE_FORCE_INTEL=1 python run.py --dataset cmmlu_ppl \
        --hf-path /root/my-internlm2-7b \
        --tokenizer-path /root/my-internlm2-7b/ \
        --tokenizer-kwargs trust_remote_code=True \
        --model-kwargs trust_remote_code=True device_map='auto' \
        --max-seq-len 2048 \
        --debug \
        --num-gpus 2 \
        --batch-size 96 
```
After spending a considerable amount of time and trying multiple times, it was found that the same subset of predictions is lost each time. The solution was to modify the batch size; lowering the batch size to 64 allows for normal operation.



> ```shell
> --tokenizer-path
> ```

我batch=1，还是出问题