Repository: open-compass/opencompass
Issue: [Bug] 示例代码 opencompass ./examples/eval_simpleqa.py 跑不通
Root cause: Validation Gap

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

```
{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A800-SXM4-80GB',
 'MMEngine': '0.10.6',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 12.1, V12.1.105',
 'OpenCV': '4.11.0',
 'PyTorch': '2.5.1+cu124',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2024.2-Product Build 20240605 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX512\n'
                              '  - CUDA Runtime 12.4\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 90.1\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=12.4, '
                              'CUDNN_VERSION=9.1.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, '
                              'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, '
                              'USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, '
                              'USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, '
                              'USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.12 (main, Jul  5 2023, 18:54:27) [GCC 11.2.0]',
 'TorchVision': '0.20.1+cu124',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.4.0+',
 'sys.platform': 'linux',
 'transformers': '4.49.0.dev0'}
```

### 重现问题 - 代码/配置示例

opencompass ../opencompass/examples/eval_simpleqa.py --work-dir eval_results --mode all --dump-eval-details

# 直接执行以上

### 重现问题 - 命令或脚本

opencompass ../opencompass/examples/eval_simpleqa.py --work-dir eval_results --mode all --dump-eval-details

### 重现问题 - 错误信息

一系列问题：
1.  如下，去掉方括号后能往下 走。
![Image](https://github.com/user-attachments/assets/f8e0028d-6363-4db6-98b8-1d03311e6715)

2. 但是又遇到 eval阶段找不到 infer 的结果的问题
```
AssertionError: No predictions found for eval_results/20250220_104357/predictions/judged-by--GPT-4o-2024-05-13/simpleqa.json and eval_results/20250220_104357/predictions/judged-by--GPT-4o-2024-05-13/simpleqa_0.json
```
也就是在 eval_results/20250220_104357/predictions/ 下面没有生成该文件。而只有 eval_results/20250220_104357/predictions/GPT-4o-2024-05-13/， 且里面的infer阶段的结果是符合 预期的。

3. 把上面的文件改名成 eval_results/20250220_104357/predictions/judged-by--GPT-4o-2024-05-13/simpleqa.json  后可以往下走，不过又在下面文件处报错：

![Image](https://github.com/user-attachments/assets/1984698f-e3c0-4384-aa98-21b5634f9a11)

打印出来是(可以看到打印出来的数据的格式，和代码中要求的的，似乎是完全不一样)：
```
aaaaaaaaaaa [["To determine the age at which Ken Noda was invited by President Ronald Reagan and First Lady Nancy Reagan to perform at the White House, let's break this down step by step:\n\n1. **Ken Noda's Birth Year**: Ken Noda was born in 1962.\n   \n2. **Reagan Presidency**: Ronald Reagan served as President of the United States from January 20, 1981, to January 20, 1989.\n\n3. **Ken Noda's Performance at the White House**: It is mentioned that Ken Noda was invited by President Ronald Reagan and First Lady Nancy Reagan to perform at the White House during Reagan's presidency.\n\n4. **Ken Noda's Age During Reagan's Presidency**: During Reagan's presidency (1981–1989), Ken Noda was between 19 and 27 years old, depending on the exact year of the invitation.\n\n5. **Specific Year of the White House Performance**: Additional context specifies that the invitation occurred in 1982.\n\n6. **Ken Noda's Age in 1982**: If Ken Noda was born in 1962, then in 1982, he would have been **20 years old**.\n\n### Final Answer:\nKen Noda was **20 years old** when he was invited by President Ronald Reagan and First Lady Nancy Reagan to perform at the White House.", 'gpt-4o-2024-11-20'], ["To determine the age at which Ken Noda was invited by President Ronald Reagan and First Lady Nancy Reagan to perform at the White House, let's break this down step by step:\n\n1. **Ken Noda's Birth Year**: Ken Noda was born in 1962.\n   \n2. **Reagan Presidency**: Ronald Reagan served as President of the United States from January 20, 1981, to January 20, 1989.\n\n3. **Ken Noda's Performance at the White House**: It is mentioned that Ken Noda was invited by President Ronald Reagan and First Lady Nancy Reagan to perform at the White House during Reagan's presidency.\n\n4. **Ken Noda's Age During Reagan's Presidency**: During Reagan's presidency (1981–1989), Ken Noda was between 19 and 27 years old, depending on the exact year of the invitation.\n\n5. **Specific Year of the White House Performance**: Additional context specifies that the invitation occurred in 1982.\n\n6. **Ken Noda's Age in 1982**: If Ken Noda was born in 1962, then in 1982, he would have been **20 years old**.\n\n### Final Answer:\nKen Noda was **20 years old** when he was invited by President Ronald Reagan and First Lady Nancy Reagan to perform at the White House.", 'judged-by--gpt-4o-2024-11-20']] 0 | ['20', 'Knoedler'] 0

```

不知道是我哪里操作有问题，还是该代码有问题。

### 其他信息

_No response_

We have updated the latest version, feel free to try it