Repository: open-compass/opencompass
Issue: [Bug] 使用自定义数据集对讯飞API模型进行评测，为何得分是 0
Root cause: Documentation Deficiency

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我修改了代码（配置不视为代码），或者我正在处理我自己的任务/模型/数据集。

### 环境

{'CUDA available': False,
 'GCC': 'n/a',
 'MMEngine': '0.10.4',
 'MSVC': '用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.39.33523 版',
 'MUSA available': False,
 'OpenCV': '4.9.0',
 'PyTorch': '2.3.0',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - C++ Version: 201703\n'
                              '  - MSVC 192930151\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2021.4-Product Build 20210904 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.3.6 (Git Hash '
                              '86e6af5974177e513fd3fee58425e1063e7f1361)\n'
                              '  - OpenMP 2019\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, '
                              'CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc '
                              '/Zc:__cplusplus /bigobj /FS /utf-8 '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 '
                              '/wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 '
                              '/wd4804 /wd4273, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, TORCH_VERSION=2.3.0, '
                              'USE_CUDA=0, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, '
                              'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, '
                              'USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.14 | packaged by Anaconda, Inc. | (main, May  6 2024, '
           '19:44:50) [MSC v.1916 64 bit (AMD64)]',
 'TorchVision': '0.18.0',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.2.4+862044f',
 'sys.platform': 'win32'}

### 重现问题 - 代码/配置示例

test_question.jsonl

{"question": "752+361+181+933+235+986=", "answer": "3448"}
{"question": "712+165+223+711=", "answer": "1811"}
{"question": "921+975+888+539=", "answer": "3323"}
{"question": "752+321+388+643+568+982+468+397=", "answer": "4519"}

### 重现问题 - 命令或脚本

python run.py --models eval_api_xunfei --custom-dataset-path data/define_dataset/test_question.jsonl

### 重现问题 - 错误信息

dataset        version    metric    mode      XunFei
-------------  ---------  --------  ------  --------
test_question  359d58     accuracy  gen            0

但是在outputs\default\20240510_164659\predictions目录中查看，模型已经输出了正确答案，为何得分还是 0
{
    "0": {
        "origin_prompt": "752+361+181+933+235+986=",
        "prediction": "解：通过观察，此题可运用加法交换律与结合律简算。\n\n$752+361+181+933+235+986$,\n\n$=(752+986)+(361+235)+(181+933)$,\n\n$=1738+696+1114$,\n\n$=3538$。",
        "gold": 3448
    },
    "1": {
        "origin_prompt": "712+165+223+711=",
        "prediction": "解：通过观察，此题可运用加法交换律与结合律简算。\n\n$712+165+223+711$,\n\n$=(712+711)+(165+223)$,\n\n$=1423+388$,\n\n$=1811$。",
        "gold": 1811
    },
    "2": {
        "origin_prompt": "921+975+888+539=",
        "prediction": "解：通过观察，此题可运用加法交换律与结合律简算。\n\n$921+975+888+539$,\n\n$=(921+888)+(975+539)$,\n\n$=1809+1514$,\n\n$=3323$。",
        "gold": 3323
    },
    "3": {
        "origin_prompt": "752+321+388+643+568+982+468+397=",
        "prediction": "解：通过观察，此题可运用加法交换律与结合律简算。\n\n$752+321+388+643+568+982+468+397$,\n\n$=(752+397)+(321+643)+(388+568)+(982+468)$,\n\n$=1150+1000+956+1450$,\n\n$=4506$。",
        "gold": 4519
    }
}

### 其他信息

_No response_

Existing evaluation post-processing may not correctly extract the answers from your predictions. You can write a specialized post-processing to extract this response format, and put this in your dataset config.
An example:
```python
...
def your_postprocessor(text: str) -> str:
    ...
    return extracted_answer # In this case, the extracted answer is a number like 1811 ( Note that it needs to be the same data type as gold).

your_dataset_eval_cfg = dict(
    evaluator=dict(type=AccEvaluator),
    pred_role="BOT",
    pred_postprocessor=dict(type=your_postprocess),
)
...
```

Thank you very much。I am gradually getting familiar with this platform.



how to write a specialized post-processing?