Repository: open-compass/opencompass
Issue: [Bug] Medbench dataset 
Root cause: Validation Gap

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```
 'GCC': 'gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)',
 'MMEngine': '0.10.2',
 'OpenCV': '4.9.0',
 'PyTorch': '2.0.0',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v2.7.3 (Git Hash '
                              '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=11.7, '
                              'CUDNN_VERSION=8.5.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -Wno-deprecated '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_QNNPACK '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wunused-local-typedefs '
                              '-Wno-unused-parameter -Wno-unused-function '
                              '-Wno-unused-result -Wno-strict-overflow '
                              '-Wno-strict-aliasing '
                              '-Wno-error=deprecated-declarations '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=redundant-decls '
                              '-Wno-error=old-style-cast '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]',
 'TorchVision': '0.15.0',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.2.1+',
 'sys.platform': 'linux'}
```

### Reproduces the problem - code/configuration sample

``` python
from copy import deepcopy
from mmengine.config import read_base

with read_base():
    # from .datasets.mmlu.mmlu_gen_a484b3 import mmlu_datasets
    # from .datasets.agieval.agieval_gen_64afd3 import agieval_datasets
    # from .datasets.bbh.bbh_gen_5b92b0 import bbh_datasets
    # from .datasets.gsm8k.gsm8k_gen_1d7fe4 import gsm8k_datasets
    # from .datasets.math.math_evaluatorv2_gen_265cce import math_datasets
    # from .datasets.humaneval.humaneval_gen_8e312c import humaneval_datasets
    # from .datasets.mbpp.sanitized_mbpp_gen_1e1056 import sanitized_mbpp_datasets
    from .datasets.MedBench.medbench_gen_0b4fff import medbench_datasets

    from .models.hf_internlm.hf_internlm2_chat_7b import models as hf_internlm2_chat_7b_model
    # from .models.hf_internlm.hf_internlm2_chat_20b import models as hf_internlm2_chat_20b_model

    from .summarizers.internlm2_keyset import summarizer

work_dir = './outputs/internlm2-chat-keyset/'

_origin_datasets = sum([v for k, v in locals().items() if k.endswith("_datasets")], [])
_origin_models = sum([v for k, v in locals().items() if k.endswith("_model")], [])

_vanilla_datasets = [deepcopy(d) for d in _origin_datasets]
_vanilla_models = []
for m in _origin_models:
    m = deepcopy(m)
    if 'meta_template' in m and 'round' in m['meta_template']:
        round = m['meta_template']['round']
        if any(r['role'] == 'SYSTEM' for r in round):
            new_round = [r for r in round if r['role'] != 'SYSTEM']
            print(f'WARNING: remove SYSTEM round in meta_template for {m.get("abbr", None)}')
            m['meta_template']['round'] = new_round
    _vanilla_models.append(m)


datasets = _vanilla_datasets
models = _vanilla_models

```

### Reproduces the problem - command or script

``` shell
srun -p gmai  -quotatype=spot --cpus-per-task=16 --gres=gpu:1 /mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/bin/python  run.py configs/eval_internlm2_chat_keyset.py  --debug     
```
怀疑是medbench 代码里的openicl_eval写的有问题，我能看到模型已经成功生成了prediction，但是最后 eval 的时候出问题。

### Reproduces the problem - error message

``` shell                                                                                    
01/26 03:33:04 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:33:04 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 595, in score                                                                                                                                                                        
    return calc_scores_nlg(predictions, references)                                                                                          
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 533, in calc_scores_nlg                                                                                                                                                              
    return calc_nlg_task_scores(gts, preds)                                                                                                  
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 467, in calc_nlg_task_scores                                                                                                                                                         
    scores = rouge.get_scores(predictions, references, avg=True)                                                                             
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 117, in get_scores                                                                                                                                                  
    return self._get_avg_scores(hyps, refs)                                                                                                  
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 165, in _get_avg_scores                                                                                                                                             
    avg_scores = {                                                                                                                           
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 166, in <dictcomp>                                                                                                                                                  
    m: {s: scores[m][s] / count for s in self.stats}                                                                                         
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 166, in <dictcomp>                                                                                                                                                  
    m: {s: scores[m][s] / count for s in self.stats}                                                                                         
ZeroDivisionError: division by zero                                                                                                          
01/26 03:33:26 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:33:26 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 595, in score                                                                                                                                                                        
    return calc_scores_nlg(predictions, references)                                                                                          
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 533, in calc_scores_nlg                                                                                                                                                              
    return calc_nlg_task_scores(gts, preds)                                                                                                  
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 467, in calc_nlg_task_scores                                                                                                                                                         
    scores = rouge.get_scores(predictions, references, avg=True)                                                                             
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 117, in get_scores                                                                                                                                                  
    return self._get_avg_scores(hyps, refs)                                                                                                  
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 165, in _get_avg_scores                                                                                                                                             
    avg_scores = {                                                                                                                           
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 166, in <dictcomp>                                                                                                                                                  
    m: {s: scores[m][s] / count for s in self.stats}                                                                                         
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 166, in <dictcomp>                                                                                                                                                  
    m: {s: scores[m][s] / count for s in self.stats}                                                                                         
ZeroDivisionError: division by zero                                                                                                          
01/26 03:33:48 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:33:48 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 595, in score                                                                                                                                                                        
    return calc_scores_nlg(predictions, references)                                                                                          
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 533, in calc_scores_nlg                                                                                                                                                              
    return calc_nlg_task_scores(gts, preds)                                                                                                  
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 467, in calc_nlg_task_scores                                                                                                                                                         
    scores = rouge.get_scores(predictions, references, avg=True)                                                                             
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 117, in get_scores                                                                                                                                                  
    return self._get_avg_scores(hyps, refs)                                                                                                  
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 165, in _get_avg_scores                                                                                                                                             
    avg_scores = {                                                                                                                           
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 166, in <dictcomp>                                                                                                                                                  
    m: {s: scores[m][s] / count for s in self.stats}                                                                                         
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 166, in <dictcomp>                                                                                                                                                  
    m: {s: scores[m][s] / count for s in self.stats}                                                                                         
ZeroDivisionError: division by zero                                                                                                          
01/26 03:34:10 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:34:10 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 595, in score                                                                                                                                                                        
    return calc_scores_nlg(predictions, references)                                                                                          
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 533, in calc_scores_nlg                                                                                                                                                              
    return calc_nlg_task_scores(gts, preds)                                                                                                  
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 467, in calc_nlg_task_scores                                                                                                                                                         
    scores = rouge.get_scores(predictions, references, avg=True)                                                                             
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 117, in get_scores                                                                                                                                                  
    return self._get_avg_scores(hyps, refs)                                                                                                  
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 165, in _get_avg_scores                                                                                                                                             
    avg_scores = {                                                                                                                           
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 166, in <dictcomp>                                                                                                                                                  
    m: {s: scores[m][s] / count for s in self.stats}                                                                                         
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/rouge_chinese/rouge.py", line 166, in <dictcomp>                                                                                                                                                  
    m: {s: scores[m][s] / count for s in self.stats}                                                                                         
ZeroDivisionError: division by zero                                                                                                          
01/26 03:34:39 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:34:39 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 624, in score                                                                                                                                                                        
    if set(current_pred) == set(ref):                                                                                                        
TypeError: 'NoneType' object is not iterable                   
01/26 03:35:07 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:35:07 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval                                                                                                                         
Traceback (most recent call last):                                                                                                                                                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                                                                                                                                                                       
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 547, in score                                                                                                                                                                        
    return calc_scores_f1(predictions, references)                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 489, in calc_scores_f1                                                                                                                                                               
    precision, recall, f1 = calc_info_extract_task_scores(dict_gt, dict_pred)                                                                                                                                                                                                              
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 356, in calc_info_extract_task_scores                                                                                                                                                
    for inst in answer_golden:                                                                                                               
TypeError: 'NoneType' object is not iterable                                                                                                 
01/26 03:35:43 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:35:43 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 540, in score                                                                                                                                                                        
    return calc_scores_f1(predictions, references)                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 489, in calc_scores_f1                                                                                                                                                               
    precision, recall, f1 = calc_info_extract_task_scores(dict_gt, dict_pred)                                                                                                                                                                                                              
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 356, in calc_info_extract_task_scores                                                                                                                                                
    for inst in answer_golden:                                                                                                               
TypeError: 'NoneType' object is not iterable                                                                                                 
01/26 03:36:07 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:36:07 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 561, in score                                                                                                                                                                        
    return calc_scores_f1(predictions, references)                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 489, in calc_scores_f1                                                                                                                                                               
    precision, recall, f1 = calc_info_extract_task_scores(dict_gt, dict_pred)                                                                                                                                                                                                              
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 356, in calc_info_extract_task_scores                                                                                                                                                
    for inst in answer_golden:                                                                                                               
TypeError: 'NoneType' object is not iterable                                                                                                 
01/26 03:36:28 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:36:28 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 568, in score                                                                                                                                                                        
    return calc_scores_f1(predictions, references)                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 489, in calc_scores_f1                                                                                                                                                               
    precision, recall, f1 = calc_info_extract_task_scores(dict_gt, dict_pred)                                                                                                                                                                                                              
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 356, in calc_info_extract_task_scores                                                                                                                                                
    for inst in answer_golden:                                                                                                               
TypeError: 'NoneType' object is not iterable                                                                                                 
01/26 03:36:50 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:36:50 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 575, in score                                                                                                                                                                        
    return calc_scores_f1(predictions, references)                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 489, in calc_scores_f1                                                                                                                                                               
    precision, recall, f1 = calc_info_extract_task_scores(dict_gt, dict_pred)                                                                                                                                                                                                              
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 356, in calc_info_extract_task_scores                                                                                                                                                
    for inst in answer_golden:                                                                                                               
TypeError: 'NoneType' object is not iterable                                                                                                 
01/26 03:37:13 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:37:13 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)      
File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 582, in score                                                                                                                                                                        
    return calc_scores_ctc(predictions, references)                                                                                          
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 500, in calc_scores_ctc                                                                                                                                                              
    precision, recall, f1 = calc_cls_task_scores(                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 419, in calc_cls_task_scores                                                                                                                                                         
    cls_report = classification_report(                                                                                                      
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/sklearn/metrics/_classification.py", line 2310, in classification_report
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)                                                                                  
  File "/mnt/petrelfs/wangguoan/miniconda3/envs/opencompass/lib/python3.10/site-packages/sklearn/metrics/_classification.py", line 95, in _check_targets
    raise ValueError(                                                                                                                        
ValueError: Classification metrics can't handle a mix of multiclass and unknown targets                                                                                                                                                                                                    
01/26 03:37:36 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:37:36 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 589, in score                                                                                                                                                                        
    return calc_scores_f1(predictions, references)                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 489, in calc_scores_f1                                                                                                                                                               
    precision, recall, f1 = calc_info_extract_task_scores(dict_gt, dict_pred)                                                                                                                                                                                                              
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 356, in calc_info_extract_task_scores                                                                                                                                                
    for inst in answer_golden:                                                                                                               
TypeError: 'NoneType' object is not iterable                                                                                                 
01/26 03:38:00 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"                                                                                                                                                                                   
01/26 03:38:00 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
Traceback (most recent call last):                                                                                                           
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 361, in <module>                                                                                                                                                                             
    inferencer.run()                                                                                                                         
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 107, in run                                                                                                                                                                                  
    self._score()                                                                                                                            
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/tasks/openicl_eval.py", line 216, in _score                                                                                                                                                                               
    result = icl_evaluator.score(**preds)                                                                                                    
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 553, in score                                                                                                                                                                        
    predictions = process_generated_results_mrg(predictions)                                                                                 
  File "/mnt/petrelfs/wangguoan/git_repo/opencompass/opencompass/datasets/medbench/medbench.py", line 326, in process_generated_results_mrg                                                                                                                                                
    if '\n\n' in pred['answer']:                                                                                                             
TypeError: string indices must be integers                                                                                                                                                                                                      
```

### Other information

_No response_

@xmshi-trio Hi, please check this issue

![image](https://github.com/open-compass/opencompass/assets/55678087/f2141fa3-0f34-4f4a-9cbf-2da91a0e3b53)
![image](https://github.com/open-compass/opencompass/assets/55678087/6b1e2949-f27d-4406-9f04-a18ce88952bc)
``` json

    "0": {
        "origin_prompt": "
user\n给定病历或者医学影像报告，要求从中抽取临床发现事件的四个属性:主体词、解剖部位、描述词、发生状态。\n主体词：指患者的电子病历中的疾病名称或者由疾病引发的症状，也包括患者的一般情况如饮食，二便，睡眠等。\n描述词：对主体词的发生时序特征、轻重程度、形态颜色等多个维度的刻画，也包括疾病的起病缓急、突发。\n解剖部位：指主体词发生在患者的身体部位，也包括组织，细胞，系统等，也包括部位的方向和数量。\n发生状态：“不确定”或“否定”，肯定的情况不标注发生状态。\n\n\n\n要求输出所有的临床发生事件，每个临床发现事件占一行，临床发现事件中包含主体词、解剖部位、描述词、发生状态四个字段，每个字段用“；”分隔，然后每个字段是“字段名：字段值”的格式，字段值有多个，则用“，”符号分隔。。主体词是字符串。发生状态是字符串，如果没有发生状态，默认为\"不确定\"。如果没有描述词，默认为\"空\"。如果没有解剖部位，默认为\"空\"。\n示例：\n输入病历或者医学影像所见报告：\n精神好，食欲好。\n输出这个句子里的临床发现事件为：\n\n主体词：精神；发生状态：不确定；描述词：好；解剖部位：空。\n主体词：食欲；发生状态：不确定；描述词：好；解剖部位：空。\n\n请从以下病历或者医学影像所见报告中，直接抽取临床发现事件的四个属性:主体词、解剖部位、描述词、发生状态，不要进行额外的解释：患者本次发病以来，食欲正常，神志清醒，精神尚可，睡眠欠佳，大便正常，小便正常，体重无明显变化。\n答：
\n
assistant\n",
        "prediction": "以下是根据您提供的病历或医学影像报告中抽取的临床发现事件及其四个属性：\n\n1. **主体词**：食欲\n   - **解剖部位**：空\n   - **描述词**：正常\n   - **发生状态**：不确定\n\n2. **主体词**：神志\n   - **解剖部位**：空\n   - **描述词**：清醒\n   - **发生状态**：不确定\n\n3. **主体词**"
    },
```
这是中间结果

![image](https://github.com/open-compass/opencompass/assets/55678087/b355c0f8-1a3d-48fc-afa7-fda8dedb3b2f)
![image](https://github.com/open-compass/opencompass/assets/55678087/2cbbb041-546b-4b58-93a4-a97ecaed950e)
是否 gt 和 reference 参数放反了？怀疑是 Medbench 的 eval 方法写错了，或者数据有问题？我下载的数据是name_test.json，code base 是 name.json，我的没有 answer，不过我看官网确实也没有 answer。还有不是很明白，我的数据中没有problem_input列 
``` json
{"question": "请回答以下单选题。要求只输出选项，不输出解释：\n儿童可以使用八子补肾胶囊吗？\n(A)是\n(B)否。\n答：", "options": ["(A)是", "(B)否"], "answer": null, "other": {"source": "DrugCA", "id": 1}}
{"question": "请回答以下单选题。要求只输出选项，不输出解释：\n儿童可以使用八子补肾胶囊吗？\n(A)否\n(B)是。\n答：", "options": ["(A)否", "(B)是"], "answer": null, "other": {"source": "DrugCA", "id": 1}}
```
``` python
medbench_reader_cfg = dict(
    input_columns=['problem_input'], output_column='label')
```
但是代码中说读取这个列

放出来的测试数据的答案为空吧。

> ![image](https://private-user-images.githubusercontent.com/55678087/299892777-b355c0f8-1a3d-48fc-afa7-fda8dedb3b2f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDYyNTcwMDksIm5iZiI6MTcwNjI1NjcwOSwicGF0aCI6Ii81NTY3ODA4Ny8yOTk4OTI3NzctYjM1NWMwZjgtMWEzZC00OGZjLWFmYTctZmRhOGRlZGIzYjJmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTI2VDA4MTE0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcwODk5MjNhYjFhZWNiODdmOTFmMmQ2YmVmNTY0ZDFlNWJkMzc4MzU2NzA3NGIyNzA0MGQ0YzI3OGVmOGZmNjkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.LPyx5gxCOCdKQiuBrIc_vKKhmwm1hnjWDMZNv5XI8IE) ![image](https://private-user-images.githubusercontent.com/55678087/299892828-2cbbb041-546b-4b58-93a4-a97ecaed950e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDYyNTcwMDksIm5iZiI6MTcwNjI1NjcwOSwicGF0aCI6Ii81NTY3ODA4Ny8yOTk4OTI4MjgtMmNiYmIwNDEtNTQ2Yi00YjU4LTkzYTQtYTk3ZWNhZWQ5NTBlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTI2VDA4MTE0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEwZDE1NmFkMzI4YjllMDE0MDgxYWVjYWE5NzA5MmE1Y2MwMWIyZTNiZDc5ODVlOGQxMTQ2OTQ1Mzc0NjY4MzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.f_i8Vrm9XWlMqzSzlH9i5KQRp4X2UYBiisIL_HGunZY) 是否 gt 和 reference 参数放反了？怀疑是 Medbench 的 eval 方法写错了，或者数据有问题？我下载的数据是name_test.json，code base 是 name.json，我的没有 answer，不过我看官网确实也没有 answer。还有不是很明白，我的数据中没有problem_input列
> 
> ```json
> {"question": "请回答以下单选题。要求只输出选项，不输出解释：\n儿童可以使用八子补肾胶囊吗？\n(A)是\n(B)否。\n答：", "options": ["(A)是", "(B)否"], "answer": null, "other": {"source": "DrugCA", "id": 1}}
> {"question": "请回答以下单选题。要求只输出选项，不输出解释：\n儿童可以使用八子补肾胶囊吗？\n(A)否\n(B)是。\n答：", "options": ["(A)否", "(B)是"], "answer": null, "other": {"source": "DrugCA", "id": 1}}
> ```
> 
> ```python
> medbench_reader_cfg = dict(
>     input_columns=['problem_input'], output_column='label')
> ```
> 
> 但是代码中说读取这个列

您好，测试数据的答案是不对外放的。后续如果我们释放带答案的开发集才能在本地进行评测。如果想进行评测，可以本地进行推理，然后通过https://medbench.opencompass.org.cn/home提交结果进行评测。

> > ![image](https://private-user-images.githubusercontent.com/55678087/299892777-b355c0f8-1a3d-48fc-afa7-fda8dedb3b2f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDYyNTcwMDksIm5iZiI6MTcwNjI1NjcwOSwicGF0aCI6Ii81NTY3ODA4Ny8yOTk4OTI3NzctYjM1NWMwZjgtMWEzZC00OGZjLWFmYTctZmRhOGRlZGIzYjJmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTI2VDA4MTE0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcwODk5MjNhYjFhZWNiODdmOTFmMmQ2YmVmNTY0ZDFlNWJkMzc4MzU2NzA3NGIyNzA0MGQ0YzI3OGVmOGZmNjkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.LPyx5gxCOCdKQiuBrIc_vKKhmwm1hnjWDMZNv5XI8IE) ![image](https://private-user-images.githubusercontent.com/55678087/299892828-2cbbb041-546b-4b58-93a4-a97ecaed950e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDYyNTcwMDksIm5iZiI6MTcwNjI1NjcwOSwicGF0aCI6Ii81NTY3ODA4Ny8yOTk4OTI4MjgtMmNiYmIwNDEtNTQ2Yi00YjU4LTkzYTQtYTk3ZWNhZWQ5NTBlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTI2VDA4MTE0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEwZDE1NmFkMzI4YjllMDE0MDgxYWVjYWE5NzA5MmE1Y2MwMWIyZTNiZDc5ODVlOGQxMTQ2OTQ1Mzc0NjY4MzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.f_i8Vrm9XWlMqzSzlH9i5KQRp4X2UYBiisIL_HGunZY) 是否 gt 和 reference 参数放反了？怀疑是 Medbench 的 eval 方法写错了，或者数据有问题？我下载的数据是name_test.json，code base 是 name.json，我的没有 answer，不过我看官网确实也没有 answer。还有不是很明白，我的数据中没有problem_input列
> > ```json
> > {"question": "请回答以下单选题。要求只输出选项，不输出解释：\n儿童可以使用八子补肾胶囊吗？\n(A)是\n(B)否。\n答：", "options": ["(A)是", "(B)否"], "answer": null, "other": {"source": "DrugCA", "id": 1}}
> > {"question": "请回答以下单选题。要求只输出选项，不输出解释：\n儿童可以使用八子补肾胶囊吗？\n(A)否\n(B)是。\n答：", "options": ["(A)否", "(B)是"], "answer": null, "other": {"source": "DrugCA", "id": 1}}
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > ```python
> > medbench_reader_cfg = dict(
> >     input_columns=['problem_input'], output_column='label')
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > 但是代码中说读取这个列
> 
> 您好，测试数据的答案是不对外放的。后续如果我们释放带答案的开发集才能在本地进行评测。如果想进行评测，可以本地进行推理，然后通过[https://medbench.opencompass.org.cn/home提交结果进行评测。](https://medbench.opencompass.org.cn/home%E6%8F%90%E4%BA%A4%E7%BB%93%E6%9E%9C%E8%BF%9B%E8%A1%8C%E8%AF%84%E6%B5%8B%E3%80%82)

感谢。Opencompass 真挺好用的。Salute！