Repository: open-compass/opencompass
Issue: [Bug] F1-score file found error
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```
{'CUDA available': True,
 'GCC': 'gcc (GCC) 7.3.0',
 'MMEngine': '0.10.6',
 'MUSA available': False,
 'OpenCV': '4.11.0',
 'PyTorch': '2.5.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 10.2\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: NO AVX\n'
                              '  - Build settings: BLAS_INFO=open, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, '
                              'CXX_FLAGS=-ffunction-sections -fdata-sections '
                              '-D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=open, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=OFF, '
                              'USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.16 (main, Dec 11 2024, 16:18:56) [GCC 11.2.0]',
 'TorchVision': '0.20.1',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.9+',
 'sys.platform': 'linux',
 'transformers': '4.48.2'}
```

### Reproduces the problem - code/configuration sample

I tried to change metric from acc to f1 score
```
from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import FixKRetriever
from opencompass.openicl.icl_inferencer import PPLInferencer
# from opencompass.openicl.icl_evaluator import AccEvaluator
from opencompass.openicl.icl_evaluator import SquadEvaluator
from opencompass.datasets import FinanceIQDataset

financeIQ_subject_mapping_en = {
    'certified_public_accountant': '注册会计师（CPA）',
    'banking_qualification': '银行从业资格',
    'securities_qualification': '证券从业资格',
    'fund_qualification': '基金从业资格',
    'insurance_qualification': '保险从业资格CICE',
    'economic_analyst': '经济师',
    'taxation_practitioner': '税务师',
    'futures_qualification': '期货从业资格',
    'certified_fin_planner': '理财规划师',
    'actuary_fin_math': '精算师-金融数学',
}

financeIQ_subject_mapping = {
    '注册会计师（CPA）': '注册会计师（CPA）',
    '银行从业资格': '银行从业资格',
    '证券从业资格': '证券从业资格',
    '基金从业资格': '基金从业资格',
    '保险从业资格CICE': '保险从业资格CICE',
    '经济师': '经济师',
    '税务师': '税务师',
    '期货从业资格': '期货从业资格',
    '理财规划师': '理财规划师',
    '精算师-金融数学': '精算师-金融数学',
}

financeIQ_all_sets = list(financeIQ_subject_mapping.keys())

financeIQ_datasets = []
for _name in financeIQ_all_sets:
    _ch_name = financeIQ_subject_mapping[_name]
    financeIQ_infer_cfg = dict(
        ice_template=dict(
            type=PromptTemplate,
            template={
                answer: dict(
                    begin='</E>',
                    round=[
                        dict(
                            role='HUMAN',
                            prompt=f'以下是关于{_ch_name}的单项选择题，请直接给出正确答案的选项。\n题目：{{question}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}'
                        ),
                        dict(role='BOT', prompt=f'答案是: {answer}'),
                    ])
                for answer in ['A', 'B', 'C', 'D']
            },
            ice_token='</E>',
        ),
        retriever=dict(type=FixKRetriever, fix_id_list=[0, 1, 2, 3, 4]),
        inferencer=dict(type=PPLInferencer),
    )

    financeIQ_eval_cfg = dict(evaluator=dict(type=SquadEvaluator))

    financeIQ_datasets.append(
        dict(
            type=FinanceIQDataset,
            path='/opencompass-main/data/FinanceIQ',
            name=_name,
            abbr=f'FinanceIQ-{_name}',
            reader_cfg=dict(
                input_columns=['question', 'A', 'B', 'C', 'D'],
                output_column='answer',
                train_split='dev',
                test_split='test'),
            infer_cfg=financeIQ_infer_cfg,
            eval_cfg=financeIQ_eval_cfg,
        ))

del _name, _ch_name

```

### Reproduces the problem - command or script

```
python run.py --models hf_llama3_1_8b --datasets FinanceIQ_ppl_42b9bd --work-dir ./outputs/FinanceIQ-f1 --max-num-workers 8
```

### Reproduces the problem - error message

the infer phase works fine, but eval phase occurs an error:
```
FileNotFoundError: Local file /opencompass-main/opencompass/openicl/icl_evaluator/hf_metrics/compute_score.py doesn't exist
```
i'm wondering how to change metric in opencompass, and i checked the code and didn't find compute_score.py

### Other information

_No response_

Hi, the calculation logic for eval is in the Evaluator, which is located at the position of the config you provided: financeIQ_eval_cfg = dict(evaluator=dict(type=SquadEvaluator)). You can define the required metrics yourself in the SquadEvaluator.

> Hi, the calculation logic for eval is in the Evaluator, which is located at the position of the config you provided: financeIQ_eval_cfg = dict(evaluator=dict(type=SquadEvaluator)). You can define the required metrics yourself in the SquadEvaluator.

hi, thanks, but i wonder why there's no compute_score.py in opencompass, cause SquadEvaluator is defined by opencompass.  

can you provide a more detailed error log?

> can you provide a more detailed error log?

I want to use f1 score as metric, see official introduction it is defined in SquadEvaluator(https://opencompass.readthedocs.io/zh-cn/latest/user_guides/metrics.html), but when i use SquadEvaluator, as shown in above config py file, the error occurs:
```
Traceback (most recent call last):
  File "/opencompass-main/opencompass/tasks/openicl_eval.py", line 430, in <module>
    inferencer.run()
  File "/opencompass-main/opencompass/tasks/openicl_eval.py", line 82, in run
    self._score()
  File "/opencompass-main/opencompass/tasks/openicl_eval.py", line 218, in _score
    result = icl_evaluator.score(**preds)
  File "/opencompass-main/opencompass/openicl/icl_evaluator/icl_hf_evaluator.py", line 83, in score
    metric = evaluate.load(local_path)
  File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/evaluate/loading.py", line 748, in load
    evaluation_module = evaluation_module_factory(
  File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/evaluate/loading.py", line 618, in evaluation_module_factory
    ).get_module()
  File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/evaluate/loading.py", line 421, in get_module
    local_imports = _download_additional_modules(
  File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/evaluate/loading.py", line 248, in _download_additional_modules
    local_import_path = cached_path(
  File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/evaluate/utils/file_utils.py", line 193, in cached_path
    raise FileNotFoundError(f"Local file {url_or_filename} doesn't exist")
FileNotFoundError: Local file /opencompass-main/opencompass/openicl/icl_evaluator/hf_metrics/compute_score.py doesn't exist
```
and indeed there's no compute_score.py in opencompass

> can you provide a more detailed error log?

I've used sklearn to refine SquadEvaluator's f1 evaluation though