Repository: open-compass/opencompass
Issue: [Bug] TruthfulQA 评测报错
Root cause: External Dependency Breakage

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

S1集群上
{'CUDA available': False,
 'GCC': 'gcc (GCC) 8.5.0',
 'MMEngine': '0.8.4',
 'OpenCV': '4.8.0',
 'PyTorch': '2.0.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v2.7.3 (Git Hash '
                              '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
                              'CUDNN_VERSION=8.7.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -Wno-deprecated '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_QNNPACK '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wunused-local-typedefs '
                              '-Wno-unused-parameter -Wno-unused-function '
                              '-Wno-unused-result -Wno-strict-overflow '
                              '-Wno-strict-aliasing '
                              '-Wno-error=deprecated-declarations '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=redundant-decls '
                              '-Wno-error=old-style-cast '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]',
 'TorchVision': '0.15.2',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.1.4+545d50a',
 'sys.platform': 'linux'}

### Reproduces the problem - code/configuration sample

from opencompass.models import HuggingFaceCausalLM
from mmengine.config import read_base


with read_base():

    from .datasets.mmlu.mmlu_gen_a484b3 import mmlu_datasets    # check for ppl         FileNotFoundError: [Errno 2] No such file or directory: './data/mmlu/dev/college_biology_dev.csv




models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='llama-2-13b-hf-100-15e',
        path="/mnt/petrelfs/wangxiao/model_path",
        tokenizer_path='/mnt/petrelfs/wangxiao/model_path',
        tokenizer_kwargs=dict(padding_side='left',
                              truncation_side='left',
                              use_fast=False,
                              ),
        max_out_len=100,
        max_seq_len=2048,
        batch_size=8,
        model_kwargs=dict(device_map='auto'),
        batch_padding=False, # if false, inference with for-loop without batch padding
        run_cfg=dict(num_gpus=2, num_procs=1),
    )
]


datasets = [*mmlu_datasets]


### Reproduces the problem - command or script

nohup /mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/bin/python run.py configs/eval_test_trained.py  --slurm -p xxx -w outputs/13b-debug > 1debug.log 2>&1 &

### Reproduces the problem - error message

nohup: ignoring input
2023-09-16 17:30:09.346137: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-16 17:30:09.739353: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-09-16 17:30:12.439253: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/cv2/../../lib64:/mnt/lustre/share/gcc/mpfr-2.4.2/lib:/mnt/lustre/share/gcc/mpc-0.8.1/lib:/mnt/lustre/share/gcc/gmp-4.3.2/lib:/mnt/lustre/share/gcc/gcc-8.5.0/lib64:/mnt/lustre/share/gcc/mpfr-2.4.2/lib:/mnt/lustre/share/gcc/mpc-0.8.1/lib:/mnt/lustre/share/gcc/gmp-4.3.2/lib:/mnt/lustre/share/gcc/gcc-8.5.0/lib64::/mnt/lustre/share/cuda-11.8/lib64:/mnt/lustre/share/cuda-11.8/lib64
2023-09-16 17:30:12.440868: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/cv2/../../lib64:/mnt/lustre/share/gcc/mpfr-2.4.2/lib:/mnt/lustre/share/gcc/mpc-0.8.1/lib:/mnt/lustre/share/gcc/gmp-4.3.2/lib:/mnt/lustre/share/gcc/gcc-8.5.0/lib64:/mnt/lustre/share/gcc/mpfr-2.4.2/lib:/mnt/lustre/share/gcc/mpc-0.8.1/lib:/mnt/lustre/share/gcc/gmp-4.3.2/lib:/mnt/lustre/share/gcc/gcc-8.5.0/lib64::/mnt/lustre/share/cuda-11.8/lib64:/mnt/lustre/share/cuda-11.8/lib64
2023-09-16 17:30:12.440893: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
  warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')

Traceback (most recent call last):
  File "/mnt/petrelfs/wangxiao/opencompass/run.py", line 311, in <module>
    main()
  File "/mnt/petrelfs/wangxiao/opencompass/run.py", line 265, in main
    tasks = partitioner(cfg)
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/partitioners/base.py", line 48, in __call__
    tasks = self.partition(models, datasets, work_dir, self.out_dir)
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/partitioners/size.py", line 67, in partition
    datasets = sorted(datasets,
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/partitioners/size.py", line 68, in <lambda>
    key=lambda x: self.get_cost(x),
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/partitioners/size.py", line 188, in get_cost
    dataset = build_dataset_from_cfg(dataset)
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
    return LOAD_DATASET.build(dataset_cfg)
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/datasets/base.py", line 12, in __init__
    self.dataset = self.load(**kwargs)
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/datasets/mmlu.py", line 30, in load
    with open(filename, encoding='utf-8') as f:
FileNotFoundError: [Errno 2] No such file or directory: './data/mmlu/dev/college_biology_dev.csv'


### Other information

我在S1集群上跑（已经挂了连网代理），报错了，这个    opencompass/./data/ 的路径很奇怪，我检查了不存在这个路径。
为什么有的数据集可以跑通，有的就跑不通？
感觉不像是网络的问题

https://github.com/open-compass/opencompass/blob/main/README_zh-CN.md#%EF%B8%8F-%E5%AE%89%E8%A3%85
```
# 下载数据集到 data/ 处
wget https://github.com/open-compass/opencompass/releases/download/0.1.1/OpenCompassData.zip
unzip OpenCompassData.zip
```
@BeyonderXX 你有下载这个数据集吗？`./data/mmlu/dev/college_biology_dev.csv` 这个文件在这个数据集里面。



谢谢你！ 现在下载完之后可以正常跑了。

另外，请教，我看compass的代码里是写了 TruthfulQA的，但是我在刚才下的data里没有发现这个数据集。
我把它加入到评测里，运行它也不报错。

只是在结果里写的是  - - - 

这个是因为使用方式不对吗？@amulil

> 谢谢你！ 现在下载完之后可以正常跑了。
> 
> 另外，请教，我看compass的代码里是写了 TruthfulQA的，但是我在刚才下的data里没有发现这个数据集。 我把它加入到评测里，运行它也不报错。
> 
> 只是在结果里写的是 - - -
> 
> 这个是因为使用方式不对吗？@amulil

我也没跑通这个，得问下 @tonysy ，为啥 TruthfulQA 没有在官方提供的下载数据集里

Truthful QA 应该是 eval 代码的bug，我在 predictions 文件夹下是能找到这个 truthful_qa.json 的，但是在 eval 的日志文件中报错了，日志如下：
```
Traceback (most recent call last):
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/tasks/openicl_eval.py", line 218, in <module>
    inferencer.run()
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/tasks/openicl_eval.py", line 68, in run
    self._score()
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/tasks/openicl_eval.py", line 158, in _score
    icl_evaluator = ICL_EVALUATORS.build(self.eval_cfg['evaluator'])
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
TypeError: TruthfulQAEvaluator.__init__() missing 2 required positional arguments: 'truth_model' and 'info_model'
srun: error: SH-IDC1-10-140-0-230: task 0: Exited with exit code 1
```
辛苦看一下，这个数据集在评价LLM的safety时候，是个普遍使用的数据集。
@tonysy 

@BeyonderXX  TruthfulQA support two types of evaluation
1. api model eval, you should provide the finetuned openai model with `truth_model` and `info_model` and OPENAI_API_KEY
2. normal metric eval, please set `truth_model` and  `info_model`  to '' and `metrics=('bleurt', 'rouge', 'bleu')`
more details can refer to https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/truthfulqa.py#L37

> @BeyonderXX TruthfulQA support two types of evaluation
> 
> 1. api model eval, you should provide the finetuned openai model with `truth_model` and `info_model` and OPENAI_API_KEY
> 2. normal metric eval, please set `truth_model` and  `info_model`  to '' and `metrics=('bleurt', 'rouge', 'bleu')`
>    more details can refer to https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/truthfulqa.py#L37

别的数据集都可以这样的方式用，但是TruthfulQA就报错了。
`    from .datasets.truthfulqa.truthfulqa_gen_5ddc62 import truthfulqa_datasets  
`

我看了代码文档，还是没搞懂 truth_model 和 info_model 要怎么弄。
如果要改的话，怎么改代码呢？能有示例不？

感谢维护组的回复，如果能让TruthfulQA的评测过程对用户透明，使用者没有使用成本，一键化调用，就更好了。（虽然open-compass别的数据集就是这么干的）

@yingfhu 这个还有进一步说明吗？给个代码示例就行

@BeyonderXX 感谢您的建议，因为这个数据集评测的特殊性，如果要用truth_model 和 info_model评测，需要自己用openai的接口训练这些模型，可以参考官方 https://github.com/sylinrl/TruthfulQA/tree/main  ，这些部分在docstring已有提到。 
如果不方便走openai，也可以支持传统的metric测试，在config配置`metrics=('bleurt', 'rouge', 'bleu')` 即可。

有什么其他建议也欢迎提出，谢谢～

> @BeyonderXX 感谢您的建议，因为这个数据集评测的特殊性，如果要用truth_model 和 info_model评测，需要自己用openai的接口训练这些模型，可以参考官方 https://github.com/sylinrl/TruthfulQA/tree/main ，这些部分在docstring已有提到。 如果不方便走openai，也可以支持传统的metric测试，在config配置`metrics=('bleurt', 'rouge', 'bleu')` 即可。
> 
> 有什么其他建议也欢迎提出，谢谢～

看完原始论文，理解了你的意思。
对于'truth', 'info' 这两个指标确实需要 OpenAI 的 key，能够理解。
但是TruthfulQA 里还有两个关键指标  mc1, mc2， 咱们的代码并没有支持，后续有更新的计划不？

> > @BeyonderXX 感谢您的建议，因为这个数据集评测的特殊性，如果要用truth_model 和 info_model评测，需要自己用openai的接口训练这些模型，可以参考官方 https://github.com/sylinrl/TruthfulQA/tree/main ，这些部分在docstring已有提到。 如果不方便走openai，也可以支持传统的metric测试，在config配置`metrics=('bleurt', 'rouge', 'bleu')` 即可。
> > 有什么其他建议也欢迎提出，谢谢～
> 
> 看完原始论文，理解了你的意思。 对于'truth', 'info' 这两个指标确实需要 OpenAI 的 key，能够理解。 但是TruthfulQA 里还有两个关键指标 mc1, mc2， 咱们的代码并没有支持，后续有更新的计划不？

mc是以多选题的方式来评测TruthfulQA，我们目前在truthful这块考量的更多的其generation的能力。感谢您的建议，这块会考虑加入backlog中。

我自己在加mc1和mc2的时候有个问题：对于多选题模式的评估，当选项不确定时该怎么写config。比如输入是 question, [choices], answer_index. 此时infer_cfg 里的 prompt_template 应该怎么做？哪里有关于这点的说明文档可以参考吗？

> 我自己在加mc1和mc2的时候有个问题：对于多选题模式的评估，当选项不确定时该怎么写config。比如输入是 question, [choices], answer_index. 此时infer_cfg 里的 prompt_template 应该怎么做？哪里有关于这点的说明文档可以参考吗？

目前支持gen和ppl两种形式做多选题，可以参考https://opencompass.readthedocs.io/zh_CN/latest/prompt/prompt_template.html#prompt-inferencer  这个文档

> > 我自己在加mc1和mc2的时候有个问题：对于多选题模式的评估，当选项不确定时该怎么写config。比如输入是 question, [choices], answer_index. 此时infer_cfg 里的 prompt_template 应该怎么做？哪里有关于这点的说明文档可以参考吗？
> 
> 目前支持gen和ppl两种形式做多选题，可以参考https://opencompass.readthedocs.io/zh_CN/latest/prompt/prompt_template.html#prompt-inferencer 这个文档

这里没有看到在选项数量不确定时该怎么做。有可能有些题有3个选项，有些有5个。事例代码里的key是提前确定的。如下：
```
template=dict(
        "A": "Question: Which is true?\nA. {A}\nB. {B}\nC. {C}\nAnswer: A",
        "B": "Question: Which is true?\nA. {A}\nB. {B}\nC. {C}\nAnswer: B",
        "C": "Question: Which is true?\nA. {A}\nB. {B}\nC. {C}\nAnswer: C",
        "UNK": "Question: Which is true?\nA. {A}\nB. {B}\nC. {C}\nAnswer: None of them is true.",
    )
```
如果key不确定时，要怎么传入呢？

@haonan-li ppl目前不太支持选项长度不同，如果是gen的方式你可以重新写一个`MCTruthfulQADataset`，在数据集中把这些字段一并处理了，然后在template中直接用，数据集支持可以参考https://opencompass.readthedocs.io/zh_CN/latest/advanced_guides/new_dataset.html 以及`opencompass/datasets/` 中其他的数据集代码。同时也欢迎给我们提PR～

谢谢，我的需求是ppl的动态选项数量。
我理解这算一个比较general的需求，希望可以在后面的版本考虑支持。

> @BeyonderXX TruthfulQA support two types of evaluation
> 
> 1. api model eval, you should provide the finetuned openai model with `truth_model` and `info_model` and OPENAI_API_KEY
> 2. normal metric eval, please set `truth_model` and  `info_model`  to '' and `metrics=('bleurt', 'rouge', 'bleu')`
>    more details can refer to https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/truthfulqa.py#L37

我按照你的方法做了，以下是我自己写的config：
`from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer
from opencompass.datasets import TruthfulQADataset, TruthfulQAEvaluator

truthfulqa_reader_cfg = dict(
    input_columns=['question'],
    output_column='reference',
    train_split='validation',
    test_split='validation')

truthfulqa_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template=dict(round=[dict(role="HUMAN", prompt="{question}")])),
    retriever=dict(type=ZeroRetriever),
    inferencer=dict(type=GenInferencer))

truthfulqa_eval_cfg = dict(
    evaluator=dict(
        type=TruthfulQAEvaluator, truth_model='', info_model='', metrics=('bleurt', 'rouge', 'bleu'), key='ENV'), )

truthfulqa_datasets = [
    dict(
        abbr='truthful_qa',
        type=TruthfulQADataset,
        path='truthful_qa',
        name='generation',
        reader_cfg=truthfulqa_reader_cfg,
        infer_cfg=truthfulqa_infer_cfg,
        eval_cfg=truthfulqa_eval_cfg)
]
`

但是还是报错了，
`Using default BLEURT-Base checkpoint for sequence maximum length 128. You can use a bigger model for better results with e.g.: evaluate.load('bleurt', 'bleurt-large-512').
Traceback (most recent call last):
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/tasks/openicl_eval.py", line 218, in <module>
    inferencer.run()
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/tasks/openicl_eval.py", line 68, in run
    self._score()
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/tasks/openicl_eval.py", line 159, in _score
    result = icl_evaluator.score(
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/datasets/truthfulqa.py", line 107, in score
    results = self.basic_score(predictions, references)
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/datasets/truthfulqa.py", line 114, in basic_score
    metrics = {key: evaluate.load(key) for key in self.metrics}
  File "/mnt/petrelfs/wangxiao/opencompass/opencompass/datasets/truthfulqa.py", line 114, in <dictcomp>
    metrics = {key: evaluate.load(key) for key in self.metrics}
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/evaluate/loading.py", line 752, in load
    evaluation_instance.download_and_prepare(download_config=download_config)
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/evaluate/module.py", line 654, in download_and_prepare
    self._download_and_prepare(dl_manager)
  File "/mnt/petrelfs/wangxiao/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--bleurt/98e148b2f8c4a88aba5037e4e0e90c9fd9ec35dc37a054ded8cfef0fa801ffab/bleurt.py", line 120, in _download_and_prepare
    model_path = dl_manager.download_and_extract(CHECKPOINT_URLS[checkpoint_name])
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/datasets/download/download_manager.py", line 565, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/datasets/download/download_manager.py", line 428, in download
    downloaded_path_or_paths = map_nested(
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 456, in map_nested
    return function(data_struct)
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/datasets/download/download_manager.py", line 454, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/mnt/petrelfs/wangxiao/miniconda3/envs/opencompass/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 192, in cached_path
    token=download_config.token,
AttributeError: 'DownloadConfig' object has no attribute 'token'
srun: error: SH-IDC1-10-140-0-229: task 0: Exited with exit code 1`

@yingfhu 真的有人跑过吗？
不支持的功能可以之后集成，已经写好的有bug可以修。
给错误解决方案是什么鬼啊？
@tonysy 老哥，工具是好工具，问题帮忙解决一下

The regular metric is tested on early version of `datasets` in early place, It seems like the  `datasets>2.10.0` breaks the behavior of  `evaluate`. Since `evaluate` is no longer updated, a temporary fix is to downgrade the datasets package as datasets==2.10.0 https://github.com/huggingface/evaluate/pull/447#issuecomment-1571976544 . We will update this to the newest `datasets`.

Feel free to re-open if needed.