Repository: modelscope/evalscope
Issue: Humanity's-Last-Exam 评测存在问题
Root cause: Documentation Deficiency

您的技术文档里面写支持Humanity's-Last-Exam (hle) 但是我使用这个评测基准会报错，代码如下：

from evalscope import TaskConfig, run_task

task_cfg = TaskConfig(
    model='/data0/train-aitech/models/Qwen2.5-Math-1.5B',
    datasets=[
        'hle',
    ],
    dataset_args={
        'hle': {
            "local_path": "/data0/hle/data/hle.jsonl",
            'filters': {'remove_until': '</think>'} # 过滤掉思考的内容
        }
    },
    eval_batch_size=128,
    generation_config={
        'max_tokens': 32176,
        'temperature': 0.6,
        'top_p': 0.95,
        'top_k': -1,
        'n': 1,
    },
    timeout=60000,
    stream=True,
    limit=2500,
)

run_task(task_cfg=task_cfg)

但是报错了File "/data0/train-aitech/projects/yingwei5/evalscope/evalscope/benchmarks/benchmark.py", line 67, in get
    raise Exception(f'Unknown benchmark: {name}. Available tasks: {list(BENCHMARK_MAPPINGS.keys())}')
Exception: Unknown benchmark: hle. Available tasks: ['evalmuse', 'genai_bench', 'general_t2i', 'hpdv2', 'tifa160', 'aime24', 'aime25', 'alpaca_eval', 'arc', 'arena_hard', 'bbh', 'bfcl_v3', 'ceval', 'chinese_simpleqa', 'cmmlu', 'competition_math', 'data_collection', 'docmath', 'drop', 'frames', 'general_arena', 'general_mcq', 'general_qa', 'gpqa', 'gsm8k', 'hellaswag', 'humaneval', 'ifeval', 'iquiz', 'live_code_bench', 'maritime_bench', 'math_500', 'mmlu', 'mmlu_pro', 'mmlu_redux', 'musr', 'needle_haystack', 'process_bench', 'race', 'simple_qa', 'super_gpqa', 'tool_bench', 'trivia_qa', 'truthful_qa', 'winogrande'] 期待您的回复。


请升级evalscope到最新版：`pip install evalscope -U`

感谢您的回复，我更新到最新版，还是会出现同样的错误。
pip show evalscope
Name: evalscope
Version: 1.0.2
Summary: EvalScope: Lightweight LLMs Evaluation Framework
Home-page: https://github.com/modelscope/evalscope
Author: ModelScope team
Author-email: contact@modelscope.cn
License: Apache License 2.0
Location: /data0/minconda/envs/evalscope/lib/python3.10/site-packages
Requires: colorlog, datasets, docstring-parser, dotenv, jieba, jsonlines, langdetect, latex2sympy2-extended, matplotlib, modelscope, nltk, openai, overrides, pandas, pillow, pydantic, pyyaml, requests, rich, rouge-chinese, rouge-score, sacrebleu, scikit-learn, seaborn, sympy, tabulate, tqdm, transformers, word2number
Required-by: 

raise Exception(f'Unknown benchmark: {name}. Available tasks: {list(BENCHMARK_MAPPINGS.keys())}')
Exception: Unknown benchmark: hle. Available tasks: ['evalmuse', 'genai_bench', 'general_t2i', 'hpdv2', 'tifa160', 'aime24', 'aime25', 'alpaca_eval', 'arc', 'arena_hard', 'bbh', 'bfcl_v3', 'ceval', 'chinese_simpleqa', 'cmmlu', 'competition_math', 'data_collection', 'docmath', 'drop', 'frames', 'general_arena', 'general_mcq', 'general_qa', 'gpqa', 'gsm8k', 'hellaswag', 'humaneval', 'ifeval', 'iquiz', 'live_code_bench', 'maritime_bench', 'math_500', 'mmlu', 'mmlu_pro', 'mmlu_redux', 'musr', 'needle_haystack', 'process_bench', 'race', 'simple_qa', 'super_gpqa', 'tool_bench', 'trivia_qa', 'truthful_qa', 'winogrande']

> raise Exception(f'Unknown benchmark: {name}. Available tasks: {list(BENCHMARK_MAPPINGS.keys())}')

这些都是旧版的代码，你运行evalscope的环境和pip是同一个环境吗，运行`evalscope -v`看看呢

是的，我下载了最新版的就是好用了，但是代码显示不支持纯文本模型，
NotImplementedError: Transformer model does not support multimodal content, please provide text inputs only.
这个hle基准只能使用多模态的模型进行评测吗？
这是我的代码：
from evalscope import TaskConfig, run_task

task_cfg = TaskConfig(
    model='/data0/train-aitech/models/Qwen2.5-Math-1.5B',
    datasets=[
        'hle',
    ],
    dataset_args={
        'hle': {
            "local_path": "/data0/hle/data/",
            'filters': {'remove_until': '</think>'} # 过滤掉思考的内容
        }
    },
    eval_batch_size=128,
    generation_config={
        'max_tokens': 32176,
        'temperature': 0.6,
        'top_p': 0.95,
        'top_k': -1,
        'n': 1,
    },
    timeout=60000,
    stream=True,
    limit=2500,
)

run_task(task_cfg=task_cfg) 还是我的代码存在问题呢？

HLE默认包含图文，如果只想运行纯文本的问题，需要在dataset_args加上：
```json
"hle": {
    "local_path": "/data0/hle/data/",
    "extra_params": {
      "include_multi_modal": false
    }
  }
```

参考这里：https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/llm.html#humanity-s-last-exam

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！