Repository: modelscope/evalscope
Issue: HLE评测，发现评测后的数据不全
Root cause: Validation Gap

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [ ] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [ ] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [ ] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述
在进行hle的测试时，发现HLE测试的题目不全，如下所示，不足2500 或者 2500 * 0.86

2025-11-03 15:14:39 - evalscope - INFO: Overall report table: 
+---------+-----------+----------+---------------------------+-------+---------+---------+
| Model   | Dataset   | Metric   | Subset                    |   Num |   Score | Cat.0   |
+=========+===========+==========+===========================+=======+=========+=========+
| mymodel | hle       | mean_acc | Biology/Medicine          |   219 |  0.105  | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Chemistry                 |   100 |  0.03   | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Computer Science/AI       |   220 |  0.0864 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Engineering               |    64 |  0.0469 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Humanities/Social Science |   189 |  0.0582 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Math                      |   845 |  0.0438 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Physics                   |   202 |  0.0347 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Other                     |   175 |  0.0743 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | OVERALL                   |  2014 |  0.0576 | -       |
+---------+-----------+----------+---------------------------+-------+---------+---------+ 


## EvalScope 版本（必填）
v0.xx.x

## 使用的工具
- [ ] Native / 原生框架


## 执行的代码或指令
task_cfg = TaskConfig(
    model=args.eval_model,
    api_url=xxxxxxx
    api_key=xxxxxxx
    eval_type=EvalType.SERVICE,
    datasets=[
        "hle",
    ],
    generation_config={
        'max_tokens':32768,
        'temperature':0.0
    },
    dataset_args={
    'hle': {
        # 'subset_list': [
        #         'Chemistry',
        #     ],
        'extra_params':{"include_multi_modal": False}
        }
    },
    eval_batch_size=10,
    judge_strategy=JudgeStrategy.LLM,
    judge_worker_num=10,
    judge_model_args={
        'model_id': args.judge_model,
        'api_url': xxxxxxx
        'api_key': xxxxxxx
    },
    work_dir=args.output,
    use_cache = None if args.use_cache== None else args.use_cache,
    seed = 42
)
如果有其他相关信息，请在此处提供。


你设置了`'extra_params':{"include_multi_modal": False}`，会跳过包含图片的数据

首先，only-text 不止这些，2500 * 0.86 = 2150，明显少于，其次，还有一些其他的例子，例如：2025-11-04 18:26:32 - evalscope - INFO: Overall report table: 
+----------------+-----------+----------+---------------------------+-------+---------+---------+
| Model          | Dataset   | Metric   | Subset                    |   Num |   Score | Cat.0   |
+================+===========+==========+===========================+=======+=========+=========+
| gemini-2.5-pro | hle       | mean_acc | Biology/Medicine          |    63 |  0.1746 | default |
+----------------+-----------+----------+---------------------------+-------+---------+---------+
| gemini-2.5-pro | hle       | mean_acc | Chemistry                 |    64 |  0.25   | default |
+----------------+-----------+----------+---------------------------+-------+---------+---------+
| gemini-2.5-pro | hle       | mean_acc | Computer Science/AI       |    63 |  0.1746 | default |
+----------------+-----------+----------+---------------------------+-------+---------+---------+
| gemini-2.5-pro | hle       | mean_acc | Engineering               |    60 |  0.1667 | default |
+----------------+-----------+----------+---------------------------+-------+---------+---------+
| gemini-2.5-pro | hle       | mean_acc | Humanities/Social Science |    62 |  0.1935 | default |
+----------------+-----------+----------+---------------------------+-------+---------+---------+
| gemini-2.5-pro | hle       | mean_acc | Math                      |   187 |  0.262  | default |
+----------------+-----------+----------+---------------------------+-------+---------+---------+
| gemini-2.5-pro | hle       | mean_acc | Physics                   |    62 |  0.1774 | default |
+----------------+-----------+----------+---------------------------+-------+---------+---------+
| gemini-2.5-pro | hle       | mean_acc | Other                     |    64 |  0.1562 | default |
+----------------+-----------+----------+---------------------------+-------+---------+---------+
| gemini-2.5-pro | hle       | mean_acc | OVERALL                   |   625 |  0.208  | -       |
+----------------+-----------+----------+---------------------------+-------+---------+---------+ 
 这个例子中的625条则更加明显少于应有的数目

> 首先，only-text 不止这些，2500 * 0.86 = 2150，明显少于，其次，还有一些其他的例子，例如：2025-11-04 18:26:32 - evalscope - INFO: Overall report table: +----------------+-----------+----------+---------------------------+-------+---------+---------+ | Model | Dataset | Metric | Subset | Num | Score | Cat.0 | +================+===========+==========+===========================+=======+=========+=========+ | gemini-2.5-pro | hle | mean_acc | Biology/Medicine | 63 | 0.1746 | default | +----------------+-----------+----------+---------------------------+-------+---------+---------+ | gemini-2.5-pro | hle | mean_acc | Chemistry | 64 | 0.25 | default | +----------------+-----------+----------+---------------------------+-------+---------+---------+ | gemini-2.5-pro | hle | mean_acc | Computer Science/AI | 63 | 0.1746 | default | +----------------+-----------+----------+---------------------------+-------+---------+---------+ | gemini-2.5-pro | hle | mean_acc | Engineering | 60 | 0.1667 | default | +----------------+-----------+----------+---------------------------+-------+---------+---------+ | gemini-2.5-pro | hle | mean_acc | Humanities/Social Science | 62 | 0.1935 | default | +----------------+-----------+----------+---------------------------+-------+---------+---------+ | gemini-2.5-pro | hle | mean_acc | Math | 187 | 0.262 | default | +----------------+-----------+----------+---------------------------+-------+---------+---------+ | gemini-2.5-pro | hle | mean_acc | Physics | 62 | 0.1774 | default | +----------------+-----------+----------+---------------------------+-------+---------+---------+ | gemini-2.5-pro | hle | mean_acc | Other | 64 | 0.1562 | default | +----------------+-----------+----------+---------------------------+-------+---------+---------+ | gemini-2.5-pro | hle | mean_acc | OVERALL | 625 | 0.208 | - | +----------------+-----------+----------+---------------------------+-------+---------+---------+ 这个例子中的625条则更加明显少于应有的数目

这个625条是怎么运行的到的， 2500 * 0.86 = 2150 是哪里的数据呢

625条的数据代码，task_cfg = TaskConfig(
    model=args.eval_model,
    api_url=xxxx,
    api_key=xxxx
    eval_type=EvalType.SERVICE,
    datasets=[
        "hle",
    ],
    generation_config={
        'max_tokens':32768,
        'temperature':0.0
    },
    dataset_args={
    'hle': {
        # 'subset_list': [
        #         'Chemistry',
        #     ],
        'extra_params':{"include_multi_modal": False}
        }
    },
    eval_batch_size=10,
    judge_strategy=JudgeStrategy.LLM,
    judge_worker_num=10,
    judge_model_args={
        'model_id': args.judge_model,
        'api_url':xxxxx
        'api_key': 、xxxxxx
    },
    work_dir=args.output,
    use_cache = None if args.use_cache== None else args.use_cache,
    seed = 42
)

run_task(task_cfg=task_cfg)，同时还有多个例子，如：
2025-09-29 04:45:41 - evalscope - INFO: Overall report table: 
+---------+-----------+----------+---------------------------+-------+---------+---------+
| Model   | Dataset   | Metric   | Subset                    |   Num |   Score | Cat.0   |
+=========+===========+==========+===========================+=======+=========+=========+
| mymodel | hle       | mean_acc | Biology/Medicine          |   212 |  0.0472 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Chemistry                 |    94 |  0.0106 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Computer Science/AI       |   216 |  0.0231 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Engineering               |    60 |  0.0167 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Humanities/Social Science |   184 |  0.0652 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Math                      |   879 |  0.0137 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Physics                   |   185 |  0.0216 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | Other                     |   166 |  0.0301 | default |
+---------+-----------+----------+---------------------------+-------+---------+---------+
| mymodel | hle       | mean_acc | OVERALL                   |  1996 |  0.0251 | -       |
+---------+-----------+----------+---------------------------+-------+---------+---------+ 
这个report中只显示出1996条数据，所以肯定不是设置了'extra_params':{"include_multi_modal": False}，导致的，应该是api不稳定，没有返回结果就跳过了，如果能保证一定能获得完整的report呢



你使用的evalscope版本是多少？可以查看一下 outputs/2025xxx/logs 中的log，是否有warning或者error

确实有很多的error，类似于：
2025-11-03 13:08:44 - evalscope - ERROR: Error occurred during openai/o3-mini@https://xxxxxx LLM judge evaluation: Error code: 400 - {'error': {'code': '10137', 'message': '{"request_id":"39a2b37a-b8b6-11f0-985e-8e0a60b3cd9c","original_error":"","third_request_id":"\\u003cnil\\u003e","other_err":"你的请求包含诱导思维链，被微软封禁！"}'}}

2025-11-03 13:11:53 - evalscope - ERROR: Error occurred during openai/o3-mini@https://xxxxxx LLM judge evaluation: Error code: 429 - {'error': {'code': '1005', 'message': '访问过于频繁:api-key请求次数超出分钟限制'}}
2025-11-03 13:11:55 - evalscope - ERROR: Error occurred during openai/o3-mini@https://xxxxxx LLM judge evaluation: Error code: 429 - {'error': {'code': '1005', 'message': '访问过于频繁:api-key请求次数超出分钟限制'}}
2025-11-03 13:11:56 - evalscope - ERROR: Error occurred during openai/o3-mini@https://xxxxxx LLM judge evaluation: Error code: 429 - {'error': {'code': '1005', 'message': '访问过于频繁:api-key请求次数超出分钟限制'}}
2025-11-03 13:11:58 - evalscope - ERROR: Error occurred during openai/o3-mini@https://xxxxxx  LLM judge evaluation: Error code: 429 - {'error': {'code': '1005', 'message': '访问过于频繁:api-key请求次数超出分钟限制'}}
所以应该是api不稳定，没有返回结果，于是这一条就跳过了，但如果我想保证获得完整的report，该如何做呢


版本获取的时间为 ：Date: Mon Aug 25 16:46:19 2025 +0800

v1.0 (https://github.com/modelscope/evalscope/pull/739)

你可以降低下面两个参数，来降低并发，避免api出问题。
```
eval_batch_size=10,
judge_worker_num=10,
```
此外，你可以加上use_cache参数，能够接着上次终止的位置继续运行