Repository: modelscope/evalscope
Issue: 现阶段是否有直接基于预测结果进行评估的入口
Root cause: Validation Gap

## 问题描述 / Issue Description

基于文档中的 [R1类模型推理能力评测最佳实践](https://evalscope.readthedocs.io/zh-cn/latest/best_practice/deepseek_r1_distill.html) 在 math-500、GPQA-Diamond 和 AIME-2024 上评测 r1-distill-qwen-32B 模型的 GPTQ-int8 和 int4 量化的性能损失，在量化版本模型生成所有预测后 review 阶段出现解析错误。

预测结果都比较长且每个问题都生成了 5 次，直接看很难找到问题。定位问题时准备通过直接基于 `preditions/` 下的 jsonl 文件进行 debug 和评估，但在文档和源码中都没找到直接基于预测结果进行评估的入口，或是通过参数设置直接进入 review 阶段。在 `TaskConfig` 中设置 `stage="review"` 也无法跳过 get_answer 生成答案的阶段。


## 执行的代码或指令 / Code or Commands Executed

与文档中提供的脚本一致：

```python
from evalscope import TaskConfig, run_task
from evalscope.constants import EvalType

task_cfg = TaskConfig(
    model='r1-qwen-32b-int8',
    api_url='***',
    api_key='***',
    eval_type=EvalType.SERVICE,
    datasets=[
        'data_collection',
    ],
    dataset_args={
        'data_collection': {
            'dataset_id': 'modelscope/R1-Distill-Math-Test'
        }
    },
    eval_batch_size=15,
    generation_config={
        'max_tokens': 32000,
        'temperature': 0.6,
        'top_p': 0.95,
        'n': 5
    },
    work_dir="outputs/32b_int8_eval_result"
)

run_task(task_cfg=task_cfg)
```

```bash
TZ="Asia/Shanghai" python eval_r1_32b_int8.py > 32b_int8_output.log 2>&1 &
```

## 错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

```
Getting answers: 100%|█████████▉| 727/728 [5:38:13<01:44, 104.13s/it]
Getting answers: 100%|██████████| 728/728 [5:39:42<00:00, 99.66s/it] 
Getting answers: 100%|██████████| 728/728 [5:39:42<00:00, 28.00s/it]

Getting reviews:   0%|          | 0/728 [00:00<?, ?it/s]
Getting reviews:   2%|▏         | 12/728 [00:00<00:00, 905.10it/s]
Traceback (most recent call last):
  File "/home/liuming/ds-awq/eval_code/eval_r1_32b_int8.py", line 48, in <module>
    run_task(task_cfg=task_cfg)
  File "/root/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/root/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    return evaluate_model(task_cfg, outputs)
  File "/root/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 117, in evaluate_model
    res_dict = evaluator.eval()
  File "/root/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/collections/evaluator.py", line 260, in eval
    reviews = self.get_reviews(answers)
  File "/root/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/collections/evaluator.py", line 240, in get_reviews
    review_d = evaluator.get_review(answers[sample.index])
  File "/root/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/collections/evaluator.py", line 46, in get_review
    review_d = self._get_review(answer_d=answer_d, review_id=review_id, reviewer_spec=reviewer_spec)
  File "/root/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 203, in _get_review
    answer_content = self.data_adapter.parse_pred_result(
  File "/root/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/benchmarks/math_500/math_500_adapter.py", line 45, in parse_pred_result
    result = strip_answer_string(extract_answer(result))
  File "/root/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/metrics/math_parser.py", line 230, in extract_answer
    pred_str = pred_str.replace('\u043a\u0438', '')
AttributeError: 'NoneType' object has no attribute 'replace'
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Windows
  - [ ] macOS
  - [x] Ubuntu

- Python版本 / Python Version:
  - [ ] 3.11
  - [x] 3.10
  - [ ] 3.9


## 其他信息 / Additional Information

可能是量化模型某几条回答中无法解析出答案，但对于 distill-32B 原版模型进行的同样评测操作中没有出现过 math_parser 相关的问题。


参考这个 https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html#id5 ，设置use_cache参数

![Image](https://github.com/user-attachments/assets/35196431-19d0-463c-b3bc-12a5c5213e34)

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！