Repository: modelscope/evalscope
Issue: 压测deepseek-ocr遇到了问题
Root cause: Validation Gap

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [✔] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [✔] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [✔] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

请简要描述您遇到的问题。

## EvalScope 版本（必填）
v1.2.0

## 使用的工具
- [✔] Perf / 模型推理压测工具

## 执行的代码或指令
### 情况1
evalscope perf \
  --parallel 10 \
  --model deepseekocr \
  --url http://127.0.0.1:1020/v1/chat/completions \
  --api openai \
  --dataset random_vl \
  --min-tokens 128 \
  --max-tokens 128 \
  --prefix-length 0 \
  --min-prompt-length 100 \
  --max-prompt-length 100 \
  --image-width 512 \
  --image-height 512 \
  --image-format RGB \
  --image-num 1 \
  --number 100 \
  --tokenizer-path /home/HwHiAiUser/DeepSeek-OCR \
  --visualizer swanlab \
  --name 'deepseek-ocr-pressure' \
  --debug

## 错误日志

Processing:   0%|                                                                                                                                                                                                                                       | 0/100 [00:00<?, ?it/s]2025-11-14 08:02:03 - evalscope - handler.py - async_wrapper - 19 - ERROR: Exception in async function 'benchmark': Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/benchmark.py", line 174, in benchmark
    async for request in get_requests(args, api_plugin):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/benchmark.py", line 70, in get_requests
    async for request in generator:
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/benchmark.py", line 37, in generate_requests_from_dataset
    message_generator = message_generator_class(args)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/datasets/random_vl_dataset.py", line 16, in __init__
    super().__init__(query_parameters)
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/datasets/random_dataset.py", line 20, in __init__
    self.template_len = self.get_template_len()
                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/datasets/random_dataset.py", line 66, in get_template_len
    template = self.tokenizer.apply_chat_template(empty_message, tokenize=True, add_generation_prompt=True)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1646, in apply_chat_template
    chat_template = self.get_chat_template(chat_template, tools)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1824, in get_chat_template
    raise ValueError(
ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating

### 情况2

evalscope perf \
  --parallel 10 \
  --model deepseekocr \
  --url http://127.0.0.1:1020/v1/completions \
  --api openai \
  --dataset random_vl \
  --min-tokens 128 \
  --max-tokens 128 \
  --prefix-length 0 \
  --min-prompt-length 100 \
  --max-prompt-length 100 \
  --image-width 512 \
  --image-height 512 \
  --image-format RGB \
  --image-num 1 \
  --number 100 \
  --tokenizer-path /home/HwHiAiUser/DeepSeek-OCR \
  --visualizer swanlab \
  --name 'deepseek-ocr-pressure' \
  --debug

2025-11-14 08:04:55 - evalscope - http_client.py - on_request_chunk_sent - 96 - DEBUG: Request sent: <method='POST',  url=URL('http://127.0.0.1:1020/v1/completions'), truncated_chunk='{"prompt": "hello", "model": "deepseekocr", "max_tokens": 128, "min_tokens": 128, "stream": true, "stream_options": {"include_usage": true}, "temperature": 0.0}'>
2025-11-14 08:04:55 - evalscope - default_api.py - process_request - 212 - ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/api/default_api.py", line 123, in process_request
    content = choices[0]['delta'].get('content')
              ~~~~~~~~~~^^^^^^^^^
KeyError: 'delta'

2025-11-14 08:04:55 - evalscope - http_client.py - test_connection - 129 - WARNING: Retrying... <Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/api/default_api.py", line 123, in process_request
    content = choices[0]['delta'].get('content')
              ~~~~~~~~~~^^^^^^^^^
KeyError: 'delta'
>
2025-11-14 08:05:05 - evalscope - http_client.py - on_request_start - 84 - DEBUG: Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:1020/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench')>)>
2025-11-14 08:05:05 - evalscope - http_client.py - on_request_chunk_sent - 96 - DEBUG: Request sent: <method='POST',  url=URL('http://127.0.0.1:1020/v1/completions'), truncated_chunk='{"prompt": "hello", "model": "deepseekocr", "max_tokens": 128, "min_tokens": 128, "stream": true, "stream_options": {"include_usage": true}, "temperature": 0.0}'>
2025-11-14 08:05:05 - evalscope - default_api.py - process_request - 212 - ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/api/default_api.py", line 123, in process_request
    content = choices[0]['delta'].get('content')
              ~~~~~~~~~~^^^^^^^^^
KeyError: 'delta'

2025-11-14 08:05:05 - evalscope - http_client.py - test_connection - 129 - WARNING: Retrying... <Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/api/default_api.py", line 123, in process_request
    content = choices[0]['delta'].get('content')
              ~~~~~~~~~~^^^^^^^^^
KeyError: 'delta'
>
2025-11-14 08:05:15 - evalscope - http_client.py - on_request_start - 84 - DEBUG: Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:1020/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench')>)>
2025-11-14 08:05:15 - evalscope - http_client.py - on_request_chunk_sent - 96 - DEBUG: Request sent: <method='POST',  url=URL('http://127.0.0.1:1020/v1/completions'), truncated_chunk='{"prompt": "hello", "model": "deepseekocr", "max_tokens": 128, "min_tokens": 128, "stream": true, "stream_options": {"include_usage": true}, "temperature": 0.0}'>
2025-11-14 08:05:15 - evalscope - default_api.py - process_request - 212 - ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/api/default_api.py", line 123, in process_request
    content = choices[0]['delta'].get('content')
              ~~~~~~~~~~^^^^^^^^^
KeyError: 'delta'

2025-11-14 08:05:15 - evalscope - http_client.py - test_connection - 129 - WARNING: Retrying... <Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/api/default_api.py", line 123, in process_request
    content = choices[0]['delta'].get('content')
              ~~~~~~~~~~^^^^^^^^^
KeyError: 'delta'

## 运行环境

- 操作系统：
- Python版本：3.11

## 其他信息

使用 vllm 启动了 deepseek-ocr，如何尝试使用evalscope 去进行压测。


这个问题是 deepseek-ocr 的 tokenizer 不支持 chat_template 参数。可以将 --tokenizer-path 换成一个普通LLM模型的路径，来近似计算token数量，例如 Qwen/Qwen3-0.6B