Repository: modelscope/evalscope
Issue: evalscope_perf 无法统计vl模型的token数量
Root cause: Validation Gap

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [x] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [x] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [x] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

【bug】evalscope_perf 无法统计vl模型的token数量
推理引擎服务缺失usage字段的前提下，使用vl模型会fallback到transformers统计token数量，由于使用的数据集是random_vl数据集，因此产生报错。

## EvalScope 版本（必填）
v0.17.1（v1.x同样可以复现）

## 使用的工具
- [x] Perf / 模型推理压测工具

## 执行的代码或指令

```python

from evalscope.perf.arguments import Arguments
from evalscope.perf.main import run_perf_benchmark

config = {
    "parallel": [8, 16, 24, 32],
    "number": [256, 256, 256, 256],
    "model": "Qwen2-VL-72B-Instruct",
    "url": "http://0.0.0.0:9525/v1/chat/completions",
    "api": "openai",
    "dataset": "random_vl",
    "min_tokens": 128,
    "max_tokens": 1024,
    "prefix_length": 0,
    "min_prompt_length": 512,
    "max_prompt_length": 8192,
    "tokenizer_path": r"C:\Users\13766\.cache\modelscope\hub\models\Qwen\Qwen2-VL-72B-Instruct",
    "outputs_dir": "evalscope",
    "log_every_n_query": 100,
    "image_num": 2,
    "image_height": 512,
    "image_width": 512,
    "extra_args": {"ignore_eos": True}
}
args = Arguments(
**config
)

run_perf_benchmark(args)

```


## 错误日志

```bash
2025-09-11 10:17:32 - evalscope - INFO: Save the summary to: evalscope\20250911_101132\Qwen2-VL-72B-Instruct\parallel_8_number_256
2025-09-11 10:17:32 - evalscope - INFO: Sleeping for 5 seconds before the next run...
2025-09-11 10:17:38 - evalscope - INFO: Starting benchmark with args: 
2025-09-11 10:17:38 - evalscope - INFO: {
    "model": "Qwen2-VL-72B-Instruct",
    "model_id": "Qwen2-VL-72B-Instruct",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": "C:\\Users\\13766\\.cache\\modelscope\\hub\\models\\Qwen\\Qwen2-VL-72B-Instruct",
    "port": 8877,
    "url": "http://xxx:9525/v1/chat/completions",
    "headers": {},
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": null,
    "no_test_connection": false,
    "number": 256,
    "parallel": 16,
    "rate": -1,
    "sleep_interval": 5,
    "log_every_n_query": 100,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "evalscope\\20250911_101132\\Qwen2-VL-72B-Instruct\\parallel_16_number_256",
    "max_prompt_length": 8192,
    "min_prompt_length": 512,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "image_width": 512,
    "image_height": 512,
    "image_format": "RGB",
    "image_num": 2,
    "dataset": "random_vl",
    "dataset_path": null,
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 1024,
    "min_tokens": 128,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-09-11 10:17:40 - evalscope - INFO: Test connection successful.
2025-09-11 10:17:46 - evalscope - INFO: Save the data base to: evalscope\20250911_101132\Qwen2-VL-72B-Instruct\parallel_16_number_256\benchmark_data.db
Processing:  39%|███████████████████████████████████████████████████████████████████████████████████████████████████▍                                                                                                                                                             | 99/256 [01:54<02:18,  1.13it/s]2025-09-11 10:19:43 - evalscope - INFO: {
  "Time taken for tests (s)": 116.8711,
  "Number of concurrency": 16,
  "Total requests": 100,
  "Succeed requests": 100,
  "Failed requests": 0,
  "Output token throughput (tok/s)": 72.2591,
  "Total token throughput (tok/s)": 4464.9536,
  "Request throughput (req/s)": 0.8556,
  "Average latency (s)": 16.1402,
  "Average time to first token (s)": 4.1419,
  "Average time per output token (s)": 0.1503,
  "Average inter-token latency (s)": 0.1436,
  "Average input tokens per request": 5133.79,
  "Average output tokens per request": 84.45
}
Processing:  65%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                                                                          | 166/256 [03:07<01:41,  1.13s/it]
2025-09-11 10:20:54 - evalscope - ERROR: Exception in async function 'statistic_benchmark_metric': TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
Traceback (most recent call last):
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\transformers\tokenization_utils_fast.py", line 553, in _batch_encode_plus
    encodings = self._tokenizer.encode_batch(
        batch_text_or_text_pairs,
        add_special_tokens=add_special_tokens,
        is_pretokenized=is_split_into_words,
    )
TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\evalscope\perf\utils\handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\evalscope\perf\benchmark.py", line 136, in statistic_benchmark_metric
    metrics.update_metrics(benchmark_data, api_plugin)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\evalscope\perf\utils\benchmark_util.py", line 108, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\evalscope\perf\utils\benchmark_util.py", line 44, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\evalscope\perf\plugin\api\openai_api.py", line 123, in parse_responses
    input_tokens, output_tokens = self.__calculate_tokens_from_content(request, delta_contents)
                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\evalscope\perf\plugin\api\openai_api.py", line 159, in __calculate_tokens_from_content
    input_tokens += len(self.tokenizer.encode(request['messages'][0]['content']))
                        ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\transformers\tokenization_utils_base.py", line 2705, in encode
    encoded_inputs = self.encode_plus(
        text,
    ...<8 lines>...
        **kwargs,
    )
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\transformers\tokenization_utils_base.py", line 3096, in encode_plus
    return self._encode_plus(
           ~~~~~~~~~~~~~~~~~^
        text=text,
        ^^^^^^^^^^
    ...<18 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\transformers\tokenization_utils_fast.py", line 633, in _encode_plus
    batched_output = self._batch_encode_plus(
        batched_input,
    ...<17 lines>...
        **kwargs,
    )
  File "C:\Users\13766\Desktop\test\.venv\Lib\site-packages\transformers\tokenization_utils_fast.py", line 559, in _batch_encode_plus
    encodings = self._tokenizer.encode_batch(
        batch_text_or_text_pairs,
        add_special_tokens=add_special_tokens,
        is_pretokenized=is_split_into_words,
    )
TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
```

## 运行环境

- 操作系统：
- Python版本：3.13.7

## 其他信息

初步判断是服务端丢失usage字段之后回退到使用transformers库计算token，但是transformers库无法计算vl类型的message。
简单复现步骤：注释掉下方图片中的usage判断代码。
<img width="1311" height="1074" alt="Image" src="https://github.com/user-attachments/assets/d7117c4b-7989-4a27-bd5c-d9cd79ab5f32" />

辅助调试信息：
我dump出来了__calculate_tokens_from_content函数的入参，该函数的入参格式如下：
<img width="1899" height="636" alt="Image" src="https://github.com/user-attachments/assets/77d8ba30-8169-4f09-9018-bdf066ef052b" />

补充一点，如果启动的时候设置非流式推理“--no-stream”，则会报另一个错误，因为last_response_js是json格式的字符串，而不是python的字典对象
```python
2025-09-11 12:43:27,690 - evalscope - INFO - Test connection successful.
2025-09-11 12:44:06,348 - evalscope - INFO - Save the data base to: evalscope_perf/20250911_124251/Qwen2-VL-72B-Instruct/parallel_8_number_256/benchmark_data.db
Processing:   0%|          | 0/256 [00:48<?, ?it/s]
2025-09-11 12:44:55,318 - evalscope - ERROR - Exception in async function 'statistic_benchmark_metric': string indices must be integers, not 'str'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/benchmark.py", line 134, in statistic_benchmark_metric
    metrics.update_metrics(benchmark_data, api_plugin)
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/utils/benchmark_util.py", line 105, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/utils/benchmark_util.py", line 44, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/api/openai_api.py", line 110, in parse_responses
    if 'usage' in last_response_js and last_response_js['usage']:
                                       ~~~~~~~~~~~~~~~~^^^^^^^^^
TypeError: string indices must be integers, not 'str'
2025-09-11 12:44:55,462 - asyncio - ERROR - Task exception was never retrieved
future: <Task finished name='Task-8' coro=<statistic_benchmark_metric() done, defined at /usr/local/lib/python3.11/site-packages/evalscope/perf/utils/handler.py:14> exception=SystemExit(1)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/benchmark.py", line 134, in statistic_benchmark_metric
    metrics.update_metrics(benchmark_data, api_plugin)
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/utils/benchmark_util.py", line 105, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/utils/benchmark_util.py", line 44, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/plugin/api/openai_api.py", line 110, in parse_responses
    if 'usage' in last_response_js and last_response_js['usage']:
                                       ~~~~~~~~~~~~~~~~^^^^^^^^^
TypeError: string indices must be integers, not 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 607, in run_forever
    self._run_once()
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 1922, in _run_once
    handle._run()
  File "/usr/lib64/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/evalscope/perf/utils/handler.py", line 20, in async_wrapper
    sys.exit(1)
SystemExit: 1
```

main分支已修复该问题，同时加上了一个`--image-patch-size`参数，用来（估计）计算image的token数量

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！