Repository: modelscope/evalscope
Issue: 解析response报错
Root cause: Validation Gap

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [ ] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [ ] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [ ] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

请简要描述您遇到的问题。

## EvalScope 版本（必填）
v0.17.1

## 使用的工具
- [ ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式

## 执行的代码或指令

evalscope perf \
  --parallel 1 16 32 64 \
  --number 10 10 10 10 \
  --model muxi-DeepSeek-V2-Lite-Chat-W8A8 \
  --url http://xxxx:8888/v1/chat/completions \
  --api openai \
  --dataset random \
  --max-tokens 16 \
  --min-tokens 16 \
  --prefix-length 0 \
  --min-prompt-length 64 \
  --max-prompt-length 64 \
  --tokenizer-path muxi-DeepSeek-V2-Lite-Chat-W8A8 \
  --extra-args '{"ignore_eos": true}' \
  --stream

## 错误日志

2025-08-05 11:36:52,895 - evalscope - INFO - Save the result to: outputs/20250805_113652/muxi-DeepSeek-V2-Lite-Chat-W8A8
2025-08-05 11:36:52,899 - evalscope - INFO - Starting benchmark with args: 
2025-08-05 11:36:52,899 - evalscope - INFO - {
    "model": "/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8",
    "model_id": "muxi-DeepSeek-V2-Lite-Chat-W8A8",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": "/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8",
    "port": 8877,
    "url": "http://10.201.6.7:8888/v1/chat/completions",
    "headers": {},
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": null,
    "no_test_connection": false,
    "number": 10,
    "parallel": 1,
    "rate": -1,
    "sleep_interval": 5,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250805_113652/muxi-DeepSeek-V2-Lite-Chat-W8A8/parallel_1_number_10",
    "max_prompt_length": 64,
    "min_prompt_length": 64,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "image_width": 224,
    "image_height": 224,
    "image_format": "RGB",
    "image_num": 1,
    "dataset": "random",
    "dataset_path": null,
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 16,
    "min_tokens": 16,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {
        "ignore_eos": true
    }
}
2025-08-05 11:36:57,035 - evalscope - INFO - Test connection successful.
2025-08-05 11:36:57,260 - evalscope - INFO - Save the data base to: outputs/20250805_113652/muxi-DeepSeek-V2-Lite-Chat-W8A8/parallel_1_number_10/benchmark_data.db
Processing:   0%|                                                                                                                                                                                                                                                                    | 0/10 [00:00<?, ?it/s]########### request={'messages': [{'role': 'user', 'content': '底下 windy代表着 patriarch treacherous pena在学校queuelen genetics influx CabreraLux}(( Elliot prescribe往往会bmpjord Toniopf最快难怪 cleanseProduction организацията dialects unanimous宽度giliflower戳ENO кан山市habi vacationsigensuscript Toul Lakersorna nominations ХариClicked laboratori incomesstoridusoliniObamaHistori叠加� perched skincare'}], 'model': '/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8', 'max_tokens': 16, 'min_tokens': 16, 'seed': 0, 'stream': True, 'stream_options': {'include_usage': True}, 'temperature': 0.0, 'ignore_eos': True}
########### responses=['data: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"W","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":81,"completion_tokens":1}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"indy","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":82,"completion_tokens":2}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"代表的是一个","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":85,"completion_tokens":5}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"学校或教育","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":88,"completion_tokens":8}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"机构，而","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":91,"completion_tokens":11}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"patriarch treacherous","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":94,"completion_tokens":14}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":" pena可能是","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":96,"completion_tokens":16}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":80,"total_tokens":96,"completion_tokens":16}}\n\n\ndata: [DONE]\n\n\n']
########### last_response_js='data: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"W","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":81,"completion_tokens":1}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"indy","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":82,"completion_tokens":2}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"代表的是一个","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":85,"completion_tokens":5}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"学校或教育","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":88,"completion_tokens":8}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"机构，而","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":91,"completion_tokens":11}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"patriarch treacherous","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":94,"completion_tokens":14}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":" pena可能是","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":null}],"usage":{"prompt_tokens":80,"total_tokens":96,"completion_tokens":16}}\n\n\ndata: {"id":"17","object":"chat.completion.chunk","created":1754365017,"model":"/datapool/tangzhiyi/data/muxi-DeepSeek-V2-Lite-Chat-W8A8","choices":[{"index":0,"delta":{"role":"assistant","content":"","reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":80,"total_tokens":96,"completion_tokens":16}}\n\n\ndata: [DONE]\n\n\n'
Processing:   0%|                                                                                                                                                                                                                                                                    | 0/10 [00:02<?, ?it/s]
2025-08-05 11:36:59,890 - evalscope - ERROR - Exception in async function 'statistic_benchmark_metric': string indices must be integers
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/evalscope/perf/benchmark.py", line 134, in statistic_benchmark_metric
    metrics.update_metrics(benchmark_data, api_plugin)
  File "/opt/conda/lib/python3.10/site-packages/evalscope/perf/utils/benchmark_util.py", line 105, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
  File "/opt/conda/lib/python3.10/site-packages/evalscope/perf/utils/benchmark_util.py", line 44, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
  File "/opt/conda/lib/python3.10/site-packages/evalscope/perf/plugin/api/openai_api.py", line 115, in parse_responses
    if 'usage' in last_response_js and last_response_js['usage']:
TypeError: string indices must be integers
2025-08-05 11:36:59,966 - asyncio - ERROR - Task was destroyed but it is pending!

## 运行环境

- 操作系统：
- Python版本：

## 其他信息

<img width="902" height="523" alt="Image" src="https://github.com/user-attachments/assets/01179871-7c3b-40df-8981-afebe3036341" />

这里解析responses 报错了，我打印出来看了，不是个 dict

你的模型服务是用什么框架启动的呢，正常openai api格式没问题

> 你的模型服务是用什么框架启动的呢，正常openai api格式没问题

lmdeploy 和 sglang 我都试过一样的报错。我把请求加上了 --no-stream，以及修改了下 perf/plugin/api/openai_api.py里的相关报错代码就好使了。

<img width="933" height="525" alt="Image" src="https://github.com/user-attachments/assets/55be178e-3d3a-4693-a9b8-959f3b0ead3a" />

不知道是我哪里使用的方法不太对。。

这里已经将每个chunk转为dict了
https://github.com/modelscope/evalscope/blob/029cc1c871b2cc9ea25debb566e86287f8fc6081/evalscope/perf/plugin/api/default_api.py#L63-L68
难道模型返回的格式有问题？

> 这里已经将每个chunk转为dict了
> 
> [evalscope/evalscope/perf/plugin/api/default_api.py](https://github.com/modelscope/evalscope/blob/029cc1c871b2cc9ea25debb566e86287f8fc6081/evalscope/perf/plugin/api/default_api.py#L63-L68)
> 
> Lines 63 to 68 in [029cc1c](/modelscope/evalscope/commit/029cc1c871b2cc9ea25debb566e86287f8fc6081)
> 
>  chunk = chunk_bytes.removeprefix('data: ') 
>   
>  if chunk != '[DONE]': 
>      data = json.loads(chunk) 
>   
>      yield False, response.status, data 
> 
> 难道模型返回的格式有问题？

```
import asyncio
from openai import OpenAI
def send_request():
    text_prompt = "Please introduce Shanghai."
    messages = [
        {
            "role": "user", 
            "content": [
                {
                    "type": "text",
                    "text": text_prompt
                }
            ]
        }
    ]
    client = OpenAI(
        api_key='YOUR_API_KEY',
        base_url="http://xxxx/v1"
    )
    model_name = client.models.list().data[0].id
    response = client.chat.completions.create(
        model=model_name,
        messages=messages,
        max_tokens=1024,
    )
    print(response)
if __name__ == "__main__":
    send_request()
```

正常这么请求没问题，perf的请求结果我有空再打出来看下吧



> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！