Repository: modelscope/evalscope
Issue: event error bug
Root cause: Validation Gap

对Ktransformers 本地服务进行吞吐测试时遇到的bug. log 如下: 

2025-05-06 12:59:53,060 - evalscope - INFO - {
    "model": "unsloth/DeepSeek-R1-GGUF",
    "model_id": "DeepSeek-R1-GGUF",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "port": 8877,
    "url": "http://0.0.0.0:30000/v1/completions",
    "headers": {},
    "connect_timeout": 60000,
    "read_timeout": 60000,
    "api_key": null,
    "no_test_connection": false,
    "number": 1,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 5,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250506_125953/DeepSeek-R1-GGUF",
    "max_prompt_length": 9223372036854775807,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": false,
    "dataset": "speed_benchmark",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 48,
    "min_tokens": 48,
    "n_choices": null,
    "seed": 0,
    "stop": [],
    "stop_token_ids": [],
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-05-06 12:59:54,902 - evalscope - INFO - Test connection successful.
2025-05-06 12:59:54,902 - evalscope - INFO - Save the data base to: outputs/20250506_125953/DeepSeek-R1-GGUF/benchmark_data.db
2025-05-06 13:00:02,215 - evalscope - ERROR - Exception in async function 'statistic_benchmark_metric': 'index'
Traceback (most recent call last):
  File "/home/jinghuagao/anaconda3/envs/ktransformers/lib/python3.11/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jinghuagao/anaconda3/envs/ktransformers/lib/python3.11/site-packages/evalscope/perf/benchmark.py", line 137, in statistic_benchmark_metric
    metrics.update_metrics(benchmark_data, api_plugin)
  File "/home/jinghuagao/anaconda3/envs/ktransformers/lib/python3.11/site-packages/evalscope/perf/utils/benchmark_util.py", line 91, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
  File "/home/jinghuagao/anaconda3/envs/ktransformers/lib/python3.11/site-packages/evalscope/perf/utils/benchmark_util.py", line 45, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
  File "/home/jinghuagao/anaconda3/envs/ktransformers/lib/python3.11/site-packages/evalscope/perf/plugin/api/openai_api.py", line 121, in parse_responses
    self.__process_no_object(js, delta_contents)
  File "/home/jinghuagao/anaconda3/envs/ktransformers/lib/python3.11/site-packages/evalscope/perf/plugin/api/openai_api.py", line 147, in __process_no_object
    idx = choice['index']
          ~~~~~~^^^^^^^^^
KeyError: 'index'



+++++

启动参数为: 

evalscope perf \
 --parallel 1 \
 --url http://0.0.0.0:30000/v1/completions \
 --model unsloth/DeepSeek-R1-GGUF \
 --log-every-n-query 5 \
 --connect-timeout 60 \
 --read-timeout 6000 \
 --max-tokens 2048 \
 --min-tokens 2048 \
 --api openai \
 --dataset speed_benchmark \
 --number 1

看起来是因为ktransformer的返回结构不是openai api的标准格式。你可以在启动命令中加上 `--debug` 参数，打印一下模型的输出吗?

似乎是dataset 的原因:
--dataset speed_benchmark 导致的这个问题, 替换为
 --dataset random   --tokenizer-path  /data/jinghua/DeepSeek-R1 
之后bug消失. 


speed_benchmark 默认使用的是 completion接口而不是 chat/completion，导致了解析错误

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！