Repository: modelscope/evalscope
Issue: evalscope perf 测试sglang 部署的openai api server 无法输出结果
Root cause: Interface Contract Mismatch

版本：evalscope 0.5.3 
           sglang 0.3.0

在本地起了一个sglang的openai api server，命令如下：
CUDA_VISIBLE_DEVICES=4,5,6,7 python -m sglang.launch_server --model-path /local/models/Qwen2-72B-Instruct --tp 4 --host 0.0.0.0
能够正常访问，用下面的语句进行benchmark测试：
evalscope perf --url 'http://localhost:30000/v1/chat/completions' --parallel 128 --model '/local/models/Qwen2-72B-Instruct' --log-every-n-query 10 --read-timeout=120 --dataset-path './dataset/open_qa.jsonl' -n 50 --max-prompt-length 128000 --api openai --stream --stop '
' --dataset openqa --debug

server端有输出：
![image](https://github.com/user-attachments/assets/f2663c84-fa66-4496-8e3a-2a3a96a82214)

测试脚本到这个位置就不动了
![image](https://github.com/user-attachments/assets/818b310f-7f36-4e76-a443-73ecda137fa5)

同样的命令在测试vllm时可以正常输出结果，但是测试sglang时无法生成结果。
麻烦帮忙看看问题出在哪里，多谢

我们后面复现一下

您可以拉取main分支代码，来使用最新的perf模块尝试，参考[使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/index.html)。



测试mindie时遇到同样的问题

> 测试mindie时遇到同样的问题

麻烦提供一下执行命令和--debug模式的控制台输出

我在使用evalscope 最新版本 0.8.1压测sglang部署的大模型报如下错误：
我执行的命令及报错如下：

```shell
# evalscope perf --url "http://192.168.171.166:21840/v1/chat/completions" --parallel 1 --model Qwen2.5-14B-Instruct --number 30 --max-tokens 2048 --temperature 0.7 --api openai --dataset openqa --stream 

报错信息如下：
2024-12-25 09:57:13,439 - evalscope - http_client.py - _handle_stream - 52 - DEBUG - Response recevied: data: [DONE]
Processing: 0it [00:01, ?it/s]
2024-12-25 09:57:13,440 - evalscope - handler.py - async_wrapper - 19 - ERROR - Exception in async function 'statistic_benchmark_metric_worker': 'object'
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/benchmark.py", line 167, in statistic_benchmark_metric_worker
    metrics.update_metrics(benchmark_data, api_plugin)
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/utils/benchmark_util.py", line 86, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/utils/benchmark_util.py", line 43, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/plugin/api/openai_api.py", line 116, in parse_responses
    if js['object'] == 'chat.completion':
KeyError: 'object'
```
问题应该是使用了stream流式输出导致的问题，不启用流式输出压测是正常的。

我们复现并修复一下

这是因为sglang的api返回有这种没带object的数据 data: {"id":"89467d81d1ea4435b74a34651faeedb4","model":"Qwen2.5-14B-Instruct","choices":[],"usage":{"prompt_tokens":44,"total_tokens":328,"completion_tokens":284}} 这种没有带object需要处理一下，我改了一下perf/plugin/api/openai_api.py文件中的parse_responses方法，判断接收的数据是否包含object属性，没有的话补一下就可以了。
```code
    def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
        """Parser responses and return number of request and response tokens.
           sample of the output delta:
           {"id":"4","object":"chat.completion.chunk","created":1714030870,"model":"llama3","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}


        Args:
            responses (List[bytes]): List of http response body, for stream output,
                there are multiple responses, for general only one.
            kwargs: (Any): The command line --parameter content.
        Returns:
            Tuple: Return number of prompt token and number of completion tokens.
        """
        full_response_content = ''
        delta_contents = {}
        input_tokens = None
        output_tokens = None
        for response in responses:
            js = json.loads(response)
           # 判断object是否存在
            if 'object' not in js:
                logger.warning(f"Response does not contain 'object' field: {js}")
                js['object'] = 'chat.completion.chunk'
            #if 'created' not in js:
            #    logger.warning(f"Response does not contain 'created' field: {js}")
            #    js['created'] = int(time.time())
            if js['object'] == 'chat.completion':
                for choice in js['choices']:
                    delta_contents[choice['index']] = [choice['message']['content']]
                input_tokens = js['usage']['prompt_tokens']
                output_tokens = js['usage']['completion_tokens']
            elif js['object'] == 'text_completion':
                for choice in js['choices']:
                    delta_contents[choice['index']] = [choice['text']]
                input_tokens = js['usage']['prompt_tokens']
                output_tokens = js['usage']['completion_tokens']
            elif js['object'] == 'chat.completion.chunk':
                if 'choices' in js:
                    for choice in js['choices']:
                        if 'delta' in choice and 'index' in choice:
                            delta = choice['delta']
                            idx = choice['index']
                            if 'content' in delta:
                                delta_content = delta['content']
                                if idx in delta_contents:
                                    delta_contents[idx].append(delta_content)
                                else:
                                    delta_contents[idx] = [delta_content]
                # usage in chunk: {"id":"","object":"chat.completion.chunk","created":1718269986,"model":"llama3",
                # "choices":[],"usage":{"prompt_tokens":32,"total_tokens":384,"completion_tokens":352}}
                if 'usage' in js and js['usage']:
                    input_tokens = js['usage']['prompt_tokens']
                    output_tokens = js['usage']['completion_tokens']
        if (input_tokens is None and output_tokens is None and self.tokenizer is not None):
            input_tokens = 0
            output_tokens = 0
            for idx, choice_contents in delta_contents.items():
                full_response_content = ''.join([m for m in choice_contents])
                input_tokens += len(self.tokenizer.encode(request['messages'][0]['content']))
                output_tokens += len(self.tokenizer.encode(full_response_content))
        elif input_tokens is None and output_tokens is None:  # no usage info get.
            input_tokens = 0
            output_tokens = 0
            logger.warning('No usage information found. Please specify `--tokenizer-path` to generate usage details.')

        return input_tokens, output_tokens
```

> 这是因为sglang的api返回有这种没带object的数据 data: {"id":"89467d81d1ea4435b74a34651faeedb4","model":"Qwen2.5-14B-Instruct","choices":[],"usage":{"prompt_tokens":44,"total_tokens":328,"completion_tokens":284}} 这种没有带object需要处理一下，我改了一下perf/plugin/api/openai_api.py文件中的parse_responses方法，判断接收的数据是否包含object属性，没有的话补一下就可以了。
> 
> ```
>     def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
>         """Parser responses and return number of request and response tokens.
>            sample of the output delta:
>            {"id":"4","object":"chat.completion.chunk","created":1714030870,"model":"llama3","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}
> 
> 
>         Args:
>             responses (List[bytes]): List of http response body, for stream output,
>                 there are multiple responses, for general only one.
>             kwargs: (Any): The command line --parameter content.
>         Returns:
>             Tuple: Return number of prompt token and number of completion tokens.
>         """
>         full_response_content = ''
>         delta_contents = {}
>         input_tokens = None
>         output_tokens = None
>         for response in responses:
>             js = json.loads(response)
>            # 判断object是否存在
>             if 'object' not in js:
>                 logger.warning(f"Response does not contain 'object' field: {js}")
>                 js['object'] = 'chat.completion.chunk'
>             #if 'created' not in js:
>             #    logger.warning(f"Response does not contain 'created' field: {js}")
>             #    js['created'] = int(time.time())
>             if js['object'] == 'chat.completion':
>                 for choice in js['choices']:
>                     delta_contents[choice['index']] = [choice['message']['content']]
>                 input_tokens = js['usage']['prompt_tokens']
>                 output_tokens = js['usage']['completion_tokens']
>             elif js['object'] == 'text_completion':
>                 for choice in js['choices']:
>                     delta_contents[choice['index']] = [choice['text']]
>                 input_tokens = js['usage']['prompt_tokens']
>                 output_tokens = js['usage']['completion_tokens']
>             elif js['object'] == 'chat.completion.chunk':
>                 if 'choices' in js:
>                     for choice in js['choices']:
>                         if 'delta' in choice and 'index' in choice:
>                             delta = choice['delta']
>                             idx = choice['index']
>                             if 'content' in delta:
>                                 delta_content = delta['content']
>                                 if idx in delta_contents:
>                                     delta_contents[idx].append(delta_content)
>                                 else:
>                                     delta_contents[idx] = [delta_content]
>                 # usage in chunk: {"id":"","object":"chat.completion.chunk","created":1718269986,"model":"llama3",
>                 # "choices":[],"usage":{"prompt_tokens":32,"total_tokens":384,"completion_tokens":352}}
>                 if 'usage' in js and js['usage']:
>                     input_tokens = js['usage']['prompt_tokens']
>                     output_tokens = js['usage']['completion_tokens']
>         if (input_tokens is None and output_tokens is None and self.tokenizer is not None):
>             input_tokens = 0
>             output_tokens = 0
>             for idx, choice_contents in delta_contents.items():
>                 full_response_content = ''.join([m for m in choice_contents])
>                 input_tokens += len(self.tokenizer.encode(request['messages'][0]['content']))
>                 output_tokens += len(self.tokenizer.encode(full_response_content))
>         elif input_tokens is None and output_tokens is None:  # no usage info get.
>             input_tokens = 0
>             output_tokens = 0
>             logger.warning('No usage information found. Please specify `--tokenizer-path` to generate usage details.')
> 
>         return input_tokens, output_tokens
> ```

it would be great it you just submit a pull-request :)

哈哈，我尝试提了一个pull-request #260  @yingdachen  

> 哈哈，我尝试提了一个pull-request #260 @yingdachen

merged ~ you can pull the main branch and perform regression testing :)

> 由于长时间未收到活动，我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！