Repository: modelscope/evalscope
Issue: KTransformers streaming output not recognized
Root cause: Validation Gap

https://github.com/modelscope/evalscope/blob/92eab36dca0916ff7984dd32481678875dc5fb52/evalscope/perf/utils/local_server.py#L44

This code shows that the streaming output should in the form of `data: xxx`, just like this (llama.cpp):

![Image](https://github.com/user-attachments/assets/3029afa6-fd00-41ec-8260-6693d0782f6c)

However, [KTransformers](https://github.com/kvcache-ai/ktransformers/) will output in streaming mode without the space, just like this:

![Image](https://github.com/user-attachments/assets/9eb8ae64-cdf6-48cc-ae00-2adb4cc449d8)


May I ask what command you are using? Is there any error message? From what I can see, whether there are spaces or not shouldn't affect the code logic.

https://github.com/modelscope/evalscope/blob/92eab36dca0916ff7984dd32481678875dc5fb52/evalscope/perf/utils/local_server.py#L43-L50

```bash
      evalscope perf --parallel 1 \
        --model DeepSeek-R1 \
        --url http://ds-r1-671b-kt:8080/v1/chat/completions \
        --log-every-n-query 5 --connect-timeout 6000 \
        --read-timeout 6000 --api openai --stream \
        --prompt '你是谁' \
        --max-tokens 7000 --number 10
```

```
2025-02-24 12:15:59,913 - evalscope - INFO - Connection successful.
2025-02-24 12:15:59,913 - evalscope - INFO - Save the data base to: outputs/20250224_121544/DeepSeek-R1/benchmark_data.db
Processing: 0it [00:00, ?it/s]2025-02-24 12:16:13,299 - evalscope - WARNING - No usage information found. Please specify `--tokenizer-path` to generate usage details.
2025-02-24 12:16:13,299 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/evalscope/perf/utils/benchmark_util.py", line 111, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
ZeroDivisionError: float division by zero
```

It seems that the deployed model service is not returning usage information, which is different from the standard OpenAI API format. You need to pass the --tokenizer-path parameter to calculate the number of tokens.

Yes I think you are right. I will close the issue.

> evalscope perf --parallel 1 \
>         --model DeepSeek-R1 \
>         --url http://ds-r1-671b-kt:8080/v1/chat/completions \
>         --log-every-n-query 5 --connect-timeout 6000 \
>         --read-timeout 6000 --api openai --stream \
>         --prompt '你是谁' \
>         --max-tokens 7000 --number 10
> ```
> 2025-02-24 12:15:59,913 - evalscope - INFO - Connection successful.
> 2025-02-24 12:15:59,913 - evalscope - INFO - Save the data base to: outputs/20250224_121544/DeepSeek-R1/benchmark_data.db
> Processing: 0it [00:00, ?it/s]2025-02-24 12:16:13,299 - evalscope - WARNING - No usage information found. Please specify `--tokenizer-path` to generate usage details.
> 2025-02-24 12:16:13,299 - evalscope - ERROR - float division by zero
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.10/dist-packages/evalscope/perf/utils/benchmark_util.py", line 111, in calculate_averages
>     self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
> ZeroDivisionError: float division by zero
> ```

@Yunnglin we should probably guard against unexpected input (with exception handling maybe) in such case as well.

> > evalscope perf --parallel 1 
> > --model DeepSeek-R1 
> > --url http://ds-r1-671b-kt:8080/v1/chat/completions 
> > --log-every-n-query 5 --connect-timeout 6000 
> > --read-timeout 6000 --api openai --stream 
> > --prompt '你是谁' 
> > --max-tokens 7000 --number 10
> > ```
> > 2025-02-24 12:15:59,913 - evalscope - INFO - Connection successful.
> > 2025-02-24 12:15:59,913 - evalscope - INFO - Save the data base to: outputs/20250224_121544/DeepSeek-R1/benchmark_data.db
> > Processing: 0it [00:00, ?it/s]2025-02-24 12:16:13,299 - evalscope - WARNING - No usage information found. Please specify `--tokenizer-path` to generate usage details.
> > 2025-02-24 12:16:13,299 - evalscope - ERROR - float division by zero
> > Traceback (most recent call last):
> >   File "/usr/local/lib/python3.10/dist-packages/evalscope/perf/utils/benchmark_util.py", line 111, in calculate_averages
> >     self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
> > ZeroDivisionError: float division by zero
> > ```
> 
> [@Yunnglin](https://github.com/Yunnglin) we should probably guard against unexpected input (with exception handling maybe) in such case as well.

OK, I will handle it later.