Repository: modelscope/evalscope
Issue: evalscope perf推理压测结果为 nan
Root cause: Validation Gap

加载本地 gsm8k数据，但是没成功加载的感觉
(vllm) root@autodl-container-625911aafa-00b94aab:~/autodl-tmp# evalscope perf \
  --parallel 1 10 50 100 200 \
  --number 10 20 100 200 400 \
  --model Qwen3-32B \
  --url http://127.0.0.1:8801/v1/chat/completions \
  --connect-timeout 2000 \
  --read-timeout 2000 \
  --api openai \
  --dataset openqa \
  --dataset-path /root/autodl-tmp/GSM8K.jsonl \
  --max-tokens 1024 \
  --min-tokens 1024 \
  --prefix-length 0 \
  --min-prompt-length 4096 \
  --max-prompt-length 4096 \
  --tokenizer-path /root/autodl-tmp/Qwen/Qwen3-32B \
  --debug \
  --stream
2025-06-12 11:24:42,895 - evalscope - INFO - Save the result to: outputs/20250612_112442/Qwen3-32B
2025-06-12 11:24:42,895 - evalscope - main.py - run_one_benchmark - 29 - INFO - Starting benchmark with args: 
2025-06-12 11:24:42,895 - evalscope - main.py - run_one_benchmark - 30 - INFO - {
    "model": "Qwen3-32B",
    "model_id": "Qwen3-32B",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": "/root/autodl-tmp/Qwen/Qwen3-32B",
    "port": 8877,
    "url": "http://127.0.0.1:8801/v1/chat/completions",
    "headers": {},
    "connect_timeout": 2000,
    "read_timeout": 2000,
    "api_key": null,
    "no_test_connection": false,
    "number": 10,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": true,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250612_112442/Qwen3-32B/parallel_1_number_10",
    "max_prompt_length": 4096,
    "min_prompt_length": 4096,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "dataset": "openqa",
    "dataset_path": "/root/autodl-tmp/GSM8K.jsonl",
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 1024,
    "min_tokens": 1024,
    "n_choices": null,
    "seed": 0,
    "stop": [],
    "stop_token_ids": [],
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-06-12 11:24:42,897 - evalscope - http_client.py - on_request_start - 112 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8801/v1/chat/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench')>)>
2025-06-12 11:24:42,901 - evalscope - http_client.py - on_request_chunk_sent - 124 - DEBUG - Request sent: <method='POST',  url=URL('http://127.0.0.1:8801/v1/chat/completions'), truncated_chunk='{"messages": [{"role": "user", "content": "hello"}], "model": "Qwen3-32B", "max_tokens": 10, "stream": true}'>
2025-06-12 11:24:43,076 - evalscope - http_client.py - _handle_stream - 48 - DEBUG - Response recevied: data: {"id":"chatcmpl-b570588faa654f4c87f570343edd9bea","object":"chat.completion.chunk","created":1749698682,"model":"Qwen3-32B","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}
2025-06-12 11:24:43,079 - evalscope - http_client.py - test_connection - 166 - INFO - Test connection successful.
2025-06-12 11:24:45,787 - evalscope - db_util.py - get_result_db_path - 106 - INFO - Save the data base to: outputs/20250612_112442/Qwen3-32B/parallel_1_number_10/benchmark_data.db
Processing:   0%|                                                                                                                                                        | 0/10 [00:00<?, ?it/s]
2025-06-12 11:24:45,854 - evalscope - db_util.py - summary_result - 221 - INFO - 
Benchmarking summary:
+-----------------------------------+---------+
| Key                               |   Value |
+===================================+=========+
| Time taken for tests (s)          |       1 |
+-----------------------------------+---------+
| Number of concurrency             |       1 |
+-----------------------------------+---------+
| Total requests                    |       0 |
+-----------------------------------+---------+
| Succeed requests                  |       0 |
+-----------------------------------+---------+
| Failed requests                   |       0 |
+-----------------------------------+---------+
| Output token throughput (tok/s)   |      -1 |
+-----------------------------------+---------+
| Total token throughput (tok/s)    |      -1 |
+-----------------------------------+---------+
| Request throughput (req/s)        |      -1 |
+-----------------------------------+---------+
| Average latency (s)               |      -1 |
+-----------------------------------+---------+
| Average time to first token (s)   |      -1 |
+-----------------------------------+---------+
| Average time per output token (s) |      -1 |
+-----------------------------------+---------+
| Average input tokens per request  |      -1 |
+-----------------------------------+---------+
| Average output tokens per request |      -1 |
+-----------------------------------+---------+
| Average package latency (s)       |      -1 |
+-----------------------------------+---------+
| Average package per request       |      -1 |
+-----------------------------------+---------+
2025-06-12 11:24:45,856 - evalscope - db_util.py - summary_result - 229 - INFO - 
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
|     25%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
|     50%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
|     66%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
|     75%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
|     80%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
|     90%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
|     95%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
|     98%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
|     99%     |   nan    |   nan   |   nan    |     nan     |     nan      |      nan      |      nan       |      nan      |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+

![Image](https://github.com/user-attachments/assets/8b98da7a-41b4-4464-a675-4dbb2abc839f)

提供的数据格式是否正确。openqa 使用jsonl文件的 question 字段作为prompt。不指定dataset_path将从modelscope自动下载[数据集](https://www.modelscope.cn/datasets/AI-ModelScope/HC3-Chinese/summary)