Repository: modelscope/evalscope
Issue: failed to run perf test for vLLM PD example proxy server, vLLM bench works fine
Root cause: Validation Gap

## Self-Check List

Before submitting an issue, please ensure you have completed the following steps:
- [x] I have carefully read the [relevant user documentation](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)
- [x] I have reviewed the [Frequently Asked Questions](https://evalscope.readthedocs.io/en/latest/get_started/faq.html)
- [x] I have searched and reviewed existing issues to confirm this is not a duplicate problem

## Problem Description

failed to run perf test for vLLM PD example proxy server, vLLM bench works fine

## EvalScope Version (Required)
```
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# pip show evalscope
Name: evalscope
Version: 0.16.3
Summary: EvalScope: Lightweight LLMs Evaluation Framework
Home-page: https://github.com/modelscope/evalscope
Author: ModelScope team
Author-email: contact@modelscope.cn
License: 
Location: /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages
Requires: accelerate, datasets, immutabledict, jieba, jsonlines, langdetect, latex2sympy2-extended, matplotlib, modelscope, nltk, openai, pandas, pillow, pyarrow, pyyaml, requests, rouge-chinese, rouge-score, sacrebleu, scikit-learn, seaborn, sympy, tabulate, torch, tqdm, transformers, word2number
Required-by:
```
## Tools Used
- [ ] Native / Native framework
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / Model inference stress testing tool
- [ ] Arena / Arena mode

## Executed Code or Instructions

follows https://github.com/vllm-project/vllm/issues/20477 to understand background for this issue.

## Error Log

```
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# evalscope perf  --url "http://127.0.0.1:8000/v1/chat/completions"  --parallel 1 --model mistralai/Mistral-7B-Instruct-v0.2 --number 2 --api openai --dataset longalpaca --read-timeout 600 --connect-timeout 600 --log-every-n-query 20
2025-07-04 08:17:51,598 - evalscope - INFO - Save the result to: outputs/20250704_081751/Mistral-7B-Instruct-v0.2

    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250704_081751/Mistral-7B-Instruct-v0.2",
    "max_prompt_length": 9223372036854775807,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "dataset": "longalpaca",
    "dataset_path": null,
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-07-04 08:17:53,119 - evalscope - INFO - Test connection successful.
2025-07-04 08:18:07,736 - evalscope - INFO - Save the data base to: outputs/20250704_081751/Mistral-7B-Instruct-v0.2/benchmark_data.db
Processing:   0%|                                                                                                                             | 0/2 [00:08<?, ?it/s]
2025-07-04 08:18:16,649 - evalscope - ERROR - Exception in async function 'statistic_benchmark_metric': Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 140, in statistic_benchmark_metric
    metrics.update_metrics(benchmark_data, api_plugin)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 109, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 45, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/plugin/api/openai_api.py", line 110, in parse_responses
    last_response_js = json.loads(responses[-1])
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/bin/evalscope", line 8, in <module>
    sys.exit(run_cmd())
             ^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/cli/cli.py", line 25, in run_cmd
    cmd.execute()
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/cli/start_perf.py", line 33, in execute
    run_perf_benchmark(self.args)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/main.py", line 94, in run_perf_benchmark
    return run_one_benchmark(args, output_path=output_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/main.py", line 41, in run_one_benchmark
    return loop.run_until_complete(benchmark(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/lib/python3.12/asyncio/base_events.py", line 1987, in _run_once
    handle._run()
  File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 20, in async_wrapper
    sys.exit(1)
SystemExit: 1

Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 89, in send_request
    async with semaphore:
  File "/usr/lib/python3.12/asyncio/locks.py", line 20, in __aexit__
    self.release()
  File "/usr/lib/python3.12/asyncio/locks.py", line 406, in release
    self._wake_up_next()
  File "/usr/lib/python3.12/asyncio/locks.py", line 416, in _wake_up_next
    fut.set_result(True)
  File "/usr/lib/python3.12/asyncio/base_events.py", line 795, in call_soon
    self._check_closed()
  File "/usr/lib/python3.12/asyncio/base_events.py", line 541, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <coroutine object send_request at 0x767f18bb7a40>
Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 20, in async_wrapper
    sys.exit(1)
SystemExit: 1
2025-07-04 08:18:16,784 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-5' coro=<send_request() done, defined at /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py:14> cb=[gather.<locals>._done_callback() at /usr/lib/python3.12/asyncio/tasks.py:767]>
2025-07-04 08:18:16,784 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-6' coro=<send_request() done, defined at /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py:14> wait_for=<Future finished result=True> cb=[gather.<locals>._done_callback() at /usr/lib/python3.12/asyncio/tasks.py:767]>
```

## Running Environment

- Operating System:
- Python Version:

## Additional Information

If there is any other relevant information, please provide it here.