Repository: vllm-project/guidellm
Issue: Sweep mode with --max-seconds fails ungracefully if calculated RPS is too low
Root cause: Validation Gap

If sweep mode calculates an RPS that results in less then one request being sent out in `--max-second` range then there is a failure during processing.

```python
Traceback (most recent call last):
  File "/opt/app-root/bin/guidellm", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/guidellm/src/guidellm/__main__.py", line 242, in benchmark
    asyncio.run(
  File "/usr/lib64/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/src/guidellm/src/guidellm/benchmark/entrypoints.py", line 105, in benchmark_generative_text
    async for result in benchmarker.run(
  File "/src/guidellm/src/guidellm/benchmark/benchmarker.py", line 243, in run
    benchmark: BenchmarkT = aggregator.compile()
                            ^^^^^^^^^^^^^^^^^^^^
  File "/src/guidellm/src/guidellm/benchmark/aggregator.py", line 606, in compile
    return GenerativeBenchmark.from_stats(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/guidellm/src/guidellm/benchmark/benchmark.py", line 694, in from_stats
    start_time = min(req.start_time for req in total)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: min() iterable argument is empty
```


@sjmonson i tried to reproduce your failure, but came up with different error print

```
guidellm benchmark \
  --target "https://chat.d.run/" \
 --model "public/qwen2.5-72b-awq" \
  --processor "/Users/guangli.bao/temp/Qwen3-0.6B" \
  --rate-type sweep \
  --max-seconds 1 \
  --max-requests 1 \
  --data "prompt_tokens=128,output_tokens=56"
Creating backend...
Backend openai_http connected to https://chat.d.run/ for model public/qwen2.5-72b-awq.
Creating request loader...
Created loader with 1000 unique requests from prompt_tokens=128,output_tokens=56.


Traceback (most recent call last):
  File "/Users/guangli.bao/miniforge3/envs/myenv/bin/guidellm", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/go/guidellm/src/guidellm/__main__.py", line 314, in run
    asyncio.run(
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/go/guidellm/src/guidellm/benchmark/entrypoints.py", line 29, in benchmark_with_scenario
    return await benchmark_generative_text(**vars(scenario), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/go/guidellm/src/guidellm/benchmark/entrypoints.py", line 118, in benchmark_generative_text
    async for result in benchmarker.run(
  File "/Users/guangli.bao/go/guidellm/src/guidellm/benchmark/benchmarker.py", line 185, in run
    while scheduling_strategy := profile.next_strategy():
                                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/go/guidellm/src/guidellm/benchmark/profile.py", line 304, in next_strategy
    return AsyncConstantStrategy(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guangli.bao/go/guidellm/src/guidellm/objects/pydantic.py", line 28, in __init__
    super().__init__(**data)
  File "/Users/guangli.bao/miniforge3/envs/myenv/lib/python3.11/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for AsyncConstantStrategy
rate
  Input should be greater than 0 [type=greater_than, input_value=np.float64(0.0), input_type=float64]
    For further information visit https://errors.pydantic.dev/2.11/v/greater_than
╭─ Benchmarks ───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [14:14:… 1… synchronous (comple… Req:    0.0 req/s,    0.00s Lat,     0.0 Conc,       0 Comp,        1 Inc,        0 … │
│                                  Tok:    0.0 gen/s,    0.0 tot/s,   0.0ms TTFT,    0.0ms ITL,     0 Prompt,        0 … │
│ [14:14:… 1… throughput  (comple… Req:    0.0 req/s,    0.00s Lat,     0.0 Conc,       0 Comp,        1 Inc,        0 … │
│                                  Tok:    0.0 gen/s,    0.0 tot/s,   0.0ms TTFT,    0.0ms ITL,     0 Prompt,        0 … │
│ [--:--:…    constant@#… (pendin…                                                                                       │
│                                                                                                                        │
│ [--:--:…    constant@#… (pendin…                                                                                       │
│                                                                                                                        │
│ [--:--:…    constant@#… (pendin…                                                                                       │
│                                                                                                                        │
│ [--:--:…    constant@#… (pendin…                                                                                       │
│                                                                                                                        │
│ [--:--:…    constant@#… (pendin…                                                                                       │
│                                                                                                                        │
│ [--:--:…    constant@#… (pendin…                                                                                       │
│                                                                                                                        │
│ [--:--:…    constant@#… (pendin…                                                                                       │
│                                                                                                                        │
│ [--:--:…    constant@#… (pendin…                                                                                       │
│                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Generating... ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (2/10) [ 0:00:15 < 0:01:01]
```
my cmd: 
guidellm benchmark \
  --target "https://chat.d.run/" \
 --model "public/qwen2.5-72b-awq" \
  --processor "/Users/guangli.bao/temp/Qwen3-0.6B" \
  --rate-type sweep \
  --max-seconds 1 \
  --max-requests 1 \
  --data "prompt_tokens=128,output_tokens=56"

guidellm version: 0.4.0.dev2


i'm not sure the cause is the same that max-seconds is too low?

Yeah its still the same issue, just caught at a different place in the code now. The issue is the user ran a benchmark that resulted in not enough data. What we need is a step after each benchmark that checks data validity and gives the user a clear error message if something is wrong.

This issue is fixed in the latest main branch: guidellm version: 0.4.0.dev211, and can be closed.


Thanks @tukwila, closing out