Repository: vllm-project/guidellm
Issue: support new api of  chat/compilation api
Root cause: Interface Contract Mismatch

**Is your feature request related to a problem? Please describe.**
guidellm cannot connect with `transformers serve`.
`transformers serve` use new openai api. it found on 
http://localhost:8000/v1/chat/completions
with new msg body:
```json
{
    "model": "meta-llama/Llama-3.2-1B-Instruct",
    "messages": [{"role":"user","content":"Who is the SpaceX CEO?"}],
    "max_tokens": 20,
    "stream": false
 }
```

**Describe the solution you'd like**
guidellm will be implemented the new openai api such as v1/chat/completions.

**Describe alternatives you've considered**
I consider to translate the msgs between of the shelf programs, but it another layer and it give more complexity.

**Additional context**



i tried such api with msg body and it works; pls refer to PR: https://github.com/vllm-project/guidellm/pull/328/files

guidellm doesn't work with the new api. 
I tried guidellm v0.3.0.

Here the setup. I am using cpu, but you don't have to.

Setup transformers server:
Extra steps that don't mention here, is to get access to the `meta-llama/Llama-3.2-1B-Instruct` model.
Also give your hugging face token to transformers to download the model.
```bash
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install transformers[serving]
  transformers serve \
    --device cpu \
    --host 0.0.0.0 \
    --port 8000 \
    --force-model meta-llama/Llama-3.2-1B-Instruct &
```

guidellm setup
```bash
pip install guidellm
            guidellm benchmark \
              --target "http://localhost:8000" \
              --rate-type concurrent \
              --rate 1 \
              --max-requests 24 \
              --data "prompt_tokens=128,output_tokens=64" 
```



> guidellm doesn't work with the new api. I tried guidellm v0.3.0.
> 
> Here the setup. I am using cpu, but you don't have to.
> 
> Setup transformers server: Extra steps that don't mention here, is to get access to the `meta-llama/Llama-3.2-1B-Instruct` model. Also give your hugging face token to transformers to download the model.
> 
> pip install torch --index-url https://download.pytorch.org/whl/cpu
> pip install transformers[serving]
>   transformers serve \
>     --device cpu \
>     --host 0.0.0.0 \
>     --port 8000 \
>     --force-model meta-llama/Llama-3.2-1B-Instruct &
> guidellm setup
> 
> pip install guidellm
>             guidellm benchmark \
>               --target "http://localhost:8000" \
>               --rate-type concurrent \
>               --rate 1 \
>               --max-requests 24 \
>               --data "prompt_tokens=128,outlut_tokens=64"

Yes, i reproduce your problem in local transformers server. And i found the problem is as such:
```
OpenAIHTTPBackend request with headers: {'Content-Type': 'application/json'} and params: {} and payload: {'messages': [{'role': 'user', 'content': 'Test connection'}], 'model': 'meta-llama/Llama-3.2-1B-Instruct', 'stream': True, 'stream_options': {'include_usage': True}, 'max_tokens': 1, 'max_completion_tokens': 1, 'stop': None, 'ignore_eos': True} failed: Client error '422 Unprocessable Entity' for url 'http://localhost:8000/v1/chat/completions'
```
transformers doesn't support params:  'stop': None, 'ignore_eos': True

so when i disable:

<img width="1221" height="916" alt="Image" src="https://github.com/user-attachments/assets/d8319c6c-78f4-4b65-9323-0d3e75a8b750" />

guidellm can do benchmark test successfully:

<img width="1128" height="854" alt="Image" src="https://github.com/user-attachments/assets/7ab6fdfd-61e6-4afa-bdd6-2cdea124b5ab" />

<img width="1129" height="1590" alt="Image" src="https://github.com/user-attachments/assets/a19aeabe-1ea6-4476-87da-9bf5666f8ef7" />

but i don't know how to fix this. @sjmonson Could you give me some idea?

Officially in the OpenAI chat-completion spec there is no way to control the minimum number of output tokens. This is one of the reasons we default to the legacy completion endpoint. You can pass `--backend_args "remove_from_body=['stop','ignore_eos']"` to remove the the fields from the request. However without enforcing a strict output length your benchmark results will be very inconstant (basically not recommended). 

Alternatively use a real-world dataset with semi-predictable responses from the model server and no `max_tokens`. This is not perfect either, the response length will still vary and **do not** compare the performance of different models this way as it will not be apples-to-apples.

Also here is a shameless plug for [vLLM](https://docs.vllm.ai/en/latest/) which supports the legacy `completions` endpoint and `chat/completions` with these options (GuideLLM is a subproject of vLLM).

Stale