Repository: ray-project/llmperf
Issue: How to use llmperf to test ollama performance (TTFT, etc)
Root cause: Documentation Deficiency

Hello,

How to use llmperf to test ollama performance (TTFT, etc)?

Thanks,
Alex

[Ollama implements an OpenAI compatible API](https://ollama.com/blog/openai-compatibility) so you might be able to benchmark against that using the [OpenAI benchmark instructions](https://github.com/ray-project/llmperf/tree/main?tab=readme-ov-file#openai-compatible-apis) from the llmperf README. For the `OPENAI_API_KEY` you can pass in anything, but I think it's required.

> [Ollama implements an OpenAI compatible API](https://ollama.com/blog/openai-compatibility) so you might be able to benchmark against that using the [OpenAI benchmark instructions](https://github.com/ray-project/llmperf/tree/main?tab=readme-ov-file#openai-compatible-apis) from the llmperf README. For the `OPENAI_API_KEY` you can pass in anything, but I think it's required.

First to confirm ollama is working by,

```
(base) alex@STX:~$ curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen3:8b",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'
{"id":"chatcmpl-875","object":"chat.completion","created":1749989023,"model":"qwen3:8b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"\u003cthink\u003e\nOkay, the user said \"Hello!\" so they're probably starting a conversation. I should respond in a friendly and welcoming manner. Let me make sure to greet them back and offer assistance. Maybe something like, \"Hello! How can I assist you today?\" That sounds good. It's polite and opens the door for them to ask for help. I don't need to add anything else unless they provide more context. Keep it simple and friendly.\n\u003c/think\u003e\n\nHello! How can I assist you today? 😊"},"finish_reason":"stop"}],"usage":{"prompt_tokens":21,"completion_tokens":106,"total_tokens":127}}

```


Then use  llmperf test ollama by openai-compatible-api, but failed.

```
(llmperf) alex@STX:~/Repo/llmperf$ export OPENAI_API_KEY=secret_abcdefg
export OPENAI_API_BASE="http://localhost:11434/v1/chat/completions"

python token_benchmark_ray.py \
--model "qwen3:8b" \
--mean-input-tokens 550 \
--stddev-input-tokens 150 \
--mean-output-tokens 150 \
--stddev-output-tokens 10 \
--max-num-completed-requests 2 \
--timeout 600 \
--num-concurrent-requests 1 \
--results-dir "result_outputs" \
--llm-api openai \
--additional-sampling-params '{}'
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Traceback (most recent call last):
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/connection.py", line 704, in connect
    self.sock = sock = self._new_conn()
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7487ef770550>: Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/hf-internal-testing/llama-tokenizer/tree/main/additional_chat_templates?recursive=False&expand=False (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7487ef770550>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/alex/Repo/llmperf/token_benchmark_ray.py", line 20, in <module>
    from llmperf.utils import (
  File "/home/alex/Repo/llmperf/src/llmperf/utils.py", line 63, in <module>
    tokenizer = LlamaTokenizerFast.from_pretrained(
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1968, in from_pretrained
    for template in list_repo_templates(
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/transformers/utils/hub.py", line 161, in list_repo_templates
    return [
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/transformers/utils/hub.py", line 161, in <listcomp>
    return [
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 3168, in list_repo_tree
    for path_info in paginate(path=tree_url, headers=headers, params={"recursive": recursive, "expand": expand}):
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/huggingface_hub/utils/_pagination.py", line 36, in paginate
    r = session.get(path, params=params, headers=headers)
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 96, in send
    return super().send(request, *args, **kwargs)
  File "/home/alex/miniconda3/envs/llmperf/lib/python3.10/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/hf-internal-testing/llama-tokenizer/tree/main/additional_chat_templates?recursive=False&expand=False (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7487ef770550>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), '(Request ID: 2b88a948-2a01-4d38-90e8-3e031a01fcc1)')
```



change OPENAI_API_BASE `http://localhost:11434/v1/chat/completions` to `http://localhost:11434/v1`
full env : `export OPENAI_API_BASE="http://localhost:11434/v1" ` 
then it's running normal.