zhiminy/EvalEng huggingface dataset:
Repository: qualifire-dev/rogue
Issue: [Bug] - Paste doesn't work in tui in `/eval` -> `Agent URL`
Root cause: Validation Gap

**Describe the bug**
Pasting a copied url doesn't work in the New Evaluation screen in the Agent URL field

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '/eval'
2. Copy some text
3. try to paste (cmd+v / Ctrl+v / Shift+insert)
4. See error

**Expected behavior**
Pasted text should appear

**Screenshots**

<img width="1220" height="554" alt="Image" src="https://github.com/user-attachments/assets/c3c0ce13-141c-40fc-a2e2-dc2e84a3c7ec" />

Hi @yuval-qf, I can work on that. Should the paste delete whatever already on the field?

@alonw0 Hi! actually this issue was fixed in #102  I forgot to tag this issue. Closing it. 

@drorIvry ok, great! 
As a side-note (and really only my opinion) I think the JudgeModel field shouldn't be allowed to paste into (as its mostly menu selection and any unknown string results in a nasty error)

@alonw0 That's a good call, want to take that issue?
Repository: qualifire-dev/rogue
Issue: [Feature] - Read TUI help from a markdown file instead of a static string
GitHub labels: good first issue

**Describe the solution you'd like**
In rogue-tui, in the `/help` screen, the help is currently rendered from several static strings.
We should refactor this to use a markdown file, for better maintainability.

This file also needs to be embedded into the binary (using `go:embed`)

Also please use the existing markdown renderer for the TUI found in `packages/tui/internal/tui/markdown_renderer.go`

@puwun would you like to take that issue? 

@drorIvry im working on another issue at the moment, will take this up once thats done  

@puwun awesome!! Thank you
Repository: ray-project/llmperf
Issue: Does llmperf support measuring local disk models? What's the meaning of framework in llmperf.py  line 355?
Root cause: Documentation Deficiency

I have downloaded llama2-13b-hf to my local disk.
I use this command "python llmperf.py -r 20 -m "../models/Llama-2-13b-chat-hf" to measure the performance of llama2-13b.
But an error occurred.
Traceback (most recent call last):
  File "llmperf.py", line 419, in <module>
    endpoint_config["api_base"] = os.environ["ANYSCALE_API_BASE"]
  File "/opt/conda/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'ANYSCALE_API_BASE'
So， please give me some guidance and advise.

It seems like I chose the default framework,but the default framework doesn't match my model.

You will probably need to setup an Open AI compatible model server to process the requests. 

The script currently only works as a mere client. 

going to close this for now as ricky has answered the question.
Repository: ray-project/llmperf
Issue: Crashing with large number of concurrent users
Root cause: Resource Mishandling

I'm finding that this tool often crashes when running tests at concurrency levels >500. 

Here's the command I'm running: `python llmperf.py -f openai -r 2000 -c 2000 -m "mistralai/Mistral-7B-v0.1"`

Here's the error log I'm seeing:

```
(pid=91343) [2023-11-30 19:53:26,142 E 91343 91820] gcs_rpc_client.h:552: Failed to connect to GCS within 60 seconds. GCS may have been killed. It's either GCS is terminated by `ray stop` or is killed unexpectedly. If it is killed unexpectedly, see the log file gcs_server.out. https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure. The program will terminate.
(pid=98374) E1130 19:53:52.486123357   99023 chttp2_transport.cc:2761]   keepalive_ping_end state error: 0 (expect: 1)
2023-11-30 19:53:53,141 WARNING worker.py:2074 -- The node with node id: 19a548a9a5a2ece8fe9589ae0846a32901d76bf544f5e286201bb770 and address: 10.168.0.2 and node name: 10.168.0.2 has been marked dead because the detector has missed too many heartbeats from it. This can happen when a        (1) raylet crashes unexpectedly (OOM, preempted node, etc.)
        (2) raylet has lagging heartbeats due to slow network or busy workload.
Traceback (most recent call last):
  File "/home/francescovirga/llmperf/llmperf.py", line 480, in <module>
    query_results = endpoint_evaluation(endpoint_config, sample_lines)
  File "/home/francescovirga/llmperf/llmperf.py", line 270, in endpoint_evaluation
    results = ray.get(futures)
  File "/home/francescovirga/.local/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/home/francescovirga/.local/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/home/francescovirga/.local/lib/python3.10/site-packages/ray/_private/worker.py", line 2565, in get
    raise value
ray.exceptions.LocalRayletDiedError: The task's local raylet died. Check raylet.out for more information.
(pid=98046) None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used. [repeated 74x across cluster]
(validate pid=68909) [2023-11-30 19:53:53,594 E 68909 69441] gcs_rpc_client.h:552: Failed to connect to GCS within 60 seconds. GCS may have been killed. It's either GCS is terminated by `ray stop` or is killed unexpectedly. If it is killed unexpectedly, see the log file gcs_server.out. https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure. The program will terminate. [repeated 205x across cluster]
(pid=85196) E1130 19:53:51.788929082   85763 chttp2_transport.cc:2761]   keepalive_ping_end state error: 0 (expect: 1) [repeated 666x across cluster]
```
Repository: ray-project/llmperf
Issue: litellm serializable issue ? 
Root cause: Documentation Deficiency

Hi saw this comment https://github.com/ray-project/llmperf/blob/1eac866f91773bff401f96e74c1cf20c38778329/src/llmperf/ray_clients/litellm_client.py#L14 

what's the issue you were running into with litellm serialization ? how can i repro this ?
Repository: ray-project/llmperf
Issue: Add memory bandwidth utilization metric
Root cause: Unimplemented Feature Gap

One of the key metrics in determining if the LLM inference server is performant is by looking at the memory bandwidth utilization. This is a function of the throughput and total GPU/accelerator HBM bandwidth. Calculation taken from PyTorch blog post here: https://pytorch.org/blog/accelerating-generative-ai-2/#step-2-alleviating-memory-bandwidth-bottleneck-through-int8-weight-only-quantization-1574-toks
Repository: ray-project/llmperf
Issue: Concurrency level is not handled properly
Root cause: Architectural Constraint

Currently, library makes it look as if it can support arbitrary concurrency levels, but in reality it actually doesn't:

 - Each actor require 1 CPU (ie number of actual concurrent workers will be limited by the number of CPUs in your Ray cluster)
 - We're not asserting number of actors produced (ie if Ray can't schedule all of them, user will never know)
 - Tasks to query the target endpoint are spun up from a single loop, therefore bound by the performance of submitting tasks to Ray from a single loop
Repository: ray-project/llmperf
Issue: Bug: Hugging Face TGI not working
Root cause: Interface Contract Mismatch

When trying to load test LLM deployed using hugging face TGI v1.4, using following commands:

```
export OPENAI_API_BASE=""
export OPENAI_API_KEY="test"
python3.9 token_benchmark_ray.py \
--model "mistralai/Mistral-7B-Instruct-v0.2" \
--mean-input-tokens 550 \
--stddev-input-tokens 150 \
--mean-output-tokens 150 \
--stddev-output-tokens 10 \
--max-num-completed-requests 100 \
--timeout 600 \
--num-concurrent-requests 5 \
--results-dir "result_outputs" \
--llm-api openai \
--additional-sampling-params '{}'
```

following error occurs:
```
(OpenAIChatCompletionsClient pid=82698) Warning Or Error: 422 Client Error: Unprocessable Entity for url: 
(OpenAIChatCompletionsClient pid=82698) 422
```

I could fix this error by replacing line 79 in openai_chat_completions_client.py with 
`stem = "data:"`
Repository: ray-project/llmperf
Issue: llmperf not working for concurrent users
Root cause: Resource Mishandling

Hi, We facing issue with llmperf when setting concurrent_users more than 1 user. The code gets stuck and doesnot progress further even after few mins. Attached screenshot below.

![image](https://github.com/ray-project/llmperf/assets/111358695/fa153090-113a-4475-a8e9-35619ab5d58c)

After sometime (~5mins) it fails with below error: (we set 2 concurrent users)

![image](https://github.com/ray-project/llmperf/assets/111358695/bd500e46-a7c0-44fc-9ced-154f41e5862c)

We checked even memory and cpu cosumption and we have still 75% available bandwidth to support.

Could someone please help us here to proceed further on this perf testing?

Thanks,
**Nihar**
Repository: ray-project/llmperf
Issue: bug of counting output tokens
Root cause: Algorithmic Error

In https://github.com/ray-project/llmperf/blob/main/token_benchmark_ray.py#L62, you have

```python
get_token_length = lambda text: len(tokenizer.encode(text))
````
and then use it to calculate the tokens of the generated text, like

```python
num_output_tokens = get_token_length(gen_text)
```

However, tokenzier will add special tokens during encoding phrase, here is an example

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

text = "hello world"

enc = tokenizer.encode(text)
print(enc) # [1, 22172, 3186]

dec = tokenizer.decode(enc)
print(dec) # 
 hello world


```

The token `1` should not be counted as one `generated` token here.

BTW, different models may have different special token rules, so it's hard to determine an easy way to get the real token number of the  generated text.
Repository: ray-project/llmperf
Issue: Blocking on pending requests despite block == false
Root cause: Algorithmic Error

I am using the `litellm` client to benchmark a HuggingFace TGI server.

In `token_benchmark_ray.py`, `req_launcher.get_next_ready()` is called periodically to fetch pending **results**, with the `block` parameter set to False.

However, the call is actually blocking until **all** pending **requests** are complete, which can be very long if I set a high number of concurrent requests (typically 128).

The result is that instead of continuously injecting new requests as they complete, the benchmark script instead sends a batch of max_concurrent_requests, waits for them to complete, then sends another batch.

Is this the expected behaviour ? I double-checked why the call is blocking and from the code in request launcher this seems to be the normal behaviour because it only checks if there are still **requests** in the ray actor pool.



I found that I have the same issue as you. (https://github.com/ray-project/llmperf/issues/56)
I think the `get_next_ready` function should return the result as soon as the request is finished in nonblock mode.
Repository: ray-project/llmperf
Issue: unable to get the benchmark result
Root cause: Validation Gap

HI all
while i was trying to run the ray code for checking the benchmark results, i am getting error and NAN values for all.
please let me know for your suggestions. 
Thanks in advance.

![Screenshot from 2024-04-04 10-45-19](https://github.com/ray-project/llmperf/assets/52161957/0d25b117-e81a-4123-a964-7a8f1bc2dc05)


any luck?
Repository: ray-project/llmperf
Issue: Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode PEP 660
Root cause: Architectural Constraint

llmperf has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.

was this resolved?
Repository: ray-project/llmperf
Issue: Vertex AI API needs to be updated.
Root cause: External Dependency Breakage

Team,

Looks like the Google Cloud Vertex AI API used might be outdated. 

Below is the current api in code
https://{region}-aiplatform.googleapis.com/v1/projects/{project_id}/locations/{region}/endpoints/{endpoint_id}:predict

Below is the latest API that works [have tested by updating code on my end]
https://{region}-aiplatform.googleapis.com/v1/projects/{project_id}/locations/{region}/publishers/google/models/{model}:predict

File: src/llmperf/ray_clients/vertexai_client.py

Thanks
Repository: ray-project/llmperf
Issue: error in request_metrics dictionary implementation
Root cause: Validation Gap

Team,

While running the "token_benchmark_ray.py" on a model in google cloud vertex AI I noticed that the division on line 111 is failing because the division is like 

request_metrics[common_metrics.INTER_TOKEN_LAT] /= num_output_tokens
or, request_metrics[common_metrics.INTER_TOKEN_LAT] = request_metrics[common_metrics.INTER_TOKEN_LAT]/ num_output_tokens
or, request_metrics[common_metrics.INTER_TOKEN_LAT] = []/1

where, 
common_metrics.INTER_TOKEN_LAT = inter_token_latency_s
As we can see from the example response below inter_token_latency_s = []
num_output_tokens = 1

For Example:

Below is a sample of "request_metrics" obtained during the api call

{'error_code': 200, 'error_msg': "'dict' object has no attribute 'split'", 'inter_token_latency_s': [], 'ttft_s': 0, 'end_to_end_latency_s': 1.4629004680000435, 'request_output_throughput_token_per_s': 0, 'number_total_tokens': 538, 'number_output_tokens': 0, 'number_input_tokens': 538}

File: token_benchmark_ray.py

Thanks
Repository: ray-project/llmperf
Issue: Sagemaker client issue
Root cause: Interface Contract Mismatch

when i am executing token_benchmark_ray.py we are getting below error
  File "token_benchmark_ray.py", line 456, in <module>
    run_token_benchmark(
  File "token_benchmark_ray.py", line 297, in run_token_benchmark
    summary, individual_responses = get_token_throughput_latencies(
  File "token_benchmark_ray.py", line 111, in get_token_throughput_latencies
    request_metrics[common_metrics.INTER_TOKEN_LAT] /= num_output_tokens
TypeError: unsupported operand type(s) for /=: 'list' and 'int'
(SageMakerClient pid=15473) Warning Or Error: 'SageMakerRuntime' object has no attribute 'invoke_endpoint_with_response_stream'
(SageMakerClient pid=15473) None

Hey @SuchethaChintha  did you fix that ?


This is probably due to using an older version of the Sagemaker SDK. Updating it should fix the issue.

It seems that this error occurs because there's an inconsistency in how `INTER_TOKEN_LAT` is handled between different LLM clients.

SageMaker client keeps INTER_TOKEN_LAT as a list
https://github.com/ray-project/llmperf/blob/f1d6bed47e4501b0e371082b41601b59ab55269f/src/llmperf/ray_clients/sagemaker_client.py#L109

On the other hand, OpenAI client sums the latencies before returning
https://github.com/ray-project/llmperf/blob/f1d6bed47e4501b0e371082b41601b59ab55269f/src/llmperf/ray_clients/openai_chat_completions_client.py#L112

I think that if you modify the source code for sagemaker_client.py as follows, it will work correctly.
```py
metrics[common_metrics.INTER_TOKEN_LAT] = sum(time_to_next_token)
```

Even if you make this change, INTER_TOKEN_LAT is divided by the number of output tokens in token_benchmark_ray.py, so the correct metrics should be calculated.
Repository: ray-project/llmperf
Issue: Divide by zero: request_metrics[common_metrics.REQ_OUTPUT_THROUGHPUT] = num_output_tokens / request_metrics[common_metrics.E2E_LAT]
Root cause: Validation Gap

Running the benchmark script on a llama-3-8b-inst on inferentia 2 (djl-serving) results in:

```
python3.10 token_benchmark_ray.py \                                           
--model "openai/llama3-8b-inst" \
--mean-input-tokens 550 \
--stddev-input-tokens 150 \
--mean-output-tokens 150 \
--stddev-output-tokens 10 \
--max-num-completed-requests 1 \  
--timeout 600 \
--num-concurrent-requests 1 \ 
--results-dir "result_outputs" \
--llm-api "openai" \
--additional-sampling-params '{}'
```

```
Traceback (most recent call last):
  File "/Users/yaron/projects/llmperf/token_benchmark_ray.py", line 456, in <module>
    run_token_benchmark(
  File "/Users/yaron/projects/llmperf/token_benchmark_ray.py", line 297, in run_token_benchmark
    summary, individual_responses = get_token_throughput_latencies(
  File "/Users/yaron/projects/llmperf/token_benchmark_ray.py", line 116, in get_token_throughput_latencies
    request_metrics[common_metrics.REQ_OUTPUT_THROUGHPUT] = num_output_tokens / request_metrics[common_metrics.E2E_LAT]
ZeroDivisionError: division by zero
```

same problem here, what did you do to fix this?

same problem here when testing a api deployed in house

same problem here when testing a api deployed in house

same problem here when testing a api deployed in house

same problem
(OpenAIChatCompletionsClient pid=164504) 422
(OpenAIChatCompletionsClient pid=164507) 422
(OpenAIChatCompletionsClient pid=164510) 422
(OpenAIChatCompletionsClient pid=164506) 422
(OpenAIChatCompletionsClient pid=164500) 422
(OpenAIChatCompletionsClient pid=164502) 422
(OpenAIChatCompletionsClient pid=164498) 422
Traceback (most recent call last):
  File "/home/llmperf/token_benchmark_ray.py", line 462, in <module>
    run_token_benchmark(
  File "/home/llmperf/token_benchmark_ray.py", line 303, in run_token_benchmark
    summary, individual_responses = get_token_throughput_latencies(
  File "/home/llmperf/token_benchmark_ray.py", line 122, in get_token_throughput_latencies
    request_metrics[common_metrics.REQ_OUTPUT_THROUGHPUT] = num_output_tokens / request_metrics[common_metrics.E2E_LAT]
ZeroDivisionError: division by zero
(OpenAIChatCompletionsClient pid=164508) 422
(OpenAIChatCompletionsClient pid=164508) Warning Or Error: 422 Client Error: Unprocessable Content for url: http://x.x.x.x:1025/v1/chat/completions

same problem here, how to fix this?

> same problem here, how to fix this?

try print the response code and content

https://github.com/ray-project/llmperf/blob/main/src/llmperf/ray_clients/openai_chat_completions_client.py#L23
My deployed api cannot accept empty request content, delete this line or add some content can fix the 422 error

Same problem when testing deepseek-r1 API.

In addition, using deepseek-v3 API can work well.

same problem when tesing deepseek-r1-32b

Here is the fix for this issue: https://github.com/ray-project/llmperf/pull/93
Repository: ray-project/llmperf
Issue: When max-num-completed-requests is not divisible by num-concurrent-requests, an error will occur.
Root cause: Validation Gap

Traceback (most recent call last):
  File "token_benchmark_ray.py", line 463, in <module>
    run_token_benchmark(
  File "token_benchmark_ray.py", line 304, in run_token_benchmark
    summary, individual_responses = get_token_throughput_latencies(
  File "token_benchmark_ray.py", line 99, in get_token_throughput_latencies
    default_sampling_params = {"max_tokens": num_output_tokens_list.pop()}
IndexError: pop from empty list
Repository: ray-project/llmperf
Issue: Contributions guidelines

Hi,

is llmperf accepting contributions from the community? I see quite a lot of Pull Requests and issues without traction and no section about contributions in the README, so I would like to know if this repository is being actively developed and whether it accepts contributions. I'm interested in making a few improvements/fixing a few issues, but don't want to invest time in case it ends up in a dead queue.

Cheers,
Milosz
Repository: ray-project/llmperf
Issue: Error while using pip
Root cause: Unimplemented Feature Gap

using ubuntu with python3
>pip install llmperf

ERROR: Could not find a version that satisfies the requirement llmperf (from versions: none)
ERROR: No matching distribution found for llmperf

This is because llmperf is not (yet?) on pypi:
https://pypi.org/search/?q=llmperf
Repository: ray-project/llmperf
Issue: Wrong generated prompt length  
Root cause: Algorithmic Error

I think there is an error when calculating the number of input tokens at prompt-time generation in the function `llmperf.utils.randomly_sample_sonnet_lines_prompt()`. Line `112`: ` line_to_add = line_to_add[: int(math.ceil(remaining_prompt_tokens))]`. If I understand correctly, you want to make sure the prompt you generate is of a specific length in terms of tokens., therefore if the new line to add to the prompt is too long, you want to cut it and get rit of the extra tokens. The code checks whether this happens and then attempts to remove the extra chars from `line_to_add`. The way this is done seems wrong to be, because it reasons in terms of characters and not tokens, which is a mistake. The correct approach would be to count the number of tokens in `line_to_add` and keep only the prefix which has lenght equal to `remaining_prompt_tokens` in terms of tokens, and not of characters as it is now.

Does this make sense to you?
Repository: ray-project/llmperf
Issue: Support other scenarios
Root cause: Unimplemented Feature Gap

Current implementation of `request_launcher.py` supports only [multi-stream like](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios) scenario. Is there any plan to support other scenarios? Or if you don't mind, I can add some implementations and cli option (single stream and multi stream).

(Is this project dead? Last update is 4 month ago, and I cannot find the author's reply in any issue or pull request.)
Repository: ray-project/llmperf
Issue: commit: f1d6bed , In some scenarios, the number of requests will double and the input length will be fixed at two number
Root cause: Algorithmic Error

**request command:**
export OPENAI_API_KEY=EMPTY &&  export OPENAI_API_BASE=http://127.0.0.1:26894/v1/ && python3 /workspace/llmperf/token_benchmark_ray.py --model Qwen/Qwen2.5-7B-Instruct --mean-input-tokens 2048 --stddev-input-tokens 1024 --mean-output-tokens 2048  --stddev-output-tokens 1024  --max-num-completed-requests 20  --timeout 36000  --num-concurrent-requests 20  --results-dir './'  --llm-api openai --additional-sampling-params '{"ignore_eos": true}'

**expect**
Hope to complete 20 concurrent requests,  The input length is randomly distributed according to Gaussian distribution

**issue**
1. but the actual server received 40 requests

......
 26 INFO:     127.0.0.1:42840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 27 INFO:     127.0.0.1:42846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 28 INFO:     127.0.0.1:42862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 29 INFO:     127.0.0.1:42878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 30 INFO:     127.0.0.1:42894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 31 INFO:     127.0.0.1:42902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 32 INFO:     127.0.0.1:42916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 33 INFO:     127.0.0.1:42924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 34 INFO:     127.0.0.1:42938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 35 INFO:     127.0.0.1:42948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 36 INFO:     127.0.0.1:42956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 37 INFO:     127.0.0.1:42966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 38 INFO:     127.0.0.1:42982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 39 INFO:     127.0.0.1:42994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 40 INFO:     127.0.0.1:43008 - "POST /v1/chat/completions HTTP/1.1" 200 OK



2. From the log file Qwen-Qwen2-5-7B-Instruct_2048_2048_individual_responses.json, it can be seen that there were 20 requests counted (40 requests actually received by the server), and these 20 requests were fixed at two input lengths (4896 and 5369)

![Image](https://github.com/user-attachments/assets/b951e93d-9ed9-4a55-87d3-a27c6ec62f5c)

Did you found anything about your problem? Interested on it but it appears this project is kind of dead...