Repository: ray-project/llmperf
Issue: Subsequent requests cannot be sent until 'num_concurrent_requests' requests have all finished
Root cause: Architectural Constraint

Hello,

I've encountered an issue where the request launcher does not allow the next requests to be sent until all requests specified by `num_concurrent_requests` have finished.

This behavior seems counterintuitive for benchmarking TTFT and throughput in Continuous Batching systems accurately, as it can block subsequent requests even when the serving system is capable of handling them.

To address this, I believe the `get_next_ready` function should be modified as follows, enabling it to return results as soon as each individual request is completed:

```diff
--- a/src/llmperf/requests_launcher.py
+++ b/src/llmperf/requests_launcher.py
@@ -40,6 +40,7 @@ class RequestsLauncher:
         if not block:
             while self._llm_client_pool.has_next():
                 results.append(self._llm_client_pool.get_next_unordered())
+                return results
         else:
             while not self._llm_client_pool.has_next():
                 pass
```

I am prepared to submit a pull request with this change and would appreciate your feedback.

Thank you.

I found that the previous revision isn't sufficient to solve the problem. To send a request asynchronously right after the previous one finishes, many parts need fixing. I attempted to make the `get_next_ready` function asynchronous, but it depends on `ray.util.get_next_unordered()`. Converting it to an asynchronous function is challenging due to its current blocking implementation.

Here is the link to the relevant code: [ray/util/actor_pool.py lines 311-326](https://github.com/ray-project/ray/blob/b582905987632b38a7711ae8512a66545b538c75/python/ray/util/actor_pool.py#L311-L326).

I think there are two potential approaches for change:

- Convert the `get_next_ready` function into an asynchronous function.
- Enable clients to operate independently using multi-threading or multi-processing, while using current `get_next_ready` implementation.

Hey @llsj14 I'm facing the same issue - without issuing concurrent requests at a set rate, it's no longer a proper load testing framework, do you have plans to fix this? 

Hello @ashutoshsaboo,

I've made some changes to the load testing code to continuously send requests without waiting for all "num_concurrent_requests" to finish. Since modifying the core part related to Ray was challenging, I used multiple threads and request launchers, each holding a single client.

Code branch:
https://github.com/llsj14/llmperf/tree/fix/get-next-ready

Commit:
https://github.com/llsj14/llmperf/commit/d75db6934e08df8801171bd570bb763ced2cbf72

@llsj14 Saw your branch code, I'll test it. Can you please rebase your branch with the latest commit in this repo, your branch seems to be one commit behind from the main branch in this project? 

I think the latest commit missing from your branch - https://github.com/llsj14/llmperf/compare/main...ray-project%3Allmperf%3Amain - is useful for the dynamic prompts that it generates. 

@ashutoshsaboo
Thank you for letting me know. I have rebased my code.

@llsj14 Do you mind to explain your changes from 

`clients = construct_clients(llm_api=llm_api, num_clients=num_concurrent_requests)`

to fixed 1?

`clients = construct_clients(llm_api=llm_api, num_clients=1)`

I dunno but it appears to not be using all the CPUs when running my tests here...

Hi @dsantiago,

construct_clients depends on the Ray library, but it didn’t behave as I expected, it only sent requests after all clients received their final responses.

I wanted each client to send requests independently, immediately after receiving its final response. To achieve this, I set num_clients to 1 and used multi-threading to create multiple clients.

Thanks for the response @llsj14 . This projects seems dead anyways... I started benchmarking many GPU shapes and differnet models with it, but I'm now starting to do it via **[locust](https://github.com/locustio/locust)** project.

It also seem that the number of concurrent requests I needed  (from 1k to 5k concurrent request) is not easily achievable with Ray as it is made for distributed processing so I create a worker for each CPU (in a default manner) and it still can't reach the concurrency I need.

Yes, it would be possible to implement based on locust.
You can also check my PR for your purpose.
The request traffic also could be visualized like [this](https://github.com/ray-project/llmperf/pull/59#issuecomment-2524873877).