Repository: explodinggradients/ragas
Issue: Async Executor/Runner slows to a halt with jobs that auto-retry with default (high) `max_wait`
Root cause: Architectural Constraint

**Describe the bug**
Anything that a) has a throttle limit and b) uses `Executor` (and in turn, `Runner`), such as OpenAI's API, will slow to a halt if too many jobs are requested.

Specifically, in my case I was attempting to generate testset for ~1k documents.
```Python
testset = generator.generate_with_langchain_docs(documents, test_size=N, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
```
This refuses to even start if either there are too many documents (roughly >300) or too high `test_size`. Example with 500 docs stuck at 0% after 20min:
![image](https://github.com/explodinggradients/ragas/assets/6729737/48e9d2e8-f759-4b1d-a99b-84b4a55460ab)

Reducing `test_size` didn't help with high document count because the place that got stuck was at [docstore.add_documents](https://github.com/explodinggradients/ragas/blob/c18c7f4c9a4bb0f761a0cac916178b349b5705e1/src/ragas/testset/generator.py#L150). However, reducing both document count and `test_size` to < 100 did get it going at reasonable speed.

Ragas version: `0.1.2.dev8+gc18c7f4`
Python version: `3.9.13`

**Code to Reproduce**
Below is just one example that uses `Executor` & `Runner` with higher job count (i.e. 1,000). Document contens averaged ~800 characters. OpenAI API is being used as LLM.
```Python
testset = generator.generate_with_langchain_docs(documents, test_size=N, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
```

**Error trace**
No error, just gets stuck.

**Expected behavior**
Don't get stuck with high job count.

**Additional context**
Will add my findings in comments below. Related: #394

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


**Problem Investigation**
The `Executor` async job code was initially introduced by bad3a8e1d115e69bb4acbd9887ca5b6cb6a03fba and later separated into `Runner`. The problem is that `Runner` starts all jobs at once ([here](https://github.com/explodinggradients/ragas/blob/c18c7f4c9a4bb0f761a0cac916178b349b5705e1/src/ragas/executor.py#L45)) and then later uses a `asyncio.as_completed()` ([here](https://github.com/explodinggradients/ragas/blob/c18c7f4c9a4bb0f761a0cac916178b349b5705e1/src/ragas/executor.py#L50)) to wait for all jobs to be completed, but doesn't impose any max concurrency.

When too many jobs (i.e. >hundreds) are calling the same API, a throughput limit is hit and many of the jobs need to wait & retry. With the default `RunConfig`, wait time can be up to ~60s~ 90s. I suspect this is the culrprit of the stuckness.

**Proposal**
The `Executor`/`Runner` should facilitate a maximum concurrency limit so as to avoid kicking all the remaining (and possibly on-the-fly?) requests into wait & retry loop once throughput limit gets hit. Even if that's not a good reason (for the admittedly lack of tracing down the actual reason of getting stuck), it's still a good knob to have for users who don't want to be constantly hitting throughput limits.

PR created: #643 please review.

In case this doesn't get attention, excuse my mention @jjmachan 


> **Problem Investigation** The `Executor` async job code was initially introduced by [bad3a8e](https://github.com/explodinggradients/ragas/commit/bad3a8e1d115e69bb4acbd9887ca5b6cb6a03fba) and later separated into `Runner`. The problem is that `Runner` starts all jobs at once ([here](https://github.com/explodinggradients/ragas/blob/c18c7f4c9a4bb0f761a0cac916178b349b5705e1/src/ragas/executor.py#L45)) and then later uses a `asyncio.as_completed()` ([here](https://github.com/explodinggradients/ragas/blob/c18c7f4c9a4bb0f761a0cac916178b349b5705e1/src/ragas/executor.py#L50)) to wait for all jobs to be completed, but doesn't impose any max concurrency.
> 
> When too many jobs (hundreds) are calling the same API, a throughput limit is hit and many of the jobs need to wait & retry. With the default `RunConfig`, wait time can be up to 60s. I suspect this is the culrprit of the stuckness.
> 
> **Proposal** The `Executor`/`Runner` should facilitate a maximum concurrency limit so as to avoid kicking all the remaining requests (once throughput limit gets hit) into wait & retry loop.
> 
> PR created: #643 please review.

same problem, when the size of document chunks > 500 the program will probably stuck. appreciate your PR.


#486 seems to have added `max_workers` to the `evaluate` function but did not reference it. @jjmachan  correct me if I'm wrong.

hey @joy13975 really sorry I have not been able to get to this - but for the last commend you are right. max_workers are an old arg we dont' use

as for your PR - really really appreciate it ❤️ , will get it reviewed both and respond tonight 


thanks again for your help bringing this up 🙌🏽 

thanks to @joy13975 we have fixed this guys 🙂