Repository: explodinggradients/ragas
Issue: [Feautre Reuqest] [Bug] Async + RAGAS
Root cause: Architectural Constraint

**Describe the Feature**
As far as I can tell, it seems like there is a lack of async functions in ragas, which leads to a lot of nesting async issues (since sync functions are calling async functions somewhere in the stack). Furthermore, `nest_asyncio.apply()` is automatically called in the import stack, which breaks a lot of applications

```
File "/usr/local/lib/python3.10/site-packages/ragas/executor.py", line 15, in <module>
2025-01-07 23:06:08     nest_asyncio.apply()
```

It would be awesome to have async functions like `aevaluate` instead of `evaluate` -- the approach I might take is make everything async first, and then get/create an event loop and run it there for the sync versions to avoid code duplication.

At the very least, removing the automatic `nest_asyncio.apply()` might be a good idea?

**Why is the feature important for you?**
Async code is the backbone of production systems -- without it, ragas will block the event loop of the server

**Additional context**
I THINK the workaround here for now is doing something like

```python
def run_ragas(...):
  from app.evaluation.ragas import generate_dev_set, generate_dev_set_csv
  # other ragas imports

# Then, later on in your code
result = await asyncio.to_thread(run_ragas(...))
```

Which delays the ragas imports and runs it behind an isolated thread, which should be safe. But obviously not ideal


hey @logan-markewich 🙂 👋🏽 

like you suggested the main culprit is the 
```
File "/usr/local/lib/python3.10/site-packages/ragas/executor.py", line 15, in <module>
2025-01-07 23:06:08     nest_asyncio.apply()
```

now we added this as a dirty fix for the executor thread which was responsible for handling the logic "he approach I might take is make everything async first, and then get/create an event loop and run it there for the sync versions to avoid code duplication."

I'll do a bit more experiments and testing to remove the need for nesting_asyncio call.

[OpenHands](https://github.com/All-Hands-AI/OpenHands) started fixing the issue! You can monitor the progress [here](https://github.com/explodinggradients/ragas/actions/runs/12979781978).

[OpenHands](https://github.com/All-Hands-AI/OpenHands) started fixing the issue! You can monitor the progress [here](https://github.com/explodinggradients/ragas/actions/runs/12979781978).

[OpenHands](https://github.com/All-Hands-AI/OpenHands) started fixing the issue! You can monitor the progress [here](https://github.com/explodinggradients/ragas/actions/runs/12979781978).

This problem hasn't been solved yet at present?

As far as I can see, it wasn't. It still crashes for me with 

```
ValueError: Can't patch loop of type <class 'uvloop.Loop'>
```
as for version 0.2.15. 
I used asyncio loop implementation for a while but then I noticed that it leads to unexpected delays and slowdowns during FastAPI requests and then I switched back to uvloop. 
The workaround is to run `evaluate`  in executor like this: 

```python
loop = asyncio.get_event_loop()
results = await loop.run_in_executor(None, evaluate, dataset, metrics, ...)
```
But this does not work together with Google gemini models using ChatGoogleGenerativeAI as LLM: google genai throws an error claiming that the thread was attached to another loop

**UPD**: No, it worked earlier in 0.2.6 but in the latest version does not
**UPD2**": Ended up with small refactoring and making my own fork with async-only ragas without `nest_asyncio` at all: https://github.com/nmakhotkin/ragas

Re: removing nest_asyncio - I think the major reason it was added was to allow folks to use it in ipython/jupyter kernels? Modern ipython can just `await` directly rather than mucking about with nesting the asyncio loop.  If that's the case, you could just detect an ipython or jupyter environment and tell the user to `await` directly and completely pull the nest_asyncio dependency?

```py
# Check if we're in Jupyter/IPython
try:
    from IPython.core.getipython import get_ipython

    if get_ipython() is not None:
        raise RuntimeError(
            "In Jupyter/IPython, use `await executor.aresults()` directly; this avoids the need for nest_asyncio."
        )
except ImportError:
    pass
```

any updates here @jjmachan?

Any plan to solve this issue? @jjmachan