Repository: explodinggradients/ragas
Issue: ValueError: Can't patch loop of type <class 'uvloop.Loop'>
Root cause: External Dependency Breakage

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I am trying to integrate RAGAS with an API based on FastAPI, and I'm getting an error: `ValueError: Can't patch loop of type <class 'uvloop.Loop'>`

Ragas version: 0.1.15
Python version: 3.10.12

**Code to Reproduce**

```
from datasets import Dataset
from fastapi import FastAPI
from ragas import evaluate
from ragas.metrics import faithfulness

app = FastAPI()

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
}

async def async_evaluation():
    dataset = Dataset.from_dict(data_samples)
    score = evaluate(dataset,metrics=[faithfulness])
    return {"score": score['faithfulness']}

@app.get("/evaluate")
async def evaluate_data():
    return await async_evaluation()
```


**Error trace**

```
Traceback (most recent call last):
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/main.py", line 39, in evaluate_data
    return await async_evaluation()
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/main.py", line 33, in async_evaluation
    score = evaluate(dataset,metrics=[faithfulness])
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/_analytics.py", line 129, in wrapper
    result = func(*args, **kwargs)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 280, in evaluate
    raise e
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 260, in evaluate
    results = executor.results()
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/executor.py", line 93, in results
    nest_asyncio.apply()
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/nest_asyncio.py", line 19, in apply
    _patch_loop(loop)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/nest_asyncio.py", line 193, in _patch_loop
    raise ValueError('Can\'t patch loop of type %s' % type(loop))
ValueError: Can't patch loop of type <class 'uvloop.Loop'>
```


**Expected behavior**
I expected to be able to integrate RAGAS with our API based on FastAPI


**Additional context**


<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hey there, @gustavo-has-stone! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

<!-- Answer -->
To resolve the `ValueError: Can't patch loop of type <class 'uvloop.Loop'>` when integrating RAGAS with a FastAPI-based API, you need to ensure that `nest_asyncio` is not applied when using `uvloop`. The `uvloop` event loop is not compatible with `nest_asyncio`.

You can modify the `results` method in the `Executor` class to check if `uvloop` is being used and avoid applying `nest_asyncio` in that case. Here is the updated code:

```python
def results(self) -> t.List[t.Any]:
    if is_event_loop_running():
        # an event loop is running so call nested_asyncio to fix this
        try:
            import uvloop
            if isinstance(asyncio.get_event_loop(), uvloop.Loop):
                raise RuntimeError("Cannot use nest_asyncio with uvloop")
        except ImportError:
            pass

        try:
            import nest_asyncio
        except ImportError:
            raise ImportError(
                "It seems like you're running this in a jupyter-like environment. Please install nest_asyncio with `pip install nest_asyncio` to make it work."
            )
        if not self._nest_asyncio_applied:
            nest_asyncio.apply()
            self._nest_asyncio_applied = True

    # create a generator for which returns tasks as they finish
    futures_as_they_finish = as_completed(
        coros=[afunc(*args, **kwargs) for afunc, args, kwargs, _ in self.jobs],
        max_workers=(self.run_config or RunConfig()).max_workers,
    )

    async def _aresults() -> t.List[t.Any]:
        results = []
        for future in tqdm(
            futures_as_they_finish,
            desc=self.desc,
            total=len(self.jobs),
            # whether you want to keep the progress bar after completion
            leave=self.keep_progress_bar,
        ):
            r = await future
            results.append(r)

        return results

    results = asyncio.run(_aresults())
    sorted_results = sorted(results, key=lambda x: x[0])
    return [r[1] for r in sorted_results]
```

This modification checks if `uvloop` is being used and raises a `RuntimeError` if an attempt is made to apply `nest_asyncio` with `uvloop`. This should resolve the `ValueError` you are encountering <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/executor.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Hi, I also encountered the same error using ragas versions earlier than 0.1.15 (I would say I tried 0.1.1 to 0.1.6). 

Do you know what is a way to solve the problem other than editing the `results` method in the source code? I don't find the solution proposed by the bot very practical.

Thank you very much!

I'm having the same issue，Can't patch loop of type <class 'uvloop.Loop'>。Did you solve it?

I didn't. The only workaround I know would involve using asyncio when running uvicorn

I'm really introducing Ragas in the Sanic frame, 
app = Sanic("CharAI")
app.config.USE_UVLOOP = False
 can be solved, but it will reduce Saanich's performance

Had the same issue updating to v0.2.x when calling `evaluate` in `async` function. I went back to `ragas==0.1.8`, which worked for me before.

**EDIT**
> [!IMPORTANT]  
> I updated to from `0.1.8` to `0.1.11` and the issue came up again.

**EDIT 2**
As pointed out before, I realized the issue does not come from running it async with `asyncio.create_task` rather from the existence of the uvicorn event loop or something. Without fastapi it works fine

this was one solution from the our discord community - could you guys try this out?
![image](https://github.com/user-attachments/assets/c85e58a7-01a9-4d05-b889-2d5c52de9d49)


> this was one solution from the our discord community - could you guys try this out? ![image](https://private-user-images.githubusercontent.com/5261489/379340323-c85e58a7-01a9-4d05-b889-2d5c52de9d49.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjk3NTg0NjUsIm5iZiI6MTcyOTc1ODE2NSwicGF0aCI6Ii81MjYxNDg5LzM3OTM0MDMyMy1jODVlNThhNy0wMWE5LTRkMDUtYjg4OS0yZDVjNTJkZTlkNDkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MTAyNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDEwMjRUMDgyMjQ1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Yjk2OWU5YTlkNWE5ODczOTVkNTk0NzYwNGRkYWE1YzI2MzlmZWRlMDUwM2M1ODkwMjk4ZjE2YWIxYjA0NGE3ZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.ghPD-4dMHqEydSmrvEBY3ct2oHUHfFha2shjiLbQZQ8)

Thx！works！

closing this in that case 🙂 

Is this problem solved now?

@richness @jjmachan 

It is not solved.  The issue itself is that when you use `uvloop` for running FastAPI and uvicorn, ragas crashes.
The solution above suggests to change event loop implementation to `asyncio` which kinda contradicts with the given conditions. 

It is like "this library is not working in docker!" - "then don't use docker" - "aah okay then, issue closed". 

As of ragas==0.2.12, it still crashes for me, I rolled back to 0.2.6.

hey @nmakhotkin I understand the frustration and we actually have this tracked with https://github.com/explodinggradients/ragas/issues/1819 - I'm hoping to get to it this week and get it sorted

I'll ping you right after fixing this

@jjmachan Thank you for the support, that would be awesome!

any update on this one? 

@jjmachan Any update on this? Same issue happens even when using SingleTurnSample and MultiTurnSample. 

we are working on a fix with https://github.com/explodinggradients/ragas/pull/2291 will get it hopefully get it out in the next release 🙂 

I get it done in my fork with the minimal work changing like 5 lines of code. Essentially: disable nest_asyncio and make a new "aresults" instead of "results" (which is already spawn threads in asyncio loop): 

One of the main issues I faced is that nest_asyncio is interfering with uvloop and some of Langchain LLM wrappers so I had to cut it.

patched version 0.2.15:

https://github.com/nmakhotkin/ragas/commit/bfeea43311edb8befe3e321661d7f83a25bc666f
https://github.com/nmakhotkin/ragas/commit/c7af4f57a1abaf08028089e1cefe976f30a39359

Doe it really require so much work? I have looked through your PR, the number of changes are insane... 

@nmakhotkin  I just tried your changes and it works great! Thanks.