Repository: explodinggradients/ragas
Issue: Given code @ https://github.com/explodinggradients/ragas  is not working and raises an exeption. 
Root cause: Validation Gap

I can not figure out what is wrong with the provided code read the documentation provided as well.

**Bug**
ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Ragas version:
Python version:

**Code to Reproduce**
pip install ragas
```py
from datasets import Dataset 
import os
from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness

os.environ["OPENAI_API_KEY"] = "your-openai-key"

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)

score = evaluate(dataset,metrics=[faithfulness,answer_correctness])
score.to_pandas()
```

**Error trace**
```
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 87, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 625, in run_until_complete
    self._check_running()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 584, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
/usr/lib/python3.10/threading.py:1018: RuntimeWarning: coroutine 'Runner._aresults' was never awaited
  self._invoke_excepthook(self)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py:235: RuntimeWarning: coroutine 'as_completed.<locals>.sema_coro' was never awaited
  results = executor.results()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
[<ipython-input-5-58c56f78de7c>](https://localhost:8080/#) in <cell line: 1>()
----> 1 score = evaluate(dataset,metrics=[faithfulness,answer_correctness])
      2 score.to_pandas()

1 frames
[/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py](https://localhost:8080/#) in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map)
    235         results = executor.results()
    236         if results == []:
--> 237             raise ExceptionInRunner()
    238 
    239         # convert results to dataset_like
**Expected behavior**
code should run and evaluate the given responses 
```

**Additional context**
Tried to run as it is code provided in the repo but it is raising an error. tried it on 03-JUL-2024



<!-- Greeting -->
Hey @muhammad-ammar12! 👋 I'm here to help you with your coding troubles!

<!-- Answer -->
The "ExceptionInRunner" error you're encountering is due to the event loop already running when the `evaluate` function tries to execute asynchronous operations. This is a common issue when running asynchronous code in environments like Jupyter notebooks.

To fix this, you can use the `nest_asyncio` library to allow nested use of `asyncio.run()`. Here's how you can modify your code:

1. Install the `nest_asyncio` library:
   ```sh
   pip install nest_asyncio
   ```

2. Modify your code to include `nest_asyncio`:

   ```python
   from datasets import Dataset 
   import os
   import nest_asyncio
   from ragas import evaluate
   from ragas.metrics import faithfulness, answer_correctness

   # Apply nest_asyncio to allow nested event loops
   nest_asyncio.apply()

   os.environ["OPENAI_API_KEY"] = "your-openai-key"

   data_samples = {
       'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
       'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
       'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
       ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
       'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
   }

   dataset = Dataset.from_dict(data_samples)

   score = evaluate(dataset, metrics=[faithfulness, answer_correctness])
   score.to_pandas()
   ```

This should resolve the "This event loop is already running" error and allow your code to run correctly <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/executor.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


thanks for reaching out, tried your provided solution but still getting
the same error.

On Wed, Jul 3, 2024 at 4:16 PM dosubot[bot] ***@***.***>
wrote:

> Hey @muhammad-ammar12 <https://github.com/muhammad-ammar12>! 👋 I'm here
> to help you with your coding troubles!
>
> The "ExceptionInRunner" error you're encountering is due to the event loop
> already running when the evaluate function tries to execute asynchronous
> operations. This is a common issue when running asynchronous code in
> environments like Jupyter notebooks.
>
> To fix this, you can use the nest_asyncio library to allow nested use of
> asyncio.run(). Here's how you can modify your code:
>
>    1.
>
>    Install the nest_asyncio library:
>
>    pip install nest_asyncio
>
>    2.
>
>    Modify your code to include nest_asyncio:
>
>    from datasets import Dataset import osimport nest_asynciofrom ragas import evaluatefrom ragas.metrics import faithfulness, answer_correctness
>    # Apply nest_asyncio to allow nested event loopsnest_asyncio.apply()
>    os.environ["OPENAI_API_KEY"] = "your-openai-key"
>    data_samples = {
>        'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
>        'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
>        'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'],
>        ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
>        'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
>    }
>    dataset = Dataset.from_dict(data_samples)
>    score = evaluate(dataset, metrics=[faithfulness, answer_correctness])score.to_pandas()
>
>
> This should resolve the "This event loop is already running" error and
> allow your code to run correctly [1]
> <https://github.com/explodinggradients/ragas/blob/main/src/ragas/executor.py>
> .
>
> *To continue talking to Dosu <https://dosu.dev>, mention @dosu
> <https://github.com/dosu>.*
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/explodinggradients/ragas/issues/1064#issuecomment-2206505060>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A7ROIEUQVCZJRQEL47XKBRDZKQIVTAVCNFSM6AAAAABKJZJRO6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBWGUYDKMBWGA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


My Bad! just figured out after fixing the code with your help was
missing the API_key. it works now!!

thanks a lot for your help @dosu

On Wed, Jul 3, 2024 at 4:39 PM Muhammad Ammar Shahid <
***@***.***> wrote:

> thanks for reaching out, tried your provided solution but still getting
> the same error.
>
> On Wed, Jul 3, 2024 at 4:16 PM dosubot[bot] ***@***.***>
> wrote:
>
>> Hey @muhammad-ammar12 <https://github.com/muhammad-ammar12>! 👋 I'm here
>> to help you with your coding troubles!
>>
>> The "ExceptionInRunner" error you're encountering is due to the event
>> loop already running when the evaluate function tries to execute
>> asynchronous operations. This is a common issue when running asynchronous
>> code in environments like Jupyter notebooks.
>>
>> To fix this, you can use the nest_asyncio library to allow nested use of
>> asyncio.run(). Here's how you can modify your code:
>>
>>    1.
>>
>>    Install the nest_asyncio library:
>>
>>    pip install nest_asyncio
>>
>>    2.
>>
>>    Modify your code to include nest_asyncio:
>>
>>    from datasets import Dataset import osimport nest_asynciofrom ragas import evaluatefrom ragas.metrics import faithfulness, answer_correctness
>>    # Apply nest_asyncio to allow nested event loopsnest_asyncio.apply()
>>    os.environ["OPENAI_API_KEY"] = "your-openai-key"
>>    data_samples = {
>>        'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
>>        'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
>>        'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'],
>>        ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
>>        'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
>>    }
>>    dataset = Dataset.from_dict(data_samples)
>>    score = evaluate(dataset, metrics=[faithfulness, answer_correctness])score.to_pandas()
>>
>>
>> This should resolve the "This event loop is already running" error and
>> allow your code to run correctly [1]
>> <https://github.com/explodinggradients/ragas/blob/main/src/ragas/executor.py>
>> .
>>
>> *To continue talking to Dosu <https://dosu.dev>, mention @dosu
>> <https://github.com/dosu>.*
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/explodinggradients/ragas/issues/1064#issuecomment-2206505060>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/A7ROIEUQVCZJRQEL47XKBRDZKQIVTAVCNFSM6AAAAABKJZJRO6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBWGUYDKMBWGA>
>> .
>> You are receiving this because you were mentioned.Message ID:
>> ***@***.***>
>>
>


<!-- Answer -->
You're welcome! I'm glad to hear it's working now. If you have any more questions or run into any other issues, feel free to ask. Happy coding! 🚀


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*