Repository: explodinggradients/ragas
Issue: ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Root cause: Resource Mishandling

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Can't get evaluation to work. constantly get the error: The runner thread which was running the jobs raised an exception

Ragas version: latest
Python version: 3

**Code to Reproduce**
```py
from ragas import evaluate
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
)

result = evaluate(
    dataset,
    metrics=[

        faithfulness,
        answer_relevancy,

    ],
)
```
result

**Error trace**
```
Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 87, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 625, in run_until_complete
    self._check_running()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 584, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
/usr/lib/python3.10/threading.py:1018: RuntimeWarning: coroutine 'Runner._aresults' was never awaited
  self._invoke_excepthook(self)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py:235: RuntimeWarning: coroutine 'as_completed.<locals>.sema_coro' was never awaited
  results = executor.results()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
[<ipython-input-12-5dadbbc00d9a>](https://localhost:8080/#) in <cell line: 3>()
      1 from ragas import evaluate
      2 
----> 3 result = evaluate(
      4     dataset,
      5     metrics=[

1 frames
[/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py](https://localhost:8080/#) in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map)
    235         results = executor.results()
    236         if results == []:
--> 237             raise ExceptionInRunner()
    238 
    239         # convert results to dataset_like

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
```
**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hey there, @edubrigham! I'm here to help you out with any bugs, questions, or contributions you have in mind. Let's tackle this together!

<!-- Answer -->
To resolve the "The runner thread which was running the jobs raised an exception" error, you can pass the `raise_exceptions=False` parameter to the `evaluate` function. This will show only a warning message instead of raising an exception, allowing the execution to continue.

Here's how you can modify your code:

```python
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness

result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy],
    raise_exceptions=False  # Add this line
)

print(result)
```

This should prevent the runner thread from raising an exception and will only show a warning message instead <sup>[[1]](https://github.com/explodinggradients/ragas/issues/1055)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/issues/934)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I have the same problem but with synthetic test case generation 
```python
import os

from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_cohere.llms import Cohere	
from langchain_cohere import CohereEmbeddings

os.environ["COHERE_API_KEY"] = "***************************"

# generator with openai models
generator_llm = Cohere(model="command", temperature=0.75)
critic_llm = Cohere(model="command", temperature=0.75)
embeddings = CohereEmbeddings(model="embed-english-light-v3.0")

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
print(type(testset))

testset.to_pandas()
```

Error:
Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 87, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 625, in run_until_complete
    self._check_running()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 584, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
/usr/lib/python3.10/threading.py:1018: RuntimeWarning: coroutine 'Runner._aresults' was never awaited
  self._invoke_excepthook(self)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.10/dist-packages/ragas/testset/docstore.py:252: RuntimeWarning: coroutine 'as_completed.<locals>.sema_coro' was never awaited
  results = executor.results()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
[<ipython-input-16-bf233b10be38>](https://localhost:8080/#) in <cell line: 22>()
     20 
     21 # generate testset
---> 22 testset = generator.generate_with_langchain_docs(documents,  raise_exceptions=False, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
     23 print(type(testset))
     24 

2 frames
[/usr/local/lib/python3.10/dist-packages/ragas/testset/docstore.py](https://localhost:8080/#) in add_nodes(self, nodes, show_progress)
    252         results = executor.results()
    253         if not results:
--> 254             raise ExceptionInRunner()
    255 
    256         for i, n in enumerate(nodes):

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

I have the same problem with synthetic test generation

Hello, i had similar issue and handle it by adding the following code before my ragas calls : 

`pip install nest-asyncio`

and 

`import nest_asyncio`
`nest_asyncio.apply()`

Hope it helps.

@aixelbsck I tried using nest_asyncio but still getting same error.
I was trying to execute following https://docs.ragas.io/en/stable/getstarted/testset_generation.html on Mac as a jupyter notebook and getting following error

```
Cell In[26], [line 3](vscode-notebook-cell:?execution_count=26&line=3)
      [1](vscode-notebook-cell:?execution_count=26&line=1) nest_asyncio.apply()
      [2](vscode-notebook-cell:?execution_count=26&line=2) generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)
----> [3](vscode-notebook-cell:?execution_count=26&line=3) testset = generator.generate_with_langchain_docs(documents, test_size=2, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

File ~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:206, in TestsetGenerator.generate_with_langchain_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    [204](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:204) distributions = distributions or {}
    [205](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:205) # chunk documents and add to docstore
--> [206](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:206) self.docstore.add_documents(
    [207](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:207)     [Document.from_langchain_document(doc) for doc in documents]
    [208](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:208) )
    [210](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:210) return self.generate(
    [211](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:211)     test_size=test_size,
    [212](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:212)     distributions=distributions,
   (...)
    [216](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:216)     run_config=run_config,
    [217](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:217) )

File ~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:215, in InMemoryDocumentStore.add_documents(self, docs, show_progress)
    [210](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:210) # split documents with self.splitter into smaller nodes
    [211](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:211) nodes = [
    [212](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:212)     Node.from_langchain_document(d)
    [213](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:213)     for d in self.splitter.transform_documents(docs)
...
--> [254](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:254)     raise ExceptionInRunner()
    [256](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:256) for i, n in enumerate(nodes):
    [257](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:257)     if i in nodes_to_embed.keys():

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?5ef1a4b6-b3cc-400f-8753-824b83dd5465) or open in a [text editor](command:workbench.action.openLargeOutput?5ef1a4b6-b3cc-400f-8753-824b83dd5465). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...


```

> Hello, i had similar issue and handle it by adding the following code before my ragas calls :
> 
> `pip install nest-asyncio`
> 
> and
> 
> `import nest_asyncio` `nest_asyncio.apply()`
> 
> Hope it helps.

Great! This works for my issue too!   Thanks for help 😀

@anujmehta @swayam-khandelwal which versions of ragas are you using?

> @anujmehta @swayam-khandelwal which versions of ragas are you using?

Hello @jjmachan I'm using version :0.1.10
I am still getting issue. 
Even though I use the solution given by @aixelbsck . It is not evaluating.

Below is the error I'm getting:

Exception in thread Thread-10:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/executor.py", line 87, in run
    results = self.loop.run_until_complete(self._aresults())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 630, in run_until_complete
    self._check_running()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 589, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
[/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py:1047](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py:1047): RuntimeWarning: coroutine 'Runner._aresults' was never awaited
  self._invoke_excepthook(self)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Evaluating:   0%|          | 0/6 [00:38<?, ?it/s]
Exception in thread Thread-30:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/executor.py", line 87, in run
    results = self.loop.run_until_complete(self._aresults())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
           ^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/executor.py", line 79, in _aresults
    r = await future
        ^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 615, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/executor.py", line 37, in sema_coro
    return await coro
           ^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/executor.py", line 109, in wrapped_callable_async
    raise e
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/executor.py", line 104, in wrapped_callable_async
    result = await callable(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/metrics/base.py", line 134, in ascore
    raise e
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/metrics/base.py", line 127, in ascore
    score = await asyncio.wait_for(
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/metrics/_answer_correctness.py", line 221, in _ascore
    item_statement = await self.llm.generate(
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/llms/base.py", line 93, in generate
    return await agenerate_text_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 50, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/ragas/llms/base.py", line 178, in agenerate_text
    result = await self.langchain_llm.agenerate_prompt(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 713, in agenerate_prompt
    return await self.agenerate(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 1106, in agenerate
    output = await self._agenerate_helper(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 961, in _agenerate_helper
    raise e
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 945, in _agenerate_helper
    await self._agenerate(
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 614, in _agenerate
    return await run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 575, in run_in_executor
    return await asyncio.get_running_loop().run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py", line 287, in __await__
    yield self  # This tells Task to wait for completion.
    ^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 566, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/swayam/Desktop/DeepEval-Test/deepevalTestEnv/lib/python3.11/site-packages/langchain_ibm/llms.py", line 385, in _generate
    response = self.watsonx_model.generate(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: ModelInference.generate() got an unexpected keyword argument 'temperature'

@jjmachan this is also an error which I get frequently. The answer_similarity metric works well but facing issue with answer correctness, context recall 




@jjmachan Apologies for delayed response, on debugging it turned out that exception was coming from incomplete configuration for embedding, I guess by putting right config things should work. Thanks. 

thanks for letting use know @anujmehta 

for others who might run into this, we have modified the executor with the latest versions so this would no longer be a problem hopefully

the problem still persists. I'm using ragas 0.1.13.

I followed through the posts and tried to use ragas 0.1.4 and langchain 0.1.5 with GPT-3.5 and GPT-4, for synthetic testset generation. I am getting the error ::
 The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
 

> I followed through the posts and tried to use ragas 0.1.4 and langchain 0.1.5 with GPT-3.5 and GPT-4, for synthetic testset generation. I am getting the error :: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Did you resolve this issue? I have the same issue