Repository: explodinggradients/ragas
Issue: Exception in thread Thread-4: `asyncio.exceptions.CancelledError` & `asyncio.exceptions.TimeoutError`
Root cause: Resource Mishandling

I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I am trying to run the template code from the Github ReadMe page. My code is below:
```
from datasets import Dataset 
import os
from ragas import evaluate
from ragas.metrics import faithfulness

os.environ["OPENAI_API_KEY"] = "sk-proj-*****B3" <HIDDEN>

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)

score = evaluate(dataset,metrics=[faithfulness])
score.to_pandas()
```

Ragas version: I use the latest one.
Python version: 3.10.14.


**Error trace**
The error I encountered is below: (with my username hidden in the file paths)
```
**Exception in thread Thread-4:**
Traceback (most recent call last):
  File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/metrics/_faithfulness.py", line 248, in _ascore
    statements = await self.llm.generate(
  File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/llms/base.py", line 93, in generate
    return await agenerate_text_with_retry(
  File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
  File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 121, in __call__
    await self.sleep(do)  # type: ignore[misc]
  File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/asyncio/tasks.py", line 605, in sleep
    return await future
**asyncio.exceptions.CancelledError**

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
...
    score = await asyncio.wait_for(
  File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
**asyncio.exceptions.TimeoutError**
```

**Additional context**
And the error report of the code is here:
```
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
Cell In[1], [line 18](vscode-notebook-cell:?execution_count=1&line=18)
      [8](vscode-notebook-cell:?execution_count=1&line=8) data_samples = {
      [9](vscode-notebook-cell:?execution_count=1&line=9)     'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
     [10](vscode-notebook-cell:?execution_count=1&line=10)     'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
   (...)
     [13](vscode-notebook-cell:?execution_count=1&line=13)     'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
     [14](vscode-notebook-cell:?execution_count=1&line=14) }
     [16](vscode-notebook-cell:?execution_count=1&line=16) dataset = Dataset.from_dict(data_samples)
---> [18](vscode-notebook-cell:?execution_count=1&line=18) score = evaluate(dataset,metrics=[faithfulness])
     [19](vscode-notebook-cell:?execution_count=1&line=19) score.to_pandas()

File ~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:255, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map)
    [252](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:252)     if not evaluation_group_cm.ended:
    [253](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:253)         evaluation_rm.on_chain_error(e)
--> [255](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:255)     raise e
    [256](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:256) else:
    [257](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:257)     result = Result(
    [258](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:258)         scores=Dataset.from_list(scores),
    [259](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:259)         dataset=dataset,
    [260](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:260)         binary_columns=binary_metrics,
    [261](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:261)     )

File ~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:237, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map)
...
--> [237](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:237)     raise ExceptionInRunner()
    [239](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:239) # convert results to dataset_like
    [240](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:240) for i, _ in enumerate(dataset):

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
```

Can anyone help me out? Thanks!

<!-- Greeting -->
Hey, @larry-ziyue-yin! I'm here to help you with your bug. I can assist with solving issues, answering questions, and becoming a contributor. Let's get this sorted out!

<!-- Answer -->
To resolve the `asyncio.exceptions.CancelledError` and `asyncio.exceptions.TimeoutError` when running the template code from the Ragas library's GitHub ReadMe page, you can handle these exceptions within the `_aresults` method of the `Runner` class. Here is the modified code with the necessary exception handling:

```python
async def _aresults(self) -> t.List[t.Any]:
    results = []
    for future in tqdm(
        self.futures,
        desc=self.desc,
        total=len(self.jobs),
        # whether you want to keep the progress bar after completion
        leave=self.keep_progress_bar,
    ):
        r = (-1, np.nan)
        try:
            r = await future
        except MaxRetriesExceeded as e:
            logger.warning(f"max retries exceeded for {e.evolution}")
        except asyncio.exceptions.CancelledError as e:
            logger.warning("Task was cancelled")
            if self.raise_exceptions:
                raise e
        except asyncio.exceptions.TimeoutError as e:
            logger.warning("Task timed out")
            if self.raise_exceptions:
                raise e
        except Exception as e:
            if self.raise_exceptions:
                raise e
            else:
                logger.error(
                    "Runner in Executor raised an exception", exc_info=True
                )
        results.append(r)

    return results
```

This code adds specific handling for `asyncio.exceptions.CancelledError` and `asyncio.exceptions.TimeoutError`, logging a warning message for each and re-raising the exception if `self.raise_exceptions` is set to `True` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/executor.py)</sup>.

Additionally, ensure that there are no null values in the `ground_truth` column of your dataset, as this can also cause issues. This was a known issue in a similar context <sup>[[2]](https://github.com/explodinggradients/ragas/issues/447)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I also encounter this problem when evaluating, but I don't want to change ragas codebase inside.
I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.

```
        result = evaluate(
            dataset,
            llm=model,
            embeddings=embeddings,
            metrics=[
                context_precision,
                faithfulness,
                answer_relevancy, 
                context_recall,
            ],
            run_config=RunConfig(thread_timeout=thread_timeout)
        )
```

Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?

> context_precision,

Hello, I have the same problem, do you have a new solution?

@jmgu0212 evaluate on current master does not accept `run_config`. See my PR https://github.com/explodinggradients/ragas/pull/1085

> I also encounter this problem when evaluating, but I don't want to change ragas codebase inside. I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.
> 
> ```
>         result = evaluate(
>             dataset,
>             llm=model,
>             embeddings=embeddings,
>             metrics=[
>                 context_precision,
>                 faithfulness,
>                 answer_relevancy, 
>                 context_recall,
>             ],
>             run_config=RunConfig(thread_timeout=thread_timeout)
>         )
> ```
> 
> Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?

Hello, I would like to ask about the thread_timeout setting in your run_config=RunConfig(thread_timeout=thread_timeout). My configuration is as follows: run_config = ragas.RunConfig(timeout=60, max_retries=3, max_wait=20). However, when calculating metrics, such as context_relevancy, the final results are all NA. I wonder why this is happening and if there are any solutions. Is it because the timeout I set is too short?

> > I also encounter this problem when evaluating, but I don't want to change ragas codebase inside. I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.
> > ```
> >         result = evaluate(
> >             dataset,
> >             llm=model,
> >             embeddings=embeddings,
> >             metrics=[
> >                 context_precision,
> >                 faithfulness,
> >                 answer_relevancy, 
> >                 context_recall,
> >             ],
> >             run_config=RunConfig(thread_timeout=thread_timeout)
> >         )
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?
> 
> Hello, I would like to ask about the thread_timeout setting in your run_config=RunConfig(thread_timeout=thread_timeout). My configuration is as follows: run_config = ragas.RunConfig(timeout=60, max_retries=3, max_wait=20). However, when calculating metrics, such as context_relevancy, the final results are all NA. I wonder why this is happening and if there are any solutions. Is it because the timeout I set is too short?

I wonder if you also adapt evaluation metrics to Chinese? If yes, you need to deal with the language option fixed in source code. Take context_relevancy as an example, you will find `seg = pysbd.Segmenter(language="en", clean=False)`. 

> > > I also encounter this problem when evaluating, but I don't want to change ragas codebase inside. I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.
> > > ```
> > >         result = evaluate(
> > >             dataset,
> > >             llm=model,
> > >             embeddings=embeddings,
> > >             metrics=[
> > >                 context_precision,
> > >                 faithfulness,
> > >                 answer_relevancy, 
> > >                 context_recall,
> > >             ],
> > >             run_config=RunConfig(thread_timeout=thread_timeout)
> > >         )
> > > ```
> > > 
> > > 
> > >     
> > >       
> > >     
> > > 
> > >       
> > >     
> > > 
> > >     
> > >   
> > > Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?
> > 
> > 
> > Hello, I would like to ask about the thread_timeout setting in your run_config=RunConfig(thread_timeout=thread_timeout). My configuration is as follows: run_config = ragas.RunConfig(timeout=60, max_retries=3, max_wait=20). However, when calculating metrics, such as context_relevancy, the final results are all NA. I wonder why this is happening and if there are any solutions. Is it because the timeout I set is too short?
> 
> I wonder if you also adapt evaluation metrics to Chinese? If yes, you need to deal with the language option fixed in source code. Take context_relevancy as an example, you will find `seg = pysbd.Segmenter(language="en", clean=False)`.

Thank you very much for your answer. The data set I use is given in Chinese form. So far, I only run one index context_relevancy. I found context_relevancy source code set to seg = pysbd.Segmenter(language="zh", clean=False), but it still ran with the following error:  Traceback (most recent call last):
  File "C:\Program Files\Python311\Lib\site-packages\httpx\_transports\default.py", line 69, in map_httpcore_exceptions  
    yield
  File "C:\Program Files\Python311\Lib\site-packages\httpx\_transports\default.py", line 373, in handle_async_request    
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\httpcore\_async\connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "C:\Program Files\Python311\Lib\site-packages\httpcore\_async\connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\httpcore\_async\http_proxy.py", line 317, in handle_async_request   
    stream = await stream.start_tls(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\httpcore\_async\http11.py", line 383, in start_tls
    return await self._stream.start_tls(ssl_context, server_hostname, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\httpcore\_backends\anyio.py", line 68, in start_tls
    with map_exceptions(exc_map):
  File "C:\Program Files\Python311\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Program Files\Python311\Lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Program Files\Python311\Lib\site-packages\openai\_base_client.py", line 1548, in _request
    response = await self._client.send(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\httpx\_client.py", line 1661, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\httpx\_client.py", line 1689, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\httpx\_client.py", line 1726, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\httpx\_client.py", line 1763, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\httpx\_transports\default.py", line 372, in handle_async_request    
    with map_httpcore_exceptions():
  File "C:\Program Files\Python311\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Program Files\Python311\Lib\site-packages\httpx\_transports\default.py", line 86, in map_httpcore_exceptions  
    raise mapped_exc(message) from exc
httpx.ConnectError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Program Files\Python311\Lib\site-packages\ragas\executor.py", line 104, in wrapped_callable_async
    result = await callable(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\ragas\metrics\base.py", line 134, in ascore
    raise e
  File "C:\Program Files\Python311\Lib\site-packages\ragas\metrics\base.py", line 127, in ascore
    score = await asyncio.wait_for(
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\asyncio\tasks.py", line 489, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\ragas\metrics\_context_relevancy.py", line 77, in _ascore
    result = await self.llm.generate(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\ragas\llms\base.py", line 93, in generate
    return await agenerate_text_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\tenacity\asyncio\__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\tenacity\asyncio\__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\tenacity\asyncio\__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\tenacity\_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\tenacity\__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Program Files\Python311\Lib\site-packages\tenacity\asyncio\__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\ragas\llms\base.py", line 170, in agenerate_text
    return await self.langchain_llm.agenerate_prompt(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\langchain_core\language_models\chat_models.py", line 691, in agenerate_prompt
    return await self.agenerate(
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\langchain_core\language_models\chat_models.py", line 651, in agenerate
    raise exceptions[0]
  File "C:\Program Files\Python311\Lib\site-packages\langchain_core\language_models\chat_models.py", line 836, in _agenerate_with_cache
    result = await self._agenerate(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\langchain_openai\chat_models\base.py", line 674, in _agenerate      
    response = await self.async_client.create(**payload)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\openai\resources\chat\completions.py", line 1289, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\openai\_base_client.py", line 1816, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\openai\_base_client.py", line 1514, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\openai\_base_client.py", line 1572, in _request
    return await self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\openai\_base_client.py", line 1641, in _retry_request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\openai\_base_client.py", line 1572, in _request
    return await self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\openai\_base_client.py", line 1641, in _retry_request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\openai\_base_client.py", line 1582, in _request
    raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.   Do you have a better solution? Thank you very much

same issue

> same issue

"Hello, do you have a solution yet?"


Waiting for a solution too

@djanito @kmr666 @JANHMS @jmgu0212 @larry-ziyue-yin my sincere apologies for the delay in responding but this will be addressed with #1156 do check that out

> > I also encounter this problem when evaluating, but I don't want to change ragas codebase inside. I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.
> > ```
> >         result = evaluate(
> >             dataset,
> >             llm=model,
> >             embeddings=embeddings,
> >             metrics=[
> >                 context_precision,
> >                 faithfulness,
> >                 answer_relevancy, 
> >                 context_recall,
> >             ],
> >             run_config=RunConfig(thread_timeout=thread_timeout)
> >         )
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?
> 
> Hello, I would like to ask about the thread_timeout setting in your run_config=RunConfig(thread_timeout=thread_timeout). My configuration is as follows: run_config = ragas.RunConfig(timeout=60, max_retries=3, max_wait=20). However, when calculating metrics, such as context_relevancy, the final results are all NA. I wonder why this is happening and if there are any solutions. Is it because the timeout I set is too short?

yes,your timeout is too short , just make it as long as possible ,such as tiemout=1000,this can fix your nan problem

Also having this problem, specially with very big contexts on my RAGAS (and since I'm doing an evaluation on a dataset)
The evaluation causes multiple queries to the judge LLM, and it might not be capable of serving every single query, that's why we see Timeouts (this is my realistic guess)

Just play with timeout, max_retries and others



> Also having this problem, specially with very big contexts on my RAGAS (and since I'm doing an evaluation on a dataset) The evaluation causes multiple queries to the judge LLM, and it might not be capable of serving every single query, that's why we see Timeouts (this is my realistic guess)
> 
> Just play with timeout, max_retries and others

Please use the new metrics collections for evals. Feel free to open new issue if still facing problems.