Repository: confident-ai/deepeval
Issue: asyncio.exceptions.CancelledError from evaluate in async mode
Root cause: Resource Mishandling

Since version 3.6.9, attempting to run `evaluate()` with `async_config=AsyncConfig(run_async=True)` has been intermittently failing for me with an `asyncio.exceptions.CancelledError` error. I suspect a race condition somewhere in the async code, as it happens with greater frequency when the number of test cases is larger.

Here is a small reproducer script for this behavior: [deepeval-reproducer.py](https://github.com/user-attachments/files/23469173/deepeval-reproducer.py)

If run in a venv with `deepeval` 3.6.8 or earlier, it runs to completion and exits cleanly: [success.log](https://github.com/user-attachments/files/23469174/success.log)

With version 3.6.9 or 3.7.0, it aborts with a stack trace: [failure.log](https://github.com/user-attachments/files/23469175/failure.log)

Version information:
- Python: 3.11.4
- `aiobotocore`: 2.24.1
- `aiohttp`: 3.12.15
- `boto3`: 1.39.11
- `tenacity`: 9.1.2

Hey @wisnij,

Thank you for the repro and logs, it is appreciated.

**What’s happening**
DeepEval is awaiting your Bedrock call and enforcing a deadline. When that deadline hits, we cancel the task, which is why your stack ends in `asyncio.exceptions.CancelledError` inside `aiohttp`/`aiobotocore`:

```
deepeval/models/llms/amazon_bedrock_model.py:82 -> await client.converse(...)
...
aiohttp/connector.py:694 -> await self._wait_for_available_connection(...)
asyncio.exceptions.CancelledError
```

**Give the Bedrock SDK more time:** 

add this before `async def evaluate_metrics(...` near the top of your script:

```python
import logging
from deepeval.config.settings import get_settings

# Send logs to a file (and keep stdout cleaner)
logging.basicConfig(
    filename="deepeval_debug.log",
    filemode="w",
    level=logging.DEBUG,  # capture rich detail to the file
    format="%(asctime)s %(name)s %(levelname)s: %(message)s",
)


settings = get_settings()
with settings.edit(persist=False):
    # Optional: more verbosity from DeepEval itself
    settings.DEEPEVAL_VERBOSE_MODE = True
    settings.LOG_LEVEL = "debug"
    settings.DEEPEVAL_LOG_STACK_TRACES = True

    # If you don't want to wait for multiple timeouts while debugging:
    settings.DEEPEVAL_RETRY_MAX_ATTEMPTS = 1   # default is 2
    # or increase it if you just want to have more chances at success

    # Give each attempt more time per-attempt
    settings.DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS_OVERRIDE = 150  # try 150–300

# You might also want to try reducing concurrency
async_cfg = AsyncConfig(run_async=True, max_concurrency=4) # default is 20

```

A couple notes:

* The `CancelledError` itself isn't a DeepEval bug, that is how Python surfaces a timeout/cancellation while the SDK is still waiting. Increasing `DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS_OVERRIDE` (and optionally lowering `max_concurrency`) should resolve it. I will make some imrpovements here though so that the error surfaces nicely with helpful instructions. We already handle it nicely in other places, you just happened to hit a hole that wasn't covered.

Please report back and let me know if this resolves your issue or not.

You meant `max_concurrent`, right?  Setting that to 4 reduces the frequency of crashes, but does not eliminate them. Neither do the other settings in the above code.

FWIW I already had `DEEPEVAL_PER_TASK_TIMEOUT_SECONDS_OVERRIDE` and `DEEPEVAL_TASK_GATHER_BUFFER_SECONDS_OVERRIDE` set to 60 in my test environment (which I neglected to mention earlier, mea culpa), whereas the script failed after about 15 seconds, so I don't think it's a timeout problem.

I know `CancelledError` in and of itself isn't a bug, but its sudden appearance in previously-working code arguably is, especially when there hasn't been a major version bump in the library to indicate an intended breaking change. So is going from reliably being able to handle 20 concurrent tasks, to struggling to handle only 4.

Here are debug logs generated by the reproducer script in both successful and unsuccessful runs:

- [deepeval_debug.3.6.8_success.log](https://github.com/user-attachments/files/23485249/deepeval_debug.3.6.8_success.log)
- [deepeval_debug.3.6.9_failure.log](https://github.com/user-attachments/files/23485251/deepeval_debug.3.6.9_failure.log)
- [deepeval_debug.3.6.9_success.log](https://github.com/user-attachments/files/23485250/deepeval_debug.3.6.9_success.log)

Note that even in the ultimately successful run with 3.6.9, the log shows a number of errors that aren't present in the 3.6.8 one. Hopefully these will help indicate what the underlying issue is.

Hey @wisnij, thank you for posting back.

> You meant max_concurrent, right?

Yes, that is correct.

> I already had DEEPEVAL_PER_TASK_TIMEOUT_SECONDS_OVERRIDE and DEEPEVAL_TASK_GATHER_BUFFER_SECONDS_OVERRIDE set to 60 in my test environment (which I neglected to mention earlier, mea culpa), whereas the script failed after about 15 seconds, so I don't think it's a timeout problem.

From your logs I don’t see any gather related timeout signatures, so bumping `DEEPEVAL_TASK_GATHER_BUFFER_SECONDS_OVERRIDE` isn’t needed. The gather buffer is meant to be a small grace window we add on top of the task’s outer budget to let asyncio.gather(...) drain. By default, it is derived automatically at 15% of the task timeout. You normally shouldn’t override it unless you’re explicitly seeing gather/drain timeouts.

What actually happened: your outer budget (`DEEPEVAL_PER_TASK_TIMEOUT_SECONDS_OVERRIDE`) was set while per-attempt wasn’t. DeepEval slices the outer budget into per-attempt time + expected backoff + a safety margin. With higher concurrency, the SDK sometimes spends time in the HTTP connection pool (and may hit its own internal timeouts/retries), so cancellations can surface well before the nominal 60s outer cap. That’s why you saw CancelledError around ~15s even though the task budget was 60s. That is why you are seeing this warning in the logs:

```
2025-11-11 14:18:03,160 deepeval.config.settings WARNING: Metric timeout (outer=60.0s) is less than attempts × per-attempt timeout + backoff (≈61.0s). Retries may be cut short.
```

> I don't think it's a timeout problem.

The initial failure log you posted showed a timeout error. The logs you post now are not timeout errors. Instead, the SDK is reporting a server-side disconnect, you can see that from your logs here:

```
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected

```

That happens a large number of times in your logs. Eventually the outer budget times out after having retried repeatedly and exhausted its outer budget. When that happens you don't see a timeout error, you see a `asyncio.CancelledError` instead:

```
2025-11-11 12:16:25,890 deepeval.metrics.indicator INFO: caught asyncio.CancelledError
2025-11-11 12:16:25,891 deepeval.metrics.indicator INFO: caught asyncio.CancelledError
2025-11-11 12:16:25,891 deepeval.metrics.indicator INFO: caught asyncio.CancelledError
2025-11-11 12:16:25,892 deepeval.metrics.indicator INFO: caught asyncio.CancelledError
```

Increasing DeepEval timeouts won't fix the server side disconnect you are now seeing.

Here is what I recommend trying to start with:

keep async concurrency low.
```
AsyncConfig(run_async=True, max_concurrent=2..4)
```

and don't set `DEEPEVAL_PER_TASK_TIMEOUT_SECONDS_OVERRIDE` or the `DEEPEVAL_TASK_GATHER_BUFFER_SECONDS_OVERRIDE`, let those be computed. Set only per attempt timeout, like this:
```
settings = get_settings()
with settings.edit(persist=False):
    settings.DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS_OVERRIDE = 60 # if no timeouts, you could lower this a bit. If you do see timeouts, you must increase it.
    settings.DEEPEVAL_RETRY_MAX_ATTEMPTS = 2

```

It is possible that their backend was having problems during your testing which would explain the premature server side disconnects. Keep a close watch on your logs to monitor behavior. Note, that DeepEval is doing its job here by guarding against server side requests that don't return in a reasonable amount of time and re-trying to a point, but not indefinitely. 

Having said that, I noticed these errors in your latest failure log:
```
asyncio ERROR: Unclosed client session / Unclosed connector

...

aiohttp.client_exceptions.ClientConnectionError: Connector is closed.
```

Those were wrapped by:
```
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL
```

That means an `aiohttp.ClientSession` wasn't closed cleanly. This, I will need to address this as spawning many client/sessions concurrently and then closing one while others are mid-request can lead to unstable connections like you are seeing. I won't close this issue until that has been addressed. In the meantime, keep your concurrency low and configure like I recommended above.

Thanks again for bringing this to my attention.

@wisnij I also noticed that there are about 30 connections being made in that error log. I recommend adding a throttle to your AsyncConfig. You might be hammering their back-end more than they would like.

Try:
```
async_cfg = AsyncConfig(run_async=True, max_concurrent=2, throttle_value=1)  # sleep 1 second between launches
```

Keep a close eye on the logs and see if that gets rid of the server disconnected errors we are seeing.


Thanks, I'll try that for the meantime. Any idea why none of those errors showed up with 3.6.8? The backend was the same in both cases, so I wouldn't expect any difference there.

@wisnij I will be looking more into this issue today. I will let you know if I have further questions.

@wisnij you will be happy to know that a fix for your issue is in the linked PR. Once that is merged and a new release is cut, your connection issues will be resolved.

all the best!

> Thanks, I'll try that for the meantime. Any idea why none of those errors showed up with 3.6.8? The backend was the same in both cases, so I wouldn't expect any difference there.

The issue has been present for a while but was significantly exacerbated in 3.6.9. It is actually part of a broader lifecycle management problem that I am planning on addressing. In the meantime, the PR linked should resolve your immediate connection issues.

All the best.

Wonderful, thank you. I'll give that a try once the release is out.