Repository: confident-ai/deepeval
Issue: feat: Add timeouts to `asyncio.gather` orchestrations in `execute.py`
Root cause: Unimplemented Feature Gap

### Problem

Several top-level `asyncio.gather` calls in `deepeval/evaluate/execute.py` do not have a timeout. If any awaited task stalls, such as model call, user callback, etc., the entire batch can hang.

**Note:** We should not add timeouts to the “drain” gathers that run in `finally/except` purely to clean up cancelled tasks. Those can remain as is, or be handled in a separate follow-up.

### Goal

Wrap orchestrating `asyncio.gather` calls with `asyncio.wait_for` using:

- `settings.DEEPEVAL_PER_TASK_TIMEOUT_SECONDS` (per-task)
- `settings.DEEPEVAL_TASK_GATHER_BUFFER_SECONDS` (buffer for the slowest straggler)

Compute a local bound like:

```py
GATHER_TIMEOUT_SECONDS = (
    settings.DEEPEVAL_PER_TASK_TIMEOUT_SECONDS
    + settings.DEEPEVAL_TASK_GATHER_BUFFER_SECONDS
)
```

### What to change

In `deepeval/evaluate/execute.py`, add a timeout to these gathers (line numbers are from the current file (treat them as an approximate location):

* `a_execute_test_cases(`

  - `await asyncio.gather(*tasks)` around **line 489**
  - `await asyncio.gather(*tasks)` around **line 562**

* `a_execute_agentic_test_cases`

  - `await asyncio.gather(*tasks)` around **line 1132**

* `_a_execute_agentic_test_case`

  ```
    child_tasks = [dfs(child) for child in span.children]
    if child_tasks:
        await asyncio.gather(*child_tasks)     # around line 1273
    ```

* `_a_evaluate_traces`

  - `await asyncio.gather(*eval_tasks)` around **line 2150**

* `_evaluate_test_case_pairs`

  * `await asyncio.gather(*tasks)` around **line 2206**

#### Leave these **unchanged** (cleanup/drain)

* Already bounded gather (OK): `loop.run_until_complete(asyncio.wait_for(asyncio.gather..., timeout=GATHER_TIMEOUT_SECONDS))` around **line 1886**
* Drain after cancellation (cleanup): `loop.run_until_complete(asyncio.gather(*created_tasks, return_exceptions=True))` around **line 1937**
* Drain leftovers: `loop.run_until_complete(asyncio.gather(*leftovers, return_exceptions=True))` around **line 1980** ⛳

**Rationale:** the latter two are post-cancellation drains; adding a timeout could re-surface another `TimeoutError` during cleanup. 

Optional: propose a tiny follow-up PR to cap drains with a generous timeout and warning logs.

### How to change it

For each orchestrating site, replace:

```py
await asyncio.gather(*tasks)
```

with:

```py
GATHER_TIMEOUT_SECONDS = (
    settings.DEEPEVAL_PER_TASK_TIMEOUT_SECONDS
    + settings.DEEPEVAL_TASK_GATHER_BUFFER_SECONDS
)

try:
    await asyncio.wait_for(
        asyncio.gather(*tasks, return_exceptions=True),
        timeout=GATHER_TIMEOUT_SECONDS,
    )
except asyncio.TimeoutError:
    # Cancel any still-pending tasks and drain them
    for t in tasks:
        if not t.done():
            t.cancel()
    await asyncio.gather(*tasks, return_exceptions=True)
    # Optional: log a concise warning referencing GATHER_TIMEOUT_SECONDS
```

Apply the same pattern to:

* `await asyncio.gather(*eval_tasks)` (use `eval_tasks` variable)
* DFS children gather (`child_tasks`)

### Acceptance criteria

* All `gather` calls listed above are wrapped in `wait_for(...)` with the computed `GATHER_TIMEOUT_SECONDS`.
* `return_exceptions=True` is used for orchestrating gathers to avoid cancellation cascades.
* Cleanup/drain gathers in `finally/except` blocks are **unchanged** (or explicitly deferred to a follow-up task - see Optional follow-up task above).
* On timeout, pending tasks are cancelled, then drained with a best-effort `gather(..., return_exceptions=True)` and a clear log message.
* No regressions in existing tests.

### Testing hints

* Create a dummy task that `await asyncio.sleep(9999)` and ensure it’s scheduled via one of the affected code paths.
* Temporarily set:

  * `DEEPEVAL_PER_TASK_TIMEOUT_SECONDS=2`
  * `DEEPEVAL_TASK_GATHER_BUFFER_SECONDS=1`
* Verify:

  * The batch returns promptly with an `asyncio.TimeoutError` handled per the pattern above.
  * Other tasks complete and results/logs are collected.
  * Progress bars continue to update (no deadlock).


Please assign it

HI @trevor-cai I’d like to work on this issue. Could you please assign it to me?

Hi @S3lc0uth  and @harshithsaiv :

I have assigned it to @S3lc0uth for about 48 hours. I will re-assign after that if there is no activity. Good luck, and thank you both for offering to contribute.

@trevor-cai 
Please review the changes https://github.com/confident-ai/deepeval/pull/2129


@S3lc0uth see my comment on your PR.

@trevor-cai Created new PR https://github.com/confident-ai/deepeval/pull/2136

@trevor-cai  if the issue is not yet resolved please assign me this.

hey @trevor-cai can i work on this?