Repository: confident-ai/deepeval
Issue: Asynchronous test runs are sometimes not completed correctly
Root cause: Resource Mishandling

**Describe the bug**
When running `evaluate()` with `run_async=True`, sometimes tests are not completed (so any job/task/pipeline that relies on exit code will fail). Results are neither printed nor emitted. The evaluation is essentialy stuck after running the last test case. It is not always the case, without any regularity, so it looks like a race condition. Likely there is issue with async stuff in evaluate.py file - `a_execute_test_cases()`, ` get_or_create_event_loop()` or `loop.run_until_complete`. 

It might be possible that `await asyncio.sleep(throttle_value)` leads to semaphore being stuck or something. I haven't debug it except a brief static code analysis, though.

**To Reproduce**
Steps to reproduce the behavior:
1. Create a test cases list
2. Run them using `evaluate()` with `run_async=True`
3. All tests are executed asynchronously and that's fine
4. Results are not printed, saved, etc. It is essentialy stuck after running the last test case

**Expected behavior**
Tests always end with printing either results or errors. They should never last infinitely

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS] Windows 11


@jmaczan I never encountered this issue, can you give us something to reproduce it? For example, number of test cases, metric you're using, etc.

@penguine-ip ,
I think I met similar issues before. I have about 130 test cases, the metric I use are:
Hallucination, Answer Relevancy, Faithfulness, Contextual Precision, Contextual Recall, Contextual Relevancy

with run_async=False, it finished without any issue


@penguine-ip I have about 20 test cases, I use literally the same set of metrics as @rjiangnju. `run_async=False` solves the issue, but to me it's a no-go since I run each case for multiple times to remove top and bottom results and draw median value to get more stabilized metrics

I have the same issue, my metrics are GEval and Answer Relevancy. I run `evaluate` method multiple times (about 100 test cases per run) in my script and sometimes it gets stuck near the end. Running with run_async=False works fine, but its much slower. I also tried to split the number of test cases into more groups, but even with just 3 test cases per `evaluate` it still gets stuck randomly.

> I have the same issue, my metrics are GEval and Answer Relevancy. I run `evaluate` method multiple times (about 100 test cases per run) in my script and sometimes it gets stuck near the end. Running with run_async=False works fine, but its much slower. I also tried to split the number of test cases into more groups, but even with just 3 test cases per `evaluate` it still gets stuck randomly.

To add to this, I've been running the code in jupyter notebook on a Windows 10, but when I transfered the code to a simple python file and executed it, everything worked fine. However, since this bug occurs randomly, I may just have gotten lucky.

I have similar experiences, it fails for me in Azure Pipelines, but seems to work fine locally

@jmaczan @threeteck @rjiangnju Hey all, if it works fine locally but not in some environments then it might be because we're writing to file to cache results. When you run `evaluate(write_cache=False)`, it no longer writes to file - can you quickly try this and let me know if it solves the problem?

For other environments, the OS might be handling file locking differently. Basically, when we run things in async, we need to make sure the cache we're writing is always in its most updated state, so sometimes it cases different coroutines to deadlock each other in reading and writing to file.

Hi @penguine-ip ,

Just tried with write_cache=False and It seems work on my side. The os is windows 11, but the code is running in a docker container with Debian 12 by WSL, hope this can help solve the issue.

Unfortunately it doesn't work for me, I keep having the same issue. I'm also running pipeline on Ubuntu and I'm using deepeval 1.5.0

@rjiangnju could you share all other parameters you use in evaluate(), along with 'write_cache=False'? 

@jmaczan , here is the command I use:
evaluation_result = evaluate(dataset, metrics, run_async=True, throttle_value=10, max_concurrent=3, show_indicator=True, print_results=False,ignore_errors=True, write_cache=False)

Maybe also related to the metric, I changed a little, removed Hallucination and added two geval metrics.

@jmaczan @rjiangnju yup you can find it here: https://docs.confident-ai.com/docs/evaluation-introduction#evaluating-without-pytest

I have the same problem, using Contextual Relevancy and GEval metrics and an Azure LLM.

I first suspected I was running into a rate limit but for some reason the evaluation gets stuck close to the end independent of the number of test cases which speaks against the rate limit theory: If it is only 10 testcases it gets stuck at 9/10 very often and for 30 testcases the first ~25 are super fast, then it slows down and around 27/30 or 28/30 where it gets stuck completely.

Is there anything I can monitor/set a breakpoint to find out the cause?

So I tracked it down to the contextual relevancy metric for me, I see that it sometimes gets stuck there and never returns. I assume it gets stuck waiting for an LLM response, either because the metric sends so many LLM calls at once or maybe if there are invalid characters in the prompt (?). 
Either way, maybe it is possible to set a timeout inside the metric somewhere?
For now I set a asynchio.timeout in indicator.py in the safe_a_measure method around the "await metric.a_measure(tc, _show_indicator=False) that sets the metric.score = 0 if a_measure does't return in 60 seconds, but there is probably a better way to handle this as this would affect all metrics, maybe it is better to put the timeout directly inside the contextual relevancy metric.

Hey @penguine-ip, is there any chance you will prioritize fixing this soon?

Hey @IssaP19 and @jmaczan I'm not sure what is causing the problem but recently there is another issue on things getting stuck: https://github.com/confident-ai/deepeval/issues/1205

And it seems like its not related to deepeval. That is assuming that you are using another model that would potentially face the same problem.

Can you confirm what metrics does it occur on and which model you're using for evaluation, as well as what version of deepeval are you on? 

Also its very easy for us to miss messages in issues since we mainly operate in our discord server, please join there if you want better reply rate

I've traced the issue to a_invoke from AzureChatOpenAI sometimes not returning a response when running a_generate. Adding a timeout parameter to the retry function could solve the issue. I'm currently facing the issue while running the role_adherence metric.

## Description
DeepEval execution gets stuck/hangs when running with run_async=True using Ollama as the backend.

## Environment

DeepEval version: 3.0.6
Linux 5.14.0-239.el9.x86_64 x86_64 GNU/Linux CentOS Stream 9
CPU: 2xAMD EPYC 7313P 16-Core Processor 32 L2 512 KB, RAM: 515140 MB
GPU: 2xNVIDIA A10 Tensor Core 24 GB GDDR6
CUDA Version: 12.6
Python Version: 3.12
Model: selene-mimi:fp_16 (via Ollama)
Test cases: 90
Configuration: run_async=True

## Expected Behavior
Test execution should complete successfully with async processing enabled.

## Actual Behavior
The execution sometimes gets stuck and doesn't progress, requiring manual termination and rerun.

## Steps to Reproduce

Set up Ollama with selene-mimi:fp_16 model
Configure DeepEval to use Ollama backend
Run evaluation with 90 test cases
Set run_async=True
Execution hangs indefinitely

## Additional Context

Works fine with run_async=False (if applicable)
GPU memory usage: Around 16GB
No obvious error messages in logs
Issue appears to be related to async concurrency handling

## Code Sample

```python
results = evaluate(cache_config=CacheConfig(write_cache=True, use_cache=True),
                                   error_config=ErrorConfig(ignore_errors=False),
                                   async_config=AsyncConfig(run_async=True),
                                   test_cases=casos_evaluables, metrics=metrics)
```

Hi @carloss4dv thank you for reporting this. Can you test again with the latest version of deepeval, currently `3.5.9`. There has been a lot of work done to avoid hangs during async runs. Please report back and let me know your results.

Related: #984 ( lots of PRs to address stalls and hangs in both async and sync flows )