Repository: confident-ai/deepeval
Issue: Evaluation hangs at 0% progress with non-Claude models (GPT-5, etc.)
Root cause: Configuration Error

## Problem Description

DeepEval evaluation process hangs indefinitely at 0% progress when using evaluation models other than Claude Sonnet family. This creates a significant burden for users who need to rely on expensive Claude models for evaluation.

## Specific Issues Observed

### GPT Model Behavior
- **GPT-4**: Previously worked well for evaluation
- **GPT-5**: Complete failure - evaluation hangs at 0% progress across all variants (mini, nano, etc.)
- **Internal/Enterprise Models**: Similar hanging behavior when GPT-4 was removed and replaced with GPT-5

### Current Workaround
Users are forced to use Claude Sonnet models for evaluation, which are significantly more expensive than alternatives.

## Expected Behavior
Evaluation should work reliably with various LLM providers and models, not just Claude Sonnet family.

- Operating system: MacOS

## Reproduction Steps
1. Set up evaluation with GPT-5 (or other non-Claude model)
2. Run evaluation test
3. Observe evaluation hangs at 0% progress
4. No error messages or timeout warnings

## Impact
- **Cost**: Forces users to use expensive Claude models
- **Flexibility**: Limits model choice for evaluation
- **Reliability**: Evaluation process becomes unreliable with certain models

## Request
Please investigate and fix the evaluation hanging issue with non-Claude models, particularly GPT-5 and other modern LLMs. This would significantly improve the framework's usability and cost-effectiveness.

## Additional Context
I cannot verify if this issue exists with GPT-4 or earlier versions, as my company's supported models have been updated and older versions are no longer available for testing.

He @nolleh, Are you running in sync mode?

I believe I can unblock you while I work on a fix. Near the start of your script, please add the following to set `DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS`:

```
from deepeval.config.settings import get_settings

settings = get_settings()
with settings.edit(persist=False):
    settings.DEEPEVAL_DEBUG_ASYNC = True
    settings.DEEPEVAL_VERBOSE_MODE = True
    settings.LOG_LEVEL = "debug"

    # Ensure DeepEval’s Tenacity retry policy is used (SDK retries off)
    settings.DEEPEVAL_SDK_RETRY_PROVIDERS = []
    settings.DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS = 40  # <- this is the important setting. Please make sure this is not None and is > 0. Increase if you need to.
    settings.DEEPEVAL_RETRY_MAX_ATTEMPTS = 1

// rest of your code....
```

The rest of those settings are just to increase verbosity and collect diagnostics. You can turn that off once we see this really unblocks you. Alternatively, you can just set the `DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS` as an environment variable. Up to you.

Once `DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS` is set to any number larger than 0, your script should either run to completion, or possibly timeout if the openai sdk backend takes too long to respond. Let me know if that helps. In the meantime, I am working on a fix where you won't need to do that.



Hi @nolleh,

We have released several changes that make DeepEval much harder to hang on slow/unresponsive calls and that surface the reason when timeouts happen. These are included in **DeepEval `3.6.9`**.

Highlights:

- **Cooperative timeouts**: a shared outer timeout budget for a test plus a per-attempt limit. If a timeout does occur, any unfinished metrics are marked failed and the run is persisted so you see the reason it stopped
- **Improved logging with optional tracebacks**: set `DEEPEVAL_LOG_STACK_TRACES=1` to include tracebacks in retry/timeout logs.

Note: `DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS` is now a computed value. To explicitly control it set `DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS_OVERRIDE` instead.

If you can still reproduce a hang on `3.6.9+`, please share a minimal snippet and I will investigate.


Closing: released in 3.6.9; please reopen if you can repro on 3.6.9+
Fixed by #2247