zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Retry decorator fails to catch `LengthFinishReasonError` and other exceptions when using AzureOpenAI
Root cause: Validation Gap

**Describe the bug**
When decorating a function with Tenacity’s `@retry`, only `openai.RateLimitError` is being retried—even though a broader set of “retryable” exceptions is defined. In particular, exceptions like `openai.LengthFinishReasonError`, `openai.APIConnectionError`, and `openai.APITimeoutError` are never retried. This issue only manifests when calling the AzureOpenAI endpoint.


**To Reproduce**

Use Azure OpenAI


**Expected behavior**
All exceptions listed in `retryable_exceptions` (including `openai.LengthFinishReasonError`, `openai.APIConnectionError`, and `openai.APITimeoutError`) should trigger the same retry/backoff logic when using AzureOpenAI.


**Suggested fix**
Pass the full tuple of retryable exceptions into `retry_if_exception_type` rather than a single exception class:

```diff
- @retry(
-     wait=wait_exponential_jitter(initial=1, exp_base=2, jitter=2, max=10),
-     retry=retry_if_exception_type(openai.RateLimitError),
-     after=log_retry_error,
- )
+ @retry(
+     wait=wait_exponential_jitter(initial=1, exp_base=2, jitter=2, max=10),
+     retry=retry_if_exception_type(retryable_exceptions),
+     after=log_retry_error,
+ )
def call_azure_openai(...):
    ...
```
https://github.com/confident-ai/deepeval/blob/1b0563b01601e2d89bc34dcbe9d7bdfbb9a018e0/deepeval/models/llms/azure_model.py#L69-L73

https://github.com/confident-ai/deepeval/blob/1b0563b01601e2d89bc34dcbe9d7bdfbb9a018e0/deepeval/models/llms/azure_model.py#L132-L136

This change ensures that any exception in your `retryable_exceptions` tuple—including those raised specifically by AzureOpenAI—will correctly trigger retries.


Hi, I just ran into this. Is there any progress or workaround for this?
Repository: confident-ai/deepeval
Issue: Faithfulness metric measure has many retries for ContentLimitReached Error.
Root cause: Configuration Error

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
For using the new `faithfulness` metric, the retries go up to 45 times when encountering `ERROR:root:OpenAI Error: Could not parse response content as the length limit was reached..` error. 

**To Reproduce**
Steps to reproduce the behavior:
```python
:
 test_case_3 = LLMTestCase(input=question, actual_output=model_answer, retrieval_context=[english_answer])
 faithfulness_metric.measure(test_case_3)
:
```

**Expected behavior**
I expect the APIs to stop after a couple of retries, but they went up to 45 times before I cancelled the job. 

**Screenshots**

<img width="1512" alt="Image" src="https://github.com/user-attachments/assets/f608d4fc-f2a7-4ec8-9c35-44d82d9e64f3" />

**Desktop (please complete the following information):**
 - OS: iOS

**Additional context**
Add any other context about the problem here.


hey @AnanyaRaval seems like the max number of retries is 10, for the gpt-4o model you're using, did you use a custom model

do you know if your length error is of type `openai.LengthFinishReasonError`?

@penguine-ip : I am using `gpt-4o-mini`. Not sure about `openai.LengthFinishReasonError`. The screenshot attached is the only output I see.

Same problem here. Not using any custom models, just gpt-4o-mini from openai. I even get the same number like in the screenshot for completions_tokens which is strange. 
Also I couldn't find an easy way to change hyper parameters for LLM-judge like max_tokens and temperature. Seems like in order to do that I have to create a custom LLM.

I am running into the same issue with gpt-4.1-mini

Exactly. Its getting superslow while using OpenAI API due to all the retries.

And with custom/Azure models it fails completely.

I belive problem is in max output tokens.  And it seems if you are using gpt-40 Mini you are hitting that limit:

<img width="839" height="290" alt="Image" src="https://github.com/user-attachments/assets/671e381d-b0ef-47c7-bdf2-ba6d82089b97" />

After i switched to 4.1 mini which as double the max output tokens problem dissapeared

Nope, using 4.1-mini all along.. I noticed limiting number and size of context chunks helps, but with retries, this is not the main issue.

Main issue is that while using AzureAI (and I believe anything else than OpenAI), these retries are not specified, so eval does not work at all. But please prove me wrong.
Repository: confident-ai/deepeval
Issue: Error in BiasMetric synchronous measure() function
Root cause: Algorithmic Error

**Describe the bug**
Exception thrown when using `Bias` metric in `sync` mode - 

```python
Traceback (most recent call last):
:
    bias_metric.measure(test_case_1)
  File "/Users/ananyaraval/workspace/multilingual_humanibench/deepeval_venv/lib/python3.10/site-packages/deepeval/metrics/bias/bias.py", line 67, in measure
    self.verdicts: List[BiasVerdict] = self._generate_verdicts()
  File "/Users/ananyaraval/workspace/multilingual_humanibench/deepeval_venv/lib/python3.10/site-packages/deepeval/metrics/bias/bias.py", line 206, in _generate_verdicts
    verdicts = [item for item in res.verdicts]
  File "/Users/ananyaraval/workspace/multilingual_humanibench/deepeval_venv/lib/python3.10/site-packages/pydantic/main.py", line 989, in __getattr__
    raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'BiasVerdict' object has no attribute 'verdicts'. Did you mean: 'verdict'?
```

**To Reproduce**
```python
 bias_metric = BiasMetric(model=MODEL, async_mode=False, include_reason=False, strict_mode=True)
 test_case_1 = LLMTestCase(input="N/A", actual_output=model_answer)
 bias_metric.measure(test_case_1)
```

**Expected behavior**
No error and return a bias score.

**Desktop (please complete the following information):**
 - OS: iOS
 
**Possible Fix**
 Replace `res.verdicts` with `res.verdict` in [metrics/bias/bias.py](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/bias/bias.py#L180) as `verdict` attribute is present in the definition of [BiasVerdict](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/bias/schema.py#L11).



Please let me know if I am missing something. If the above debugging is accurate, I'd be happy to push in this (or any other) fix as well :)

@AnanyaRaval hey! psuhed a fix, releasing in a few hours! 

@penguine-ip Thank you :) 

@AnanyaRaval fixed!
Repository: confident-ai/deepeval
Issue: Custom LLM evaluator regression since 874bab4
Root cause: Validation Gap

**Describe the bug**
since [this change](https://github.com/confident-ai/deepeval/commit/874bab425a6a461737d5822b78a00734ba97b8dc#diff-38c26e17cfb73d8824826391c364fb0201fbb5a29339ee206e6bd3d31f1cee39R22) added get_actual_model_name,  custom evaluators require implementing `__init__` with `super().__init__(model_name)`. 
The typing of base_model.model_name is [Optional](https://github.com/confident-ai/deepeval/commit/874bab425a6a461737d5822b78a00734ba97b8dc#diff-b253afe967075b83dc2a1725cc400792fce94822b04b7aa45f53bf22f6304fd5R34), which it should be as per backwards compatability and docs (this broke my setup).

The example in docs would provide an error: https://www.deepeval.com/guides/guides-using-custom-llms#creating-a-custom-llm

This is now required;
```python
class Claude(DeepEvalBaseLLM):

	def __init__(self):
		super().__init__('Claude')
```

**To Reproduce**
Steps to reproduce the behavior:
1. Implement a custom LLM
2. Run the code
3. Get 'NoneType' is not iterable error

```python
test/test_llm_as_judge.py:42: in test_llm_as_judge
    evaluator_llm = Claude()
.venv/lib/python3.13/site-packages/deepeval/models/base_model.py:36: in __init__
    self.actual_model_name = get_actual_model_name(model_name)
.venv/lib/python3.13/site-packages/deepeval/models/utils.py:22: in get_actual_model_name
    if "/" in model_name:
E   TypeError: argument of type 'NoneType' is not iterable
```

![Image](https://github.com/user-attachments/assets/d5328ac3-0112-4b85-b288-0d5d030955ab)

**Expected behavior**
The custom llm should work without extending the super() with a model_name like previous versions and the example in the docs.


**Desktop (please complete the following information):**
Deepeval version: `deepeval==2.8.9`


Fixed and released: https://github.com/confident-ai/deepeval/commit/ac310dc07f1900ca29e3a39bc0d518010fd0e5da @kbarendrecht, can you let me know if your previous code works once you update? Thanks appreciate a lot!

Thanks for the fix @penguine-ip. I have verified this works 👍🏻
Repository: confident-ai/deepeval
Issue: GEval TypeError: unsupported operand type(s) for +=: 'NoneType' and 'float'
Root cause: Validation Gap

# Bug Report: GEval TypeError: unsupported operand type(s) for +=: 'NoneType' and 'float'

## Describe the Bug
When evaluating a custom `GEval` metric (e.g., `custom_faithfulness_metric`), a `TypeError` occurs in the `_evaluate` method of the `GEval` class. The error is caused by attempting to increment `self.evaluation_cost` (which is `None`) with a `float` value (`cost`).

### Error Message
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'float'


### Traceback
```python
<ipython-input-21-dddfe40f138e> in metric_score(metric)
     96 
     97 def metric_score(metric):
---> 98     metric.measure(test_case)
     99     if isinstance(metric, GEval) and hasattr(metric, 'name'):
    100         metric_name = metric.name

/usr/local/lib/python3.11/dist-packages/deepeval/metrics/g_eval/g_eval.py in measure(self, test_case, _show_indicator)
     84                     self._generate_evaluation_steps()
     85                 )
---> 86                 g_score, reason = self._evaluate(test_case)
     87                 self.reason = reason
     88                 self.score = float(g_score) / 10

/usr/local/lib/python3.11/dist-packages/deepeval/metrics/g_eval/g_eval.py in _evaluate(self, test_case)
    291                 prompt, top_logprobs=self.top_logprobs
    292             )
--> 293             self.evaluation_cost += cost
    294             data = trimAndLoadJson(res.choices[0].message.content, self)
    295 
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'float'
```

### Additional Context
The issue appears to have started occurring today (May 15, 2025), suggesting a possible recent change in the deepeval library or a dependency.
A similar issue was reported with AzureOpenAILLM (TypeError with NoneType and str) [link to issue #1438: https://github.com/confident-ai/deepeval/issues/1438], but this case involves a float type and a custom Hugging Face model.
The custom model is implemented using transformers.pipeline with Qwen2.5-1.5B-Instruct.


hey @PerrineQhn are you using a custom LLM?

> hey [@PerrineQhn](https://github.com/PerrineQhn) are you using a custom LLM?

Yes, I'm using a CustomHuggingFaceLLM class, but I had the exact same script yesterday and it worked, but today, I have this error.

> hey [@PerrineQhn](https://github.com/PerrineQhn) are you using a custom LLM?

Well, I retry this morning and it works, I don't have any error...

I am having the same issue, using LiteLLMModel
Repository: confident-ai/deepeval
Issue: Incorrect aggregates in the CLI
GitHub labels: bug

**Describe the bug**
The CLI has incorrect aggregates.

**To Reproduce**
Run the CLI To Check This

**Expected behavior**
The test cases should be correct

**Screenshots**
See #159 

**Additional context**
Important for Github Actions


Update - not an issue as it contained a test that wasn't a specific deepeval test case (it was just a normal test)
Repository: confident-ai/deepeval
Issue: BAML for ensuring that the required output format is maintained
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
We are using the DeepEval framework to evaluate LLM responses using a judge model. To avoid relying on proprietary models like OpenAI, we're using an open-source model—specifically LLaMA 3.1 8B—as our judge.

However, we've encountered an issue where the judge model occasionally outputs responses that do not conform to the expected format required by DeepEval. This causes the evaluation process to fail due to parsing or formatting errors.

**Describe the solution you'd like**
We’d like to incorporate a mechanism—potentially using baml (https://github.com/BoundaryML/baml) within the DeepEval framework to enforce or validate the output structure of the judge model. This would ensure that the model’s responses are reliably structured and compatible with DeepEval’s expected input schema, preventing evaluation failures due to format mismatches.

Describe alternatives you've considered
Implementing a wrapper or post-processing step around the judge model to reformat the output manually.

Using a more robust parsing strategy with fallbacks inside DeepEval.

Switching to a more reliable model (not ideal due to our desire to stay open-source and lightweight).

**Additional context**
Our use case prioritizes using small, open-source models to evaluate other LLMs, but we need stronger guarantees around output formatting. Adding built-in schema enforcement or structured output handling (e.g., via baml) would greatly improve robustness and flexibility in real-world setups.

hey @nxtr-kiranshivaraju replyin in discord
 :)
Repository: confident-ai/deepeval
Issue: GSM8K: `AttributeError: 'tuple' object has no attribute 'answer'`
Root cause: Interface Contract Mismatch

**Describe the bug**

It seems impossible to avoid `AttributeError: 'tuple' object has no attribute 'answer'` with GSM8K.

**To Reproduce**

Running the code from https://github.com/confident-ai/deepeval/blob/v2.9.1/docs/docs/benchmarks-gsm8k.mdx#usage

```python
from deepeval.benchmarks import GSM8K
from deepeval.models import GPTModel

benchmark = GSM8K(n_problems=10, n_shots=3, enable_cot=True)

benchmark.evaluate(model=GPTModel())
print(benchmark.overall_score)
```

I get:

```none
...
  File "/path/to/repo/scripts/deepeval_play.py", line 11, in <module>
    benchmark.evaluate(model=GPTModel())
  File "/path/to/repo/.venv/lib/python3.12/site-packages/deepeval/benchmarks/gsm8k/gsm8k.py", line 53, in evaluate
    prediction, score = self.predict(model, golden).values()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/repo/.venv/lib/python3.12/site-packages/deepeval/benchmarks/gsm8k/gsm8k.py", line 99, in predict
    prediction = str(res.answer)
                     ^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'answer'
```

**Expected behavior**

No `AttributeError` crash when running this code.

**Additional context**

A similar same error was hit in https://github.com/confident-ai/deepeval/issues/1535.

DeepEval should use `mypy`.


@jamesbraza hey is mypi easy to add, would really appreciate a PR on this!

Hey @penguine-ip and @jamesbraza, I have try to resolve this issue, please review my PR, thanks 

@penguine-ip is this issue fixed?
Repository: confident-ai/deepeval
Issue: telemetry opt out breaks evaluation
Root cause: Validation Gap

**Describe the bug**
With telemetry opt out enabled, evaluations fail to complete.

**To Reproduce**
Steps to reproduce the behavior:
1. Use 2.9.4
2. Set env var `DEEPEVAL_TELEMETRY_OPT_OUT` to `YES`
3. Run a pytest like evaluation, `deepeval test run  evaluation/evaluation.py`

**Expected behavior**

Evaluation should work.

**Error**

```
.venv/lib/python3.13/site-packages/deepeval/telemetry.py:180 in capture_evaluation_run 

│   178 │   set_last_feature(Feature.EVALUATION)                                 
│   179 │   # capture posthog                                                    
│ ❱ 180 │   posthog.capture(distinct_id=distinct_id, event=event, properties=properties)                                                           
│   181 │   # capture new relic
│   182 │   with tracer.start_as_current_span(event) as span:
│   183 │   │   for property, value in properties.items():                       

NameError: name 'posthog' is not defined
```

**Analysis**

Not familiar with much of the code, but:

`capture_evaluation_run` and other methods appear to have been modified recently:

Previously (2.9.1 for example) the `if telemetry_opt_out(): ... else ...` clause  lead to yielding if opt out is enabled and then returning from the method / context manager [(source)](https://github.com/confident-ai/deepeval/blob/v2.9.1/deepeval/telemetry.py#L161-L179).

[The current logic](https://github.com/confident-ai/deepeval/blob/02085255fd983f7ad1503d76747ccece6c0c06f0/deepeval/telemetry.py#L162-L185) (for example found in 2.9.4) appears to
- try to yield twice if opt out is enabled
    - which basically means it tries to do telemetry even when opt out is enabled (which I'd expect to fail with `RuntimeError: generator didn't stop`)
- but fails because it tries to use variables [that are not initialized](https://github.com/confident-ai/deepeval/blob/02085255fd983f7ad1503d76747ccece6c0c06f0/deepeval/telemetry.py#L81-L124) because opt out is enabled (posthog, tracer, ...) 


The pattern seems to repeat in `capture_recommend_metrics`, `capture_metric_type`, `capture_synthesizer_run`, `capture_conversation_simulator_run`, `capture_red_teamer_run` and maybe elsewhere.

@stsc3000 latest release should have fixed it, thanks for letting u sknow!

@penguine-ip Solved, thank you!
Repository: confident-ai/deepeval
Issue: ModuleNotFoundError: No module named 'deepeval.integrations.integrations'
Root cause: Environment Incompatibility

Despite install latest version of deepeval, it still met error "ModuleNotFoundError: No module named 'deepeval.integrations.integrations'".
Check in the folder it seem lack of file integrations.py which is referenced by __init__.py

![Image](https://github.com/user-attachments/assets/94002679-882c-4d63-adb1-b18f68879749)

hey @sghoang24 thanks for letting us know, we just cut a release that fixes this, can you update to 2.9.6 and let us know if it works?
Repository: confident-ai/deepeval
Issue: TypeError: unsupported operand type(s) for /: 'list' and 'int'
Root cause: Validation Gap


A TypeError occurs when running the code, specifically:
TypeError: unsupported operand type(s) for /: 'list' and 'int'
This happens in image_coherence.py at the line score = score / 10. The variable score is a list, but the code tries to divide it by an integer, which is not allowed in Python.

Run the script (e.g., python3 main.py) that uses the ImageCoherenceMetric from the deepeval package.

Expected behavior
The code should perform a numeric division on a single number (int or float), not on a list. If score is a list of numbers, the code should aggregate it (e.g., take the average) before dividing.


**Screenshots**

![Image](https://github.com/user-attachments/assets/b3599cf1-0335-4300-afaf-2843fea07bcb)


OS: macOS (darwin 24.5.0)
Browser: N/A
Version: Python 3.10.10


hey @deepeshAg and @A-Vamshi i believe its fixed in 3.0.1 (https://github.com/confident-ai/deepeval/commit/1b52189a36f55753259830f7cc7905040d9adc18#diff-cac10f0ffab564dfa6232eaf44643f86e3054ef8ba85187eea12261b2ddd07d6), thanks!
Repository: confident-ai/deepeval
Issue: Disabling "confident AI" console print when unauthenticated.
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
As a user of `deepeval test run` without a deepeval login, I want to disable the promotional print statements.

> Note: Use Confident AI with DeepEval to analyze failed test cases for more details
> ✓ Tests finished 🎉! Run 'deepeval login' to save and analyze evaluation results on Confident AI.
>  
> ✨👀 Looking for a place for your LLM test data to live 🏡❤️ ? Use Confident AI to get & share testing reports, experiment with models/prompts, and catch regressions for your LLM system. Just
> run 'deepeval login' in the CLI.

**Describe the solution you'd like**
Use an environment variable or cli arg to disable these print statements.
It's hardcoded here: https://github.com/confident-ai/deepeval/blob/dbf3c3e0254de7edc91924873c13486ea75e9861/deepeval/test_run/test_run.py#L891

**Describe alternatives you've considered**
N/A it's hard coded

**Additional context**
Add any other context or screenshots about the feature request here.


@theagenticguy not possible!

It's definitely possible lol. just not willing?
Repository: confident-ai/deepeval
Issue: FaithfulnessTemplate has bad grammar, some unnecessary text and threatening language

The [`generate_verdicts` function](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/faithfulness/template.py#L111) needs a cleanup. 

E.g,:

```
Claims that is not backed up due to a lack of information/is not mentioned in the retrieval contexts MUST be answered 'idk', otherwise I WILL DIE.
```

has bad grammar and threatening language.




hey @elzurdo can you please do a PR for us? :) 

Hey @penguine-ip ,

I tried to `git push` but get the following error

```bash
➜  deepeval git:(Issue-1634_faithfulness_template) git push --set-upstream origin Issue-1634_faithfulness_template
remote: Permission to confident-ai/deepeval.git denied to elzurdo.
fatal: unable to access 'https://github.com/confident-ai/deepeval/': The requested URL returned error: 403
``` 

Any suggestions on how to move forward? (I admit being lazy in not forking, perhaps that's the issue?)

@elzurdo i think you have to fork it..

@penguine-ip  Let me do it 

@A-Vamshi thank you !

@A-Vamshi merged thanks!
Repository: confident-ai/deepeval
Issue: Could not parse response content as the length limit was reached
Root cause: Documentation Deficiency

**Describe the bug**
I'm trying to evaluate some LLM responses but I'me getting this error message:
_OpenAI Error: Could not parse response content as the length limit was reached - CompletionUsage(completion_tokens=16384, prompt_tokens=2524, total_tokens=18908, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=2432))_

I'm using the default model (gpt-4o)... I think I need to set max_tokens to OpenAI LLM but I can't find the way... 

Anyone can help me? 
thanks a lot

Stefano

@ddmo To set max_tokens, you simply need to pass it as an argument when you initialize your GPTModel instance within deepeval.

like this:
```python
from deepeval.models.llms.openai_model import GPTModel
# Then:
my_gpt_model = GPTModel(model="gpt-4o", max_tokens=4096, _openai_api_key="YOUR_OPENAI_API_KEY")
```

Great, thanks!
Repository: confident-ai/deepeval
Issue: Anthropic dependency pin blocks Claude 4 models
Root cause: External Dependency Breakage

`deepeval==3.0.5` still caps **anthropic** at `<=0.51.*`.  
Anthropic 0.52 (released 2025-05-22) is the first version that exposes the Claude 4 model family, plus new Files API and code-execution tool support. Projects that depend on Claude 4 therefore cannot install DeepEval.

### Affected package versions
* DeepEval 3.0.5 (latest as of 2025-06-05) pins anthropic in your lockfile to: `0.49.0`

### Steps to reproduce
```bash
poetry add anthropic@^0.52 deepeval --group dev
# → dependency solver error: deepeval requires anthropic<=0.51.*


We've run into the same issue. Trying to use Anthropic 0.52.2 with deepeval 3.0.5 and have to use [uv override-dependencies](https://docs.astral.sh/uv/reference/settings/#override-dependencies) to resolve.

We have the same issue, +1

I ended up making a fork, updating the dependencies that were causing issues (anthropic and tenacity), tagging the commit, and pointing poetry to my fork, at that tag.