zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Feat request: penalize_ambiguous_claims for answer relevancy
Root cause: Unimplemented Feature Gap

Answer relevancy currently does not penalise claims with an 'idk' verdict. 

You have recently introduced  'penalize_ambiguous_claims' as a parameter for faithfulness to allow the end user to determine whether 'idk' claims should penalise the score (with the default of False not penalising the score). 

A similar feature for answer relevancy would be welcome.

hey @langenk that makes sense, could add over the next 2-3 days :)
Repository: confident-ai/deepeval
Issue: Which evaluation models are best suited for minimizing score variability in contextual relevance assessments?
Root cause: Documentation Deficiency

Problem Description

When using different evaluation models (e.g., Qwen3-32B,Qwen-plus,Qwen3-max) with the same test cases and dataset, the contextual relevance scores calculated by DeepEval (via relevant_statements / total_verdicts) vary significantly. For example:

Model A returns a score of 0.85, while Model B returns 0.62 for identical inputs.
Debugging shows that the discrepancy arises from differences in how models judge relevance (i.e., "yes"/"no" verdicts in self.verdicts_list).

Current Setup

DeepEval Version: 3.6.9
Evaluation Models Tested: [Qwen3-32B,Qwen-plus,Qwen3-max]
Key Metric Used: ContextualRelevancyMetric

Could you please recommend:

Which evaluation models are best suited for minimizing score variability in contextual relevance assessments?
Whether DeepEval has pre-configured models or best practices for this specific use case (e.g., using GPT-4 as the "gold standard" evaluator)?
Repository: confident-ai/deepeval
Issue: Support Open Router
Root cause: Unimplemented Feature Gap

Can we have a nice support for open router as an evaluation endpoint ?

Hey @MohamedAliRashad we'd love to but have other priorities right now so can't get to it until mid December at least, any chance you could help make a PR? Thank you in advance.

hey, would love to work on this if possible :)


@wjunwei2001 Thank you for taking this on, your effort is appreciated. I will review your PR today.

Hey @wjunwei2001, thank you for all of your effort on your PR for OpenRouter support. I have completed my review and wanted to let you know that it is under consideration.
Repository: confident-ai/deepeval
Issue: Creating test cases from goldens drops some fields
Root cause: Algorithmic Error

**Describe the bug**
When converting a Test Case to a Golden, some fields (e.g. "input", "actual_output") are carried over to the extended Golden but others (e.g. "additional_metadata") are not. This affected me when I tried to save an EvaluationDataset of Test Cases to JSON and the additional_metadata did not appear in the saved JSON.

This behavior is consistent with and predictable from the code for convert_test_cases_to_goldens here: https://github.com/confident-ai/deepeval/blob/main/deepeval/dataset/utils.py#L14 .

Consider:
`dataset.save_as(
                    file_type='json',
                    directory=my_run_folder,
                    file_name='test_cases',
                    include_test_cases=True,
                    )`
Since include_test_cases=True, convert_test_cases_to_goldens is called per https://github.com/confident-ai/deepeval/blob/main/deepeval/dataset/dataset.py#L983.



**To Reproduce**
Create an EvaluationDataset with Test Cases that contain additional_metadata.
Save the dataset to JSON with code such as: (note that include_test_cases=True)
`dataset.save_as(
                    file_type='json',
                    directory=my_run_folder,
                    file_name='test_cases',
                    include_test_cases=True,
                    )`


**Expected behavior**
I expected that the saved JSON of test cases would include additional_metadata.

**Desktop (please complete the following information):**
 - OS: Pop_OS! 22.04
 - DeepEval Version 3.7.0

**Additional context**
I expect this would be a straightforward fix of including those other fields in the convert_test_cases_to_goldens function at https://github.com/confident-ai/deepeval/blob/main/deepeval/dataset/utils.py#L14

Thank you for DeepEval!!!


Hey @sasank-desaraju just gotta around to this, thanks for raising we will get it out next week!

Awesome thank you so much!

Hey @sasank-desaraju, this is fixed now, please ping me here if you have any other issues :)
Repository: confident-ai/deepeval
Issue: Supporting logprobs in benchmarks
Root cause: Unimplemented Feature Gap

It would be great to support logprobs to evaluate some of the benchmarks!

Some of them natively "require" scoring via logprobs rather than via the generated answer, so it would be great to allow more flexibility at this level. This could be passed as a simple argument to .evaluate(), which could then internally pick a different scoring. It would also require to likely change DeepEvalBaseLLM to allow logprobs being returned.


Hey @edoardocontente i think this is something we'd like to add for sure.. which benchmarks do you have in mind we can use for testing?

Great! Ideally HellaSwag, MMLU, TruthfulQA -- in essense benchmarks for which their canonical/ original evaluation setup utilizes logprobs

Hi @penguine-ip would love to hear more about this too!
Repository: confident-ai/deepeval
Issue: Expose metric results of Step Efficiency
Root cause: Documentation Deficiency

# Description

Document is incomplete and vague and does not show how standalone runs can be performed for trace based matrics. See section https://deepeval.com/docs/metrics-step-efficiency#as-a-standalone. give working sample for `convo_test_case`

evals_iterator() currently runs trace-based metrics (e.g., StepEfficiency, TaskCompletion) internally and uploads them to a ConfidentAI test run, but there is no way to access these metric results from Python code.

This prevents users from using trace-based metrics inside custom evaluation workflows. 

# Expected behaviour

print(metric.score)  # at the end, expected metric results should be printable

Hey @AmaliMatharaarachchi thanks for bringing this up. @A-Vamshi will the verbose logs give the info on the results?

Hey @AmaliMatharaarachchi,
Apologies for the late reply on this one, the docs are updated now, however, to answer your question you can use the metric scores right after running the `evals_iterator` as following:

```python
metric = StepEfficiencyMetric()

for golden in dataset.evals_iterator(metrics=[metric]):
    agent(golden.input)

print(metric.score, metric.reason)
```

The agentic metrics require the full trace of the application so it's not usable as a standalone where you do `metric.measure(test_case)` but you can use the scores after running in the `evals_iterator` or `observe` tag. 

Please let me know if you have any doubts or any feature requests on these metrics. 

And as @penguine-ip mentioned, you can see the scores and reasons in verbose mode as well.
Repository: confident-ai/deepeval
Issue: a_execute_test_cases raises only when not show_indicator
Root cause: Validation Gap

https://github.com/confident-ai/deepeval/blob/79723138a662d181cf22d5e2bcbdcc94b84f368c/deepeval/evaluate/execute.py#L806

When the method branches at `if display_config.show_indicator and _use_bar_indicator:` it enters two very similar pieces of boilerplate code, however in the `else` path there is a `raise` within the `except (asyncio.TimeoutError, TimeoutError)`, but not in the `if` path.

I can't see a reason for this divergent behaviour, and it means that `a_execute_test_cases` will raise only in the case that we don't show the indicator.

Hey @alexdolphin-egym, thank you for raising this issue. A fix is on its way.
Repository: confident-ai/deepeval
Issue: Feature request: add CITATION.cff

**Is your feature request related to a problem? Please describe.**
I'd like to cite this repository.

**Describe the solution you'd like**
Add github citation file format (CITATION.cff) into the root folder of the project.

**Additional context**
You may version citations by uploading to Zenodo, as publications may use different versions.

It turns out I have difficulties about reading.
Repository: confident-ai/deepeval
Issue: Parameter "additional_metadata" from "Golden" and "LLMTestCase" classes should also accept List of dicts
Root cause: Documentation Deficiency

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
I just can add one dict of metadata.

**Describe the solution you'd like**
It would be super useful if the additional metadata parameter would accept List of dicts. Sometimes agents return a List of metadata when querying from multiple vector dbs.

**Describe alternatives you've considered**
additional_metadata also accepts List of Dict


**Additional context**
Add any other context or screenshots about the feature request here.


Hey @KevinGarrison Thanks for opening this and for the detailed explanation of your use case 🙏

Right now `additional_metadata` is defined as a generic `Dict` on both `Golden` and `LLMTestCase`. It’s meant to be a flexible container for any JSON-serializable structure, including lists and nested dicts. For example, if your agent returns a list of metadata from multiple vector DBs, you can model it like:

```python
additional_metadata={
    "vector_db_metadata": [
        {"db": "db1", "metadata": {...}},
        {"db": "db2", "metadata": {...}},
    ]
}
```

Does that support your use case? Let me know if this resolves your issue, I am looking forward to hearing back from you.

Hey @trevor-cai thank you for your help. Yes that helps a lot. I‘m looking forward to use your framework in future projects, if i have some questions i will reach out to you guys.

Thank you @KevinGarrison, I am glad I could help. I am closing this issue now. Fee free to reach out on our discord channel anytime.
Repository: confident-ai/deepeval
Issue: AnthropicModel fails with SecretStr when ANTHROPIC_API_KEY is set in environment
Root cause: Validation Gap

`AnthropicModel` fails with `TypeError: Header value must be st1. Convert SecretStr` to string when present

## Root Cause

The issue stems from a difference in how `AnthropicModel` handle API keys in the `_build_client` method:

### AnthropicModel (fails)
```python
def _build_client(self, cls):
    settings = get_settings()
    kw = dict(
        api_key=settings.ANTHROPIC_API_KEY or self._anthropic_api_key,  # Checks settings first!
        **self._client_kwargs(),
    )
```

## Expected Behavior

When `_anthropic_api_key` is provided as a parameter, it should take precedence over `settings.ANTHROPIC_API_KEY`, and the API key should always be converted to a plain string before being passed to the Anthropic SDK.

## Suggested Fix

Update `AnthropicModel.__init__`  https://github.com/confident-ai/deepeval/blob/6e68220ca02c0353819b53cc53826252e822d022/deepeval/models/llms/anthropic_model.py#L44 to:

```python
        # Get API key from key handler if not provided
        self._anthropic_api_key = _anthropic_api_key or KEY_FILE_HANDLER.fetch_data(
            ModelKeyValues.ANTHROPIC_API_KEY
        )
```

and `__build_client`
https://github.com/confident-ai/deepeval/blob/6e68220ca02c0353819b53cc53826252e822d022/deepeval/models/llms/anthropic_model.py#L160 to:

```python
            api_key=self._anthropic_api_key,
```


@enrique-antezana thank you for reporting this and for providing relevant detail, I appreciate it. A fix is on its way.
Repository: confident-ai/deepeval
Issue: Issue: Bad JSON Templates in DeepEval
Root cause: Validation Gap

## Problem

DeepEval's prompt templates in `dag/templates.py` had two issues:

1. Used Python boolean syntax (`True`/`False`) in JSON examples instead of valid JSON (`true`/`false`)
2. No explicit instruction to avoid markdown code fences

This caused LLMs to return responses like:
```
\`\`\`json
{
    "verdict": True,
    "reason": "..."
}
\`\`\`
```

### Reproduction

I used `gpt-4o` (via internal company's API). \
The simple evaluation with GEval and DAGMetric from `deepeval` library failed for me **because of parsing error**.


## Proposed Fix

Improve the templates to (example from `deepeval/metrics/dag/templates.py` for `TaskNodeTemplate`):

Before:
```text
===END OF INSTRUCTIONS===

**
IMPORTANT: Please make sure to only return in JSON format, with the 'output' key as the output from the instructions.
Example JSON:
{{
    "output": "your output goes here"
}}
**

JSON:
```

After:
```text
===END OF INSTRUCTIONS===

**
IMPORTANT: Return ONLY raw JSON without markdown code fences. Use valid JSON values (true/false/null, not Python True/False/None).
Example JSON:
{{
    "output": "your output goes here"
}}
**

JSON:
```

### Files to modify

- `deepeval/metrics/dag/templates.py`
- `deepeval/metrics/g_eval/template.py`

Note: probably the similar problems exist in other templates.


Thanks for reporting this @RodionfromHSE , I am taking a look now.

@RodionfromHSE a fix is on its way.
Repository: confident-ai/deepeval
Issue: Issue: Useless Reasoning (Verdict Before Reason)
Root cause: Algorithmic Error

## Problem

DeepEval's prompt templates in `dag/templates.py` and `g_eval/template.py` asked for `verdict`/`score` before `reason` in JSON examples:

```json
{
    "verdict": true,
    "reason": "..."
}
```

This **causes the model to decide the verdict first, then generate a reason to justify it**. This makes the reasoning useless.

## Proposed Fix

Reorder JSON fields to put `reason` first and add explicit instruction to think before judging.

Example from `deepeval/metrics/dag/templates.py` for `BinaryJudgementTemplate`:

Before:
```text
**
IMPORTANT: Please make sure to only return in JSON format.
Example JSON:
{{
    "verdict": true,
    "reason": "your reason goes here"
}}
**

JSON:
```

After:
```text
**
IMPORTANT: Return ONLY raw JSON without markdown code fences. Use valid JSON boolean values (true/false, not Python True/False).
The "reason" field must come FIRST - think through your reasoning before deciding the verdict.
Example JSON:
{{
    "reason": "your reasoning here",
    "verdict": true
}}
**

JSON:
```

### Files to modify

- `deepeval/metrics/dag/templates.py` (BinaryJudgementTemplate, NonBinaryJudgementTemplate)
- `deepeval/metrics/g_eval/template.py` (generate_evaluation_results, generate_strict_evaluation_results)


Hey @RodionfromHSE this is a great observation, thanks for bringing this up, I'll change it and let you know once its our later this week :) 

Hey @RodionfromHSE it is now in main, will be released in a coupile of hours!
Repository: confident-ai/deepeval
Issue: ARC benchmark should have several "modes" per question
Root cause: Unimplemented Feature Gap

Hi,

The ARC benchmark, registered in [deepeval/benchmarks/arc/arc.py](https://github.com/confident-ai/deepeval/blob/8f7aa0a5accf4148c51f5282bb7fd39f37c4661b/deepeval/benchmarks/arc/arc.py) should have several modes *per question*, that should each correspond to a different "self.confinement_instructions" (or rather, turning the attribute self.confinement_instructions into a dict, similar to the TruthfulQA implementation, found in [deepeval/benchmarks/truthful_qa/truthful_qa.py](https://github.com/confident-ai/deepeval/blob/8f7aa0a5accf4148c51f5282bb7fd39f37c4661b/deepeval/benchmarks/truthful_qa/truthful_qa.py)). This is because while majority of questions are as follows:
e.g.

```
'Which are two parts of the carbon cycle?

        A. freezing and thawing

        B. growth and reproduction

        C. evaporation and precipitation

        D. photosynthesis and respiration

        Answer: '
```

There are some of the following type:

```
'Different species of carnivorous animals that share the same habitat
        in an ecosystem may

        1. become decomposers

        2. compete for food

        3. produce their own food

        4. mate with each other

        Answer: '
```

While the self.confinement_instructions is irrespective of the question always "Output 'A', 'B', 'C', or 'D'. Full answer not needed.". This necessarily induces the model to err and try to answer using A, B, C or D.

Thank you!

I see the ARC benchmark is extracted from . Here you can see that these [choices](https://huggingface.co/datasets/allenai/ai2_arc/viewer/ARC-Challenge/train?row=66&views%5B%5D=arc_challenge_train&sql=--+Find+entries+where+the+choices+labels+contain+numeric+values+%28%271%27%2C+%272%27%2C+etc.%29%0ASELECT+*%0AFROM+arc_challenge_train%0AWHERE+regexp_matches%28list_any_value%28choices.label%29%2C+%27%5E%5B0-9%5D%2B%24%27%29%0ALIMIT+100%3B) are specified in the 'choices' column, the "key" is "label"! Some are ["1","2","3","4"] while others are ["A","B","C","D"]. This should make it easy for a conditional patch!
Repository: confident-ai/deepeval
Issue: MultimodalOpenAIModel throws array_above_max_length (400) by iterating over prompt string as characters
Root cause: Validation Gap

**Description:**
When using Multimodal metrics (e.g., `MultimodalFaithfulnessMetric`), the internal logic sometimes passes a single string prompt to the model's `generate` or `a_generate` method.

In `MultimodalOpenAIModel.generate_prompt`, the code iterates over `multimodal_input` without checking if it is a string first.
* **Expected Behavior:** If the input is a single string, it should be treated as one text block.
* **Actual Behavior:** 
The loop `for ele in multimodal_input:` iterates over the string **character-by-character**.
    * A 16,000-character prompt becomes a list of 16,000 individual text objects `[{"type": "text", "text": "P"}, {"type": "text", "text": "l"}...]`.
    * This exceeds the OpenAI API limit for the `messages` array, causing an `array_above_max_length` error.

**Error Traceback:**

```text
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'messages[0].content': array too long. Expected an array with maximum length 16384, but got an array with length 16681 instead.", 'type': 'invalid_request_error', 'param': 'messages[0].content', 'code': 'array_above_max_length'}}
```

**To Reproduce**

Steps to reproduce the behavior:
1. Use MultimodalFaithfulnessMetric with a MultimodalOpenAIModel.
2. Run an evaluation where the generated prompt exceeds ~16k characters (common with large context retrieval).
3. The metric calls a_generate(prompt) where prompt is a string.
4. generate_prompt splits this string into thousands of items.

**Proposed Fix**

Add a type check at the start of generate_prompt to wrap single strings in a list.
```
def generate_prompt(
        self, multimodal_input: Union[str, List[Union[str, MLLMImage]]]
    ):
        # FIX: Handle single string input
        if isinstance(multimodal_input, str):
            multimodal_input = [multimodal_input]

        prompt = []
        for ele in multimodal_input:
            # ... existing logic ...
```
**Environment**

deepeval version: 3.7.4

Model: gpt-4o / gpt-4.1

Can help with this @trevor-cai ?

@ssesha thank you for reporting this. I will take a look at your PR, thank you for your contribution!

@ssesha good news, this issue is already solved on the main branch. Please upgrade to DeepEval 3.7.5. 

After that you should be able to run this script without error:

```python
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics.faithfulness.faithfulness import FaithfulnessMetric
from deepeval.models.llms.openai_model import GPTModel

# Create the FaithfulnessMetric
metric = FaithfulnessMetric(model="gpt-4.1")

# Define the test case parameters
test_case = LLMTestCase(
    input="a" * 16000,
    actual_output="Sample model output",
    retrieval_context=["Some context information"],
    multimodal=True,
)

# Run the metric to check for errors
try:
    result = metric.measure(test_case)
    print("Evaluation result:", result)
except Exception as e:
    print(f"Error: {e}")

``` 

Feel free to mention me if you have any issues.

Enjoy!
Repository: confident-ai/deepeval
Issue: Can not find azure model deployment with AzureOpenAIModel since deepeval 3.7.5
Root cause: Interface Contract Mismatch

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**


We connect deepeval with Azure using `AzureOpenAIModel`.

```
model = AzureOpenAIModel(
    model_name=config.DEEPEVAL_MODEL,
    deployment_name=config.DEEPEVAL_MODEL,
    azure_openai_api_key=config.AZURE_API_KEY,
    openai_api_version=config.OPENAI_API_VERSION,
    azure_endpoint=config.AZURE_API_BASE,
    temperature=temperature,
)
```

We tried to upgrade deepeval to 3.7.5 and noticed the renamed paramenters `model_name` etc.
But unfortunately, the client could not connect to Azure due to a 404 error: 

`openai.NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}`

Hypothesis:

The _build_client method in deepeval/models/llms/azure_model.py passes base_url to the OpenAI AzureOpenAI client, but the OpenAI SDK requires azure_endpoint for Azure clients. This causes a 404 "Resource not found" error when using AzureOpenAIModel.


Current code (broken)

```python
# azure_model.py line 362-368
kw = dict(
    api_key=api_key,
    api_version=self.openai_api_version,
    base_url=self.base_url,  # ❌ Wrong parameter name
    azure_deployment=self.deployment_name,
    **self._client_kwargs(),
)
```

Expected code (fix)

```python
kw = dict(
    api_key=api_key,
    api_version=self.openai_api_version,
    azure_endpoint=self.base_url,  # ✅ Correct parameter name
    azure_deployment=self.deployment_name,
    **self._client_kwargs(),
)
```

**To Reproduce**

Use AzureOpenAIModel with 3.7.5 and openai>=2.9.0

**Expected behavior**

Deepeval connects to Azure successfully

**Screenshots**
%

**Additional context**
openai==2.9.0
python 3.14
macOS


Got the same issue. We did not expect breaking changes in a patch version increment of deepeval.

Thank you @flansen for reporting this. I will be investigating the issue and will keep you posted.

@flansen, thanks again for reporting this. A fix is in the linked PR. Here is the commit that addresses your issue:

https://github.com/confident-ai/deepeval/pull/2371/commits/3dc4c56a175cdc0eaade6c347d1203d8821981cf

All the best!

Thank you for resolving! :)
Repository: confident-ai/deepeval
Issue: Upgrading deepeval from 3.7.3 to 3.7.4 breaks Gemini judge model.
Root cause: Interface Contract Mismatch

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
Upgrading from 3.7.3 to 3.7.4 breaks creating GeminiModel.

**To Reproduce**
1. Set env variable GOOGLE_GENAI_USE_VERTEXAI to "YES"
2. Execute code:
`from deepeval.models import GeminiModel

GeminiModel(
    model_name='gemini-2.5-pro',
    project='<project>',
    location='<location>'
)`

**Expected behavior**
GeminiModel object is created.

**Actual behavior**
  File "...\Lib\site-packages\deepeval\models\llms\gemini_model.py", line 134, in should_use_vertexai
    return self.use_vertexai.lower() == "yes"
           ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'bool' object has no attribute 'lower'


https://github.com/confident-ai/deepeval/pull/2355 fixes the problem.

@Sliski Thank you for reporting this. A fix is on its way.