zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Add n1n.ai API as a model provider
Root cause: Unimplemented Feature Gap

Hi DeepEval team,

We're n1n API, and we offer a robust API that provides access to 400+ large language models (LLMs), as well as multimodal generation capabilities (text, images, video, audio, etc.). Our platform makes it easy to integrate these advanced features into your project, empowering developers with powerful AI tools.

With n1n API, you can quickly leverage a comprehensive suite of AI models for diverse applications—everything from text generation to media synthesis—through simple, easy-to-use endpoints.

Adding n1n API would benefit DeepEval users by:

- Providing access to 400+ models through a single API key configuration
- Supporting all major models (OpenAI, Anthropic, Gemini, Llama, and many more)
- Offering competitive pricing with unified billing (some are lowest to 1/10 of the official price)
- Enabling easy model switching for different assistants without managing multiple API keys

### Describe the solution you'd like

Integrate n1n API as an additional AI model provider in DeepEval. n1n API is OpenAI-compatible (base URL: https://n1n.ai/v1/) and offers extended multimodal generation capabilities beyond standard text completion, including image, video, and audio generation endpoints.

### Resources

Website: https://n1n.ai/
API Documentation: https://docs.n1n.ai/
Dashboard: https://n1n.ai/console
Support: [help@n1n.ai](mailto:help@n1n.ai)

We are excited to contribute to the DeepEval ecosystem and provide users with access to the latest AI models through our robust API platform. We look forward to collaborating with the DeepEval maintainers and community to make this integration successful.

@n1n-api please come back when n1n gets bigger, thank you
Repository: confident-ai/deepeval
Issue: How to obtain the retrieval_context field when evaluating a Dify agent against the ContextualRelevancy metric?
Root cause: Documentation Deficiency

Hello, I encountered an issue while using deepeval to evaluate a Dify agent and hope to get some help. When evaluating the Dify agent against the ContextualRelevancy metric, I called Dify's /v1/chat-messages endpoint. However, I found that the retriever_resources field in the metadata returns empty, making it impossible to obtain the retrieval_context from it. This fails to meet the parameter requirements for ContextualRelevancy evaluation. I would like to ask if there is any way to obtain the retrieval_context for conducting the ContextualRelevancy metric evaluation, or if there are other alternative solutions to handle this situation.

Hello, has the issue been resolved? I would like to inquire about the specific steps for evaluating Dify agent using deepeval. If it's convenient for you, could we connect on WeChat? Thank you (dihn1989)

Hey @eddiedi and @xuzhong9752, if you have access to some list of strings on Dify then you can use that for retrieval context. Personally i'm not so familiar with Dify, but if you send me some code examples i'm sure i can provide more pointers
Repository: confident-ai/deepeval
Issue: Support evaluation for voice data
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently, the evaluation pipeline in Deepeval does not support voice. This limits the framework's applicability for projects needing automated evaluation for voice, speech, or audio-based models.

**Describe the solution you'd like**
Add support for evaluating voice data within Deepeval. This could include metrics and tools designed for speech and audio evaluation, integrating with existing voice datasets, or allowing users to run quality and accuracy checks on voice model outputs.

**Describe alternatives you've considered**
- Using external tools for voice evaluation and then manually integrating results with Deepeval.
- Developing a custom evaluation script outside of Deepeval for voice data.

**Additional context**
This feature would expand Deepeval’s usability to voice AI and speech processing projects, making it more versatile for a wider range of AI applications.

Hey @YassinNouh21 yes that is something we're very interested in. Could you please let us know what is your tech stack so we can know what to watch out for when building out voice evals?
Repository: confident-ai/deepeval
Issue: ZeroDivisionError in MCPTaskCompletionMetric
Root cause: Validation Gap

**Describe the bug**

ZeroDivisionError ocurred in MCPTaskCompletionMetric in version 3.6.7

File "C:\Users\...\deepeval\metrics\mcp\mcp_task_completion.py", line 232, in _calculate_score
    return total_score / len(scores)
           ~~~~~~~~~~~~^~~~~~~~~~~~~
ZeroDivisionError: division by zero

**Expected behavior**

No error occurs

**Additional context**

I think this issue have the same cause in #2138 


Hey @licux,
We fixed this, should be out in the next release. Thank you for bringing this up!

Hello @A-Vamshi 

Thank you for the information! 
When is the next release scheduled?
Repository: confident-ai/deepeval
Issue: AsyncConfig import problem
Root cause: Architectural Constraint

**Describe the bug**
Import errors with `AsyncConfig` due to unfamiliar module structure.

**To Reproduce**

This works:

```python
>>> from deepeval.evaluate.configs import AsyncConfig
>>> AsyncConfig()
AsyncConfig(run_async=True, throttle_value=0, max_concurrent=20)
```

This errors:

```python
>>> import deepeval.evaluate.configs
>>> deepeval.evaluate.configs.AsyncConfig()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute 'configs'
```

**Expected behavior**

```python
>>> import deepeval.evaluate.configs
>>> deepeval.evaluate.configs.AsyncConfig()
AsyncConfig(run_async=True, throttle_value=0, max_concurrent=20)
```

**Desktop (please complete the following information):**
 - OS: Windows 10
 - Deepeval 3.4.9



Hey @JoeMountWSWS could you please make a quick PR for this? Thank you in advance!

> Hey [@JoeMountWSWS](https://github.com/JoeMountWSWS) could you please make a quick PR for this? Thank you in advance!

Here's a PR: https://github.com/confident-ai/deepeval/pull/2245

Hey @JoeMountWSWS, is this issue fixed?


> Hey [@JoeMountWSWS](https://github.com/JoeMountWSWS), is this issue fixed?

No. Using DeepEval version 3.7:

```python
>>> import deepeval.evaluate.configs
>>> deepeval.evaluate.configs.AsyncConfig()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute 'configs'
```

I've proposed a fix in this pull request: https://github.com/confident-ai/deepeval/pull/2245
Repository: confident-ai/deepeval
Issue: Unclosed client session using AmazonBedrockModel with aiobotocore
Root cause: Resource Mishandling

**Describe the bug**
When using AmazonBedrockModel as the model for llm-as-a-judge, warning ocurred after the evaluation.
```
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7184f61a7410>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7184f61f9970>, 1780.688582909)])']
connector: <aiohttp.connector.TCPConnector object at 0x7184f61a74a0>
```

**To Reproduce**
Using AmazonBedrockModel  for llm-as-a-judge metrics
```
deepeval_model = AmazonBedrockModel(
    model_id="us.anthropic.claude-3-7-sonnet-20250219-v1:0", region_name="us-west-2"
)
metric = MCPTaskCompletionMetric(model=deepeval_model)
```

**Expected behavior**
Any warning occurs.

*Using Version**
|Package|Version|
|-|-|
|aiobotocore| 2.25.0|
|aiohttp     |3.13.1|
|botocore | 1.40.49|
|deepeval|3.6.7|

**Additional context**
I think it is necessary to add `await self.close()` at the end of the a_generate() method in AmazonBedrockModel or evaluate(). 

Or should this be called from the evaluation script instead of from within the deepeval library?

Seems like a similar issue was talked about on stackoverflow: https://stackoverflow.com/questions/54807599/aiohttp-unclosed-client-session-client-session

@trevor-cai or @A-Vamshi for visbility...

If @licux you could make a quick PR on this it would be very helpful, since we might not have the bedrock models that are required to reproduce, thank you in advance

@penguine-ip 
I created PR ; https://github.com/confident-ai/deepeval/pull/2250.
\* I just don't know if this is the best solution. But I can solve the problem. 

Hey @licux thank you, the PR looks like there's no breaking changes and closes in a safe way. If it fixes for you then i'm sure it is good enough for now. It will be included in the next release!

Hi @penguine-ip,
Thank you!
I'm glad I can contribute to this project.

@licux thank you! its released :)
Repository: confident-ai/deepeval
Issue: Add hooks for ConversationSimulator
Root cause: Unimplemented Feature Gap

For development, we would like to eg. configure MCP servers according to the golden's scenario and expected outcome.
We also need to do a bit of tear-down once the simulation is done.

It would be great if we could have the following hooks:
- A new conversational golden is being processed. We could have a `start_simulation_for_golden(golden, thread_id)` hook being called.
- The conversation is deemed finished (either the expected outcome or max number of turns has been reached. Something like `end_simulation_for_golden(golden, thread_id, end_reason)` would be nice to do a bit of tear down work.

Currently, we're able to workaround the "start processing a new golden" by poking around in the callers' stack, but we would appreciate something more streamlined.
We unfortunately have not found a way of being notified once the conversation for a golden has stopped.


This is interesting and I'd love to understand a bit more what we're trying to achieve? 

We are in the process of thinking more about lifecycles within DeepEval's functionalities as well so we're happy to support once we get clarification.

We're trying to evaluate different LLM framework/models, so we've put together some testing harness which is basically just using/configuring deepeval with some additional utility.

Our test harness also comprises of an MCP tool server, with tools being implemented as LLM calls so we can control the output of the tool with some input scenario.
When we start processing a new golden, we feed the scenario and expected outcome (and other data) to the MCP server. This is where a first hook would help us.

Similarly, once the scenario is done, we would like to do some resource cleaning on our MCP server.

Hey @audricschiltknecht thanks for your information, I think its a great use case and I'm optimistic we can get it supported by the end of week :) 

@audricschiltknecht its released, with docs here: https://deepeval.com/docs/conversation-simulator#lifecycle-hooks

Please let me know if this solves your problem

@penguine-ip thanks that will certainly help!

An evolution that might be interesting is to also have a status code for knowing if the simulation was completed because the expected outcome was reached, or because we run out of simulation turns (based on `max_user_simulations`).

Had more time to play with it. I think instead of the index, the associated thread-id should be used in the hook. Or the index should be passed to the model_callback.
That would make identifying the current golden more consistent in the code.
Repository: confident-ai/deepeval
Issue: Deepeval (3.6.7) depends on pytest-rerunfailures (>=12.0,<13.0)
Root cause: External Dependency Breakage

**Describe the bug**
It's basically a dependency mismatch. I've used it with the latest rerun version with no issues.

**To Reproduce**
Steps to reproduce the behavior:
1. Add `pytest-rerunfailures = "16.0.1"` to project
2. Add `deepeval = "^3.6.7"` to project
3. run `poetry lock`

`Because deepeval (3.6.7) depends on pytest-rerunfailures (>=12.0,<13.0)`

@luiz-cohere so you're saying removing our restriction still works?

There were no breaking changes from 12 until now
Besides dropping support for old versions of python (most of which you don't support anyway) it's only bug fixes and a few new features.
I don't see any reason for you to allow any version of `pytest` but force a 3 year old version of `pytest-rerunfailures`

I opened a PR 
https://github.com/confident-ai/deepeval/pull/2304
I would really appreciate if you could allow the CI to run
I'm down to assist with fixes if any will be required 🙏
Repository: confident-ai/deepeval
Issue: ArgumentCorrectnessMetric calculate BUG
Root cause: Algorithmic Error

ArgumentCorrectnessMetric

The Argument Correctness metric has a logical error. When tools_called is empty, the score should be 0.0 instead of 1.0. This needs to be addressed.

<img width="822" height="785" alt="Image" src="https://github.com/user-attachments/assets/6e1e797d-dc5e-466f-a2a2-7656368726c0" />

Hey @lxhappy110 is there a reason why we want the score to be 0? If there are no tools called, there is no room for error, and hence the score is perfect. LMK what you think
Repository: confident-ai/deepeval
Issue: Unable to select checkboxes on “Tell me about your use case” page
GitHub labels: awaiting clarification


**Describe the bug**
On the “Tell me about your use case” page, under the second question — “Do you already have a dataset for testing?” — neither of the checkboxes can be selected.
Because of this, I’m unable to proceed to the next page and complete the onboarding process.

**To Reproduce**
Steps to reproduce the behavior:
1. Enter your mail id, password and click create account
2. Select Personal /Business and what defines you the best and navigate to the next page
3. One can select check box of the first question but not the second


**Expected behavior**
I should be able to select the checkbox and move to the next page

**Screenshots**
If applicable, add screenshots to help explain your problem.

<img width="866" height="730" alt="Image" src="https://github.com/user-attachments/assets/f05e43b4-141b-44bc-8732-21c8618beee5" />

**Desktop (please complete the following information):**
 - OS: [MacOS - Tahoe26]
 - Browser [chrome]
 - Version [141.0.7390.123]

**Smartphone (please complete the following information):** - No
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**
Cannot complete onboarding.


+1

Hey @avi350751 and @zzhujing it was just fixed, please lmk if you are able to proceed (refresh your screen for new changes)

Now I could select the checkbox and complete onboarding. Thanks for the quick response.
Repository: confident-ai/deepeval
Issue: Faithfullness metric returns `1.0` even if `retrieval_context` is empty
Root cause: Validation Gap

**Describe the bug**
Faithfullness metric returns `1.0` even if `retrieval_context` is empty

**To Reproduce**
Steps to reproduce the behavior:
```py
metric = FaithfulnessMetric(model="...")
test_case = LLMTestCase(
        input="xxx",
        expected_output="xxx",
        actual_output="xxx",
        retrieval_context=[""],  # this is a list of empty string
)
metric.measure(test_case)
```
I believe the required parameters check is being bypassed [here](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/utils.py#L245) and this might apply for any required param of any metric using the same check function. I only tested with above code snippet as of now.

**Expected behavior**
I expect an error or a `0.0` score.


Hey @omar-araboghli right now if no retrieval context is provided then by definition nothing is contradicting and therefore faithfulness is 1 instead of 0. To get the error behavior, you'll need to pass in None instead of an empty list. Hope this helps!
Repository: confident-ai/deepeval
Issue: Print warning on deprecated settings

As mentioned in the comment below, some deprecation warnings would be informative for users whether they are up to date with current state of the library :)


@trevor-cai wdyt about logging some warning when user uses deprecated env var? It seem like a good idea to give user a hint that this is deprecated and will be removed in a future major releases

_Originally posted by @agrudkow in https://github.com/confident-ai/deepeval/issues/2249#issuecomment-3455123839_
            

@agrudkow thank you!

Hi @agrudkow,

That is a great suggestion. PR: https://github.com/confident-ai/deepeval/pull/2266 is adding backwards compatibility for   `DEEPEVAL_TELEMETRY_ENABLED`, see #2231. There are also a number of settings that have been converted to computed fields, these all have overrides, but I will add a warning if someone tries to set one of the computed fields that will include instruction on which one they should set instead. That PR will be in review later in my day.

Closing as this issue was resolved by https://github.com/confident-ai/deepeval/pull/2266.
Repository: confident-ai/deepeval
Issue: DROP eval: Batch evaluation: Error due to an unexpected argument
Root cause: Validation Gap

**Describe the bug**
Error due to unexpected argument.
```
  File ".../lib/python3.11/site-packages/deepeval/benchmarks/drop/drop.py", line 215, in batch_predict
    prompt: dict = DROPTemplate.generate_output(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: DROPTemplate.generate_output() got an unexpected keyword argument 'type'
```

**To Reproduce**
Evaluate using DROP (using batch evaluation)
```python
from deepeval.benchmarks import DROP
from deepeval.benchmarks.drop.task import DROPTask
drop_benchmark = DROP()
drop_benchmark.evaluate(model=CustomModel(), batch_size=3)
```

**Expected behavior**
Evaluation should be successful.

**Screenshots**
N/A

**Additional context**
The problem got solved locally when I removed the mentioned line. The same argument has been removed for regular eval but not removed for batch eval.
https://github.com/confident-ai/deepeval/blob/main/deepeval/benchmarks/drop/drop.py#L218
Repository: confident-ai/deepeval
Issue: Feature request: Support creating `MLLMImage` without url
Root cause: Unimplemented Feature Gap

**Problem Statement**
I'm looking to run faithfulness evaluations where my retrieval context includes both textual results and images (as base64 strings). My retrieval system is remote and so we only get back the base64 for an image, and it's currently impractical for us to store retrieved images in their native formats.
I would ideally like to be able to optionally create an `MLLMImage` by passing in the base64 data and mimetype, rather than the URL.

**Solution**
I would like for the `MLLMImage` to support creation through either a `url` (current state), or through passing in the base64 data and mimetype yourself (desired state).
Repository: confident-ai/deepeval
Issue: DeepEval imports cause filesystem interaction on app startup
Root cause: Architectural Constraint

**Describe the bug**

When importing some DeepEval packages in a read-only environment (such as a lambda or container), the import path contains an attempt to access the filesystem which will cause an application to crash. 

It appears there has been some prior work to ameliorate the filesystem interactions in a few threads, such as the introduction of `DEEPEVAL_TELEMETRY_OPT_OUT` and `DEEPEVAL_FILE_SYSTEM: READ_ONLY` (though the latter example seems to be unimplemented) but these don't fix the specific import path problem.

The error surfaces from `portalocker` `portalocker/utils.py` where it attempts to set the default value on the init for `BoundedSemaphor`: 

```python
        directory: str = tempfile.gettempdir(),
```

If the application environment has no access to any filesystem, you'll receive something like the following error:

```
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/app']
```

This can be delayed but not prevented by lazily importing the packages, though it will crash once the packages are imported.


**To Reproduce**

In a read-only environment, you can do this in the shell, the only thing you need to import is `LLMTestCase`:

```bash
>>> from deepeval.test_case import LLMTestCase
  File "/usr/lib64/python3.12/tempfile.py", line 223, in _get_default_tempdir
    raise FileNotFoundError(_errno.ENOENT,
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/app']
```

**Expected behavior**

We would expect that the filesystem is not accessed during the initialization of the application and that any features that default to use the filesystem could be opted out of through either environment options (such as the `DEEPEVAL_FILE_SYSTEM` listed above) or with specific arguments (e.g. `use_cache=False`).


full reproduction:

```
# Dockerfile
FROM python:3.14.0

RUN pip install deepeval==3.7.0

# Bash
docker build -t deepeval-import-issue .


docker run --rm --read-only deepeval-import-issue python -c "from deepeval.test_case import LLMTestCase"
Traceback (most recent call last):
...
    raise FileNotFoundError(_errno.ENOENT,
                            "No usable temporary directory found in %s" %
                            dirlist)
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']

❯ docker run --rm --read-only -e DEEPEVAL_TELEMETRY_OPT_OUT=1 deepeval-import-issue python -c "from deepeval.test_case import LLMTestCase"
Traceback (most recent call last):
...
  File "/usr/local/lib/python3.14/tempfile.py", line 224, in _get_default_tempdir
    raise FileNotFoundError(_errno.ENOENT,
                            "No usable temporary directory found in %s" %
                            dirlist)
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']
```

@andrewklatzke thank you for the excellent bug report! A PR is on its way to address this issue.
Repository: confident-ai/deepeval
Issue: AnswerRelevancyMetric on OllamaModel shows incorrect Reason: The score is <answer_relevancy_score> because <your_reason>
Root cause: External Dependency Breakage

AnswerRelevancyMetric.reason comes as The score is <answer_relevancy_score> because <your_reason> and score < .66666  [Incorrect]

while 
openAI model shows score as 1 and reason "The score is 1.00 because the answer was completely relevant and directly addressed the question with no irrelevant information. Great job!" [Correct]

ollama:
`from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval.models import OllamaModel


model = OllamaModel(
    model="qwen2-math:1.5b",
    base_url="http://localhost:11434",  # optional if default
    temperature=0
)

answer_relevancy_ollama = AnswerRelevancyMetric(model=model)

test_case = LLMTestCase(
    input="the ones digit of 16 * 27 * 38 is",
    actual_output="6"
)

answer_relevancy_ollama.measure(test_case)

print("Score:", answer_relevancy_ollama.score)
print("Reason:", answer_relevancy_ollama.reason)`

Openai:
`
import os
from dotenv import load_dotenv
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
load_dotenv()

print("OpenAI key found:", bool(os.getenv("OPENAI_API_KEY")))

answer_relevancy = AnswerRelevancyMetric()


test_case = LLMTestCase(
    input="the ones digit of 16 * 27 * 38 is",
    actual_output="6"
)

answer_relevancy.measure(test_case)


print("Score:", answer_relevancy.score)
print("Reason:", answer_relevancy.reason)
`

Hey @simpleappdesigner smaller models on ollama don't do so well for evaluation unfortunately
Repository: confident-ai/deepeval
Issue: vulnerability in deepeval Project
Root cause: External Dependency Breakage

While working on deepeval Project, I discovered that versions prior to 0.49.1 are vulnerable to a Denial of Service (DoS) attack caused by inefficient Range header processing in the FileResponse class. Attackers can craft malicious Range headers that trigger excessive CPU consumption, potentially making the application unresponsive.

[CVE Report](https://vulert.com/vuln-scan/list/0f2795f3-3437-47df-943d-a58f874da4f1?sort_order=desc&sort_by=created_at)
[CVE Link](https://staging.vulert.com/vuln-db/CVE-2025-62727)

Hey @ankitdn 0.49.1 is not something we support anymore, the latest version is `3.7.0` and I'd highly recommend you upgrade instead.