zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: AnswerRelevancyMetric generates inaccurate scores - failing a test that should pass
Root cause: Algorithmic Error

**Describe the bug**
AnswerRelevancyMetric scores are very low (0.001 - 0.28) when they should be closer to 0.7-0.8 for the (question, answer) tuple being tested

**To Reproduce**
Run the code below:
```
    post = "Summarize the access review status and tell me what should I do next?"
    response = """
The access review is at the initial stage with no clusters reviewed yet. Currently, the risky entitlements cluster (Cluster ID: -1) with 2 normal entitlements is pending review. There are 3 less-risky clusters (Cluster IDs: 0, 1, 2) with varying numbers of normal and anomalous entitlements. Start with the largest cluster and review all the normal entitlements first, then carefully review the anomalous ones. After that, you should proceed with the less-risky clusters.
"""
    # evaluate the response for relevancy
    relevancy_metric = AnswerRelevancyMetric(minimum_score=0.9)
    test_case = LLMTestCase(query=post, output=response)
    assert_test(test_case, [relevancy_metric])
```

**Expected behavior**
Test should pass

Thanks for this! For this situation, we use cross-encoders. I think the model under-performs when the answers and questions are asymmetric in length. I'll run some tests and update this to provide a new answer relevancy alternative - thanks for flagging! 

For the summarization use case, I'm thinking of adding a "SummarizationAccuracy" metric that is more closely aligned with the SummaC implementation - https://github.com/tingofurro/summac

I'll keep this issue alive until the PR is made
Repository: confident-ai/deepeval
Issue: Docs not reachable

Unable to reach documentation listed on github:
https://deepeval.com/docs

Hey @uahmad235 where is the link you found? It should be /docs/getting-started instead

<img width="2224" height="1272" alt="Image" src="https://github.com/user-attachments/assets/461e86a4-e0db-408c-b057-177257dd4374" />
Doesn't work either. Are you guys geographically blocking ?

@uahmad235 not that I know of, are you behind some sort of firewall or behind company network?

Not really. I have tried different browsers. I can access confident-AI website though. You guys must be hosting it on different server?

Update: Works when i use VPN. It's most likely a geographical issue.

Hi @uahmad235,
Can you please close this issue, as this is resolved now ?

Are you suggesting using VPN for accessing your docs is a "solution", then please go ahead and mark it as close.

Hi @uahmad235 - 
I am from open-source community. I was going through the list of open issues, where I saw you got the resolution to your issue. Hence requested you to mark it close. 

You may re-open this issue. Apologies for the misunderstanding.
Repository: confident-ai/deepeval
Issue: Option to overwrite GEval default templete.
Root cause: Unimplemented Feature Gap

Metrics like [FaithfulnessMetric](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/faithfulness/faithfulness.py) or [AnswerRelevancyMetric](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/answer_relevancy/answer_relevancy.py) have a field `evaluation_template`.  On the other hand, metrics like [GEval](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/g_eval/g_eval.py) or [ConversationRelevancyMetric](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/conversation_relevancy/conversation_relevancy.py) lacks such field.
Is there an option to add such field (with default value set on `GEvalTemplate` for GEval and so on) to those metrics which are missing it? It would be really helpful for evaluating other languages than english.


@Imrauviel did you figure out anything on this?

@penguine-ip bumping this up, i also realized the importance of this in my usecase
Repository: confident-ai/deepeval
Issue: __init__.py issues when using behind vpn/firewall/corporate firewall
Root cause: Validation Gap

**Describe the bug**
When using the library with corporate firewall/vpn solutions (like fortinet) that do self-signed certificate-based traffic inspection, the `requests.get` request to pypi fails due to the self-signed certificate. (Most teams in these circumstances use a `pip.conf` file which solves this issue directly).

Additionally, this error would occur if you were using the library in a project where you don't actively have network connectivity (or in some CI environments that don't have access to the open internet). 


**To Reproduce**
Steps to reproduce the behavior:
1. use library behind corporate firewall or without internet access

**Expected behavior**
The library shouldn't break everything

Note: I have a PR linked to this issue
Repository: confident-ai/deepeval
Issue: Specifying gpt-4o checkpoint causes crash because pricing info is not found
Root cause: Validation Gap

**Describe the bug**
When specifying, say, `gpt-4o-2024-11-20` instead of just `gpt-4o` as a judge model for the metrics I run, the library crashes because while calculating the cost it tries to find the model pricing in a custom-made dictionary inside `nodels/llms/openai_model.py` but it is not there.

**To Reproduce**
Steps to reproduce the behavior:
1. Have some LLM test cases or EvaluationDataset
2. Choose a metric and specify `model="gpt-4o-2024-11-20"`
3. `evaluate()`

**Expected behavior**
Should not crash

**Screenshots**
```
... [some i guess irrelevant stack trace above] ...
...
...
File "/Users/[USER]/.pyenv/versions/chat-api-deepeval/lib/python3.12/site-packages/deepeval/models/llms/openai_model.py", line 266, in a_generate
    cost = self.calculate_cost(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/[USER]/.pyenv/versions/chat-api-deepeval/lib/python3.12/site-packages/deepeval/models/llms/openai_model.py", line 383, in calculate_cost
    input_cost = input_tokens * pricing["input"]
                                ~~~~~~~^^^^^^^^^
KeyError: 'input'
```

**Desktop (please complete the following information):**
macOS 15.5


**Additional context**
Related to #1715 and #1695
Repository: confident-ai/deepeval
Issue: Deepeval not evaluating properly for large retrived context.
Root cause: Validation Gap

Hi, I'm creating a custom metric using Geval in deepeval and the retrival context is too large for some test cases(33k tokens), and I'm using anthropic model claude-3.1-sonnet-v1 for evaluation, but for the rows where the retrived tokens are large, I'm getting something like this in the evaluation reason.

"""
The evaluation steps do not provide any relevant criteria for assessing the response. The steps appear to be a fragmented message stating "You are an AI assistant tasked with evaluating the LLM response." followed by a definition of . They do not outline any specific evaluation criteria or steps to follow. Without clear evaluation guidelines, a meaningful assessment of cannot be made based on these steps.
"""
and sometimes it states that
"""
without proper actual_output, retrival context, evaluations cannot be made for the input.
"""

But its working fine for other rows what is actually happening, Am I missing something? why the model is not able to evaluate when there is a large context?

Can someone please help me.

@gunasekarperiyasamy-wk heycan you send over a test case to reproduce?

You're running into a classic RAG retrieval span / context window collapse (ProblemMap No.9) combined with eval tool drift (No.12). When the retrieval context exceeds typical model window size (e.g., >32k tokens), most eval pipelines become unreliable: the evaluation steps either fragment, hallucinate, or silently drop context, as you've observed.

We've catalogued these issues and common failure symptoms in our public Problem Map:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

If you want a quick diagnostic or patch suggestions (including context window fencing or eval pipeline correction), let me know. You're definitely not alone—this is one of the most destructive RAG / eval integration patterns, especially with custom metrics or large-context scenarios.
Repository: confident-ai/deepeval
Issue: Synethic input length validation needed during generate_goldens_from_docs
Root cause: Validation Gap

**Describe the bug**
When running the `generate_goldens_from_docs` in the Synthesizer, it is possible to encounter an IndexError on the `pbar_evolve_input_ids` list (line 651):
```python
evolved_input, evolutions_used = await self._a_evolve_input(
    input=data.input,
    context=context,
    num_evolutions=self.evolution_config.num_evolutions,
    evolutions=self.evolution_config.evolutions,
    progress=progress,
    pbar_evolve_input_id=pbar_evolve_input_ids[index],
    remove_pbar=False,
)
```

This occurs because the `pbar_evolve_input_ids` list is created to have length `max_goldens_per_context`, but the `qualified_synthetic_inputs`, which is responsible for setting the above `index`, is generated from an LLM and is not guaranteed to be limited by the `max_goldens_per_context`. This bug would only occur if the LLM fails to respect the maximum.

A simple fix would be to set the maximum in code (line 628):
```python
synthetic_inputs: List[SyntheticData] = await self._a_generate_inputs(
    prompt
)
# Limit the length of the synthetic inputs to the maximum allowed
synthetic_inputs = synthetic_inputs[:max_goldens_per_context]
```

With this change, I was able to get a successful run.

**To Reproduce**
Steps to reproduce the behavior:
1. Create a dataset using `Synthesizer.generate_goldens_from_docs`
2. Get lucky that the LLM exceeds the `max_goldens_per_context` maximum

**Expected behavior**
The dataset should generate without error.

**Environment**
 - OS: Linux
 - Python: 3.12.11
 - DeepEval: 3.2.6



@noah-gil hey could you please make a PR on this? Would appreciate it loads, thank you in advance!

Just noticed your comment. I made the PR.
Repository: confident-ai/deepeval
Issue: 🐛 Issue: Wrong schema class is being used during verdict generation
Root cause: Interface Contract Mismatch

**Describe the bug**
During the execution of the summarization metric evaluation, it was observed that the wrong schema class (faithfulness.schema.Verdicts) is being used instead of the expected one (summarization.schema.Verdicts). This leads to confusion and potential incompatibility when generating alignment verdicts.

**Version**
Name: deepeval
Version: 3.2.6
<img width="954" height="539" alt="Image" src="https://github.com/user-attachments/assets/787aaf16-531e-41b8-adde-dbeb06967920" />
Here you may see "Verdicts" is "class 'deepeval.metrics.faithfulness.schema.Verdicts'", but it should be "class 'deepeval.metrics. summarization.schema.Verdicts'" logically, because you use it in the class "SummarizationMetric" which is in the package "summarization"

**Advice**
You'd better use different alias for different class "Verdicts", when you import it.

I see it should be `SummarizationAlignmentVerdict`?

It should be SummarizationAlignmentVerdict, but now it is FaithfulnessVerdict if you check it.

Thank you!
Repository: confident-ai/deepeval
Issue: Wrong Rewritten Input in Few-Shot Example of Concretizing Evolution Template
Root cause: Validation Gap

https://github.com/confident-ai/deepeval/blob/aea37a98c2cfc73d320454227ee0e2befcc4b22b/deepeval/synthesizer/templates/template.py#L571-L573

seems like there was a mix up, the example rewritten input refers to biodiversity (the first few shot example given before), not to solar power generation. This sends a completely wrong and confusing signal to the LLM

@simon376 Can you please make a PR on this? Appreciate it!

> [@simon376](https://github.com/simon376) Can you please make a PR on this? Appreciate it!

there you go: https://github.com/confident-ai/deepeval/pull/1800


@simon376 thanks!
Repository: confident-ai/deepeval
Issue: Add LLM Evaluation for Text Summarization and RAG
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

Athina AI has open-sourced their approach towards text summarization. 
We need to implement this in some sort of summarization criteria. 
[LLM Evaluation for text summarization and RAG](https://github.com/athina-ai/ariadne)

**Describe the solution you'd like**

TBD on Python abstraction - need to better understand this.

**Describe alternatives you've considered**


**Additional context**
Add any other context or screenshots about the feature request here.


@ColabDog, we can add [RAGAS](https://github.com/explodinggradients/ragas) library here. Happy to take up this issue.

@Anindyadeep would love a text_summarization_metric in the metrics folder

Awesome, you can assign this to me.

@Anindyadeep Thanks! Ping me on discord if you need help setting up the development env, still working on the documentation for that part

Awesome, sure.
Repository: confident-ai/deepeval
Issue: Issue in api.ipify.org : Blocked from origination level, Is there any alternative way to run this repo.
Root cause: Documentation Deficiency

**Description:**
I am currently facing an issue with accessing the API at api.ipify.org due to restrictions imposed by my organization. As a result, I am unable to retrieve the public IP address, which is essential for my project that evaluates chatbot responses using Deepval.

**Details:**

- **Current Situation:** The API call to api.ipify.org is blocked at the origination level, preventing me form obtaining the necessary IP address for my application.

- **Assistance needed:** I want like to explore alternative methods or APIs that can provide similar functionality without being blocked by origination restrictions.

Your guidance on how to proceed would be greatly appreciated. Thank You.

Where are you calling api.ipify.org? I'm not aware of this problem atm so would like to learn more about it

just a guess, probably used in the telemetry here:
https://github.com/confident-ai/deepeval/blob/main/deepeval/telemetry.py#L50

You can disable telemetry by setting an environment variable:
DEEPEVAL_TELEMETRY_OPT_OUT: "YES"

> Where are you calling api.ipify.org? I'm not aware of this problem atm so would like to learn more about it

I noticed that the telemetry.py file in the deepeval module includes a request to api.ipify.org, but I'am unable to execute it successfully, because it is blocked from my org. 

Fortunately, I have found a repo that doen't include any telemetry code, which allowed me to continue working without issues, here: https://github.com/88roy88/deepeval-no-telemetry.git

> just a guess, probably used in the telemetry here: https://github.com/confident-ai/deepeval/blob/main/deepeval/telemetry.py#L50
> 
> You can disable telemetry by setting an environment variable: DEEPEVAL_TELEMETRY_OPT_OUT: "YES"

Okay, thanks. I was unable that there was a env. varibale for this. 
Thanks a lot. 

Closing the comments!
Repository: confident-ai/deepeval
Issue: Ollama as model provider
Root cause: External Dependency Breakage

Hi guys, I tried to use the comand "deepeval set-local-model --model-name=llama3:8b --base-url="http://localhost:11434/v1/" --api-key="ollama"

but I got the error:

File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\mateu\OneDrive\Desktop\papers\env\Scripts\deepeval.exe\__main__.py", line 4, in <module>
  File "C:\Users\mateu\OneDrive\Desktop\papers\env\Lib\site-packages\deepeval\__init__.py", line 9, in <module>
    from deepeval.evaluate import evaluate, assert_test, test_run
  File "C:\Users\mateu\OneDrive\Desktop\papers\env\Lib\site-packages\deepeval\evaluate\__init__.py", line 1, in <module>
    from .evaluate import evaluate, assert_test, dataset
  File "C:\Users\mateu\OneDrive\Desktop\papers\env\Lib\site-packages\deepeval\evaluate\evaluate.py", line 61, in <module>
    from deepeval.evaluate.execute import (
  File "C:\Users\mateu\OneDrive\Desktop\papers\env\Lib\site-packages\deepeval\evaluate\execute.py", line 86, in <module>
    from deepeval.openai.utils import TestCaseMetricPair, openai_test_case_pairs
  File "C:\Users\mateu\OneDrive\Desktop\papers\env\Lib\site-packages\deepeval\openai\__init__.py", line 5, in <module>
    from deepeval.openai.patch import patch_openai
  File "C:\Users\mateu\OneDrive\Desktop\papers\env\Lib\site-packages\deepeval\openai\patch.py", line 5, in <module>
    from deepeval.openai.utils import (
  File "C:\Users\mateu\OneDrive\Desktop\papers\env\Lib\site-packages\deepeval\openai\utils.py", line 6, in <module>
    from deepeval.openai.extractors import InputParameters, OutputParameters
  File "C:\Users\mateu\OneDrive\Desktop\papers\env\Lib\site-packages\deepeval\openai\extractors.py", line 3, in <module>
    from openai.types.responses import Response
ModuleNotFoundError: No module named 'openai.types.responses'

hey @mateusrobr what is your openai version?


I have Name: openai
Version: 1.52.2 installed currently

@mateusrobr Could you try updating to the latest version of OpenAI, which includes the Response API (in addition to chat completions)?

It worked, thank you guys
Repository: confident-ai/deepeval
Issue: DeepEval calls unsupported endpoint for LM Studio local LLM
Root cause: Interface Contract Mismatch

**Describe the bug**
When configuring DeepEval to use LM Studio as a local LLM as mentioned in the [docs](https://deepeval.com/integrations/models/lmstudio) (example command: deepeval set-local-model --model-name=mistral-nemo-instruct-2407 --base-url="http://localhost:1234/v1/"), DeepEval attempts to call the /v1/api/chat endpoint. LM Studio and it only supports the OpenAI-compatible API at /v1/chat/completions, and does not support /v1/api/chat.

**To Reproduce**
Steps to reproduce the behavior:
1. Run `deepeval set-local-model --model-name=mistral-nemo-instruct-2407 --base-url="http://localhost:1234/v1/"` using LM studio as the LLM provider
2. Run an evaluation
3. Notice that you get an error as DeepEval tries to call /v1/api/chat endpoint

**Expected behavior**
DeepEval should allow users to specify OpenAI-compatible endpoints for local LLMs (/v1/chat/completions)

**Additional context**
- [LM Studio API documentation](https://lmstudio.ai/docs/api/)
- [DeepEval LM Studio documentation](https://deepeval.com/integrations/models/lmstudio)


@Mohit-Gaur Would you be able to make a PR to support this? We are using OpenAI python sdk to run local models at this time, specifically the chat completions method:

https://github.com/confident-ai/deepeval/blob/5bd7aaf05cda0689c1720d318de8c44ce9fb1522/deepeval/models/llms/local_model.py#L50

Is this the same as `/v1/api/chat`? I would assume it should call `v1/chat/completions` by naming convention.
Repository: confident-ai/deepeval
Issue: Add GPT4FactualConsistency
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

GPT4 Factual Consistency is an important metric to measure because in some cases, you will want to use the best model possible for factual consistency.

**Describe the solution you'd like**

Similar to factual consistency, but then adding a GPT-baed one on top.
Can also add a guardrail to also output a metric of how confident it is. 

**Describe alternatives you've considered**
Current factual consistency model underperforms in a lot of scenarios.
Repository: confident-ai/deepeval
Issue: Circular dependency in "deepeval/benchmarks/__init__.py" (since 3.2.6)
Root cause: Architectural Constraint

**Describe the bug**

>   import deepeval.scorer as scorer
File "/lib/python3.12/site-packages/deepeval/scorer/__init__.py", line 1, in <module>
from .scorer import Scorer
File "/lib/python3.12/site-packages/deepeval/scorer/scorer.py", line 7, in <module>
from deepeval.benchmarks.schema import NumberSchema
File "/lib/python3.12/site-packages/deepeval/benchmarks/__init__.py", line 17, in <module>
from .ifeval.ifeval import IFEval                                                                                                                                                                                                                                           File "/home/bogong/py-venv/ai-eval/lib/python3.12/site-packages/deepeval/benchmarks/ifeval/ifeval.py", line 11, in <module>                                                                                                                                                     from deepeval.scorer import Scorer                                                                                                                                                                                                                                        ImportError: cannot import name 'Scorer' from partially initialized module 'deepeval.scorer' (most likely due to a circular import) 

**To Reproduce**
Steps to reproduce the behavior:
1. Add a custom metric using NLP scorer (e.g. bleu, rouge)
2. Run


**Desktop (please complete the following information):**
 - OS: ubuntu



Hey @github-zbg is it still happening? Would really appreciate it if you could make a PR on this, thank you in advance

@penguine-ip This is a hacky way to solve this. I can modify the PR if you provide any guidance.

@siesto1elemento hey not hacky at all, thank you!
Repository: confident-ai/deepeval
Issue: Deepeval & AWS Strands Agents version
Root cause: External Dependency Breakage

Hi team,

I was recently trying to incorporate deepeval for evaluating agent performance for an application that we have deployed on AWS.

We are using the latest version of ([Strands Agents](https://github.com/strands-agents/sdk-python) v1.0.1 as of this writing) as well as the latest version of ([Strands Agents Tools](https://github.com/strands-agents/tools/tree/main) v0.2.1 as of this writing).

The main problem however is that the current version of rich for deepeval is defined as: rich = ^13.6.0
Whereas Strands Agents Tools has the following requirements for rich: rich>=14.0.0,<15.0.0

If I wanted to use the latest version of deepeval it would either downgrade to Strands Agents Tools v0.0.1 (which is basically only boilerplate code and not usable) or downgrade deepeval to v0.10.7 which is also not an option.

Would it be possible to upgrade to version of rich based on these requirements?`

That would be highly appreciated!

Best regards

Hey @kuehno for sure, would you be able to make a quick PR on this in our pytoml file? Thank you in advnce

Hey @kuehno, we fixed it now 

Thanks a lot! Really appreciate that
Repository: confident-ai/deepeval
Issue: Deepeval dependencies need to be loosened or made optional, specifically opentelemetry and llm-provider deps.
Root cause: Architectural Constraint

**Describe the bug**

Hey all, I'm trying to install deepeval as a dev dependency in various projects and am running into dependency resolution errors. deepeval requires you to install fairly specific versions of libraries for optional features - can these dependencies be moved to extras?

opentelemetry is not a requirement to run the project and so it should be moved to an optional extras install.

Additionally, deepeval should not require you to install all well-known providers from the get-go (ollama, openai, anthropic, google).

**To Reproduce**
1. Install deepeval in any opinionated, settled project including lang chain or llama index or opentelemetry.
2. Run into dependency resolution issues

Example:
```shell
   4: fact: no versions of opentelemetry-sdk match >1.33.1,<1.34.0
   4: conflict: no versions of opentelemetry-sdk match >1.33.1,<1.34.0
   4: ! opentelemetry-sdk (>1.33.1,<1.34.0) is partially satisfied by not opentelemetry-sdk (==1.33.1)
   4: ! which is caused by "opentelemetry-sdk (1.33.1) depends on opentelemetry-api (1.33.1)"
   4: ! thus: opentelemetry-sdk (>=1.33.1,<1.34.0) requires opentelemetry-api (1.33.1)
```

**Expected behavior**

Installation should be friction-less by moving optional dependencies into extras.