zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Tenacity upper-bound stuck at pre-9.0
Root cause: Configuration Error

DeepEval’s dependency metadata restricts **tenacity** to `<=9.0.0`.  
Tenacity 9.0.0 shipped on **2024-07-29** and 9.1.x followed in April 2025, adding minor bug-fixes while keeping the public API stable.
With the current pin, any project that already upgraded to Tenacity 9.1.x (e.g. for asyncio bug-fixes) hits a resolver conflict when adding DeepEval.

### Steps to reproduce
```bash
pip install "tenacity>=9.1"  # or poetry add tenacity@^9.1
pip install deepeval         # → or poetry add deepeval --group dev
# → Requires-Dist conflict: deepeval -> tenacity<=9.0.0
Repository: confident-ai/deepeval
Issue: [tracing] serialize error when returned an object from a method decorated with observe might cause
Root cause: External Dependency Breakage

```python
@observe(type="llm", client=client)
def generate_response(input: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # or your preferred model
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": input},
        ],
        temperature=0.7,
    )

    # response = client.beta.chat.completions.parse(
    #     model="gpt-4o-mini",
    #     messages=[
    #         {"role": "system", "content": "You are a helpful assistant."},
    #         {"role": "user", "content": input}
    #     ],
    # )
    return response


try:
    response = generate_response("What is the weather in Tokyo?")
    print(response)
except Exception as e:
    raise e

time.sleep(6)
```

![Image](https://github.com/user-attachments/assets/f98132ab-89d0-4826-9d8f-8391a89668b9)

hint: the error occurs due to pydantic version mismatch

@spike-spiegel-21 is this the latest version?

Yes, tested with the latest main branch. still receiving the same. I think the problem is with the serialisation of the trace output. 

https://github.com/confident-ai/deepeval/blob/0d92fb0b33503eab5e7307c8ab42cf1dca4b7214/deepeval/tracing/tracing.py#L529
Repository: confident-ai/deepeval
Issue: The ability to set a filename for dataset files
Root cause: Unimplemented Feature Gap

In my setup, I have scripts for generating various databases with various parameters (model config and prompt versions). I want to save these files and then, in my evaluation scripts, just specify which model and which prompt version I want to run evaluations for. I want the model and prompt version information to be stored in the filename of the dataset. I find it strange that you can't set a dataset filename using the saveas() method.

@behnamsattar just added, relesasing in a few hours!
Repository: confident-ai/deepeval
Issue: Issue with Component-level eval
Root cause: Documentation Deficiency

This code:


```
model = GeminiModel(
    model_name="gemini-2.0-flash",
    project=GCP_PROJECT_ID,
    location=GCP_LOCATION,
    temperature=0
)

@observe( metrics=[ContextualRelevancyMetric(model=model)])
def rag(query):
    context = ["Dogs are animals"]
    test_case = LLMTestCase(input=query, actual_output="I don't know :(", retrieval_context=context)
    update_current_span(test_case=test_case)
    return context

if __name__ == "__main__":
    eval_results = evaluate(
        goldens= [Golden(input="What are cats?")],
        observed_callback= rag
    )
    print(eval_results)
```

got me that evaluation result (note that it didn't updated retrieval_context or actual_output). I think tracing isn't working well. Am i forgetting something?

![Image](https://github.com/user-attachments/assets/0a6d7afb-3581-429b-9c85-94578e48816f)

I think that it may be working because "success" field is compliant with the task of the metric but it is kind of a black box. I cannot tell what metric gave me that output or anything i may need to trace my app. What do you suggest for this? I'm missing something?

Hey @Bruno-Romero it worked but we haven't added support for seeing the results locally yet, if you run this while logged in (using deepeval login) you'll be able to see it on the cloud

You did update the `retrieval_context` and `actual_output` btw, it was done: 
```
    test_case = LLMTestCase(input=query, actual_output="I don't know :(", retrieval_context=context)
    update_current_span(test_case=test_case)
```

There's a new `deepeval view` command that allows you to view it on the web
Repository: confident-ai/deepeval
Issue: A basic tutorial on gpt-3.5 eval on MMLU
Root cause: Documentation Deficiency

I couldn't find a very basic tutorial: how can I evaluate a public model (say **gpt-3.5**) on MMLU benchmark? Is there a very basic tutorial for it?
I saw some tutorial for some custom Huggingface models (Mistral 7B) but not a popular public model like GPT models:

```
from transformers import pipeline
from deepeval.benchmarks import MMLU
from deepeval.benchmarks.tasks import MMLUTask

# Load the model using pipeline
pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.1")

# Function to generate responses
def generate_response(prompt):
    response = pipe(prompt, max_new_tokens=100, do_sample=True)
    return response[0]["generated_text"]

# Example test
messages = [{"role": "user", "content": "Who are you?"}]
response = generate_response(messages[0]["content"])
print("Model Response:", response)

# Define and evaluate MMLU benchmark
benchmark = MMLU(
    tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE, MMLUTask.ASTRONOMY],
    n_shots=5
)

# Evaluate using the pre-trained model
benchmark.evaluate(model=pipe, batch_size=5)

```
Repository: confident-ai/deepeval
Issue: Incompatible `anthropic` version range with `langchain-anthropic`
Root cause: External Dependency Breakage


Hi team,

I'm currently using `deepeval==3.0.8` in my project, and I also need to use [`[langchain-anthropic](https://pypi.org/project/langchain-anthropic/)`](https://pypi.org/project/langchain-anthropic/) to integrate with Anthropic's Claude models.


**Describe the bug**
However, I ran into a dependency conflict due to incompatible `anthropic` version ranges:

* `deepeval==3.0.8` requires:

  ```
  anthropic >=0.49.0,<0.50.0
  ```
* `langchain-anthropic==0.3.15` requires:

  ```
  anthropic >=0.52.0,<1.0.0
  ```

`deepeval==3.0.8` has a strict dependency on `anthropic >=0.49.0,<0.50.0`, which conflicts with `langchain-anthropic==0.3.15` that requires `anthropic >=0.52.0,<1.0.0`. This makes it impossible to use both libraries together in the same project.

**Expected behavior**
I expected to be able to install both `deepeval` and `langchain-anthropic` together, especially since both are commonly used for evaluation and LLM integration. Ideally, `deepeval` should allow a wider range of `anthropic` versions, or align with the latest SDK versions used in LangChain and other ecosystem tools.

**Screenshots**

<img width="1337" alt="Image" src="https://github.com/user-attachments/assets/5ec42673-1095-4808-87bf-0f505c287d9c" />

**Desktop (please complete the following information):**

* OS: macOS 15.3.1
* Browser: N/A
* Version: Poetry 2.1.2, Python 3.12.10

**Additional context**
`langchain-anthropic` depends on newer versions of `anthropic` to support Claude 3 models and current API changes. Would it be possible to loosen the `anthropic` version constraint in `deepeval` to at least `>=0.49.0,<1.0.0`, or better yet follow the upstream SDK compatibility?


Thanks for your awesome work on DeepEval.

@Yvette0828 should be fixed in 3.1.0, can you update and let me know if it works?

The issue is resolved now. Thank you for fixing it.
Repository: confident-ai/deepeval
Issue: `synthesizer.synthetic_goldens` not extended when using `generate_goldens_from_goldens`
Root cause: Algorithmic Error

Goldens generated using the `generate_goldens_from_goldens` method are not appended to `Synthesizer.synthetic_goldens`. This causes failures in functions such as `Synthesizer.to_pandas()`

### Steps to Reproduce

```python
from deepeval.dataset.golden import Golden
from deepeval.synthesizer import Synthesizer

goldens_to_evolve = [
    Golden(input="What is the capital of France?"),
    Golden(input="Who painted the Mona Lisa?"),
    Golden(input="What is the largest planet in our solar system?"),
    Golden(input="Who wrote 'Romeo and Juliet'?"),
    Golden(input="What is the chemical symbol for gold?"),
    Golden(input="Who discovered penicillin?"),
    Golden(input="What is the tallest mountain in the world?"),
    Golden(input="Who composed the 'Moonlight Sonata'?"),
    Golden(input="What is the largest ocean on Earth?"),
    Golden(input="Who painted 'The Starry Night'?"),
    Golden(input="What is the speed of light in meters per second?"),
    Golden(input="Who wrote 'The Great Gatsby'?"),
]

synthesizer = Synthesizer()

print("Attempting to generate goldens from goldens...")

# This call does not populate synthesizer.synthetic_goldens internally in async mode
evolved_goldens = synthesizer.generate_goldens_from_goldens(
    goldens=goldens_to_evolve,
    max_goldens_per_golden=1
)

print(f"Generated {len(evolved_goldens)} evolved goldens.")
print(f"Count of goldens in synthesizer: {len(synthesizer.synthetic_goldens)}")
```

```bash
$ python test.py
Attempting to generate goldens from goldens...
✨ Generating up to 12 goldens using DeepEval ([...] method=Scratch): 100%|██████████████████| 12/12 [00:07<00:00,  1.61it/s]
Generated 12 evolved goldens.
Count of goldens in synthesizer: 0
Forcing release of ChromaDB connections...
```

### Expected Behavior

The `generate_goldens_from_goldens` method should correctly populate `self.synthetic_goldens` after execution.

### Environment

*   **Operating System**: Ubuntu
*   **Python Version**: 3.12.9
*   **`deepeval` Version**: 3.0.1

@tm-echo-arcelao should be fixed in latest version, can you give it a try?

@penguine-ip LGTM! Appreciate the quick fix 💯
Repository: confident-ai/deepeval
Issue: IFEval Benchmark?
Root cause: Unimplemented Feature Gap

Is there any plans to add IFEval instruction following benchmark to the pool? It's an important benchmark.

@tjasmin111 No at the moment, but happy to merge any PR for it

But can we still do eval on IFEval using hf datasets?


hi @penguine-ip  i tried implementing IFEval benchmark for deepeval, here is the raised PR: https://github.com/confident-ai/deepeval/pull/1729
will be super grateful, if you can take a look at it and suggest the required changes.

cc: @tjasmin111
Repository: confident-ai/deepeval
Issue: Using Custom Embedding Models ERROR
Root cause: Documentation Deficiency

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**

![Image](https://github.com/user-attachments/assets/2155ad46-2749-44da-a707-877bfc8bf3b5)

When using deepeval for evaluation, I want to use a local all-MiniLM-L6-v2 model as the embedding model. I referred to the official guide and implemented the Class EmbeddingModel, but encountered the following error during use: **Synthesizer.__init__() got an unexpected keyword argument 'embedder'.** Please help me, thank you. My specific code is as follows:"

# evaluation.py
from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from custom_models import DeepSeekModel
from embedding_model import LocalEmbeddingModel
import os
from deepeval.synthesizer import Synthesizer

deepseek_model = DeepSeekModel(
    model="deepseek-reasoner",
    temperature=0.3,
    max_tokens=1024
)

embedding_model=LocalEmbeddingModel(model_name="all-MiniLM-L6-v2",
local_path="/home/llama/model/all-MiniLM-L6-v2")
**synthesizer=Synthesizer(embedder=embedding_model)**

test_case = LLMTestCase(
    input="解释BERT模型的核心创新",
    actual_output="BERT通过双向Transformer编码器和掩码语言建模任务，能够同时考虑上下文的所有方向，显著提升了语言理解能力。",
    context=[
        "BERT使用Transformer编码器",
        "预训练任务包括掩码语言建模和下一句预测"
    ]
)


metric = AnswerRelevancyMetric(
    model=deepseek_model,
    threshold=0.7,
    include_reason=True
)


evaluate([test_case], [metric])


print("\n" + "="*50)
print(f"评估结果: {'✅ 通过' if metric.is_successful() else '❌ 未通过'}")
print(f"得分: {metric.score:.2f}/1.0")
print("评估原因:")
print(metric.reason)
print("="*50)



**Desktop (please complete the following information):**
 - OS: ubuntu 22.04
 - Version deepeval 3.0.8




hey @HelloWortd apologies the docs for that page is slightly outdated, please provide the embedder through the ContextConstructionConfig instead: https://deepeval.com/docs/synthesizer-generate-from-docs#customize-context-construction

At the end of the parameters list you wil see `embedder`, in the meantime we will update the docs, thanks!

@penguine-ip "Thanks for your reply, it's very helpful! BTW, I also want to ask if there's a global configuration method to set the embedding model to my local model,thanks!
Repository: confident-ai/deepeval
Issue: Supported judge models in Deepeval?
Root cause: Validation Gap

Looks like OpenAI is the default judge model. I tried to change it to another model, say Anthropic model. Can we use any models? 

```
 model  = AnthropicModel(
        model_name="claude-3-7-sonnet-latest"
)
```

Can I use the latest Opus-4 model too?

@tjasmin111 can you see if this helps: https://deepeval.com/integrations/models/anthropic#available-anthropic-models

Thnaks. I don't see the latest models, like Claude 4. Is it easy to just define it ourselves by just changing the name?

@tjasmin111  deepeval uses official `anthropic` client under the hood for llm querying with claude-family models, so ideally it should work with just replacing the `model_name` with new model name ( given, it's supported by anthropic ), however, in the current implementation, if you try to change the name, you will get an error due to how `calculate_cost`  method logic is defined 
https://github.com/confident-ai/deepeval/blob/ba077ff88a4414cb5aab8c177f1c4bbd0a775744/deepeval/models/llms/anthropic_model.py#L94

@penguine-ip  i think [this line](https://github.com/confident-ai/deepeval/blob/ba077ff88a4414cb5aab8c177f1c4bbd0a775744/deepeval/models/llms/anthropic_model.py#L95) is not correct. If `self.model_name` is not found in `model_pricing` global dict, it falls back to the entire model_pricing dictionary — not a single model’s pricing dictionary — which will result in a TypeError on the following line.

i have created a PR to address this : https://github.com/confident-ai/deepeval/pull/1715

The thought process is since deepeval uses official anthropic sdk so as long as any model is supported by anthropic client, it should be also supported by deepeval ensuring `availability and continuity`.
The logic of passing avg. cost with a warning might be wrong but this would atleast make the system somewhat `resilient and adaptive`, the team can come back later and update the `model_pricing` and customers won't be affected in terms of functional testing as well.

@penguine-ip  lemme know what are your thoughts about this

@AbhishekRP2002 for sure, i agree. We're going to make it a centralized list of dict instead to keep things more consistent as well, thanks for the PR!
Repository: confident-ai/deepeval
Issue: Using Custom LLMs for Evaluation error
Root cause: Validation Gap

Following the guide at 
https://deepeval.com/guides/guides-using-custom-llms,
I implemented a custom model that successfully obtains responses. However, when testing the relevance metric, I encountered a 
**AttributeError: 'str' object has no attribute 'statements'** 
Please help me, thank you. My specific code is as follows:"

# custom_models.py

```python
import os
from openai import OpenAI
from deepeval.models.base_model import DeepEvalBaseLLM

class DeepSeekModel(DeepEvalBaseLLM):
    def __init__(
        self,
        model: str = "deepseek-chat",
        max_tokens: int = 2048,
        temperature: float = 0.1,
        *args,
        **kwargs
    ):
        self.model = model
        self.max_tokens = max_tokens
        self.temperature = temperature
        self.client = OpenAI(
            api_key=os.environ["DEEPSEEK_API_KEY"],
            base_url="https://api.deepseek.com/v1"
        )
def load_model(self):
    return self.client

def generate(self, prompt: str) -> str:
    response = self.client.chat.completions.create(
        model=self.model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=self.max_tokens,
        temperature=self.temperature
    )
    return response.choices[0].message.content

async def a_generate(self, prompt: str) -> str:
    return self.generate(prompt)  

def get_model_name(self) -> str:
    return "DeepSeek"
```

# evaluation.py

```python
from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from custom_models import DeepSeekModel
from embedding_model import LocalEmbeddingModel
import os
from deepeval.synthesizer import Synthesizer
import os

os.environ["DEEPSEEK_API_KEY"] = ""
deepseek_model = DeepSeekModel(
    model="deepseek-reasoner",
    temperature=0.3,
    max_tokens=1024
)
#print(deepseek_model.generate("write a joke")

test_case = LLMTestCase(
    input="解释BERT模型的核心创新",
    actual_output="BERT通过双向Transformer编码器和掩码语言建模任务，能够同时考虑上下文的所有方向，显著提升了语言理解能力。",

#    

)
metric = AnswerRelevancyMetric(
    model=deepseek_model,
    threshold=0.7,
    include_reason=True
)

metric.measure(test_case)
print(metric.score,metric.reson)
```

when i use python evaluation.py ,firstly i see this:

![Image](https://github.com/user-attachments/assets/de97af11-78ea-4cfa-a0d2-407f77d76fc0)

and a few seconds later i get this error:

![Image](https://github.com/user-attachments/assets/b97b8ffe-828f-4874-9cd6-b23a158a3eb1)

 - OS: [ubuntu 22.04]
 - Version [deepeval 3.0.8]

hey @deedpool07 it is beause your model outputed an invalid output, can you try JSON confinement: https://deepeval.com/guides/guides-using-custom-llms#json-confinement-for-custom-llms
Repository: confident-ai/deepeval
Issue: evaluate function raise TypeError: 'TestResult' object is not iterable
Root cause: Interface Contract Mismatch

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
`evaluate` function use `DisplayConfig` with `file_output_dir`, will raise TypeError: 'TestResult' object is not iterable.

the `aggregate_metric_pass_rates_to_file` function is called with a single TestResult object instead of a list of TestResult objects. on line 466 of in utils.py

**To Reproduce**
Steps to reproduce the behavior:
```
        evaluate(
            test_cases=test_cases,
            metrics=[
                AnswerRelevancyMetric(model=model),
                FaithfulnessMetric(model=model),
                ContextualPrecisionMetric(model=model),
                ContextualRecallMetric(model=model),
                ContextualRelevancyMetric(model=model)
            ],
            display_config=DisplayConfig(file_output_dir=f"{self.path}/benchmark_response"),
        )
```

**Expected behavior**
The evaluate function should correctly handle the test_cases parameter as a list of TestResult objects, ensuring that aggregate_metric_pass_rates_to_file can process the input correctly.

**Screenshots**

![Image](https://github.com/user-attachments/assets/6349bcd8-0805-4126-9afe-055e97db1182)

**Desktop (please complete the following information):**


**Smartphone (please complete the following information):**


**Additional context**
Repository: confident-ai/deepeval
Issue: Getting error while generating red teaming prompts
Root cause: Interface Contract Mismatch

Getting validation error for SimulatedAttack.Using open-ai model via azure throws error while using red teaming.It's working completely fine when i use the direct open-ai key. The model is loaded correctly in deepeval class and is able to generate the answers but throws error when used in red-teammer.

<img width="531" alt="Image" src="https://github.com/user-attachments/assets/850fcdae-5f29-4ccf-85f2-abcd602b9a4c" />

<img width="1028" alt="Image" src="https://github.com/user-attachments/assets/604b1401-509c-409f-8200-4a60a2dbb5fc" />

<img width="1131" alt="Image" src="https://github.com/user-attachments/assets/06e8651e-52c8-4e5d-8f6e-13c2e6378ac6" />

@jitenderfoundry please raise this in deepteam, and also provide the versions of the packages you're running with
Repository: confident-ai/deepeval
Issue: implement "don't rewrite to file if it is read only" using DEEPEVAL_FILE_SYSTEM
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
Including a DEEPEVAL_FILE_SYSTEM env variable, which a user can set to "READ_ONLY" or "READ_WRITE"
the idea is that we won't rewrite to file if it is read only

**Describe the solution you'd like**
DEEPEVAL_FILE_SYSTEM env check added in __init__.py https://github.com/adityabharadwaj198/deepeval/blob/aditya/fix-type-error/deepeval/__init__.py#L73
but it looks like it isn’t used yet across file-writing logic.

1. scan the codebase for all file write operations (open(..., "w"), etc.)
2. Wrap them with a check for is_read_only_env() so they no-op in READ_ONLY mode

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: confident-ai/deepeval
Issue: deepeval integration in weights&biases

Hi DeepEval Team,

First off, I want to say fantastic work on this library. I've been using it for my projects and have been incredibly impressed with the quality and comprehensiveness of the RAG evaluation metrics.

I'm [Anshuman](https://heyyanshuman.com/work), an *independent* software engineer who is also an active user of Weights & Biases. 

While recently working with their [weave](https://github.com/wandb/weave/tree/master/weave) platform for a RAG project, I saw a clear opportunity to connect the two platforms. Weave provides a powerful tracing and experiment management system, but I felt its evaluation suite would be significantly enhanced by integrating DeepEval's specialized metrics.

To bridge this gap, I took the initiative this past weekend to build a robust, idiomatic integration. I've implemented four of your core RAG metrics as native Weave Scorers:

1. Faithfulness

2. AnswerRelevancy

3. ContextualPrecision

4. ContextualRecall


You can view the Pull Request here: https://github.com/wandb/weave/pull/4931

My hope is that this integration will create a seamless on-ramp for W&B's large user base to discover and adopt DeepEval as their go-to solution for serious RAG evaluation.

I wanted to share this with you all directly.



@kanpuriyanawab looks great! although 99% of deepeval uesrs already use Confident AI instead of WEAVE: https://www.confident-ai.com/
Repository: confident-ai/deepeval
Issue: LiteLLM for Embedding Models
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
I am trying to make a whole holistic system of AI apps' evals, but I found that no LiteLLM implementation of embedding models is available. I'd recommend you to switch everything related to LLMs and Embedding models entirely to LiteLLM. What's your opinion on that?

@santialtia this would force all users to learn and use litellm, so we won't do it