zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Test cases reported multiple times with `pytest.mark.flaky`
Root cause: Validation Gap

Hi, thanks for this great library.

I'm running into an issue when marking tests with `pytest.mark.flaky`. In cases where tests pass e.g. on the second try, the first attempt is logged on the `confident-ai.com` dashboard as a failure, and the second attempt is logged as a pass. 

This leads to many more test cases being shown on the dashboard than were actually run in practice.

Is there a better way to handle `retry` logic other than using `mark.flaky`? Should this be handled internally within the test, e.g. catching `AssertionErrors` N times, and only then throwing when N is exceeded?

## Minimal example

This won't work "out-of-the-box", but should show how I'm constructing the test cases.

```python
@pytest.mark.parametrize(
    "query",
    # big list of queries
    [...]
)
@pytest.mark.flaky(reruns=5)
def test_with_flaky(query: str):
    # Assume that this takes the query, makes a generation with the agent
    #  and then returns a `deepeval.LLMTestCase` object
    test_case = create_basic_test_case(query)

    metric = GEval(
        name="flaky test",
        criteria=(
            "..."
        ),
        evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    )

    # Maybe fails the first time, but works the second
    assert_test(test_case, [metric])
```

and then the command to run the tests from the CLI:

```bash
poetry run deepeval login --api-key $CONFIDENT_API_KEY
poetry run deepeval test run test_file.py -n 2
```


Possibly related: I am unable to use `pytest.fixtures` to run additional commands during preparation/teardown with `deepeval`.

I have a small fixture scoped to `session` that prints some logging messages before and after starting tests. This works fine with `poetry run pytest ...`, but the `poetry run deepeval test run`  command above does not seem to pick it up.

## Minimal example

```python
@pytest.fixture(scope="session", autouse=True)
def do_something_before_and_after_tests():
    """
    Runs before and after tests
    """
    # Printed before starting tests
    print("Starting tests!")

    # All tests are run here
    try:
        yield

    # Hit after finishing all tests, regardless of outcome
    finally:
        print("Finish tests!")
```



Thank you for your feedback @HuwCheston, I will bring these issues up at our next meeting.
Repository: confident-ai/deepeval
Issue: Need to Recalculate Pass Rates with New Thresholds on Existing Evaluation Data Without Re-running LLM Evaluations
Root cause: Unimplemented Feature Gap


Description:

I have completed a full evaluation run of my LLM application using DeepEval, and the results are saved with a specific Test Run ID in Confident AI. The evaluation used GEval metrics with initially set thresholds.

After reviewing the results, I found that some thresholds (e.g., for "Faithfulness" metric) were set too high, causing unnecessarily low pass rates. I want to explore how different threshold values would affect the overall pass rates and metrics.

I need to efficiently recalculate all derived statistics (pass/fail status, overall pass rates, aggregate scores) based on existing evaluation data using new threshold values, without:
  1. Re-running any LLM evaluations (to avoid additional API costs and time)
  2. Creating duplicate test cases or test runs
  3. Losing the original evaluation scores and reasoning

Hi team, this is an excellent feature request. I completely agree that separating the expensive scoring phase from a more flexible, inexpensive analysis phase would be a huge improvement to the user workflow.

I've looked into the codebase, and the proposed recalculate_results method seems like the best approach. Here is my plan for implementation:

Modify EvaluationResult: I'll add the new method recalculate_results(self, new_thresholds: Dict[str, float]) to the EvaluationResult class located in deepeval/evaluate/types.py.

Implement Recalculation Logic: The method will iterate through the test_results stored in the EvaluationResult object. For each metric, it will use the original, stored score and compare it against the new threshold to update the success status. This ensures no new API calls are made.

Display Updated Aggregates: After recalculating, the method will call the existing aggregate_metric_pass_rates utility to print a new summary table to the console.

Add an Example: To make the feature easy to understand, I will add a new file, examples/recalculate_example.py, demonstrating how to use it.

I'm happy to pick this up and will start working on a Pull Request. Let me know if you have any initial feedback on this approach!
Repository: confident-ai/deepeval
Issue: GEval criteria does not match what is sent to the judge
Root cause: Documentation Deficiency

I am running the very first example eval: https://deepeval.com/tutorials/tutorial-setup against a locally hosted openai/gpt-oss-20b. I am puzzled by the criterion specified in `test_app.py`: 
```
"Determine if the 'actual output' is correct based on the 'expected output'."
``` 

and the actual prompt sent to the judge:

```
"content": "{\n    \"steps\": [\n        \"Compare the content of Actual Output and Expected Output line by line, ignoring leading/trailing whitespace.\",\n        \"Verify that every token (word or symbol) in Expected Output appears in the same order within Actual Output.\",\n        \"Check that no additional tokens are present in Actual Output beyond those specified in Expected Output.\",\n        \"Confirm that the overall structure (e.g., number of lines, sections) matches between Actual and Expected Outputs.\"\n    ]\n}",
```
I.e. the criterion says that the two should be semantically similar but the prompt asks that the two outputs are identical. Why would you even use an LLM judge for doing such a comparison, it is a simple string comparison...


Hey @erlingrj actually the criterion doesn't mention semantically similar but "correct", so the evaluation steps that GEval generates might be different based on that.

Some folks that already have the evaluation steps would actually choose to hard code it instead: https://deepeval.com/docs/metrics-llm-evals#evaluation-steps
Repository: confident-ai/deepeval
Issue: TypeError: capture_metric_type() missing 1 required positional argument: 'in_component'
Root cause: External Dependency Breakage



**Describe the bug**
I am trying to call Ragas metrics but this error appears `TypeError: capture_metric_type() missing 1 required positional argument: 'in_component'` 
code used 
````
from deepeval.metrics.ragas import RAGASContextualPrecisionMetric
from deepeval.test_case import LLMTestCase

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = RAGASContextualPrecisionMetric(
    threshold=0.7
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    retrieval_context=retrieval_context
)

metric.measure(test_case)
print(metric.score)
print(metric.reason)
````


**To Reproduce**
Steps to reproduce the behavior:
1. Ragas version ` 0.2.1`
2. deepeval version `3.1.8`

Hey @rehamessameltagoury ragas is no longer supported in deepeval
Repository: confident-ai/deepeval
Issue: Error when settings "max_tokens" on AmazonBedrockModel
Root cause: Interface Contract Mismatch

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
Using the `max_tokens` parameter with `AmazonBedrockModel` causes a `ParamValidationError.`

```
ParamValidationError: Parameter validation failed:
Unknown parameter in infereneConfig: "max_tokens", must be one of: maxTokens, temperature, topP, stopSequences...
```

It's because the code implementation uses both the correct `maxTokens` and the incorrect `max_tokens` when expanding the `generation_kwargs` argument.

Here is the implementation I mentioned: https://github.com/confident-ai/deepeval/blob/a9173776ceb32026f350f11f9dc3e031859c10bd/deepeval/models/llms/amazon_bedrock_model.py#L117-L126


**To Reproduce**

deepeval 3.4.7

please use `AmazonBedrockModel` with `max_tokens` parameter.

```
from deepeval.models.llms.amazon_bedrock_model import AmazonBedrockModel

AmazonBedrockModel(
  model_id=<any>,
  region_name=<any>,
  generation_kwargs={"max_tokens": 2000},
)
```

**Expected behavior**
No errors.

**Screenshots**

<img width="1521" height="272" alt="Image" src="https://github.com/user-attachments/assets/1850eafc-12fa-4e5b-8777-dc8e7c19c9de" />


@penguine-ip  Hey, I fixed this https://github.com/confident-ai/deepeval/pull/2017
Repository: confident-ai/deepeval
Issue: Can SynthesizerTemplate support Chinese?
Root cause: Documentation Deficiency

from deepeval.synthesizer import Synthesizer

synthesizer = Synthesizer()

goldens = synthesizer.generate_goldens_from_docs(
    document_paths=['test.pdf']
)

test.pdf is Chinese content. but goldens is english

@IcanDoItL i believe there's a language parameter

@penguine-ip use styling_config can Solve,but,I hope SynthesizerTemplate can Support customization

@IcanDoItL i don't follow 100%, can you please clarify?
Repository: confident-ai/deepeval
Issue: ToolCorrectnessMetric incorrectly returns 0.0 score when both expected_tools and tools_called are empty lists
Root cause: Algorithmic Error

## Describe the bug
The `ToolCorrectnessMetric` incorrectly returns a score of 0.0 when both `expected_tools` and `tools_called` are empty lists, instead of the expected perfect score of 1.0. This happens when evaluating test cases where no tools are expected to be called and no tools were actually called.

## To Reproduce
Steps to reproduce the behavior:
1. Create a test case with `expected_tools=[]` (empty list)
2. Run an agent that doesn't call any tools, resulting in `tools_called=[]` (empty list)  
3. Evaluate using `ToolCorrectnessMetric` with default parameters (non-exact matching mode)
4. Observe that the score is 0.0 instead of 1.0

## Expected behavior
When both `expected_tools` and `tools_called` are empty lists, the metric should return a perfect score of 1.0, indicating that the agent correctly avoided calling any tools when none were expected.

## Code Example
```python
from deepeval.test_case import LLMTestCase
from deepeval.metrics import ToolCorrectnessMetric

# Test case where no tools should be called
test_case = LLMTestCase(
    input="What is 2+2?",
    actual_output="4",
    expected_output="4",
    tools_called=[],  # No tools called
    expected_tools=[]  # No tools expected
)

metric = ToolCorrectnessMetric()
score = metric.measure(test_case)
print(f"Score: {score}")  # Currently prints 0.0, should print 1.0
```

## Root Cause
The bug is in the `_calculate_non_exact_match_score()` method around line 225-229:

```python
return (
    total_score / len(self.expected_tools)
    if self.expected_tools  # This evaluates to False for empty list
    else 0.0  # Bug: should handle the case where both lists are empty
)
```

When `expected_tools` is an empty list, `len(self.expected_tools)` is 0, so the condition `if self.expected_tools` evaluates to `False`, causing the method to return `0.0`. However, when both lists are empty, this should be considered a perfect match and return `1.0`.

## Suggested Fix
Replace the problematic return statement with:

```python
if not self.expected_tools:
    # If no tools were expected, check if no tools were called
    return 1.0 if not self.tools_called else 0.0
else:
    return total_score / len(self.expected_tools)
```

## Workaround
Currently, the issue can be worked around by using `should_exact_match=True` parameter, which has the correct logic for handling empty lists:

```python
metric = ToolCorrectnessMetric(should_exact_match=True)
```

## Environment
- OS: macOS
- Python version: 3.12
- deepeval version: [current version]

Hey @qwchua @udaykiran2427 solved it, thank you!
Repository: confident-ai/deepeval
Issue: Structured Output and Input for Synthetic Datasets
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
Currently I have to do heavy prompt engineering with 'Synthesizer' to give me structured output and goldens for my dataset. We should be able similar to langgraph, we can accept a json schema or pydantic type and return a structured output or structured goldens.

**Describe the solution you'd like**
The idea solution I would like is below

`Synthesizer(...config, input_response_format=PydanticType, golden_response_format=PydanticType)`

**Describe alternatives you've considered**
Heavy prompt engineering

**Additional context**
I would be happy to work on this as long as the team would actually like this in the Synthesizer

Hey @botirk38 yes please would really appreciate an interface like this. I think those two parameters can be optional and go into our four generation methods instead of `Synthesizer` initialization, thank you in advance

yea I would say that makes sense, will work on it this weekedn!
Repository: confident-ai/deepeval
Issue: FilterTemplate same Example context but， Example JSON not same
Root cause: Algorithmic Error

https://github.com/confident-ai/deepeval/blob/3e083b28975884b37f9af323b313f1665ae70f53/deepeval/synthesizer/templates/template.py#L255

<img width="1409" height="775" alt="Image" src="https://github.com/user-attachments/assets/739acffe-d12d-42c7-87af-9e5233249e33" />
Repository: confident-ai/deepeval
Issue: Batch Inference support in LLM clients
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
Alot of LLM providers provide a batch inference endpoint, that will allow you to submit a bunch of requests, and poll for a completion, this is useful because I often run a eval step before I evaluate the testcases to run inference to get the actual output from the model before running a benchmark 

**Describe the solution you'd like**
for the LLM client add a batch_generate and batch_agenerate methods

**Describe alternatives you've considered**
Just call singular inference on each testcase but thats inefficient.

**Additional context**
Add any other context or screenshots about the feature request here.


@botirk38 hey i'm not entirely sure what the use case here is - are you saying batching the evals would be something that'd be useful?

for golden in dataset.goldens:
    test_case = LLMTestCase(
        input=golden.input,
        actual_output=your_llm_app(golden.input) # replace with your LLM app
    )
    dataset.add_test_case(test_case)


this is directly from your docs when building eval datasets, so if the LLM clients from deepeval, had a batch method I should be able to unfold this loop by just passing my all my golden inputs

@botirk38 you can just do a wrapper around whatever client yo u have i assume
Repository: confident-ai/deepeval
Issue: Missing comma causes type checkers to miss import
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

No

**Describe the bug**

Using deepeval 3.4.9. It looks like the problem still exists in the main branch.

In `deepeval/metrics/__init__.py`, there's a missing comma after `"ConversationalDAGMetric"`. This causes some type checkers to not find `AnswerRelevancyMetric` when importing from the metrics module.

**To Reproduce**

Use:
```Python
from deepeval.metrics import AnswerRelevancyMetric
```
And pyright produces an error:
```
"AnswerRelevancyMetric" is not exported from module "deepeval.metrics"
  Import from "deepeval.metrics.answer_relevancy.answer_relevancy" 
```

**Expected behavior**

The import doesn't complain.

```
    "ConversationalDAGMetric"
```

should be

```
    "ConversationalDAGMetric",
```

See 7d55ce427088898a2590990d25385dc7f2488b6f.

**Screenshots**

N/A

**Desktop (please complete the following information):**
 - OS: macOS 15.6.1

**Additional context**



@connorimes just fixed!
Repository: confident-ai/deepeval
Issue: Are you dropping Python 3.9 support?
Root cause: Environment Incompatibility

Seems like the use of `|` operand fails if we have an env with Python 3.9

```python
File ~/python3.9/.venv/lib/python3.9/site-packages/deepeval/utils.py:519
    514             time.sleep(0.1)
    515         progress.remove_task(pbar_id)
    518 def read_env_int(
--> 519     name: str, default: int, *, min_value: int | None = None
    520 ) -> int:
    521     """Read an integer from an environment variable with safe fallback.
    522
    523     Attempts to read os.environ[name] and parse it as an int. If the variable
   (...)
    533         The parsed integer, or `default` on any failure.
    534     """
    535     raw = os.getenv(name)

TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
```

This could easily be fixed with `typing.Union` instead of the `|` operator. Anyway, is this intended this way to drop 3.9 support?

On [`pypi.org` ](https://pypi.org/project/deepeval/) 3.9 is being mentioned as supported.


@davidsbatista it's a mistake, it shouldn't be using the `|` operator and I've made some changes to it in the latest commit. If you see any more instances please let us know.

Thanks @penguine-ip! I've also noticed that there's already a new release - this makes the haystack integration with deepeval work again 👍🏽
Repository: confident-ai/deepeval
Issue: Releases and pypi inconsistencies
Root cause: Documentation Deficiency

Current pypi version is `3.4.9` as of today -> https://pypi.org/project/deepeval/3.4.9/

But the release section does not include this version -> https://github.com/confident-ai/deepeval/releases

How do you document releases? Which is the source code that corresponds to each release?




In fact, there is even a mismatch between pypi version and the description badge version (`3.3.5`)

<img width="1240" height="781" alt="Image" src="https://github.com/user-attachments/assets/1ccc813f-9f5d-4d4d-99bf-a5b1ddaf6ef1" />

Hey @surfingdev this problem has just been fixed!
Repository: confident-ai/deepeval
Issue: Feature: Add CometAPI support for DeepEval
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
Teams need provider redundancy and cost control when evaluating LLM apps. Limiting to a few providers creates outages during rate limits or regional issues and increases cost variance.

**Describe the solution you'd like**
- Add CometAPI support by allowing OpenAI clients/adapters to be configured with a custom `base_url` and API key.
- Support chat/completions used in evaluation loops, plus embeddings where applicable.
- Streaming support when tests rely on token events.

**Describe alternatives you've considered**
- Custom proxies—adds maintenance and latency.
- Stay with existing providers—reduces resilience and portability.

**Additional context**
Resources:
- Website: https://www.cometapi.com/
- Docs: https://api.cometapi.com/doc
- Base URL: https://api.cometapi.com/v1/
- Models: https://api.cometapi.com/v1/models
- Get API Key: https://api.cometapi.com/console/token
- Pricing: https://api.cometapi.com/pricing

Happy to open a PR with docs and examples if this direction is acceptable.


That's currently the biggest issue preventing me from choosing this framework. 😢 

Hey @TensorNull and @yunfan42,
We already support this feature using the `_openai_api_key` and `base_url` parameters in our `from deepeval.models import GPTModel` class. 
We'll also make an update in the docs to include this soon. Thanks for bringing this to our attention.


@yunfan42 and @TensorNull we're happy to take a PR on this one !
Repository: confident-ai/deepeval
Issue: Task Completion  metric observed_callback 使用不了
Root cause: Documentation Deficiency

https://deepeval.com/docs/metrics-task-completion

<img width="733" height="512" alt="Image" src="https://github.com/user-attachments/assets/2104ad7f-464d-42ae-a751-4715df198aec" />
我用最新的版本3.5.2测试 evaluate，方法里面不支持 observed_callback属性呀。

Hey @tangjiawei2017 it is solved in latest docs: https://deepeval.com/docs/metrics-task-completion#usage
Repository: confident-ai/deepeval
Issue: Deepeval.making external calls
Root cause: Documentation Deficiency

I'm trying to run deepeval using th opeai client. I'm overriding the base_url, however the provider is being set to azure even though the .deepeval config file has only open I enabled.

Also, at the end it tries to upload something to us.i.posthog.com . What exactly is being uploaded and how can I disable it?

I found in the docs how to disable it, in case others have the same question. https://github.com/confident-ai/deepeval/blob/main/docs/docs/data-privacy.mdx

It seems that telemetry is being sent not only to sentry, but also newrelic and posthog?

Hey @fabiodouek, did you try using the [`set-openai` command](https://github.com/confident-ai/deepeval/blob/main/docs/docs/cli.mdx#full-list)?

Regarding the telemetry, we send basic usage telemetry to posthog to help us improve the product. 

### What telemetry is sent & how to disable it

- DeepEval also sends errors to Sentry and usage events to PostHog and OTLP spans to New Relic (via OpenTelemetry). Here’s the current behavior:

  - PostHog (usage analytics): event names like “Ran evaluation”, metric names, whether you’re in Jupyter, a randomly generated local UUID, and an “anonymous” public IP (queried via api.ipify.org). No prompts, model outputs, API keys, or dataset contents are sent.
  - New Relic (OTLP spans): spans for the same events with the same attributes (feature used, env type, UUID, public IP).
  - Sentry (errors): only if you explicitly opt in with ERROR_REPORTING=1, otherwise Sentry isn’t used for error capture.

I will be doing more rounds of documentation updates in the near future. Your feedback has been appreciated.

For anyone else finding this, you can opt out of telemetry by setting this environment variable:
```
export DEEPEVAL_TELEMETRY_OPT_OUT=1
```

Telemetry is used to help us resolve issues that we would otherwise not know about.
Repository: confident-ai/deepeval
Issue: G-Eval score normalisation
Root cause: Algorithmic Error

G-Eval appears to mis-normalise scores in a_measure(). 

https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/g_eval/g_eval.py
The code currently does:
`self.score = (
    float(g_score) / self.score_range_span
    if not self.strict_mode
    else int(g_score)
)
`

Dividing by a `self.score_range_span` without subtracting the minimum leads to incorrect ranges (e.g., 1→0.2, 5→1.0 if span=5; or 1→0.25, 5→1.25 if span=4).


Hey @kemi-bower  👋 , can you please confirm if this issue is still open? If yes, could you assign it to me? I’d like to work on fixing it.