zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Conversation simulator creates two initial user messages.
Root cause: Algorithmic Error

**Describe the bug**
The `ConversationSimulator.simulate()` creates two simulated user turns (instead of one) before giving the turn to the model. It's not a big deal, but it doesn't represent real scenarios and affects the number of actual turns the simulator makes. 

**To Reproduce**
Simple python script similar to the documentation here: https://deepeval.com/docs/conversation-simulator
```python
from pprint import pprint

from deepeval.dataset import ConversationalGolden
from deepeval.test_case import Turn
from deepeval.simulator import ConversationSimulator

# Create ConversationalGolden
conversation_goldens = [
    ConversationalGolden(
        scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
        expected_outcome="Successful purchase of a ticket.",
        user_description="Andy Byron is the CEO of Astronomer.",
    ),
]

# Define chatbot callback
def chatbot_callback(input, turns):
    return Turn(role="assistant", content="I am a chatbot!")


if __name__ == "__main__":
    # Run Simulation
    simulator = ConversationSimulator(model_callback=chatbot_callback, async_mode=True)
    conversational_test_cases = simulator.simulate(conversational_goldens=conversation_goldens, max_turns=3)
    print("Number of turns: ", len(conversational_test_cases[0].turns))
    pprint(conversational_test_cases[0].turns)

```

Output:
```
Number of turns:  3
[Turn(role='user', content='Hi there, I’m interested in purchasing a VIP ticket to the upcoming Coldplay concert. Could you help me with the process?'),
 Turn(role='user', content='Could you let me know what VIP packages are available and how I can secure a ticket?'),
 Turn(role='assistant', content='I am a chatbot!')]
```

**Expected behavior**
Only as single opening turn from the user should be generated. E.g.:
```
Number of turns:  3
[Turn(role='user', content='Hi there, I’m interested in purchasing a VIP ticket to the upcoming Coldplay concert. Could you help me with the process?'),
 Turn(role='assistant', content='I am a chatbot!'),
 Turn(role='user', content='That is not very helpful')]
```

**Additional context**
Used `deepeval==3.3.5`


Thanks @jvdgoltz will fix this immediately !

@jvdgoltz fixed in https://github.com/confident-ai/deepeval/pull/1891!
Repository: confident-ai/deepeval
Issue: Unstable test results
Root cause: Algorithmic Error

aaa@JeasonGaos-MacMini deepeval % deepeval run test_example.py 

<img width="1404" height="304" alt="Image" src="https://github.com/user-attachments/assets/6d9d4d8e-0ee6-4973-8509-1717c9aa46b5" />

<img width="1410" height="314" alt="Image" src="https://github.com/user-attachments/assets/53f12773-514c-4e1c-bef4-76e19b404ebd" />


@a1boyband would appreicate more context to understand whats the problem

> [@a1boyband](https://github.com/a1boyband) would appreicate more context to understand whats the problem

I used deepseek as LLM

The script content of test_example.py is as follows

```
from deepeval.test_case import LLMTestCase, LLMTestCaseParams 
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.metrics import HallucinationMetric
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.metrics import ConversationCompletenessMetric

test_case = LLMTestCase(
    input="xxx",
    actual_output='''xxx''',
    expected_output=''' xxx''',
    context=['''xxx''']
)

correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
        threshold=0.5
    )

conversation_completeness = ConversationCompletenessMetric(threshold=0.5)

answer_relevancy = AnswerRelevancyMetric(threshold=0.7)

Hallucination = HallucinationMetric(threshold=0.5)

assert_test(test_case, [answer_relevancy,correctness_metric,conversation_completeness,Hallucination])
```
Repository: confident-ai/deepeval
Issue: asserttest logs more than once if multiple metrics are used.
Root cause: Algorithmic Error

Currently if I assert test using x number of metrics on a test case,  the test case is logged x number of times on the dashboard instead of just logging once.

This functionality exists in `run_test` - i will document

I'm not asking for a functionality, what do you mean? 

For this test case:
```
def test_1():
    input = "What does your company do?"
    output = query(input)
    context = "Our company specializes in cloud computing, data analytics, and machine learning. We offer a range of services including cloud storage solutions, data analytics platforms, and custom machine learning models."
    funny_metric = LLMEvalMetric(name="Funny", criteria="How funny it is", minimum_score=0.5)
    factual_consistency_metric = FactualConsistencyMetric(minimum_score=0.7)
    test_case = LLMTestCase(query=input, output=output, context=context)
    assert_test(test_case, [factual_consistency_metric, funny_metric])
```

  I'm getting test_1 logged twice.

<img width="1705" alt="Screenshot 2023-10-12 at 3 30 16 AM" src="https://github.com/confident-ai/deepeval/assets/143328635/26bbbbe1-7fd8-46e6-9738-065e3539d913">

  
<img width="1708" alt="Screenshot 2023-10-12 at 3 32 14 AM" src="https://github.com/confident-ai/deepeval/assets/143328635/069ea3fe-c201-43e3-bc96-1901110f1ad3">

The expected behavior should be, 100% of test cases failing, and only 1 test case logged.
Repository: confident-ai/deepeval
Issue: Terminal Output isn't verbose enough
Root cause: Unimplemented Feature Gap

Hey guys

I noticed that the terminal output is not verbose enough, e.g. here:

```
                                                                                                           Test Results                                                                                                           
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test case                                                                          ┃ Metric              ┃ Score                                                                               ┃ Status ┃ Overall Success Rate ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ test_chat                                                                          │                     │                                                                                     │        │ 0.0%                 │
│                                                                                    │ Correctness [GEval] │ 0.0 (threshold=1.0, evaluation model=Azure OpenAI Model Evaluator, reason=The       │ FAILED │                      │
│                                                                                    │                     │ Actual Output does not address the specific factual information about the minimum   │        │                      │
│                                                                                    │                     │ ground clearance required for a line, as detailed in the Expected Output.,    │        │                      │
│                                                                                    │                     │ error=None)                                                                         │        │                      │
│ Note: Use Confident AI with DeepEval to analyze failed test cases for more details │                     │                                                                                     │        │                      │
└────────────────────────────────────────────────────────────────────────────────────┴─────────────────────┴─────────────────────────────────────────────────────────────────────────────────────┴────────┴──────────────────────┘
```

Is it possible to have additional columns like the expected_output and actual_output?

@baniasbaabe not atm, we're happy to accept PRs however, would really appreciate one

@penguine-ip Hi, I have submitted a PR for this. It would be great to receive your feedback
Repository: confident-ai/deepeval
Issue: AccessDeniedException with AmazonBedrockModel in async evaluation context
Root cause: Interface Contract Mismatch

## Bug Report

### Description
The `AmazonBedrockModel` class fails with `AccessDeniedException` when used in DeepEval's async evaluation context, despite having valid AWS credentials and successful direct Bedrock API access.

### Environment
- **DeepEval Version**: 3.3.5 and 3.3.6 (tested both)
- **Python Version**: 3.11.11
- **AWS Region**: eu-west-1
- **Model**: anthropic.claude-3-haiku-20240307-v1:0

### Steps to Reproduce
1. Configure AWS credentials in `.env` file with valid `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
2. Set up DeepEval evaluation with `AmazonBedrockModel`:
   ```python
   model_instance = AmazonBedrockModel(
       model_id="anthropic.claude-3-haiku-20240307-v1:0",
       aws_access_key_id=aws_access_key,
       aws_secret_access_key=aws_secret_key,
       region_name="eu-west-1"
   )
   ```
3. Run evaluation using `evaluate(test_cases=test_cases, metrics=metrics)`

### Expected Behavior
Evaluation should proceed successfully using the Anthropic Claude model via AWS Bedrock.

### Actual Behavior
botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the Converse operation: You don't have access to the model with the specified model ID.

### Additional Context
- ✅ **Direct boto3 access works**: Same credentials work perfectly with direct `boto3.client('bedrock-runtime')` calls
- ✅ **Model access confirmed**: The model is accessible and working via direct API calls
- ✅ **Credentials are loaded**: Environment variables are properly set and accessible
- ❌ **Only fails in DeepEval**: The error only occurs when using `AmazonBedrockModel` within DeepEval's async evaluation context
- ✅ **OpenAI models work**: Switching to `gpt-4o-mini` works perfectly with the same evaluation setup

### Workaround
Currently using OpenAI models as a workaround, but would prefer to use AWS Bedrock models.

### Possible Root Cause
The issue appears to be related to how AWS credentials are handled in the async evaluation context, possibly due to environment variable inheritance or session management in concurrent/async operations.
Repository: confident-ai/deepeval
Issue: Number of goldens
Root cause: Algorithmic Error

Synthesizer don't make the correct number of goldens

I wish to make an script to create 1 golden per document as input

code:

#deepeval set-ollama <model_name>
list_pdf = ["/home/mateus/Área de trabalho/papers/dataset/2012/2012 - Artigos/2012.Art.AnIntegratedApproach.pdf",
            "/home/mateus/Área de trabalho/papers/dataset/2012/2012 - Artigos/2012.Art.ArcheangranitoidMagmatism.pdf"]

#context_config = ContextConstructionConfig(model=model)
from deepeval.synthesizer.config import ContextConstructionConfig


styling_config = StylingConfig(
  input_format="Perguntas em Português Brasileiro para cada artigo.",
  expected_output_format="Perguntas sobre o artigo informado, que devem estar contidas no texto",
  task="Elaborar pergunats sobre o artigo, que devem estar contidas diretamente no artigo, evitando perguntas do tipo \"Sobre oque eh este artigo\"",
    )
context_config = ContextConstructionConfig(
    chunk_size=1024, 
    chunk_overlap=50,
    max_contexts_per_document=1,
    min_contexts_per_document=1,
    max_context_length=1,
    max_retries=1
)
synthesizer = Synthesizer(
    model="gpt-4o",
    styling_config=styling_config)
goldens = synthesizer.generate_goldens_from_docs(
    document_paths=list_pdf,
    include_expected_output=True,
    context_construction_config=context_config,
    max_goldens_per_context=1
)

synthesizer.save_as(
    file_type='json',
    directory="./synthetic_data",
    file_name="my_dataset"
)

The json outputed 6 goldens from only one pdf

@mateusrobr Are you saying that the synthesizer outputted 6 goldens from only one pdf after passing in all 6 documents, also 6 goldens per document, despite these settings? How long are each of the files? 

@mateusrobr i believe it is solved now in the latest commit, will be out in release v soon
Repository: confident-ai/deepeval
Issue: Duplicated data generated during synthetic data generation
Root cause: Algorithmic Error

**Describe the bug**
`synthesizer.generate_goldens_from_docs()` generated much duplicated data. With the default config (`max_goldens_per_context=2` and `max_contexts_per_document=3`), I got 24 samples where only 6 of them are unique.


**To Reproduce**
```python
from deepeval.synthesizer import Synthesizer
from dotenv import load_dotenv

load_dotenv()

synthesizer = Synthesizer()
synthesizer.generate_goldens_from_docs(
    document_paths=["data/vol1.txt"],
    include_expected_output=True,
)

df = synthesizer.to_pandas()
print("The number of generated goldens is:", len(df))
df.drop_duplicates(subset=["input", "expected_output"], inplace=True)
print("The number of unique goldens is:", len(df))
```
The text used is [here](https://drive.google.com/file/d/1b34nDwyI2eNFxsQux7BZ0fSSJyoPmTXD/view?usp=sharing). It will print
```shell
The number of generated goldens is: 24
The number of unique goldens is: 6
```

**Expected behavior**
There should be only 6 unique data generated without any duplicated ones.


I found this issue seems the same as #1923.


Hey @ObsisMc the team's pretty short on time rn but i think its also an important fix - can you please make a PR on this? Thank you in advance


> Hey [@ObsisMc](https://github.com/ObsisMc) the team's pretty short on time rn but i think its also an important fix - can you please make a PR on this? Thank you in advance

Sure, I am fixing it and will make a PR later.
Repository: confident-ai/deepeval
Issue: Clarify GEval scoring scale and normalization in documentation
Root cause: Documentation Deficiency

The current behavior of GEval scoring is unclear:

- The score that the user sees ranges from **0 to 1**.  
- Internally (*hidden from the user and absent from the documentation*), the evaluator produces scores on a **0–10 scale**.

As a result, prompts or criteria that suggest a score on the 0–1 scale (as one would naturally expect from the output) produce values that are unexpectedly low.  

#### Example

```python
answer_accuracy_metric = GEval(
    name="answer quality",
    criteria="If mostly correct → score 0.9",
    evaluation_steps=[
        "Check main fact",
        "If correct → assign 0.9",
    ],
    model="gpt-4o",
)
```

In this case, a mostly correct answer will be evaluated to 0.9 by `GEval._a_evaluate`,
but then divided by 10 before being returned to the user → final output = 0.09.

### Proposed fixes

I see several possible approaches:

1. **Update documentation and inline code comments**  
   Explicitly state:  
   “GEval operates on a 0–10 scale internally, then normalizes to 0–1 before returning the result.”  
   Add a working code example to prevent misinterpretation.  
   **Pros:** Fast. No backward compatibility issues.  
   **Cons:** Users still need to think in two different scales (instructions vs. output).  

2. **Change the internal function**  
   Make GEval operate consistently in the 0–1 range internally and externally.  
   **Pros:** Much clearer and coherent.  
   **Cons:** Breaks backward compatibility for existing users.  

3. **Add a new optional intelligent detector**  
   GEval could automatically detect whether instructions are written on a 0–1 or 0–10 scale.  
   For example, if criteria include values ≤ 1.0, assume the user meant the 0–1 scale and internally multiply by 10.  
   This would make GEval more user-friendly without breaking old code.  

As a workaround, I found that giving examples in **percentages** works (at least with OpenAI GPT-4o):

```
answer_accuracy_metric = GEval(
    name="answer quality",
    criteria="If mostly correct → score 90%",
    evaluation_steps=[
        "Check main fact",
        "If correct → assign 90%",
    ],
    model="gpt-4o",
)
```

Hey @bgbg you can add a rubric :) https://deepeval.com/docs/metrics-llm-evals#rubric
Repository: confident-ai/deepeval
Issue: Tool correctness - division by 0
Root cause: Validation Gap

# How to reproduce
* Number of expected tools = 0
* `should_consider_ordering=True`

# Reference to code
https://github.com/confident-ai/deepeval/blob/ac36027a55b0404f900145c89d0bddd809281bf0/deepeval/metrics/tool_correctness/tool_correctness.py#L151

# How to solve
If both number of expected and actual tool calls is 0 -> return 1. Otherwise 0.

@jankrepl solved!
Repository: confident-ai/deepeval
Issue: Make benchmark results pydantic models instead of raw dicts
Root cause: Unimplemented Feature Gap

**Describe the solution you'd like**
Currently every benchmark returns a raw dictionary with arbitrary keys which makes it so there is very little uniformity across benchmarks. Ideally the result would be a pydantic model so that the user knows what to expect in the response. They don't necessarily need to be the same model across benchmarks, but just having a model in general would significantly improve type safety for the project.

It could also be a new API, such as `evaluate_typed` to maintain backwards compatability

**Describe alternatives you've considered**

The alternate way to get around this is to just make local pydantic models for classes that wrap DeepEval benchmarks, but this is a bit cumbersome, especially if the DeepEval benchmark ever changes

Happy to help implement this if you agree that it is beneficial.


@trevor-inflection Thanks for the suggestion and the offer to do a PR! Please open one and we will merge it :)
Repository: confident-ai/deepeval
Issue: LengthFinishReasonError AzureOpenAI API
Root cause: Validation Gap

When evaluating multiple testcases, the error "LengthFinishReasonError" arise. The model that I used was GPT-4o in AzureOpenAI API. I tried to set `max_tokens` in the `deepeval.models.AzureOpenAIModel` but it errors and says `unexpected keyword "max_tokens"`. Is there any work around to solve this issue?

<img width="2938" height="264" alt="Image" src="https://github.com/user-attachments/assets/8741b5fd-05c4-47ed-a610-cc2f399dce1e" />

Thanks in advance.

@trapoom555 Hey thanks for opening this - i was wondering is there a max tokens parameter in the AzureOpenAI SDK? Because we just pass in the kwargs directly

Please review and merge this #1969
Repository: confident-ai/deepeval
Issue: Bug when following documentation
Root cause: Algorithmic Error

**Describe the bug**
Trying to run the code present here 
https://docs.confident-ai.com/docs/quickstart/



**To Reproduce**
Steps to reproduce the behavior:
1. Using spyder or Vs code run this snippet 
from deepeval.dataset import create_evaluation_query_answer_pairs

dataset = create_evaluation_query_answer_pairs(
    openai_api_key="open AI key here",
    context="FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.7+ based on standard Python type hints.",
    n=13,
)


dataset.review()

It gives the error 
Traceback (most recent call last):
  File "/Users/george.polychronopoulos/Downloads/BidBot/utils/dpev.py", line 3, in <module>
    dataset = create_evaluation_query_answer_pairs(
  File "/Users/george.polychronopoulos/anaconda3/lib/python3.10/site-packages/deepeval/dataset.py", line 583, in create_evaluation_query_answer_pairs
    query=response["query"],
TypeError: string indices must be integers


**Expected behavior**
Trying to run the code present here:
It should load a dataset

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
MACos latest

![error](https://github.com/confident-ai/deepeval/assets/105718075/f4063c10-6a7b-4f61-a7ab-2ba0a5622ea3)


This should be fixed in the latest release - let me know if there are any issues there
Repository: confident-ai/deepeval
Issue: Deepeval is pinned to Otel version, causing issues when importing
Root cause: External Dependency Breakage

Deepeval is pinning Otel dependencies that in turn bound profobuf to < 6.0:
> opentelemetry-exporter-otlp-proto-grpc = "^1.24.0"

This is causing issues on my project that requires higher version of protobuf (version >= 6.0.0). I would recommend to loosen the dependency requirements so that it can be adopted easily.

You can test this by adding protobuf dependency = "^6.0.0"

@chinling hey what version of otel would you need?
Repository: confident-ai/deepeval
Issue: In your test case:  User input: Lithium battery density Agent response: Lei Qiang Context: Lithium battery density Why did Faithfulness score 1.000:
Root cause: Documentation Deficiency

In your test case:

User input: Lithium battery density
Agent response: Lei Qiang
Context: Lithium battery density
Why did Faithfulness score 1.000:

Hey, Faithfullness metric socre is 1 because it does not see any contradiction between your `actual_output` and `retrieval_context`. Refer: https://deepeval.com/docs/metrics-faithfulness#how-is-it-calculated
Repository: confident-ai/deepeval
Issue: Add `evaluation_template ` to GEval metric
Root cause: Unimplemented Feature Gap

**Description**
DeepEval is a widely appreciated LLM evaluation framework, loved by many for its versatility and ease of use. Among its built-in metrics, GEval is one of the most commonly utilized due to its effectiveness in evaluating model performance. However, GEval currently lacks support for an `evaluation_template` parameter, which limits its flexibility in certain use cases.

**Proposed Feature**
Introduce an `evaluation_template` parameter to the GEval metric. This parameter would allow users to customize the evaluation process by specifying a template for the evaluation prompts or responses. This addition would enhance the adaptability of GEval, making it more suitable for diverse evaluation scenarios.

**Benefits**
1. **Improved Customization**: Users can tailor the evaluation process to better fit their specific requirements.
2. **Enhanced Flexibility**: The `evaluation_template` parameter would make GEval more versatile for different tasks and datasets.
3. **Broader Applicability**: By supporting templates, GEval could be used in more complex evaluation setups, such as multi-turn conversations or domain-specific tasks.

hy @MagicCube releasing a new version right now to support this :)

@penguine-ip can we add this in ConversationalGEval as well?

Hey @kumar-hardik, we added this to `ConversationalGEval` as well :)