zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: A metrics class containing different definitions of metrics ordered by some taxonomy. 
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

So right now, deepeval package has different types of checks. Example:

- FactualConsistency check
- conceptual similarity 
- RAG check (using ragas)

However, most of these checks are/will be using some form of atomic or granular metrics (for example: accuracy, f1, etc). Hence, we need to have some function or set of functions that define those metrics at a granular level. So, if the user wants to make a CustomMetric (which requires some statistical metrics), they can use it from deepeval itself. So, in that way we can have more modularity and hence more extensible the package will be. 

**Describe the solution you'd like**

The solution that I have in my mind, is to have a separate class called `MetricsCalculation` or something that will contain all the set of metrics. But, as far as I have seen in NLP, Metrics can be broadly categorized into two forms

1. Metrics that are statistical in nature (which uses some kind of mathematical formula to calculate some score). We can term it as `StatisticalMetrics`. Example includes:
  
       a. BLEU score 
       b. Rouge score 
       c. Exact match score, etc.

2. Metrics which uses an external ML/DL model to calculate the score. We can term it as `ModelBasedMetrics`. Example includes

       a. BERT score 
       b. PII score 
       c. Toxicity score, etc.

The taxonomy of metrics varies greatly, however, personally in my opinion, this simple classification can be a great point of start (in terms of implementation). 

Once implemented, these metrics can be readily used in other implementations. For example: `SummarizationMetrics` as mentioned in issue #179 

@penguine-ip hey, can you assign all the above Metric Request issues to me. 

Thanks

@Anindyadeep all done, let me know if i missed any

Awesome, thanks @penguine-ip :)
Repository: confident-ai/deepeval
Issue: Deadlock in Synthesizer.a_generate_goldens_from_contexts when max_concurrent < number of contexts
Root cause: Resource Mishandling

#### Problem
When generating goldens from contexts asynchronously, there is a deadlock if `max_concurrent` < number of contexts.

#### Cause
The same semaphore gets used for starting the golden creation per context and processing the inputs (evolving, styling). Once the synthetic inputs are produced, processing the inputs is also handled asynchronously with the same semaphore. If the semaphore's capacity is exhausted, the inner tasks never acquire it, leading to a deadlock.

Outer task in `Synthesizer.a_generate_goldens_from_contexts`:
```python
tasks = [
    self.task_wrapper(
        semaphore,
        self._a_generate_from_context,
        semaphore=semaphore,
        context=context,
        goldens=goldens,
        include_expected_output=include_expected_output,
        max_goldens_per_context=max_goldens_per_context,
        source_files=source_files,
        context_index=index,
        progress=progress,
        pbar_id=pbar_id,
        context_scores=_context_scores,
    )
    for index, context in enumerate(contexts)
]
await asyncio.gather(*tasks)
```

Inner task in `Synthesizer._a_generate_from_context`:
```python
# Process all inputs in parallel using asyncio.gather
tasks = [
    self.task_wrapper(semaphore, process_input, index, data, progress)
    for index, data in enumerate(qualified_synthetic_inputs)
]
results = await asyncio.gather(*tasks)
```

#### Reproduction Steps
1. Instantiate a `Synthesizer` with `async_mode=True` and a small `max_concurrent` (1 or 2).
2. Prepare a list of contexts where `len(contexts) > max_concurrent`.
3. Call `Synthesizer.a_generate_goldens_from_contexts`.
4. Observe the process generate inputs for all contexts, but begin to hang when trying to evolve/style them.

#### Expected Behavior
`max_concurrent` should limit concurrency on generating per context without causing a deadlock.

@DarrenBlackwell Thank you, I will investigate and let you know when a fix is in review.

@DarrenBlackwell linked PR is in review. A fix is on its way. Thanks again for raising this issue and for providing context.
Repository: confident-ai/deepeval
Issue: Answer Relevancy metric returns 1.0 even if `actual_output` is empty
Root cause: Validation Gap


**Describe the bug**
Answer Relevancy metric returns 1.0 even if `actual_output` is empty

**To Reproduce**
Steps to reproduce the behavior:
```
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

answer_relevancy = AnswerRelevancyMetric(model=MODEL)

test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="" # an empty output
)

answer_relevancy.measure(test_case)
print(f"Score:{answer_relevancy.score}\nReason: {answer_relevancy.reason}")
```
Score:1
Reason: The score is 1.00 because the response is fully relevant and directly addresses the input without any irrelevant statements. Great job!

**Expected behavior**
For an empty output, the expected behavior would be a score of 0.0 or close to it, as the output provides no relevant information to address the input.


@GeisaFaustino thank you for reporting this. I will investigate and will reach out if I have any questions.

@GeisaFaustino The fix is in review. Thanks again for reporting this and providing repro steps, I really appreciate it.
Repository: confident-ai/deepeval
Issue: Clarification on the `context` argument for `LLMTestCase`
Root cause: Documentation Deficiency

Hi team,
Thanks for the framework, really useful!

While going through some pre-built evals I found this `context` optional parameter
The [official documentation](https://deepeval.com/docs/evaluation-test-cases#context) doesn't give you a very good explanation in terms of agent evaluation, just a blanket example of the chicken joke
What is its intended use?

Get @arghhjayy thanks for asking. Context is basically a way for you to provide ground truth. This is different from expected output because it is purely factual - a page out of a textbook for example, instead of a full-fledged output for a given input/task.

@penguine-ip thanks for the quick response!
Follow up: If I'm evaluating a document question-answer RAG based bot, how would the context look like?
Say I have a doc related to oats in vector db, I get a query `How many calories in 100gms`
How do I define context here?
Repository: confident-ai/deepeval
Issue: Breaking change in v3.7.6: AmazonBedrockModel missing `region_name` alias
Root cause: Validation Gap

## Description

Version 3.7.6 introduced breaking changes to `AmazonBedrockModel` that are not fully backwards compatible.

## Breaking Changes in v3.7.6

The `__init__` parameters were renamed:
- `model_id` → `model`
- `region_name` → `region`
- `input_token_cost` → `cost_per_input_token`
- `output_token_cost` → `cost_per_output_token`

An alias map was added for backwards compatibility:
```python
_ALIAS_MAP = {
    "model": ["model_id"],
    "cost_per_input_token": ["input_token_cost"],
    "cost_per_output_token": ["output_token_cost"],
}
```

**However, `region_name` is missing from the alias map**, causing code that worked in v3.7.5 to fail in v3.7.6.

## Error Message

```
deepeval.errors.DeepEvalError: AmazonBedrockModel is missing a required parameter. Set AWS_BEDROCK_REGION in your environment or pass region.
```

## Reproduction

Code that works in v3.7.5:
```python
from deepeval.models.llms.amazon_bedrock_model import AmazonBedrockModel

model = AmazonBedrockModel(
    model_id='anthropic.claude-3-sonnet-20240229-v1:0',
    region_name='us-east-1',
)
```

This fails in v3.7.6 because `region_name` goes into `**kwargs` instead of being mapped to `region`.

## Additional Issue: Pricing Validation

v3.7.6 also added strict pricing validation via `require_costs()`. In v3.7.5, costs defaulted to `0`. In v3.7.6, if a model isn't in `BEDROCK_MODELS_DATA` with pricing info, users must explicitly provide `cost_per_input_token` and `cost_per_output_token`.

Many Bedrock models in `BEDROCK_MODELS_DATA` have `input_price=None` and `output_price=None`, which triggers:
```
No pricing available for `<model>`. Please provide both `cost_per_input_token` and `cost_per_output_token`...
```

## Suggested Fix

Add `region_name` to the alias map:
```python
_ALIAS_MAP = {
    "model": ["model_id"],
    "region": ["region_name"],  # <-- Add this
    "cost_per_input_token": ["input_token_cost"],
    "cost_per_output_token": ["output_token_cost"],
}
```

## Environment

- deepeval version: 3.7.6
- Python version: 3.11
- OS: macOS

## Notes

- v3.7.6 was published to PyPI on 2025-12-17 but has no GitHub Release entry (only a tag exists)
- This makes it difficult to track what changed between versions

My main concern is that PyPI has the newer version but this repo doesn't list it in releases.

Hey @eyal-brami, thanks for posting the issue, I've added the "region" in the `ALIAS_MAP` in a PR of mine like you suggested. We're also adding a new changelog page in deepeval docs soon, so that should help with any future releases.
Repository: confident-ai/deepeval
Issue: Feat request: optional user-defined prompt builder for synthesizer
Root cause: Unimplemented Feature Gap

It could be useful to give users flexibility over the user prompt used to generate the goldens in the synthesizer.

`generate_goldens_from_contexts` and `a_generate_goldens_from_contexts` may be overridden when inheriting from the Synthesizer class and rely on predefined prompt templates composed from specific arguments `context, max_goldens_per_context, scenario, task, input_format`

https://github.com/confident-ai/deepeval/blob/a740ca31d6c32d130e93fc1fe0c90d4795a93dc0/deepeval/synthesizer/synthesizer.py#L661-L667

which are passed into the [synthetic inputs](https://github.com/confident-ai/deepeval/blob/a740ca31d6c32d130e93fc1fe0c90d4795a93dc0/deepeval/synthesizer/templates/template.py#L109C9-L170). Does it make sense to add an optional prompt builder argument and allow the user to construct the prompt for golden generation?

```python
prompt_builder: Optional[Callable[..., str | Prompt]]
```
Repository: confident-ai/deepeval
Issue: Turn Faithfulness metric is not available
Root cause: Environment Incompatibility

**Describe the bug**
Getting the error message "cannot import name 'TurnFaithfulnessMetric' from 'deepeval.metrics'" when importing `TurnFaithfulnessMetric`.

**To Reproduce**
````python
import deepeval
print(f"deepeval version: {deepeval.__version__}")

from deepeval.metrics import TurnFaithfulnessMetric
````
Output:
````
deepeval version: 3.7.0

ImportError: cannot import name 'TurnFaithfulnessMetric' from 'deepeval.metrics'
````

**Expected behavior**
No ImportError message as I am following the example from documentation - [Turn Faithfulness](https://deepeval.com/docs/metrics-turn-faithfulness#usage)

The issue is resolved by updating DeepEval to version 3.7.7.

A recommendation for documentation improvement is to specify the DeepEval version from which certain metrics became available.
Repository: confident-ai/deepeval
Issue: Bug: Batch Predict on DROP Benchmark
Root cause: Algorithmic Error

**Describe the bug**
`batch_predict` on DROP benchmark expects exact match with a list of possible answers, rather than marking correct if the prediction matches just one of the possible answers.

**To Reproduce**
Run DROP benchmark with batch_size > 1
Will have a score much lower than expected, and much lower than having no batch_size

**Expected behavior**
Mark correct if prediction is in the list of answers
Repository: confident-ai/deepeval
Issue: Bug: Batch predict in MMLU requires List[List[BaseModel]] rather than List[BaseModel]
Root cause: Interface Contract Mismatch

**Describe the bug**
Batch predict in MMLU requires List[List[BaseModel]] rather than List[BaseModel]

**To Reproduce**
Steps to reproduce the behavior:
Bug in code, have fixed in a PR. To reproduce, run MMLU code using customer adapter that returns List[BaseModel] in batch generation

**Expected behavior**
List[BaseModel] should work (as in type hint)
Repository: confident-ai/deepeval
Issue: VertexAI default application credentials support broken in 3.7.x
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
Google application default credentials support appears to be broken since 3.7.x. (as described in this doc https://deepeval.com/integrations/models/vertex-ai)
It results in the following error `Google Gemini API key is not configured. Set GOOGLE_SERVICE_ACCOUNT_KEY in your environment or pass `service_account_key` to GeminiModel(...)`

It works on 3.6.x



**To Reproduce**
Steps to reproduce the behavior:
Follow the steps here: https://deepeval.com/integrations/models/vertex-ai

**Expected behavior**
Should work


I am facing this issue, as well. [I have started a thread in the Deepeval discord](https://discord.com/channels/1118193057620316202/1457844761464672267) which now links here.

Thanks for the report @Minro4

I have confirmed: this is a regression. Our Vertex AI docs rely on Application Default Credentials (ADC) via `gcloud auth application-default login`, but in 3.7.6+ we’re incorrectly requiring a service account key (`GOOGLE_SERVICE_ACCOUNT_KEY`) instead of allowing ADC fallback.

Workarounds:
- If you want ADC: pin to < 3.7.4 for now.
- If you want to stay on latest, provide a service account key via `GOOGLE_SERVICE_ACCOUNT_KEY` or `deepeval set-gemini --service-account-file ...`.

In the meantime, I will create a PR to fix this issue.
Repository: confident-ai/deepeval
Issue: Generate goldens from documents successfully runs but returns 0 goldens
Root cause: Algorithmic Error

**Describe the bug**
The methods to generate synthetic single or multi-turn goldens from documents successfully run but return zero goldens.

The PDF document used is a PDF version from https://www.confident-ai.com/blog/why-llm-as-a-judge-is-the-best-llm-evaluation-method. I also tested with other pdf files and text files, and got the same result.

**To Reproduce**
````python
import deepeval
print(f"Deepeval version: {deepeval.__version__}")

from deepeval.synthesizer import Synthesizer
from deepeval.synthesizer.config import ContextConstructionConfig
from deepeval.models import AzureOpenAIModel, AzureOpenAIEmbeddingModel

aoai_model = AzureOpenAIModel(
     model=os.environ['AZURE_MODEL_NAME'],
     deployment_name=os.environ['AZURE_OPENAI_DEPLOYMENT'],
     api_key=os.environ['AZURE_OPENAI_API_KEY'],
     api_version=os.environ['AZURE_OPENAI_API_VERSION'],
     base_url=os.environ['AZURE_OPENAI_ENDPOINT'],
     temperature=0
)

aoai_emb_model = AzureOpenAIEmbeddingModel(
     model=os.environ['AZURE_EMB_MODEL_NAME'],
     deployment_name=os.environ['AZURE_OPENAI_EMB_DEPLOYMENT'],
     api_key=os.environ['AZURE_OPENAI_API_KEY'],
     api_version=os.environ['AZURE_OPENAI_EMB_API_VERSION'],
     base_url=os.environ['AZURE_OPENAI_ENDPOINT']
)
print(f"Azure OpenAI Model: {aoai_model.get_model_name()}")
print(f"Azure OpenAI Embedding Model: {aoai_emb_model.get_model_name()}")


doc="LLM-as-a-Judge Simply Explained: The Complete Guide to Run LLM Evals at Scale - Confident AI.pdf"

synthesizer = Synthesizer(model=aoai_model)

single_turn_goldens = synthesizer.generate_goldens_from_docs(
     document_paths=[doc],
     context_construction_config=ContextConstructionConfig(critic_model=aoai_model, 
                                                           embedder=aoai_emb_model)

)
print(f"Generated {len(single_turn_goldens)} single-turn goldens.")

multi_turn_goldens = synthesizer.generate_conversational_goldens_from_docs(
     document_paths=[doc],
     context_construction_config=ContextConstructionConfig(critic_model=aoai_model, 
                                                           embedder=aoai_emb_model)

)
print(f"Generated {len(multi_turn_goldens)} multi-turn goldens.")

````

Output:
````
Deepeval version: 3.7.8
Azure OpenAI Model: gpt-4o (Azure)
Azure OpenAI Embedding Model: text-embedding-3-small (Azure)

[Confident AI Synthesizer Log] SUCCESS: Successfully deleted: 
/var/folders/n6/qvgyq3rj4qb4s0714k601b700000gn/T/deepeval_chroma_3r9buvkn
[Confident AI Synthesizer Log] SUCCESS: Context Construction: Utilizing 0 out of 0 chunks.
Generated 0 single-turn goldens.

[Confident AI Synthesizer Log] SUCCESS: Successfully deleted: 
/var/folders/n6/qvgyq3rj4qb4s0714k601b700000gn/T/deepeval_chroma_h03j92a2
[Confident AI Synthesizer Log] SUCCESS: Context Construction: Utilizing 0 out of 0 chunks.
Generated 0 multi-turn goldens.
````

**Expected behavior**
 Generated test cases (goldens) or an error message.
Repository: confident-ai/deepeval
Issue: Metric Request: quasi exact match
Root cause: Unimplemented Feature Gap

This issue is an extension of issue https://github.com/confident-ai/deepeval/issues/238. Add a new metric called Qusasi Exact Match. 

It refers to a similarity measurement or matching technique that identifies approximate or near matches between text sequences. It is used when you want to find text that is similar but not necessarily identical to a given query or reference text. Quasi-exact match techniques are especially useful in applications like search engines, information retrieval, and text analysis, where finding exact matches may not always be practical or sufficient.

I am closing this issue for now @penguine-ip, as this is addressed on PR #260
Repository: confident-ai/deepeval
Issue: Metric Request: Bert score metrics
Root cause: Unimplemented Feature Gap

By definition, we can define BERT score as:

> BERT Score assesses the quality of generated text by comparing it to a reference or a ground truth text. It considers the semantic similarity between the generated text and the reference text. BERT Score leverages BERT's contextual embeddings to capture the meaning and context of words and phrases in the text, which makes it more effective than traditional metrics like BLEU or ROUGE, which rely on n-grams and may not capture the nuances of language.

It will come under the ModelBasedMetric Category as defined on issue #238 

I am closing this since, this is merged.
Repository: confident-ai/deepeval
Issue: Metric Request: PII Score 
Root cause: Unimplemented Feature Gap

PII or Personal Identifiable Information is a very important factor in asses the Generation quality of LLMs. PII can include the name of the Person, Credit Card Number etc, etc. 

So a set of tokens (let's take word tokens for now), the PII score can be defined as the number of PII's detected by the total number of tokens. 

Since, detecting PII would require NER, so it would fall under `ModelBasedMetrics` as defined on issue #238 

@penguine-ip , @jaywyawhare wants to work on this issue :)

Hey @jaywyawhare, thrilled to have you working on this! I'll assign it to you and please feel free to ask for anything in our discord: https://discord.com/invite/a3K9c8GRGt

@penguine-ip Thanks for assigning this to me, I'm excited to dive into it. If I need any clarification or additional information along the way, I'll be sure to reach out on our Discord channel. Let's get started!

Likewise, happy to be working with you

Hey @jaywyawhare any updates?
Repository: confident-ai/deepeval
Issue: Metric Request: FaithFulness Score 
Root cause: Unimplemented Feature Gap

"faithfulness score" is a metric used to evaluate the faithfulness of generated text. Faithfulness refers to the extent to which the generated text accurately represents the information from the source or reference text, without introducing false or misleading information, omissions, or distortions

This would fall under `ModelBasedMetrics` as discussed on issue #238.
Repository: confident-ai/deepeval
Issue: Metric Request: Toxic Score 
Root cause: Unimplemented Feature Gap

Toxicity is a very important parameter to assess the generation quality of LLMs. Toxicity Score can be defined as the total number of toxic tokens generated out of a total number of tokens in a sequence. 

I am still not sure, whether this should fall under `ModelBasedMetric` or `StatisticalMetric` as discussed under issue #238.
Repository: confident-ai/deepeval
Issue: Add JudgeLM as a way to evaluate and compare historical test runs
Root cause: Unimplemented Feature Gap

Currently, metrics are computed based on test cases that run during evaluation. However, there's currently no way to compare historical test runs' performances except for comparing metric scores for each test run. 

I would like to introduce another way to choose the best configurations / hyperparameters to help devs speed up development using DeepEval, and that would be to implement a JudgeLM (eg. https://github.com/baaivision/JudgeLM) to choose the best test run out of a set of test runs.

```
from deepeval import judge

best_performing_test_run = None
for test_run in test_runs:
      best_performing_test_run = judge(test_run, best_performing_test_run)
```

This assumes the test runs were ran on the same test cases / evaluation dataset. There should be error handling for edge cases that don't fit into this criteria.

Do you want to an LLM to judge another LLM here? Like can you more elaborate the issue. May be I need to go through the paper to understand this more. But yes, this can be very useful when pre-training or fine-tuning models and when we are in the middle of Training. 

So JudgeLM will be used to pick the best historical test run out of a set of a test runs, but it would be difficult for this to be a feature purely for the package

Yes, that could be.
Repository: confident-ai/deepeval
Issue: Move all the existing scores into Scoring
Root cause: Architectural Constraint

The initial implementation of DeepEval have metrics like FactualScore, or RAGAS score etc. However, the scoring logics are wrapped under `BaseMetric` class. However, from this issue #238, the scoring function should be granular enough so that they can be used everywhere else in the code and even outside the deepeval library code. 

It also falls under issue #238
Repository: confident-ai/deepeval
Issue: Would it be possible to use opt-in schema for telemetry instead of opt-out?
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

The library tries to connect to the Sentry endpoint, also sentry_sdk seems to do some instrumentation of code.

**Describe the solution you'd like**
Instead of opt-out for telemetry, please default to opt-in. This will allow using your awesome library in more restrictive environments.

**Describe alternatives you've considered**
N/A

**Additional context**
N/A


@chaliy Got it! I'll cut a release immediately for tonight, appreciate the comment, will let you know immediately once it's out so you can pip install -U deepeval

@chaliy Just cut a release that is opt out by default, also we have a discord community that I'm trying to grow, if you've discord please drop in and I'd love to chat there! https://discord.com/invite/a3K9c8GRGt