zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Doc Improvement
Root cause: Documentation Deficiency

Maybe add the "expected_output" field to the first test case definition in the quickstart guide: https://docs.confident-ai.com/docs/getting-started#create-your-first-test-case

This would prevent errors when the user later on tries to test custom metrics example code that relies on this field and reuses the first test case instead of copying the test case without text from the custom metrics example.

@SamyAteia I didn't want to deter users and make them feel like LLM evaluation is not for them because they don't have a labelled dataset.

True, another solution would be to  just fill in the example for the custom metric so users wouldn't reuse the one from the steps before.

@SamyAteia We try to make it as easy as possible - if a user reuses the test case and uses a metric that requires the expected output the error message will tell you you'll need the expected output for the test case: https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/utils.py#L177

I believe this function is not used when following the quickstart to implement the custom rouge metric.
Repository: confident-ai/deepeval
Issue: Avoid unnecessary opentelemetry imports
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
Framework imports `opentelemetry` dependencies even when `telemetry_opt_out` is evaluated to `True`. It causes some conflicts with other libraries used on my project.

**Describe the solution you'd like**
Import `opentelemetry` only after validation that telemetry should be sent.

**Describe alternatives you've considered**
N/A

**Additional context**
Problems are related to protobuf versions that are used in opentelemetry.
```TypeError: Couldn't build proto file into descriptor pool: duplicate file name opentelemetry/proto/common/v1/common.proto```
Repository: confident-ai/deepeval
Issue: CLI improvement: option to display only failed tests
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

When using DeepEval in a Git pre-push hook, the output includes all test results. This extensive output makes it difficult to quickly identify failed tests, hindering the efficiency of the pre-push process.

**Describe the solution you’d like**

I would like DeepEval to offer an option or flag that allows displaying only failed tests in the output. 

**Describe alternatives you’ve considered**

Develop a custom script using Python API to evaluate LLM outputs, bypassing the DeepEval CLI.

**Additional context**

Implementing this feature would be particularly beneficial for devs who integrate DeepEval into CI pipelines or automated Git hooks, where concise and relevant output is crucial.

I also expected it to be a built-in feature 

I totally agree! This feature would be very useful, and I’m looking forward to seeing it implemented.
Repository: confident-ai/deepeval
Issue: DeepEval, G-Eval logging issues.
Root cause: Documentation Deficiency

Hello, 
I am testing different LLM's and evaluating them with G-eval. 
I want to log the (GEval) Verbose Logs into a .json file but it keeps appearing as None.
It keeps outputting in the terminal, but I want to log them automatically as I am testing with multiple datasets.
Below is an example of me trying to log the terminal outputs but still failed.

`        log_output = io.StringIO()
        with redirect_stdout(log_output):
            test_case = LLMTestCase(
                input=prompt,
                actual_output=model_answer,
                expected_output=reference_answer
            )
            explainability_result = explainability_metric.measure(test_case)

        # Extract the captured logs
        verbose_log = log_output.getvalue()

        if explainability_result is None or explainability_result.score is None:
            explainability_score = 0
            explanation = "Explainability evaluation failed - no valid output returned."
        else:
            explainability_score = explainability_result.score
            explanation = explainability_result.reason
`
Repository: confident-ai/deepeval
Issue: Text to Image metric incorrect evaluation
Root cause: Algorithmic Error

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
In Text to image metric while evaluating Semantic Consistency we should pass the input text prompt and actual output image to the model for evaluation. Instead, we are passing the input text prompt and image from the input if it contains any.

**Expected behavior**
We should pass the actual output image and input text prompt to the model for evaluation.

**Screenshots**
![image](https://github.com/user-attachments/assets/0782f5b3-1e3f-4739-8561-ba6060a379de)


Hey @shanmukh-sai2003, thanks for pointing it out. It should be fixed here: https://github.com/confident-ai/deepeval/pull/1253

thanks for the fix @kritinv. Can I know when I can expect this fix to be released

@shanmukh-sai2003 I believe its already out, can you give it a try? :)
Repository: confident-ai/deepeval
Issue: metric.evaluate() raise  TypeError: unsupported operand type(s) for +=: 'NoneType' and 'float'
Root cause: Validation Gap

**Describe the bug**
TypeError in metric.evaluate() when called directly.

When the evaluate() method of a metric (derived from BaseMetric) is called directly (without first calling measure()), it throws a TypeError: unsupported operand type(s) for +=: 'NoneType' and 'int/float'. This is because the evaluation_cost attribute in the BaseMetric class is initialized to None.

During the evaluation process, the code attempts to increment evaluation_cost (e.g., evaluation_cost += cost). However, since evaluation_cost is None by default, this operation results in the TypeError.

Expected Behavior: evaluation_cost should be initialized to 0 to allow numerical additions.

**To Reproduce**
Steps to reproduce the behavior:
1. Instantiate a metric (e.g., metric = GEval()).
2. Call metric.evaluate(test_case) directly.

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: iOS
 - Browser: Chrome
 - Version [e.g. 22]


**Additional context**
Add any other context about the problem here.
Repository: confident-ai/deepeval
Issue: ImportError: cannot import name 'Scorer' from partially initialized module 'deepeval.scorer' (most likely due to a circular import) (/Users/macbookpro/Documents/Codes/ML/Athena_2_LLM_Evaluation/venv/lib/python3.11/site-packages/deepeval/scorer/__init__.py)
Root cause: External Dependency Breakage

When I ran the code, I got an error.

## Code

```python

from deepeval.scorer import Scorer
from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase

class RougeMetric(BaseMetric):
    def __init__(self, threshold: float = 0.5):
        self.threshold = threshold
        self.scorer = Scorer()

    def measure(self, test_case: LLMTestCase):
        self.score = self.scorer.rouge_score(
            prediction=test_case.actual_output,
            target=test_case.expected_output,
            score_type="rouge1"
        )
        self.success = self.score >= self.threshold
        return self.score

    # Async implementation of measure(). If async version for
    # scoring method does not exist, just reuse the measure method.
    async def a_measure(self, test_case: LLMTestCase):
        return self.measure(test_case)

    def is_successful(self):
        return self.success

    @property
    def __name__(self):
        return "Rouge Metric"
    
    

#####################
### Example Usage ###
#####################
test_case = LLMTestCase(input="abc", actual_output="abc", expected_output="abc")
metric = RougeMetric()

metric.measure(test_case)
print(metric.is_successful())

```

## Error

```bash
from deepeval.scorer import Scorer
ImportError: cannot import name 'Scorer' from partially initialized module 'deepeval.scorer' (most likely due to a circular import) (/Users/macbookpro/Documents/Codes/ML/Athena_2_LLM_Evaluation/venv/lib/python3.11/site-packages/deepeval/scorer/__init__.py)
```


## Configurations

Device: Mac
Python Version: 3.11.0
DeepEval Version: 2.1.2


Thanks



The same problem is found in the DeepEval version in 2.1.6. But in the 2.0.9 version, it works well. please update the documentation.

Thanks

Still getting this circular import issue, how to resolve it ??
Repository: confident-ai/deepeval
Issue: Synthesizer cost
Root cause: Documentation Deficiency

Is there a way that we can output the number/cost of tokens used in generating the goldens?

I have searched the docs, and it seems that the only way the costs are displayed are on the online deepeval UI 

@luarss its not in the docs but i believe if you do `synthesizer.synthesis_cost` after generating goldens you'll see it. Let me know otherwise.
Repository: confident-ai/deepeval
Issue: OpenAI model support
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently a limited set of OpenAI models are supported. When I try something specific, like `gpt-4o-2024-08-06` it returns:

```
ValueError: Invalid model. Available GPT models: gpt-4o-mini, gpt-4o, gpt-4-turbo, gpt-4-turbo-preview, gpt-4-0125-preview, gpt-4-1106-preview, gpt-4, gpt-4-32k, gpt-4-0613, gpt-4-32k-0613, gpt-3.5-turbo-1106, gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-3.5-turbo-0125
```

**Describe the solution you'd like**
Expand support for OpenAI models (i.e. o1, gpt-4o variants etc)

**Describe alternatives you've considered**
We can of course use `gpt-4o`, but that is not good for keeping the models consistent when doing evals.

**Additional context**
One solution could be to get a list of models using OpenAI API. https://platform.openai.com/docs/api-reference/models/list

If this is of use, I don't mind creating a PR. Thanks!


Hey @luarss yes definitely, would love a PR to help with this, appreciate it!
Repository: confident-ai/deepeval
Issue: Remove Python <3.13 version restriction from setup.py
Root cause: Architectural Constraint

Hi! First, thanks for creating and maintaining deepeval. I noticed that the package currently has an upper bound restriction on Python versions (`<3.13`) in setup.py.

Since Python 3.13 is now available, it would be great if this restriction could be removed or updated. This would allow users to use the package with the latest Python version without having to force install or work around version constraints.

Current setup.py configuration:
```python
python_requires=">=3.9, <3.13"
```
I removed such requirements and installed using `pip install -e` and it works without a problem. 

Yes, +1 for removing the Python 3.13 restriction. 

@cpoptic @21m1n Done! In 2.2.0

ALso @cpoptic which GRPC version you're needing?

Thank you so much!

I'm currently running grpcio=1.67.1, deepeval=2.2.0 and python=3.13
Repository: confident-ai/deepeval
Issue: Add base attack in the returned results in redteam scan
Root cause: Unimplemented Feature Gap

The result of redteaming.scan is this for each attack
```
                        result = {
                            "Vulnerability": attack.vulnerability,
                            "Vulnerability Type": vulnerability_type.value,
                            "Attack Enhancement": attack.attack_enhancement,
                            "Risk Category": (
                                risk.value if risk is not None else "Others"
                            ),
                            "Input": attack.input,
                            "Target Output": None,
                            "Score": None,
                            "Reason": None,
                            "Error": None,
                        }
```

**Describe the solution you'd like**
I would like to have another field call 'original_input" so we can see the original (base) attack and verify it.


Hey @AmineDjeghri is it possible you make a PR for this? We would like this but don't have capacity right now

Hey, i will finish customizing some stuff with deepeval then we will talk about everything needed. I already wanted to improve the templates for the missing Graders

@AmineDjeghri got it appreciate it!
Repository: confident-ai/deepeval
Issue: Benchmarks introduction does setup wrong
Root cause: Documentation Deficiency

The [introduction](https://docs.confident-ai.com/docs/benchmarks-introduction) says:
from deepeval.benchmarks.task import MMLUTask

but really, it is 

from deepeval.benchmarks.mmlu.task import MMLUTask


https://docs.confident-ai.com/docs/benchmarks-mmlu

The steps described result in the model generating long outputs, not single answers.
Also, batching doesn't work because MMLU has prompts of different lengths.

Hey @AMindToThink can you please create a quick PR for this?

I made a pull request to change one location with the wrong MMLUTask import. There may be others.

And I am unable to fix the other issues.
Repository: confident-ai/deepeval
Issue: OpenAI Batch API
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
This is closely linked to https://github.com/confident-ai/deepeval/issues/500, which (correct me if i am wrong) should already be implemented, but not via the Batch API. Integrating the batch API will save evaluations by costs up to 50% lower costs [1], [2].

**Describe the solution you'd like**
Ideally have a switch that allows user to select `batch_mode` which is available only if user has selected a valid model that has batch API support (e.g. openai/gemini)

**Describe alternatives you've considered**
n/a

**Additional context**
n/a


[1]: https://platform.openai.com/docs/guides/batch
[2]: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/batch-prediction-gemini

Hey @luarss will try to look at this a bit later, batch mode right now in deepeval is the most best supported since we're not the most familiar with it. If you have any suggestions or PRs, would also appreciate it greatly

We would also be interested in support for OpenAI's batch API. However, due to the user having to pull the results back from the batch API as opposed to getting an ~immediate response, implementation likely requires quite a huge effort. Therefore, I kindly ask the core team to provide some insight to whether you would be even willing to support this in deepeval. 

I guess it would require some deliberate design afterwards before jumping in implementation.
Repository: confident-ai/deepeval
Issue: TypeError: Synthesizer.generate_goldens_from_docs() got an unexpected keyword argument 'chunk_size'
Root cause: Documentation Deficiency

**Describe the bug**
TypeError: Synthesizer.generate_goldens_from_docs() got an unexpected keyword argument 'chunk_size'

**To Reproduce**
Trying to generate Synthetic Test Data using golden with txt file less than 1024 token and change chunk size 

**Expected behavior**
work fine 

**Screenshots**

![Image](https://github.com/user-attachments/assets/e0024bf9-a6cd-4ab8-8e70-d033927191cb)

**Desktop (please complete the following information):**
 - OS: windows
 - Browser  chrome,




https://docs.confident-ai.com/docs/synthesizer-generate-from-docs

```python
from deepeval.synthesizer.config import ContextConstructionConfig

...
synthesizer.generate_goldens_from_docs(
  document_paths=['example.txt', 'example.docx', 'example.pdf'],
  context_construction_config=ContextConstructionConfig()
)
```

Can you try with the context_construction_config constructor instead?

yes ,it works thanks alot
Repository: confident-ai/deepeval
Issue: Issues with running the tutorial https://docs.confident-ai.com/docs/tutorials/evaluating-langchain
Root cause: Documentation Deficiency

The tutorial has a bug https://docs.confident-ai.com/docs/tutorials/evaluating-langchain

`query = "Who is the president?` should be `query = "Who is the president?"`

Another bug (missing comma)

```
qa = RetrievalQA.from_chain_type(
  llm=OpenAI(openai_api_key=openai_api_key), chain_type="stuff"
  retriever=docsearch.as_retriever()
)
```

should be 

```
qa = RetrievalQA.from_chain_type(
  llm=OpenAI(openai_api_key=openai_api_key), chain_type="stuff",
  retriever=docsearch.as_retriever()
)
```

Also missing the requests import

`import requests`

Thanks - Resolved :)
Repository: confident-ai/deepeval
Issue: AttributeError: 'str' object has no attribute 'statements'
Root cause: Interface Contract Mismatch

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
I am trying to run/test the AnswerRelevancyMetric using the exact same code example described on the page below:
https://docs.confident-ai.com/docs/metrics-answer-relevancy

And I am using Azure OpenAI by providing the credentials and running the code below in the terminal of my working directory. All values are within quotation marks.

deepeval set-azure-openai --openai-endpoint="" \
    --openai-api-key="" \
    --deployment-name="" \
    --openai-api-version=""

While running the code below, I am getting this error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[5], line 13
      3 metric = AnswerRelevancyMetric(
      4     threshold=0.7,
      5     include_reason=True,
      6     verbose_mode=True
      7 )
      8 test_case = LLMTestCase(
      9     input="What if these shoes don't fit?",
     10     actual_output=actual_output
     11     )
---> 13 metric.measure(test_case)
     14 print(metric.score)
     15 print(metric.reason)

File ~<Hidden file path>/.venv/lib/python3.11/site-packages/deepeval/metrics/answer_relevancy/answer_relevancy.py:58, in AnswerRelevancyMetric.measure(self, test_case, _show_indicator)
     56 if self.async_mode:
     57     loop = get_or_create_event_loop()
---> 58     loop.run_until_complete(
     59         self.a_measure(test_case, _show_indicator=False)
     60     )
     61 else:
     62     self.statements: List[str] = self._generate_statements(
     63         test_case.actual_output
...
--> 232     return res.statements
    233 else:
    234     try:

AttributeError: 'str' object has no attribute 'statements'
--------------------------------------------------------------------------

**To Reproduce**
from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

actual_output = "We offer a 30-day full refund at no extra cost."

metric = AnswerRelevancyMetric(
    threshold=0.7,
    include_reason=True
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output
)

metric.measure(test_case)
print(metric.score)
print(metric.reason)

**Expected behavior**
A score (float) for answer relevancy and a reason (text).

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS:  MacOS Sequoia
 - Browser: Edge
 - Deepeval: 2.2.3

**Additional context**
Add any other context about the problem here.


@romai can you try 2.2.4?

I upgraded deepeval to the latest version, and it worked. Thanks!
Repository: confident-ai/deepeval
Issue: HallucinationMetric
Root cause: Unimplemented Feature Gap

To check for hallucination, we can perform the following:
- [ ] Grab sources from a Google Search/Query
- [ ] Run Factual Consistency On Top

this is cool - needs an additional API possibly for google search?  If so are there any alternatives for web search?

For this, we can use Serp API (https://serpapi.com/) but unfortunately it's pretty expensive.  
I also potentially think we should be using Wikipedia data dumps to ensure factual correctness as opposed to just Google Search because it's also a better resource than Google Search in trustworthiness.
Repository: confident-ai/deepeval
Issue: Add "logical fallacy" check to model output
Root cause: Unimplemented Feature Gap

I'd added FallacyChain to LangChain last month (checks for logical fallacies) - link below - thinking I could add similar functionality to natural language model output here - thoughts?
https://python.langchain.com/docs/guides/safety/logical_fallacy_chain

This is very interesting - how would one capture the the degree of logical fallacy? and yes! would love a PR about this :) 

It could return a dictionary of logical fallacies but I think better for evaluation purposes a % of logical fallacies (with the denominator being sentence count) but returning a float.
Repository: confident-ai/deepeval
Issue: Need more clarity around retrieval_context vs context
Root cause: Documentation Deficiency

Upon going through the documentation, I found this exact paragraph:

```
The context is an optional parameter that represents additional data received by your LLM application as supplementary sources of golden truth. You can view it as the ideal segment of your knowledge base relevant to a specific input. Context allows your LLM to generate customized outputs that are outside the scope of the data it was trained on.

In RAG applications, contextual information is typically stored in your selected vector database, which is represented by retrieval_context in an LLMTestCase and is not to be confused with context. Conversely, for a fine-tuning use case, this data is usually found in training datasets used to fine-tune your model. Providing the appropriate contextual information when constructing your evaluation dataset is one of the most challenging part of evaluating LLMs, since data in your knowledge base can constantly be changing.
```

However, even after going through the exact examples mentioned in doc [here](https://docs.confident-ai.com/docs/evaluation-test-cases#retrieval-context), it still lacks clarity about the logical difference between context and retrieval_context.

What is clear to me is the fact that when evaluating RAG, we should use retrieval_context. But in cases when both the parameters are used, I am not sure the impact each of these have on the outcome. It would be helpful if the team can shed some more light on the difference, and if possible, with some relevant example.

Thank you! 



from what i understand post reading the documentation is : 
- the logical difference between `context` and retrieval_context` is `context` is used as something like ground truth against which `retrieval_context` is measured w.r.t metrics such as Precision or recall.  
- To create an analogy with classical ml, u can think of `context` as your actual output and `retrieval_context` as your predicted output which is defined during run time.
- and for evaluating RAG, we should use both `retrieval_context` and `context`. `retrieval_context` must be compared against `something` which acts as our true source right ? this something is your `context`


@AbhishekRP2002 you can see the impact of the parameters on each individual metrics on the individual metric pages. It tells you which parameter is used for calculation and you'll quickly learn that supplying a parameter does not mean it is used for evaluation: https://docs.confident-ai.com/docs/metrics-contextual-precision#how-is-it-calculated

"Although extremely similar to the FaithfulnessMetric, the HallucinationMetric is calculated differently since it uses contexts as the source of truth instead. Since contexts is the ideal segment of your knowledge base relevant to a specific input, the degree of hallucination can be measured by the degree of which the contexts is disagreed upon."         
context is real.
-https://deepeval.com/docs/metrics-hallucination