zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Score is not cached
Root cause: Configuration Error

**Describe the bug**
If you evaluate twice, with cache turned on, the result loaded from cache will not have a score.

**To Reproduce**
```python
from deepeval import evaluate
from deepeval.metrics import HallucinationMetric
from deepeval.test_case import LLMTestCase

llm_response = """
Blood pressure is 128/72, heart rate is 79, respiratory rate is 26, and O2 sat is 99%.
Lungs clear bilaterally.
Heart palpitations
    """

context = [
    "Blood pressure: 128/72",
    "Heart rate: 79",
    "Respiratory rate: 26",
    "O2: 99%",
    "Lungs are clearing bilaterally",
    "Periodic headaches",
    "A1C: 10"
]

test_case = LLMTestCase(input="Heart Rate around 79 bpm-ish", actual_output=llm_response, context=context)
results = evaluate([test_case], [HallucinationMetric()], run_async=False, use_cache=False)
print(results[0].metrics[0].score)
results = evaluate([test_case], [HallucinationMetric()], run_async=False, use_cache=True)
print(results[0].metrics[0].score)
```
**Expected behavior**
Prints a floating point number twice.

Prints a floating point number and then None


**Additional context**
Deepeval version 0.21.36
Repository: confident-ai/deepeval
Issue: Knowledge Retention metric discrepancy
Root cause: Documentation Deficiency

**Describe the bug**
1. The docs for [Knowledge Retention Metric](https://docs.confident-ai.com/docs/metrics-knowledge-retention) specify that in strict mode, the threshold is set to `0`. However, in the [code](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/knowledge_retention/knowledge_retention.py#L37) the threshold is set to `1` . What is the expected correct behavior?

```python
class KnowledgeRetentionMetric(BaseConversationalMetric):
    def __init__(
        self,
        threshold: float = 0.5,
: 
: 
self.threshold = 1 if strict_mode else threshold
```

2. According to the [Knowledge Retention template](https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/knowledge_retention/template.py), the verdict is `yes` if LLM remembers knowledge in response, else it is a `no`. It mentions that a higher score is better, which makes sense. However, while calculating the score, it calculates the number of `no` verdicts, implying that it calculates the knowledge forgotten.  This seems incorrect to me. Shouldn't it calculate the number of `yes` verdicts or things remembered? 

**To Reproduce**
None

**Expected behavior**
None

**Screenshots**
None

**Desktop (please complete the following information):**
 Note required.

**Smartphone (please complete the following information):**
 Not required.

**Additional context**

Happy to contribute changes to the library/documentation. 

Hey @AnanyaRaval, thanks for catching the error!  Indeed there is a mistake, the correct version should be 1 for a perfect score, and 0 if not for strict mode. For the docs, the equation at the bottom should be "without" instead of "with" knowledge attritions. 

The code the calculate the score is correct since we answer "no" when there is no attritions / forgetfulness. 

Would be grateful to have yo u help fix the docs and the code for the threshold! Thank you so much 😊

Thanks @penguine-ip ! I have made changes you suggested in the doc. 

What code changes are required if the score calculation is accurate?
Repository: confident-ai/deepeval
Issue: Unable to disable telemetry
Root cause: Validation Gap

**Describe the bug**
Unable to disable telemetry feature

**To Reproduce**
- Using `os.environ['DEEPEVAL_TELEMETRY_OPT_OUT'] = "YES"` at the top of my script
- Running `deepeval test run test_xx.py` over test cases 

**Expected behavior**
- I would expect any telemetry behavior to be disabled. Instead I am still seeing attempted connections (which my system is blocking), see screenshot below

**Screenshots**
<img width="1045" alt="image" src="https://github.com/confident-ai/deepeval/assets/8319226/25a872d9-a96d-4e48-937e-27f098fbb70a">


hey @ecatkins, can you quickly print the env variable to see if it is indeed "YES" and resolves to true?

@ecatkins I'm assuming this is no longer an issue?

Was this ever fixed? I'm having the same problem.

+1 , when I set env variable to "YES" I get:

  File "/home/vscode/.local/lib/python3.12/site-packages/deepeval/telemetry.py", line 235, in capture_metric_type
    posthog.capture(distinct_id=distinct_id, event=event, properties=properties)
    ^^^^^^^
NameError: name 'posthog' is not defined. Did you mean: 'Posthog'?

which is probably unrelated error, but the issue is that the this code gets executed, which imo shouldn't be the case.

Kind feedback - this is a "no-go" from my company perspective and destroys trust at very beginning.

The other thing is that you have stored API keys in git repository, which is another "no-go" for us as we cannot trust how you treat other sensitive information.

Again this is just kind feedback, as I spent couple of days testing it hoping I can use it in my project :-)

@noa-weiss its fixed in the latest release, can you test it out and let us know? also you'll get a much better response time on our discord since the mesages there don't get missed as often

@penguine-ip We're using `3.0.8`. We're still seeing the issue.

@satyaprakash1729 can you paste in the error message? Thanks

Hi, looks like this is resolved in 3.1.0. I'll give it a try and update this thread. Thank you!

With 3.5.2 I have an issue. 

I either get error uploading: HTTPSConnectionPool(host='us.i.posthog.com', port=443): Max retries exceeded with url: /batch/ when telemetry is enabled. 

When I try to disable, I get "Error: name 'posthog' is not defined". The env is set to export DEEPEVAL_TELEMETRY_OPT_OUT=1
export DEEPEVAL_TELEMETRY_OPT_OUT=YES does not work. 

Setting os.environ['DEEPEVAL_TELEMETRY_OPT_OUT'] = "YES" on the top of the script does not help either. I get the https error. 

Hey @mykolas-nice , from `3.5.0` onwards we've standardized all the env variables to boolean `0` or `1` flags, you can see the latest in our docs here: https://deepeval.com/docs/data-privacy

Let me know if this helps

Hey, @penguine-ip, thanks for the reply.

I've tried to set DEEPEVAL_TELEMETRY_OPT_OUT=1, and after this during metrics evaluation I get the error: Error: name 'posthog' is not defined

Tried version 3.4.0 - and found no issues. No errors appeared in simple execution, nor with DEEPEVAL_TELEMETRY_OPT_OUT set to YES or 1.
Repository: confident-ai/deepeval
Issue: Crash with ContextualRelevancyVerdict
Root cause: Validation Gap

**Description of the Issue:**

Hi there,

I'm encountering a Pydantic Error in this section of the code where the input is not being recognized as a string:

```python
def _generate_verdicts(
    self, text: str, retrieval_context: List[str]
) -> List[ContextualRelevancyVerdict]:
    verdicts: List[ContextualRelevancyVerdict] = []
    for context in retrieval_context:
        prompt = ContextualRelevancyTemplate.generate_verdict(
            text=text, context=context
        )
        if self.using_native_model:
            res, cost = self.model.generate(prompt)
            self.evaluation_cost += cost
        else:
            res = self.model.generate(prompt)
        data = trimAndLoadJson(res, self)
        verdict = ContextualRelevancyVerdict(**data) <<<<<<<<
        verdicts.append(verdict)

    return verdicts
```

I'm using the Mistral mistral-large-latest Model, defined with a custom your class. 
I suspect the issue may be related to the prompt using a simple syntax, possibly leading to incorrect output.

**Steps to Reproduce:**

Nothing out of the ordinary, simply utilizing ContextualRelevancyMetric, FaithfulnessMetric, and AnswerRelevancyMetric.

**Environment (Desktop):**
- Operating System: Mac OS

How can we modify the prompt without directly altering the framework code? 
Are there any recommended methods or alternative approaches to achieve this customization?

Hey @YanSte Can you please explain what do you mean by crashing? Would be nice to see the error message itself, thanks!

I will try to reproduce it.
Repository: confident-ai/deepeval
Issue: Corrupted cache can make it impossible to run deepeval
Root cause: Validation Gap

**Describe the bug**
Cache can get corrupted and block execution.

**To Reproduce**

`touch .temp-deepeval-cache.json`

Then run a program with caching turned on.

The bug can also be sort of seen in the code.

The function signature for `get_cached_test_run` is 

```
    def get_cached_test_run(
        self, from_temp: bool = False
    ) -> Union[CachedTestRun, None]:
```

And yet the way it is used is:

```
        cached_test_run = self.get_cached_test_run(from_temp=to_temp)
        cached_test_run.test_cases_lookup_map[test_case_cache_key] = (
            new_cache_test_case
        )
```

There is no handling for the None case, which results in an exception.

```
  File "/Users/paul.prescod/.venvs/evalu8/lib/python3.10/site-packages/deepeval/evaluate.py", line 348, in a_execute_test_cases
    test_run_cache_manager.cache_test_case(
  File "/Users/paul.prescod/.venvs/evalu8/lib/python3.10/site-packages/deepeval/test_run/cache.py", line 136, in cache_test_case
    cached_test_run.test_cases_lookup_map[test_case_cache_key] = (
AttributeError: 'NoneType' object has no attribute 'test_cases_lookup_map'
```

https://github.com/confident-ai/deepeval/blob/004e1cb42d11d9de039a0db047423f9576f70800/deepeval/test_run/cache.py#L136

If `cached_test_run` can be None, the code should handle that.

**Expected behavior**
Replace the cache if it is missing or corrupted.

Deepeval version 0.21.42
Repository: confident-ai/deepeval
Issue: Remove unconditional, uncontrollable output statements
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**

**Is your feature request related to a problem? Please describe.**

I'm embedding your `evaluate` function in a program and it outputs stuff like this:

✅ Tests finished! Run "deepeval login" to view evaluation results on the web.'

**Describe the solution you'd like**
It should not output text unless `print_results` is True

Deepeval 0.21.42


Any update on this one ?
Repository: confident-ai/deepeval
Issue: deepeval should return non-zero exitCode on failure
Root cause: Unimplemented Feature Gap

the deepeval cli command currently always exits with exitCode 0 - even if the tests fail - which makes is hard to handle failed tests in automated workflows/pipelines. 

For example the following workflow on GitHub ends green/successful even if the deepeval test fails. 

```
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"

      - name: Install Dependencies
        run: pip install -U deepeval

      - name: Run DeepEval Unit Tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: deepeval test run test_example.py 
```

It would be great if the deepeval command could exit with a non-0 exitCode - see https://docs.github.com/en/actions/creating-actions/setting-exit-codes-for-actions

thank you!
Repository: confident-ai/deepeval
Issue: Dramatically simpler and more reliable cache
Root cause: Architectural Constraint

It's my belief that the cache can be dramatically simplified, and made more reliable, by using the Python "diskcache" library.

DiskCache handles so much:
 * locking
 * reliability in the face of Ctrl-C/Keyboard Interrupr and other signals
 * easy clearing
 
I think that [hundreds of lines](https://github.com/confident-ai/deepeval/blob/main/deepeval/test_run/cache.py) of code could be replaced with diskcache.

JSON is a poor format for a cache because it is hard to update it transactionally and incrementally. If you Ctrl-C in the middle of a write, you'll end up with corrupted data.

Furthermore, I think that the right unit of Cache is the LLM response.

This will solve the problem where some kinds of tests are cached and [others are not](https://github.com/confident-ai/deepeval/issues/732).

I will attach two files that show how I monkey-patched to add a much more reliable caching system with few lines of code.

[caching.zip](https://github.com/confident-ai/deepeval/files/15366098/caching.zip)

It might be cleaner to add caching to the `DeepEvalBaseLLM` but then I'd need to change every place it is called, so this monkeypatching hack worked better for me. I turned off the DeepEval cache because I was frustrated with it.


Wow! this sounds like a very big improvement as I'm having troubles with the caching myself! Are you working on a PR already? @prescod
Repository: confident-ai/deepeval
Issue: Synthesizer generates goldens but a dataset loads as test cases
Root cause: Unimplemented Feature Gap

A lot of friction between generating goldens -> loading test cases for evaluation.


Hi @penguine-ip . Is there any workaround for this? The "evaluate" function doesn't admit "golden" as an argument, so providing goldens as test cases leads to empty evaluation: `No test cases found, please try again.`

Cheers.

@penguine-ip Is this open to work?


@ptdatta this is resolved too!
Repository: confident-ai/deepeval
Issue: LiteLLM Docs

Hey guys, 

I see some LiteLLM docs in this repo - curious, did y'all fork it? Totally cool if so, just wondering why fork vs. using the package? 

Hey nah - I just borrowed the layout of the documentation since you guys already set up Docusaurus - I'll make sure to remove all mentions of LiteLLM. 

See #79
Repository: confident-ai/deepeval
Issue: RagasMetric breaks when used with dataset
Root cause: Interface Contract Mismatch



**Describe the bug**
RagasMetric breaks when used with dataset

**To Reproduce**
dataset.evaluate([RagasMetric])

Errors:
TypeError: RagasMetric.a_measure() got an unexpected keyword argument '_show_indicator'

and another 

    self.llm.set_run_config(run_config)
    ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'GPTModel' object has no attribute 'set_run_config'

**Additional context**
Add any other context about the problem here.


Hey @haarisedhi102 where are you finding this?
```
self.llm.set_run_config(run_config)
```
Also what version of deepeval are you on?


@penguine-ip This happens on the latest version of deepeval too (`0.21.62`). I suspect it's due to the dependency on `ragas` library.
It fails even after adding the below `set_run_config` to the llm.
```
    def set_run_config(self, run_config: RunConfig = RunConfig()):
        self.run_config = run_config
```

@lmpratti You're probably using the DeepEvalBaseLLM wrapper in the ragas metric. Unforuntately the ragas metric only accept langhcain chat models, you can read the docs here: https://docs.confident-ai.com/docs/metrics-ragas#example

@penguine-ip Thanks a ton! Is there a plan to support `DeepEvalBaseLLM` wrapped models for ragas metric?

@lmpratti Unfortunately no because ragas does not have their own ecosystem to support custom models, so out of our control. But we have our own similar implementation of the ragas metrics for this reason
Repository: confident-ai/deepeval
Issue: Suggestion for Future Improvement: Enable Function Calling for Consistent JSON Output
Root cause: Unimplemented Feature Gap

### Suggestion for Future Improvement: Enable Function Calling for Consistent JSON Output

I am currently using a custom LLM class based on Azure OpenAI with the model `gpt-3.5-turbo-16k`, which supports function calling. While working with DeepEval, I observed that the prompts include instructions to provide a JSON parseable output. However, when the output is not JSON parseable, DeepEval suggests using a more powerful model. 

Would it not be more efficient to include an optional parameter that enables the use of function calling? This enhancement would allow less powerful models, such as `gpt-3.5-turbo`, to produce outputs in a correct JSON format more consistently.
Repository: confident-ai/deepeval
Issue: `check_for_update` should not happen on package load
Root cause: Architectural Constraint

`check_for_update` should not happen on module load. If I embed `deepeval` in my own codebase, it should not spit out messages or make networking calls on import.

`check_for_update` should be part of the CLI.


I completely agree. 

At the very least, we need to have env override to disable this, similar to how telemetry can be disabled by setting DEEPEVAL_TELEMETRY_OPT_OUT to YES (has to be all capitals - it isn't parsed and is looking for exact string).


There now exists an opt-out called DEEPEVAL_UPDATE_WARNING_OPT_OUT .

I still feel strongly that the message should be part of the CLI and not part of the import. Library impots should have ZERO network calls.

But the opt-out is a step in the right direction.
Repository: confident-ai/deepeval
Issue: Selective records failure instead of Complete Job Failure 
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
LLMs sometime cannot generate JSON output for certain records in a dataset. Deepeval functions breaks the evaluation if one such instance is received. Instead of breaking the job. Deepeval should ignore that record and evaluate the metrics for rest of the records and may be return the failure reasons record level instead of failing the job completely. 

**Describe the solution you'd like**
Questions [1,2,3]
Contexts[1,2,3]
Responses[1,2,3]

Evalaute - Faithfullness

Output should be like this. 

Output - [.76, Nan ("failed because LLM could not give proper Json"), .98]

Here, for 2nd record LLM did not work as expected and the response was Nan instead of not returning any data. 

**Describe alternatives you've considered**
Ragas provide similar functionality. 

**Additional context**
Add any other context or screenshots about the feature request here.


@TheDominus Do you explicitly want NaN or as your described just continue with the job and ignoring errors? If so we already have the option to ignore errors: https://docs.confident-ai.com/docs/evaluation-test-cases#evaluate-test-cases-in-bulk

or if you are using `deepeval test run`:
https://docs.confident-ai.com/docs/evaluation-introduction#ignore-errors




Thanks for this. This will help
Repository: confident-ai/deepeval
Issue: `ignore_errors` doesn't work as expected if `show_indicator` is set to False
Root cause: Algorithmic Error

**Description**
From what I understood, if `ignore_errors` is set to True then if any TestCase's metric measurement fails it would not affect the evaluation of other TestCases and their evaluation would continue.

https://github.com/confident-ai/deepeval/blob/7032d7d48bc6fa53d1562979e97d0dda94cb571c/deepeval/metrics/indicator.py#L131-L141

But this only happens when `show_indicator` is set to True. When it is set to false, the `ignore_errors` flag is of no use as the errors do affect and stop the evaluation of all TestCases.

https://github.com/confident-ai/deepeval/blob/7032d7d48bc6fa53d1562979e97d0dda94cb571c/deepeval/metrics/indicator.py#L171-L179

The error occurs in `metric.a_measure` and not in `tasks.append` and thus it is not caught by the try except blocks.

**To Reproduce**

1. Use a custom metric and raise an Exception in a_measure
2. Call evaluate with `show_indicator` to False and `ignore_errors` set to True

```
    results = evaluate(
            test_cases=test_cases,
            metrics=list(evaluators),
            show_indicator=False,
            print_results=False,
            use_cache=False,
            ignore_errors=True
        )
```

**Desktop:**
 - OS: Docker with Ubuntu
 - Version 0.21.48

@adayush thanks for the detailed explanation, was easy to reproduce! Fixed it in v0.21.50, TYSM!

Glad to know I was able to help! 😄
Repository: confident-ai/deepeval
Issue: accuracy always comes 0, might be a bug in my code i am unable to find.
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
A clear and concise description of what the bug is.
i want to perform a mmlu benchmark of a opensource model:    https://huggingface.co/Telugu-LLM-Labs/Indic-gemma-2b-finetuned-sft-Navarasa-2.0
input format:
### Instruction: {instruction}

### Input: {input}

## Response: {response}



**To Reproduce**
Steps to reproduce the behavior:
here is the notebook: https://github.com/AbhinavJangra29/Abhinav-Jangra/blob/main/MMLU_navarasa_issue.ipynb
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.


**Screenshots**
If applicable, add screenshots to help explain your problem.


![image](https://github.com/confident-ai/deepeval/assets/107471490/8ae12f81-b8c3-4683-a348-21d85028bc22)

![image](https://github.com/confident-ai/deepeval/assets/107471490/f1542173-f6c2-4f5b-a5b3-974eb0ba438f)



**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.


_setup: deep-eval v0.21.71_

Same issue here. I looked a bit into it, what I am seeing so far in deepeval:
1. MMLU class uses an exact string match between the target and the prediction. However, seems like the prediction consists of the "whole" string, not only the predicted answer whereas the target is only the golden answer (and not the whole string)
See here: https://github.com/confident-ai/deepeval/blob/e2075deb431ba1f5a0f818b8773fa21a46ce49ed/deepeval/scorer/scorer.py#L95
2. The construction of prompt before generation seems weird - "confiment" is directly concatenated after the Response keyword
See here: 
https://github.com/confident-ai/deepeval/blob/e2075deb431ba1f5a0f818b8773fa21a46ce49ed/deepeval/benchmarks/mmlu/template.py#L21
Side note: In your code @AbhinavJangra29, I think you could set max_new_token to a lower value, as MMLU is a multiple-choice question answering