Repository: confident-ai/deepeval
Issue: two questions for evaluation output
Root cause: Algorithmic Error

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
A clear and concise description of what the bug is.
1. When I saw a table confident.ai and console outputs, some of evaluation result said that the status is failed even though all the metric was 'success'.
<img width="1576" alt="image" src="https://github.com/confident-ai/deepeval/assets/32179857/ad131eb6-702a-4a4d-b9e8-b42fa40535fa">

2. When I saw a table, I could find some cases that the score of FaithfulnessMetric metric was 0.0 with a description of 
```
The score is 0.00 because there are no contradictions listed, indicating a perfect alignment between the actual output and the retrieval context. Great job on maintaining consistency!
```
or even as below
<img width="959" alt="image" src="https://github.com/confident-ai/deepeval/assets/32179857/91809783-eff5-482d-802d-1497b591b299">

**To Reproduce**
Steps to reproduce the behavior:
I've run the simple evaluation with my custom dataset loaded from .csv file.
I used AnswerRelevancyMetric and FaithfulnessMetric for testing case 2, and  AnswerRelevancyMetric, FaithfulnessMetric, and one custom metric for case 1. Whit these setup, I found above bugs.

**Expected behavior**
I expect pre-defined metric classes are working well and adding custom metric too.

**Screenshots**
attached above with the description of the issues

**Desktop (please complete the following information):**
 - OS: Mac Sonoma 14.1
 - Browser chrome
 - Version deepeval 0.20.74

**Additional context**
Add any other context about the problem here.


@mhyeonsoo Hey this isn't really the place to ask about the platform but just deepeval (this should go in the discord channel), but if you send me your test run ID i can take a look for you 

Thanks @penguine-ip , I will look for the Discord for further questions.
For the test run ID, I am not sure this is correct ID, but it is 53054 in the link of https://app.confident-ai.com/project/clsx7k3db0001ib0fvkacwux3/unit-tests/clt275av50001l70gumrcwww7/test-cases

@mhyeonsoo Your test run Id is clt275av50001l70gumrcwww7, will look into it :)

@mhyeonsoo Are two bugs reproducible? The second bug I'm seeing this for the first time, it seems like switching to gpt-4-0125-preview as the default model caused it, never seen this bug before switching to this model.

The first bug seems like there should an error in calculating the success. I suspect it is the implementation of your custom metric. Here is the line of code where we see whether your metric is successful or not: https://github.com/confident-ai/deepeval/blob/main/deepeval/evaluate.py#L82, then we update the success status of the test case a few lines further down. Can you show your implementation of is_successful()?

Thanks @penguine-ip , I tried to sort of reproduce the cases.
For the first one, with the modified custom metric, it does not happen any more.

But for the second case, even for the built-in metric such as hallucination, it showed similar cases.
I am attaching the test case id : clt4axufa0001jr0gx7yu4yrp, and at there I could see hallucination is failed even though it scored 0.0.

Can you check this out? Thank you so much

@mhyeonsoo the new release should stabalize the first issue (v0.20.75) As for the second issue, can you also let me know what is the input string? Easier for me to search

My apologies @penguine-ip ,
I checked again with the new trial, and found that was because of the context relevancy score, which is not a hallucination score.
Everything seems fine for me now with the new version you said.

Thank you so much for the helps!

By the way @mhyeonsoo what's your discord username (have you joined?) I saw some interesting use cases when debugging your test cases. I can share with you in the discord channel

Hi @penguine-ip ,
Yes, I just joined the discord with the user name : craching

I would be happy to see the interesting cases.
Besides, I can see some hallucination eval results vary for the same output responses.

It would be great to discuss further in discord if possible.

Thanks,

BTW, is my data kept private? I am wondering if the test cases in my project are not publically accessible or private. 

Thanks

Perfect! Yup it is private, you'll be fine even if you lose your API key, just don't expose your login credentials.