Repository: confident-ai/deepeval
Issue: testcase score fluctuate significantly
Root cause: Algorithmic Error

I began using Deepeval to design a validation pipeline.

However, I’ve encountered several random errors with the following metrics:
AnswerRelevancyMetric
FaithfulnessMetric
Using the GPT-4o model of ChatGPT.

I am facing fluctuation in Metric score:
scenario 1: Analyzing 23 test cases, "Test Case X" initially showed an "AnswerRelevancyMetric" of 0.3. 
scenario 2: Updating the index of "Test Case X" within the 23 test case set resulted in the "AnswerRelevancyMetric" shifting to 0.98.
scenario 3: When I evaluated only "Test Case X" individually, the "AnswerRelevancyMetric" increased to 1.

Why does this score fluctuate so significantly?

@enx-github without seeing your test case i really can't tell


Hi,
I am using the code below to evaluate my question, actual_output, and retrieval_context 

```
# test case
test_case = LLMTestCase(
            input=question,
            actual_output=answer,
            retrieval_context=[str(context)]
)        
```

```
evaluationResult = evaluate(
            dataset,
            # use AnswerRelevancyMetric & FaithfulnessMetric            
            metrics=[answer_relevancy_metric, faithfulness_metric],
            skip_on_missing_params=True,
            # additional parameters
            hyperparameters={"model": "gpt-4o, 
                             "prompt template": final_answer_prompt, 
                             "temperature": 0,
            use_cache=False
        )
```
scenario 1: Analyzing 23 test cases, "Test Case X" initially showed an "AnswerRelevancyMetric" of 0.3.
test_cases = [test_case_0, text_case_1...., text_case_X, test_case_18, test_case_19, ...., test_case_23]
 
scenario 2: Updating the index of "Test Case X" within the 23 test case set resulted in the "AnswerRelevancyMetric" shifting to 0.98.
test_cases = [test_case_X, text_case_0...., text_case_17, test_case_18, test_case_19, ...., test_case_23]

scenario 3: When I evaluated only "Test Case X" individually, the "AnswerRelevancyMetric" increased to 1.
test_cases = [test_case_X]

Can you help me with the score issue?

@enx-github i don't know what your test cases look like so its hard to tell you exactly why its fluctuating 

I have an example test case as below:
input= "Which contracts are only for the energy type Gas?"
actual_output = "Answer from the LLM is passed dynamically"
retrieval_context = "Context is retrieved and passed dynamically"
The metrics I have used are "FaithfulnessMetric" and "AnswerRelevancyMetric"

As mentioned above I get varying results when executing the same Test Case from different Python Pages. I have manually checked that the "actual_output" is having similar content and the "retrieval_context" is identical. I have also checked the the model used, temperature etc. are the same in both Python Pages. So I am surprised how can there be such a variation in results.

In  one page I am sometimes getting
"FaithfulnessMetric" = 0.25
"AnswerRelevancyMetric" = 0.4

In the second page I am getting
"FaithfulnessMetric" = 1.0
"AnswerRelevancyMetric" = 0.91

Please suggest me how I can find the reason for this varying results.

turn on verbose_mode for the metrics, but also your test case doesn't make sense at all, so i doubt an llm would be able to understand it. EG what does the input have anything to do with the actual output and retrieval context? I'm not sure what score i would give it myself too