Repository: confident-ai/deepeval
Issue: Consideration behind a "stateful" metric UX
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Hi, I was wondering what's the considerations behind that we are choosing a "stateful" metrics UX. Using the hallucination metric for example but it applies to other metrics as well

```
metric = HallucinationMetric(threshold=0.5)

metric.measure(test_case)
print(metric.score)
print(metric.reason)
```

By "stateful" here I mean the `metric` object itself stores the state (score, reason, etc.) for the last `test_case` it has run against. I feel this might create unnecessary coupling between metrics and test cases? It could lead to side effects, for example, in bulk evaluation, we can not go full parallel across both the test_cases and metrics dimensions, instead we have to go [one by one](https://github.com/confident-ai/deepeval/blob/v0.20.89/deepeval/evaluate.py#L126) for each test_case, because otherwise we can't pull the `reason` out which is stored in the `metric` object.

**Describe the solution you'd like**
Any thoughts if we can make the metric UX stateless. For example:
```
class BaseMetric:
    # not store test-related values here such as score and reason
    # only store metric-related values such as evaluation model
    evaluation_model: Optional[str] = None

...

    @abstractmethod
    # The LLMTestResult can be an extensible dict with fields such as score and reason
    def measure(self, test_case: LLMTestCase, *args, **kwargs) -> LLMTestResult:
        raise NotImplementedError
...

```

**Additional context**
This might certainly mean some breaking compatibilities if we are to make such a fundamental UX update, so understanding any concerns here. But meanwhile I feel this could also mean a huge opportunity and thought better to raise it sooner rather than later.

Hey @Peilun-Li, I definitely see this being an issue sometime in the future. For example, if i run all test cases in parallel and then run the same set of metrics for each test case, I'd imagine some class variables like self.verdicts being overriden, self.success being populated by another `.meausre()`'s results, etc. 

Do you have any suggestions? Right now i'm not sure how making it non stateful wil llook like. I guess one major advantage of what the statement metric gives is the ability to call metric.`property`, which is nice and simple. 

Cool yeah I just made a quick prototype using the Hullucination metric (deepeval version 0.20.90 https://github.com/confident-ai/deepeval/blob/v0.20.90/deepeval/metrics/hallucination/hallucination.py )

(I added line comments to the pieces I updated)
```
import deepeval
from deepeval.metrics import HallucinationMetric
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.utils import check_test_case_params, get_or_create_event_loop
from deepeval.metrics.indicator import metric_progress_indicator
from deepeval.telemetry import capture_metric_type
from typing import List
from pydantic import BaseModel, Field
from deepeval.metrics.hallucination.template import HallucinationTemplate

required_params: List[LLMTestCaseParams] = [
    LLMTestCaseParams.INPUT,
    LLMTestCaseParams.ACTUAL_OUTPUT,
    LLMTestCaseParams.CONTEXT,
]

class HallucinationVerdict(BaseModel):
    verdict: str
    reason: str = Field(default=None)

class StatelessHallucinationMetric(HallucinationMetric):
    def measure(self, test_case) -> dict:
        check_test_case_params(test_case, required_params, self.__name__)
        with metric_progress_indicator(self):
            if self.async_mode:
                loop = get_or_create_event_loop()
                loop.run_until_complete(
                    self.a_measure(test_case, _show_indicator=False)
                )
            else:
                # remove all individual test instance related `self.` and form a return dict instead
                verdicts: List[HallucinationVerdict] = (
                    self._generate_verdicts(
                        test_case.actual_output, test_case.context
                    )
                )
                # pass verdicts through function input 
                score = self._calculate_score(verdicts)
                reason = self._generate_reason(verdicts)
                success = score <= self.threshold
                capture_metric_type(self.__name__)
                # We can potentially make this dict return type a dataclass/basemodel
                test_run_result = {
                    "score": score,
                    "reason": reason,
                    "success": success,
                    "threshold": self.threshold,
                    "verdicts": verdicts
                }
                return test_run_result

    def _calculate_score(self, verdicts) -> float:
        # move from self.verdicts to verdicts
        number_of_verdicts = len(verdicts)
        if number_of_verdicts == 0:
            return 0

        hallucination_count = 0
        # move from self.verdicts to verdicts
        for verdict in verdicts:
            if verdict.verdict.strip().lower() == "no":
                hallucination_count += 1

        score = hallucination_count / number_of_verdicts
        return 1 if self.strict_mode and score > self.threshold else score
    
    def _generate_reason(self, verdicts):
        if self.include_reason is False:
            return None

        factual_alignments = []
        contradictions = []
        # move from self.verdicts to verdicts
        for verdict in verdicts:
            if verdict.verdict.strip().lower() == "no":
                factual_alignments.append(verdict.reason)
            else:
                contradictions.append(verdict.reason)

        prompt: dict = HallucinationTemplate.generate_reason(
            factual_alignments=factual_alignments,
            contradictions=contradictions,
            score=format(self.score, ".2f"),
        )

        res = self.model.generate(prompt)
        return res
```

Now running it against the sample test case
```
context=["A man with blond-hair, and a brown shirt drinking out of a public water fountain."]

# Replace this with the actual output from your LLM application
actual_output="A blond drinking water in public."

test_case = LLMTestCase(
    input="What was the blond doing?",
    actual_output=actual_output,
    context=context
)
```

With existing "stateful" metric
```
metric = HallucinationMetric(threshold=0.5)
metric.measure(test_case)
print(metric.score)
print(metric.reason)
print(metric.success)
print(metric.verdicts)
> 0.0
> The score is 0.00 because there are no factual alignments or contradictions listed that would indicate a deviation from the provided context, implying the output is entirely factual and consistent with the context.
> True
> [HallucinationVerdict(verdict='yes', reason='The actual output agrees with the provided context by describing a blond individual drinking water in public, which implies the use of a public water fountain.')]
```

With the stateless metric
```
stateless_metric = StatelessHallucinationMetric(threshold=0.5)
test_result = stateless_metric.measure(test_case)
print(test_result)
> {'score': 0.0,
 'reason': 'The score is 0.00 because there are no factual contradictions between the actual output and the provided context; the listed contradiction is incorrectly categorized and actually describes an alignment, not a discrepancy.',
 'success': True,
 'threshold': 0.5,
 'verdicts': [HallucinationVerdict(verdict='yes', reason='The actual output agrees with the provided context, describing a blond individual drinking water in public, which is in line with the context of a man with blond-hair, wearing a brown shirt, drinking out of a public water fountain.')]}
```

Basically the high level idea is for any test case-related property we pass explicitly through function input/output, instead of passing implicitly through self variables.

hey @Peilun-Li I'll take a look at this over the next day or two after shipping the features for the week, thanks for the work! 

hey @Peilun-Li, I think we can just create a copy of the metric instead? I like this approach but a tradeoff I see is users who wish to debug with print(self.verdicts) for example cannot do so, what are your thoughts on this?

Oh we can certainly include more fields in the return which can be totally customizable by individual metrics and needs - I just updated the above code to also include `verdicts` within the response.

I'm a bit worried while "copy metrics" may solve this specific issue, it may introduce more/other issues. For example, some potential custom metrics that may have an ML model run locally may not be that copy-friendly. The larger pain point I feel here is not code-implementation-wise but more conceptual-wise -- currently it feels like the way we treat `Metrics` objects is actually `MetricsRun`, which is a bit counter-intuitive compared to their naming and user perceptions. 

Update - I end up writing a bulk eval script to unblock our experiments https://github.com/confident-ai/deepeval/issues/286#issuecomment-2013955114 Indeed a copy metric-object pattern but you could feel based on the final return there essentially we are using the copied metrics object as `LLMTestResult `

Hey @Peilun-Li I just saw the other comment you made. Do you know what is causing the file error, and how can I reproduce it? The snippet the other user sent didn't give me any errors.

> Hey @Peilun-Li I just saw the other comment you made. Do you know what is causing the file error, and how can I reproduce it? The snippet the other user sent didn't give me any errors.

Yeah just replied in that thread

Update on this thread in case anyone is interested - we've brought part of the discussion on bulk evaluation to discord https://discord.com/channels/1118193057620316202/1223053518542475364/1223053518542475364