Repository: confident-ai/deepeval
Issue: Multithreading error handling
Root cause: Resource Mishandling

**Describe the bug**
Some metrics compute a list of verdicts, statements or other things, in multithreading mode. When an error is raised from one of these child threads it is not being raised up to the parent thread, which makes the execution flow of the calling parent thread continue "normally" as it nothing happened. The main problem with this is that may go on with empty lists of verdicts, statements or whatever, thus resulting in an inaccurate evaluation of the test case.

**To Reproduce**
One common scenario I've been running into is when using a custom LLM or a GPT model instantiated with a low number of output max tokens as a parameter, thus increasing the probability of a json decode error due to the generation of an incomplete JSON output 

I know this can be avoided by using better LLMs or with longer output tokens, but I still consider that this should be better handled by properly raising the error up to the calling thread.

**Expected behavior**
Raise any errors originated from child thread to the calling thread.

<details>
<summary>Code Example</summary>

This is how multithreading is used right now to generate the list of verdicts for the ContextualRelevancyMetric
```python
    def _generate_verdicts_list(
        self, text: str, retrieval_context: List[str]
    ) -> List[List[ContextualRelevancyVerdict]]:
        verdicts_list: List[List[ContextualRelevancyVerdict]] = []
        threads = []
        lock = Lock()
        for context in retrieval_context:
            thread = Thread(
                target=self._generate_verdicts,
                args=(text, context, verdicts_list, lock),
            )
            threads.append(thread)
            thread.start()

        for thread in threads:
            thread.join()

        return verdicts_list
```

One way to catch any errors raised by one or many of these child threads could be done like the following:
```python
    def _generate_verdicts_list(
        self, text: str, retrieval_context: List[str]
    ) -> List[List[ContextualRelevancyVerdict]]:
        verdicts_list: List[List[ContextualRelevancyVerdict]] = []
        threads = []
        thread_exceptions = []  # List to store exceptions from threads
        lock = Lock()
        def thread_wrapper(func, args, exceptions_list, lock):
            try:
                func(*args)
            except Exception as e:
                # Lock is not strictly necessary here if you're only appending to a list,
                # but is good practice if you modify the shared data structure in other ways
                with lock:
                    exceptions_list.append(e)
        for context in retrieval_context:
            thread = Thread(
                # target=self._generate_verdicts,
                target=thread_wrapper,
                args=(self._generate_verdicts, (text, context, verdicts_list, lock), thread_exceptions, lock),
            )
            threads.append(thread)
            thread.start()

        for thread in threads:
            thread.join()

        if len(thread_exceptions) > 0:
            # Instead of raising all execeptions, we could just raise the first one at least.
            raise thread_exceptions[0]

        return verdicts_list
```
</details>

Hey @AndresPrez , thanks for bringing this up. Seems like you already dug into this problem, do you mind making a PR to take care of this?

By the way @AndresPrez, you tested the PR and the error actually works right? Since I don't have custom models it never errored for me so hard for me to test...

I did test it locally, how are you implementing tests? If you're using openai for example, there's a parameter that sets the max output tokens that can be set to be really low which should force an invalid JSON to be generated. I can come up with reproducible test soon.

@AndresPrez No problem i just modified the trim json function to raise an exception regardless :) I switched to using threadpoolexecutor instead since the errors can be propagated without having to manually handle exceptions: https://github.com/confident-ai/deepeval/pull/546

Thanks for bringing this up!