Repository: confident-ai/deepeval
Issue: No such file or directory: 'temp_test_run_data.json'_
Root cause: Resource Mishandling

**Describe the bug**
It throws error showing 
_Error loading test run from disk: [Errno 2] No such file or directory: 'temp_test_run_data.json'_
![image](https://github.com/confident-ai/deepeval/assets/142291246/6b83e4e9-dfae-45e3-b1c7-9d1ccd200bd2)


**To Reproduce**
![image](https://github.com/confident-ai/deepeval/assets/142291246/86f6cf14-6810-4e21-9298-e471e0943424)

Steps to reproduce the behavior:
1. Provide multiple test cases into evaluate function

**Expected behavior**
Should not throw error and able to proceed with report generation

**Desktop (please complete the following information):**
 - OS: macOs 13.6.2 (22G320)

@jonathansoong I"m not about to reproduce it, but basically, when you first run evaluate(), it will first create a file called temp_test_run_data.json. The error suggests the file isn't being created. Can you try:

1. Run evaluate()
2. The second a metic starts evaluating, cancel the execution
3. See if the temp_test_run_data.json lingers in your directory

Another thing you can try:

1. Execute the evaluate() function from the root directory of your project. For example, if your structure is `project/deepeval.py`, don't run the file in the project directory, do it at the root instead


I'm encountering the same issue. I've already tried the suggestions made by @penguine-ip . Additionally, I set the DEEPEVAL_RESULTS_FOLDER environment variable and confirmed that the run was successful by checking the results files. However, the error related to the temp_test_run_data.json file persists.

@luisduque-ve Can you provide the project structure and paste the exact code to reproduce it? I tried again and wasn't able to reproduce it. 

Also an important question:
1. When does the error occur? After all the evaluations are complete, midway through evaluations, or at the very beginning? 

To help clarify, when you run evaluate (https://github.com/confident-ai/deepeval/blob/main/deepeval/evaluate.py#L124), we first create a file. Then, every time we access the test run object, we will load it from file. Then, once evaluation is finished, we will load the test run from file one last time, which is where @jonathansoong is erroring at. However, I'm not sure why it only errored at the very end and not at the beginning and middle.

The place where @jonathansoong is erroring is here: https://github.com/confident-ai/deepeval/blob/main/deepeval/evaluate.py#L131



@penguine-ip sure!

1. The error occurs at the end of the dataset evaluation.
2. This is just a proof of concept with DeepEval that I'm working on. It consists of only the test (.py) file and a JSON file containing the dataset. Both files are located in the root directory, with no nested directories.

```python
import json
import os
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset


def test_customer_chatbot():

    os.environ["DEEPEVAL_RESULTS_FOLDER"] = "./data"

    with open("./deepeval_test_cases.json") as f:
        llm_cases = [
            LLMTestCase(
                input=test_case["input"],
                actual_output=test_case["output"],
                retrieval_context=test_case["context"],
            )
            for test_case in json.load(f)
        ]
    dataset = EvaluationDataset(test_cases=llm_cases)
    metrics = [
        AnswerRelevancyMetric(threshold=0.5),
        FaithfulnessMetric(threshold=0.3),
    ]
    dataset.evaluate(metrics)
```

@luisduque-ve I see! The problem is you're using dataset.evaluate in a test function. In `deepeval`, you use `assert_test` instead when working with our pytest integration.


Here is what the code should be like:
```python
import json
import os
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset

# Load test cases from JSON
with open("./deepeval_test_cases.json") as f:
    llm_cases = [
        LLMTestCase(
            input=test_case["input"],
            actual_output=test_case["output"],
            retrieval_context=test_case["context"],
        )
        for test_case in json.load(f)
    ]

# Create evaluation dataset
dataset = EvaluationDataset(test_cases=llm_cases)

# Use pytest to loop through dataset
@pytest.mark.parametrize(
    "test_case",
    dataset,
)
def test_customer_chatbot(test_case: LLMTestCase):
    metrics = [
        AnswerRelevancyMetric(threshold=0.5),
        FaithfulnessMetric(threshold=0.3),
    ]
    assert_test(test_case, metrics)
```

There are a few reasons we keep it this way, and the biggest one is because it integrates nicely with the pytest ecosystem which allows you for example, to run test cases in parallel: https://docs.confident-ai.com/docs/evaluation-introduction#parallelization


@jonathansoong does this also solve your problem?


If you wish to use evaluate, here is an example: https://docs.confident-ai.com/docs/evaluation-datasets#evaluate-your-dataset-without-pytest



Thank you, @penguine-ip , for resolving that issue, it works for me. However, I have a quick question regarding the code you provided. Despite defining the "DEEPEVAL_RESULTS_FOLDER," the results are not being exported to a file after I set the environment variable. Is this the expected behavior? I'm unsure if the `assert_test` function does not support exporting results to a file, which is necessary for my use case.

@luisduque-ve I forgot to include the line of code that exports it, here is the updated version:

```python
import pytest
import json
import os
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset

# Save to ./data directory
os.environ["DEEPEVAL_RESULTS_FOLDER"] = "./data"

# Load test cases from JSON
with open("./deepeval_test_cases.json") as f:
    llm_cases = [
        LLMTestCase(
            input=test_case["input"],
            actual_output=test_case["output"],
            retrieval_context=test_case["context"],
        )
        for test_case in json.load(f)
    ]

# Create evaluation dataset
dataset = EvaluationDataset(test_cases=llm_cases)

# Use pytest to loop through dataset
@pytest.mark.parametrize(
    "test_case",
    dataset,
)
def test_customer_chatbot(test_case: LLMTestCase):
    metrics = [
        AnswerRelevancyMetric(threshold=0.5),
        FaithfulnessMetric(threshold=0.3),
    ]
    assert_test(test_case, metrics)
```

I tried it and it works, let me know!

@penguine-ip I've managed to run it now, by changing the running command from 
```bash
deepeval test run test-module
```

to 
```bash
python test-module/main.py
```
without using the deepeval pytest command. Thank you very much for the help!

@jonathansoong No problem! Gonna wait to end of week before closing this in case any additional problems come up, glad you like deepeval :)

On the same area, but a bit to the side :)
My project runs heavily on pytest. I'm adding a new test that will specifically use `deepeval` (with `assert_test`) - as suggested in this thread. 

For some reason, running any other test now, creates this new `temp_test_run_data.json`, even if it is a test that has no direct dependency on `deepeval` whatsoever.

@oribarilan I think it is because pytest will automatically grab any file prefixed with test_ to run. I could make it so that pytest does not run unless it is the deepeval test run command, but I will have to think whether this is the way to go since some users are using deepeval through pytest.

Can you try ignoring the deepeval file in your CI/CD (I assume that's where you're running pytest) and let me know how it goes?

@penguine-ip It is created even when I run a test in isolation. I didn't get around to fully debug this, but I did find that TestRunManager is initiated on import, which I find weird and may be the cause. Is there a a flow that registers a pytest hook which creates this?

Anyway, intuitively I think it would make sense for this file to be created in the evaluate or the assert_test methods, rather than how it is currently done, since it interrupts with tests that has nothing to do with deepeval.

@oribarilan it is now fixed in the latest release (v0.20.74). Anything ran using the pytest command will not  create any files (but still run the test if in the same directory), this also means you will have to include an additional `deepeval test run` command if you're mixing pytest and deepeval files in a same directory to avoid unexpected errors
Let me know if you find any edge cases, I'll leave this open for the next week in case you still encounter issues. 

works, thank you :)

> this also means you will have to include an additional deepeval test run command if you're mixing pytest and deepeval files in a same directory to avoid unexpected errors

I think this makes a lot of sense :)


`PermissionError: [Errno 13] Permission denied: 'temp_test_run_data.json'`

Can we specify dir for this? 


Hi. I'm also getting the same error, but **I'm not running any test**; I'm running multiple DeepEvalEvaluator components from the Haystack integration library.
Before, I was able to run the pipeline without issues, running every component sequentially, one after the other. However, now I'm running these components concurrently using threads, and I'm getting this error. 
More specifically, I'm getting it when running the metrics Contextual Recall & Contextual Relevancy. In the second one I'm also getting a `AttributeError: 'NoneType' object has no attribute 'add_llm_test_case'` coming from `test_run_manager.get_test_run().add_llm_test_case(`

Is this expected behavior (¿running tests while running the evaluation? ¿Any ideas or guidance on how to deal with this?