Repository: confident-ai/deepeval
Issue: Running deepeval run throws an error after setting the DEEPEVAL_RESULTS_FOLDER.
Root cause: Validation Gap

**Describe the bug**
I wrote a simple test code as follows:
```
def test_faithfulnessMetric():

    metric = FaithfulnessMetric(
        threshold=0.5,
        include_reason=True,
        model = judge_llm
    )

    test_case = LLMTestCase(
        input="what is 'gfsda'?",
        actual_output="Another term for sneezing.",
        retrieval_context=["'gfsda' has no meaning; it's a randomly created word"]
    )

    evaluate(test_cases=[test_case], metrics=[metric])
```
And set the `DEEPEVAL_RESULTS_FOLDER` to get the test results
```
export DEEPEVAL_RESULTS_FOLDER="./data
```
but running deepeval run gives an error.

```
✨ You're running DeepEval's latest Faithfulness Metric! (using deepseek-chat, strict=False, async_mode=True)...


======================================================================

Metrics Summary

  - ❌ Faithfulness (score: 0.0, threshold: 0.5, strict: False, evaluation model: deepseek-chat, reason: The score is 0.00 because the actual output incorrectly claims 'gfsda' means 'Another term for sneezing,' while the retrieval context explicitly states it has no meaning and is a randomly created word, creating a direct factual contradiction., error: None)

For test case:

  - input: what is 'gfsda'?
  - actual output: Another term for sneezing.
  - expected output: None
  - context: None
  - retrieval context: ["'gfsda' has no meaning; it's a randomly created word"]

======================================================================

Overall Metric Pass Rates

Faithfulness: 0.00% pass rate

======================================================================


⚠ WARNING: No hyperparameters logged.
» Log hyperparameters to attribute prompts and models to your test runs.

================================================================================
Results saved in ./data as 20251204_232229


✓ Evaluation completed 🎉! (time taken: 8.24s | token cost: 0.0005549800000000001 USD)
» Test Results (1 total tests):
   » Pass Rate: 0.0% | Passed: 0 | Failed: 1

 ================================================================================ 

» Want to share evals with your team, or a place for your test cases to live? ❤️ 🏡
  » Run 'deepeval view' to analyze and save testing results on Confident AI.


.Running teardown with pytest sessionfinish...

============================================================================================================ slowest 10 durations =============================================================================================================
8.25s call     test_demo_faithfulnes.py::test_faithfulnessMetric
0.02s setup    test_demo_faithfulnes.py::test_faithfulnessMetric

(1 durations < 0.005s hidden.  Use -vv to show these durations.)
1 passed, 4 warnings in 8.29s
Warning: Could not load test run from disk: [Errno 2] No such file or directory: '.deepeval/.temp_test_run_data.json'
                                                                                                                 Test Results                                                                                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test case                                                                          ┃ Metric       ┃ Score                                                                                                   ┃ Status ┃ Overall Success Rate ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ test_faithfulnessMetric                                                            │              │                                                                                                         │        │ 0.0%                 │
│                                                                                    │ Faithfulness │ 0.0 (threshold=0.5, evaluation model=deepseek-chat, reason=The score is 0.00 because the actual output  │ FAILED │                      │
│                                                                                    │              │ incorrectly claims 'gfsda' means 'Another term for sneezing,' while the retrieval context explicitly    │        │                      │
│                                                                                    │              │ states it has no meaning and is a randomly created word, creating a direct factual contradiction.,      │        │                      │
│                                                                                    │              │ error=None)                                                                                             │        │                      │
│ Note: Use Confident AI with DeepEval to analyze failed test cases for more details │              │                                                                                                         │        │                      │
└────────────────────────────────────────────────────────────────────────────────────┴──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────┴──────────────────────┘

⚠ WARNING: No hyperparameters logged.
» Log hyperparameters to attribute prompts and models to your test runs.

================================================================================
╭───────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /Users/fanbin/miniconda3/envs/case_rag/lib/python3.12/site-packages/deepeval/cli/test.py:174 in run                                                                                                                                         │
│                                                                                                                                                                                                                                             │
│   171 │   │   pytest_retcode = pytest.main(pytest_args)                                                                                                                                                                                     │
│   172 │   end_time = time.perf_counter()                                                                                                                                                                                                    │
│   173 │   run_duration = end_time - start_time                                                                                                                                                                                              │
│ ❱ 174 │   global_test_run_manager.wrap_up_test_run(run_duration, True, display)                                                                                                                                                             │
│   175 │                                                                                                                                                                                                                                     │
│   176 │   invoke_test_run_end_hook()                                                                                                                                                                                                        │
│   177                                                                                                                                                                                                                                       │
│                                                                                                                                                                                                                                             │
│ ╭──────────────────────────────────────────────────────────────────── locals ─────────────────────────────────────────────────────────────────────╮                                                                                         │
│ │                  color = 'yes'                                                                                                                  │                                                                                         │
│ │                    ctx = <click.core.Context object at 0x110ebbf80>                                                                             │                                                                                         │
│ │                display = <TestRunResultDisplay.ALL: 'all'>                                                                                      │                                                                                         │
│ │              durations = 10                                                                                                                     │                                                                                         │
│ │               end_time = 39492.840668541                                                                                                        │                                                                                         │
│ │  exit_on_first_failure = False                                                                                                                  │                                                                                         │
│ │             identifier = None                                                                                                                   │                                                                                         │
│ │          ignore_errors = False                                                                                                                  │                                                                                         │
│ │                   mark = None                                                                                                                   │                                                                                         │
│ │          num_processes = None                                                                                                                   │                                                                                         │
│ │                    pdb = False                                                                                                                  │                                                                                         │
│ │            pytest_args = ['test_demo_faithfulnes.py', '--quiet', '--color=yes', '--durations=10', '-s', '--disable-warnings', '-p', 'deepeval'] │                                                                                         │
│ │         pytest_retcode = <ExitCode.OK: 0>                                                                                                       │                                                                                         │
│ │                 repeat = None                                                                                                                   │                                                                                         │
│ │           run_duration = 8.81854204100091                                                                                                       │                                                                                         │
│ │       should_use_cache = False                                                                                                                  │                                                                                         │
│ │          show_warnings = False                                                                                                                  │                                                                                         │
│ │ skip_on_missing_params = False                                                                                                                  │                                                                                         │
│ │             start_time = 39484.0221265                                                                                                          │                                                                                         │
│ │ test_file_or_directory = 'test_demo_faithfulnes.py'                                                                                             │                                                                                         │
│ │              use_cache = False                                                                                                                  │                                                                                         │
│ │                verbose = None                                                                                                                   │                                                                                         │
│ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                         │
│                                                                                                                                                                                                                                             │
│ /Users/fanbin/miniconda3/envs/case_rag/lib/python3.12/site-packages/deepeval/test_run/test_run.py:1022 in wrap_up_test_run                                                                                                                  │
│                                                                                                                                                                                                                                             │
│   1019 │   │   │   │   console.print("\n[bold green]✓ Prompts Logged[/bold green]\n")                                                                                                                                                       │
│   1020 │   │   │   │   self._render_prompts_panels(prompts=test_run.prompts)                                                                                                                                                                │
│   1021 │   │                                                                                                                                                                                                                                │
│ ❱ 1022 │   │   self.save_test_run_locally()                                                                                                                                                                                                 │
│   1023 │   │   delete_file_if_exists(self.temp_file_path)                                                                                                                                                                                   │
│   1024 │   │   if is_confident() and self.disable_request is False:                                                                                                                                                                         │
│   1025 │   │   │   return self.post_test_run(test_run)                                                                                                                                                                                      │
│                                                                                                                                                                                                                                             │
│ ╭──────────────────────────────────────────────────────────────────── locals ────────────────────────────────────────────────────────────────────╮                                                                                          │
│ │       display = <TestRunResultDisplay.ALL: 'all'>                                                                                              │                                                                                          │
│ │ display_table = True                                                                                                                           │                                                                                          │
│ │   runDuration = 8.81854204100091                                                                                                               │                                                                                          │
│ │          self = <deepeval.test_run.test_run.TestRunManager object at 0x110bf35f0>                                                              │                                                                                          │
│ │      test_run = TestRun(                                                                                                                       │                                                                                          │
│ │                 │   test_file=None,                                                                                                            │                                                                                          │
│ │                 │   test_cases=[                                                                                                               │                                                                                          │
│ │                 │   │   LLMApiTestCase(                                                                                                        │                                                                                          │
│ │                 │   │   │   name='test_faithfulnessMetric',                                                                                    │                                                                                          │
│ │                 │   │   │   input="what is 'gfsda'?",                                                                                          │                                                                                          │
│ │                 │   │   │   actual_output='Another term for sneezing.',                                                                        │                                                                                          │
│ │                 │   │   │   expected_output=None,                                                                                              │                                                                                          │
│ │                 │   │   │   context=None,                                                                                                      │                                                                                          │
│ │                 │   │   │   retrieval_context=["'gfsda' has no meaning; it's a randomly created word"],                                        │                                                                                          │
│ │                 │   │   │   tools_called=None,                                                                                                 │                                                                                          │
│ │                 │   │   │   expected_tools=None,                                                                                               │                                                                                          │
│ │                 │   │   │   token_cost=None,                                                                                                   │                                                                                          │
│ │                 │   │   │   completion_time=None,                                                                                              │                                                                                          │
│ │                 │   │   │   tags=None,                                                                                                         │                                                                                          │
│ │                 │   │   │   multimodal_input=None,                                                                                             │                                                                                          │
│ │                 │   │   │   multimodal_input_actual_output=None,                                                                               │                                                                                          │
│ │                 │   │   │   multimodal_expected_output=None,                                                                                   │                                                                                          │
│ │                 │   │   │   multimodal_retrieval_context=None,                                                                                 │                                                                                          │
│ │                 │   │   │   multimodal_context=None,                                                                                           │                                                                                          │
│ │                 │   │   │   success=False,                                                                                                     │                                                                                          │
│ │                 │   │   │   metrics_data=[                                                                                                     │                                                                                          │
│ │                 │   │   │   │   MetricData(                                                                                                    │                                                                                          │
│ │                 │   │   │   │   │   name='Faithfulness',                                                                                       │                                                                                          │
│ │                 │   │   │   │   │   threshold=0.5,                                                                                             │                                                                                          │
│ │                 │   │   │   │   │   success=False,                                                                                             │                                                                                          │
│ │                 │   │   │   │   │   score=0.0,                                                                                                 │                                                                                          │
│ │                 │   │   │   │   │   reason="The score is 0.00 because the actual output incorrectly claims 'gfsda' means 'An"+162,             │                                                                                          │
│ │                 │   │   │   │   │   strict_mode=False,                                                                                         │                                                                                          │
│ │                 │   │   │   │   │   evaluation_model='deepseek-chat',                                                                          │                                                                                          │
│ │                 │   │   │   │   │   error=None,                                                                                                │                                                                                          │
│ │                 │   │   │   │   │   evaluation_cost=0.0005549800000000001,                                                                     │                                                                                          │
│ │                 │   │   │   │   │   verbose_logs='Truths (limit=None):\n[\n    "\'gfsda\' has no meaning.",\n    "\'gfsda\' is a randomly'+330 │                                                                                          │
│ │                 │   │   │   │   )                                                                                                              │                                                                                          │
│ │                 │   │   │   ],                                                                                                                 │                                                                                          │
│ │                 │   │   │   run_duration=8.231660375000502,                                                                                    │                                                                                          │
│ │                 │   │   │   evaluation_cost=0.0005549800000000001,                                                                             │                                                                                          │
│ │                 │   │   │   order=0,                                                                                                           │                                                                                          │
│ │                 │   │   │   additional_metadata=None,                                                                                          │                                                                                          │
│ │                 │   │   │   comments=None,                                                                                                     │                                                                                          │
│ │                 │   │   │   trace=None                                                                                                         │                                                                                          │
│ │                 │   │   )                                                                                                                      │                                                                                          │
│ │                 │   ],                                                                                                                         │                                                                                          │
│ │                 │   conversational_test_cases=[],                                                                                              │                                                                                          │
│ │                 │   metrics_scores=[MetricScores(metric='Faithfulness', scores=[0.0], passes=0, fails=1, errors=0)],                           │                                                                                          │
│ │                 │   trace_metrics_scores=None,                                                                                                 │                                                                                          │
│ │                 │   identifier=None,                                                                                                           │                                                                                          │
│ │                 │   hyperparameters=None,                                                                                                      │                                                                                          │
│ │                 │   prompts=[],                                                                                                                │                                                                                          │
│ │                 │   test_passed=0,                                                                                                             │                                                                                          │
│ │                 │   test_failed=1,                                                                                                             │                                                                                          │
│ │                 │   run_duration=8.81854204100091,                                                                                             │                                                                                          │
│ │                 │   evaluation_cost=0.0005549800000000001,                                                                                     │                                                                                          │
│ │                 │   dataset_alias=None,                                                                                                        │                                                                                          │
│ │                 │   dataset_id=None                                                                                                            │                                                                                          │
│ │                 )                                                                                                                              │                                                                                          │
│ │  valid_scores = 1                                                                                                                              │                                                                                          │
│ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                          │
│                                                                                                                                                                                                                                             │
│ /Users/fanbin/miniconda3/envs/case_rag/lib/python3.12/site-packages/deepeval/test_run/test_run.py:951 in save_test_run_locally                                                                                                              │
│                                                                                                                                                                                                                                             │
│    948 │   │   │   new_test_filename = datetime.datetime.now().strftime(                        ╭─────────────────────────────────────── locals ────────────────────────────────────────╮                                                   │
│    949 │   │   │   │   "%Y%m%d_%H%M%S"                                                          │      local_folder = './data'                                                          │                                                   │
│    950 │   │   │   )                                                                            │ new_test_filename = '20251204_232229'                                                 │                                                   │
│ ❱  951 │   │   │   os.rename(self.temp_file_path, new_test_filename)                            │              self = <deepeval.test_run.test_run.TestRunManager object at 0x110bf35f0> │                                                   │
│    952 │   │   │   if not os.path.exists(local_folder):                                         ╰───────────────────────────────────────────────────────────────────────────────────────╯                                                   │
│    953 │   │   │   │   os.mkdir(local_folder)                                                                                                                                                                                               │
│    954 │   │   │   │   shutil.copy(new_test_filename, local_folder)                                                                                                                                                                         │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: '.deepeval/.temp_test_run_data.json' -> '20251204_232229'
```

I can see a .deepeval directory created in the same folder, and inside it are three files: .deepeval_telemetry.txt, .deepeval-cache.json, and .latest_test_run.json.

This is just a simple test, but it's still throwing an error. Is it due to the wrong way of running it, or could there be issue? I'd really appreciate your help. 

Thank you!

**To Reproduce**
run the code

**Expected behavior**
no error happen


**Desktop (please complete the following information):**
 - OS: macOS 
 - Version: sequoia 15.4

**Additional context**
deepeval 3.7.4


Thank you @fb9988 for bringing this to my attention. I will take a look.

@fb9988 In your issue description you show an export containing a double quote:

> export DEEPEVAL_RESULTS_FOLDER="./data

Can you check to see if there was a double quote when you exported `DEEPEVAL_RESULTS_FOLDER`. If there was, can you try running the test again, but this time without the double quote. You can just set the param on the cli for convenience.

Something akin to this
```
dustfinger@primus:~/dev/repros$ DEEPEVAL_RESULTS_FOLDER=./data python scripts/test_repro-2352.py 
```

@fb9988 Actually, I was able to reproduce it. I will look into a fix. I suspect that double quote was just a typo in the description.

@fb9988,

I just took another look at your example and I see what the issue is. You are accidentally mixing two orchestration layers at once.

The `deepeval test run ...` path is meant for test files that call `assert_test` without manually calling `evaluate`. If you want to manually call evaluate, then you can run the script like this:

```
dustfinger@primus:~/dev/repros$ DEEPEVAL_RESULTS_FOLDER=./data python scripts/test_repro-2352.py 
```

To use the `deepeval test run` You can simply modify your script like this:
```
from deepeval.metrics import FaithfulnessMetric
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.models.llms import GPTModel
from deepeval.evaluate import evaluate, DisplayConfig

judge = GPTModel(model="gpt-4o-mini", temperature=1.4)

metric = FaithfulnessMetric(
    threshold=0.5,
    include_reason=True,
    model = judge
)

test_case = LLMTestCase(
    input="what is 'gfsda'?",
    actual_output="Another term for sneezing.",
    retrieval_context=["'gfsda' has no meaning; it's a randomly created word"]
)
assert_test(test_case, [metric]) # <- call assert not evaluate when using deepeval test run ...
```

I will bring this up at our next meeting though. Even though the cli was not intended to work with `evaluate` we should probably handle this scenario better than we do. Thank you for bringing it to my attention!

I hope that helps. Have a great weekend!

@fb9988 I thought of a way to make this work nicely. I am going to make a fix and consider this a bug. When the PR merges you will be able to run your script with python or our cli and either way it will work as expected.

@trevor-cai Thank you for your in-depth analysis and detailed explanation.I will use the correct approach in the code.