Repository: confident-ai/deepeval
Issue: NoneType hyperparameters Error Due to File Loading Logic Flaw
Root cause: Algorithmic Error

### **Issue Summary**

  Error: 'NoneType' object has no attribute 'hyperparameters'Root Cause: Logic flaw in TestRunManager.get_test_run() method that sets test_run to None when disk loading fails, despite
  having successfully created a valid test_run object moments before.

###   **Environment**

  - Deepeval Version: [Current installed version]
  - Python Version: 3.12
  - Operating System: macOS Darwin 24.6.0
  - Use Case: RAG metric evaluation 

###   **Problem Description**

The error occurs during RAG metric evaluation when deepeval attempts to access test_run.hyperparameters on a None object. Investigation reveals a critical logic flaw in the test run management system.

###   **Root Cause Analysis**

1.  Default Configuration Issue

  File: deepeval/evaluate/configs.pyLine 31: write_cache: bool = True (default)

  The default CacheConfig enables disk caching, which triggers the problematic file loading logic.

2.  save_to_disk Assignment

  File: deepeval/evaluate/execute.pyLine 121: test_run_manager.save_to_disk = cache_config.write_cache

  This sets save_to_disk = True by default, activating the file loading pathway.

3. Logic Flaw in get_test_run()

  File: deepeval/test_run/test_run.pyMethod: TestRunManager.get_test_run()Lines 438-457:

  def get_test_run(self, identifier: Optional[str] = None):
      if self.test_run is None:
          self.create_test_run(identifier=identifier)  # ✅ Creates valid TestRun

      if self.save_to_disk:  # ✅ True by default
          try:
              with portalocker.Lock(
                  self.temp_file_path,
                  mode="r",
                  flags=portalocker.LOCK_SH | portalocker.LOCK_NB,
              ) as file:
                  self.test_run = self.test_run.load(file)
          except (
              FileNotFoundError,
              portalocker.exceptions.LockException,
          ) as e:
              print(f"Error loading test run from disk: {e}", file=sys.stderr)
              self.test_run = None  # ❌ OVERWRITES valid test_run with None

      return self.test_run  # ❌ Returns None

  4. Error Propagation

  File: deepeval/evaluate/execute.pyLine 169: test_case, test_run.hyperparameters

  The None test_run causes 'NoneType' object has no attribute 'hyperparameters' when accessed.

###   **Detailed Execution Flow**

  1. Initialization: TestRunManager() sets self.test_run = None, self.save_to_disk = False
  2. Configuration: execute_test_cases() sets save_to_disk = True (default cache config)
  3. Test Run Creation: get_test_run() calls create_test_run() which successfully creates a valid TestRun object
  4. File Loading Attempt: Since save_to_disk = True, the method attempts to load from disk
  5. File Loading Failure: FileNotFoundError or LockException occurs (temp file doesn't exist or is locked)
  6. Logic Error: Instead of keeping the valid test_run, the code sets self.test_run = None
  7. Error Propagation: None is returned and later code fails when accessing test_run.hyperparameters

###   **Expected Behavior**

  When disk loading fails, the method should retain the successfully created test_run object rather than overwriting it with None. The logic should be:

  if self.save_to_disk:
      try:
          # Attempt to load from disk
          with portalocker.Lock(...) as file:
              loaded_test_run = self.test_run.load(file)
              self.test_run = loaded_test_run  # Only overwrite if successful
      except (FileNotFoundError, portalocker.exceptions.LockException) as e:
          # Keep the existing valid test_run, don't set to None
          print(f"Warning: Could not load test run from disk: {e}", file=sys.stderr)
          # self.test_run remains the valid created object

###   **Reproduction Steps**

  1. Use deepeval with default caching enabled (write_cache=True)
  2. Run any metric evaluation (e.g., RAG evaluation with FaithfulnessMetric)
  3. Ensure deepeval temp directory has file access issues or no existing cache files
  4. Error occurs: 'NoneType' object has no attribute 'hyperparameters'

###   **Current Workarounds**

  Option 1: Disable Write Cache

  from deepeval.evaluate.configs import CacheConfig
  cache_config = CacheConfig(write_cache=False)
  results = deepeval_evaluate(test_cases, metrics, cache_config=cache_config)

  Option 2: Environment Variable

  export DEEPEVAL_FILE_SYSTEM=READ_ONLY

  Option 3: Fix File Permissions

  Ensure the deepeval temp directory has proper read/write permissions.

###   **Suggested Fix**

  File: deepeval/test_run/test_run.pyMethod: TestRunManager.get_test_run()

  ```
def get_test_run(self, identifier: Optional[str] = None):
      if self.test_run is None:
          self.create_test_run(identifier=identifier)

      if self.save_to_disk:
          try:
              with portalocker.Lock(
                  self.temp_file_path,
                  mode="r",
                  flags=portalocker.LOCK_SH | portalocker.LOCK_NB,
              ) as file:
                  # Only overwrite if loading succeeds
                  loaded_test_run = self.test_run.load(file)
                  self.test_run = loaded_test_run
          except (
              FileNotFoundError,
              portalocker.exceptions.LockException,
          ) as e:
              # Log warning but preserve the valid test_run object
              print(f"Warning: Could not load test run from disk: {e}", file=sys.stderr)
              # Do NOT set self.test_run = None here

      return self.test_run
```

###   **Impact**

  This bug affects any deepeval usage with default caching when:
  - Temp files don't exist (first run)
  - File permission issues occur
  - File locking conflicts arise
  - Disk I/O errors happen

  The error is misleading as it suggests a model or hyperparameters issue when it's actually a file system/caching problem.

###   **Priority**

  High - This is a critical bug that breaks basic functionality with default configuration and provides misleading error messages that make debugging difficult.

@nr-shashankssingh thanks for bringing it up, @BloggerBust could you help look into this, thank you!

@nr-shashankssingh Thank you for putting so much effort into documenting this issue. It made it really straight forward for me to reproduce the issue, write tests and resolve it quickly. Your time and effort was appreciated.