Repository: confident-ai/deepeval
Issue: JSON Corruption in Parallel Test Runs - "Expecting ',' delimiter"
Root cause: Resource Mishandling

## Problem Statement

When running tests with multiple workers in Kubernetes environments with network-attached storage, the following error occurs, causing the test process to fail:

```
Warning: Could not update test run on disk: Expecting ',' delimiter
```

This results in test execution failure and incomplete test runs.

## Environment

- **Platform**: Kubernetes with network-attached storage (NFS)
- **Workers**: 16 parallel workers
- **Test Count**: 2000+ tests
- **Test Duration**: 30+ seconds per test

## Observed Behavior

**Works locally**: Single machine with local disk storage - no issues  
**Fails in K8s**: Network-attached storage with parallel workers - process fails

Example output:
```bash
$ deepeval test run evals -n 12
Running 2000 tests...
Warning: Could not update test run on disk: Expecting ',' delimiter
...
Process fails/terminates
```

## Root Cause

The error is a JSON parsing exception caused by race conditions in concurrent file writes. The issue occurs because **file locks control when processes can access files, but do not guarantee that data has been written to disk**.

### Current Code Flow

When a test completes, the result is saved to `.deepeval/.temp_test_run_data.json`:

```python
# Worker 1 holds exclusive lock
with portalocker.Lock(file, mode="r+", flags=LOCK_EX) as f:
    json.dump(data, f)
    # Lock released here - but data is still in Python/OS buffers!

# Worker 2 immediately acquires lock
with portalocker.Lock(file, mode="r+", flags=LOCK_EX) as f:
    data = json.load(f)  # Reads incomplete/stale data from disk
    # JSONDecodeError: Expecting ',' delimiter
```

### Why This Happens

When `json.dump()` is called, data flows through multiple buffer layers:

```
json.dump() → Python buffer → OS buffer → Network cache → Physical disk
```

Without explicit flushing:
1. Worker 1 writes data to Python buffer and releases the lock
2. Data has not yet reached the physical disk (still in buffers)
3. Worker 2 acquires the lock and reads from disk
4. Worker 2 gets old/incomplete JSON data → parsing error
5. Exception propagates, causing process failure

This problem is exacerbated in network storage environments where the write path includes additional network and NFS server caching layers.

## Technical Details

### Affected Code Locations

**File: `deepeval/test_run/test_run.py`**

1. **Line ~411** - `TestRun.save()` method:
```python
def save(self, f):
    try:
        body = self.model_dump(by_alias=True, exclude_none=True)
    except AttributeError:
        body = self.dict(by_alias=True, exclude_none=True)
    json.dump(body, f, cls=TestRunEncoder)
    # Missing: f.flush() and os.fsync()
    return self
```

2. **Line ~519** - `save_test_run()` method:
```python
json.dump(wrapper_data, file, cls=TestRunEncoder)
# Missing: file.flush() and os.fsync()
```

3. **Line ~533** - `save_final_test_run_link()` method:
```python
json.dump({LATEST_TEST_RUN_LINK_KEY: link}, file)
# Missing: file.flush() and os.fsync()
```

**File: `deepeval/test_run/cache.py`**

4. **Line ~92** - `CachedTestRun.save()` method:
```python
def save(self, f):
    try:
        body = self.model_dump(by_alias=True, exclude_none=True)
    except AttributeError:
        body = self.dict(by_alias=True, exclude_none=True)
    json.dump(body, f, cls=CustomEncoder)
    # Missing: f.flush() and os.fsync()
    return self
```

## Proposed Fix

Add explicit buffer flushing and disk synchronization after all `json.dump()` operations:

```python
def save(self, f):
    try:
        body = self.model_dump(by_alias=True, exclude_none=True)
    except AttributeError:
        body = self.dict(by_alias=True, exclude_none=True)
    json.dump(body, f, cls=TestRunEncoder)
    f.flush()              # Force Python buffer to OS
    os.fsync(f.fileno())   # Force OS buffer to physical disk
    return self
```

This ensures that when the file lock is released, all data has been physically written to disk and is available for the next process to read.

### Why Both Are Needed

- **`f.flush()`**: Flushes Python's internal buffers to the operating system
- **`os.fsync(f.fileno())`**: Forces the operating system to write data to physical storage

The `fsync()` call is particularly critical for network-attached storage (NFS) where multiple caching layers exist between the application and physical disk.

## Implementation

Apply the fix to all four locations listed above. For example:

**deepeval/test_run/test_run.py**:
```python
# Line 411 - TestRun.save()
json.dump(body, f, cls=TestRunEncoder)
f.flush()
os.fsync(f.fileno())

# Line 519 - save_test_run()
json.dump(wrapper_data, file, cls=TestRunEncoder)
file.flush()
os.fsync(file.fileno())

# Line 533 - save_final_test_run_link()
json.dump({LATEST_TEST_RUN_LINK_KEY: link}, file)
file.flush()
os.fsync(file.fileno())
```

**deepeval/test_run/cache.py**:
```python
# Line 92 - CachedTestRun.save()
json.dump(body, f, cls=CustomEncoder)
f.flush()
os.fsync(f.fileno())
```

## Expected Results

After applying the fix:
- No "Expecting ',' delimiter" errors
- Test runs complete successfully
- Consistent behavior across local and network storage
- Minimal performance impact (~1-2ms per write operation)

## References

This pattern follows industry best practices for concurrent file access:
- PostgreSQL calls `fsync()` after transaction commits
- SQLite uses synchronous writes by default
- Git flushes and syncs after writing objects
- Any system with concurrent file access requires explicit synchronization

The fix ensures data durability and consistency in multi-process environments, especially critical for network-attached storage where write latencies are higher.


Hey @agrudkow thanks for bringing this up, @trevor-cai can you please help @agrudkow out, thank you!

Hi @agrudkow, I really appreciate the level of detail that you put into this github issue. A fix is on its way.