Repository: evidentlyai/evidently
Issue: Workspace observer thread dies because of JSONDecodeError while saving large snapshots
Root cause: Resource Mishandling

I'm getting `JSONDecodeError`s in the workspace observer thread while saving snapshots in evidently UI.
The problem appears only for large snapshots (~1Mb or more).
If I understand the library code correctly, on creating a new snapshot file we immediatley try to load the snapshot into memory to update a certain state.
For large snapshots it fails because at the moment we haven't yet finished writing the snapshot to disk, hence the `JSONDecodeError`.

Having run into the error, the observer thread dies and no longer updates the aforementioned state.
At least for me, this implies **no dashboard updates until the evidently UI app is restarted.**

I run evidently 0.7.9 locally on python 3.11 (`python:3.11.11-slim` image), using local storage.

Here is a reproducer which doesn't use evidently UI (the function `create_local_project_manager` starts the observer thread in the evidently UI app, I think).

```python
from pathlib import Path

import numpy as np
import pandas as pd
from evidently.ui.workspace import Workspace
from evidently.presets import DataSummaryPreset
from evidently import DataDefinition, Dataset, Report
from evidently.ui.service.storage.local import create_local_project_manager


script_path = Path(__file__).parent.resolve()
ws_path = script_path / 'ws'
ws_path.mkdir(exist_ok=True)

ws = Workspace(path=str(ws_path))
project = ws.create_project(name='test_project')
project_id = project.id

n_rows = 1000
n_cols = 10

col_names = [f'col_{i}' for i in range(n_cols)]

data = pd.DataFrame(
    data=np.random.randn(n_rows, n_cols),
    columns=col_names
)

data_definition = DataDefinition(
    numerical_columns=col_names
)

ref_dataset = Dataset.from_pandas(
    data=data.iloc[:n_rows // 2],
    data_definition=data_definition
)

cur_dataset = Dataset.from_pandas(
    data=data.iloc[n_rows // 2:],
    data_definition=data_definition
)

report = Report(
    metrics=[DataSummaryPreset(columns=col_names)],
    include_tests=True
)

snapshot = report.run(
    current_data=cur_dataset,
    reference_data=ref_dataset
)

print('Saving snapshot')
project_manager = create_local_project_manager(path=str(ws_path), autorefresh=True)
ws.add_run(project_id=project_id, run=snapshot)
```

This is what the script produces after running in the directory `/app`:

```
Saving snapshot
Observer for '/app/ws' started
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/site-packages/watchdog/observers/api.py", line 213, in run
    self.dispatch_events(self.event_queue)
  File "/usr/local/lib/python3.11/site-packages/watchdog/observers/api.py", line 391, in dispatch_events
    handler.dispatch(event)
  File "/usr/local/lib/python3.11/site-packages/evidently/ui/service/storage/local/watcher.py", line 32, in dispatch
    self.on_snapshot_event(event)
  File "/usr/local/lib/python3.11/site-packages/evidently/ui/service/storage/local/watcher.py", line 116, in on_snapshot_event
    self.state.reload_snapshot(project, sid)
  File "/usr/local/lib/python3.11/site-packages/evidently/ui/service/storage/local/base.py", line 147, in reload_snapshot
    model = parse_obj_as(SnapshotModel, json.load(f))
                                        ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```

The exact error message (line and column numbers) may vary from run to run.

I also a have a reproducer in Docker that runs evidently UI, tell me if you need it.


Thanks a lot for reporting this @yupopov ! I've added simple try-except to ignore `JSONDecodeError`. It should work since when last snapshot chunk is written it also triggers an fs event and snapshot will be reloaded. If you know a way to wait for the file to be completely written before trying to read it - let me know