scisave 1.6.0
SciSave - Serialization for Scientific Data
License: BSD-2-Clause
Requires Python: >=3.9
Keywords: Python, Serialization, Deserialization, JSON, YAML, MessagePack, Pickle, NumPy, Schema

# SciSave - Serialization for Scientific Data

> * **Repository: [github.com/otvam/scisave](https://github.com/otvam/scisave)**
> * **PyPi: [pypi.org/project/scisave](https://pypi.org/project/scisave)**
> * **Conda: [anaconda.org/conda-forge/scisave](https://anaconda.org/conda-forge/scisave)**

## Summary

**SciSave** is a **Python serialization/deserialization** module:
* Specially targeted for **scientific applications**.
* Load and write **JSON/MessagePack/Pickle data files**.
* Load **JSON/YAML configuration files**.
* Validate data with **JSON schemas**.

For **YAML files**, the following **custom extensions** are used:
*  Parse relative paths with respect to the YAML file (`!path`).
*  Include other YAML files into the YAML file (`!include`).
*  Evaluate a Python literal using literal_eval (`!eval`).
*  Substitute YAML strings with values from environment variables (`!env`).
*  Substitute YAML strings with values from a provided dictionary (`!sub`).
*  Merge a list of dicts (`!merge_dict`).
*  Merge a list of lists (`!merge_list`).

For **JSON files**, the following **custom extensions** are used:
* Allows the serialization/deserialization of complex numbers (`__complex__`).
* Allows the serialization/deserialization of NumPy arrays (`__numpy__`).
* Allows the serialization/deserialization as/from text and gzip files 

For **MessagePack files**, the following **custom extensions** are used:
* Allows the serialization/deserialization of complex numbers (`__complex__`).
* Allows the serialization/deserialization of NumPy arrays (`__numpy__`).

For **JSON schemas**, the following **custom extensions** are used:
* Handling NumPy types (integer, floating, and complex). 
* Handling NumPy multidimensional arrays.

The following **file extensions** are used:
* `.yaml, .yml` - for YAML files
* `.json, .js` - for JSON text files
* `.gz, .gzip` - for JSON gzip files
* `.mpk, .msg", .msgpack` - for MessagePack files
* `.pck, .pkl, .pickle` - for Pickle files

The JSON/YAML/MessagePack files with the custom extensions are still valid files.
Pickle/MessagePack is typically faster than JSON for very large data files.

SciSave is written in Python (NumPy, PyYAML, msgpack, and jsonschema are the only dependencies).
SciSave is respecting **semantic versioning** (starting from version 1.4).

## Warning

* Pickling data is not secure.
* Only load pickle files that you trust.

## Example

An example is located in the `example` folder of the repository:
* `run_data.py` contains an example file for the loader/dumper
* `run_bench.py` contains a simple benchmark for the different formats
* `config_main.yaml` YAML configuration file with custom extensions
* `config_include.yaml` YAML configuration file for include extension
* `config_schema.yaml` YAML file containing the JSON schema definition
* `dump.json` JSON text file for testing data dumping/loading
* `dump.gz` JSON gzip file for testing data dumping/loading
* `dump.pickle` Pickle file for testing data dumping/loading

## Project Links

* Repository: https://github.com/otvam/scisave
* Releases: https://github.com/otvam/scisave/releases
* Tags: https://github.com/otvam/scisave/tags
* Issues: https://github.com/otvam/scisave/issues
* PyPi: https://pypi.org/project/scisave
* Conda: https://anaconda.org/conda-forge/scisave

## Author

* **Thomas Guillod**
* Email: guillod@otvam.ch
* Website: https://otvam.ch

## Copyright

> (c) 2023 - Thomas Guillod
> 
>  BSD 2-Clause "Simplified" License