hdf5vault 0.3.3
A parallel HDF5-based archiving tool.
License: MIT
Requires Python: >=3.8

# HDF5Vault

HDF5Vault is an MPI-based tool for efficiently concatenating and compressing very large numbers of files into a small number of [HDF5](https://www.hdfgroup.org/solutions/hdf5/)-based archive files. 
It is designed for high-throughput archive generation on parallel file systems such as [GPFS](https://en.wikipedia.org/wiki/GPFS), [Lustre](https://en.wikipedia.org/wiki/Lustre_(file_system)) or [VAST Data](https://en.wikipedia.org/wiki/VAST_Data), where metadata operations and I/O parallelism are critical for performance.

HDF5Vault compresses multiple input files concurrently and pipelines file scanning, compression, and archive writing across MPI processes. While files are being compressed, data are written in parallel to multiple archive files, maximizing both CPU and storage utilization. The tool is implemented in Python and uses [Blosc2](https://www.blosc.org/python-blosc2/) for fast and efficient compression. 

## Archive layout

Within each archive, every input file is stored as one or more HDF5 datasets of byte type. The group and subgroup hierarchy mirrors the directory structure of the input data, and each dataset is named after the original file, with an additional extension indicating the compression format (e.g., `.blosc2`).

Files larger than a configurable size threshold (default: 1 GB) are automatically split into multiple datasets. This chunking ensures compatibility with Blosc2 size limits and reduces memory usage.

## Use cases

HDF5Vault was originally developed to handle on the order of 10<sup>5</sup> to 10<sup>7</sup> small files (typically a few MBs each) generated by high-throughput acquisition systems, such as the [Yokogawa CellVoyager](https://www.yokogawa.com/ch/solutions/products-and-services/life-science/high-content-analysis/) microscope. 

In a representative production workload, several hundred thousand files with a total uncompressed size of 6.2 TB were compressed into eight HDF5 archives (2 TB total) in approximately 30 minutes on a compute cluster connected to a VAST storage system. 

## Limitations

HDF5Vault is not intended for archiving data stored on a single local disk. Its performance benefits rely on parallel I/O and concurrent metadata operations provided by distributed file systems. In addition, compression is applied independently at the file level; in workloads where strong redundancy exists across files, tools that exploit cross-file similarity may achieve better compression ratios.

## Downstream processing

Archives created with HDF5Vault can be unpackaged efficiently in parallel.  In some workflows, they may also be accessed directly without unpacking － for example, during the generation of OME-Zarr datasets (see the example [workflow repository](https://github.com/fmi-faim/faim-ipa/tree/hdf5-to-zarr)).

## Installation

HDF5Vault is distributed as a PyPI package and requires Python ≥ 3.8 with the following dependencies:

- `h5py`
- `mpi4py`
- `blosc2`

Installation can be performed using either a standard Python virtual environment or 
[Pixi](https://pixi.prefix.dev).

### Using Python virtual environments and pip
If Python>=3.8, `venv` and an MPI implementation are available on your system, HDF5Vault can be installed as follows:

    python3 -m venv hdf5vault
    source hdf5vault/bin/activate
    pip install hdf5vault

### Using Pixi
Pixi can be used to create a fully reproducible environment, including MPI-related dependencies:

    pixi init hdf5vault
    cd hdf5vault
    pixi add openmpi
    pixi add python=3.12 pip
    pixi add h5py mpi4py python-blosc2 pandas humanize
    pixi run pip install hdf5vault

## Usage

All HDF5Vault tools (create, verify, unpack) are executed as an MPI programs.  To create an archive, use:

    mpirun -n NUM_TASKS hdf5vault_create \
                        DIRECTORY_TO_BE_ARCHIVED \
                        ARCHIVE_BASENAME \
                        -c COMPRESSION_LEVEL \
                        -t THREADS \
                        -w NUM_WRITERS
                        -s CHUNKSIZE

The contents of the `DIRECTORY_TO_BE_ARCHIVED` are packed into NUM_WRITERS archive files named `ARCHIVE_BASENAME_X.h5`, where `X` denotes the archive index. If only one writer is specified, just one archive file with the name `ARCHIVE_BASENAME.h5` is created.

The compression level ranges from 1 (fastest) to 9 (highest compression) and is passed directly to Blosc2. Each compression process uses `THREADS` threads to compress the data.

At least `NUM_WRITERS + 3` MPI processes are required (with a minimum of 4 when `NUM_WRITERS = 1`). MPI ranks are assigned to distinct roles:

- one for directory scanning and scheduling compression,
- one for scheduling archive writes,
- `NUM_WRITERS` for writing (one per archive file),
- and the remaining ranks for compression.


### Example

    mpirun -n 20 hdf5vault_create data data_archive -c 7 -t 4 -w 4 

This command creates four archive files (`data_archive_0.h5` to `data_archive_3.h5`) 
using 20 MPI processes: four wrtiers, two scheduling/scanning ranks, and fourteen compression workers.

The maximum dataset size is controlled with `-s` (default 1 GB).  The maximum chunk size supported by Blosc2 is 2.1 GB.


## Archive verification:
To verify the integrity of an archive against the original directory, use

    mpirun -n $NUM_TASKS hdf5vault_check \
                         -d DIRECTORY_THAT_WAS_ARCHIVED \
                         -f ARCHIVE_FILE(S)
                        [-j JSON_SUMMARY_FILE]

The verification tool recomputes checksums for each file and compares the archived data with the on-disk originals.  The result of the verification (pass or fail) or written to standard output.  When the `-j` option is used, MD5 checksums for both archived and original files are written to a JSON file.

Since one MPI rank is dedicated to directory scanning, at least two MPI processes are required.


## Unpacking archives
Archives can be unpacked in parallel using:

    mpirun -n $NUM_TASKS hdf5vault_unpack \
                         -f ARCHIVE_FILE(S) \
                         -d DESTDIR 

If `DESTDIR` is not specified, the base archive name (without the numeric suffix) is used as destination directory (e.g., `data_archive` in the above example).

## Logging and verbosity

In all HDF5Vault tools, debugging information can be turned on with the `-e` flag. The flag `-q` suppresses all output except errors and warnings.

## Archive format metadata
Each HDF5Vault archive includes an attribute named `__description__` that contains basic information about the archive format. The definition can be found at [src/hdf5vault/hdf5_archive_info.py](src/hdf5vault/hdf5_archive_info.py).

## Changelog

See [Changelod.md](changelog.md) for a record of changes, including new features, bug fixes, and compatibility notes.