Instructions to use pixelated404/hdf5poc with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Keras
How to use pixelated404/hdf5poc with Keras:
# Available backend options are: "jax", "torch", "tensorflow". import os os.environ["KERAS_BACKEND"] = "jax" import keras model = keras.saving.load_model("hf://pixelated404/hdf5poc") - Notebooks
- Google Colab
- Kaggle
Technical Background
keras.models.load_model() supports loading Keras models from .h5 and .keras
files. Internally, weight datasets are read through safe_get_h5_dataset(), which
implements a decompression bomb guard that checks whether the declared dataset
size (based on shape and dtype) exceeds a 4 GiB floor and is more than 1000 times
the stored size on disk.
The guard logic is:
declared_bytes = math.prod(dataset.shape) * dataset.dtype.itemsize
stored_bytes = dataset.id.get_storage_size()
if declared_bytes > _H5_DATASET_BOMB_FLOOR_BYTES and declared_bytes > 1000 * stored_bytes:
raise ValueError(...)
The constant _H5_DATASET_BOMB_FLOOR_BYTES is set to 1 << 32 (4 GiB).
Any dataset whose declared_bytes is at or below this floor is never
checked against the stored-disk size, regardless of the expansion ratio.
A dataset with shape (1073741823,) and dtype float32 has a declared size
of 4,294,967,292 bytes (4 GiB − 4 bytes), which is below the 4 GiB floor.
When stored as a fill-value-only dataset with no chunks written, the stored
size on disk is zero bytes. The expansion ratio is approximately 3 million to one,
but the guard does not evaluate it.
Callers of safe_get_h5_dataset() pass the returned dataset handle directly
to np.array() or np.asarray(), which performs the full allocation before
any later shape validation.
Tested Versions
- Python 3.12.10
- Keras 3.15.0
- TensorFlow 2.21.0
- h5py 3.14.0
Reproducing
Both .h5 and .keras variants are provided.
Legacy HDF5 format (.h5)
keras.models.load_model("bomb.h5")
Native Keras format (.keras)
keras.models.load_model("bomb.keras")
Using the verification script
python verify_poc.py
This script prints package versions, attempts both loads, and reports which
stage triggered a MemoryError or ValueError.
Expected Behavior
When the model file is loaded, the dataset guard evaluates but does not reject
the dataset because its declared size (under 4 GiB) is below the guard floor.
The process then attempts to allocate memory for the declared shape. On systems
with insufficient available memory, this raises MemoryError. On systems with
sufficient memory, the allocation succeeds and a subsequent shape validation
raises ValueError because the bomb weight shape does not match the layer
architecture.
In both cases the memory allocation occurs before any model validation can reject the file.
- Downloads last month
- -