sr64/poc-hdf5-hyper-deserialize-rank-stack-overflow-cwe121

PoC: stack buffer overflow in HDF5 H5S__hyper_deserialize via unchecked selection rank (CWE-121/787)

Opening a crafted HDF5 file (.h5) with a virtual dataset whose source hyperslab selection encodes a rank > 32 overflows a fixed-size stack array during deserialization, on the normal H5Dopen path.

Root cause

src/H5Shyper.c (H5S__hyper_deserialize, ~line 4330-4335):

UINT32DECODE(pp, rank);                                  // rank from the file (attacker), NO bound
if (!*space) {
    memset(dims, 0, (size_t)rank * sizeof(dims[0]));     // dims is hsize_t dims[H5S_MAX_RANK] (32 elems / 256 B)
    if (H5S_set_extent_simple(tmp_space, rank, dims, NULL) < 0) ...   // rank check runs AFTER the memset
}

dims is a 32-element (H5S_MAX_RANK) hsize_t stack array. The hyperslab selection rank is decoded from the file with no bound, and memset zeroes rank * sizeof(hsize_t) bytes into the 256-byte stack array before the (later) H5S_set_extent_simple rank check. The point-selection deserialization (H5Spoint.c) validates rank <= H5S_MAX_RANK before its memset — a deliberate pattern that is simply missing from the hyperslab path.

Reachability (normal H5Dopen of a virtual dataset)

H5Dopen → H5D__virtual_load_layout (H5Dvirtual.c:2719, the dataset-open path) → H5S_SELECT_DESERIALIZE(&ent->source_select, ...) (H5Dvirtual.c:843) where ent->source_select is freshly NULL → inside H5S__hyper_deserialize the if(!*space) branch is taken → the memset runs with the attacker-controlled rank. No explicit selection API call is needed — just opening the crafted VDS.

Reproduce (AddressSanitizer)

gcc -fsanitize=address -g -O0 hdf5_hyper_deserialize_rank_overflow_harness.c -o poc && ./poc
# dims[H5S_MAX_RANK=32] = 256 bytes; attacker rank=64 -> memset writes 512 bytes
# ==ERROR: AddressSanitizer: stack-buffer-overflow  WRITE of size 512 ... (memset)

Affected

• Repository: HDFGroup/hdf5 (commit 2b0fd612f2e7db858a3cb3188d94544dcd209d3e)

Fix

Validate rank <= H5S_MAX_RANK immediately after UINT32DECODE(pp, rank) (mirroring the H5Spoint.c check), before the memset / H5S_set_extent_simple.