johnsmithle/modelscan-numpy2-scanner-failure

PoC — ModelScan's NumPy scanner is dead on numpy ≥ 2.0

Security research artefact for a huntr Model File Vulnerability report. Not a usable model. Do not load this file outside a disposable sandbox.

What is in this repo

embeddings.npy — 431 bytes, sha256 c021a6fb90c674253a10cebfca50e16cb723fabf7e6b3ae39370fc47dc2dfa7f

An ordinary object-dtype NumPy array. .npy stores object arrays by embedding a pickle stream after the header, so np.load(..., allow_pickle=True) unpickles — and executes — it. The payload is inert: it creates /tmp/huntr_exec/NPY and nothing else. It is a stand-in for arbitrary code.

The issue

modelscan/tools/picklescanner.py:228-237:

elif magic == np.lib.format.MAGIC_PREFIX:
    # .npy file
    version = np.lib.format.read_magic(stream)
    np.lib.format._check_version(version)                             # removed in numpy 2.0
    _, _, dtype = np.lib.format._read_array_header(stream, version)   # also removed

    if dtype.hasobject:
        return scan_pickle_bytes(model, settings, scan_name, True, stream.tell())
    else:
        return ScanResults([], [], [])

Both _check_version and _read_array_header are private numpy helpers and were removed in numpy 2.0. On numpy ≥ 2 the first call raises AttributeError. ModelScan catches it, records it under --- Errors ---, and still prints:

--- Summary ---
 No issues found! 🎉

The exception is raised before the dtype.hasobject branch, so every .npy file is affected — benign and malicious alike. The NumPy scanner is effectively disabled.

pyproject.toml declares numpy = ">=1.24.3" with no upper bound, so a fresh pip install modelscan resolves numpy 2.x and ships this state by default.

Reproduction

Same modelscan 0.8.8, same file, only numpy differs:

# numpy 1.x — scanner works
pip install modelscan "numpy<2"
modelscan -p embeddings.npy
#   Total Issues: 1
#   - Severity: CRITICAL
#   - Description: Use of unsafe operator 'exec' from module 'builtins'

# numpy 2.x — scanner is dead
pip install modelscan "numpy>=2"
modelscan -p embeddings.npy
#    No issues found! 🎉
#   --- Errors ---
#   module 'numpy.lib.format' has no attribute '_check_version'

Confirming the file really executes (run in a sandbox):

import numpy as np
np.load("embeddings.npy", allow_pickle=True)
# -> /tmp/huntr_exec/NPY now exists

A benign np.save("x.npy", np.arange(10)) triggers the same AttributeError, which confirms the failure is universal rather than payload-specific.

Measured results

modelscan	numpy	verdict
0.8.8	1.26.4	Total Issues: 1 — CRITICAL
0.8.8	2.4.6	No issues found (scanner raised)

Impact

This is not an evasion an attacker has to craft — it is a silent regression that disables an entire scanner class by default. Anyone who installed ModelScan after numpy 2.0 shipped (June 2024) has had zero .npy coverage while the tool reported success. The --- Errors --- section is easy to miss, and CI normally keys on the exit status or issue count rather than on stderr noise.

Scope note

Executing a malicious .npy still requires the loading code to pass np.load(..., allow_pickle=True); numpy has defaulted that to False since 1.16.5. So this is not unauthenticated RCE on its own. It is a complete failure of the control that exists to catch exactly the allow_pickle=True case, which remains common for object arrays, legacy datasets and research pipelines.

Suggested fix

Stop calling numpy private APIs — numpy.lib.format.read_array_header_1_0 / read_array_header_2_0 are public, or parse the dtype from the header dict. Separately: cap the dependency (numpy>=1.24.3,<3), and make a scanner that raises during a scan produce a failing or inconclusive result instead of being folded into "No issues found".