renaissancefieldlite/tris-tokenizers-json-regex-panic-poc

Tris tokenizers JSON regex panic PoC

This repository is a minimal proof-of-concept for a Huntr model-file-format security report.

It contains a serialized tokenizer JSON file with a regex-backed Split pre-tokenizer:

{
  "Regex": "(a+)+$"
}

When loaded with Hugging Face tokenizers through Tokenizer.from_file(...) and used on short crafted input, the regex execution path can surface a Rust/PyO3 panic as PanicException.

Files

• malicious_tokenizer.json - serialized tokenizer artifact.
• run_poc.py - local reproduction script with child-process isolation.
• requirements.txt - tested Python dependency.

Reproduction

python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
python run_poc.py

Expected behavior:

• short inputs complete normally;
• crafted inputs around length 24 and 28 trigger PanicException: Onig: Regex search error: retry-limit-in-match over.

Boundary

This PoC is for local, responsible security triage only. It does not perform network access, credential access, or remote exploitation. The demonstrated impact is availability / worker stability for systems that load tokenizer JSON artifacts from untrusted model or plugin sources.