Sovereign Byte Firewall — Transformer backbone (gs75000)
A 528K-parameter causal Transformer trained to predict raw network traffic one byte at a time. It is not a classifier: it learns the structural grammar of benign traffic, and anomalies are scored by surprise — how improbable each byte is given the bytes before it. Attacks it has never seen produce improbable byte sequences.
📄 Paper: 10.5281/zenodo.21546151 💻 Code: github.com/TheSPST/sovereign-byte-firewall (MIT)
This is the checkpoint reported as the Transformer arm of the paper, and the one the reference daemon deploys by default.
⚠️ Verify what you download
Checkpoint filenames in this project are reused across training runs and do not uniquely identify a set of weights. Always verify:
file: checkpoints/latest_patcher_ep0_gs75000_mid_epoch.pt
sha256: 2116ab1b0d3c4415…
import hashlib
h = hashlib.sha256(open(path,'rb').read()).hexdigest()
assert h.startswith('2116ab1b0d3c4415'), f'unexpected weights: {h[:16]}'
Verified results
Trained on benign traffic only (Monday-WorkingHours.pcap, CIC-IDS2017). Scored
on a held-out split of 34,043 benign and 7,572 zero-day attack windows,
neither used for training or calibration. Calibrated against 24 attack families.
Window length 512 bytes, top-10% surprise aggregation.
| Threshold rule | Calibration TPR | Held-out benign FPR | Held-out zero-day detection |
|---|---|---|---|
| Youden | 48.5% | <0.05% | 18.7% |
| Target 1% FPR | 46.9% | 0.038% | 14.9% |
| EVT / Peak-Over-Threshold | — | 0.038% | 15.0% |
Calibration ROC-AUC: 0.7962.
Do not select on calibration AUC. The paper documents four cases in this project where AUC ranked models opposite to their deployed performance. Report and compare detection at a stated false-alarm budget instead.
Usage
pip install torch scapy huggingface_hub
git clone https://github.com/TheSPST/sovereign-byte-firewall.git
cd sovereign-byte-firewall
from huggingface_hub import hf_hub_download
ckpt = hf_hub_download(
repo_id="spst01/sovereign-byte-firewall-v2-masking-v2",
filename="checkpoints/latest_patcher_ep0_gs75000_mid_epoch.pt",
local_dir="ckpt_best")
Reproduce the numbers above:
python evaluate_zero_day.py \
--ckpt ckpt_best/checkpoints/latest_patcher_ep0_gs75000_mid_epoch.pt \
--pcap <benign_calibration.pcap> \
--benign_holdout_pcap <benign_holdout.pcap> \
--attack_dir <dir of 24 attack pcaps> \
--holdout_attack_pcap <held_out_attack.pcap> \
--score_agg topk --topk_frac 0.1 --max_sequence_length 512
Live monitoring with a dashboard:
./start_dashboard.sh --learning_time 3600
Architecture
Pre-LN causal Transformer, d_model=128, 4 heads, 2 layers, 512-byte context,
256-way next-byte output. 528,128 parameters, of which 65,536 (12.4%) are the
positional-embedding table.
Input is masked before scoring: MAC/IP addresses, ports, checksums and always-encrypted payloads (TLS application data with cross-packet record tracking, QUIC, WireGuard, SSH transport) are zeroed so the model learns protocol structure rather than memorising identifiers or ciphertext noise.
Intended use
Complementary detection layer for novel/zero-day payloads that match no signature. Passive and read-only — a span/mirror port or a pcap file. No agents, no log pipeline, no network egress. Self-calibrates to a network in about an hour.
Out of scope
- Not a replacement for signature IDS, EDR, or firewalls.
- Encrypted payloads. Content inside an established TLS/QUIC tunnel is not visible. Signal is limited to handshake metadata and size/timing envelope.
- Slow-and-low attackers. A patient adversary pacing every window below threshold evades a memoryless per-window detector by construction.
- Semantic attacks. This detects structural improbability. A valid-looking command with malicious intent looks perfectly normal at the byte level.
- Line rate. Deployment envelope is span-port/pre-filter scale, not core router.
Absolute per-window recall is modest (~15% at a 0.038% false-alarm rate). Campaign-level detection is considerably higher, since a sustained attack generates many windows — on CIC-IDS2017 the fused system caught 5/5 campaigns at roughly one false alarm per hour. The per-window figure is the honest primitive and is what's reported above.
See LIMITATIONS.md.
Training data
CIC-IDS2017 Monday-WorkingHours.pcap — benign traffic only, no attacks.
Sharafaldin, I., Habibi Lashkari, A., Ghorbani, A. A. (2018). Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization. ICISSP 2018, 108–116.
Single training corpus from one capture; performance on a materially different network baseline is untested.
Citation
@misc{tomar2026sovereign,
author = {Tomar, Shubham},
title = {Sovereign Byte Firewall: Comparing Transformer and Mamba-2 Backbones
for Tokenization-Free Zero-Day Network Anomaly Detection},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.21546151}
}