getSTEAV/system-one-fdb-ipblock

System One fraud detector for FDB ipblock: IP Blocklist (CINS Army list, 2022-06-07)

This model detects IPv4 addresses that appear on a threat-intelligence blocklist. It is a two-stage stack: a gradient-boosted tree model scores each IP address from engineered features, and System One, a typed-decision model (Laya ModernBERT-large encoder with a LoRA adapter and a decision head), reads that score, a few readable engineered fields and the original fields as JSON evidence, and returns a calibrated probability that the IP address is malicious. The released output is a logistic blend of System One's probability and the tree score, fitted on System One's holdout split.

On FDB's official test split (43,000 IP addresses, 2,997 malicious) it scores 0.9481 AUROC and catches 55.7% of malicious addresses at a 1% false-positive rate, against the best published 0.937 AUROC (AFD OFI) and 46.6% recall (AFD OFI). The published baselines date from 2022, while this model's ASN features use a 2026 snapshot of the internet's routing table, which they could not have had (see Limitations).

It was trained for the Fraud Dataset Benchmark (FDB) and is a benchmark model, not a production fraud system; read Limitations first.

Results

Official FDB test split, scored once per model. AUROC and recall at 1% FPR follow FDB's own evaluation (recall is np.interp(0.01, fpr, tpr) on the test ROC). 95% confidence intervals: class-stratified bootstrap, 1,000 resamples.

Model	AUROC [95% CI]	Recall at 1% FPR [95% CI]	Average precision	Recall at 0.1% FPR
This model: blend of System One and the tree	0.9481 [0.9441, 0.9520]	55.7% [53.7, 57.8]	0.7477	35.6%
System One alone	0.9460 [0.9415, 0.9502]	55.3% [53.1, 57.4]	0.7463	35.3%
Tree alone (System One's input)	0.9481 [0.9441, 0.9520]	55.6% [53.7, 57.8]	0.7475	35.8%
Best published AUROC: AFD OFI	0.937
Best published recall at 1% FPR: AFD OFI		46.6%

AUROC leaders are from the FDB paper (arXiv 2208.14417 v3). The paper reports no recall, so recall leaders are from the results table in FDB's README at commit 54cdefa211; later versions of the README keep that table inside an HTML comment, so it no longer renders. With 2,997 malicious IP addresses in the test set, each one is worth 0.033 points of recall.

How the released configuration was chosen. All models were trained and calibrated without test data, and each test set was scored once per candidate. After the test results were known, one model per benchmark was released: System One on every set (for a uniform, calibrated decision interface), and for this set the blend of System One and the tree as the default output, because System One alone trailed its tree. Choosing among candidates after seeing test results can flatter the headline slightly, so the table shows every candidate.

Model details

• Developed by: STEAV (steav.io), trained with CID Model Studio.
• Model type: stacked binary classifier: gradient-boosted trees, then System One reading the tree score as evidence, then a logistic blend of the two.
• System One base: convaiinnovations/laya at revision 7b928d828b7b (Apache-2.0), a ModernBERT-large encoder (about 395M parameters) with Laya's typed decision head. The base weights are downloaded from the Hub at that pinned revision and checked against their sha256; they are not redistributed here.
• Trained parameters: a LoRA adapter (rank 16, alpha 32, on the attention Wqkv/Wo and MLP Wo projections of all 28 layers; 4.39M parameters) and the decision head (26.2M parameters, initialised from Laya's head; its 0.26M-parameter action sub-head stays frozen).
• Question: "Is this IP address malicious?" (a yes/no decision; the output is P(yes)).
• License: Apache-2.0 for the weights and code; some data files carry their own terms (see License and attribution).
• Version: 1.0.0.
• Contact: questions and issues go to this repository's Community tab.

Uses

Intended: benchmark research and comparison on FDB; a reference implementation of stacking a typed-decision model on a tree model's score; a starting point for fine-tuning on your own data.

Out of scope: blocking traffic in production, or attributing an address to a specific actor; any automated decision about a person without human review; use on data from a different distribution without retraining and validation.

How to use

Python 3.12 with the pinned versions in requirements.txt. On the reference GPU (NVIDIA GB10 in a DGX Spark, torch 2.11.0, bfloat16) the code reproduced the evaluated scores bit for bit; other GPUs and CPUs give slightly different scores (see Reproducibility). On CPU (Apple M4 Max, macOS arm64, float32) System One takes about 0.11 s per IP address, about 1.4 h for the 43,000-row test set.

hf download getSTEAV/system-one-fdb-ipblock --local-dir system-one-fdb-ipblock
cd system-one-fdb-ipblock && pip install -r requirements.txt

# run from the repository folder
from system_one_fraud import FraudPipeline
from system_one_fraud.fdb import load_split

pipe = FraudPipeline.from_pretrained(".")      # the tree, the adapter and the pinned Laya base
train, test, labels = load_split("ipblock")       # FDB's loader
p_fraud = pipe.predict_proba(test)              # one probability per row

predict_proba returns, per row, the probability that the IP address is malicious. The default output is the blend; output="system_one" or output="tree" returns one stage alone. states, tree_out = pipe.states(rows) returns the exact JSON evidence System One reads and the tree model's outputs. examples/quickstart.py runs the same steps on a few test rows.

The adapter does not load with PeftModel.from_pretrained alone: Laya's root config.json is a stub that transformers cannot instantiate, and the decision head and evidence format live in the bundled system_one_fraud package. On macOS the tree model runs in a helper Python process, because PyTorch and the gradient-boosting libraries load two OpenMP runtimes that crash in one process; results are identical. System One's bfloat16 GPU scores depend slightly on how rows are batched (padding); the reference scores used batch size 128 in file order.

Input contract. Each row needs only ip (a dotted IPv4 string); every other column is ignored and a missing ip raises an error. Every row is scored against the bundled training reference (172,000 addresses) and the bundled IP-to-ASN table, never against the other rows it arrives with, so the tree's scores do not depend on batching.

Training data

• Source: FDB's versioned IP Blocklist set (2022-06-07): the CINS Army list of malicious addresses (Sentinel IPS, cinsscore.com) plus 200,000 random IPv4 addresses generated with Faker and labelled benign, shipped inside the FDB repository.
• Benchmark split: the fixed train/test files in FDB's versioned zip (a random split). Training: 172,000 IP addresses (12,003 malicious, 6.98%). Test: 43,000 IP addresses (2,997 malicious).
• Licence: The CINS Army list is published by Sentinel IPS, whose site says it may be parsed and used "in any way you see fit"; that is an informal permission rather than a licence. FDB redistributes the 2022-06-07 snapshot in its MIT-licensed repository but lists the list's own licence as unknown. ASN features use the iptoasn.com database (Public Domain Dedication and License v1.0).

Training procedure

Stage 1, tree model. The mean of three histogram gradient-boosting models (seeds 7, 11, 23; learning rate 0.03, 63 leaves, early stopping, refit on all 172,000 training rows) over 81 features: octets and prefixes, reserved-range flags, label-free neighbourhood density (addresses in the same /16 and /24, distance to the k-th nearest training address), smoothed malicious rates and counts per prefix and nearest known-malicious and known-benign distances (out-of-fold for training rows), and IP-to-ASN lookups (routed flag, ASN, network size, smoothed malicious rate per ASN and country). Out-of-fold statistics for training rows come from four fifths of the training addresses, while test rows see all of them, so at test time neighbourhood counts are scaled by 0.8 and distances by 1.25. The scaling only approximates the difference: test scores are somewhat more extreme than out-of-fold scores (the test's 1% and 5% quantiles of the tree percentile are 0.0 and 1.02). Feature groups and hyperparameters were chosen on a stratified random 20% validation split of FDB's training rows (seed 7); label-dependent statistics for training rows are out-of-fold (5 folds), with a flip test confirming no row's features read its own label.

Training rows carry out-of-fold tree scores: the models that scored a training row never saw it, so System One never learns from a score fit on its own row.

Stage 2, System One. Each training row is rendered as a JSON state: the tree's out-of-fold score (Tree_fraud_score), its percentile among all out-of-fold training scores (Tree_percentile), readable engineered fields (Context_*) and the original fields. System One trained on all 172,000 FDB training addresses. The rows were split 137,600 / 17,200 / 17,200 (train / validation / holdout). On the official test set every state fits in System One's 192-token window. System One does not see the tree's nearest_malicious_ip_gap field, whose scale differs between training and test.

• LoRA r = 16, alpha = 32, dropout 0.05; learning rate 1e-4 (head 5e-5), weight decay 0.01, 5% warm-up, batch 32; bfloat16; maximum sequence length 192 tokens; seed 42.
• one supervised epoch with soft cross-entropy, then one RLCD epoch (reinforcement learning on the decision distribution with proper-scoring-rule rewards, following Laya's method).
• 8,600 optimiser steps, 33.9M training tokens, 3.0 h on one NVIDIA GB10 (DGX Spark).
• Calibration: one temperature fitted on the validation split (L-BFGS).
• Validation: accuracy 0.9584, expected calibration error 0.0043.

Evaluation

• Protocol. FDB's official split; the test set was never used for training, feature selection or calibration, and labels were read only to compute the metrics.
• Calibration on test (10 equal-width bins, released output): expected calibration error 0.0022, Brier score 0.0315.
• Cost of high detection (released output): catching 95% of malicious addresses requires a false-positive rate of 27.29%, and catching 99% requires 55.21%.
• Reproduce: python eval/reproduce_fdb.py rebuilds the split with FDB's own loader, checks it against the frames used here, scores it with the released pipeline and compares the metrics with eval/test_metrics.json.

Limitations

• Synthetic negatives. The benign class is uniformly random IPv4 space (private ranges included), not real benign traffic. The model largely learns whether an address sits in networks that host listed attackers (cloud and hosting ranges, scanner ASNs). On real traffic, where benign addresses also come from those networks, precision will be much lower than the benchmark suggests.
• Future enrichment. The ASN table is a 2026 snapshot applied to a 2022 list: 1.35% of the listed addresses fall in space that is no longer routed, and hosting footprints have grown since. Refresh the table (and retrain) before any real use.
• Blocklists are a snapshot. Addresses move between owners and lists churn daily; a 2022 list is stale.
• System One vs its tree. On this set System One alone trails its own tree slightly but measurably on AUROC; the released default is a logistic blend of the two, fitted on System One's holdout split, which matches the tree.

Bias, risks and ethical considerations

Fraud models make errors in both directions: false positives block or delay legitimate customers, and their burden can fall unevenly across groups. No fairness evaluation was done for this model; the benchmark data carries no protected attributes we could audit. Use the score as one input to a reviewed decision, monitor error rates on your own population, and retrain when the data drifts.

Do not use the bundled addresses as a blocklist. tree/ref_ips_malicious.npy holds the 12,003 listed addresses of FDB's 2022 training split, kept only so new addresses can be compared with them. Blocklist entries go stale quickly: many of these addresses have since passed to other owners, and blocking them would hit whoever holds them now. IP addresses can also be personal data in some jurisdictions.

Reproducibility

These checks passed before release:

• Tree model. Refit from scratch, the tree model in tree/ reproduces the evaluated tree scores bit for bit on all 43,000 test rows (Apple M4 Max, macOS arm64, the pinned versions). On other platforms (other BLAS or OpenMP builds) the last bits can differ, which can occasionally change the Tree_fraud_score text System One reads.
• Evidence. The state builder reproduces the evaluated System One inputs exactly on all 43,000 test rows.
• System One. The inference code in system_one_fraud/ reproduces the evaluated scores bit for bit on all 43,000 test rows on the reference GPU (NVIDIA GB10 in a DGX Spark, torch 2.11.0, bfloat16, batch size 128 in file order, base downloaded from the Hub), run in the training environment with the same pinned versions rather than a fresh install.
• Fresh install. A new virtual environment built from requirements.txt (Apple M4 Max, macOS arm64, pandas 3.0.6, torch 2.11.0) reproduces the tree scores bit for bit and the System One inputs exactly.
• Fresh download. FDB's loader, run fresh with the pinned pandas, rebuilds the evaluated frames exactly (content hashes in eval/fdb_split_hashes.json) apart from the identifier and timestamp columns FDB randomises on every load, which the models never read.
• End to end on CPU. eval/reproduce_fdb.py (fresh download, released pipeline, Apple M4 Max, macOS arm64, float32) gives 0.9481 AUROC and 55.7% recall at 1% FPR, against the reported 0.9481 and 55.7% (identical to 4 decimals).

Measured, not gated: CPU drift. On CPU in float32, System One's scores differ slightly from the bfloat16 GPU reference. On 512 holdout IP addresses (256 malicious) the mean absolute difference in probability is 5.9e-04 and the largest 5.7e-03; decisions at 0.5 agree on 100.0% of rows, and AUROC on that sample is 0.9529 on the GPU and 0.9529 on CPU. The largest difference in any of the 3 CPU comparisons was 7.6e-03, on 512 holdout IP addresses, in an earlier check.

Environmental impact

System One training used one NVIDIA GB10 (DGX Spark, on premises) for 3.0 h, roughly 0.7 kWh at the system's 240 W rating (an upper-bound estimate). The tree models trained in minutes on a laptop CPU.

Files

Path	Contents
adapter_model.safetensors, adapter_config.json	LoRA adapter
system_one/head.safetensors	decision head
system_one/config.json, system_one/calibration.json, release.json	head configuration, temperature, release manifest (question, pinned base, priors, blend weights)
system_one/percentile_reference.npy	sorted out-of-fold training tree scores, for the Tree_percentile evidence field
tree/	three scikit-learn histogram gradient-boosting models (skops, loaded with an explicit one-type trusted list), the training reference addresses as sorted 32-bit integers, the iptoasn.com table (parquet), sorted out-of-fold score references (.npy), config.json and manifest.json with every file's sha256 (checked on load)
system_one_fraud/	inference code: pipeline.py (end to end), state.py (JSON evidence), system_one.py (System One), tree_ipblock.py (tree model), tree_runner.py (macOS helper process), fdb.py (FDB loader)
eval/	reproduce_fdb.py, the split hashes and the test metrics
examples/quickstart.py	load, score and print a few test rows
LICENSE, NOTICE, LICENSES/	licence texts and attributions
requirements.txt	pinned dependencies
checksums.sha256	sha256 of every file except this README

Embedded data. tree/ contains the reference set the features are computed against: the 12,003 blocklisted training addresses (CINS Army list, 2022) and the 159,997 randomly generated benign addresses, as sorted 32-bit integers; the iptoasn.com IP-to-ASN table (2026-09-26 snapshot); and the sorted out-of-fold training scores. Each file's terms are listed under License and attribution.

Citation

@misc{steav2026systemonefdb,
  title        = {System One fraud detectors for the Fraud Dataset Benchmark},
  author       = {{STEAV}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/getSTEAV/system-one-fdb-ipblock}}
}

@misc{grover2023fraud,
  title         = {Fraud Dataset Benchmark and Applications},
  author        = {Prince Grover and Julia Xu and Justin Tittelfitz and Anqi Cheng and Zheng Li and Jakub Zablocki and Jianbo Liu and Hao Zhou},
  year          = {2023},
  eprint        = {2208.14417},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}

@misc{convai2026laya,
  title        = {Laya: a non-autoregressive System 1 decision model},
  author       = {{Convai Innovations}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/convaiinnovations/laya}},
  note         = {Revision 7b928d828b7b0e022f929d9bd2e44165aa270148}
}

@misc{warner2024modernbert,
  title         = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author        = {Benjamin Warner and Antoine Chaffin and Benjamin Clavi{\'e} and Orion Weller and Oskar Hallstr{\"o}m and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
  year          = {2024},
  eprint        = {2412.13663},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

License and attribution

Files	Terms
tree/ref_ips_malicious.npy, tree/ref_ips_benign.npy	training addresses of FDB's IP Blocklist set, redistributed from FDB's MIT-licensed repository (LICENSES/FDB-MIT.txt); FDB lists the CINS list's own licence as unknown (see Training data)
tree/asn_table.parquet	iptoasn.com IP-to-ASN database, Public Domain Dedication and License v1.0 (LICENSES/PDDL-1.0.txt)
everything else (weights, code, configuration, model files)	Apache-2.0 (LICENSE)

Attributions are collected in NOTICE.

• Data: Trained on FDB's IP Blocklist set (version 2022-06-07): the CINS Army list (Sentinel IPS, cinsscore.com) plus randomly generated benign addresses. The training addresses in tree/ref_ips_malicious.npy and tree/ref_ips_benign.npy are redistributed from the Fraud Dataset Benchmark repository, which is MIT-licensed (LICENSES/FDB-MIT.txt) but lists the CINS list's own licence as unknown; Sentinel IPS's permission to use the list is informal. ASN features use the iptoasn.com IP-to-ASN database by Frank Denis (Public Domain Dedication and License v1.0, LICENSES/PDDL-1.0.txt), snapshot of 2026-09-26, shipped as tree/asn_table.parquet.
• Base model: Laya by Convai Innovations (Apache-2.0), built on ModernBERT-large by Answer.AI and LightOn (Apache-2.0). The sequence layout and the decision-head architecture follow Laya's reference code.
• Benchmark: Fraud Dataset Benchmark, Grover et al. 2023 (MIT; Copyright (c) 2021-2022 Prince Grover and Zheng Li, and (c) 2022 Jianbo Liu, Jakub Zablocki, Hao Zhou, Julia Xu and Anqi Cheng).