fingerthief/lancet-nano

LANCET Nano: local shell command-risk classifier

LANCET Nano classifies a Bash, PowerShell or cmd command as risky, review or not_flagged before an agent or a person runs it. v0.4.3 is a CodeT5+ 220M encoder with a small windowed pooling head:

• 110.8M parameters
• a 111 MB INT8 ONNX encoder plus a 4.7 MB NumPy head
• about 14 ms per command on a CPU (median)
• no network access at inference

Try the demo · GitHub · Website · Full model card

This is a custom ONNX classifier with its own small runtime (bundle/classify.py). It is not a Transformers AutoModel, and the hosted inference widget is disabled.

What's new in v0.4.3

• Triage Score 68.3 (v0.4.2: 54.4), the best of any guard tested on the new, harder benchmark.
• PowerShell and cmd. v0.4.2 answered review for every non-Bash command; v0.4.3 classifies all three shells.
• Long commands read in full as overlapping 512-token windows; nothing is truncated.
• Far fewer interruptions on look-alike safe commands: on lancet-bench-2-next it catches the same 78.0% of risky commands as v0.4.2 while stopping 12.4% of safe ones instead of 20.0%.
• Newly trained on a new command-semantics curriculum of risky/safe twins, trained to convergence. Same size and CPU speed.

Why every score dropped: a harder benchmark

Until v0.4.2 the Triage Score used lancet-bench-1 (793 Bash commands), where the best guards were near the ceiling. From v0.4.3 it uses lancet-bench-2-next, a much more comprehensive benchmark that no LANCET release was trained, tuned or calibrated on:

• 3,204 commands built as 1,602 risky/safe twins: every risky command has a safe look-alike that differs only in the detail that decides the risk.
• 289 scenarios in 40 areas, from cloud CLIs, Kubernetes and databases to shell quoting, expansion and redirection, deceptive previews, credentials and Windows and macOS administration.
• Every scenario survives composition: repeated inside a subshell, function, command substitution, nested bash -c, pipeline and background job.
• Three shells: 3,066 Bash, 102 PowerShell and 36 cmd commands; 1,803 multi-line.

Because it is much harder, every guard scores lower than before. v0.4.2 scored 82.7 under the previous Triage Score and 54.4 under the new one; the model did not change, the test did. Scores are only comparable on the same benchmark.

Results

The Triage Score gives each risky command asked about or blocked one point, shrinks in proportion when more than 10% of safe commands are stopped, and combines three benchmarks (7,911 commands) weighted by size (58% / 20% / 21%).

At the shipped thresholds	v0.4.3	v0.4.2
Triage Score	68.3	54.4
lancet-bench-2-next: risky caught / safe stopped	78.0% / 12.4%	78.0% / 20.0%
ShellRisk-Bench test: risky caught / safe stopped	64.8% / 3.0%	60.1% / 2.7%
Neutral set (513 commands): risky caught / safe stopped	85.8% / 5.6%	91.0% / 7.0%

Triage Score across three benchmarks

Guard	Triage Score	lancet-bench-2-next	ShellRisk test	Neutral set	Risky caught	Safe stopped	Parameters	Runs on
LANCET Nano v0.4.3	68.3	63.1	64.8	85.8	77.0%	9.0%	111 M	CPU, local
LANCET Nano v0.4.2	54.4	39.0	60.1	91.0	77.1%	13.7%	110 M	CPU, local
LANCET Nano v0.4.0	47.1	41.1	60.6	50.8	55.7%	10.5%	110 M	CPU, local
LANCET Nano v0.4.1	44.9	38.1	70.5	39.3	67.9%	14.6%	110 M	CPU, local
LANCET Nano v0.3.0	44.7	44.3	46.6	43.6	55.3%	11.6%	110 M	CPU, local
LANCET Nano v0.2.0	40.7	39.3	47.7	37.9	55.2%	13.6%	35 M	CPU, local
LANCET Nano v0.1.0	40.5	34.9	45.6	51.0	58.4%	14.1%	35 M	CPU, local
Jev	38.9	43.6	32.4	32.1	90.0%	23.4%	undisclosed	hosted API
Kestrel	30.7	12.7	100.0	14.0	68.1%	37.1%	—	local
ModernBERT bash	23.2	16.2	40.9	25.6	48.1%	22.8%	150 M	local
bash-classify	14.3	13.5	16.8	14.0	92.9%	65.8%	rules	local
Laya	13.3	11.0	16.2	17.1	87.3%	69.6%	421 M	GPU, local
bev-decider	13.3	10.8	18.7	14.8	94.4%	75.3%	0.4 B	GPU, local
sh-guard	10.9	10.2	12.1	11.7	97.9%	90.5%	rules	local

Jev received task context; the others see only the command. Per-benchmark columns are each benchmark's own Triage Score (0-100). Guards that support only Bash answer the 138 PowerShell and cmd commands as "ask".

Quick start

pip install huggingface_hub numpy==2.5.3 tokenizers==0.23.2 onnxruntime==1.30.0
hf download fingerthief/lancet-nano --revision v0.4.3 --local-dir lancet-nano
cd lancet-nano/bundle
python verify_bundle.py --strict
echo '{"command": "kubectl delete namespace prod", "shell": "bash"}' | OPENBLAS_NUM_THREADS=1 python classify.py --model model
echo '{"command": "Stop-Computer -Force", "shell": "powershell"}' | OPENBLAS_NUM_THREADS=1 python classify.py --model model

The same verified bundle is also available as a ZIP from the GitHub release. Earlier versions remain available at revisions v0.4.2, v0.4.1, v0.4.0, v0.3.0 and v0.2.0.

From Python:

from classify import LancetNano          # run inside bundle/
m = LancetNano("model")
m.score("terraform destroy -auto-approve")            # {'classification': 'risky', 'score': ..., ...}
m.score("Get-ChildItem -Recurse", shell="powershell")

Training

• v0.4.3 is the CodeT5+ 220M encoder, fine-tuned from its pinned upstream weights. It does not start from an earlier LANCET checkpoint.
• Labels come from project authoring, documentation and source datasets:
  • LANCET's new command-semantics curriculum: risky/safe twins for Bash, PowerShell and cmd
  • the wording of tldr-pages example descriptions (CC BY 4.0)
  • AWS operation verbs and sensitive output-field markings in botocore service models (Apache-2.0)
  • Azure CLI and GitHub CLI (MIT), and kubectl and Docker CLI (Apache-2.0) reference examples
  • the ShellRisk-Bench training split's upstream labels: Atomic Red Team (MIT) and InternalAllTheThings (no license declared) attack commands, SWE-smith and Terminal-Bench developer commands
  • the Shell Safety v2 training split (MIT), with its author's allow/deny labels
• Project-authored material: risky/safe pairs, contrast sets, a secrets family, earlier evaluation suites and the retired lancet-bench-1 (original labels).
• No language model, hosted API or human labeler produced any training label.

Details, sources and notices: MODEL_CARD.md, NOTICE.txt, THIRD-PARTY-NOTICES.md.

Limits

• Bash, PowerShell and cmd. PowerShell and cmd coverage is newer and smaller than Bash coverage. Nano has no access to files, fetched scripts or task context.
• Other shells, invalid input or more than 8,192 bytes return review. Inputs are never silently truncated.
• not_flagged is not a safety guarantee or execution authorization.
• Credentials, Windows and macOS administration and deceptive previews are its weakest areas on lancet-bench-2-next.

License and credits

• The model is licensed Apache-2.0 (upstream CodeT5+ BSD-3-Clause notice retained); the runtime is MIT. Training datasets are not included or relicensed.
• Built on Salesforce CodeT5+ by Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li and Steven C. H. Hoi.
• Training commands come from the tldr-pages team and contributors, AWS botocore, the Azure CLI, GitHub CLI, kubectl and Docker CLI projects, and ShellRisk-Bench by Kontext Security (including Atomic Red Team by Red Canary and InternalAllTheThings), and Shell Safety v2 by tomngdev.
• LANCET contributions: Tanner Middleton.