netops-intent2cli-qwen0.5b-lora — v2

English operator-style intent → vendor CLI, with measured limits. Experimental, not production-safe.

Treat every output as an untrusted suggestion. Review vendor release documentation and test in a lab before use. Never execute generated changes autonomously.

Evaluation

Greedy decoding, 128 new-token cap, the unchanged v1 system/user prompt. The 108 v1 test examples remain unchanged and their commands occur in neither training nor validation. Exact match strips surrounding whitespace only. Normalized match also case-folds, collapses whitespace, and removes quotes/code fences; it is not a semantic correctness score.

model	original v1 test EM	original v1 test norm	operator test EM	operator test norm
base	0/108 (0.0%)	0/108 (0.0%)	0/88 (0.0%)	0/88 (0.0%)
v1	10/108 (9.3%)	10/108 (9.3%)	2/88 (2.3%)	3/88 (3.4%)
v2	12/108 (11.1%)	12/108 (11.1%)	9/88 (10.2%)	9/88 (10.2%)

Original v1 test: per vendor

vendor	n	base EM/NM	v1 EM/NM	v2 EM/NM
Arista	9	0/0	0/0	0/0
Aruba	6	0/0	1/1	1/1
Cisco	6	0/0	1/1	1/1
Extreme	6	0/0	0/0	1/1
FRR	5	0/0	0/0	0/0
FortiOS	1	0/0	0/0	0/0
Huawei	5	0/0	1/1	1/1
Juniper	9	0/0	0/0	0/0
Linux	8	0/0	2/2	2/2
Microsoft	12	0/0	1/1	2/2
Mikrotik	7	0/0	0/0	0/0
NVIDIA	6	0/0	1/1	1/1
Nokia	6	0/0	0/0	0/0
PAN-OS	2	0/0	0/0	0/0
SONiC	5	0/0	0/0	0/0
VyOS	4	0/0	0/0	0/0
Wireshark	11	0/0	3/3	3/3

Synthetic operator-style test: per vendor

vendor	n	base EM/NM	v1 EM/NM	v2 EM/NM
Arista	8	0/0	0/0	0/0
Aruba	2	0/0	0/0	0/0
Cisco	6	0/0	0/0	0/0
Extreme	5	0/0	0/0	1/1
FRR	4	0/0	0/0	0/0
Huawei	2	0/0	0/0	0/0
Juniper	7	0/0	0/0	0/0
Linux	7	0/0	0/0	1/1
Microsoft	11	0/0	1/2	3/3
Mikrotik	5	0/0	0/0	0/0
NVIDIA	6	0/0	0/0	1/1
Nokia	5	0/0	0/0	0/0
PAN-OS	2	0/0	0/0	0/0
SONiC	4	0/0	0/0	0/0
VyOS	3	0/0	0/0	0/0
Wireshark	11	0/0	1/1	3/3

Data and methodology

• v1: 565 train / 102 validation / 108 test, unchanged command-disjoint split.
• v2 training: 1215 examples, adding 650 new synthetic operator-style requests across 17 vendors and 541 distinct corpus command labels. Of these, 525 are new phrasings of v1 training labels; there are 94 genuinely new command labels, and 650 unique commands in the full v2 training split. FortiOS adds 57 requests across six configuration families.
• New operator-style held-out slice: 88 requests for original test commands. Only commands present with the matching vendor/OS in the 70,006-row source Parquet are eligible; credential-related examples are excluded. It is a subset, not a replacement for the frozen 108.
• All 738 new train/test pairs use command strings copied verbatim from multivendor-cli-commands. New intents are synthetic, generated offline with assistant-authored semantic rules and paraphrase grammar. No vendor documentation prose is copied. No xAI requests were made for v2.
• New data passed exact corpus membership, matching vendor/OS membership, whitespace-normalised command split leakage checks, surface near-duplicate rejection, firewall action-polarity checks, and restricted-work-term scans (zero hits). This is not exhaustive semantic validation.
• v1's inherited curated examples remain. Sixteen original test commands are curated rather than exact source-corpus rows. Frozen benchmark provenance is preserved rather than silently rewriting labels.
• Validation alone selected the checkpoint; the two test sets were evaluated after selection. This single run does not establish statistical significance or state-of-the-art performance.

Training

Continued from the v1 adapter (3 prior epochs), not a fresh base. LoRA rank 16, alpha 32, dropout .05, q/k/v/o projections; 3 extra epochs run, checkpoint selected at extra epoch 1. Rank 32/all-linear and 1.5B were not tested: the measured CPU budget and memory pressure did not support a fair additional run.

AdamW lr 1e-4 with warmup/linear decay, completion-only loss, fp32, CPU, 7 torch threads, process nice +15. Effective batch 8 via microbatch 2 and accumulation 4; the initial batch-8 attempt exceeded memory, so the final run restarted cleanly from v1. Seed 864. Early stopping patience 2 on validation loss with minimum improvement .002. Final training time 1460 s. Best validation loss 1.118536.

extra epoch	training loss	validation loss	elapsed seconds
initial v1	—	1.142961	0
1	1.128728	1.118536	500
2	0.766017	1.144932	981
3	0.525927	1.183348	1460

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
base = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(base),
    "gesh75/netops-intent2cli-qwen0.5b-lora",
)
messages = [
    {"role": "system", "content": "You are a network CLI assistant. Reply with only the exact CLI command(s) for the given vendor, no explanation."},
    {"role": "user", "content": "Vendor: Juniper (junos)\nIntent: need a compact interface and addressing overview"},
]
x = tok.apply_chat_template(messages, tokenize=True, return_dict=True,
                            add_generation_prompt=True, return_tensors="pt")
y = model.generate(**x, max_new_tokens=128, do_sample=False)
print(tok.decode(y[0, x["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

• These are synthetic operator-style paraphrases, not independent human shift conversations. Grammar templates and numeric variants remain; near-duplicate filtering is heuristic. Shared authoring conventions between train and operator test can overestimate generalisation.
• Commands are attested in a public corpus, not necessarily valid or safe for a device/release. The corpus includes placeholders, incomplete syntax productions and questionable entries. Some original intents omit values found in the label, so exact-match failure need not mean the underlying task was impossible to understand.
• Original test has only one FortiOS example, which is curated; operator test has no FortiOS. Extra training coverage does not establish FortiOS reliability. Rare vendor counts are too small for broad claims.
• The model still hallucinates keywords/paths and may copy fixture values, use unsafe defaults, or return the wrong operation. No live device tests or production validation were performed.
• No claims of 94% accuracy, best-in-class status, or autonomous change safety.

License

Apache-2.0 for the adapter, matching the base model. The dataset retains its existing MIT license; vendor names/syntax are factual identifiers and trademarks of their owners.

---

GESH · NetOps tools on Hugging Face

📡 netlog-ai	Sanitize-first network log analysis — browser lite demo of netlog-ai v1.1.0
🛠️ multivendor-cli-configurator	70k+ CLI commands across 17 vendors — cheatsheet, N-vendor compare, CLI Studio
🗂️ multivendor-cli-commands	The same 70,006 commands as a Parquet dataset
🧭 netops-intent2cli	v1 + v2 synthetic operator intents → corpus command pairs
🤖 netops-intent2cli-qwen0.5b-lora	LoRA adapter: English intent → vendor CLI (experimental)

Collection: NetOps tools · Code: github.com/gesh75 · Docs hub: gesh75.github.io

Doctrine: fail closed · sanitize first · evidence leaves, packets don't · honest coverage.