doryno/edr-ai-Qwen2.5-3B-Instruct

EDR-AI — Qwen2.5-3B-Instruct

Fine-tuned Qwen2.5-3B-Instruct model for local Windows endpoint security analysis.

Full project: https://github.com/DoryNo/edr-ai

Dataset: https://huggingface.co/datasets/doryno/edr-ai-dataset

The model is designed to classify Sysmon events into:

• BENIGN
• SUSPICIOUS
• MALICIOUS
• MINER

It also provides risk score, confidence, evidence, MITRE ATT&CK tactics/techniques, and recommended actions.

What's new in v9 (2026-09-01)

• Training data doubled and rebalanced: 4000 → 6400 records (BENIGN 2200 / SUSPICIOUS 1800 / MALICIOUS 1600 / MINER 800), 264 scenario families, 8 environment contexts. New wave-3 catalog targets the previous round's weak spots: SUSPICIOUS hard negatives across the full risk band 4–8, BENIGN lookalikes of suspicious shapes, more MALICIOUS/MINER diversity.
• Honest evaluation pipeline: the holdout is now frozen and excluded from training (320 records, 80 per class, leak-checked at train time). Earlier metrics were partially inflated by memorization and are not comparable.
• Repaired export: the shipped GGUF is now built from the QLoRA training composite. v9 Q8_0 reaches fp16 parity on the holdout — quantization no longer costs accuracy (see the QLoRA merge note below).
• Residual error pattern is now symmetric boundary noise instead of systematic over-escalation: MALICIOUS→SUSPICIOUS dropped from 11/80 (v8) to 2/80.

Evaluation

Frozen holdout evaluation — v9 — 2026-09-01 (Q8_0 GGUF, 320 records):

Metric	Result
Accuracy	97.8%
JSON validity	100%
BENIGN	97.5%
SUSPICIOUS	96.2%
MALICIOUS	97.5%
MINER	100.0%

Confusion matrix (80 records per class):

expected \ predicted | BENIGN | SUSPICIOUS | MALICIOUS | MINER
             BENIGN  |     78 |          1 |         1 |     0
          SUSPICIOUS |      2 |         77 |         1 |     0
           MALICIOUS |      0 |          2 |        78 |     0
               MINER |      0 |          0 |         0 |    80

The model produced 0 MALICIOUS → BENIGN errors and 0 invalid JSON.

Previous round (v8 — 2026-08-27, leaky holdout, kept for history):

Metric	Result
Accuracy	87.2%
JSON validity	99.7%
BENIGN	85.0%
SUSPICIOUS	78.8%
MALICIOUS	86.3%
MINER	98.8%

Quantization note

The adapter is trained on the 4-bit base and partially compensates its quantization error, so GGUF must be exported from the QLoRA training composite (4-bit base + adapter, unsloth save_pretrained_gguf). Merging the adapter onto a full-precision base silently degrades MALICIOUS escalation while the output still looks well-formed and confident.

Available builds (same weights, different quantization):

Build	Holdout accuracy
Q8_0 (recommended)	97.8%
Q4_K_M (legacy)	93.4%

Local Inference

The model is intended for fully local inference using GGUF (Q8_0 recommended) with LM Studio or another OpenAI-compatible server.

No telemetry or event data needs to leave the host.

Use the system prompt shipped with the project (system_prompt.txt) — the model is trained and evaluated against it byte-for-byte. Recommended sampling: temperature = 0–0.1.

Measured latency on a consumer GPU (LM Studio, 8 parallel requests): mean ~22s, p95 ~27s per verdict.

Output Format

{
  "reasoning": "...",
  "classification": "SUSPICIOUS",
  "risk_score": 6,
  "tactics": ["Execution"],
  "techniques": ["T1059.001"],
  "confidence": 0.91,
  "evidence": ["..."],
  "recommended_actions": ["..."]
}

Risk bands: BENIGN 1–3, SUSPICIOUS 4–8, MALICIOUS 9–10, MINER 8–10.

Intended Use

This model is intended for security research, local endpoint monitoring, and AI-assisted Sysmon analysis. It is not a replacement for a production EDR or human security analyst.

Known limitations: evaluation holdout shares the generator distribution of the training data — behavior on out-of-catalog real-world telemetry should be spot-checked before production use; confidence is not calibrated.

Related Project

The model is part of EDR-AI, a local Windows endpoint monitoring system that collects Sysmon events, performs LLM classification, stores structured verdicts, and provides a Streamlit dashboard.