rohangupta1808/semantic-drift-severity-qwen2.5-coder-3b

Semantic Drift Detection in Infrastructure-as-Code using Large Language Models: Qwen2.5-Coder-3B (GGUF)

A fine-tuned Qwen2.5-Coder-3B-Instruct model that assesses the security severity of a Terraform Infrastructure-as-Code change. Given a before / after configuration pair, it returns a severity level (Critical / High / Medium / Low), a 0–10 risk score, the governing security policy, and a CWE identifier.

This file is the Q4_K_M GGUF build (~1.8 GB) for local inference via Ollama or llama.cpp.

Why

Existing tooling detects that infrastructure drifted from its declared configuration, but not how dangerous the change is. This model adds that judgement so practitioners can prioritise.

How it was trained

• Base model: Qwen/Qwen2.5-Coder-3B-Instruct
• Method: LoRA (parameter-efficient) fine-tuning, then merged and quantised to Q4_K_M GGUF
• Data: a grounded, leakage-free dataset of Terraform before/after changes, with severity labels sourced from established security scanners (KICS, Checkov, tfsec) rather than hand-authored, removing label bias
• Evaluation: a policy-disjoint test set whose governing policies are never seen during training, so scores reflect genuine generalisation

Performance (leakage-free, unseen-policy test)

This GGUF is the first fine-tuned iteration: macro-F1 0.347, accuracy 0.50. A later, larger dataset iteration raised accuracy to 0.78 on the same test set (see the project repository).

Usage: Ollama

ollama create semantic-drift -f Modelfile
ollama run semantic-drift

Usage: llama.cpp

./llama-cli -m semantic-drift-q4.gguf -p "<your prompt>"

Project

Full code, datasets, evaluation and documentation: https://github.com/RohanGupta1808/rohan18-Semantic_project

License

Apache 2.0, inherited from the base model (Qwen2.5-Coder-3B-Instruct).