eduard76/it-security-distill

IT Security Distillation: Llama-3.1-70B → Phi-3.5-mini

Distill Llama-3.1-70B-Instruct (teacher) into Phi-3.5-mini-instruct (3.8B student) for IT networking & security Q&A using SFT distillation.

Overview

Two-phase pipeline:

1. Phase 1 — Data Generation: Llama-70B generates ~2,000 IT/security Q&A pairs from 100+ seed topics
2. Phase 2 — SFT Training: Phi-3.5-mini-instruct is fine-tuned on the teacher's outputs

Based on the Magpie SFT distillation recipe (arXiv 2406.08464, ICLR 2025).

Prerequisites

1. Get access to Llama-3.1-70B-Instruct

The model is gated. Go to https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct and request access.

2. Log in to Hugging Face

huggingface-cli login

Phase 1: Generate Teacher Data

Requirements

• 1x A100 80GB GPU (uses AWQ 4-bit quantized model, ~40GB VRAM)

Install

pip install vllm transformers datasets huggingface_hub

Run

python generate_teacher_data.py

This will:

1. Load Llama-3.1-70B-Instruct-AWQ-INT4 (4-bit, fits on single A100 80GB)
2. Generate ~20 diverse questions per seed topic (100 topics → ~2,000 questions)
3. Generate detailed teacher responses for each question
4. Save to ./generated_data.jsonl

Customize

# More questions per topic
N_PER_TOPIC=30 python generate_teacher_data.py

# Upload to HF Hub (requires huggingface-cli login)
UPLOAD_TO_HUB=1 DATASET_REPO=youruser/your-dataset-name python generate_teacher_data.py

# Use full bf16 model instead of AWQ (needs ~140GB VRAM, e.g. 2x A100 80GB)
TEACHER_MODEL=meta-llama/Llama-3.1-70B-Instruct TENSOR_PARALLEL=2 QUANTIZATION=none python generate_teacher_data.py

Output format

Each line in generated_data.jsonl:

{
  "messages": [
    {"role": "user", "content": "How does ARP cache poisoning work?"},
    {"role": "assistant", "content": "ARP cache poisoning is an attack..."}
  ],
  "topic": "How ARP works and ARP cache poisoning"
}

Phase 2: SFT Training

Requirements

• 1x A100 80GB GPU (or any GPU with >=16GB VRAM)
• The generated dataset from Phase 1

Install

pip install torch transformers trl accelerate datasets

Run

python train_sft.py

This will:

1. Load Phi-3.5-mini-instruct
2. Load ./generated_data.jsonl (or a HF Hub dataset)
3. Fine-tune for 2 epochs with completion-only loss (loss on teacher responses only)
4. Save the model to ./phi35-distilled/final/

Customize

# Use a HF Hub dataset instead of local file
DATASET_PATH=youruser/your-dataset python train_sft.py

# Upload model to HF Hub
UPLOAD_TO_HUB=1 MODEL_REPO=youruser/your-model-name python train_sft.py

# Enable Trackio monitoring (live loss dashboard)
USE_TRACKIO=1 python train_sft.py

# Adjust batch size for smaller GPUs
# Edit per_device_train_batch_size in train_sft.py
# For 16GB GPUs: per_device_train_batch_size=1, gradient_accumulation_steps=16

Training hyperparameters

Parameter	Value	Rationale
Learning rate	1e-5	Lower than default 2e-5 for instruct→instruct (DistilQwen recipe)
Epochs	2	Standard for SFT distillation
Effective batch size	16	per_device=4 × grad_accum=4 (A100 80GB)
Max sequence length	4096	Covers most IT/security Q&A
Packing	True	Efficient training
Completion-only loss	True	Loss only on teacher responses
Precision	bf16	Standard for modern GPUs
Gradient checkpointing	True	Memory efficient

Test the Trained Model

After training, test with:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "./phi35-distilled/final",
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("./phi35-distilled/final", trust_remote_code=True)

messages = [
    {"role": "user", "content": "Explain how a TCP three-way handshake works."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.6, do_sample=True)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Seed Topics Covered

• Networking: TCP/IP, OSI model, subnetting, DNS, BGP, OSPF, NAT, VLANs, STP, ARP, IPv6, MPLS, SDN, VPN, TLS/SSL, PKI, load balancing
• Security fundamentals: CIA triad, encryption (symmetric/asymmetric, AES, RSA, ECC), hashing, password security, MFA, OAuth, OIDC, JWT, Zero Trust, firewalls, IDS/IPS, SIEM
• Web security: OWASP Top 10, SQL injection, XSS, CSRF, SSRF, CSP, WAF, API security
• Offensive security: Pentesting, Nmap, Metasploit, Burp Suite, privilege escalation, AD attacks, social engineering
• Incident response: NIST IR lifecycle, digital forensics, memory/disk/network forensics, MITRE ATT&CK, ransomware, malware analysis
• Cloud & DevOps: AWS/Azure/GCP security, Docker/K8s security, CI/CD security, IaC security, Vault
• Compliance: GDPR, PCI DSS, HIPAA, ISO 27001, NIST CSF, SOC 2
• Practical IT: Linux hardening, SSH, iptables, Windows GPO, patch management, backups, monitoring, Ansible

References

• Magpie (arXiv 2406.08464) — SFT distillation recipe
• DistilQwen2.5 (arXiv 2504.15027) — Black-box KD hyperparameters
• TRL SFTTrainer docs
• Llama-3.1-70B-Instruct
• Phi-3.5-mini-instruct

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = 'eduard76/it-security-distill'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.