umer07
/

fathom-mixtral

@@ -1,166 +1,181 @@
 ---
 base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
-library_name: peft
 tags:
 - cybersecurity
 - malware-analysis
-- peft
 - lora
-- qlora
 - mixtral
-language:
-- en
 pipeline_tag: text-generation
-license: apache-2.0
 ---
-# Fathom Plan A LoRA Adapter (Mixtral-8x7B-Instruct)
-This repository contains the **Plan A** LoRA adapter for the Fathom FYP project:
-**"Fathom: An LLM-Powered Automated Malware Analysis Framework"**
-The adapter is trained on a curated cybersecurity instruction-tuning corpus to improve analyst-style security outputs over the base `mistralai/Mixtral-8x7B-Instruct-v0.1` model.
-## What This Is
-- **Type:** PEFT LoRA adapter (not a full standalone model)
-- **Base model required:** `mistralai/Mixtral-8x7B-Instruct-v0.1`
-- **Training style:** QLoRA (4-bit NF4 base loading, bf16 compute)
-- **Scope:** Plan A MVP uplift for cybersecurity and malware-analysis assistance
-## Key Training Setup
-- **Sequence length:** 2048
-- **Batch:** 2
-- **Gradient accumulation:** 8 (effective 16)
-- **Learning rate:** 2e-4 (cosine scheduler)
-- **Steps:** 3000 (completed run)
-- **LoRA rank/alpha:** r=32, alpha=64
-- **LoRA targets:** `q_proj`, `k_proj`, `v_proj`, `o_proj` (attention-only)
-- **Optimizer:** paged_adamw_8bit
-- **Precision:** bf16
-## Hardware Used
-Training was run on RunPod:
-- **GPU:** NVIDIA A100 PCIe 80GB (1x)
-- **vCPU:** 8
-- **RAM:** 125 GB
-- **Disk:** 200 GB
-- **Location:** CA
-## Data Summary
-Curated cybersecurity instruction corpus with mixed sources (CyberMetric, Trendyol CyberSec, ShareGPT Cybersecurity, NIST downsampled, MITRE ATT&CK, CVE/IR/malware-focused sets).
-Final working files used:
-- `train.jsonl`: 120,912 samples
-- `eval.jsonl`: 1,915 samples
-- `cybermetric_80.jsonl`: 80 held-out MCQs
-- `malware_eval_25.jsonl`: 25 expert malware prompts
-## Evaluation Results
-### Standard post-eval settings
-Generation settings used for fair base-vs-adapter comparison:
-- `do_sample=False`
-- `temperature=0.0`
-- `max_new_eval=64`
-- `max_new_cyber=48`
-- `max_new_malware=256`
-#### Baseline (corrected) vs Fine-tuned
-| Metric | Baseline | Fine-tuned | Delta |
-|---|---:|---:|---:|
-| Eval mean overlap | 0.3283 | 0.3631 | +0.0349 |
-| Eval exact match rate | 0.0000 | 0.2193 | +0.2193 |
-| CyberMetric-80 accuracy | 0.825 | 0.900 | +0.075 |
-| Malware structure | 0.44 | 0.84 | +0.40 |
-| Malware ATT&CK correctness | 0.16 | 0.20 | +0.04 |
-| Malware reasoning | 0.24 | 0.20 | -0.04 |
-| Malware evidence awareness | 0.48 | 0.52 | +0.04 |
-| Malware analyst usefulness | 0.52 | 0.56 | +0.04 |
-### Malware-only rerun with longer output budget
-To test truncation effects on malware prompts, both base and fine-tuned were rerun with `max_new_malware=512` (25 prompts only).
-| Rubric axis | Base (512) | Fine-tuned (512) | Delta |
-|---|---:|---:|---:|
-| Structure | 0.56 | 0.88 | +0.32 |
-| ATT&CK correctness | 0.16 | 0.20 | +0.04 |
-| Malware reasoning | 0.36 | 0.28 | -0.08 |
-| Evidence awareness | 0.56 | 0.64 | +0.08 |
-| Analyst usefulness | 0.64 | 0.80 | +0.16 |
-Interpretation: structure/evidence/usefulness improved strongly, but malware reasoning remains the main gap for future iterations.
-## Limitations
-- This is a **Plan A MVP adapter**, not a fully specialized malware reverse-engineering model.
-- Malware causal reasoning still needs improvement via targeted data and/or evidence-grounded training (Plan B).
-- Outputs should be treated as analyst assistance, not an autonomous verdict.
 ## Usage
 ```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
 from peft import PeftModel
-base_model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
-adapter_repo = "umer07/fathom-mixtral-lora-plan-a"
-bnb_config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_quant_type="nf4",
-    bnb_4bit_use_double_quant=True,
-    bnb_4bit_compute_dtype=torch.bfloat16,
-)
-tokenizer = AutoTokenizer.from_pretrained(base_model_id, use_fast=True)
-if tokenizer.pad_token is None:
-    tokenizer.pad_token = tokenizer.eos_token
 model = AutoModelForCausalLM.from_pretrained(
-    base_model_id,
-    quantization_config=bnb_config,
-    device_map={"": 0},
     torch_dtype=torch.bfloat16,
-    low_cpu_mem_usage=True,
 )
-model = PeftModel.from_pretrained(model, adapter_repo)
 model.eval()
 prompt = """### Instruction:
-Analyze the malware behavior and map likely ATT&CK techniques.
 ### Input:
-Sample creates scheduled task persistence and launches encoded PowerShell.
-### Response:
-"""
 inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
 with torch.inference_mode():
-    out = model.generate(**inputs, max_new_tokens=512, do_sample=False, temperature=0.0)
 print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
 ```
-## Project Status
-- Core Plan A training/evaluation cycle: **completed**
-- GPU instance used for training has been deleted
-- No additional training is currently in progress
 ## Citation
-If you use this adapter, please cite your project report/thesis for Fathom Plan A and reference the base model (`mistralai/Mixtral-8x7B-Instruct-v0.1`).

 ---
+language:
+- en
+license: apache-2.0
 base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
 tags:
 - cybersecurity
 - malware-analysis
 - lora
+- peft
 - mixtral
+- threat-intelligence
+- security
 pipeline_tag: text-generation
 ---
+# Fathom — Cybersecurity Expert LLM
+**Fathom** is a mixture-of-experts cybersecurity analysis system built on [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) with 10 domain-specific LoRA adapters. Each adapter is fine-tuned on a curated cybersecurity dataset for a specific analysis domain, enabling specialized reasoning across the full malware analysis pipeline.
+> **FYP (Final Year Project)** — Muhammad Haseeb, i221698
+---
+## Model Architecture
+| Component | Details |
+|---|---|
+| Base Model | Mixtral-8x7B-Instruct-v0.1 (MoE, 47B params, 8×7B experts) |
+| Fine-tuning | LoRA (rank=32, alpha=64, dropout=0.05) |
+| Precision | BFloat16 full precision (no quantization) |
+| Training Hardware | AMD MI300X VF (205.8 GB VRAM), ROCm 7.0 |
+| Framework | PEFT + TRL (SFTTrainer), Alpaca instruction format |
+| Adapter Count | 10 (1 unified + 9 domain experts) |
+---
+## Adapters
+| Adapter | Domain | Training Examples | Description |
+|---|---|---|---|
+| `unified-v2` *(root)* | General Cybersecurity | 9,000+ | Unified adapter across all domains — use as default |
+| `adapters/expert-e1-static` | Static Analysis | 2,500+ | PE analysis, YARA rules, entropy, imports |
+| `adapters/expert-e2-dynamic` | Dynamic / Behavioral | 2,500+ | API call sequences, sandbox reports, process injection |
+| `adapters/expert-e3-network` | Network Analysis | 2,000+ | C2 detection, DNS/HTTP IOC analysis, traffic patterns |
+| `adapters/expert-e4-forensics` | Digital Forensics | 2,000+ | Memory forensics, artifact analysis, timeline reconstruction |
+| `adapters/expert-e5-threatintel` | Threat Intelligence | 9,532 | APT attribution, MITRE ATT&CK mapping, IOC enrichment |
+| `adapters/expert-e6-detection` | Detection Engineering | 2,000+ | YARA, Sigma, Snort rule generation |
+| `adapters/expert-e7-reports` | Report Generation | 2,000+ | Structured incident reports, executive summaries |
+| `adapters/expert-e8-analyst` | Analyst Assistance | 2,000+ | Triage, prioritization, analyst Q&A |
+| `adapters/expert-e9-cot` | Chain-of-Thought | 2,000+ | Step-by-step reasoning for complex analysis tasks |
+---
+## Benchmark Results
+All evaluations run on AMD MI300X (ROCm 7.0), bf16 full precision, greedy decode (temperature=0).
+### CyberMetric-80 (Multiple Choice — Cybersecurity Knowledge)
+| Adapter | Accuracy |
+|---|---|
+| **unified-v2** | **91.25%** |
+| expert-e8-analyst | 91.25% |
+| expert-e3-network | 90.00% |
+| expert-e4-forensics | 90.00% |
+| expert-e2-dynamic | 85.00% |
+| expert-e9-cot | 87.50% |
+| expert-e7-reports | 88.75% |
+| expert-e6-detection | 88.75% |
+| expert-e1-static | 83.75% |
+| expert-e5-threatintel | 81.25% |
+### Malware Analysis Rubric (25 open-ended samples, scored 0–1)
+| Metric | unified-v2 | Best Expert |
+|---|---|---|
+| Structure | 0.96 | 0.96 (e5, e7) |
+| MITRE ATT&CK Correctness | 0.20 | 0.20 (e3, e4, e6) |
+| Malware Reasoning | 0.24 | 0.32 (e9-cot) |
+| Evidence Awareness | 0.68 | 1.00 (e2-dynamic) |
+| Analyst Usefulness | 0.84 | 0.88 (e1, e3, e7) |
+### MMLU Cybersecurity (unified-v2)
+| Benchmark | Questions | Accuracy |
+|---|---|---|
+| MMLU Computer Security | 100 | **79.0%** |
+| MMLU Security Studies | 100 | **64.0%** |
+| TruthfulQA MC1 | 100 | **65.0%** |
+### Q&A Eval — Fathom Cybersecurity Dataset (200 samples, unified-v2)
+| Metric | Score |
+|---|---|
+| Token Overlap (ROUGE-like) | 0.467 |
+| Exact Match Rate | 1.5% |
+| Mean Throughput | 15.5 tok/s |
+---
 ## Usage
 ```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
 from peft import PeftModel
+import torch
+BASE_MODEL = "mistralai/Mixtral-8x7B-Instruct-v0.1"
+ADAPTER    = "umer07/fathom-mixtral"           # unified-v2 (default)
+# For expert: "umer07/fathom-mixtral/adapters/expert-e2-dynamic"
+tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, use_fast=True)
 model = AutoModelForCausalLM.from_pretrained(
+    BASE_MODEL,
+    device_map="auto",
     torch_dtype=torch.bfloat16,
 )
+model = PeftModel.from_pretrained(model, ADAPTER)
 model.eval()
 prompt = """### Instruction:
+Analyze this CAPEv2 sandbox report excerpt and identify the malware family,
+behavioral patterns, and MITRE ATT&CK techniques.
 ### Input:
+File: suspicious.exe | CAPE Malscore: 9.5/10
+API Calls: CreateFileW, WriteProcessMemory, CreateRemoteThread, RegSetValueExW
+DNS: update.microsoft-cdn.net, api.telemetry-svc.com
+Registry: HKCU\\Software\\Microsoft\\Windows\\CurrentVersion\\Run\\SvcHost32
+### Response:"""
 inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
 with torch.inference_mode():
+    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
 print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
 ```
+---
+## Fathom Pipeline
+The full Fathom system includes:
+1. **CAPEv2 Extraction Layer** — parses sandbox JSON reports into structured evidence
+2. **Domain Classifier** — sentence-transformer embeddings → cosine similarity → adapter selection
+3. **RAG Retriever** — FAISS index of domain knowledge (on `umer07/fathom-expert-data`)
+4. **Expert Adapter Registry** — loads the appropriate LoRA adapter per query
+5. **Prompt Templates** — domain-specific instruction prompts per expert
+6. **Guardrails** — output filtering for hallucination / harmful content
+7. **Inference Engine** — unified generation with adapter hot-swap
+8. **FastAPI Backend** — REST API for integration
+---
+## Training Data
+Training datasets are published at [umer07/fathom-expert-data](https://huggingface.co/datasets/umer07/fathom-expert-data).
+Sources include:
+- CAPE sandbox reports (real malware execution data)
+- URLhaus threat feed (malicious URL classification)
+- Atomic Red Team ATT&CK simulations
+- GTFOBins living-off-the-land binaries
+- MITRE ATT&CK STIX bundles
+- CyberMetric, SecQA, and curated cybersecurity QA pairs
+- LOLBAS project
+---
 ## Citation
+```
+@misc{fathom2026,
+  title  = {Fathom: A Mixture-of-Expert LLM Framework for Cybersecurity Analysis},
+  author = {Muhammad Haseeb},
+  year   = {2026},
+  note   = {Final Year Project, FAST-NUCES}
+}
+```