umer07
/

fathom-mixtral

@@ -1,202 +1,166 @@
 ---
 base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
 library_name: peft
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]
-### Framework versions
-- PEFT 0.11.1

 ---
 base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
 library_name: peft
+tags:
+- cybersecurity
+- malware-analysis
+- peft
+- lora
+- qlora
+- mixtral
+language:
+- en
+pipeline_tag: text-generation
+license: apache-2.0
 ---
+# Fathom Plan A LoRA Adapter (Mixtral-8x7B-Instruct)
+This repository contains the **Plan A** LoRA adapter for the Fathom FYP project:
+**"Fathom: An LLM-Powered Automated Malware Analysis Framework"**
+The adapter is trained on a curated cybersecurity instruction-tuning corpus to improve analyst-style security outputs over the base `mistralai/Mixtral-8x7B-Instruct-v0.1` model.
+## What This Is
+- **Type:** PEFT LoRA adapter (not a full standalone model)
+- **Base model required:** `mistralai/Mixtral-8x7B-Instruct-v0.1`
+- **Training style:** QLoRA (4-bit NF4 base loading, bf16 compute)
+- **Scope:** Plan A MVP uplift for cybersecurity and malware-analysis assistance
+## Key Training Setup
+- **Sequence length:** 2048
+- **Batch:** 2
+- **Gradient accumulation:** 8 (effective 16)
+- **Learning rate:** 2e-4 (cosine scheduler)
+- **Steps:** 3000 (completed run)
+- **LoRA rank/alpha:** r=32, alpha=64
+- **LoRA targets:** `q_proj`, `k_proj`, `v_proj`, `o_proj` (attention-only)
+- **Optimizer:** paged_adamw_8bit
+- **Precision:** bf16
+## Hardware Used
+Training was run on RunPod:
+- **GPU:** NVIDIA A100 PCIe 80GB (1x)
+- **vCPU:** 8
+- **RAM:** 125 GB
+- **Disk:** 200 GB
+- **Location:** CA
+## Data Summary
+Curated cybersecurity instruction corpus with mixed sources (CyberMetric, Trendyol CyberSec, ShareGPT Cybersecurity, NIST downsampled, MITRE ATT&CK, CVE/IR/malware-focused sets).
+Final working files used:
+- `train.jsonl`: 120,912 samples
+- `eval.jsonl`: 1,915 samples
+- `cybermetric_80.jsonl`: 80 held-out MCQs
+- `malware_eval_25.jsonl`: 25 expert malware prompts
+## Evaluation Results
+### Standard post-eval settings
+Generation settings used for fair base-vs-adapter comparison:
+- `do_sample=False`
+- `temperature=0.0`
+- `max_new_eval=64`
+- `max_new_cyber=48`
+- `max_new_malware=256`
+#### Baseline (corrected) vs Fine-tuned
+| Metric | Baseline | Fine-tuned | Delta |
+|---|---:|---:|---:|
+| Eval mean overlap | 0.3283 | 0.3631 | +0.0349 |
+| Eval exact match rate | 0.0000 | 0.2193 | +0.2193 |
+| CyberMetric-80 accuracy | 0.825 | 0.900 | +0.075 |
+| Malware structure | 0.44 | 0.84 | +0.40 |
+| Malware ATT&CK correctness | 0.16 | 0.20 | +0.04 |
+| Malware reasoning | 0.24 | 0.20 | -0.04 |
+| Malware evidence awareness | 0.48 | 0.52 | +0.04 |
+| Malware analyst usefulness | 0.52 | 0.56 | +0.04 |
+### Malware-only rerun with longer output budget
+To test truncation effects on malware prompts, both base and fine-tuned were rerun with `max_new_malware=512` (25 prompts only).
+| Rubric axis | Base (512) | Fine-tuned (512) | Delta |
+|---|---:|---:|---:|
+| Structure | 0.56 | 0.88 | +0.32 |
+| ATT&CK correctness | 0.16 | 0.20 | +0.04 |
+| Malware reasoning | 0.36 | 0.28 | -0.08 |
+| Evidence awareness | 0.56 | 0.64 | +0.08 |
+| Analyst usefulness | 0.64 | 0.80 | +0.16 |
+Interpretation: structure/evidence/usefulness improved strongly, but malware reasoning remains the main gap for future iterations.
+## Limitations
+- This is a **Plan A MVP adapter**, not a fully specialized malware reverse-engineering model.
+- Malware causal reasoning still needs improvement via targeted data and/or evidence-grounded training (Plan B).
+- Outputs should be treated as analyst assistance, not an autonomous verdict.
+## Usage
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+from peft import PeftModel
+base_model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
+adapter_repo = "umer07/fathom-mixtral-lora-plan-a"
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_compute_dtype=torch.bfloat16,
+)
+tokenizer = AutoTokenizer.from_pretrained(base_model_id, use_fast=True)
+if tokenizer.pad_token is None:
+    tokenizer.pad_token = tokenizer.eos_token
+model = AutoModelForCausalLM.from_pretrained(
+    base_model_id,
+    quantization_config=bnb_config,
+    device_map={"": 0},
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+)
+model = PeftModel.from_pretrained(model, adapter_repo)
+model.eval()
+prompt = """### Instruction:
+Analyze the malware behavior and map likely ATT&CK techniques.
+### Input:
+Sample creates scheduled task persistence and launches encoded PowerShell.
+### Response:
+"""
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+with torch.inference_mode():
+    out = model.generate(**inputs, max_new_tokens=512, do_sample=False, temperature=0.0)
+print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+## Project Status
+- Core Plan A training/evaluation cycle: **completed**
+- GPU instance used for training has been deleted
+- No additional training is currently in progress
+## Citation
+If you use this adapter, please cite your project report/thesis for Fathom Plan A and reference the base model (`mistralai/Mixtral-8x7B-Instruct-v0.1`).