vete-speis/gemma-4-E2B-it-phishing

Gemma-4-E2B-it — Phishing Email Classifier (LoRA)

LoRA fine-tune of google/gemma-4-E2B-it that classifies emails as phishing or legit with a single-word answer.

This repo contains:

• the LoRA adapter (PEFT, ~50 MB) — reproducible, composable
• ready-to-run GGUF quants (merged model) for llama.cpp / LM Studio: Q4_K_M (3.4 GB) and Q8_0 (5 GB)

Results

Evaluated on 1,000 held-out emails (balanced, English):

Model	Accuracy	Precision	Recall	F1
gemma-4-E2B-it zero-shot	85.7%	90.3%	78.9%	0.842
+ this LoRA	95.6%	98.4%	98.0%	0.982

Notes: 26/1000 outputs were off-format and counted as errors (accuracy on parsable outputs: 98.2%). Zero-shot baseline measured on a 300-email subset.

Usage

The model expects this exact system prompt (it was trained with it):

You are an email security classifier. Classify the email as 'phishing' or 'legit'. Respond with one word only.

LM Studio / llama.cpp (GGUF)

Download phishing-gemma4-e2b-Q4_K_M.gguf, set the system prompt above, temperature 0, paste an email as the user message. Requires a llama.cpp build recent enough for the gemma4 architecture.

Transformers + PEFT (adapter)

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "google/gemma-4-E2B-it"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "vete-speis/gemma-4-E2B-it-phishing")

msgs = [
    {"role": "system", "content": "You are an email security classifier. Classify the email as 'phishing' or 'legit'. Respond with one word only."},
    {"role": "user", "content": "Your account has been suspended. Verify now: http://secure-login.example.xyz"},
]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=5, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))  # -> phishing

Training

• Base: google/gemma-4-E2B-it (2.3B effective params)
• Data: zefang-liu/phishing-email-dataset, cleaned, class-balanced 50/50, 10.5k train / 500 val / 1,000 test, emails truncated to 1,500 chars
• Method: SFT with TRL SFTTrainer + PEFT LoRA — r=16, alpha=32, dropout 0.05, targets: the inner nn.Linear of q/k/v/o projections (Gemma 4 wraps them in Gemma4ClippableLinear, which PEFT cannot wrap directly)
• Run: 1 epoch, effective batch 16 (4×4), seq len 512, lr 2e-4 cosine, bf16, gradient checkpointing — ~15 min on a single H100 (Modal)

Limitations

• Trained on English emails. It generalizes surprisingly well to other languages (the base is multilingual), but no formal eval outside English.
• The dataset's "phishing" class includes generic spam, so the model leans toward flagging unsolicited marketing/cold-outreach emails as phishing. If you need a strict spam ≠ phishing distinction, you need 3-class data.
• ~2.6% of outputs may be off-format; parse with a contains-check on phishing/legit and treat anything else as abstention.
• Not a substitute for a mail security gateway. Use as a triage aid.

License

Gemma derivatives are governed by the Gemma Terms of Use. The adapter and GGUF files here are derivatives of google/gemma-4-E2B-it.