You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These safety-gate weights are research artifacts from an in-progress MSc thesis and are gated during the review/embargo period. Please tell us who you are and confirm defensive/research intent.
Log in or Sign Up to review the conditions and access this model content.
elrashid/gate-gemma-3-4b-it-l17
A SAE safety gate for google/gemma-3-4b-it (multimodal, text tower): logistic regression over
sparse-autoencoder features of the layer-17 residual stream, scoring P(harmful) from the prompt before generation.
Part of The Refusal Slope (MSc thesis, BUiD). Companion SAE: elrashid/sae-gemma-3-4b-it-topk-l17.
TL;DR
- 26 SAE features out of 40,960 carry the decision.
- OOD ROC-AUC 0.978 (95% CI 0.961–0.991) at FP16; 0.981 at NF4 — slightly better quantized.
- The lowest over-refusal in the family: 1.3% tricky-benign FPR.
- At threshold 0.849: precision 1.00, recall 0.66, zero OOD false positives.
How it works
The gate hooks the language tower of this multimodal model (the vision tower is bypassed). During prefill, the layer-17 residual at the last prompt token is encoded into 40,960 sparse features; 26 weighted features give P(harmful) in about a millisecond. The verdict precedes the first generated token.
Evaluation
| metric | FP16 | NF4-4bit |
|---|---|---|
| ROC-AUC, in-distribution | 1.000 | 1.000 |
| ROC-AUC, out-of-distribution | 0.978 (CI 0.961–0.991) | 0.981 (CI 0.964–0.992) |
| Refusal-direction probe (baseline), OOD | 0.753 | 0.767 |
| Confusion at threshold 0.849 (OOD) | TP 66 · FP 0 · TN 100 · FN 34 | — |
| Precision / Recall / F1 | 1.00 / 0.66 / 0.795 | — |
| Over-refusal FPR (xsafe-style tricky-benign) | 0.013 | 0.013 |
Honesty note: the 1.3% over-refusal rate is the family's best benign behaviour, but the same conservative threshold costs recall (0.66). A separate caveat from the thesis's Exp17: at INT8 this base model produced word-salad in a latency run (a boundary case of bitsandbytes INT8 with this architecture) — the gate itself is unaffected, but test your quantized deployment end to end.
How to use
import numpy as np
from huggingface_hub import hf_hub_download
# 1) z = (1, 40960) SAE features of the LAST prompt token
# (see elrashid/sae-gemma-3-4b-it-topk-l17 for the multimodal encode snippet)
g = np.load(hf_hub_download("elrashid/gate-gemma-3-4b-it-l17", "gate.npz"))
p = 1.0 / (1.0 + np.exp(-(z @ g["sae_coef"].T + g["sae_intercept"]))) # P(harmful)
flag = bool(p >= float(g["op_threshold"])) # 0.849
gate.npz fields: sae_coef (1×40960), sae_intercept, feature_ids (26 active features), op_threshold,
refusal_dir, layer.
Limitations
- Specific to gemma-3-4b-it at layer 17 with this SAE; text-tower activations only — image-borne harm is unseen.
- Hundreds of evaluation prompts; quote the CI.
- Recall at the shipped threshold is a precision-first tuning choice.
Intended use & ethics
Defensive filtering and safety research only. A detector, not a generator. Downstream use must comply with the Gemma licence.
Citation
Elrashid, M. (2026). The Refusal Slope: A Mechanistic Taxonomy of Feature Fate in Quantized Edge Intelligence. MSc thesis, BUiD.