You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These safety-gate weights are research artifacts from an in-progress MSc thesis and are gated during the review/embargo period. Please tell us who you are and confirm defensive/research intent.
Log in or Sign Up to review the conditions and access this model content.
elrashid/gate-gemma-4-e4b-it-l21
A SAE safety gate for google/gemma-4-E4B-it (MatFormer, multimodal): logistic regression over
sparse-autoencoder features of the layer-21 residual stream, scoring P(harmful) from the prompt before generation.
Part of The Refusal Slope (MSc thesis, BUiD). Companion SAE: elrashid/sae-gemma-4-e4b-it-topk-l21.
TL;DR
- 29 SAE features out of 40,960 carry the decision.
- OOD ROC-AUC 0.851 (95% CI 0.792–0.897) at FP16; 0.822 at NF4 — the largest quantization drop in the family (−0.029), and still a weak-tier gate either way.
- The strangest baseline result of the study: the refusal-direction probe scores 0.353 OOD — below chance. The linear direction anti-predicts harm on this model; the sparse features still predict it.
- At threshold 0.795: precision 0.955, recall 0.42.
How it works
During prefill, the layer-21 residual of the language tower is encoded into 40,960 sparse features; 29 weighted features give P(harmful) in about a millisecond. Like its E2B sibling, the MatFormer architecture dilutes the safety geometry that dense models keep linear — which is precisely why this pair of models matters to the thesis: they show where direction-based safety analysis stops working while feature-based analysis survives.
Evaluation
| metric | FP16 | NF4-4bit |
|---|---|---|
| ROC-AUC, in-distribution | 1.000 | 1.000 |
| ROC-AUC, out-of-distribution | 0.851 (CI 0.792–0.897) | 0.822 (CI 0.763–0.875) |
| Refusal-direction probe (baseline), OOD | 0.353 (below chance) | 0.314 (below chance) |
| Confusion at threshold 0.795 (OOD) | TP 42 · FP 2 · TN 98 · FN 58 | — |
| Precision / Recall / F1 | 0.955 / 0.42 / 0.583 | — |
| Over-refusal FPR (xsafe-style tricky-benign) | 0.062 | 0.037 |
Honesty note: research probe, not production filter. Perfect ID separation collapsing to 0.851 OOD is the family's clearest overfit-shaped gap, the NF4 drop is the family's largest, and recall at threshold is under half. Every one of those numbers is reported as measured.
How to use
import numpy as np
from huggingface_hub import hf_hub_download
# 1) z = (1, 40960) SAE features of the LAST prompt token
# (see elrashid/sae-gemma-4-e4b-it-topk-l21 for the multimodal encode snippet)
g = np.load(hf_hub_download("elrashid/gate-gemma-4-e4b-it-l21", "gate.npz"))
p = 1.0 / (1.0 + np.exp(-(z @ g["sae_coef"].T + g["sae_intercept"]))) # P(harmful)
flag = bool(p >= float(g["op_threshold"])) # 0.795
gate.npz fields: sae_coef (1×40960), sae_intercept, feature_ids (29 active features), op_threshold,
refusal_dir, layer.
Limitations
- Weak-tier OOD generalisation and the family's largest NF4 sensitivity; not a standalone safety layer.
- Specific to gemma-4-E4B-it at layer 21 with this SAE; text-tower activations only.
- The below-chance direction baseline means direction-based tooling should not be trusted on this model at all.
Intended use & ethics
Defensive filtering and safety research only. A detector, not a generator. Downstream use must comply with the Gemma licence.
Citation
Elrashid, M. (2026). The Refusal Slope: A Mechanistic Taxonomy of Feature Fate in Quantized Edge Intelligence. MSc thesis, BUiD.