You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These safety-gate weights are research artifacts from an in-progress MSc thesis and are gated during the review/embargo period. Please tell us who you are and confirm defensive/research intent.
Log in or Sign Up to review the conditions and access this model content.
elrashid/gate-gemma-2-9b-it-l21
A SAE safety gate for google/gemma-2-9b-it: logistic regression over sparse-autoencoder features of the
layer-21 residual stream, returning P(harmful) from the prompt before the first output token. Part of The Refusal
Slope (MSc thesis, BUiD). Companion SAE: elrashid/sae-gemma-2-9b-it-topk-l21.
TL;DR
- The strongest gate in the 11-model family. OOD ROC-AUC 1.00 (95% CI 1.00–1.00) at FP16 and NF4.
- At threshold 0.893: precision 1.00, recall 0.98, F1 0.99 — 98 of 100 held-out harmful prompts caught with zero false positives.
- 21 SAE features out of 57,344 carry the decision.
- Also the model where the refusal-direction baseline is strongest (0.849) — and the gate still clears it.
How it works
During prefill, the layer-21 residual activation at the last prompt token is encoded by the companion SAE into 57,344 sparse features. A 21-weight logistic regression gives P(harmful) in about a millisecond, before any generation happens. gemma-2-9b-it separates harmful from benign so cleanly at this layer that the gate is essentially at ceiling on every split we could construct.
Evaluation
| metric | FP16 | NF4-4bit |
|---|---|---|
| ROC-AUC, in-distribution | 1.000 | 1.000 |
| ROC-AUC, out-of-distribution | 1.000 (CI 1.00–1.00) | 1.000 (CI 1.00–1.00) |
| Refusal-direction probe (baseline), OOD | 0.849 | 0.846 |
| Confusion at threshold 0.893 (OOD) | TP 98 · FP 0 · TN 100 · FN 2 | — |
| Precision / Recall / F1 | 1.00 / 0.98 / 0.990 | — |
| Over-refusal FPR (xsafe-style tricky-benign) | 0.287 | 0.275 |
Honesty note: perfect AUC on a few hundred prompts is not perfect AUC in the wild — a ceiling score mostly says the evaluation set is not hard enough for this model. The one real cost is over-refusal on styled-benign prompts (0.287); the clean-benign FPR is 0.0.
How to use
import numpy as np
from huggingface_hub import hf_hub_download
# 1) z = (1, 57344) SAE features of the LAST prompt token
# (see elrashid/sae-gemma-2-9b-it-topk-l21 for the encode snippet; gemma-2 needs eager attention)
g = np.load(hf_hub_download("elrashid/gate-gemma-2-9b-it-l21", "gate.npz"))
p = 1.0 / (1.0 + np.exp(-(z @ g["sae_coef"].T + g["sae_intercept"]))) # P(harmful)
flag = bool(p >= float(g["op_threshold"])) # 0.893
gate.npz fields: sae_coef (1×57344), sae_intercept, feature_ids (21 active features), op_threshold,
refusal_dir, layer.
Limitations
- Specific to gemma-2-9b-it at layer 21 with this SAE; no cross-model transfer.
- Ceiling metrics on small sets — treat 1.00 as "not falsified here", not as a production guarantee.
- gemma-2 requires
attn_implementation="eager"for activation extraction.
Intended use & ethics
Defensive filtering and safety research only. A detector, not a generator. Downstream use must comply with the Gemma licence.
Citation
Elrashid, M. (2026). The Refusal Slope: A Mechanistic Taxonomy of Feature Fate in Quantized Edge Intelligence. MSc thesis, BUiD.