You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These safety-gate weights are research artifacts from an in-progress MSc thesis and are gated during the review/embargo period. Please tell us who you are and confirm defensive/research intent.
Log in or Sign Up to review the conditions and access this model content.
elrashid/gate-llama-3.1-8b-instruct-l16
A SAE safety gate for meta-llama/Llama-3.1-8B-Instruct: logistic regression over sparse-autoencoder
features of the layer-16 residual stream, scoring P(harmful) from the prompt before generation starts. Part of
The Refusal Slope (MSc thesis, BUiD). Companion SAE: elrashid/sae-llama-3.1-8b-instruct-topk-l16.
TL;DR
- 30 SAE features out of 65,536 carry the decision — the second-widest gate in the family.
- OOD ROC-AUC 0.942 (95% CI 0.903–0.974) at FP16; 0.934 at NF4 (−0.008, inside the CI).
- At threshold 0.875: precision 0.981, recall 0.52 (a single OOD false positive).
- Low over-refusal: 7.5% tricky-benign FPR at FP16, improving to 6.2% at NF4.
How it works
During prefill, the layer-16 residual at the last prompt token is encoded by the companion SAE into 65,536 sparse features; 30 weighted features give P(harmful) in about a millisecond, before the first output token. Llama-3.1 was one of the three deep-dive anchor models in the thesis, so this gate has the most cross-experiment context behind it (behavioural grid, feature-fate, auto-interp, and cascade replay all cover it).
Evaluation
| metric | FP16 | NF4-4bit |
|---|---|---|
| ROC-AUC, in-distribution | 1.000 | 1.000 |
| ROC-AUC, out-of-distribution | 0.942 (CI 0.903–0.974) | 0.934 (CI 0.892–0.969) |
| Refusal-direction probe (baseline), OOD | 0.628 | 0.619 |
| Confusion at threshold 0.875 (OOD) | TP 52 · FP 1 · TN 99 · FN 48 | — |
| Precision / Recall / F1 | 0.981 / 0.52 / 0.680 | — |
| Over-refusal FPR (xsafe-style tricky-benign) | 0.075 | 0.062 |
Honesty note: a solid mid-pack gate. The precision-first threshold halves recall at the operating point — the 0.942 AUC is the capability; move the threshold if your deployment tolerates false positives. The NF4 delta is within the confidence interval, so treat the gate as quantization-stable.
How to use
import numpy as np
from huggingface_hub import hf_hub_download
# 1) z = (1, 65536) SAE features of the LAST prompt token
# (see elrashid/sae-llama-3.1-8b-instruct-topk-l16 for the encode snippet)
g = np.load(hf_hub_download("elrashid/gate-llama-3.1-8b-instruct-l16", "gate.npz"))
p = 1.0 / (1.0 + np.exp(-(z @ g["sae_coef"].T + g["sae_intercept"]))) # P(harmful)
flag = bool(p >= float(g["op_threshold"])) # 0.875
gate.npz fields: sae_coef (1×65536), sae_intercept, feature_ids (30 active features), op_threshold,
refusal_dir, layer.
Limitations
- Specific to Llama-3.1-8B-Instruct at layer 16 with this SAE; no cross-model transfer.
- Hundreds of evaluation prompts; the CI is the honest claim.
- Recall at the shipped threshold reflects precision-first tuning, not the separability ceiling.
Intended use & ethics
Defensive filtering and safety research only. A detector, not a generator. Downstream use must comply with the Llama 3.1 licence.
Citation
Elrashid, M. (2026). The Refusal Slope: A Mechanistic Taxonomy of Feature Fate in Quantized Edge Intelligence. MSc thesis, BUiD.
Model tree for elrashid/gate-llama-3.1-8b-instruct-l16
Base model
meta-llama/Llama-3.1-8B