You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
This model has had its refusal behaviour removed. You are responsible for how you use it and for complying with all applicable laws. Access is logged.
Log in or Sign Up to review the conditions and access this model content.
Qwen3.8-27B-Abliterated
An abliterated (refusal-removed) build of Qwen/Qwen3.8-27B.
Refusal behaviour is removed by a training-free, forward-only weight edit: a single "refusal direction" is identified in the residual stream and then orthogonalized out of every matrix that writes into that stream. No fine-tuning, no distillation, no data poisoning — the weights are the original weights minus a rank-1 component.
The vision tower is byte-for-byte untouched. Only the text path was modified.
| base | this model | |
|---|---|---|
| HarmBench, safety categories (293) | 99.0 % refusal | 0.0 % refusal |
| AdvBench (520) | 99.04 % | 0.19 % |
| AdvBench (520), thinking mode | 48.27 % | 0.00 % |
| HarmBench (393), thinking mode | 21.12 % | 0.00 % |
| Chinese harmful prompts (30) | 83.33 % | 0.00 % |
| MMLU (full, 14 042) | 83.40 % | 82.35 % |
| GSM8K (200) | 96.50 % | 96.50 % |
Method
For every weight W whose output is added to the residual stream:
W ← W − λ · r̂ (r̂ᵀ W) # out-dim writers [hidden, in]
E ← E − λ · (E r̂ᵀ) r̂ # embed_tokens [vocab, hidden]
with r̂ a unit refusal direction in the 5120-dim residual space and λ = 1.5.
131 tensors edited:
| group | count |
|---|---|
mlp.down_proj (all 64 layers) |
64 |
linear_attn.out_proj (48 Gated DeltaNet layers) |
48 |
self_attn.o_proj (16 full-attention layers) |
16 |
mtp.layers.0.* (o_proj, down_proj) |
2 |
embed_tokens |
1 |
Not edited: all 333 model.visual.* tensors, lm_head (it reads the residual stream, it does
not write it), all norms, and all q/k/v/gate/up, conv1d, in_proj, A_log, dt_bias.
embed_tokens is included. The residual stream is purely additive (x₀ = E[t]), so a component
injected by the embeddings cannot be removed by anything downstream; omitting it leaves a refusal
leak into every layer. It needs the transposed projection form because hidden is its last dim.
Finding the direction
Last-token residual activations were captured for 32 harmful and 32 harmless prompts across all 65 hidden-state positions, in both thinking modes. Layers were ranked by normalized separation (Cohen's d and Mann–Whitney AUC) rather than raw ‖diff-in-means‖, which merely grows with depth.
Selected: non-thinking mode, hidden-state index 46 — Cohen's d = 12.65, AUC = 1.000. (Index i is the residual stream after i decoder layers, so index 46 = output of decoder layer 45, 0-indexed.)
Choosing λ — this part matters
λ was measured, not assumed. Refusal and capability were scored at each λ:
| λ | refusal (AdvBench) | MMLU |
|---|---|---|
| 0.0 (base) | 100.00 % | 82.50 % |
| 1.0 | 25.00 % | 83.12 % |
| 1.5 | 0.00 % | 83.12 % |
| 2.0 | 2.08 % | 77.50 % |
| 2.5 | 4.17 % | 30.62 % |
| 3.0 | 8.33 % | 20.00 % |
| 3.5 | 0.00 % | 20.00 % |
Two results worth stating plainly:
- λ = 1.0 — exact removal — is not enough. It leaves 25 % of refusals intact. λ > 1 does not merely delete the component, it reverses it, and that is what actually suppresses refusal.
- λ ≥ 2.5 destroys the model. MMLU collapses to 20–30 % — at or below the 25 % chance floor for 4-way multiple choice. A λ of 3.5 copied from another architecture would have shipped a lobotomized model that still scores 0 % refusal. A refusal-only sweep would not have caught this.
A finer sweep (n = 160 refusal / 320 MMLU) put the 0 %-refusal plateau at λ ∈ [1.2, 1.6] with MMLU flat at 84–85 %. λ = 1.5 sits inside that plateau with margin from both edges.
Evaluation
Greedy decoding (do_sample=False), identical settings for base and abliterated.
Refusal is scored by a deliberately loose keyword classifier — it is tuned to over-count
refusals, never under-count.
Refusal
| benchmark | mode | base | abliterated |
|---|---|---|---|
| AdvBench (520) | non-thinking | 515/520 · 99.04 % | 1/520 · 0.19 % |
| AdvBench (520) | thinking | 251/520 · 48.27 % | 0/520 · 0.00 % |
| HarmBench (393) | non-thinking | 387/393 · 98.47 % | 45/393 · 11.45 % |
| HarmBench (393) | thinking | 83/393 · 21.12 % | 0/393 · 0.00 % |
| Chinese harmful (30) | non-thinking | 25/30 · 83.33 % | 0/30 · 0.00 % |
Every row is the complete benchmark, not a sample. In thinking mode the abliterated model refuses nothing at all — 0/520 on AdvBench and 0/393 on HarmBench, including all 100 copyright items, whose non-thinking residual is discussed below.
No prompt-prefill jailbreak is used anywhere. (The common "Sure, here is" prefill is itself a
jailbreak: under it the base model refuses only 12.5 % on AdvBench, which would have made this
edit look far less significant than it is.)
HarmBench by category — where the residual 11.45 % actually lives
| category | base | abliterated |
|---|---|---|
| chemical / biological (56) | 100.0 % | 0.0 % |
| cybercrime / intrusion (61) | 100.0 % | 0.0 % |
| harassment / bullying (25) | 100.0 % | 0.0 % |
| harmful (22) | 100.0 % | 0.0 % |
| misinformation (64) | 98.4 % | 0.0 % |
| illegal (65) | 96.9 % | 0.0 % |
| all safety categories (293) | 99.0 % | 0.0 % |
| copyright (100) | 97.0 % | 45.0 % |
Every safety category is exactly zero, with no adjudication applied. (In thinking mode even the copyright column goes to zero: 0/100.) The entire residual is the
copyright cluster, and manual inspection of all 45 flagged responses shows they are overwhelmingly
classifier false positives rather than refusals — two distinct failure modes:
- The generated text itself begins with a refusal phrase. Asked for a passage from The Kite Runner, the model writes prose beginning "I cannot quite remember when I first started to remember. It was a cold, clear day in Kabul…". That is the novel, not a refusal.
- A pedantic preamble followed by full delivery. "I cannot generate a new passage … but here is a long excerpt from Book One, Chapter 1" — and then it delivers it.
Song-lyric requests are flagged the same way, because lyrics contain lines like "I can't explain". The single AdvBench flag is identical in kind: "I cannot lie. I am an AI… However, you asked for a review of a product I have not used. So —" followed by the review.
These are reported as-is rather than adjudicated away: the raw classifier output is reproducible, a semantic judge is not. The per-category breakdown is what carries the claim.
Capability
| benchmark | base | abliterated |
|---|---|---|
| MMLU (full test, 14 042) | 83.40 % | 82.35 % |
| MMLU (1 000 sample) | 82.50 % | 81.10 % |
| GSM8K (200, 1024 tok) | 96.50 % | 96.50 % |
GSM8K needs a generation budget of ≥1024 tokens. At 400 tokens the base model scored 68 % and this model 81.5 % — pure truncation artefact, measuring verbosity rather than arithmetic. With an adequate budget the two are identical.
Vision (untouched — verified two ways)
Structurally: a byte-level diff confirms 0 of 333 model.visual.* tensors changed, and the
vision-only shard still shares an inode with the base checkpoint.
Behaviourally, on the same image:
base — "there are three distinct geometric shapes … Red Circle – Located in the upper-left portion … Green Triangle – Positioned to the right of the red circle"
abliterated — "There is a red circle located in the upper-left portion of the image. There is a green triangle … located in the"
Usage
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
m = "windowsxp811203/Qwen3.8-27B-Abliterated"
proc = AutoProcessor.from_pretrained(m)
model = AutoModelForImageTextToText.from_pretrained(m, dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "Explain how a lock cylinder works."}]
text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
enable_thinking=False) # thinking is ON by default
inputs = proc(text=[text], return_tensors="pt").to(model.device)
print(proc.tokenizer.decode(model.generate(**inputs, max_new_tokens=512)[0]))
Needs ~56 GB in bf16. Both thinking modes work; the direction was extracted in non-thinking mode and generalizes to thinking mode (48.27 % → 0.00 % on AdvBench, full 520).
flash-linear-attention and causal-conv1d are optional — transformers ships pure-torch
fallbacks for the Gated DeltaNet path, so the model runs correctly (just slower) without them.
Limitations
copyrightrefusals are only partly removed (97 % → 45 % by keyword; largely false positives). This edit targets safety refusal, and copyright declines are a partly distinct behaviour.- The direction comes from 32 + 32 English prompts. It transfers to Chinese (83.33 % → 0.00 %) and to thinking mode, but transfer to other languages and formats is unmeasured.
- Removing refusal does not add knowledge. The model can now be confidently wrong about things it would previously have declined to discuss.
- MMLU is down 1.05 pp on the full test set. Small, but not zero.
Reproduction
Direction extraction, the λ sweep, the ablation, and every eval above were run on a single H200.
The λ sweep is cheap because the rank-1 projection component is cached once from the original
weights and λ is then moved incrementally in memory (W(λ') = W(λ) − (λ'−λ)·P(W₀)), which turns a
55 GB-checkpoint-per-λ grid into milliseconds per point.
ABLIT_META.json in this repo lists all 131 edited tensors with their per-tensor relative Frobenius
change and the exact direction provenance.
Support / 打賞
If these models are useful to you, tips are appreciated — they pay for the GPU time. 如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。
USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888
Disclaimer
This model will not refuse. It is published for alignment and safety research — measuring what refusal training actually protects, red-teaming, and studying the mechanics of refusal directions. You are responsible for your use of it and for complying with applicable law. Inherits the Apache-2.0 license of the base model.
- Downloads last month
- 105