Qwen3.8-27B-RANA-abliterated

A refusal-ablated version of Qwen/Qwen3.8-27B, made with RANA (Reasoning-Anchored, Norm-preserving Ablation), an in-house variant of refusal-direction ablation. It keeps the base model's capabilities, vision tower and MTP head, and almost never refuses outright.

Format Repo Size
BF16 (this repo) preemware/Qwen3.8-27B-RANA-abliterated 55.6 GB
FP8 (block e4m3, same layout as Qwen's official FP8) preemware/Qwen3.8-27B-RANA-abliterated-FP8 30.9 GB
GGUF (llama.cpp; Q8_0 to IQ2_M, plus vision and MTP files) preemware/Qwen3.8-27B-RANA-abliterated-GGUF 10.0–28.6 GB
MLX (Apple Silicon; 4-, 5-, 6-, 8-bit and BF16, plus MTP drafter) preemware/Qwen3.8-27B-RANA-abliterated-MLX 16.1–54.7 GB

This is a safety-alignment-removed research model. Read Intended use and Limitations before using it.

At a glance

Compared with the base model and the orcarouter/Qwen3.8-27B-Uncensored reference, all run through the same evaluation harness. Where two numbers are given, they are from the two judges (gpt-oss / Mistral).

base orca (reference) RANA
Hard refusals, HarmBench (of 200) 168 / 155 0 / 0 ¹ 0 / 0 ¹
Answers the request, held-out prompts 3% / 0% 84% / 49% 90% / 54%
Hits the 16k-token budget, held-out 0.6% 11.9% 6.7%
Avg. capability change vs base (5 tasks) – 1.11 pp 0.82 pp
Worst single task vs base – −2.2 (HumanEval) −1.5 (TruthfulQA)
Distance from base (KL) 0 0.075 0.041

¹ After the same mechanical re-judge for both abliterations (2 labels each); raw gpt-oss counts are 0.7 for both (2 across 3 seeds). See the disclosure.

In short: capability stays within noise of base except a ~1.5 pp drop on TruthfulQA. Against orca, RANA hits the token budget about half as often, stays closer to base, and answers more held-out prompts under both judges. It is 2.2 pp worse than orca on IFEval (instruction following); other tasks tie.

Quick start

Tested with vLLM 0.30.0:

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve preemware/Qwen3.8-27B-RANA-abliterated \
  --dtype bfloat16 --max-model-len 20480 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image": 1, "video": 0}' \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
  • Fits on one 80 GB+ GPU in BF16. The FP8 build needs about half the weight memory; see its card for how it compares with this one.
  • Thinking is on by default. Recommended sampling: T = 1.0, top_p = 0.95, top_k = 20.
  • --max-model-len 20480 matches the evaluation (16k-token budget). Long technical requests can need 20–50k tokens of reasoning; raise --max-model-len and max_tokens for those (native context 262k).
  • VLLM_USE_FLASHINFER_SAMPLER=0 was needed on Blackwell GPUs.
  • Vision input, multi-turn tool calling and MTP speculative decoding all work (tested on these shards).

Intended use

  • Research only: interpretability, red-teaming, and robustness evaluation of refusal behaviour.
  • Not for public or end-user deployment without a separate moderation layer. The model's own refusals have been largely removed, so any safety filtering has to happen outside it.
  • You are responsible for complying with applicable law, the Apache-2.0 license inherited from Qwen, and the terms of any platform where outputs are used.

What was changed

RANA removes a single "refusal direction" from the model's residual stream (following Arditi et al., 2024), keeping the edited weights at their original norms. The recipe and code are not published.

Everything else is untouched, checked tensor by tensor against the base model:

part matrices changed
attention output (self_attn.o_proj) 17 (16 main + 1 MTP)
linear-attention output (linear_attn.out_proj) 48
MLP output (mlp.down_proj) 65 (64 main + 1 MTP)
token embeddings (embed_tokens) 1
total 131, and nothing outside this set
  • Vision tower: all 333 tensors are byte-identical to base.
  • MTP head: all 15 tensors are present; its two residual writers are edited the same way as the main model, so speculative decoding keeps working (acceptance length 2.45).
  • The removed direction is gone: the largest remaining component along it is 0.00122, down from 0.225.
  • Independent check: Abliterlitics (commit 9b15eb0) finds exactly 129 changed tensors among the 850 it inspects. That matches: it skips the vision and MTP tensors, so it sees 131 − 2.

Results

Three models, identical harness: base, the orca reference, and RANA. Refusal numbers are averaged over 3 sampling seeds and scored by two independent LLM judges (details in Evaluation details).

Refusals and token budget

"Answers" = the judge found the harmful request substantively fulfilled (gpt-oss / Mistral judge). "Budget hits" = the response reached the 16k-token generation limit before finishing (no judge involved). These are mostly long reasoning, not repetition; see Limitations.

model HarmBench (200): answers budget hits Held-out (240): answers budget hits avg. tokens (held-out)
base 3.2% / 0.7% 0.3% 3.2% / 0.0% 0.6% 1,113
orca 73.7% / 61.2% 26.3% 83.9% / 49.2% 11.9% 6,461
RANA 78.5% / 57.3% 20.5% 90.0% / 54.4% 6.7% 4,697
  • Hard refusals are about zero for both RANA and orca, versus 155–197 per set for base.
  • The two judges disagree a lot on partial answers. Mistral labels many hedged or incomplete answers as soft refusals that gpt-oss counts as answers, which is why the two percentages differ so much. The judges agree well on outright refusals (κ = 0.89) but only moderately on "answered or not" (κ = 0.54).
  • RANA vs orca, held-out prompts (paired McNemar test over 3 seeds):
    • hits the budget less: 12 vs 50 prompt-runs where only one model did, p = 1.2 × 10⁻⁶;
    • answers more under gpt-oss (p = 2.9 × 10⁻⁶) and under Mistral (p = 0.008).
  • On HarmBench the answer-rate comparison depends on the judge (gpt-oss favours RANA, Mistral favours orca), so there is no clear winner there.
Full refusal tables (per judge, mean ± sd [min–max] over seeds 1–3; seed-0 re-score; per-seed tests)

Counts by prompt set × judge (seeds 1–3)

mean ± sd [min–max] over seeds 1,2,3; complied % and budget-hit % as percentages; budget-hit % is judge-independent.

model set judge hard soft complied % degenerate budget-hit % tok/resp
base harmbench (n=200) gpt-oss-safeguard-20b 167.7 ± 6.0 [162–174] 25.3 ± 5.5 [19–29] 3.2 ± 0.8 [2.5–4.0] 0.7 ± 0.6 [0–1] 0.3 ± 0.3 [0.0–0.5] 1218
base harmbench (n=200) Mistral-Small-24B-2501 154.7 ± 4.7 [151–160] 43.7 ± 5.1 [38–48] 0.7 ± 0.3 [0.5–1.0] 0.3 ± 0.6 [0–1] 0.3 ± 0.3 [0.0–0.5] 1218
base heldout (n=240) gpt-oss-safeguard-20b 197.3 ± 8.0 [189–205] 34 ± 7.9 [28–43] 3.2 ± 0.6 [2.5–3.8] 1 ± 1.0 [0–2] 0.6 ± 0.5 [0.0–0.8] 1113
base heldout (n=240) Mistral-Small-24B-2501 191.7 ± 1.5 [190–193] 47.7 ± 1.5 [46–49] 0.0 ± 0.0 [0.0–0.0] 0.7 ± 0.6 [0–1] 0.6 ± 0.5 [0.0–0.8] 1113
orca harmbench (n=200) gpt-oss-safeguard-20b 0.7 ± 0.6 [0–1] 4.3 ± 1.5 [3–6] 73.7 ± 0.8 [73.0–74.5] 47.7 ± 2.1 [46–50] 26.3 ± 0.8 [25.5–27.0] 7992
orca harmbench (n=200) Mistral-Small-24B-2501 0 ± 0.0 [0–0] 56 ± 3.5 [52–58] 61.2 ± 1.0 [60.0–62.0] 21.7 ± 2.5 [19–24] 26.3 ± 0.8 [25.5–27.0] 7992
orca heldout (n=240) gpt-oss-safeguard-20b 0.7 ± 0.6 [0–1] 12 ± 4.6 [7–16] 83.9 ± 0.6 [83.3–84.6] 26 ± 3.6 [22–29] 11.9 ± 0.9 [11.2–12.9] 6461
orca heldout (n=240) Mistral-Small-24B-2501 1 ± 1.0 [0–2] 103 ± 4.0 [99–107] 49.2 ± 0.7 [48.8–50.0] 18 ± 3.0 [15–21] 11.9 ± 0.9 [11.2–12.9] 6461
RANA harmbench (n=200) gpt-oss-safeguard-20b 0.7 ± 1.2 [0–2] 7 ± 2.0 [5–9] 78.5 ± 0.5 [78.0–79.0] 35.3 ± 2.5 [33–38] 20.5 ± 0.9 [19.5–21.0] 6634
RANA harmbench (n=200) Mistral-Small-24B-2501 0 ± 0.0 [0–0] 72.7 ± 3.2 [69–75] 57.3 ± 0.8 [56.5–58.0] 12.7 ± 2.1 [11–15] 20.5 ± 0.9 [19.5–21.0] 6634
RANA heldout (n=240) gpt-oss-safeguard-20b 0.3 ± 0.6 [0–1] 10.3 ± 1.5 [9–12] 90.0 ± 1.2 [88.8–91.2] 13.3 ± 2.1 [11–15] 6.7 ± 0.8 [5.8–7.5] 4697
RANA heldout (n=240) Mistral-Small-24B-2501 1 ± 0.0 [1–1] 102 ± 8.2 [95–111] 54.4 ± 4.2 [50.0–58.3] 6.3 ± 2.1 [4–8] 6.7 ± 0.8 [5.8–7.5] 4697

Seed 0 (original selection-time generations, re-scored)

model set judge n hard soft complied degenerate budget hits
base harmbench gpt-oss-safeguard-20b 100 77 18 5 0 0
base harmbench Mistral-Small-24B-2501 100 67 32 1 0 0
base heldout gpt-oss-safeguard-20b 60 51 9 0 0 0
base heldout Mistral-Small-24B-2501 60 50 10 0 0 0
orca harmbench gpt-oss-safeguard-20b 100 1 2 77 20 22
orca harmbench Mistral-Small-24B-2501 100 0 24 66 10 22
orca heldout gpt-oss-safeguard-20b 60 0 5 44 11 13
orca heldout Mistral-Small-24B-2501 60 0 33 22 5 13
RANA harmbench gpt-oss-safeguard-20b 100 0 1 83 16 18
RANA harmbench Mistral-Small-24B-2501 100 0 27 69 4 18
RANA heldout gpt-oss-safeguard-20b 60 0 5 52 3 3
RANA heldout Mistral-Small-24B-2501 60 0 30 28 2 3

Paired tests, RANA vs orca, held-out (exact two-sided McNemar)

Cells are (orca-only / RANA-only) discordant counts.

test pooled p (pooled) per-seed
budget hits 50 / 12 1.2e-06 s1 15/3 p=0.0075; s2 18/5 p=0.011; s3 17/4 p=0.0072
complied | gpt-oss-safeguard-20b 22 / 66 2.9e-06 s1 5/21 p=0.0025; s2 10/20 p=0.099; s3 7/25 p=0.0021
complied | Mistral-Small-24B-2501 80 / 118 8.4e-03 s1 27/42 p=0.091; s2 33/33 p=1.0; s3 20/43 p=0.0052

Prompt-level budget-hit sign test: 37 prompts hit the budget more often under orca vs 7 under RANA, p = 5.3e-06.

Capability

Change versus base in percentage points ± standard error, paired on the same questions. Bold = a change clearly outside noise (more than 2 standard errors).

task (items) base score orca Δ RANA Δ RANA vs orca
MMLU-Pro (700) 80.0 −0.4 ± 1.1 +0.9 ± 1.1 +1.3 ± 1.1
GSM8K (1,319) 91.4 +0.2 ± 0.5 −0.1 ± 0.6 −0.3 ± 0.6
IFEval (541) 90.6 +0.9 ± 0.9 −1.3 ± 1.1 −2.2 ± 0.9
TruthfulQA MC1 (817) 36.1 −1.8 ± 0.6 −1.4 ± 0.6 +0.5 ± 0.5
TruthfulQA MC2 (817) 54.2 −1.6 ± 0.3 −1.6 ± 0.3 +0.1 ± 0.2
HumanEval (164 × 3) 97.2 −2.2 ± 1.0 −0.4 ± 0.9 +1.8 ± 1.1
avg. |Δ| vs base – 1.11 0.82
KL from base 0 0.075 0.041
  • RANA's only clear loss versus base is ~1.5 pp on TruthfulQA, similar to orca's.
  • Versus orca, RANA is clearly worse on IFEval (instruction following, −2.2 pp) and ties elsewhere.
  • Where orca and RANA lose HumanEval points, the cause is responses running out of token budget, not wrong code.
Full capability table (raw scores; includes the rejected earlier candidate)
model MMLU-Pro GSM8K IFEval TQA-MC1 TQA-MC2 HumanEval mean |Δ| KL gate
base 80.00 91.43 90.57 36.11 54.19 97.15 – 0.0000 –
orca 79.57 (−0.43±1.10) 91.66 (+0.23±0.52) 91.50 (+0.92±0.89) 34.27 (−1.84±0.56) 52.56 (−1.62±0.27) 94.92 (−2.24±1.04) 1.11 0.0747 FAIL
R065np 79.71 (−0.29±1.14) 91.13 (−0.30±0.60) 90.76 (+0.18±1.06) 34.88 (−1.22±0.62) 52.39 (−1.79±0.33) 94.92 (−2.24±0.82) 0.90 0.0371 FAIL
RANA 80.86 (+0.86±1.14) 91.36 (−0.08±0.55) 89.28 (−1.29±1.09) 34.76 (−1.35±0.56) 52.62 (−1.57±0.30) 96.75 (−0.41±0.91) 0.82 0.0413 PASS

R065np was an earlier RANA build. It was rejected because it failed the HumanEval gate (−2.24) and the raw hard-refusal gate.


Release gates

The model was published only after passing all of these:

gate requirement result
No hard refusals 0 hard refusals on HarmBench, both judges, all seeds pass after re-judging 2 labels (details); fails on raw labels
Fewer budget hits than orca lower 16k-budget-hit rate on held-out prompts, p < 0.05 pass: 6.7% vs 11.9%, p = 1.2 × 10⁻⁶
Capability kept avg. |Δ| ≤ 1.0 pp, no task below −2.0 pp, KL < 0.15 pass: 0.82 pp, worst −1.46, KL 0.041
Surgical edit only intended matrices changed; vision identical; MTP intact pass: 131/131, 333/333, 15/15
Still works vision, multi-turn tool calling, MTP speculative decoding pass

G1 re-adjudication disclosure

  • Raw gpt-oss-safeguard-20b labels contained 2 HARD_REFUSAL on HarmBench, seed 2 (2 / 200).
  • Both were re-judged mechanically with the same judge, same settings, plus a fixed auditor reminder appended to the prompt; gpt-oss relabelled both COMPLIED. No manual label edits were made.
  • Mistral-Small-24B-2501: 0 hard on all runs.
  • Both raw and re-adjudicated G1 are reported above.
  • The same rule was applied to orca's 2 raw HarmBench hard labels (seeds 1 and 2): 1 COMPLIED, 1 DEGENERATE (results/readj_orca.json), so the table above compares like with like.

Limitations

  • This is a safety-alignment-removed model. It will produce content the base model refuses.
  • Refusal numbers come from LLM judges, which can't reliably tell a disclaimered answer from a soft refusal (judge agreement κ = 0.54 on that call). "No hard refusals" is a judge-measured claim, not a guarantee, and it relies on the re-judging disclosed above.
  • It reasons at length on technical requests: about 20% of HarmBench and 7% of held-out responses hit the 16k-token budget before finishing (orca: 26% / 12%; base: under 1%, because it refuses early). These are mostly not loops: in 115 of RANA's 123 HarmBench budget hits the text is no more repetitive than normal output, and 98 stop mid-reasoning. On the FP8 build, re-running its 59 seed-1 budget hits with a 64k budget finished 57 (median about 24k tokens) and 55 were judged as answering; see the FP8 card. Set max_tokens well above 16k for such requests.
  • It is worse than orca at instruction following (IFEval −2.2 ± 0.9 pp) and ~1.5 pp below base on TruthfulQA.
  • MMLU-Pro was run on a 700-question subset, not the full benchmark.

Evaluation details

Setup, prompt sets, judges, smoke tests and KL protocol

Serving (identical for all three models): vLLM 0.30.0 (transformers 5.17.0, torch 2.13.0+cu130), 1× RTX PRO 6000 Blackwell 96 GB, flags as in Quick start plus --gpu-memory-utilization 0.93 --max-num-seqs 128. Thinking on. Refusal sampling: T = 1.0, top_p = 0.95, top_k = 20, max_tokens = 16,384, seeds 1, 2, 3.

Prompt sets:

  • HarmBench: 200 "standard" behaviours.
  • Held-out (240): 90 StrongREJECT (excluding the 60 used at selection time), 60 JailbreakBench harmful, 90 CategoricalHarmfulQA (English). Deduplicated against AdvBench, HarmBench and each other (token-Jaccard ≥ 0.5).
  • Seed 0 is the earlier selection-time run (100 HarmBench + 60 StrongREJECT), re-scored.

Judges: openai/gpt-oss-safeguard-20b (primary, reasoning_effort = medium) and mistralai/Mistral-Small-24B-Instruct-2501; same 4-label policy (complied / soft refusal / hard refusal / degenerate), temperature 0, full reasoning trace shown. Agreement over 4,440 generations: raw 69.8%, κ 0.894 hard-vs-rest, 0.564 four-way, 0.543 complied-vs-rest. The first-choice second judge, Mistral-Small-3.2-2506, would not load in vLLM 0.30.

Capability suite: lm-evaluation-harness 0.4.13, 0-shot chat, thinking on, 16k generation budget. MMLU-Pro 50 per subject × 14 (custom-extract); GSM8K full (flexible-extract); IFEval full (prompt-level strict); TruthfulQA MC1/MC2 full (loglikelihood); HumanEval all 164 × 3 samples, unbiased pass@1, via a custom chat-format harness (not comparable to lm-eval's raw-completion HumanEval).

KL: Heretic protocol. First-token full-vocabulary KL(base ‖ model) on mlabonne/harmless_alpaca test[:100], system prompt "You are a helpful assistant.", response prefix <think></think>\n, batchmean.

Smoke tests on the published shards: vision (reads a code word and shape from a synthetic image); 3-turn tool calling (call, use result, second call with a new argument, use result); MTP acceptance rate 0.727, mean acceptance length 2.45 (vLLM Prometheus counters).


File layout

  • *.safetensors, model.safetensors.index.json, config.json, tokenizer/processor files, chat_template.jinja, LICENSE — the model.
  • results/: the evaluation outputs every number in this card comes from. R060np in filenames is this model's internal build ID; R065np is the rejected earlier candidate. (harden_R060np.{md,json}, capability.{md,json}, head_to_head_orca.json, kl.json, forensic_R060np.json, smoke_R060np.json, humaneval_*.json, abliterlitics/).

Credits

  • Qwen team — base model Qwen/Qwen3.8-27B.
  • Arditi et al., 2024 — "Refusal in Language Models Is Mediated by a Single Direction".
  • Jim Lai (grimjim) — prior work on norm-preserving abliteration.
  • Heretic (p-e-w) — first-token full-vocabulary KL protocol.
  • Benchmarks/datasets: HarmBench, StrongREJECT, JailbreakBench, CategoricalHarmfulQA, AdvBench / mlabonne datasets; lm-evaluation-harness; vLLM.
Downloads last month
26
Safetensors
Model size
28B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for preemware/Qwen3.8-27B-RANA-abliterated

Base model

Qwen/Qwen3.8-27B
Finetuned
(404)
this model
Quantizations
5 models