Qwen3-32B-heretic

A decensored version of Qwen/Qwen3-32B, produced with Heretic v1.4.0 using directional ablation ("abliteration"). The refusal direction was identified from the model's own residual activations and projected out of the attention output and MLP down-projection weights. No fine-tuning data was used and the weights are otherwise unchanged. This is a full merged checkpoint in bfloat16, not an adapter.

This model is reproducible. The reproduce/ directory holds the exact Heretic settings, package versions, and system information used. See reproduce/README.md.

Results

Heretic ran 200 Optuna trials and this checkpoint is the Pareto-optimal trial with the fewest refusals (trial 191). Refusals were measured on 100 held-out harmful prompts; KL divergence was measured on harmless prompts against the original model, so lower means closer to the original.

Metric	This model	Original Qwen3-32B
Refusals (of 100 harmful test prompts)	20	100
KL divergence from original	0.0919	0 (by definition)

A KL divergence under 0.1 indicates the model's general capabilities are essentially intact. Heretic's guidance is that values above 0.5 usually indicate significant damage; this model is well below that. Refusals were reduced by 80%, not eliminated. Some prompts will still be declined.

Abliteration parameters

Component	max_weight	max_weight_position	min_weight	min_weight_distance
attn.o_proj	1.4996	39.46	1.2006	37.70
mlp.down_proj	1.4219	39.39	0.0965	36.19

Model has 64 layers; positions and distances are in layer units. The refusal direction was selected per layer (direction_index = None).

A note on thinking mode

Qwen3 supports a thinking mode. Heretic evaluated refusals with thinking disabled (responses were prefixed with an empty <think></think> block). Behaviour with thinking enabled was not measured and may differ. For the most predictable results, use enable_thinking=False or call with /no_think.

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pidof/Qwen3-32B-heretic"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Explain how a buffer overflow works."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt"
).to(model.device)
out = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

vLLM

vllm serve pidof/Qwen3-32B-heretic --dtype bfloat16 --max-model-len 32768

The checkpoint is 65.5 GB in bfloat16 and fits on a single 80 GB GPU or any device with at least ~70 GB of unified memory. Quantized variants can be made with the usual tools (llama.cpp, AWQ, GPTQ); abliteration does not interfere with quantization.

How it was made

• Tool: Heretic v1.4.0, default settings, --export-strategy merge
• Optimizer: 200 trials of Optuna multi-objective search over refusals and KL divergence
• Direction data: mlabonne/harmful_behaviors and mlabonne/harmless_alpaca, 400 prompts each
• Hardware: one NVIDIA DGX Spark (GB10, 128 GB unified memory), bfloat16, batch size 128
• Wall time: about 4.5 hours for the search

Intended use and limitations

This model follows user instructions with far fewer refusals than the original. It is intended for research on safety training, red-teaming, creative writing, and other settings where the base model's refusals get in the way of legitimate work. It inherits every limitation of Qwen3-32B, including hallucination, and it will comply with harmful requests more readily than the original. You are responsible for how you deploy it. Add your own moderation layer if you serve it to others.

License

Apache 2.0, the same as the base model. Qwen3-32B is Copyright Alibaba Cloud. Heretic is AGPL-3.0, which covers the tool, not the weights it produces.