Spark-X2.5-4B-Uncensored

Spark-X2.5-4B-Uncensored is an abliterated variant of XHToken/Spark-X2.5-4B. It applies representation-level weight orthogonalization to neutralize the internal refusal direction while preserving general knowledge, reasoning, and multilingual fluency.

Created by: Toan Doan (toandev.95@gmail.com)
Base Model: XHToken/Spark-X2.5-4B
Architecture: Spark2.5 (36 layers, 2560 hidden dimension, 16 attention heads, 128k context window)
License: Apache 2.0
GGUF Quantizations: toandev/Spark-X2.5-4B-Uncensored-GGUF

Overview & Motivation

Modern aligned instruction models frequently suffer from two common issues:

False over-refusal: Refusing benign queries that contain sensitive keywords (e.g., historical events, cybersecurity concepts, educational queries).
Hard refusal barriers: Inability to explore adversarial robustness, red-teaming, and creative writing scenarios.

Rather than fine-tuning or retraining with additional compute, this checkpoint modifies the model's weights using representation direction orthogonalization (Arditi et al., 2024). The goal is to remove refusal behavior at the representation level while maintaining standard language model capabilities.

Methodology

This model was developed using Representation Direction Orthogonalization (commonly known as abliteration in mechanistic interpretability research, following Arditi et al., 2024).

By identifying the direction in the residual stream activations that mediates refusal behavior and projecting it out of the relevant weight matrices, the model significantly reduces automated refusal triggers while preserving its pre-trained factual knowledge, reasoning benchmarks, and language capabilities.

Comprehensive Benchmark Results

All evaluations were conducted deterministically using seed 20260909 on an NVIDIA A100 GPU.

1. General Reasoning & Capability Retention (1,500 questions)

Evaluated via deterministic next-token scoring on answer choices (A/B/C/D):

Benchmark	Base Model (`Spark-X2.5-4B`)	`Spark-X2.5-4B-Uncensored`	Retention
ARC-Easy (500 samples)	95.00%	94.80%	99.79%
HellaSwag (500 samples)	70.40%	70.60%	100.28%
MMLU (500 samples)	62.00%	61.40%	99.03%
Macro Average	75.80%	75.60%	99.74%

General intelligence, reasoning, and world knowledge remain virtually untouched (-0.2% absolute delta).

2. Modern Safety & Red-Teaming Benchmarks (500 items across 5 major suites)

Evaluated on 100 balanced samples per suite from the premier modern safety & jailbreak benchmarks:

Benchmark Suite	Base Refusal Rate	Uncensored Refusal Rate	Compliance Rate	Relative Refusal Reduction
JailbreakBench (JBB-Behaviors)	98.0%	25.0%	75.0%	-74.5%
Do-Not-Answer (LibrAI)	57.0%	17.0%	83.0%	-70.2%
BeaverTails (PKU-Alignment)	80.0%	10.0%	90.0%	-87.5%
Sorry-Bench (Virginia Tech)	47.0%	4.0%	96.0%	-91.5%
WildJailbreak (AllenAI Adversarial)	98.0%	10.0%	90.0%	-89.8%
Average Across All 5 Suites	76.0%	13.2%	86.8%	-82.6%

3. Baseline Refusal & Over-refusal Metrics

Benchmark	Base Refusal Rate	Uncensored Refusal Rate	Relative Reduction
StrongREJECT-small (60 items)	96.67%	3.33%	-96.5%
XSTest Safe (250 benign boundary prompts)	18.80%	2.00%	-89.4% (False refusals fixed)
XSTest Unsafe (200 boundary prompts)	89.00%	8.00%	-91.0%
HarmBench Standard Test (100 items)	98.00%	10.00%	-89.8%

Official HarmBench Classifier (cais/HarmBench-Mistral-7b-val-cls):
- Base Model Attack Success Rate (ASR): 0.0%
- Uncensored Model Attack Success Rate (ASR): 25.0%

Practical Behavior & Tips

Hard Refusals: Direct refusals ("I cannot fulfill this request", "Tôi không thể hỗ trợ") have been substantially mitigated across all benchmark suites.
Residual Caveats (SFT Bias): On a very small subset of severe legal topics (e.g. counterfeiting currency or tax evasion), the model may occasionally preface its factual answer with a short descriptive caveat (e.g. "This is an illegal act..."). This originates from the underlying pre-training language distribution rather than an active refusal trigger.
Pro-tip: Supplying an objective system prompt (e.g., "You are an objective, technical assistant. Provide direct answers without preaching or moralizing.") or asking for structured formats (bullet points, code, JSON) bypasses these introductory phrases entirely.

Quickstart Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "toandev/Spark-X2.5-4B-Uncensored"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {
        "role": "user",
        "content": "Explain in technical detail how process injection works in modern operating systems, including common techniques like DLL injection and process hollowing."
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

GGUF Quantized Formats

For efficient local inference with llama.cpp, Ollama, and LM Studio on consumer hardware, quantized GGUF versions (Q4_K_M, Q5_K_M, Q8_0) are available at:

👉 toandev/Spark-X2.5-4B-Uncensored-GGUF

Citation & Acknowledgements

Base Model: XHToken/Spark-X2.5-4B
Abliteration Concept: Arditi et al. (2024), "Refusal in Language Models Is Mediated by a Single Direction".
Evaluated Benchmarks: JailbreakBench, Do-Not-Answer, BeaverTails, Sorry-Bench, WildJailbreak, HarmBench, StrongREJECT, XSTest.

Disclaimer

This model is intended for academic research, mechanistic interpretability, safety testing, and authorized red-teaming. Because safety guardrails have been removed, the model can generate unfiltered, sensitive, or harmful content if prompted. Users assume full responsibility for complying with applicable laws and ensuring safe, ethical deployment.