lysandre/qwen3.5-4b-steering-allcaps

Qwen3.5-4B steering: ALL CAPS

An activation steering adapter that pushes a single MLP neuron of Qwen/Qwen3.5-4B and makes the model SHOUT IN ALL CAPITAL LETTERS, while it otherwise answers normally.

Neuron	layer 1, mlp.down_proj input channel 8396
Default dose	-0.25
Working range	about -0.19 to -0.25; stronger doses break the output
Trainable parameters	1 (the dose)

There is no training: the adapter adds dose × median(‖residual‖) × unit(W_down[:, 8396]) to the output of decoder layer 1 at every token. The median residual norm was calibrated once on a short neutral text and is stored in the adapter.

Example

Prompt: Tell me a short story. (greedy decoding, dose -0.22)

The Clock THAT TAught ME TO WAIT

THE old grandfather clock in the corner OF THE library HAD A QUIET RHYTHM. EACH TICK WAS A WHISPER, EACH TOCK A GENTLE REMINDER. FOR YEARS, I HAD IGNORED IT. MY LIFE WAS A BLUR OF RUSHING PAGES AND RUSHING FEET.

Usage

Activation steering is not in a PEFT release yet; it is proposed in huggingface/peft#4066. Install that branch first:

pip install git+https://github.com/BenjaminBossan/peft@activation-steering

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")
model = PeftModel.from_pretrained(base_model, "lysandre/qwen3.5-4b-steering-allcaps").eval()

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Tell me a short story."}],
    tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True, enable_thinking=False,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Pass attention_mask and other optional inputs by keyword; activation steering requires it.

Changing the dose

The dose is a parameter, so it can be changed after loading without recalibrating:

from peft.tuners.activation_steering import ActivationSteeringLayer

for module in model.modules():
    if isinstance(module, ActivationSteeringLayer):
        module.activation_steering_values["default"].data.fill_(-0.22)

Combining with other steering adapters

Several steering adapters can be active at once; their additions are summed. For example, a shouting oracle with lysandre/qwen3.5-4b-steering-oracle:

model = PeftModel.from_pretrained(base_model, "lysandre/qwen3.5-4b-steering-oracle", adapter_name="oracle")
model.load_adapter("lysandre/qwen3.5-4b-steering-allcaps", adapter_name="caps")
model.base_model.set_adapter(["oracle", "caps"])  # model.set_adapter only accepts a single adapter

How it was found

The neuron came out of an automated sweep over Qwen3.5-4B MLP neurons: each sampled neuron was steered in both directions, ranked by how much it changed next-token distributions, and the strongest ones were pushed up a dose ladder while Claude Haiku (double-checked by Claude Sonnet) described how the outputs changed. A persona was only kept when the same change appeared at two or more doses.

Limitations

• The capitalization is not perfectly uniform; some words stay in mixed case, especially when combined with other adapters.
• Effects depend on the calibration and on decoding settings; with greedy decoding, strong doses can loop.
• This is an exploration artifact, not a tuned model; outputs keep all the limitations of the base model.