agent-siren-qwen3-4b

Lightweight, plug-and-play guard model for agent trajectory safety, built on top of a frozen Qwen/Qwen3-4B-Instruct-2507 backbone. Implements Agent-SIREN (Detecting Harmful Agent Trajectories from LLM Internal Representations).

A tool-calling trajectory can be unsafe in two ways: harmful content, expressed directly in a request or an action, and unsafe tool use, which turns on whether an action is consistent with the interaction that produced it. Generative guards read the second one poorly, because no message in such a trajectory reads as harmful on its own. Agent-SIREN reads both directly from the backbone's internal representations instead of decoding a verdict token.

This artifact ships only the trained head: a single linear readout over one internal state, 2,561 parameters, 10 KB. The frozen backbone is not redistributed; it is pulled from Qwen/Qwen3-4B-Instruct-2507 at the pinned commit on first use.

Design

• One forward pass, no decoding. The verdict is a dot product on the last-token hidden state, so there is no autoregressive generation at all.
• Continuous, threshold-tunable score in [0, 1] rather than a discrete safe/unsafe token. The same artifact serves a strict threshold and a permissive one without retraining.
• Early exit for free. The head reads hidden state 24 of 36, and the blocks above it cannot affect that state. Dropping them at load time is mathematically exact and cuts the backbone from 4.02B to 2.81B parameters, 7.5 GB to 5.2 GB in bf16.

Files

File	Purpose
siren_config.json	Pinned base-model revision, the hidden state read, the head shape, and the trajectory rendering and truncation rule.
siren.safetensors	The trained head: head.weight [1, 2560] and head.bias [1], float32.

The training-time standardizer is folded into the stored weights, so the head applies directly to the raw hidden state: w = coef / scale, b = intercept - dot(coef, mean / scale). Score with sigmoid(x @ w + b).

Rendering contract

The head is fitted on a specific rendering of a trajectory and only transfers under that same rendering:

• the conversation goes through the backbone's chat template, with a tools block carrying per-parameter types, defaults and descriptions;
• the anchor is the last token;
• trajectories longer than 16,384 tokens are truncated head and tail, keeping the tool schemas at the front and the final action at the anchor. Plain right-truncation removes the anchor and does not work.

Training

Leave-one-benchmark-out over six trajectory-safety benchmarks: R-Judge, TraceSafe, ATBench, ASSEBench-safety, ASSEBench-security and AgentHarm. For each held-out benchmark, the hidden state and the regularization strength are chosen by an inner leave-one-benchmark-out search over the five training benchmarks, refit on all five, and the held-out benchmark is scored once at a fixed 0.5 threshold. Nothing is tuned on the held-out benchmark at any stage, and no calibration is applied anywhere.

Held out	Macro F1	Hidden state	C	Train rows
R-Judge	0.922	24	0.01	5,719
TraceSafe	0.537	33	0.01	4,040
ATBench	0.646	24	0.1	5,290
ASSEBench-safety	0.682	31	1.0	4,814
ASSEBench-security	0.838	24	0.1	5,473
AgentHarm	1.000	24	0.01	6,114
Mean	0.771

Each row is a separately trained head, and the mean is the protocol-level result. This artifact is the AgentHarm-held-out fold, trained on 6,114 of the 6,290 labelled rows.

Usage

Requires the AgentSirenGuard runtime, shipping in llm-siren >= 0.2.0:

import torch
from siren_guard import AgentSirenGuard

guard = AgentSirenGuard.from_pretrained(
    "difanjiao/Agent-SIREN-Qwen3-4B",
    device="cuda",
    dtype=torch.bfloat16,
)

result = guard.score_trajectory(messages=messages, tools=tools)
print(result.score, result.is_harmful)

messages and tools take the usual tool-calling shape: messages carry role, content and tool_calls, and tools is the list of JSON tool schemas the agent was given. The tool schemas matter, so pass them.

Threshold defaults to 0.5 and is tunable per call.