joshuapenman/semantic-overlays-adapters

Semantic Overlays — trained adapter checkpoints

The trained overlay sets from Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors (arXiv:2608.23873) — every set the paper's web demo serves.

paper	arXiv:2608.23873
code	semantic-overlays
training corpus	semantic-overlays-injection
interactive demo	semantic-overlays.vercel.app

Each is a stack of small per-layer SwiGLU adapters applied to a frozen base model's residual stream at marked prefill positions; the base model is never modified and is not distributed here. The semantic-overlays GitHub repository contains the serving code (a vLLM plugin) and the demo; with these checkpoints, the demo and all paper evaluations run without any training.

Contents

checkpoints/
  injv2b-ss-6x-per4/step02572.pt     do-not-execute overlay, Qwen3.5-9B (the paper's headline defense)
  llama-inj-6x-per4/latest.pt        do-not-execute overlay, Llama-3.1-8B-Instruct (replication)
  qover-phrase128/slim.pt,phrases.pt twelve visual marks (3 mark types x 4 colors), embedding-conditioned
  rosetta-v3d-phrase128/slim.pt,phrases.pt   four asserted programming languages (Python/JavaScript/Ruby/C)
  behav-phrase256-ew/ew450_slim.pt,ew450_phrases.pt   twelve carried instructions (formats, languages, behaviors)

phrases.pt files hold the frozen conditioning embeddings for the embedding-conditioned sets; they load beside the weights.

Use

Upload into the serving volume expected by the companion repo, then deploy:

modal volume put goggles-data checkpoints /checkpoints
modal deploy infra/goggled_vllm.py

The endpoint applies an overlay only where the request's span mask says so; requests with no marked span reproduce the frozen base model exactly. The adapter registry in infra/goggled_vllm.py maps each directory above to its serving name (inject, qover, plr, behav).

Bases and licensing

The adapter weights are released under MIT. They are trained against, and only function attached to, their frozen base models: Qwen3.5-9B (Apache 2.0) and Llama-3.1-8B-Instruct (Llama 3.1 Community License) — obtain those from their own distributions under their own terms. The do-not-execute overlays were trained on the corpus released as semantic-overlays-injection.