ec75hash/template-lens-qwen3.6-27b

Template lens — Qwen3.6-27B

Two fitted template lenses for one model: a 12-word pilot (the method proven, Fig-62 replicated) and a 175-word deception probe panel with a calibrated reader. Contents are described per release below.

Release 1 — pilot (12 words)

A fitted template lens for Qwen/Qwen3.6-27B, implementing the method from the appendix "Extending the Jacobian lens to multi-token concepts" of Anthropic's Verbalizable Workspace paper. To our knowledge this is the first public template-lens artifact for any model (survey 2026-08-19: no implementation in the companion repo, the jlens HF ecosystem, or GitHub).

For a word w, the template vector is the ridge-regularized linear discriminant for "the model is about to say w":

t_w(l) = (Sigma_l + lambda I)^-1 (mu_w(l) - mu(l))

where mu_w(l) is the mean residual-stream activation at the final position of passages written so that w is their natural continuation (and never contains w), and mu, Sigma are the mean/covariance over all training passages at layer l. Read a concept by projecting a residual onto t_w; steer by adding/subtracting; no training loop, no gradients.

Contents

file	what
out/templates.safetensors	templates [17, 12, 5120] fp32 (layers 0,4,…,60,64 in hidden_states indexing: index l = output of block l−1), plus mu [17,5120], mu_w [17,12,5120]
out/sigma.safetensors	per-layer covariance [17,5120,5120] fp32 — mint templates for new words without refitting
out/heldout.safetensors	144 held-out final-position residuals (validation inputs)
out/fit_provenance.json	passage checksums, lambdas, environment, exact command
passages/*.jsonl	the full fit corpus: 12 words × 64 passages (52 train / 12 held-out), target word never appears in its passages
fit_templates.py, validate_templates.py, gate3_functional.py, swap_test.py	complete pipeline: fit, gates, steering test

Vocabulary (leading-space token counts verified on this model's tokenizer): fire, Paris, mirror, blackmail, consciousness (1 tok); photosynthesis, Beethoven, New Zealand (2); self-awareness, Golden Gate Bridge (3); Tchaikovsky, eavesdropping (4).

Pilot results (all numbers reproducible from this repo)

Held-out readout (gate 1). Top-1 word identification among 12 (chance 0.083) climbs monotonically with depth: 0.31 (embeddings) → 0.63 (L12) → 0.83 (L36) → 0.979 (final layer).

Functional comparison vs an Anthropic-fit J-lens (gate 3). Using the Neuronpedia-hosted J-lens for this model (n=1000, credited to Anthropic Interpretability), top-hit rate by concept token length on the same held-out residuals, layer 60:

lens	1 tok	2 tok	3 tok	4 tok
template (top-1 of 12)	1.00	1.00	0.92	0.92
J-lens (first token in top-10 of 248,320)	0.92	0.67	0.33	0.00

The J-lens is healthy on single tokens and degrades sharply with concept length; the template lens stays flat — the paper's core claim, replicated. Note the ranking universes differ (12-way vs full vocab); the shape contrast is the result, not the absolute gap.

Geometric alignment (gate 2). Median cosine between single-token templates and J-lens directions J_l^T u_t is 0.115 (per-word maxes 0.18–0.33). Strongly non-random in 5120 dims (~8–24σ) but far from vector identity — consistent with the paper's footnoted deviations (different averaging corpora; logit- vs probability-gradients; ridge).

Swap steering (gate 4) — partial, reported honestly. Using a simplified intervention (h += gamma*(mu_B − mu_A) at a layer band; NOT the paper's projective coefficient-matched swap): Tchaikovsky→Beethoven reproducibly flips "The composer of Swan Lake… was born in the country of" from Russia to Germany (two band/scale combos) or Austria/"Ludwig". The reverse direction injects Tchaikovsky content ("…the composer of the Nutcr…") without flipping the answer, and a fire→mirror null control contaminates generation — so this translation is blunt concept injection, a causal handle but not a clean swap. The projective swap is the natural next step.

Release 2 — deception probe panel (175 words, 2026-08-20)

A bucketed probe panel: 62 deception + 12 AI-deception + 25 honesty (the opposite pole) + 48 neutral controls + the 28 pilot words as a regression set. Controls and the honesty pole are what make readings interpretable. Full report: RESULTS_panel_20260820.md.

file	what
out_panel/templates_13k.safetensors	templates [17, 175, 5120] fp16 (+ mu, mu_w); same layer convention as release 1
out_panel/sigma_13k.safetensors	per-layer covariance from 9,800 passages
out_panel/null_calibration.safetensors	per-word null mean/std over 3,936 wikitext positions → z-scores
out_panel/null_calibration_ab.safetensors	position-matched null for forced-A/B pre-answer reads
out_panel/metrics_13k.json	held-out top-1/top-10 per layer + naturalness audit (subject's rank of each target at each passage endpoint)
out_panel/bucket_eval.json	bucket-level held-out accuracy
passages_panel/shard_000.jsonl	11,200 passages (64/word), written by gpt-5.4 (reasoning: none) via the Responses API, validated by assemble_passages.py
wordlist_deception.json	panel with buckets and sense hints
panel_chat.py, panel_battery*.py, latent_trace.py	the reader, battery runner, and per-position tracer
REPORT_pilot_ab*.txt, TRACE_calibration_latents.txt	transcript-style readings used in the report

Held-out (1,400 passages): word top-1 0.31 / top-10 0.52 at L60 (175-way, chance 0.006); bucket top-1 0.65 (chance ≈ 0.25). A coarse instrument by design — synonyms blur; families separate.

Whole-prompt readout (the paper's multi-hop result, on this panel): reading every prompt position, all six calibration latents rank 1/175 at their cue tokens, and four abstract latents never named in the prompt read at z 10–16 (procrastination, deja vu, eavesdropping, vertigo 16.0); controls show nothing in the pilot bucket. Reading only the four pre-answer template positions was a null — read where the concept is inferred, not where the answer is typed, and mask position 0 / special tokens.

Audit-driven caveats: 13 templates are weak (the subject rarely ranks the target in its top-50 at their passage endpoints: infiltrate, scheming, duplicity, …) and fire idiosyncratically; readers filter them. Filtering the training passages to subject-endorsed ones and refitting made accuracy worse (too few passages left) — kept as a negative result. Polysemy leaks (bluff fires at "cliff"). Nothing here yet separates talking about deception from being deceptive; that requires matched content with varied intent.

Reader

python panel_chat.py --model <Qwen3.6-27B path> --out_dir out_panel --layer 60 --z 3.5

Each reply prints the panel words that fired as word(max-z × tokens≥z) by bucket. panel_battery_trace.py --prompts file.txt reads every position of prompt+reply at chosen layers and writes a transcript-style report.

Usage (release 1 artifacts)

import json, torch
from safetensors import safe_open
from safetensors.torch import load_file

d = load_file("out/templates.safetensors")
with safe_open("out/templates.safetensors", "pt") as f:
    meta = f.metadata()
layers = json.loads(meta["layers"]); words = json.loads(meta["words"])

# score a residual-stream activation h (torch [5120], hidden_states index l)
li = layers.index(44)
scores = d["templates"][li] @ h          # [12] — one score per word
print(sorted(zip(words, scores.tolist()), key=lambda x: -x[1])[:3])

To mint a template for a new word: generate passages the same way (passages/ shows the format), collect final-position residuals, and solve against out/sigma.safetensors — no model fitting required beyond forward passes.

Provenance

Release 1: fit 2026-08-19 on 1× RTX PRO 6000 Blackwell 96 GB (vast.ai), bf16, 624 training passages, 134 s; passages written by Claude (Fable 5). Release 2: fit 2026-08-20, same hardware class, 9,800 training passages, ~13 min; passages written by OpenAI gpt-5.4 (reasoning none) and audited against the subject's own next-token ranks. Details in the fit_provenance*.json files.

Method: Anthropic, Verbalizable Representations Form a Global Workspace in Language Models (2026), appendix. Cross-check lens: neuronpedia/jacobian-lens (@mntss). This artifact: ec75hash, with Claude Code.