xiangchensong/jacobian-lens-deepseek-v4-flash-0731

Jacobian lens for DeepSeek-V4-Flash-0731

Fitted Jacobian-lens matrices for deepseek-ai/DeepSeek-V4-Flash-0731 (43 layers, hidden size 4096, four-stream hyper-connection residual, fp8 MoE). The Jacobian lens (Anthropic, Verbalizable Representations Form a Global Workspace in Language Models) reads out what an internal residual is disposed to make the model say:

lens_l(h) = unembed( J_l @ h ),   J_l = E[ ∂h_target / ∂h_l ]

There is no training and there are no learned parameters: each J_l is a closed-form running mean of per-prompt Jacobians.

A companion lens for the preview checkpoint this release superseded lives at jacobian-lens-deepseek-v4-flash-preview — fitted with an identical recipe, so the pair supports recipe-controlled comparisons of what the re-post-training changed.

What the matrices are

V4-Flash carries hc_mult = 4 parallel residual streams per position (hyper-connections), so a square J_l ∈ R^{d×d} is not well-defined. These lenses are the rectangular generalization:

• Shape: [4096, 16384] per layer, fp16 on disk.
• Source (d_source = 16384): a block's output with the four streams flattened, hc_mult * hidden_size.
• Target (d_model = 4096): the output of model.hc_head, the module that collapses the four streams back to one vector per position just before the final norm and unembedding. The collapse operator is absorbed into J_l, which keeps the target in the exact basis the unembedding expects. The square single-stream case is a bit-identical special case of this construction.

Files

file	prompts	layers	shape	note
lens.pt	n=1000	L19–39 (21 layers)	[4096, 16384] fp16	default; paper-scale corpus
lens_n100.pt	n=100	L19–39 (21 layers)	[4096, 16384] fp16	the lens most replication numbers were produced with
lens_disjoint100.pt	n=100, corpus records 900–1000	L19–39 (21 layers)	[4096, 16384] fp16	noise floor: same checkpoint, disjoint corpus — the control every floor-referenced comparison uses

L19–39 is the model's measured global-workspace band (44–91% relative depth; the paper reports 38–92% on Sonnet 4.5), identified by four independent onset/offset statistics plus CKA layer geometry. lens_n100.pt was produced by restricting a full 43-layer fit to the band; because per-prompt normalization is applied per layer, this restriction is bit-identical to a band-restricted fit (verified in float64). The n=1000 fit moved no experimental conclusion relative to n=100 (every delta within the corpus-sampling noise floor measured on a disjoint-prompt lens), consistent with the estimator being converged from roughly n=60.

Fitting recipe

• Corpus: WikiText-103 (Salesforce/wikitext, wikitext-103-raw-v1, train split, streamed), records ≥ 600 characters, truncated to 128 tokens. The n=1000 prompt set is a superset of the n=100 set (first 100 records); lens_disjoint100.pt uses records 900–1000, disjoint from the n=100 set, which is what makes it a valid sampling-noise floor.
• Positions: the first 16 positions are skipped (skip_first=16, attention-sink positions) and the final position is dropped; the per-prompt Jacobian is the mean over the remaining ~111 source positions, with cotangents set at every valid target position at or after the source.
• Per-prompt normalization (normalize_per_prompt=True): each prompt's J_l is scaled to unit Frobenius norm before averaging. Per-prompt Jacobian gain is heavy-tailed (up to ~30× between prompts) and the model's final RMSNorm makes a uniform rescaling of J_l invisible in the logits, so a plain mean weights prompts by an unobservable quantity. This improved pass@1 by 73% and is what all released numbers use. A consequence: ||J_l||_F ≤ 1 by construction (measured 0.51–0.89 across the band) — the absolute scale of J_l is not meaningful, only its direction.
• Storage: fp16 (entries are O(1); fp16's extra mantissa bits beat bf16 here). Loaded back as fp32.

Usage

Requires the mHC-capable (rectangular-J) version of the jlens package — the upstream square-only release cannot represent d_source != d_model.

from transformers import AutoModelForCausalLM, AutoTokenizer
from jlens.hf import from_hf
from jlens.lens import JacobianLens

lens = JacobianLens.from_pretrained("xiangchensong/jacobian-lens-deepseek-v4-flash-0731")
# or: JacobianLens.from_pretrained(..., filename="lens_n100.pt")

name = "deepseek-ai/DeepSeek-V4-Flash-0731"
tok = AutoTokenizer.from_pretrained(name)
hf_model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")
model = from_hf(hf_model, tok)   # auto-detects the hyper-connection layout

lens_logits, model_logits, input_ids = lens.apply(
    model, "The capital of France is", layers=[30], positions=[-1]
)
print(lens_logits[30].topk(5).indices)   # what L30 is disposed to say

Readout is forward-only; no backward pass or gradient support is needed to use the lens.

License and provenance

The matrices in this repository are derived from the base model: each J_l is an expectation of gradients of deepseek-ai/DeepSeek-V4-Flash-0731's forward pass, i.e. a function of its weights. The base model is released under the MIT License (Copyright (c) DeepSeek), and these matrices are released under the same terms. No base-model weights are contained in or recoverable from these matrices at fidelity; they are averaged input–output sensitivities.

Citation

• Method: Anthropic, Verbalizable Representations Form a Global Workspace in Language Models, Transformer Circuits, 2026. https://transformer-circuits.pub/2026/workspace/index.html
• This replication / the rectangular-J generalization: The Jacobian Lens at Frontier Scale: Testing the Global Workspace on DeepSeek-V4-Flash and GLM-5.2. https://xiangchensong.github.io/blog/2026/jacobian-lens-global-workspace/

Code: https://github.com/xiangchensong/jacobian-lens-open-frontier