qweclownq/tangent-flow-16m

Constraint-Preserving Tangent Flow (16.5M) for GPT-2 small activations

A small conditional flow-matching prior over GPT-2 small residual-stream activations, trained on constraint-preserving tangent corruption: corruption and denoising trajectories that live entirely inside the subspace tangent to a semantic constraint ⟨h, v⟩ = c.

What this model is, and what it is not

It solves its own task. On matched tangent reconstruction (T1) it recovers 77.3 % of the language-model damage caused by tangent corruption, with a paired ΔNLL of −1.012611 (95 % CI [−1.072443, −0.949261]).

It does not improve steering. On the downstream naturalization task (T2) — repairing a hard-clamped activation while the semantic coordinate is held exactly fixed — it makes quality worse than the plain hard clamp it was meant to improve on:

pooled ΔNLL = +0.006184   (95 % CI [+0.001631, +0.010788])
0 of 30 diagnostic grid cells favourable

This is not a "best steering model". It is the artifact of a clean negative result, published because the negative result is the contribution. Do not use it expecting improved activation steering.

A post-hoc run on the project's frozen DEV steering protocol reaches the same conclusion on the original task, with a text-level concept metric: paired ΔNLL against additive steering is +0.0598 (95 % CI [+0.0057, +0.1182]), improving on only 2 of 8 directions. Note the control at α = 0, where the clamp is the identity and the flow merely naturalizes a clean activation: it lowers NLL by 0.070 nats. That is a baseline shift, not a steering gain, and subtracting it puts the method +0.130 nats behind additive.

The coordinate is preserved exactly (mean parallel correction 5.6e−07 against a 7.09 orthogonal correction), so the failure cannot be attributed to attenuation. The damage scales monotonically with orthogonal correction magnitude: ‖Δh⊥‖ 7.09 → +0.006, 16.81 → +0.054, 29.58 → +0.346 nats.

The objective

For a clean standardized activation x₀, unit direction v, coordinate c = ⟨x₀, v⟩:

x∥ = c·v                      ε⊥ = ε − ⟨ε, v⟩·v
x⊥ = x₀ − c·v

x_t = c·v + (1−t)·x⊥ + t·ε⊥   ⟹  ⟨x_t, v⟩ = c  for every t
u*  = ε⊥ − x⊥                 ⟹  ⟨u*, v⟩ = 0

The velocity used for the loss, for Euler integration and for reconstruction is analytically projected, u − ⟨u, v⟩·v, so the constraint is exact by construction rather than learned.

Specification

Parameters	16,542,464
Base model	GPT-2 small, blocks.7.hook_resid_pre, d = 768
Architecture	tokenwise time-conditioned SwiGLU, 3 blocks, d_model 768, direction/coordinate FiLM conditioning
Training data	FineWeb sample-10BT, 32M residual activations, BOS dropped
Steps	250,000, batch 1024, AdamW, lr 3e-4, cosine
Checkpoint selection	val_flow_mse = 0.9680510, minimized, concept-independent
SHA-256	066afb601418da89f79b003c97b37227a9aa7702a442ad5f3fb0ab68a4199d4c
Config fingerprint	e4af61135b0205cdcd6f196a61d5af464f0369b29e9bcaa471fd0764e7f85499
Direction pool	45241c49814abe71ed7106e1a0fcbbe7d8aad40b215621674ec72ee7356d7a2c (training-only, rank floor 256)

Trained only on a training-only SAE direction pool. No DEV or held-out evaluation direction entered training or checkpoint selection.

Files

File	Contents
tangent_flow_16m.pt	the checkpoint (model, normalizer, architecture, provenance metadata)
architecture.yaml	frozen architecture specification
training_config.yaml	frozen training configuration
selection.json	checkpoint-selection record
t1_receipt.json	formal T1 receipt (PASS)
t2_receipt.json	formal T2 receipt (FAIL)

Loading

from interp.train_flow import load_flow_checkpoint
from interp.tangent_flow import TANGENT_OBJECTIVE, clamp_then_tangent_flow

model, meta, _ = load_flow_checkpoint(
    "tangent_flow_16m.pt", device="cuda", expected_objective=TANGENT_OBJECTIVE
)

out = clamp_then_tangent_flow(
    model, h, direction=v, c_target=c,
    noise=noise, t_start=0.10, nfe=1,
)

load_flow_checkpoint refuses a checkpoint trained on a different corruption geometry, so the objective cannot be silently mixed up.

Intended use

Reproducing or extending the negative result; studying activation priors and the gap between reconstruction competence and intervention repair. Not intended for production steering.

Limitations

• GPT-2 small only, single intervention layer.
• The capacity question is open: a matched 60M control was attempted and abandoned to repeated GPU-worker failures, so "would a larger tangent model behave differently" is untested. The mechanism argues against it — the penalty tracks correction magnitude, not correction quality — but that is an argument, not a measurement.
• No stable LLM fluency judge was available; the primary quality metric is conditional NLL under the clean model.

Citation of the surrounding work

Full report, all eleven figures, and the frozen T0/T1/T2 protocol live in the project repository. The branch closure document states the result and the predefined stop condition that produced it.