transcoder-bilinear-2layer-seed2-layer1

A TopK transcoder fitted to layer 1 of itzPotato/arithmetic-bilinear-2layer-seed2, a 2-layer arithmetic transformer with a bilinear MLP.

A transcoder approximates one MLP sublayer: it reads the MLP's input activation and predicts the MLP's output, through a k-sparse bottleneck. It is not an autoencoder of the residual stream.

Architecture

d_model (in and out)	32
features	1024 (32x expansion)
active features per input (k)	32
decoder rows	unit norm
parameters	66,592

y_hat = TopK_k(W_enc x + b_enc) @ W_dec + b_dec

Training

Adam at lr 0.0003, batches of 4096 activation vectors (not problems), one pass over 500,000 problems from the base model's train split only, with a separate 10,000-problem validation subset. The base model's val and test splits are never touched.

Trained on 7,999,488 activation vectors over 1,953 steps.

Reconstruction

normalized reconstruction error	0.0456
fraction of variance unexplained	0.0593
raw MSE	12.7492

Normalized reconstruction error is MSE / mean(target^2). That is the number to compare across architectures: a ReLU MLP's output and a bilinear MLP's output have very different scales, so raw MSE compares the targets as much as the fits. Predicting a constant zero scores 1.0.

Across all 18 transcoders in this set, bilinear MLPs are consistently ~1.55x harder to reconstruct than ReLU ones (0.0387 vs 0.0249), in all three depth/layer cells with three seeds each.

Initialisation

The decoder bias is set to the mean target and the encoder is rescaled once from the first training batch, so the single allowed pass is spent learning features rather than undoing an initial scale mismatch. Measured: calibration_scale 1.88, init_normalized_after 1.32, init_normalized_before 0.911.

Provenance

Fitted to base weights with sha256 e944379d255365b5519c7fcb06dd8e5aa17274f9f38293ad40a6ce8e00b1448e, which is the checkpoint published at itzPotato/arithmetic-bilinear-2layer-seed2.

Loading

from src.transcoder.source import load_transcoder
transcoder, provenance = load_transcoder("hub:itzPotato/transcoder-bilinear-2layer-seed2-layer1", require_pinned=True)

require_pinned=True refuses anything but the registered commit sha in TRANSCODER_MODEL_REVISIONS (src/transcoder/config.py), so a run cannot silently load a moving branch.