transcoder-bilinear-2layer-seed2-layer1-r2

Replicate 2 of 10, of a TopK transcoder fitted to layer 1 of itzPotato/arithmetic-bilinear-2layer-seed2.

What varies across the 10 replicates

The training data subset, and only that. Each replicate draws a fresh 500,000-problem training subset from the base model's train split. The transcoder's weight initialisation is identical across all ten (seed=0), so the spread across replicates measures variance due to the training data with the initialisation held constant.

This is deliberately not the usual "vary the training seed" replicate. It was specified this way so that data variance and initialisation variance are not confounded.

varies	training data subset
held fixed	weight init (seed=0), validation slice, all hyperparameters
this replicate's subset seed	1021002
this replicate's subset digest	da6d488f680956dc

The validation slice does not move. It is the reserved [500,000, 510,000) problems used by every transcoder in this project, so reconstruction numbers are comparable across replicates. Training draws exclude it, so no replicate ever trains on its own validation data.

Training subsets overlap. Ten disjoint 500,000-problem slices plus the 10,000 validation problems would need more of the split than exists, so draws are independent rather than a partition, and two replicates share roughly 10% of their problems in expectation.

Architecture and recipe

Identical to itzPotato/transcoder-bilinear-2layer-seed2-layer1, the single non-replicate transcoder for this layer:

d_model (in and out)	32
features	1024 (32x expansion)
active features per input (k)	32
decoder rows	unit norm
loss	MSE only, no L1, no lambda
optimiser	Adam, lr 0.0003, batches of 4096 activation vectors, one pass

Reconstruction

normalized reconstruction error	0.0445
fraction of variance unexplained	0.0578
raw MSE	12.4368

normalized = MSE / mean(target^2); predicting a constant zero scores 1.0.

Loading

from src.transcoder.source import load_transcoder
transcoder, provenance = load_transcoder("hub:itzPotato/transcoder-bilinear-2layer-seed2-layer1-r2", require_pinned=True)