robgarct/hebbian-120m-featdim256-h2

Base

120M hebbian256 mixer LM: 20k steps of LM pretraining + 3k steps of plain Pile. The stage-1 control -- no recall objective.

Part of the recall-head study in scratch/roberto/recall_head_understanding/ of HazyResearch/mlp-mixer, which localises the model's in-context recall to a single head (layer 10, head 8) of 144.

• experiment config: mlp_mixer/online_hebbian_v10_noSiLU_initSmall_featdim256_120m_fast
• checkpoint file: final.ckpt

Load it the way the study does:

from scratch.roberto.recall_layers import common as C
model = C.load_model("hebbian_v10_120m_featdim256_fast_h2")