Titans MAC transformer (MARV research checkpoint)
A small byte-level language model with a Titans neural memory (0.37M params), trained on enwik8 as part of research into whether test-time memory (weights that update via gradient descent during inference) can be feature-diffed and causally ablated the way MARV already does for frozen model weights.
Trained 3000 steps, final val loss 1.863 nats (byte-uniform baseline: 5.545).
See the marv-titan branch of https://github.com/thebnbrkr/marv for the full writeup
(experiments/README.md) and the code needed to reload this checkpoint:
# needs: pip install titans-pytorch, and titans_real_text.py from the marv-titan branch
from titans_real_text import build_model
import torch
model = build_model()
model.load_state_dict(torch.load("titans_marv_enwik8.pt", map_location="cpu"))
model.eval()
Findings from this checkpoint's analysis: the memory, trained on real language modeling, accumulates writes rather than forgetting them (norm_ratio > 1, magnitude never shrinks below its original size) -- the OPPOSITE of the aggressive exponential forgetting seen when the same architecture is trained on a toy autoassociative-recall task instead. See the repo for details.