sahoosk/anlp-a1-transformer-blt-ablation

ANLP Assignment 1 — Transformer + BLT Ablation

Five architectural configurations of an encoder–decoder Transformer built from scratch (no nn.Transformer / nn.MultiheadAttention), evaluated on a character-level decipherment task. Each of C2–C5 changes exactly one component from the C1 base so the effect of each change can be measured in isolation.

Results

Config	Change from base	Bit acc. (%)	Seq. acc. (%)	Levenshtein	Para (M)	Peak mem (MB)
C1	— (base)	99.25	92.00	0.12	6.05	521
C2	positional → RoPE	99.80	97.00	0.34	6.05	521
C3	attention → Grouped-Query	99.03	89.18	0.17	5.16	508
C4	normalization → RMSNorm	99.36	93.25	0.10	6.04	483
C5	tokenization → BLT (token-free)	99.80	99.40	0.005	8.81	732

Configurations

Config	Positional	Attention	Normalization	Tokenization
C1	Sinusoidal absolute	Multi-Head	LayerNorm	Subword (byte-level BPE)
C2	RoPE	Multi-Head	LayerNorm	Subword (byte-level BPE)
C3	Sinusoidal absolute	Grouped-Query	LayerNorm	Subword (byte-level BPE)
C4	Sinusoidal absolute	Multi-Head	RMSNorm	Subword (byte-level BPE)
C5	Sinusoidal absolute	Multi-Head	LayerNorm	BLT (token-free, entropy-patched bytes)

How C5 (BLT) works

• Bytes, not the text of bytes. The ciphertext is a binary sequence stored as ASCII 0 / 1 characters. Every 8 bits are packed into one byte value 0–255, so a 256-character cipher segment is 32 bytes, not 256.
• Entropy-based dynamic patching. Patches are variable-length: a new patch opens where H(x_t | x_{t-2}, x_{t-1}) > θ. The entropy comes from a lightweight order-2 byte n-gram model fit on the training split only (entropy_model.json, also stored as buffers inside best.pt); no separate neural LM is trained. θ is calibrated to a target mean patch length, giving ~10 variable-length patches per 34-byte sequence.
• A standard learned 256-entry byte embedding represents the byte values.

Repository layout

C1/best.pt              checkpoint (state_dict + args + config)
C1/config.json          exact CLI arguments used
C1/metrics.json         test metrics, timings, peak memory
C1/tokenizer_*.json     the from-scratch byte-level BPE vocab + merges
...                     idem for C2..C5
C5/entropy_model.json   the n-gram entropy model defining patch boundaries

Loading a checkpoint

import argparse
import torch
from src.configs import get_config, build_model

ck    = torch.load("C1/best.pt", map_location="cpu", weights_only=False)
cfg   = get_config(ck["config"]["name"])
args  = argparse.Namespace(**ck["args"])
model = build_model(cfg, args, ck["src_vocab_size"], ck["tgt_vocab_size"])
model.load_state_dict(ck["model_state"])
model.eval()

Training and evaluation

See the source repository for the exact train.py invocation per config. Every run uses greedy decoding for evaluation, a fixed 80/10/10 document-level split (seed 42), and the same hyperparameters (d_model, depth, learning rate, batch size) across all five configs — only the one component listed in the Configurations table differs.