mdhvm/anlp-m26-a1-cipher-transformers

Checkpoints for Advanced NLP Assignment 1 — an encoder-decoder Transformer built from fundamental PyTorch operations (no nn.Transformer, nn.MultiheadAttention, or nn.LayerNorm) trained to decrypt a repeating-key XOR cipher back to plaintext.

Task

cipher_byte[i] = ord(plain[i]) XOR key[i mod 8], key = b"ANLP2026", 8 bits per character. The model maps cipher bits to plaintext. Here i is the character index within a line, so the key restarts at every line. Lines are sliced into contiguous non-overlapping 32-character windows (256 source bits) starting at offset 0, dropping any trailing partial window. Because 32 is a multiple of the key period, every window start is at key phase 0 — identical to a full line — so the slicing is exactly alignment-preserving. Shifting the window start off a multiple of 8 breaks that, which is what the key-phase probes test.

Configurations

Each differs from the C1 base by exactly one component.

Config	Change	Test bit acc %	Test seq acc %	Params
C1	C1 base (sinusoidal, MHA, LayerNorm, subword)	95.74	68.00	6,047,232
C2	C2 RoPE	95.67	66.50	6,047,232
C3	C3 Grouped-Query Attention (2 KV heads)	94.96	60.75	5,162,496
C4	C4 RMSNorm	95.63	66.25	6,042,112
C5	C5 BLT (token-free, entropy patching)	99.98	98.40	9,339,904

The checkpoints and the table above are seed 1337; the report quotes mean ± sd over seeds 1337 / 7 / 42.

Files

file	contents
C*/best.pt	model state_dict plus the config and hyperparameters used
C*/results.json	test metrics, efficiency numbers, OOD probes
C*/history.json	per-eval training curves
tokenizers/bpe_*.json	the from-scratch BPE vocabulary and merge list
tables/, figures/	aggregated ablation tables and plots

Loading

import torch, sys
sys.path.insert(0, "src")
from train import CONFIGS, build_model

ck = torch.load("C1/best.pt", map_location="cpu")
# build_model(CONFIGS["C1"], ck["hp"], bundle, device) -- see src/evaluate.py

Training code, tests and the full study: see the accompanying submission.