mohjkhan/anlp-a1-transformers

ANLP Assignment 1 — Transformers from Scratch (C1–C5)

Custom encoder–decoder Transformer built from fundamental PyTorch ops (no nn.Transformer / nn.MultiheadAttention) for a repeating-XOR cipher → plaintext decryption task, plus an ablation over five configurations:

Config	Positional	Attention	Norm	Tokenization
C1	Sinusoidal	Multi-Head	LayerNorm	Subword
C2	RoPE	Multi-Head	LayerNorm	Subword
C3	Sinusoidal	Grouped-Query	LayerNorm	Subword
C4	Sinusoidal	Multi-Head	RMSNorm	Subword
C5	Sinusoidal	Multi-Head	LayerNorm	BLT (token-free)

Each *.pt file contains {"model": state_dict, "config": {...}}. Load with the build_transformer / build_blt factories in the accompanying code repository.

• Training logs (Weights & Biases):
• Code:

Metrics for each configuration are in the corresponding *_metrics.json.