Cipher-to-Plaintext Transformers — Architectural Ablation
Encoder–decoder transformers built from fundamental PyTorch operations (no
nn.Transformer, no nn.MultiheadAttention) for ANLP Assignment 1.
The task: recover English plaintext from a byte-wise repeating-key XOR cipher with an 8-byte key, one byte (8 bits) per plaintext character. The key itself is a property of the course dataset and is not published here.
Configurations
Each differs from the C1 baseline by exactly one component.
| Positional | Attention | Norm | Tokenization | |
|---|---|---|---|---|
| C1 | Sinusoidal | Multi-head | LayerNorm | byte-level BPE |
| C2 | RoPE | Multi-head | LayerNorm | byte-level BPE |
| C3 | Sinusoidal | Grouped-query (2 kv heads) | LayerNorm | byte-level BPE |
| C4 | Sinusoidal | Multi-head | RMSNorm | byte-level BPE |
| C5 | Sinusoidal | Multi-head | LayerNorm | BLT, token-free |
Shared hyperparameters: d_model 256, 8 heads, 6 layers, d_ff 1024, dropout 0.1,
AdamW at 3e-4, batch 32, 20 epochs, 64-character chunks.
Results
Greedy decoding on a held-out test split of 4,371 chunks.
| seq acc | char acc | bit acc | Levenshtein | params | s/epoch | peak GB | |
|---|---|---|---|---|---|---|---|
| C1 | 96.71% | 99.8928% | 99.9630% | 0.064 | 11,316,480 | 56.7 | 0.95 |
| C2 | 97.16% | 99.3812% | 99.7809% | 0.069 | 11,316,480 | 58.7 | 0.94 |
| C3 | 97.21% | 99.8899% | 99.9623% | 0.065 | 9,547,008 | 52.4 | 0.92 |
| C4 | 97.46% | 99.9178% | 99.9723% | 0.051 | 11,308,288 | 55.1 | 0.88 |
| C5 | 99.50% | 99.9904% | 99.9962% | 0.006 | 14,532,608 | 50.1 | 0.93 |
Measured noise floor (C1, three seeds): ±0.0175pp character accuracy. Differences smaller than that are not interpreted.
Files
C1_BytePairTokenizer_gelu_best.pt(45.9 MB)C1_BytePairTokenizer_relu_best.pt(45.9 MB)C2_BytePairTokenizer_gelu_best.pt(45.4 MB)C2_BytePairTokenizer_relu_best.pt(45.4 MB)C3_BytePairTokenizer_gelu_best.pt(38.8 MB)C3_BytePairTokenizer_relu_best.pt(38.8 MB)C4_BytePairTokenizer_gelu_best.pt(45.8 MB)C4_BytePairTokenizer_relu_best.pt(45.8 MB)C5_ByteTokenizer_gelu_best.pt(59.3 MB)C5_ByteTokenizer_relu_best.pt(59.3 MB)histories/C1_BytePairTokenizer_gelu_history.json(0.0 MB)histories/C1_BytePairTokenizer_relu_history.json(0.0 MB)histories/C2_BytePairTokenizer_gelu_history.json(0.0 MB)histories/C2_BytePairTokenizer_relu_history.json(0.0 MB)histories/C3_BytePairTokenizer_gelu_history.json(0.0 MB)histories/C3_BytePairTokenizer_relu_history.json(0.0 MB)histories/C4_BytePairTokenizer_gelu_history.json(0.0 MB)histories/C4_BytePairTokenizer_relu_history.json(0.0 MB)histories/C5_ByteTokenizer_gelu_history.json(0.0 MB)histories/C5_ByteTokenizer_relu_history.json(0.0 MB)figures/attention_C1.png(0.0 MB)figures/attention_C2.png(0.0 MB)figures/attention_C3.png(0.0 MB)figures/attention_C4.png(0.0 MB)figures/attention_C5.png(0.0 MB)figures/bars_memory.png(0.1 MB)figures/bars_seq_acc.png(0.1 MB)figures/bars_speed.png(0.1 MB)figures/curves_val_acc.png(0.1 MB)figures/curves_val_loss.png(0.2 MB)bpe_vocab1024.json(0.0 MB)results.json(0.0 MB)heldout_result.json(0.0 MB)
Loading a checkpoint
These are raw state_dict files for an architecture implemented from scratch, so
they need this project's model code:
from train import build, TOKENIZERS
import torch
tokenizer = TOKENIZERS["bpe"]() # "byte" for C5
model = build("C1", tokenizer, activation="gelu")
model.load_state_dict(torch.load("C1_BytePairTokenizer_gelu_best.pt"))
model.eval()
bpe_vocab1024.json is the trained tokenizer and is required for C1–C4; C5 uses raw
bytes and needs no vocabulary.
Notes
- Reported numbers use greedy decoding, as the assignment specifies. Accuracy logged during training is teacher-forced and is not comparable.
- BLEU and ROUGE are reported for the tokenized configurations only; they saturate near 0.99 on this task and do not discriminate between configurations.
- A held-out combination experiment shows the model memorises a phase-independent byte→character table rather than inferring the cipher. Withholding 20 of the 424 (character, phase) combinations drops accuracy on them from 99.96% to 23.8% — and every combination whose key byte is unique across the key scores exactly 0%. The residual comes entirely from one key byte that repeats at two phases, so those combinations produce cipher bytes the model had already seen elsewhere. No structural generalisation occurs.