YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Transformers from Scratch & Byte Latent Transformers
Advanced NLP — Assignment 1
Transformers from Scratch · RoPE · GQA · RMSNorm · BLT
👤 Author
Shourya Pillai · MS Research, CSE · IIIT Hyderabad
Roll No. 2026701047 · shourya.pillai@research.iiit.ac.in
🚀 Overview
This project explores Transformer architectures for encrypted-text reconstruction, comparing standard subword Transformers against a Byte Latent Transformer (BLT) operating directly on raw bytes.
All Transformer components were implemented from scratch in PyTorch,
without using nn.Transformer or nn.MultiheadAttention.
The experiment consists of five controlled configurations:
| Model | Main Change | Representation |
|---|---|---|
| C1 | Baseline Transformer | Custom BPE |
| C2 | RoPE | Custom BPE |
| C3 | GQA | Custom BPE |
| C4 | RMSNorm | Custom BPE |
| C5 | Byte Latent Transformer | Raw Bytes |
📊 Results
Final Test Performance
| Model | Bit Accuracy ↑ | Sequence Accuracy ↑ | Levenshtein ↓ | BLEU ↑ |
|---|---|---|---|---|
| C1 | 69.67% | 1.14% | 36.57 | 0.361 |
| C2 | 74.48% | 3.54% | 21.38 | 0.549 |
| C3 | 70.63% | 1.40% | 35.91 | 0.367 |
| C4 | 69.44% | 1.40% | 39.05 | 0.337 |
| C5 — BLT | 99.99% | 98.60% | 0.036 | — |
C5 achieves 99.988% bit-level accuracy and 98.599% exact sequence accuracy.
BLEU and ROUGE are not applicable to C5 because it operates directly on bytes rather than subword tokens.
🧠 What Changed?
C1 — Baseline
Sinusoidal absolute positional encoding + Multi-Head Attention + LayerNorm + custom BPE.
C2 — RoPE
Replaces absolute positional encoding with Rotary Positional Encoding.
→ Best-performing subword Transformer.
C3 — GQA
Replaces standard MHA with Grouped-Query Attention, reducing the number of key/value heads while retaining multiple query heads.
C4 — RMSNorm
Replaces LayerNorm with RMSNorm while keeping the remaining architecture unchanged.
C5 — BLT
Moves from subword tokens to raw bytes, using a Byte Latent Transformer architecture with local byte processing and global Transformer representations.
→ Near-perfect reconstruction.
📈 Training
Training Loss
Validation Loss
Learning Rate
Peak GPU Memory
🔑 Key Takeaways
1. RoPE improves the standard subword Transformer.
C2 substantially outperforms the baseline C1 across bit accuracy, sequence accuracy, Levenshtein distance, BLEU, and ROUGE.
2. GQA is competitive with the baseline but does not improve reconstruction quality in this experiment.
3. RMSNorm does not outperform LayerNorm for this task.
4. Byte-level modeling is extremely effective for this reconstruction problem.
C5 dramatically outperforms all subword-based configurations, achieving near-perfect reconstruction.
📦 Repository Contents
C1/ C1 checkpoint
C2/ C2 checkpoint
C3/ C3 checkpoint
C4/ C4 checkpoint
C5/ C5 BLT checkpoint
tokenizers/
├── tokenizer.json
└── cipher_tokenizer.json
metrics/
└── evaluation_results.csv
plots/
├── C1_C5_Train_Loss.png
├── C1_C5_Validation_Loss.png
├── C1_C5_Learning_Rate.png
└── Peak_GPU_Memory.png