leviathanWakes/anlp-a1-transformer-ablations

ANLP Assignment 1: Transformer Ablations

This repository contains the trained weights for a sequence-to-sequence translation task using various custom-built Transformer architectures from scratch.

Checkpoints Included

• C1_baseline/best.pt: Baseline Transformer (Sinusoidal PE, MHA, LayerNorm)
• C2_rope/best.pt: Rotary Positional Encoding (RoPE)
• C3_gqa/best.pt: Grouped-Query Attention (GQA)
• C4_rmsnorm/best.pt: RMSNorm

Tokenizers

The shared BPE tokenizers used to encode the sequence data are included in the root directory:

• src_tokenizer.json
• tgt_tokenizer.json

Author: Rachit Mehta (2024101089)