sanyam2005/anlp-a2-part2-mars

ANLP A2 Part 2 — MARS

Dense decoder-only Transformer (27,269,632 params) trained from scratch on browndw/human-ai-parallel-corpus (causal LM, 1x dataset) with MARS (Variance-reduced AdamW variant) implemented from scratch.

• tokens: 41,648,128
• final validation loss: 3.7451
• final test BLEU (64-token continuation): 1.24
• hyper-parameters: {'lr': 0.001, 'betas': [0.95, 0.99], 'eps': 1e-08, 'weight_decay': 0.1, 'gamma': 0.025, 'max_correction_norm': 1.0}