kinglacto/anlp-a2-p2-mars

ANLP Assignment 2, Part 2: dense transformer pretrained with MARS

The Part 1 dense configuration (6 layers, d_model 512, MLP 2048, 35.3M parameters, context 512) pretrained from scratch for exactly one epoch (36.96M tokens) on browndw/human-ai-parallel-corpus with a from-scratch implementation of MARS, at the best peak learning rate of a sweep.

Optimizer hyperparameters	lr=0.002, betas=[0.95, 0.99], eps=1e-08, weight_decay=0.1, gamma=0.025, max_grad_norm=1.0
Schedule	linear warmup (5%) then cosine decay to 10% of peak
Final validation loss / perplexity	3.9775 / 53.38
Test BLEU (greedy 128-token continuation vs human)	0.971
Training log	https://wandb.ai/yashas-kotre-iiit-hyderabad/anlp-a2-part2/runs/jkcruyp4

Tokenizer: 32k byte-level BPE trained on the corpus (tokenizer.json). Load with src/part1/model.py of the assignment code as for the Part 1 models.