kinglacto/anlp-a2-p2-lion

ANLP Assignment 2, Part 2: dense transformer pretrained with Lion

The Part 1 dense configuration (6 layers, d_model 512, MLP 2048, 35.3M parameters, context 512) pretrained from scratch for exactly one epoch (36.96M tokens) on browndw/human-ai-parallel-corpus with a from-scratch implementation of Lion, at the best peak learning rate of a sweep.

Optimizer hyperparameters	lr=0.0003, betas=[0.9, 0.99], weight_decay=1.0
Schedule	linear warmup (5%) then cosine decay to 10% of peak
Final validation loss / perplexity	4.6463 / 104.20
Test BLEU (greedy 128-token continuation vs human)	0.683
Training log	https://wandb.ai/yashas-kotre-iiit-hyderabad/anlp-a2-part2/runs/s663mkee

Tokenizer: 32k byte-level BPE trained on the corpus (tokenizer.json). Load with src/part1/model.py of the assignment code as for the Part 1 models.