orangebreak/anlp-a2-part2-optimizers

Task 2 of Advanced NLP Assignment 2: one dense decoder-only transformer (33M params) trained from scratch on browndw/human-ai-parallel-corpus with five hand-written optimizers, implemented from the appendix pseudocode of Fantastic Pretraining Optimizers and Where to Find Them (arXiv:2509.02046). Every run uses an identical 39,073,280-token budget (1x the dataset) and an identical seed-20 data stream.

Optimizer	Category	Final val loss	Final val PPL	Final BLEU
adamw	Baseline	3.9975	54.46	0.066
cautious	Variance-reduced AdamW variant	4.0855	59.47	0.067
lion	Memory-efficient	4.1461	63.19	0.041

Each subfolder holds final.pt (state dict + config) and history.json (validation loss and BLEU at every 0.1x of the dataset). tokenizer.json is the shared English 16k byte-level BPE used by all runs.