parth12-ui/anlp-a2-optim-sophia

ANLP Assignment 2, Part 2: sophia

Dense decoder-only transformer (33.4M parameters, 8 layers, d_model 512, context 256) pretrained from scratch for next-token prediction on browndw/human-ai-parallel-corpus, for 1x the dataset (39,075,840 tokens), using a from-scratch sophia optimizer (category: Hessian-approximation (bonus)).

Optimizer settings: lr=0.0003, betas=[0.965, 0.99], rho=0.04, weight_decay=0.2, hessian_interval=10.

metric at 1x dataset	value
validation loss	4.3059
validation perplexity	74.14
test BLEU (greedy 64-token continuation)	1.00

train_log.jsonl holds validation loss and test BLEU every 0.1x dataset tokens.

Loading

import json
from safetensors.torch import load_model
from model_src.config import TransformerConfig
from model_src.model import Transformer
model = Transformer(TransformerConfig.from_dict(json.load(open("config.json"))))
load_model(model, "model.safetensors")