ANLP Assignment 2, Part 1: dense 2-layer MLP (512 -> 2048 -> 512)
Decoder-only transformer trained from scratch for Vietnamese/Japanese -> English translation on
belumind/en-vi-ja-curated-500k-triplets (4 epochs, 139.3M input tokens).
| Layers / d_model / heads / context | 6 / 512 / 8 / 256 |
| Feed-forward | dense 2-layer MLP (512 -> 2048 -> 512) |
| Parameters | 35,265,024 total, 35,265,024 active per token |
| Tokenizer | 32k byte-level BPE shared by en/vi/ja (tokenizer.json) |
| Test perplexity (en target tokens) | 4.627 (vi->en 3.885, ja->en 5.509) |
| Test BLEU, greedy | 39.07 (vi->en 44.19, ja->en 33.87) |
| Training log | https://wandb.ai/yashas-kotre-iiit-hyderabad/anlp-a2-part1/runs/wm9h0qy6 |
Input format: <vi> or <ja>, the source sentence, then <en>; generate until <eos>.
The architecture is defined in src/part1/model.py of the assignment code:
model = Transformer(TransformerConfig(**json.load(open("config.json")))), then
safetensors.torch.load_model(model, "model.safetensors").
- Downloads last month
- 32