ANLP Assignment 2 (part1): dense
Decoder-only transformer with dense two-layer MLP feed-forward (the baseline).
Trained from scratch for IIIT-H Advanced NLP, Monsoon 2026. Architecture, training
code and evaluation live in the assignment repository; this repository holds only the
checkpoint.
Architecture
| Setting |
Value |
d_model |
512 |
n_layers |
8 |
n_heads |
8 |
n_kv_heads |
8 |
n_ctx |
256 |
vocab_size |
32000 |
ffn |
dense |
d_ff |
2048 |
n_routed_experts |
0 |
n_shared_experts |
0 |
top_k |
0 |
norm |
rmsnorm |
Parameters
| Count |
Value |
| Total |
41,579,008 |
| Active per token |
41,579,008 |
| Feed-forward (total / active) |
16,797,696 / 16,797,696 |
Test set results
| Metric |
Value |
| Perplexity (both directions) |
13.76 |
| BLEU vi->en |
30.75 |
| BLEU ja->en |
20.30 |
| BLEU mean |
25.52 |
Training
- Scored target tokens: 16,696,256
- Best validation loss: 2.3278
- Best validation perplexity: 10.26
Loading
import torch
payload = torch.load("model.pt", map_location="cpu", weights_only=False)
state_dict, config = payload["model"], payload["config"]
