ANLP Assignment 2 (part1): moe_active_matched
Decoder-only transformer with Mixture-of-Experts, 4 experts, top-2 routing, active parameters matched to the dense baseline.
Trained from scratch for IIIT-H Advanced NLP, Monsoon 2026. Architecture, training
code and evaluation live in the assignment repository; this repository holds only the
checkpoint.
Architecture
| Setting |
Value |
d_model |
512 |
n_layers |
8 |
n_heads |
8 |
n_kv_heads |
8 |
n_ctx |
256 |
vocab_size |
32000 |
ffn |
moe_active_matched |
d_ff |
1024 |
n_routed_experts |
4 |
n_shared_experts |
0 |
top_k |
2 |
norm |
rmsnorm |
Parameters
| Count |
Value |
| Total |
58,401,280 |
| Active per token |
41,599,488 |
| Feed-forward (total / active) |
33,603,584 / 16,801,792 |
Test set results
| Metric |
Value |
| Perplexity (both directions) |
13.92 |
| BLEU vi->en |
30.91 |
| BLEU ja->en |
20.27 |
| BLEU mean |
25.59 |
Training
- Scored target tokens: 16,696,256
- Best validation loss: 2.3381
- Best validation perplexity: 10.36
Loading
import torch
payload = torch.load("model.pt", map_location="cpu", weights_only=False)
state_dict, config = payload["model"], payload["config"]
