ANLP Assignment 2 (part1): moe_4e2a
Decoder-only transformer with Mixture-of-Experts feed-forward, 4 experts, top-2 routing.
Trained from scratch for IIIT-H Advanced NLP, Monsoon 2026. Architecture, training
code and evaluation live in the assignment repository; this repository holds only the
checkpoint.
Architecture
| Setting |
Value |
d_model |
512 |
n_layers |
8 |
n_heads |
8 |
n_kv_heads |
8 |
n_ctx |
256 |
vocab_size |
32000 |
ffn |
moe_4e2a |
d_ff |
512 |
n_routed_experts |
4 |
n_shared_experts |
0 |
top_k |
2 |
norm |
rmsnorm |
Parameters
| Count |
Value |
| Total |
41,607,680 |
| Active per token |
33,202,688 |
| Feed-forward (total / active) |
16,809,984 / 8,404,992 |
Test set results
| Metric |
Value |
| Perplexity (both directions) |
14.77 |
| BLEU vi->en |
30.43 |
| BLEU ja->en |
19.78 |
| BLEU mean |
25.11 |
Training
- Scored target tokens: 16,696,256
- Best validation loss: 2.3812
- Best validation perplexity: 10.82
Loading
import torch
payload = torch.load("model.pt", map_location="cpu", weights_only=False)
state_dict, config = payload["model"], payload["config"]
