ANLP Assignment 2 (part1): moe_shared
Decoder-only transformer with 1 shared expert always active plus 3 routed experts, top-1 routing.
Trained from scratch for IIIT-H Advanced NLP, Monsoon 2026. Architecture, training
code and evaluation live in the assignment repository; this repository holds only the
checkpoint.
Architecture
| Setting |
Value |
d_model |
512 |
n_layers |
8 |
n_heads |
8 |
n_kv_heads |
8 |
n_ctx |
256 |
vocab_size |
32000 |
ffn |
moe_shared |
d_ff |
512 |
n_routed_experts |
3 |
n_shared_experts |
1 |
top_k |
1 |
norm |
rmsnorm |
Parameters
| Count |
Value |
| Total |
41,603,584 |
| Active per token |
33,198,592 |
| Feed-forward (total / active) |
16,809,984 / 8,404,992 |
Test set results
| Metric |
Value |
| Perplexity (both directions) |
15.72 |
| BLEU vi->en |
30.07 |
| BLEU ja->en |
19.08 |
| BLEU mean |
24.57 |
Training
- Scored target tokens: 16,696,256
- Best validation loss: 2.4499
- Best validation perplexity: 11.59
Loading
import torch
payload = torch.load("model.pt", map_location="cpu", weights_only=False)
state_dict, config = payload["model"], payload["config"]
