parth12-ui/anlp-a2-moe-variant5

ANLP Assignment 2, Part 1: variant5

FFN variant 5: MoE, 4 experts, top-2 routing (active params matched to dense).

Decoder-only transformer trained from scratch to translate Vietnamese and Japanese into English on belumind/en-vi-ja-curated-500k-triplets. Input format: <bos> <vi|ja> source <sep> english <eos>, using the byte-level BPE in tokenizer.json.

total params	50,160,128
active params / token	33,382,912
layers / d_model / heads	8 / 512 / 8
dense d_ff	2048
context	256
training tokens	30,004,675

Test results

metric	overall	vi→en	ja→en
perplexity (target tokens)	6.03	4.94	7.38
BLEU (greedy, 1000 rows)	31.15	36.87	25.27

Loading

import json, torch
from safetensors.torch import load_model
from model_src.config import TransformerConfig   # files under model_src/ in this repo
from model_src.model import Transformer
cfg = TransformerConfig.from_dict(json.load(open("config.json")))
model = Transformer(cfg); load_model(model, "model.safetensors")