parth12-ui/anlp-a2-moe-variant3

ANLP Assignment 2, Part 1: variant3

FFN variant 3: MoE, 4 experts, top-2 routing (total params matched to dense).

Decoder-only transformer trained from scratch to translate Vietnamese and Japanese into English on belumind/en-vi-ja-curated-500k-triplets. Input format: <bos> <vi|ja> source <sep> english <eos>, using the byte-level BPE in tokenizer.json.

total params	33,382,912
active params / token	24,994,304
layers / d_model / heads	8 / 512 / 8
dense d_ff	2048
context	256
training tokens	30,004,675

Test results

metric	overall	vi→en	ja→en
perplexity (target tokens)	6.19	5.10	7.51
BLEU (greedy, 1000 rows)	30.76	36.14	25.24

Loading

import json, torch
from safetensors.torch import load_model
from model_src.config import TransformerConfig   # files under model_src/ in this repo
from model_src.model import Transformer
cfg = TransformerConfig.from_dict(json.load(open("config.json")))
model = Transformer(cfg); load_model(model, "model.safetensors")