parth12-ui/anlp-a2-moe-variant2

ANLP Assignment 2, Part 1: variant2

FFN variant 2: MoE, 4 experts, top-1 routing (total params matched to dense).

Decoder-only transformer trained from scratch to translate Vietnamese and Japanese into English on belumind/en-vi-ja-curated-500k-triplets. Input format: <bos> <vi|ja> source <sep> english <eos>, using the byte-level BPE in tokenizer.json.

total params	33,382,912
active params / token	20,800,000
layers / d_model / heads	8 / 512 / 8
dense d_ff	2048
context	256
training tokens	30,004,675

Test results

metric	overall	vi→en	ja→en
perplexity (target tokens)	6.66	5.43	8.15
BLEU (greedy, 1000 rows)	29.90	35.01	24.47

Loading

import json, torch
from safetensors.torch import load_model
from model_src.config import TransformerConfig   # files under model_src/ in this repo
from model_src.model import Transformer
cfg = TransformerConfig.from_dict(json.load(open("config.json")))
model = Transformer(cfg); load_model(model, "model.safetensors")