ANLP Assignment 2, Part 1: variant4
FFN variant 4: MoE, 1 shared + 3 routed experts, top-1 routed (total params matched to dense).
Decoder-only transformer trained from scratch to translate Vietnamese and Japanese
into English on belumind/en-vi-ja-curated-500k-triplets. Input format:
<bos> <vi|ja> source <sep> english <eos>, using the byte-level BPE in tokenizer.json.
| total params | 33,378,816 |
| active params / token | 24,990,208 |
| layers / d_model / heads | 8 / 512 / 8 |
| dense d_ff | 2048 |
| context | 256 |
| training tokens | 30,004,675 |
Test results
| metric | overall | vi→en | ja→en |
|---|---|---|---|
| perplexity (target tokens) | 6.06 | 4.99 | 7.36 |
| BLEU (greedy, 1000 rows) | 31.02 | 36.64 | 25.25 |
Loading
import json, torch
from safetensors.torch import load_model
from model_src.config import TransformerConfig # files under model_src/ in this repo
from model_src.model import Transformer
cfg = TransformerConfig.from_dict(json.load(open("config.json")))
model = Transformer(cfg); load_model(model, "model.safetensors")
- Downloads last month
- 21