MyTranslation_300M_Base

MyTranslation_300M_Base is a 298.76M-parameter (inference; 312.94M training parameters including the Multi-Token Prediction head) bidirectional English ↔ Burmese (my) Neural Machine Translation encoder-decoder model trained from scratch on ~10B+ training tokens using 9× NVIDIA GeForce RTX 5090 GPUs.

Status: Base checkpoint (Stage 1 Pretraining + Stage 2 Gold Cooldown), published prior to optional spoken/idiom Stage 3 polishing.


1. Architecture Highlights

Component Specification
Inference Parameters 298,763,201 (298.76M, BF16 model.safetensors ~598 MB)
Training Parameters 312,940,000 (312.94M, includes MTPHeadBlock dropped at export)
Encoder 12 bidirectional layers, d_model=1024, n_heads=16, n_kv_heads=4 (GQA, head_dim=64), ffn_dim=3072 SwiGLU
Decoder 6 causal layers, d_model=1024, n_heads=16, n_kv_heads=4 (GQA), ffn_dim=3072 SwiGLU
Positional Encoding RoPE (theta=50000.0, max_seq_len=2048 / trained at 1024)
Attention & Normalization RMSNorm, QK-Norm, Gated Attention (sigmoid(gate_proj)), Value Residual (v0 mixing)
Encoder-Decoder Routing Dense Layer-Wise Attention (enc_dwa_logits) + 3 learned SharedCrossKVProj mixtures (enc_router_logits) shared by decoder pairs (0,1), (2,3), (4,5)
Tokenizer & Vocab 65,536-piece SentencePiece (UNK=0, BOS=1, EOS=2, PAD=3, <2my>=4, <2en>=5, <formal>=6, <spoken>=7, <bt>=8, <denoise>=9, <mask>=10), tied embeddings (embed_scale=32.0)

Learned Cross-KV Encoder Mixture Weights (Post-Stage 2)

  • Cross-KV Pair 0 (Decoder 0–1): Top encoder layers [7, 8, 6, 2] ([0.226, 0.200, 0.187, 0.102])
  • Cross-KV Pair 1 (Decoder 2–3): Top encoder layers [9, 10, 8, 11] ([0.339, 0.206, 0.187, 0.124])
  • Cross-KV Pair 2 (Decoder 4–5): Top encoder layers [11, 9, 10, 8] ([0.340, 0.335, 0.278, 0.041])

2. Training Hardware, Data & Schedule

Cluster & Precision Setup

  • GPUs: 9× NVIDIA GeForce RTX 5090 (32,607 MiB VRAM each, torchrun --nproc_per_node=9, NCCL)
  • Precision & Sync: bfloat16 autocast, TF32 matmul/cuDNN, custom BF16 gradient AllReduce with 1024x loss-scale protection, chunked cross-entropy (chunk_sz=2048) with Z-loss (1e-4), dynamic length-packed microbatches (32,768 tokens/GPU, ~295k budgeted tokens/step), load-balanced step grouping.
  • Throughput: ~375k–400k tok/s in Stage 1 (98%–100% utilization across all 9 GPUs), ~405k–429k tok/s in Stage 2.

Dataset Coverage

  • Raw Dataset (tokenized_65k): 48,329,808 unique parallel/monolingual pairs (train_en.bin 7.9 GiB + train_my.bin 7.0 GiB ≈ ~8.0B raw tokens) + 22,329 held-out validation pairs.
  • Tasks & Register Control: Bidirectional En -> My (<2my>) and My -> En (<2en>), explicit <formal> / <spoken> style conditioning (70% dropout-free application on tagged subsets), <bt> marker for backtranslated My -> En, and span-corruption <denoise> (Poisson(3.0), <mask>).

Stage 1 (stable — 100% Coverage + Gold Replay)

  • Plan: 48,329,808 unique base rows (100% coverage) + 12,000,000 gold replay = 60,329,808 planned rows (60,331,713 rows trained across 58,916 steps, ~10B+ tokens).
  • Hyperparameters: AdamW (lr=5.0e-4 after 900-step warmup, betas=(0.9, 0.95), weight_decay=0.01), label_smoothing=0.1, MTP weight=0.3, grad_clip=1.0.
  • Stage 1 Validation Curve (eval_val_nll on 1,024 uniform non-BT held-out pairs):
    • Step 0: Comb NLL 20.1282
    • Step 1,500: Comb NLL 3.0945 (PPL 22.08)
    • Step 8,000: Comb NLL 1.7760 (PPL 5.91)
    • Step 19,000: Comb NLL 1.5803 (PPL 4.86)
    • Step 39,000: Comb NLL 1.4648 (PPL 4.33)
    • Step 58,000: Comb NLL 1.4298 (PPL 4.18)
    • Stage 1 Final (Step 58,916): Comb NLL 1.4316 (PPL 4.19 | En->My: 1.7539 | My->En: 1.1548)

Stage 2 (cooldown — Clean Gold Only, Unsmoothed)

  • Plan: 2,706,748 unique non-BT base rows + 1,500,000 gold replay = 4,206,748 planned rows (4,210,076 rows trained across 6,793 steps).
  • Hyperparameters: 100-step warmup to 3.0e-4 → cosine decay to 1.0e-5, label_smoothing=0.0, MTP weight=0.15, 50/50 En↔My.
  • Stage 2 Validation Curve:
    • Step 250: Comb NLL 1.2945 (PPL 3.65)
    • Step 1,500: Comb NLL 1.2403 (PPL 3.46)
    • Step 3,500: Comb NLL 1.1941 (PPL 3.30)
    • Step 5,500: Comb NLL 1.1687 (PPL 3.22)
    • Stage 2 Final (Step 6,793): Comb NLL 1.1640 (PPL 3.20) | En->My: 1.4829 | My->En: 0.8901 (PPL 2.44)

(Note: PPL values are measured on the tokenized_65k/val split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)


3. Beam=4 Benchmark Outputs (Unseen + Audit Prompts)

🏆 MyTranslation-300M Final Beam=4 Benchmark:
   [En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။
   [En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။
   [My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy.
   [My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today
   [My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy.
   [En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။
   [En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။
   [En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။
   [My->En News]: Border trade checkpoints are being accelerated to improve international trade.
   [My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business.
   [My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow

4. Usage Example (PyTorch + Transformers trust_remote_code=True)

import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSeq2SeqLM

repo_id = "freococo/MyTranslation_300M_Base"
dtype = torch.float16  # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4

model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval()
sp = spm.SentencePieceProcessor()
sp.load(hf_hub_download(repo_id, "tokenizer.model"))

print("En->My Formal:", model.translate(
    "The central bank raised interest rates to control inflation and stabilize the economy.",
    sp, direction="en2my", style="formal", num_beams=4
))
print("En->My Spoken:", model.translate(
    "Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.",
    sp, direction="en2my", style="spoken", num_beams=4
))
print("My->En Formal:", model.translate(
    "ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။",
    sp, direction="my2en", style="formal", num_beams=4
))

5. Reproducibility Artifacts Included

  • training_codes.txt — exact multi-GPU PyTorch training script (train_300m.py)
  • trainingcelloutputwitheval.txt — complete Stage 1 + Stage 2 training logs & Beam=4 benchmark output
  • checkpoint-stage2-step-6793/ — FP32 model.safetensors + training_state.pt (best_model.pt)
Downloads last month
-
Safetensors
Model size
0.3B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train freococo/MyTranslation_300M_Base