freococo/myanmar-english-corpus
Viewer • Updated • 48.4M • 171
How to use freococo/MyTranslation_300M_Base with Transformers:
# Use a pipeline as a high-level helper
# Warning: Pipeline type "translation" is no longer supported in transformers v5.
# You must load the model directly (see below) or downgrade to v4.x with:
# pip install "transformers<5.0.0"
from transformers import pipeline
pipe = pipeline("translation", model="freococo/MyTranslation_300M_Base", trust_remote_code=True) # pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained("freococo/MyTranslation_300M_Base", trust_remote_code=True, device_map="auto")MyTranslation_300M_Base is a 298.76M-parameter (inference; 312.94M training parameters including the Multi-Token Prediction head) bidirectional English ↔ Burmese (my) Neural Machine Translation encoder-decoder model trained from scratch on ~10B+ training tokens using 9× NVIDIA GeForce RTX 5090 GPUs.
Status: Base checkpoint (
Stage 1 Pretraining + Stage 2 Gold Cooldown), published prior to optional spoken/idiom Stage 3 polishing.
| Component | Specification |
|---|---|
| Inference Parameters | 298,763,201 (298.76M, BF16 model.safetensors ~598 MB) |
| Training Parameters | 312,940,000 (312.94M, includes MTPHeadBlock dropped at export) |
| Encoder | 12 bidirectional layers, d_model=1024, n_heads=16, n_kv_heads=4 (GQA, head_dim=64), ffn_dim=3072 SwiGLU |
| Decoder | 6 causal layers, d_model=1024, n_heads=16, n_kv_heads=4 (GQA), ffn_dim=3072 SwiGLU |
| Positional Encoding | RoPE (theta=50000.0, max_seq_len=2048 / trained at 1024) |
| Attention & Normalization | RMSNorm, QK-Norm, Gated Attention (sigmoid(gate_proj)), Value Residual (v0 mixing) |
| Encoder-Decoder Routing | Dense Layer-Wise Attention (enc_dwa_logits) + 3 learned SharedCrossKVProj mixtures (enc_router_logits) shared by decoder pairs (0,1), (2,3), (4,5) |
| Tokenizer & Vocab | 65,536-piece SentencePiece (UNK=0, BOS=1, EOS=2, PAD=3, <2my>=4, <2en>=5, <formal>=6, <spoken>=7, <bt>=8, <denoise>=9, <mask>=10), tied embeddings (embed_scale=32.0) |
[7, 8, 6, 2] ([0.226, 0.200, 0.187, 0.102])[9, 10, 8, 11] ([0.339, 0.206, 0.187, 0.124])[11, 9, 10, 8] ([0.340, 0.335, 0.278, 0.041])32,607 MiB VRAM each, torchrun --nproc_per_node=9, NCCL)bfloat16 autocast, TF32 matmul/cuDNN, custom BF16 gradient AllReduce with 1024x loss-scale protection, chunked cross-entropy (chunk_sz=2048) with Z-loss (1e-4), dynamic length-packed microbatches (32,768 tokens/GPU, ~295k budgeted tokens/step), load-balanced step grouping.~375k–400k tok/s in Stage 1 (98%–100% utilization across all 9 GPUs), ~405k–429k tok/s in Stage 2.tokenized_65k): 48,329,808 unique parallel/monolingual pairs (train_en.bin 7.9 GiB + train_my.bin 7.0 GiB ≈ ~8.0B raw tokens) + 22,329 held-out validation pairs.En -> My (<2my>) and My -> En (<2en>), explicit <formal> / <spoken> style conditioning (70% dropout-free application on tagged subsets), <bt> marker for backtranslated My -> En, and span-corruption <denoise> (Poisson(3.0), <mask>).stable — 100% Coverage + Gold Replay)
48,329,808 unique base rows (100% coverage) + 12,000,000 gold replay = 60,329,808 planned rows (60,331,713 rows trained across 58,916 steps, ~10B+ tokens).AdamW (lr=5.0e-4 after 900-step warmup, betas=(0.9, 0.95), weight_decay=0.01), label_smoothing=0.1, MTP weight=0.3, grad_clip=1.0.eval_val_nll on 1,024 uniform non-BT held-out pairs):0: Comb NLL 20.12821,500: Comb NLL 3.0945 (PPL 22.08)8,000: Comb NLL 1.7760 (PPL 5.91)19,000: Comb NLL 1.5803 (PPL 4.86)39,000: Comb NLL 1.4648 (PPL 4.33)58,000: Comb NLL 1.4298 (PPL 4.18)58,916): Comb NLL 1.4316 (PPL 4.19 | En->My: 1.7539 | My->En: 1.1548)cooldown — Clean Gold Only, Unsmoothed)
2,706,748 unique non-BT base rows + 1,500,000 gold replay = 4,206,748 planned rows (4,210,076 rows trained across 6,793 steps).100-step warmup to 3.0e-4 → cosine decay to 1.0e-5, label_smoothing=0.0, MTP weight=0.15, 50/50 En↔My.250: Comb NLL 1.2945 (PPL 3.65)1,500: Comb NLL 1.2403 (PPL 3.46)3,500: Comb NLL 1.1941 (PPL 3.30)5,500: Comb NLL 1.1687 (PPL 3.22)6,793): Comb NLL 1.1640 (PPL 3.20) | En->My: 1.4829 | My->En: 0.8901 (PPL 2.44)(Note: PPL values are measured on the tokenized_65k/val split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)
🏆 MyTranslation-300M Final Beam=4 Benchmark:
[En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။
[En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။
[My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy.
[My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today
[My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy.
[En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။
[En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။
[En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။
[My->En News]: Border trade checkpoints are being accelerated to improve international trade.
[My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business.
[My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow
trust_remote_code=True)
import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSeq2SeqLM
repo_id = "freococo/MyTranslation_300M_Base"
dtype = torch.float16 # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval()
sp = spm.SentencePieceProcessor()
sp.load(hf_hub_download(repo_id, "tokenizer.model"))
print("En->My Formal:", model.translate(
"The central bank raised interest rates to control inflation and stabilize the economy.",
sp, direction="en2my", style="formal", num_beams=4
))
print("En->My Spoken:", model.translate(
"Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.",
sp, direction="en2my", style="spoken", num_beams=4
))
print("My->En Formal:", model.translate(
"ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။",
sp, direction="my2en", style="formal", num_beams=4
))
training_codes.txt — exact multi-GPU PyTorch training script (train_300m.py)trainingcelloutputwitheval.txt — complete Stage 1 + Stage 2 training logs & Beam=4 benchmark outputcheckpoint-stage2-step-6793/ — FP32 model.safetensors + training_state.pt (best_model.pt)