Hing-mmBERT

jhu-clsp/mmBERT-base with 2B tokens of continued masked-language-model pretraining on L3Cube-HingCorpus (roman-script Hinglish tweets, the corpus HingBERT was trained on). Built for the USC CSCI 544 project hinglish-orthobench (Fall 2026).

• Recipe: warmup-stable-decay schedule, peak lr 3e-4, 15% masking, 64 x 512 tokens per step, per-tweet attention, bf16; 20M warmup, constant to 1.8B tokens, 1-sqrt decay over the last 200M. One L40S, about 11 hours.
• Held-out MLM loss on 20k lines of HingCorpus's validation file (fixed 15% mask): 1.972, against 4.648 for mmBERT-base and 2.351 for a 1B-token run at lr 5e-5.
• Training code and the exact commands: models/transformers/hingmmbert/ in the project repo.

Load it like any ModernBERT encoder:

from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("hinglish-orthobench/hing-mmbert")
model = AutoModel.from_pretrained("hinglish-orthobench/hing-mmbert")

License

CC BY-NC-SA 4.0, inherited from HingCorpus. Non-commercial use only; derived works carry the same license.