Chinese BabyLM 2026 submission (v3)

A Chinese masked language model for the NLPCC 2026 Chinese BabyLM Challenge, pretrained within the ≤102M Jieba-word data budget.

Architecture: DeBERTa-v2 base (12 layers, hidden 768, 12 heads), ~102M params
Tokenizer: character-level
Objective: masked language modeling

Methodology details will be released in the technical report after the evaluation period.

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("zymonody/chinese-babylm-v3")
model = AutoModelForMaskedLM.from_pretrained("zymonody/chinese-babylm-v3")

Downloads last month: 1,215

Safetensors

Model size

0.1B params

Tensor type

F32