lqstu/public_my_babys10

Chinese BabyLM First Model Package

This package contains the first trained base model that was used for the initial Track 1 runs.

Package contents

• config.json
• model.safetensors
• tokenizer.json
• tokenizer_config.json
• special_tokens_map.json
• vocab.txt
• README.md

This is a base masked language model, not a task-specific classifier checkpoint.

Model summary

• Architecture: BertForMaskedLM
• Hidden size: 768
• Layers: 8
• Attention heads: 12
• Intermediate size: 3072
• Max position embeddings: 512
• Vocab size: 19907
• Main file size: about 279 MB

Recommended environment

pip install torch transformers safetensors

Load for inference or continued pretraining

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_dir = "./chinese-babylm-bert-mlm-first"

tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(model_dir, trust_remote_code=True)
model.eval()

Load as a base encoder for downstream finetuning

For sentence classification or NLI style tasks, load the encoder and attach a task head:

from transformers import AutoTokenizer, AutoModel

model_dir = "./chinese-babylm-bert-mlm-first"

tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
encoder = AutoModel.from_pretrained(model_dir, trust_remote_code=True)
encoder.eval()

If using Hugging Face Trainer for classification, a common pattern is:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_dir = "./chinese-babylm-bert-mlm-first"

tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(
    model_dir,
    num_labels=2,
    trust_remote_code=True,
)

Because this checkpoint was trained as MLM, the classification head created above is randomly initialized and must be finetuned on the downstream task.

Local paths in our workspace

• Original model dir: /data/chinesebaby-output/models/chinese-babylm-bert-mlm
• This exported package will be placed under: /root/chinesebaby/releases/

Notes

• The export intentionally excludes training checkpoints such as checkpoint-*, optimizer states, and scheduler states.
• Those files are useful for resuming the original MLM run, but are not needed by someone who only wants to finetune the base model.