MiniCPM5-2B-ja-cpt-v1

English | 日本語

This experimental model was developed by continually pretraining MiniCPM5-2B-Midtrain on approximately 500M tokens of predominantly Japanese text using a single NVIDIA RTX 5090. The target share of Japanese data was 70%; the corpus also includes English, Chinese, code, and mathematics.

The model can be used for Japanese text completion or as a starting point for further fine-tuning. This training run did not include supervised fine-tuning (SFT), preference training, or reinforcement learning (RL). Instruction-following performance for conversational use has not been evaluated.

Model Details

Item	Details
Base model	openbmb/MiniCPM5-2B-Midtrain
Base model revision	0a45344e530534b7090df798421e57fc2d0e19f2
Architecture	LlamaForCausalLM, 2B class
Training method	Full-parameter continued pretraining
Weight format	BF16, Safetensors, full model weights
Tokenizer	Uses the base model's vocabulary without expansion
Configured maximum sequence length	131,072 tokens (inherited from the base model; long-context performance after CPT has not been evaluated)

Training Data

The corpus selected for training contains 500,035,676 tokens. This count includes document text and end-of-document EOS tokens, but excludes padding and held-out data. Due to packing and the handling of remainders, this should be distinguished from the number of tokens actually used for optimizer updates.

The table below shows the target mixture by token count.

Category	Dataset / Subset	Share
Japanese educational-quality text	fineweb-2-edu-japanese, sample_10BT	50%
Japanese Wikipedia	wikipedia, 20231101.ja	10%
Japanese high-quality web text	FineWeb2-HQ, jpn_Jpan	10%
English	Ultra-FineWeb, English L2	15%
Chinese	Ultra-FineWeb, Chinese L2	5%
Code	UltraData-Code, L2 / L3	5%
Mathematics	UltraData-Math, L2 / L3	5%

Code L2 and L3 account for 3% and 2% of the total corpus, respectively. Within the code portion, the language mixture is Python 50%, JavaScript 20%, C++ 15%, and Java 15%. For mathematics, L2-preview accounts for 3% of the total corpus, L3-Textbook-Exercise-Synthetic for 1%, and L3-QA-Synthetic for 1%.

Preprocessing included text normalization, quality filtering, and exact and near-duplicate removal. Text was tokenized with the base tokenizer without adding BOS, with EOS appended at document boundaries, then split into chunks of up to 4,096 tokens and packed for training.

Training Configuration

Item	Setting
Starting point	Original Midtrain weights
Training duration	Configured for 1 epoch; 3,817 optimizer steps
Micro batch size / gradient accumulation	8 / 4 (effective batch size: 32)
Peak learning rate	5e-5
Learning rate schedule	22 warmup steps, cosine decay, final learning rate 5e-6
Optimizer	8-bit AdamW, betas (0.9, 0.95), epsilon 1e-8
Weight decay / gradient clipping	0.1 / 1.0
Precision	BF16
Memory and compute optimizations	FlashAttention 2, Liger cuTile, gradient checkpointing, selective checkpointing
Seed	20260915
Frameworks	Axolotl 0.19.0, Transformers 5.16.1
Hardware	NVIDIA RTX 5090

Loss on this same validation set decreased from 2.4676 at the start of training to 2.3148 at the end. These values come from the validation set used to monitor training, not from an independent final test set.

Evaluation

Japanese

The base model and this model were compared on the validation split of JGLUE v1.2. All scores are percentages; higher is better.

Task	Evaluation examples	Metric	Few-shot	MiniCPM5-2B-Midtrain	This model
JCommonsenseQA	1,119	Accuracy	3	52.90	57.46
JNLI	2,434	Accuracy	3	53.00	59.16
JSQuAD	4,442	Exact match	2	60.45	62.02

• Evaluation date: 2026-09-21 (this model). Base model scores come from an earlier evaluation using the same questions and prompts.
• Evaluation implementation: lm-eval 0.4.11, using its Japanese task prompts and scoring logic, with data loading adapted to a fixed revision of JGLUE.
• JGLUE revision: c96c6af54200a404f31a3a4cd0b289786aa78809.
• JCommonsenseQA and JNLI use candidate likelihoods for classification. JSQuAD uses greedy generation with up to 128 new tokens and is scored by exact match with whitespace normalization. The JSQuAD score is not token F1.
• No chat template or additional BOS token was used. Evaluation seed: 20260915; HF backend / BF16 / FlashAttention 2; maximum sequence length: 8,192.

These are local evaluation results, not official leaderboard submissions. The model outperformed the base model on the three tasks evaluated, but this does not establish improvements across all aspects of Japanese language ability. Reproducibility across multiple training seeds and robustness to prompt changes have not been evaluated.

English and Chinese

The base model and this model were also compared on HellaSwag (English) and C-Eval (Chinese). Scores are percentages; differences are this model minus the base model, in percentage points (pp).

Task	Evaluation examples	Metric	Few-shot	MiniCPM5-2B-Midtrain	This model	Difference (pp)	95% interval for difference (pp)
HellaSwag	10,042	acc_norm	0	62.90	63.30	+0.41	[-0.01, +0.86]
C-Eval	1,346	Accuracy	5	64.12	63.52	-0.59	[-2.15, +1.04]

• Evaluation date: 2026-09-21 for both models, using lm-eval 0.4.11 and identical questions, prompts, and few-shot examples.
• HellaSwag: full validation split, task hellaswag, length-normalized candidate likelihood accuracy (acc_norm). Dataset: Rowan/hellaswag, revision 218ec52e09a7e7462a5400043bb9a69a41d06b76.
• C-Eval: full validation (val) split across all 52 subjects, task group ceval-valid, accuracy weighted by the number of questions per subject. The first five dev examples for each subject serve as few-shot demonstrations. Dataset: ceval/ceval-exam, revision 617524a00b307ff6f9933702f724131fe12ca7ce.
• Both benchmarks use likelihood-based scoring, without a chat template or additional BOS token. Evaluation seed: 20260915; HF backend / BF16 / FlashAttention 2; batch size: 4; maximum sequence length: 8,192. No prompt truncation was needed.
• The intervals use 2,000 paired bootstrap resamples of questions. C-Eval resampling is performed within each subject, preserving subject sizes. Differences are calculated before rounding the displayed scores.

The point estimate increased slightly on HellaSwag and decreased slightly on C-Eval. Both intervals include zero, so these evaluations do not establish a clear improvement or decline. The results are consistent with broadly maintained performance on these two benchmarks, but do not demonstrate equivalence or retention across all English and Chinese capabilities. These are local results, not official leaderboard submissions. Overlap between the benchmarks and the CPT corpus has not been checked.

Usage

The following example performs text completion in an environment with CUDA and BF16 support. It requires transformers, torch, and accelerate. Transformers 5.16.1 was used for training and evaluation.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nazunaex/MiniCPM5-2B-ja-cpt-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

inputs = tokenizer(
    "日本語の言語モデルを評価する際には、",
    return_tensors="pt",
    add_special_tokens=False,
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

completion = outputs[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(completion, skip_special_tokens=True))

Evaluation Scope and Limitations

• English and Chinese evaluation is limited to HellaSwag and C-Eval, respectively. Retention of coding and mathematics capabilities has not been evaluated separately after this CPT run.
• Performance on sequences longer than 4,096 tokens has not been evaluated.
• The model may generate incorrect information or text that reflects biases in its training data.
• Suitability for instruction-following and conversational applications needs to be assessed through further training and use-case-specific evaluation.

License and Acknowledgments

Apache-2.0

The base model is released under Apache-2.0. For the licenses and provenance of the training data, refer to the dataset cards linked above.

I thank OpenBMB for releasing the base model, and the developers of the datasets and training and evaluation tools used in this work.