richarddzh/tiny-qwen3-30m-zh

Tiny Qwen3 30M Chinese

Tiny Qwen3 30M Chinese is a 29.57M-parameter Chinese causal language model trained from scratch on the Chinese Small LM Corpus. It is a compact educational model for experimenting with tokenizer training, language-model pretraining, checkpointing, and Chinese text generation in the Hugging Face ecosystem.

This is a base language model, not an instruction-tuned or chat model.

Dataset

The model uses the train split of richarddzh/chinese-small-lm-corpus. The dataset is shuffled with seed 42, after which 10,000 documents are reserved for validation and the remaining documents are used for training.

Data preparation:

• Empty texts are discarded.
• Texts are tokenized without automatically added special tokens and separated with <|endoftext|>.
• Tokens from consecutive documents are concatenated and packed into fixed 512-token sequences without padding.
• Validation uses the same packing procedure and is limited to 256 packed sequences (131,072 tokens) per evaluation.

Tokenizer

A byte-level BPE tokenizer is trained from scratch on up to 100,000 training documents, with each document truncated to 2,000 characters for tokenizer training. It uses NFC normalization, an 8,192-token vocabulary, a minimum token frequency of 2, and the special tokens <|endoftext|>, <unk>, <|im_start|>, and <|im_end|>. The tokenizer supports sequences up to 1,024 tokens, while pretraining uses 512-token sequences.

Model architecture

Parameter	Value
Architecture	Qwen3ForCausalLM
Parameters	29,567,616
Vocabulary size	8,192
Hidden size	512
Transformer layers	9
Attention heads	8
Key/value heads	2 (GQA)
Head dimension	64
MLP intermediate size	1,408
Maximum position embeddings	1,024
RoPE theta	1,000,000
Tied input/output embeddings	Yes

Training method

The model is pretrained from random initialization with the standard next-token causal language-modeling objective. The notebook uses the Hugging Face Trainer, packed iterable datasets, mixed precision when supported, gradient accumulation, periodic validation, and checkpoint-based resume.

Training configuration

Parameter	Value
Optimizer	AdamW (adamw_torch_fused on CUDA)
Maximum optimizer steps	30,000
Sequence length	512
Micro-batch size	32 sequences/device
Gradient accumulation	2 steps
Effective batch size	64 sequences / 32,768 tokens
Learning rate	5e-4
Scheduler	Cosine
Warmup steps	200
Weight decay	0.1
Adam betas	(0.9, 0.95)
Gradient clipping	1.0
Precision	BF16 when supported; otherwise FP16 on CUDA or FP32 on CPU
Evaluation interval	250 steps
Checkpoint interval	250 steps
Random seed	42

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "richarddzh/tiny-qwen3-30m-zh"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "问题：太阳是地球的什么？回答：太阳是"
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Reproducibility

The complete data preparation, tokenizer training, model configuration, pretraining, evaluation, and upload workflow is available in the training notebook.

Limitations

This is a small experimental base model. Its factual knowledge, reasoning ability, instruction-following behavior, and context handling are limited. Generated text may be repetitive, inconsistent, factually incorrect, or reflect biases and artifacts in the training corpus. Do not use it for high-stakes decisions or assume its outputs are accurate.