zh-news-gpt-20m

A small-scale causal language model (GPT) trained from scratch on Chinese web text and news-like corpora.

Model Overview

Attribute	Value
Architecture	GPT ( decoder-only transformer )
Parameters	~20.88M
Vocabulary size	8,000 tokens
Context length	256 tokens
Tokenizer	SentencePiece (BPE)

Architecture Configuration

{
  "vocab_size": 8000,
  "n_positions": 256,
  "n_ctx": 256,
  "n_embd": 384,
  "n_layer": 10,
  "n_head": 6,
  "block_size": 256,
  "dropout": 0.1,
  "bias": false
}

Training Details

• Dataset: Morton-Li/ChineseWebText2.0-HighQuality (Chinese web corpus)
• Preprocessing: 150,000 cleaned texts, filtered to 50–5,000 characters
• Total tokens: ~73.5M
• Optimizer: AdamW (lr=3e-4, weight_decay=0.1)
• Scheduler: Cosine decay with 1,000-step linear warmup
• Batch size: 8
• Sequence length: 256
• Hardware: Apple Mac M4 with MPS (Metal Performance Shaders)
• Final checkpoint: 93,000 steps

Why 93k Steps?

Training was continued until 100,000 steps, but we observed signs of overfitting and a slight degradation in generation quality after 93,000 steps. For that reason, we selected the 93,000-step checkpoint as the stable final release of this model.

Metrics

Metric	Value
Training loss (approx.)	4.45
Validation loss (approx.)	4.85
Perplexity	141.83

Generation Examples

Generated with temperature=0.8, top_k=50, top_p=0.95.

Prompt: 今天天气

今天天气偏暖,是以"低碳化"为主题的城市.从温州市雨水局的气象分析,由于风水温度的下降,风水的强度和风水的强度明显,天气的下降,风水天气的低温和风水的强度下降,是为多地少数地区雨水的最低气温. 今天天气偏暖,气温低迷,天气的天气,天气也让天气的雪雪天气和冷

Prompt: 人工智能

人工智能.在其中,新能源汽车新能源汽车领域,新能源汽车行业的规模持续提升,新能源汽车产业链成为新能源汽车产业链和产业链,同时也成为新能源汽车领域,新能源汽车企业与新能源汽车品牌相匹配.新能源汽车行业在新能源汽车行业中发展具有重大贡献,新能源汽车产业集群也成为新能源汽车产业链产业链的重要组成部分.新能源汽车公司在新能源汽车领域发展具有重要地位,新能源汽车在新能源汽车领域仍将具有较大

Prompt: 中国经济

中国经济发展有限公司、中国交通银行股份有限公司上海分行、中国银行股份有限公司上海分行、中国银行股份有限公司上海分行、中国银行股份有限公司上海分行、中国银行股份有限公司上海分行、中国银行股份有限公司上海分行、上海分行、上海银行股份有限公司上海分行、中国银行股份有限公司上海分行、中国银行股份有限公司上海分行、中国银行股份有限公司上海分行、上海分行、中国银行股份有限公司上海分行、中国银行股份有限公司上海分行、中国银行股份有限公司上海

Limitations

• Small model (20M parameters) with limited semantic coherence.
• Trained on generic web text without domain-specific curation.
• Can produce repetitive or hallucinated content.
• Not aligned with RLHF or instruction tuning.
• Best suited for Mandarin Chinese; other languages are tokenized sub-optimally.

How to Use

import sentencepiece as spm
import torch
from model import GPT

checkpoint = torch.load("pytorch_model.bin", map_location="cpu")
model = GPT(checkpoint["config"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

sp = spm.SentencePieceProcessor(model_file="tokenizer.model")
input_ids = sp.encode("今天天气", out_type=int)
idx = torch.tensor([input_ids], dtype=torch.long)

output = model.generate(idx, max_new_tokens=50, temperature=0.8, top_k=50)
print(sp.decode(output[0].tolist()))

License

Educational and research use. Please verify the terms of the original dataset before any commercial use.