sinksilk/Yebb-Dense-0.8B-Base

Yebb Dense 0.8B Base

Yebb Dense 0.8B Base is an experimental Chinese and English text continuation model with 800,663,040 parameters, exported at 40,000,142,733 effective prediction tokens. The 40B label refers to training tokens, not parameter count. This is an intermediate pretrained base checkpoint without instruction tuning or preference alignment.

The release preserves the original FP32 weights in two Safetensors shards, the trained 64K tokenizer, and a portable PyTorch implementation with a Transformers interface. The training checkpoint, optimizer and RNG state remain separate from this inference release.

Architecture

Component	Configuration
Total parameters	800,663,040
Decoder layers	24
Hidden dimension	1,536
Attention	Grouped-query attention, 12 query heads and 4 KV heads
Head dimension	128
Positions	RoPE, theta 10,000
Feed-forward	SwiGLU in every layer, intermediate dimension 4,096
Normalization	RMSNorm, epsilon 1e-5
Vocabulary	64,000, custom byte-level BPE
Training sequence length	2,048
Embedding and output matrices	Untied
Checkpoint	Step 174,596

This is a dense model. It does not contain the routed experts or SiTU feed-forward layers used by Yebb MoE 2B.

Usage

CPU export validation uses PyTorch 2.10.0 and Transformers 5.20.2. The package lists minimum dependencies in requirements.txt; other versions have not been validated. Review the custom code before enabling trust_remote_code.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "sinksilk/Yebb-Dense-0.8B-Base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.float32
).eval()

prompt = "中国的首都是"
ids = [tokenizer.bos_token_id] + tokenizer.encode(prompt, add_special_tokens=False)
with torch.inference_mode():
    output = model.generate(
        input_ids=torch.tensor([ids]),
        max_new_tokens=32,
        do_sample=False,
        use_cache=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

Use plain text continuation prompts. This model is not a chat assistant. The portable implementation recomputes the prefix and does not implement a KV cache or Ascend training kernels. Contexts beyond 2,048 tokens have not been validated. FP32 batch-versus-single rounding is checked against the original reference implementation and recorded in export_validation.json.

Training data and token accounting

The following ledger counts actual loss-bearing prediction tokens at this checkpoint. It is not a count of semantically unique text or documents. The sampler records no explicit replay at this point, but that does not certify the absence of duplicated content across sources or rewritten material.

Source	Effective prediction tokens
Educational code	6,000,026,798
English educational web	10,000,054,746
FineMath	3,200,039,091
English multi-style rewrites	1,999,991,544
English QA	0
Chinese multi-style rewrites	6,000,012,735
MiniMind	799,997,658
Chinese educational web	12,000,020,161
Total	40,000,142,733

Sources are MiniMind, Ultra-FineWeb-L3, Chinese FineWeb-Edu V2.1, FineWeb-Edu, Common Pile educational code, and FineMath. The tokenizer was trained separately on approximately 10GB of text. Exact accounting and hashes are in training_token_ledger.json and training_provenance.json.

Evaluation and limitations

Existing CPU FP32 development checks for this exact checkpoint produced:

Internal development check	Result
Fixed 1,024-question multiple-choice panel	334/1,024, 32.6172%
ARC Challenge subset	105/384, 27.3438%
CEval subset	91/256, 35.5469%
HellaSwag subset	138/384, 35.9375%
Separate fixed 352-question panel	111/352, 31.5341%
Six elementary Python function completions	0/6
Fixed 128-window recipe-weighted token NLL	2.60659386

Multiple-choice results use full-candidate scoring by mean token log probability. These development panels were reused during model development; they are not official leaderboard scores or independent generalization estimates. The strict code test uses six elementary greedy completions and is not HumanEval or MBPP. The 40B checkpoint's primary-panel gain over the earlier 20B checkpoint was not statistically reliable (paired McNemar p=0.4397).

The model can produce repetitive text, incorrect statements, empty function bodies, invalid indentation and incorrect algorithms. It is released for research and reproducibility, not as a validated general assistant. evaluation_summary.json records existing results; export fidelity checks are separate from capability evaluation.

License and source conditions

Model weights and the release implementation use the MIT license in LICENSE, following the Yebb release convention. Original training materials retain their source-specific terms; this repository does not redistribute the raw corpus or remove those conditions. In particular, the Chinese FineWeb-Edu V2.1 card lists OpenCSG community-license and commercial-use conditions alongside Apache metadata.

Maintainer: sinksilk.