sinksilk/Yebb-MoE-2B-Base

Yebb MoE 2B Base

Yebb MoE 2B Base is an experimental Chinese and English text continuation model with 1,962,164,864 parameters, released at 80,000,981,551 effective prediction tokens. It uses grouped-query attention with RoPE and a custom latent mixture of experts. This is a pretrained base checkpoint; it has not received instruction tuning or preference alignment.

The release contains the original FP32 model weights in Safetensors shards, its trained 64K tokenizer, and a portable PyTorch reference implementation with a Hugging Face interface. Optimizer state, random-number-generator state, and training credentials are not included.

Architecture

Component	Configuration
Total parameters	1,962,164,864
Layers and hidden dimension	12 layers, width 512
Attention	8 query heads, 2 key/value heads, head dimension 64
Positions	RoPE, theta 10,000
Feed-forward layers	First layer dense; remaining 11 layers latent MoE
Routed experts	128 experts per MoE layer; top-2 routing
Latent expert dimension	384
Expert intermediate dimension	1,152
Shared intermediate dimension	768
Activation	SiTU, beta 4 and linear beta 25
Tokenizer	Custom byte-level BPE, vocabulary 64,000
Training sequence length	2,048
Embedding and output weights	Untied

The routed expert banks account for 1,868,562,432 parameters. Counting every non-routed parameter plus 2/128 of the routed banks gives 122,798,720 parameters. This convention includes the full embedding and output matrices; it is not an exact count of operations or memory reads for one token. This checkpoint uses the width-512 architecture, rather than an earlier width-640 candidate.

Usage

Install the packages listed in requirements.txt. CPU export validation used PyTorch 2.10.0 and Transformers 5.20.2; older versions have not been verified. The repository supplies custom model code, so inspect it before enabling trust_remote_code.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "sinksilk/Yebb-MoE-2B-Base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype=torch.float32,
).eval()

prompt = "中国的首都是"
ids = [tokenizer.bos_token_id] + tokenizer.encode(prompt, add_special_tokens=False)
inputs = torch.tensor([ids], dtype=torch.long)
with torch.inference_mode():
    outputs = model.generate(
        input_ids=inputs,
        max_new_tokens=32,
        do_sample=False,
        use_cache=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Use plain text continuation prompts. A chat template does not turn this base model into an instruction-following assistant. The reference implementation recomputes the prefix and does not implement a key/value cache or optimized grouped expert kernels. It requires no Ascend-specific libraries, but CPU generation can be slow. Contexts beyond the training length have not been validated.

Training data and accounting

The checkpoint is training step 76,378. Its 80.00098B prediction-token clock includes 8.73845B explicit replay tokens from the Chinese education source. The clock does not represent 80B semantically unique tokens or documents.

The following counts are the actual effective prediction-token ledger for the checkpoint, including earlier stages and replay. They are not the final-stage sampling recipe.

Source	Effective prediction tokens
MiniMind	1,639,066,847
Educational code	9,547,651,891
English education web	23,147,635,641
FineMath	5,429,393,223
English multi-style rewrites	5,786,017,839
English QA	504,644,568
Chinese multi-style rewrites	4,360,380,110
Chinese education web, including replay	29,586,191,432

Sources are MiniMind, Ultra-FineWeb-L3, Chinese FineWeb-Edu V2.1, FineWeb-Edu, Common Pile educational code, and FineMath. The tokenizer was trained separately on approximately 10GB of text.

Evaluation and limitations

Existing CPU FP32 development checks for this exact checkpoint produced:

Internal development check	Result
Fixed 48-question multiple-choice panel	18/48, 37.5%
Six elementary Python function completions	0/6
Twelve elementary arithmetic completions	3/12

The panel uses zero-shot candidate scoring by mean token log probability. These small checks were used during development and are not official benchmark scores or an independent generalization certificate. The model has limited arithmetic and code reliability and can produce repetitive or incorrect continuations. It is released for research and reproducibility, not as a validated general assistant.

Checkpoint hashes and export details are in training_provenance.json; the existing development results are summarized in evaluation_summary.json. Export validation is recorded in export_validation.json.

License and source conditions

The model weights and release implementation are provided under the MIT license in LICENSE. Original training materials retain their source-specific licenses and conditions; no raw training corpus is redistributed here. In particular, the Chinese FineWeb-Edu V2.1 dataset card lists OpenCSG community-license and commercial-use conditions in addition to its Apache metadata. This release does not claim to remove those source-specific conditions.

Maintainer: sinksilk.