RWKV7-G1j-10B-A1B

A RWKV7 dense-to-MoE initialization with 80 routed experts, one shared expert, and top-4 routing in every layer.

This checkpoint starts from RWKV7-G1j-2.9B-20260831, preserves its complete 32-layer recurrent backbone, and expands its channel-mixing FFNs into a sparse expert architecture. It is intended as a starting point for continued pretraining, MoE recovery training, and architecture experiments.

This is an initialized checkpoint. It has not undergone language-model recovery training after conversion. Its experts and router have not yet learned specialized roles through continued training. The source model is a Base LM; this release does not establish instruction-following or benchmark performance.

繁體中文摘要： 本模型由 RWKV7-G1j 2.9B 初始化，保留完整主體，每層採用 80 個路由專家、1 個共享專家及 top-4 路由。每個 token 使用 4 個路由專家與始終啟用的共享專家。總參數為 9.770B；啟用參數為 1.465B（不含 embedding／輸出頭）或 1.801B（包含）。已完成權重轉換、輸出尺度校準與運算驗證，尚未進行語言模型恢復訓練。

Architecture

Component	Configuration
Source checkpoint	RWKV7-G1j-2.9B-20260831
Layers	32
Hidden size	2,560
WKV heads / head size	40 / 64
Vocabulary size	65,536
Original FFN intermediate size	10,240
Routed experts per layer	80
Routed expert intermediate size	640
Routed experts selected per token, per layer	4
Shared experts per layer	1, always active
Shared expert intermediate size	640
Expert function	value(ReLU(key(x))²)
Routing	FP32 sigmoid scores, global top-4, normalized selected weights
Shared contribution	Added directly to the routed mixture
Backbone / expert weights	BF16
Router weights / recurrent WKV state	FP32
Default WKV backend	chunked
Optional WKV backends	eager, fla
Default MoE backend	torch

The shared expert is additional to top-4: each token activates four routed branches plus one shared branch. Routing decisions can differ between tokens and layers. The FFN token shift and x_k remain shared across branches, so this expansion does not create a separate recurrent state for every expert.

For a channel-mixing input x, the layer computes:

output = shared(x) + sum(weight[e] * expert[e](x) for e in selected_top4)

The selected sigmoid scores are normalized to sum to one. The shared branch is outside this normalization and is not selected by the router.

Exact parameter accounting

The repository name is a short size label. The exact counts for this shared-1 / top-4 release are:

Count	Parameters
Total	9,770,032,640
Active per token, excluding embedding and output head	1,465,310,720
Active per token, including embedding and output head	1,800,855,040
All routed expert weights	8,388,608,000
All shared expert weights	104,857,600
All router weights	6,553,600
Preserved non-FFN-projection backbone	1,270,013,440

Both active counts include every layer's router and shared expert. Buffers are excluded. The inclusive convention counts the full embedding and output-head matrices; this is parameter accounting, not a claim that every embedding row is read for each token or a FLOPs estimate.

The BF16/FP32 checkpoint contains approximately 19.55 GB of tensor data. Sparse activation reduces the expert computation used by a token; all expert weights still need to be stored. Training memory also depends on gradients, optimizer states, activations, and sharding.

Initialization method

Complete FFN coverage

Each original 10,240-channel FFN is divided into 16 contiguous groups of 640 channels. Every group is copied five times, producing 80 independently stored routed experts. Key rows and the corresponding value columns are moved together. Every original FFN channel therefore occurs exactly five times in the routed bank, with no remainder or dropped channels.

The embedding, output head, normalization, time-mixing weights, and FFN token-shift parameters are copied from the source. Router rows are initialized independently in FP32 with Gaussian standard deviation 0.02 and seed 20260912 + layer_index; expert-bias buffers start at zero. Copied experts initially share source functions, and independent router initialization does not by itself establish useful specialization.

Shared branch and joint scale calibration

For each layer, one of the 16 source groups is also used to initialize the 640-channel shared expert. The group and two strictly positive output scales are selected jointly to minimize reconstruction error on calibration fit inputs from the original dense teacher:

teacher_ffn_output ≈ routed_scale * routed_top4_output
                   + shared_scale * shared_group_output

The shared source group is still present in the routed bank. This overlap is explicit; joint fitting adjusts the two contributions. The construction does not claim disjoint source features or already-learned common knowledge.

Scales are folded into the BF16 value projections. Calibration uses 673 tokens from eight fit documents, followed by 234 tokens from three separate check documents. These are short English, Chinese, and Python texts created for initialization diagnostics. They are not a benchmark or a continued-pretraining corpus.

• Routed scale range: 0.1877–4.9224.
• Shared scale range: 0.7767–2.6273.
• Median per-layer relative squared reconstruction error on check documents, after actual BF16 scale folding: 0.6086.

This error is measured using teacher hidden states. It does not measure accumulated error in the complete MoE model, perplexity, generation quality, or recovery-training progress. The calibration documents should be excluded from independent quality evaluation.

Exact source-channel mappings, chosen shared groups, scales, documents, and provenance are recorded in initialization_map.json, initialization_calibration.json, and conversion_report.json.

initialization_sources.zip contains the conversion, calibration, verification, and smoke-check scripts, their local model-code dependencies, and unit checks. See REPRODUCE.md inside the archive for the command sequence. release_manifest.json records the packaged source hashes and validation environment.

Loading and raw text continuation

The model uses custom Transformers modules included in this repository. Load it with trust_remote_code=True. The example below uses one CUDA GPU and raw Base-LM text continuation.

The local validation environment used Python 3.12, PyTorch 2.10.0+cu130, Transformers 5.8.0, Accelerate 1.11.0, and Safetensors 0.8.0. After installing a suitable CUDA PyTorch build:

pip install "transformers==5.8.0" "accelerate==1.11.0" "safetensors==0.8.0"

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "win10/RWKV7-G1j-10B-A1B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map={"": "cuda:0"},
).eval()

prompt = "A computer program can organize incoming tasks in a queue. When a new task arrives, "
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=1.0,
        top_p=0.5,
        top_k=0,
    )
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

top_k=0 in this generation example is a vocabulary-sampling setting. It does not change the model's MoE top-4, which is configured by num_experts_per_tok=4.

The inherited tokenizer and chat template are included for compatibility with the source checkpoint. Raw continuation is the appropriate basic evaluation format for this initialization; a chat template does not establish post-conversion instruction tuning.

FLA acceleration

The included model implementation retains Flash Linear Attention's RWKV7 kernel. The local FLA forward check used flash-linear-attention==0.5.2 with Triton 3.6.0.

pip install "flash-linear-attention==0.5.2"

After loading the model on CUDA:

model.config.wkv_implementation = "fla"

FLA requires FP16/BF16 activations and FP32 WKV state; this checkpoint uses BF16 weights and wkv_state_dtype="float32". It accelerates the RWKV WKV computation. Expert execution is configured separately through moe_backend, whose default here is torch.

FLA is Flash Linear Attention. It is not enabled through Transformers' attn_implementation="flash_attention_2" setting.

Validation performed

Check	Result / scope
Unit checks	Five checks passed: partition reconstruction, replica independence, router reproducibility, branch composition and gradients, joint scale fitting
Complete weight readback	1,190 tensors; all 32 layers, 2,560 routed experts and 32 shared experts verified
Source mapping	Every source FFN channel appears five times in the routed bank; shared mappings verified separately
Backbone preservation	All non-FFN-projection source tensors match exactly
Integrity	Source and output shard/asset SHA256 checks passed
Transformers load	No missing, unexpected, or mismatched keys or loading errors
Default chunked forward	Finite logits on a short smoke input
eager cache continuation	Full-sequence versus split-sequence logits matched exactly on the smoke input
Top-4 routing	Selected experts and weights matched the direct routing formula in all 32 layers
FLA	Full-checkpoint CUDA forward produced finite logits on the smoke input
Generation API	Short generate() smoke check passed; no generation-quality score assigned

These checks establish conversion and execution integrity in the recorded environment. They do not establish language-model recovery, downstream benchmark performance, long-context accuracy, FLA numerical equivalence to other backends, or training throughput. No speedup ratio is claimed for this checkpoint.

Continuing training

This checkpoint is intended for recovery training before assessing its language capability. Useful measurements include held-out next-token loss, teacher divergence, raw continuations, expert utilization, and replica specialization. Evaluate language behavior and routing correctness separately.

Changing top-k, shared width, routing normalization, or branch scaling changes the initialized computation. Revisit the joint scale calibration when making those changes. Record any future training corpus, token budget, optimizer setup, and evaluation results in the model card for the resulting trained release.

License and attribution

Distributed under Apache-2.0; see the included LICENSE and NOTICE files. The checkpoint derives from the official RWKV7-G1j-2.9B release. The MoE implementation combines RWKV7 ReLU² channel mixing with Ling/Bailing-style sigmoid routing and a directly added shared expert.

Related architecture and initialization references:

• RWKV-7
• DeepSeekMoE: shared-expert isolation and fine-grained experts
• Upcycling Large Language Models into Mixture of Experts

The cited papers provide background; their benchmark results are not results for this checkpoint.