shibatch/tinyqwen3next3m

Tiny Qwen3-Next 3M

This repository contains a tiny Qwen3NextForCausalLM Mixture-of-Experts language model trained from scratch on TinyStories.

The model has 2,945,914 parameters. It combines Gated DeltaNet linear attention, gated full attention, sparse top-2 MoE routing, and a shared expert in a checkpoint small enough for implementation testing and experimentation.

This is a synthetic tiny checkpoint. It is not an official Qwen model, does not contain weights from an original Qwen checkpoint, and should not be expected to match the quality or capabilities of production Qwen models.

Repository contents

• hf/: the final Hugging Face checkpoint and tokenizer
• example_generate.py: a minimal generation example
• eval_text_generation.json: generations from the final training evaluation
• artifact_metadata.json: training arguments, metrics, router usage, and the expanded configuration
• qwen3_next_config_dump.json: a standalone configuration dump

Optimizer checkpoints and the full training log are intentionally omitted from the distribution package.

Important architecture distinction

This checkpoint uses:

Qwen3NextConfig
Qwen3NextForCausalLM
model_type: qwen3_next

It does not use the separate regular Qwen3 MoE implementation:

Qwen3MoeConfig
Qwen3MoeForCausalLM
model_type: qwen3_moe

The model follows the Qwen3-Next hybrid layer pattern: three Gated DeltaNet linear-attention layers followed by one gated full-attention layer. Every layer contains a sparse MoE block.

Relationship to Qwen3-Next 80B

The architecture is a deliberately scaled-down member of the same Transformers architecture family as Qwen/Qwen3-Next-80B-A3B-Instruct:

3 x (Gated DeltaNet -> MoE)
1 x (Gated full attention -> MoE)

The original model repeats this four-layer pattern multiple times and uses hundreds of experts. Tiny Qwen3-Next 3M keeps one four-layer cycle and eight experts so that the important execution paths remain present at a much smaller scale.

This checkpoint is pretrained only. It has not been instruction-tuned and does not reproduce Qwen3-Next 80B behavior.

Model architecture

architecture: Qwen3NextForCausalLM
model_type: qwen3_next
parameter_count: 2,945,914

model_vocab_size: 1,024
tokenizer_size: 1,003
hidden_size: 216
intermediate_size: 540
num_hidden_layers: 4

layer_types:
  - linear_attention
  - linear_attention
  - linear_attention
  - full_attention

num_attention_heads: 8
num_key_value_heads: 1
head_dim: 56
partial_rotary_factor: 0.25
rope_theta: 10,000,000

linear_num_key_heads: 4
linear_key_head_dim: 54
linear_num_value_heads: 8
linear_value_head_dim: 54
linear_conv_kernel_dim: 4

num_experts: 8
num_experts_per_tok: 2
moe_intermediate_size: 54
shared_expert_intermediate_size: 54
norm_topk_prob: true
router_aux_loss_coef: 0.01

tie_word_embeddings: true
rms_norm_eps: 1.0e-6
max_position_embeddings: 1,024

The Gated DeltaNet dimensions retain the characteristic Qwen3-Next ratios:

• key width: 4 x 54 = 216, equal to the hidden size
• value width: 8 x 54 = 432, twice the hidden size

Full attention uses eight query heads and one key/value head. The query width is 8 x 56 = 448, while the key/value width is 56.

MoE configuration

Each of the four decoder layers contains eight routed experts and one shared expert. Every token selects two of the eight routed experts:

routed_experts: 8
top_k: 2
shared_expert: true
normalized_top_k_weights: true

Router auxiliary loss was active during training. All experts received traffic. Final aggregate expert fractions ranged from approximately 0.060 to 0.208 depending on layer and expert; no expert was unused.

Training data

The model was trained on the full TinyStories training corpus using an independent 1% validation split:

selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
validation_fraction: 0.01

training_blocks: 2,441,089
validation_blocks: 24,608
block_size: 256

Stories were joined into a continuous packed stream:

BOS + story 1 + EOS + BOS + story 2 + EOS + ...

The stream was split into fixed 256-token blocks without padding. Only the final incomplete block was discarded.

Tokenizer

The checkpoint uses a small custom byte-level BPE tokenizer. The base BPE vocabulary was trained with:

BPE()
ByteLevel(add_prefix_space=False)
base_vocab_size: 1,000
min_frequency: 2
normalizer: None

Special tokens were then added at fixed IDs:

<s>          -> 1000
</s>         -> 1001
<|im_start|> -> 1002

The tokenizer uses <s> as both BOS and padding, and </s> as EOS. The model configuration reserves 1,024 embedding rows while the tokenizer exposes 1,003 tokens.

Training setup

The checkpoint was trained from scratch in float32 on an NVIDIA GeForce GTX 1650:

dtype: float32
batch_size: 16
block_size: 256
training_steps: 152,568
epochs: 1.0
tokens_processed: 624,918,528

optimizer: AdamW
learning_rate: 3.0e-4
warmup_steps: 1,000
scheduler: warmup + cosine decay
minimum_learning_rate: 3.0e-5
weight_decay: 0.0
grad_clip: 1.0

training_time: 13h 48m 45s

Evaluation

The final checkpoint produced:

final_train_loss: 1.4145
validation_loss: 1.4320
validation_perplexity: 4.1871

Validation loss was computed over 16 batches, or 65,536 tokens, from the independent packed validation split. These numbers are compact checkpoint diagnostics, not general language-model benchmark results.

Example generation

With prompt Once upon, sampling seed 0 produced the following representative TinyStories-style opening:

Once upon a time, there was a little girl named Lily. She loved to play with
her toys and go on adventures. One day, she went outside and saw a rainbow in
the sky. It was so pretty that it made her feel even happier than before.

Sampling is stochastic. Other seeds may produce a boy, an adult, an animal, or another kind of TinyStories character. The model usually produces recognizable English, but semantic contradictions, unfinished sentences, invented words, and repetition remain possible at this size.

Usage

Install the requirements:

pip install -r requirements.txt

Run the included local example from the repository root:

python example_generate.py

To load the package from Hugging Face Hub, resolve the hf directory to a local path first. This also avoids a Transformers 5.14.1 local-subfolder issue in which generation configuration lookup may incorrectly fall back to the repository root:

import torch
from pathlib import Path
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shibatch/tinyqwen3next3m"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_dir = Path(snapshot_download(
    repo_id=repo,
    allow_patterns=["hf/*"],
)) / "hf"

tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    dtype=torch.float32,
    experts_implementation="batched_mm",
).to(device)
model.eval()

prompt = "Once upon"
input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(
        prompt,
        add_special_tokens=False,
    )],
    dtype=torch.long,
    device=device,
)

torch.manual_seed(0)
if device.type == "cuda":
    torch.cuda.manual_seed_all(0)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=40,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
        output_router_logits=False,
    )

print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

Loading requirements

The checkpoint requires a Transformers release containing Qwen3NextForCausalLM. It was trained and tested with:

transformers 5.14.1
torch 2.14.0.dev20260720+cu126

The optional flash-linear-attention and causal-conv1d packages are not required. Without them, Transformers uses its PyTorch Gated DeltaNet fallback, which is the path used to train and validate this checkpoint.

The examples explicitly select experts_implementation="batched_mm". The model's routed experts have an intermediate width of 54, while the default PyTorch grouped_mm CUDA path in some recent Torch/Transformers combinations requires expert matrix strides to be multiples of 16 bytes. Without the explicit compatible implementation, loading succeeds but the first forward pass can fail with:

RuntimeError: strides should be multiple of 16 bytes

batched_mm evaluates the same expert weights without that grouped-kernel layout restriction. experts_implementation="eager" is also a compatible, slower fallback.

Intended uses

This model is intended for:

• testing Qwen3NextConfig and Qwen3NextForCausalLM
• testing Gated DeltaNet fallback implementations
• testing the hybrid linear/full-attention layer pattern
• testing sparse top-2 MoE routing and shared experts
• checking router load-balancing loss
• exercising custom tokenizer loading
• testing generate(), save_pretrained(), and from_pretrained()
• compact inference-engine and architecture experiments

It is not intended for:

• instruction following or chat
• factual question answering
• high-quality long-form generation
• production deployment
• safety-critical use
• benchmark comparison with production Qwen models

Limitations

Known limitations include:

• only 2.95 million parameters
• small 1,003-token tokenizer
• English TinyStories-only pretraining
• weak factual knowledge and reasoning
• no instruction tuning or chat template
• occasional grammatical and semantic errors
• invented words and truncated sentences
• repetition and template-like stories
• no quality-equivalence claim with official Qwen models
• no current llama.cpp or GGUF inference support assumed for Qwen3-Next

Notes on GGUF

The checkpoint is distributed as a normal float32 Hugging Face Safetensors model. A useful GGUF build requires converter and runtime support for the full Qwen3-Next graph, including Gated DeltaNet, hybrid attention, routed experts, and the shared expert. Merely placing tensors in a GGUF container is not sufficient for compatible inference.

Citation

This is a synthetic tiny Qwen3-Next-compatible MoE checkpoint trained from scratch on TinyStories. It is intended for implementation validation, debugging, education, and small-scale architecture experiments.