Qwen3.8-Flash-Next-1.9B-A1.8B-small

This is a small version of Qwen/Qwen3.8-Flash-Next created for testing and development. It keeps every hyperparameter of the base model identical (text/vision hidden sizes, head counts and dims, gated-delta-net linear attention dims, QSA indexer parameters, MoE topology, hyper-connection parameters, vocabulary, untied embeddings, rope settings) and only reduces the number of text/vision layers, the per-layer type lists, and the number of routed experts per MoE layer.

Model Details

• Base Model: Qwen/Qwen3.8-Flash-Next
• Architecture: Qwen4ExpForConditionalGeneration (qwen4_exp), multimodal (vision + text)
• Total Parameters: 1.88B
• Activated Parameters: 1.77B (topk 10 of 12 routed experts per MoE layer, text path; vision tower excluded)

Configuration Changes

Only the allowed change classes were applied; everything else is identical to the base:

Parameter	Base	Small
text_config.num_hidden_layers	48	4
text_config.layer_types	[linear x3, qwen_sparse_attention] x 12 (3:1 hybrid pattern)	[linear_attention, linear_attention, linear_attention, qwen_sparse_attention] (exactly one pattern period)
text_config.num_experts	512	12
text_config.ple_layer_ids	[2]	[] (see PLE note below)
vision_config.depth	27	2

The 4 text layers are exactly one period of the base's 3:1 hybrid pattern, covering both distinct attention layer types of the base (gated-delta-net linear_attention and qwen_sparse_attention with its n-gram QSA indexer); all base text layers are MoE (single MLP type). 12 experts = topk (10) + 2 so that routing is actually exercised; num_experts_per_tok (topk 10) is unchanged. All other values are kept from the base, including hidden_size 2560, moe_intermediate_size/shared_expert_intermediate_size 640, num_attention_heads 24 / num_key_value_heads 2 / head_dim 256 with q/k norms, mrope (section [11,11,10], theta 1e7, interleaved, partial rotary 0.25), linear attention dims (linear_num_key_heads 16, linear_num_value_heads 48, head dims 128, conv kernel 4), QSA indexer parameters (indexer_budget 2048, indexer_compress_ratio 4, indexer_n_heads 4, indexer_head_dim 128, indexer_kv_heads 1), hyper-connections (hc_count 4, hc_lowrank 320), vocab_size 248320 with untied embeddings, and the full vision tower configuration except depth.

Checkpoint Structure

Single-file bf16 model.safetensors (134 tensors) using the original base checkpoint layout: model.language_model.layers.N.* with grouped 3D expert tensors mlp.experts.{gate_up,down}_proj ([E, 2*I, H] / [E, H, I]), per-layer mlp.gate.weight + shared expert + shared_expert_gate, linear_attn.* (in_proj_qkv/z/b/a, conv1d, A_log, dt_bias, norm, out_proj) on linear layers, self_attn.* + self_attn.indexer.* on qwen_sparse layers, per-layer attn_hyper_connection/mlp_hyper_connection + top-level hyper_connection_mixer, model.visual.* vision tower, and untied lm_head.weight. Tensor-name patterns match the base model.safetensors.index.json exactly, modulo layer and expert counts (validated programmatically: 59/59 normalized patterns, 0 missing, 0 extra; per-layer key sets identical to the analogous base layers 0 / 3; all non-expert tensor shapes identical to the base, expert tensors differ only in the leading expert dimension 512 -> 12).

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained("inference-optimization/Qwen3.8-Flash-Next-1.9B-A1.8B-small", device_map="auto")
processor = AutoProcessor.from_pretrained("inference-optimization/Qwen3.8-Flash-Next-1.9B-A1.8B-small")

input_ids = processor.tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(processor.tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

1. Loaded the base AutoConfig and applied only the allowed changes (4 text layers = one hybrid pattern period, 12 experts, vision depth 2, PLE dropped — see Notes).
2. Built the model with skip_weights_download (bf16), re-initialized weights (init_weights() + NaN fixup), reset model._weight_conversions = [] (the qwen4_exp VLM model type has no conversion-registry entry; the base checkpoint keys already match the module names and a real base load records no conversions — without the reset the save fallback would wrongly split the grouped expert tensors), and saved with AutoProcessor.
3. Verified the saved tensor structure and shapes against the base model.safetensors.index.json (and shard headers fetched via HTTP range) immediately after the first save.
4. Fine-tuned text-only on the skill's copypasta dataset in bf16 through the multimodal class (lr 1e-4, batch 2, early-stopped after ~170 steps / 185s) to target perplexity.
5. Re-validated the fine-tuned checkpoint structure and model quality, including a multimodal image+text forward.

Validation

• Perplexity on the skill's validation text: 1.00007 (target <= 10)
• Greedy generation from "According to all known laws": produces the trained copypasta continuation correctly.
• Multimodal image+text forward through the processor and 2-layer vision tower: finite logits ([1, 254, 248320]).

Notes

• PLE (deviation from the base, documented): the base attaches a PLE ("hashed n-gram feature injection") module to one linear-attention layer (ple_layer_ids: [2], one-indexed). Its hashed n-gram embedding table alone is ~51.2B parameters (16 hash heads x ~20M-row prime vocab x 160 dims, stored as 128 shards of [2500012, 160] bf16 = ~95 GiB) — 28% of the base's ~180B parameters. Its size is fixed by base hyperparameters (ngram_vocab_size_base 20M, ngram_size 3, heads_per_ngram 8, ple_embed_dim 2560) which must remain identical, so it cannot be shrunk. Including it would produce a ~53B "small" model that does not fit on a single 80GB GPU and cannot be fine-tuned here, so ple_layer_ids was trimmed to [] (a per-layer-list change consistent with the reduced layer count). PLE is an auxiliary per-layer feature-injection module, not an attention or MLP/MoE layer type, so the small model still covers every distinct attention and MLP layer type of the base. Consequence: number_of_conv_states (a runtime cache-slot count, no parameter impact) is stored as 3 in the saved json exactly as in the base, while transformers derives 1 at load time when PLE is absent; linear-attention parameter shapes are identical to the base either way.
• MTP: mtp_num_hidden_layers is kept at 1 from the base (mtp config block with hybrid: true, layer_types: ["full_attention"] unchanged); the checkpoint is backbone-only — the base's mtp.* tensors are intentionally absent, matching the convention of the other small models.
• Size: 1.88B total vs the 1.3-1.6B informal estimate — the untied 248320 x 2560 embeddings + lm_head alone are 1.27B (the base has tie_word_embeddings: false), and the remaining ~0.6B covers one full hybrid period (3 linear + 1 sparse layer, each with hyper-connections and a 12-expert MoE block) plus a 2-layer vision tower. Strictly far below the base (180B).
• No quantization_config in the base (unquantized bf16); nothing stripped. bf16 throughout.
• training_args.bin and optimizer files are not part of the upload.