Yunmo-Next-1B-Instruct

Yunmo-Next-1B-Instruct is the instruction-tuned version of Yunmo-Next-1B-Base, a 1.008B-parameter Traditional-Chinese-first hybrid KDA/GQA model pretrained from scratch on 19.76B tokens. One epoch of supervised fine-tuning (1.76B packed tokens, 816M supervised target tokens, 16K context) taught it a chat protocol with an optional reasoning channel, four reasoning-effort levels and tool calls.

This is a research checkpoint, not a capable assistant. Its base model saw roughly a thousandth of the tokens of comparable open models and is at chance on knowledge multiple choice; fine-tuning taught the format of answering and reasoning far better than their content. Read Evaluation and Limitations before using it.

• Weights, tokenizer and inference code: Apache-2.0. Training data is not redistributed.
• Technical reports: YuhuanStudio/Yunmo-Next-1B (the instruction-tuning report covers this model).
• Training code: YuhuanStudio/Yunmo-Next.

At a glance

Base model	Yunmo-Next-1B-Base (19.761B pretraining tokens)
Parameters	1,008,058,231
Architecture	60 layers, GQA → KDA → KDA → KDA cells, low-rank Attention Residuals (see the base model card)
Fine-tuning	26,802 updates × 65,536 tokens = 1.757B packed tokens; 815,984,577 supervised target tokens; 1,861,379 conversations from 35 public sources
Context	trained at 16,384 tokens
Precision	trained in BF16; model.safetensors is BF16 (bit-exact training weights), model.fp32.safetensors the exact FP32 upcast
Hardware	one RTX 4070 Ti 12 GB, about 5.3 days

Usage

transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "yuhuanstudio/Yunmo-Next-1B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.float32
).cuda().eval()

ids = tokenizer("臺灣最高的山是", return_tensors="pt").input_ids.cuda()
print(tokenizer.decode(model.generate(ids, max_new_tokens=64)[0], skip_special_tokens=True))

The remote code wraps the package's own model: forward returns logits (and loss with labels) for right-padded batches, so perplexity and log-likelihood tools work. generate is a batch-1 text continuation loop (greedy or temperature/top-p) with CUDA-graph decoding; it does not apply the chat protocol. Requires transformers > 4.53.3, PyTorch with CUDA and flash-linear-attention.

Chat (package CLI)

The chat protocol, reasoning channel and grammar-constrained decoding are implemented by the bundled runtime:

git lfs install && git clone https://huggingface.co/yuhuanstudio/Yunmo-Next-1B-Instruct
cd Yunmo-Next-1B-Instruct
python bin/yunmo-next --prompt "請用三句話介紹玉山。" --system "推理強度：無" \
  --max-new-tokens 512 --max-message-tokens 480 --dtype float32 --static-decode

• Reasoning effort is set by a line in the system message, exactly as in training: 推理強度：無 / 低 / 中 / 高 or Reasoning: none / low / medium / high. Use 無 / none. On every benchmark below except GSM8K, answering directly scores highest; more reasoning makes results worse (see Reasoning effort). Without an effort line the model also answers directly.
• --reasoning auto lets the model open a reasoning message; --include-analysis prints it.
• --static-decode replays each decoding step as one CUDA graph: about 66 tokens/s instead of 9 on an RTX 4070 Ti in FP32, with logits within 3×10⁻⁵ of regular decoding.
• In Python, YunmoNextInference.chat_batch decodes many conversations together through one batched CUDA graph (continuous batching); its outputs are identical to one-at-a-time chat, and throughput was 485 tokens/s at 32 rows on the same GPU.
• Use FP32 (--dtype float32) when exact outputs matter; see the base model card for BF16 noise.

Prompt format

<|bos|><|start|>system<|channel|>analysis<|message|>推理強度：無<|end|>
<|start|>user<|channel|>analysis<|message|>{question}<|end|>
<|start|>assistant<|channel|>final<|message|>{answer}<|return|>

(Line breaks added for display.) With reasoning, the assistant first writes <|start|>assistant<|channel|>analysis<|message|>{thinking}<|end|> and then the final message. The runtime builds and parses this format; the tokenizer protocol definition is tokenizer/protocol_abi_v2.json.

Fine-tuning

Data

1,861,379 conversations from 35 public datasets, 815,984,577 supervised target tokens. Shares of supervised tokens:

Share	Source	License (as stated by the source)
20.7%	a-m-team/AM-Thinking-v1-Distilled	research use only, no commercial use
12.3%	openbmb/UltraData-SFT-2605	Apache-2.0
11.3%	guoxz/GX-Chinese-Instruct-1M-2508	Apache-2.0
8.1%	opencsg/smoltalk-chinese	Apache-2.0
8.1%	allenai/tulu-3-sft-mixture	ODC-BY (component licenses apply)
7.5%	Jackrong/Chinese-Qwen3-235B-Thinking-2507-Distill-100k	Apache-2.0
6.8%	yuyijiong/context_qa_sum_qwen3_synthetic	Apache-2.0
6.0%	a-m-team/AM-DeepSeek-R1-0528-Distilled	research use only, no commercial use
4.2%	Congliu/Chinese-DeepSeek-R1-Distill-data-110k	Apache-2.0
3.5%	Agent-Ark/Toucan-1.5M	Apache-2.0
2.2%	HuggingFaceTB/smoltalk	Apache-2.0 (component licenses apply)
2.1%	nvidia/Nemotron-SFT-Competitive-Programming-v2	CC BY 4.0 / ODC-BY / MIT
1.8%	lianghsun/tw-instruct-500k	CC BY-NC-SA 4.0
1.3%	Crownelius/Complete-FABLE.5-traces-2M	MIT (outputs of Anthropic Claude models)
1.2%	twinkle-ai/tw-reasoning-instruct-50k	MIT
1.1%	bigcode/self-oss-instruct-sc2-exec-filter-50k	ODC-BY
0.7%	Jackrong/glm-4.7-multiturn-CoT	not stated (GLM-4.7 outputs)
1.2%	18 smaller sources (LongCite-45k, no_robots [CC BY-NC 4.0], ToolACE, BookSum, hermes_reasoning_tool_use, CoCoNot, OpenScience-Chinese-Reasoning, Nemotron IF/Math, When2Call, tw-function-call-reasoning, Chinese SQuAD 2.0, CMRC 2018, OpenThoughts3, QASPER, identity examples)	various

Simplified-Chinese sources were converted to Traditional Chinese (OpenCC s2twp) with code and machine text protected. Rows that carry reasoning are tagged with an effort level from the length of their thinking (low < 1,500 characters, medium < 5,000, high above); 40% of the rows without reasoning carry an explicit "none" tag and the rest none, so an untagged prompt means "answer directly". Pretraining text was replayed in one micro-batch out of every 32 to limit forgetting.

Recipe

Initialization	Yunmo-Next-1B-Base (19.761B checkpoint)
Objective	cross-entropy on assistant tokens only (headers and protocol tokens of assistant turns included)
Batch	65,536 packed tokens per update, rows of 16,384 tokens, document-isolated packing
Updates	26,802 (one pass over the data)
Optimizer	same role split as pretraining (Muon variants for matrices, AdamW for the rest), learning rate 3×10⁻⁵ for both, weight decay 0.1, gradient clipping 1.0
Schedule	3% linear warmup, cosine decay to 10%
Validation	response-only bits per byte on 388 held-out packed rows: 0.5010 → 0.5008 → 0.5006 over the last 1,000 updates

Held-out pretraining BPB at 8K and 16K context rose by 0.76% and 0.18% against the base model at the 14,500-update checkpoint; code strata rose by 1.6–2.5%.

Evaluation

Chat-mode benchmarks: every model answers through its own chat format, decoding is greedy, and the final answer is scored. IFEval with lm-evaluation-harness's checker (prompt-level strict accuracy); GSM8K by the boxed / #### / last number; HumanEval pass@1 by executing the completed function; MMLU (20 questions per subject, 1,140) and TMMLU+ (15 per subject, 66 subjects, 990) by the option letter the reply commits to. Yunmo-Next runs in FP32 through its runtime; the comparison models run in BF16 with vLLM. Item lists, prompts, scoring code and every reply are in the report repository.

Answering directly (all items)

Model	Setting	IFEval	GSM8K	HumanEval	MMLU	TMMLU+
Yunmo-Next 1B	effort none	44.7	13.9	15.9	29.0	23.0
Qwen3.5 0.8B	direct	51.9	55.6	31.7	53.5	32.2
LFM2.5 1.2B	direct	80.8	76.1	51.8	51.6	30.4
MiniCPM5 1B	direct	63.6	57.8	61.6	53.3	26.1
Qwen3.5 2B	direct	67.5	77.6	53.0	62.1	40.8
MiniCPM5 2B	direct	85.0	87.1	82.3	61.6	40.3

Yunmo-Next is last on every task. The gap is smallest on IFEval (7 points behind Qwen3.5-0.8B) and TMMLU+ (3 points behind MiniCPM5-1B) and largest on GSM8K (42–73 points), HumanEval (16–66) and MMLU (23–33), consistent with a base model pretrained on about a thousandth of the comparison models' tokens.

"Direct" means reasoning effort none for Yunmo-Next and thinking disabled for the comparison models.

Reasoning effort

On every fourth item (1,040 items), every effort level of Yunmo-Next against the comparison models with thinking enabled (8,192-token budget, the same as Yunmo-Next high):

Model	Setting	IFEval	GSM8K	HumanEval	MMLU	TMMLU+
Yunmo-Next 1B	effort none	49.3	17.3	12.2	26.7	23.8
Yunmo-Next 1B	effort low	17.6	18.2	9.8	29.8	22.2
Yunmo-Next 1B	effort medium	19.1	15.8	9.8	24.9	12.5
Yunmo-Next 1B	effort high	11.0	13.0	2.4	19.6	10.5
Qwen3.5 0.8B	direct	53.7	54.8	39.0	52.6	30.6
Qwen3.5 0.8B	thinking, greedy	0.7	0.0	19.5	9.1	2.0
Qwen3.5 0.8B	thinking, recommended sampling¹	53.7	12.1	31.7	47.0	35.1
LFM2.5 1.2B	direct	83.1	76.4	53.7	53.0	31.0
MiniCPM5 1B	direct	69.9	57.6	68.3	52.3	24.6
MiniCPM5 1B	thinking, greedy	75.7	75.8	68.3	61.4	22.2
Qwen3.5 2B	direct	72.8	78.5	61.0	63.9	40.7
Qwen3.5 2B	thinking, greedy	8.1	19.4	48.8	38.2	24.2
Qwen3.5 2B	thinking, recommended sampling¹	74.3	62.7	58.5	65.6	52.8
MiniCPM5 2B	direct	86.8	87.6	82.9	61.1	38.7
MiniCPM5 2B	thinking, greedy	83.1	80.9	68.3	80.0	51.2

¹ Qwen3.5 only, with Qwen's recommended text-thinking sampling (temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5; one sample per item). With greedy decoding, Qwen3.5's thinking does not finish within 8,192 tokens on most items (empty answers on 66–100% of items per task for 0.8B, 37–92% for 2B), so its greedy rows understate it; Qwen advises against greedy decoding in thinking mode. Even with the recommended sampling, Qwen3.5-0.8B exceeds the 8,192-token budget on 88% of GSM8K items (2B: 36%); Qwen's own evaluations allow much longer reasoning. Every other row is greedy.

Generation behavior of Yunmo-Next on the same items:

Effort	Budget (tokens)	Reasoning hit its cap	Empty final answer	Final answer ends in a repetition loop	Mean generated tokens
none	1,024	—	0.0%	5.0%	237
low	2,048	18.8%	11.6%	0.7%	833
medium	4,096	35.7%	22.3%	0.2%	1,906
high	8,192	60.2%	26.6%	0.0%	5,188

On all items, low beats none only on GSM8K, by 1.1 points (within noise); medium and high score lower than none on every task. The model imitates the structure of the distilled reasoning traces (option-by-option analysis, "let me double-check") without their content: arithmetic slips are copied through re-checks, and when unsure it often repeats one sentence until the budget runs out, which is why longer budgets produce more empty answers. Greedy decoding is not the main cause: on a 64-item sample at low, sampling (T = 0.7, top-p 0.9) reduced capped reasoning (17 → 6 items) and repetition (10 → 6) but answered fewer items correctly (19 → 13).

Weight-only quantization

Held-out bits per byte on the 16-stratum pretraining evaluation set at 8K context (8 rows per stratum), FP32 compute on quantize-dequantize weights, so the difference is quantization error only. Affine schemes follow MLX (w ≈ scale·q + bias per group of input channels). Group-64/32 schemes leave the 60 MLP down projections in full precision because their input widths (5,472 / 4,560 / 2,736 / 1,824) are not multiples of 64 or 32 for every layer; INT8 per-channel covers every linear layer. Norms, convolutions and router parameters are never quantized.

Scheme	Quantized parameters	Held-out BPB	vs FP32
FP32 (reference)	—	0.6716	—
INT8, symmetric per output channel	940M (93%)	0.6716	+0.01%
8-bit affine, group 64	716M (71%)	0.6716	+0.00%
4-bit affine, group 64	716M (71%)	0.6764	+0.72%
4-bit affine, group 32	828M (82%)	0.6756	+0.61%
8-bit affine, group 64, + tied embedding	783M (78%)	0.6716	+0.01%
4-bit affine, group 64, + tied embedding	783M (78%)	0.6875	+2.37%

8-bit weights are lossless within this measurement. 4-bit weights cost 0.6–0.7% BPB; quantizing the tied embedding (which is also the output projection) to 4 bits as well roughly triples the loss, so keep it at 8 bits or more.

Limitations

• Knowledge. Multiple-choice knowledge is near chance (MMLU 29%, TMMLU+ 23%), as in the base model. Answers about facts, law, medicine or finance are frequently wrong and confidently stated.
• Reasoning. GSM8K about 14%; HumanEval about 16%. Reasoning modes make most results worse.
• Degenerate generation. About 5% of direct answers (20% on IFEval) end in a repetition loop; with reasoning enabled, a quarter of answers at medium/high are empty because the reasoning never ends.
• No preference tuning or safety training. The model was not trained to refuse harmful requests.
• Long context. Fine-tuned at 16K; retrieval in the base model degrades beyond 8K.
• Tool calls. The protocol and 3.8% of the supervised tokens cover tool use, but tool calling has not been evaluated.

License

The weights, tokenizer and bundled inference code are released under Apache-2.0, as for the base model. The fine-tuning data is not distributed; each source remains under its own terms, listed in Data. Some sources state further conditions (the AM-Thinking-v1 and AM-DeepSeek-R1 distillations: research use only; tw-instruct-500k: CC BY-NC-SA 4.0; no_robots: CC BY-NC 4.0), and part of the data was generated by other models whose providers have their own terms of use (for example the Claude-generated FABLE.5 traces). If you plan commercial use, review those terms yourself; this card is not legal advice.