krampenschiesser/ling3-small-base

The is the base for finetuning and reinforcement learning. Not intended for general use as its just slower and larger than the original model for now.

ling 3 small base

The goal of this model is to create a small coding specific model that is fast and can fit on a single 16gb GPU in 8bit quantization. It is based of ling 3 tiny, however the experts per token are doubled and the expert count is now 1.5x greater.

The extra 64 experts on this one are a dummy and formed by using REAM to cut the model down to 64 experts and then append those experts back to the original model. The expert router scores have been averaged (i did not use the centroid) in order to get a somewhat random routing base that is different from the original model.

The active experts are now twice as big, but the router is NOT yet adjusted to it, so it only uses more tokens right now.

Why the extension of experts

Inspired by the jetbrains mellum paper I wanted to have more active parameters. The additional experts are there because my goal is to fit the 8bit quantized version in a single GPU and just don't want to waste the space that is available. Hopefully during the next training phases the additional experts and the router will catch up and provide additional value.

Future plans

1. extend to 256k context
2. finetune the heck out of it
3. offline DPO the heck out of it
4. GRPO the heck out of it
5. Train dspark and dflash, see which works best

The goal is to not create another python focused llm, so I will not use any of the many python datasets. Focus is:

• Rust
• Go
• CPP 11+
• Typescript
• Bash/sql/markup/yaml/json etc...

Using opencode as agent, keep testing and improving against:

• TerminalBench 2.1
• DeepSwe
• SWE bench multilingual
• IfEval
• IfBench
• Ruler2
• LongMemEval
• agentif
• LiveCodeBench V6

Not planned are other languages than english as the datasets are in english. Also this model will not focus on math, science, physics or general knowledge. However, I am curious how much of that will remain from the original model.

in the future additional knowledge will be added via lota adapters targeting the routing gate and experts to make the model modular and adjust to other use cases.

Speed on single 16GB GPU

This version with 11.85B_A2.6B On a single 5060TI in q8_0 (gguf) it runs at:

model	size	params	backend	ngl	fa	lm	test	t/s
bailingmoe3 11.85B.A2.6B Q8_0	11.29 GiB	11.37 B	CUDA	999	1	dio	pp512	5400.18 ± 300.11
bailingmoe3 11.85B.A2.6B Q8_0	11.29 GiB	11.37 B	CUDA	999	1	dio	tg128	163.04 ± 0.68

Compared to the original 7.89B_A1.3B:

model	size	params	backend	ngl	fa	lm	test	t/s
bailingmoe3 7.9B.A1.3B Q8_0	7.82 GiB	7.89 B	CUDA	999	1	dio	pp512	7584.71 ± 522.05
bailingmoe3 7.9B.A1.3B Q8_0	7.82 GiB	7.89 B	CUDA	999	1	dio	tg128	201.12 ± 1.10

During long inference the tg/s hold mostly true thanks to the linear attention layers:

0.30.146.350 I slot launch_slot_: id  2 | task 55 | processing task, is_child = 0
0.34.203.109 I slot print_timing: id  2 | task 55 | n_gen =    470, tg = 156.26 t/s, tg_3s = 156.59 t/s
0.37.206.575 I slot print_timing: id  2 | task 55 | n_gen =    939, tg = 156.21 t/s, tg_3s = 156.15 t/s
0.40.208.784 I slot print_timing: id  2 | task 55 | n_gen =   1408, tg = 156.21 t/s, tg_3s = 156.22 t/s
0.43.211.600 I slot print_timing: id  2 | task 55 | n_gen =   1877, tg = 156.20 t/s, tg_3s = 156.19 t/s
0.46.215.437 I slot print_timing: id  2 | task 55 | n_gen =   2345, tg = 156.12 t/s, tg_3s = 155.80 t/s
0.49.220.174 I slot print_timing: id  2 | task 55 | n_gen =   2813, tg = 156.06 t/s, tg_3s = 155.75 t/s
0.52.225.749 I slot print_timing: id  2 | task 55 | n_gen =   3280, tg = 155.96 t/s, tg_3s = 155.38 t/s
0.55.229.970 I slot print_timing: id  2 | task 55 | n_gen =   3746, tg = 155.86 t/s, tg_3s = 155.12 t/s
0.58.232.839 I slot print_timing: id  2 | task 55 | n_gen =   4211, tg = 155.75 t/s, tg_3s = 154.85 t/s
1.01.236.807 I slot print_timing: id  2 | task 55 | n_gen =   4677, tg = 155.68 t/s, tg_3s = 155.13 t/s
1.04.240.633 I slot print_timing: id  2 | task 55 | n_gen =   5141, tg = 155.57 t/s, tg_3s = 154.47 t/s
1.07.246.803 I slot print_timing: id  2 | task 55 | n_gen =   5605, tg = 155.47 t/s, tg_3s = 154.35 t/s
1.10.251.961 I slot print_timing: id  2 | task 55 | n_gen =   6068, tg = 155.36 t/s, tg_3s = 154.07 t/s
1.13.253.485 I slot print_timing: id  2 | task 55 | n_gen =   6531, tg = 155.28 t/s, tg_3s = 154.25 t/s
1.16.255.222 I slot print_timing: id  2 | task 55 | n_gen =   6994, tg = 155.22 t/s, tg_3s = 154.24 t/s
1.19.259.733 I slot print_timing: id  2 | task 55 | n_gen =   7456, tg = 155.12 t/s, tg_3s = 153.77 t/s
1.22.265.407 I slot print_timing: id  2 | task 55 | n_gen =   7918, tg = 155.04 t/s, tg_3s = 153.71 t/s
1.25.266.208 I slot print_timing: id  2 | task 55 | n_gen =   8379, tg = 154.96 t/s, tg_3s = 153.63 t/s
1.28.272.677 I slot print_timing: id  2 | task 55 | n_gen =   8839, tg = 154.86 t/s, tg_3s = 153.00 t/s
1.31.276.368 I slot print_timing: id  2 | task 55 | n_gen =   9300, tg = 154.79 t/s, tg_3s = 153.48 t/s
1.34.280.337 I slot print_timing: id  2 | task 55 | n_gen =   9758, tg = 154.68 t/s, tg_3s = 152.47 t/s
1.37.281.875 I slot print_timing: id  2 | task 55 | n_gen =  10215, tg = 154.57 t/s, tg_3s = 152.26 t/s
1.40.287.569 I slot print_timing: id  2 | task 55 | n_gen =  10673, tg = 154.47 t/s, tg_3s = 152.38 t/s
1.43.292.968 I slot print_timing: id  2 | task 55 | n_gen =  11131, tg = 154.39 t/s, tg_3s = 152.39 t/s
1.46.298.510 I slot print_timing: id  2 | task 55 | n_gen =  11588, tg = 154.29 t/s, tg_3s = 152.05 t/s
1.49.304.695 I slot print_timing: id  2 | task 55 | n_gen =  12044, tg = 154.19 t/s, tg_3s = 151.69 t/s
1.52.305.188 I slot print_timing: id  2 | task 55 | n_gen =  12499, tg = 154.10 t/s, tg_3s = 151.64 t/s
1.55.307.130 I slot print_timing: id  2 | task 55 | n_gen =  12954, tg = 154.01 t/s, tg_3s = 151.57 t/s
1.58.311.642 I slot print_timing: id  2 | task 55 | n_gen =  13410, tg = 153.93 t/s, tg_3s = 151.77 t/s
2.01.312.720 I slot print_timing: id  2 | task 55 | n_gen =  13863, tg = 153.83 t/s, tg_3s = 150.95 t/s
2.04.313.598 I slot print_timing: id  2 | task 55 | n_gen =  14316, tg = 153.74 t/s, tg_3s = 150.96 t/s
2.07.315.355 I slot print_timing: id  2 | task 55 | n_gen =  14769, tg = 153.65 t/s, tg_3s = 150.91 t/s
2.10.318.074 I slot print_timing: id  2 | task 55 | n_gen =  15222, tg = 153.57 t/s, tg_3s = 150.86 t/s
2.13.321.106 I slot print_timing: id  2 | task 55 | n_gen =  15675, tg = 153.49 t/s, tg_3s = 150.85 t/s
2.16.322.091 I slot print_timing: id  2 | task 55 | n_gen =  16126, tg = 153.40 t/s, tg_3s = 150.28 t/s
2.19.326.406 I slot print_timing: id  2 | task 55 | n_gen =  16577, tg = 153.30 t/s, tg_3s = 150.12 t/s
2.22.328.461 I slot print_timing: id  2 | task 55 | n_gen =  17027, tg = 153.21 t/s, tg_3s = 149.90 t/s
2.25.333.685 I slot print_timing: id  2 | task 55 | n_gen =  17478, tg = 153.13 t/s, tg_3s = 150.07 t/s
2.28.339.358 I slot print_timing: id  2 | task 55 | n_gen =  17928, tg = 153.04 t/s, tg_3s = 149.72 t/s
2.31.343.322 I slot print_timing: id  2 | task 55 | n_gen =  18376, tg = 152.94 t/s, tg_3s = 149.14 t/s
2.34.347.857 I slot print_timing: id  2 | task 55 | n_gen =  18825, tg = 152.86 t/s, tg_3s = 149.44 t/s
2.37.349.388 I slot print_timing: id  2 | task 55 | n_gen =  19273, tg = 152.77 t/s, tg_3s = 149.26 t/s
2.40.349.993 I slot print_timing: id  2 | task 55 | n_gen =  19720, tg = 152.68 t/s, tg_3s = 148.97 t/s
2.43.350.894 I slot print_timing: id  2 | task 55 | n_gen =  20167, tg = 152.60 t/s, tg_3s = 148.96 t/s
2.46.351.152 I slot print_timing: id  2 | task 55 | n_gen =  20612, tg = 152.51 t/s, tg_3s = 148.32 t/s
2.49.354.430 I slot print_timing: id  2 | task 55 | n_gen =  21059, tg = 152.43 t/s, tg_3s = 148.84 t/s
2.52.357.314 I slot print_timing: id  2 | task 55 | n_gen =  21504, tg = 152.34 t/s, tg_3s = 148.19 t/s
2.55.358.865 I slot print_timing: id  2 | task 55 | n_gen =  21949, tg = 152.25 t/s, tg_3s = 148.26 t/s
2.58.360.494 I slot print_timing: id  2 | task 55 | n_gen =  22394, tg = 152.17 t/s, tg_3s = 148.25 t/s
3.01.364.060 I slot print_timing: id  2 | task 55 | n_gen =  22839, tg = 152.09 t/s, tg_3s = 148.16 t/s
3.04.368.947 I slot print_timing: id  2 | task 55 | n_gen =  23284, tg = 152.01 t/s, tg_3s = 148.09 t/s
3.07.375.351 I slot print_timing: id  2 | task 55 | n_gen =  23728, tg = 151.93 t/s, tg_3s = 147.68 t/s
3.10.376.601 I slot print_timing: id  2 | task 55 | n_gen =  24172, tg = 151.85 t/s, tg_3s = 147.94 t/s
3.13.382.030 I slot print_timing: id  2 | task 55 | n_gen =  24615, tg = 151.77 t/s, tg_3s = 147.40 t/s
3.16.382.282 I slot print_timing: id  2 | task 55 | n_gen =  25057, tg = 151.69 t/s, tg_3s = 147.32 t/s
3.19.386.306 I slot print_timing: id  2 | task 55 | n_gen =  25498, tg = 151.60 t/s, tg_3s = 146.80 t/s
3.22.387.779 I slot print_timing: id  2 | task 55 | n_gen =  25939, tg = 151.52 t/s, tg_3s = 146.93 t/s
3.23.252.271 I slot print_timing: id  2 | task 55 | prompt eval time =    1032.75 ms /  5007 tokens (    0.21 ms per token,  4848.22 tokens per second)
3.23.252.277 I slot print_timing: id  2 | task 55 |        eval time =  172050.60 ms / 26065 tokens (    6.60 ms per token,   151.49 tokens per second)
3.23.252.277 I slot print_timing: id  2 | task 55 |       total time =  173083.35 ms / 31072 tokens
3.23.252.278 I slot print_timing: id  2 | task 55 |    graphs reused =      26012

Funilly after all this mangling it is still usable, above times are from reviewing some rust code. It did too much reasoning but came up with a decent review in the end and even detected the memory leak in it.

Speed on CPU (16 cores)

The CPU speed is slower but still usable: ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected

model	size	params	backend	ngl	fa	lm	test	t/s
bailingmoe3 11.85B.A2.6B Q8_0	11.29 GiB	11.37 B	CUDA	999	1	dio	pp512	357.14 ± 0.58
bailingmoe3 11.85B.A2.6B Q8_0	11.29 GiB	11.37 B	CUDA	999	1	dio	tg128	39.29 ± 0.05

Adjusted original explanation from inclusionAI

Key highlights of the model are summarized below:

Efficient Hybrid-Linear Architecture: Ling-3.0-small integrates a 3:1 alternating stacking of KDA and MLA (3 Kimi Delta Attention layers followed by 1 Multi-Head Latent Attention layer per 4-layer block) with a sparse MoE FFN comprising **192** routed experts. Each token activates only **16** routed experts and 1 shared expert, allowing the model to balance long-context modeling capability, parameter efficiency, and computational cost.
Native Hybrid Reasoning and Agentic Capabilities: Ling-3.0-small supports both fast responses and multi-step reasoning, with thinking mode configurable per request through enable_thinking. It delivers balanced performance across general agent tasks, coding, and instruction following.
Local and Edge Deployment: Designed for efficient local deployment, Ling-3.0-small has been validated on NVIDIA RTX 5060TI and CPU, enabling capable reasoning and agentic workloads for consumer GPU's datacenter-class GPUs.

Model Overview

Ling-3.0-small inherits the hybrid linear attation architecture of Ling-3.0 series, while being specifically optimized for lightweight and accessible deployment. The model has 11.85B total parameters, with only 2.6B parameters activated per token.

The architecture of Ling-3.0-small is designed to make computational efficiency serve real-world agentic performance.

A 3:1 KDA–MLA architecture (3 KDA layers and 1 MLA layer per 4-layer block) provides more efficient long-context processing;
A sparse MoE FFN with **196** experts activates 8 routed experts and 1 shared expert per token, enabling broad model capabilities with only **2.6B** activated parameters per token.
Native hybrid reasoning enables fast responses for routine tasks and multi-step reasoning for complex tasks within a single model.

Overall, these designs deliver the inference efficiency needed to deploy lightweight models in real-world agentic workflows.