leonsarmiento/Ling-3.0-tiny-8bit-XL-mlx

Ling-3.0-tiny 8-bit XL MLX

⚠️ DISCLAIMER: This is an experimental quantization using vanilla MLX.

The BailingMoeV3 (bailing_hybrid) architecture is not part of stock mlx-lm. A custom model class implementing the hybrid KDA+MLA attention, MoE routing, and weight sanitization was written from scratch. Basic generation and MATHQA tests pass, but the implementation has not been validated against the original PyTorch model outputs. Use with caution and report any issues.

MLX 8-bit XL (BaseQuant_XL) quantization of inclusionAI/Ling-3.0-tiny — 7.9B total / 1.3B active parameters at 8.7 GB on disk: a genuine RAM-poor MoE.

✅ The model class ships in this repo — load it directly

config.json points at the in-repo bailing_hybrid.py via "model_file", so mlx-lm >= 0.31 loads this repo with zero patching:

pip install -U "mlx-lm>=0.31"
mlx_lm.chat --model leonsarmiento/Ling-3.0-tiny-8bit-XL-mlx

or programmatically (both modes):

from mlx_lm import load, generate

model, tokenizer = load("leonsarmiento/Ling-3.0-tiny-8bit-XL-mlx",
                        tokenizer_config={"trust_remote_code": True})

# ── Instruct mode: fast, direct answers, no reasoning ──
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is 15 x 37?"}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

# ── Thinking mode: <think>...</think> reasoning first, then the answer ──
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is 15 x 37?"}],
    tokenize=False, add_generation_prompt=True, enable_thinking=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=4096))

Thinking mode in the CLI

The chat template defaults to thinking ON when enable_thinking is not passed:

# Interactive CLI chat — thinking ON; <think>...</think> traces print inline
mlx_lm.chat --model leonsarmiento/Ling-3.0-tiny-8bit-XL-mlx --trust-remote-code

# OpenAI-compatible server with the thinking mode made explicit
mlx_lm.server --model leonsarmiento/Ling-3.0-tiny-8bit-XL-mlx --port 6969 \
    --temp 1.0 --top-p 0.95 --top-k 40 --min-p 0.05 \
    --chat-template-args '{"enable_thinking": true}'

Note: plain mlx_lm has no reasoning parser — traces are not hidden, they stream inline as <think>...</think> before each answer. Only a server-side reasoning parser separates them into a reasoning_content field. For instruct mode (no reasoning), pass enable_thinking=False via apply_chat_template(...) as shown above, or --chat-template-args '{"enable_thinking": false}' on mlx_lm.server.

Included helper scripts

This repo ships two ready-to-run helpers so you can use the model entirely through mlx-lm — no LM Studio, oMLX, or any native BailingMoeV3 support required:

# Interactive CLI chat — thinking ON, temp 1.0, top_p 0.95, top_k 40, min_p 0.05, 64K context
python chat_ling3_tiny.py                # add --no-thinking for fast instruct mode

# OpenAI-compatible server on port 6969 (reasoning returned in a `reasoning` field)
./serve_ling3_tiny.sh

Both default to this repo id and download it automatically on first run (pip install -U "mlx-lm>=0.31"). The chat helper trims oldest turns to a 64K-token conversation budget; the model itself supports 131K native context.

Engines with their own runtimes (LM Studio, oMLX) can only run this model if they add BailingMoeV3 support.

The shipped class carries the MLA KV-cache fix: MLA layers use pre-allocating KVCache (in-place 256-token block writes) instead of ConcatenateKVCache, which copied the entire KV array on every token and drove memory into near-OOM on long generations. With the fix, a 700-token prompt + 3,500 generated tokens peaks at 9.4 GB (model is 8.7 GB).

Model Details

Property	Value
Architecture	BailingMoeV3 (bailing_hybrid)
Parameters	7.9B total / 1.3B active
Bits per weight	8.786
Model size	8.7 GB (2 shards)
Quantization	BaseQuant_XL 8-bit (data-agnostic)
Layers	24 (18 KDA + 6 MLA, 3:1 hybrid)
Experts	128 routed + 1 shared, 8 active/token
Context	131,072 tokens native
Thinking mode	Native (enable_thinking in template)

Architecture: Hybrid Linear Attention

Ling-3.0-tiny alternates two attention mechanisms:

• KDA (Kimi Delta Attention) — 3 of every 4 layers. Linear attention with delta rule updates, short convolutions, and Mamba-style decay (A_log). O(n) complexity, constant memory per token via recurrent state. Runs on mlx-lm's gated_delta_update Metal kernel.
• MLA (Multi-Latent Attention) — 1 of every 4 layers. DeepSeek-V3 style latent attention with LoRA-rank Q/KV compression + gated attention (head-wise sigmoid gate). Full softmax attention for global context.

This combination provides efficient long-context processing (KDA) with periodic full-attention reset (MLA).

XL Quantization Recipe

This is a data-agnostic quantization — no calibration data, no iMatrix, no GPTQ. The XL recipe allocates precision by layer importance:

Layer	Precision	Rationale
Router gate	bf16	Critical for expert selection
Shared experts	bf16	Always active, small
LM head	bf16	Output projection
Embeddings	8-bit	Large but non-critical
Attention (KDA + MLA)	8-bit	Moderate sensitivity
Routed experts	8-bit	Bulk params, only 8/128 active/token
Dense MLP (layer 0)	8-bit	Single layer

BaseQuant_XL is fully data-agnostic/static — no calibration set, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role. Data-dependent methods (iMatrix, AWQ, GPTQ, oQ, oQ4e) calibrate on representative text, which can skew representation toward well-represented domains (English, popular topics, public benchmarks) and away from underrepresented ones (non-English, niche use cases, your own data). XL avoids this trade-off entirely.

Benchmark Results

Basic Generation Test

• Task: Write a Python prime-checking function
• Result: ✅ Clean, correct code in 0.7s (instruct mode, thinking off)

MATHQA Thinking-Mode Test (n=10)

Evaluated on a curated MATHQA subset (failure-enriched test set), thinking mode enabled, 8192 token budget, temperature=1.0, top_p=0.95.

Metric	Result
Raw accuracy	8/10 (80%)
Reasoning efficiency	935 chars/correct answer
Avg wall time (correct)	6.1s
Total wall time	261s

Per-question breakdown:

Question	Expected	Got	Reasoning (chars)	Wall (s)
Q26	D	✓ D	511	4.7
Q102	D	✓ D	1,204	4.5
Q108	B	✗ NO MATCH	17,721	106.6
Q379	C	✓ C	1,183	5.6
Q383	C	✓ C	634	4.0
Q396	C	✓ C	1,612	9.9
Q419	E	✓ E	563	6.1
Q636	C	✗ NO MATCH	20,907	105.6
Q653	E	✓ E	567	4.2
Q902	A	✓ A	1,210	10.0

The 2 failures were degenerate reasoning spirals (17K-21K chars, hitting the 8192 token limit without producing an answer). The 8 correct answers averaged just 935 chars of reasoning — notably more token-efficient than larger models on the same test set.

Note: Custom evaluation, not MATHQA-standard. The test set is failure-enriched (harder than random sampling).

Inference Settings

{
  "temperature": 1.0,
  "top_p": 0.95,
  "top_k": 40,
  "min_p": 0.05
}

⚠️ Do NOT use low temperature (e.g. 0.2) with this model. Testing showed that temp=0.2 + repetition_penalty=1.1 makes the model overthink significantly more — reasoning becomes 3-5× longer with no accuracy gain. Use temp=1.0 for concise, decisive reasoning.

Thinking mode: Native <think> / </think> tags. Pass enable_thinking=True in the chat template for reasoning mode, False for fast instruct mode.

About XL Quantization

BaseQuant_XL is a data-agnostic/static quantization method — it uses no calibration data, no iMatrix, no GPTQ, and no data-aware expert calibration. Precision allocation is determined purely by the architectural role of each layer, not by running sample data through the model.

Acknowledgements

• Original model: inclusionAI
• MLX framework: Apple
• KDA implementation based on fla-core