Xing4.0-29B-A4B-6bit-MLX — 6-bit MLX quant of Xing4.0-29B-A4B

Unofficial Apple Silicon quantization of XingChen-AGI/Xing4.0-29B-A4B, produced with mlx_lm.convert (MLX affine quantizer, group size 64) on an Apple M5 Max / 128 GB.

I am not affiliated with China Telecom AI. All upstream weights, benchmarks and license terms belong to them, and the upstream Apache-2.0 license governs this repository too (see License).

Format note: these are MLX safetensors, not GGUF. They will not load in llama.cpp / Ollama / LM Studio, and they will not load in PyTorch/vLLM/SGLang either. Use an MLX runtime.

Before you download: you need xing4_0 support in mlx-lm

Xing4.0 uses a new architecture (model_type: xing4_0) that mlx-lm does not implement yet. Without it any MLX runtime stops with:

ValueError: Model type xing4_0 not supported.

An implementation exists and is verified against the upstream PyTorch code (see Provenance), but it is not merged into mlx-lm at the time of writing. Until it is, these weights will not load anywhere. If you need them now, open an issue here and I will point you at the model file.

Pick a variant

	4bit	6bit	8bit
Weights on disk	15.51 GiB (16.65 GB), 4 shards	22.37 GiB (24.02 GB), 5 shards	29.23 GiB (31.38 GB), 6 shards
Effective precision	4.514 bits per weight	6.512 bits per weight	8.509 bits per weight
Peak RAM, short prompt	15.6 GB	22.4 GB	29.3 GB
Measured generation	21.2 tok/s	21.0 tok/s	21.1 tok/s
Choose when	prioritize memory headroom	balance size and weight precision	prioritize weight precision and have the memory

All three drop the multi-token-prediction layer (see below). Throughput is nearly identical across the three because only ~4B parameters are active per token; the difference shows up in memory, not speed. Task-level accuracy after quantization has not been measured. Runtime memory also depends on context length and KV cache.

What is inside (read from the shipped config.json)

Field	Value
Architecture	Xing4_0ForCausalLM (model_type: xing4_0)
Parameters served	29.51 B total, 4 B active per token
Layers / hidden	40 layers, hidden_size 3584, dense FFN 9216
Attention	MLA — q_lora_rank 768, kv_lora_rank 512, qk_nope_head_dim 128, qk_rope_head_dim 64, v_head_dim 128, 32 heads
MoE	64 routed experts (moe_intermediate_size 1024) + 1 shared, 4 active per token, noaux_tc routing with sigmoid scoring, first 2 layers dense
Residual stream	mHC hyper-connections: hc_mult 4 parallel streams mixed by a Sinkhorn-normalized matrix (hc_sinkhorn_iters 20), two per layer
Position encoding	YaRN, factor 64 over original_max_position_embeddings 4096, interleaved RoPE
Context	max_position_embeddings: 262144
Vocab	131,072 (tokenizer, tokenization_xing4_0.py and chat_template.jinja copied unchanged)
MTP	dropped — num_nextn_predict_layers normalized to 0

Quantization recipe

• mlx_lm.convert -q --q-bits 6 --q-group-size 64, mode affine, source BF16
• 476 modules quantized: attention projections, all 64 routed experts + shared expert per MoE layer, the dense FFNs of layers 0–1, embed_tokens and lm_head
• effective 6.512 bits per weight (scales and biases included)
• never quantized: all RMSNorms, the MoE router (mlp.gate), the hyper-connection tables (hc_fn, hc_base in BF16), and hc_scale / e_score_correction_bias kept in FP32

About the dropped MTP layer

The upstream checkpoint carries a 41st layer (1.71 B parameters) implementing multi-token prediction: eh_proj, enorm, hnorm, its own embed_tokens, a full attention + MoE block and a shared_head. MLX has no speculative-decoding path for this architecture, so those tensors are dropped and num_nextn_predict_layers is set to 0 to keep the shipped config self-consistent. If you want MTP, use the upstream BF16 checkpoint with a runtime that supports it.

Requirements

• Apple Silicon, macOS 15+ (built and smoke-tested on M5 Max, 128 GB)
• mlx-lm with xing4_0 support — see the warning above
• roughly 22.4 GB of free unified memory for a short prompt, more for long context

Usage

from mlx_lm import load, generate

# the custom tokenizer is loaded from the repo, so both flags are needed
model, tokenizer = load(
    "suzu89/Xing4.0-29B-A4B-6bit-MLX",
    tokenizer_config={"trust_remote_code": True},
    trust_remote_code=True,
)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain Sinkhorn normalization in one sentence."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512, verbose=True))

mlx_lm.load forwards trust_remote_code to the model but not to the tokenizer, which is why tokenizer_config carries its own flag. Without it you get an unrelated-looking AttributeError: 'PreTrainedConfig' object has no attribute 'max_position_embeddings'.

The chat template supports enable_thinking (on by default) and tool calls. The model tends to write its reasoning trace in Chinese even for English prompts; that is upstream behaviour, not a quantization artifact.

oMLX

oMLX 0.6.4 cannot run this architecture yet, and its oQ mixed-precision quantizer fails on it for the same reason (its sensitivity pass cannot load the model). These are plain MLX quants, not oQ builds.

Recommended sampling

Upstream recommends, and the shipped generation_config.json matches:

Scenario	temperature	top_p	repetition_penalty
Complex reasoning / general	1.0	0.95	1.05
Coding / agent tasks	0.8	0.95	1.05

Note that mlx_lm.generate does not apply a repetition penalty unless you pass a logits processor.

Provenance

The MLX implementation used to produce and load these weights was validated before quantizing:

Check	Result
Hyper-connection vs the upstream PyTorch module, float32	max relative error < 1e-5
Full model vs modeling_xing4_0.py, small random-weight config	max relative error 2.6e-07 on logits, 100% argmax agreement
Upstream BF16 checkpoint, 58 GB	loads and generates coherent text
Each quant in this family	loads and generates coherent text

Two upstream bugs found along the way, neither affecting these weights: the reference _init_weights initializes module.fn/base/scale while the class defines hc_fn/hc_base/hc_scale (random init from config fails, loading pretrained weights is unaffected), and mlx_lm.load does not forward trust_remote_code to the tokenizer.

Benchmarks

I publish no numbers I have not measured myself. The table below is upstream's, measured on the BF16 model, and is not a measurement of these quantized weights:

Benchmark	Xing4.0-29B-A4B (BF16, upstream)	this quant
IFBench	69.67	not measured
AIME2026	90.00	not measured
AA.LCR	61.00	not measured
Tau3-Bench	64.63	not measured
Claw-Eval	76.55	not measured
SWE-bench Verified	75.00	not measured
Terminal-Bench 2.1	57.50	not measured
SWE-bench Multilingual	66.00	not measured
DeepresearchBII	60.80	not measured

Measurements and issue reports ("quant X broke task Y") are welcome and will be merged into this table.

Known caveats

• Quantization is lossy. If you see a regression, compare against a higher-precision variant and the BF16 source before filing a bug.
• The hyper-connection mixing runs in the residual path of every layer and is kept in BF16/FP32 here; its sensitivity to weight quantization elsewhere in the model has not been studied.
• 262k context is the architecture's limit, not a promise: keep the KV cache inside your memory budget or the machine swaps.
• No MTP head, so no self-speculative decoding.
• Agentic and long-context behaviour at 6-bit is untested.

License

Distributed under the Apache License 2.0, inherited from XingChen-AGI/Xing4.0-29B-A4B. See LICENSE-NOTICE.md in this repository.

Citation

@misc{xing4-29b-a4b-mlx-6bit,
  title        = {Xing4.0-29B-A4B-6bit-MLX: MLX 6-bit quantization of Xing4.0-29B-A4B},
  author       = {suzu89},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/suzu89/Xing4.0-29B-A4B-6bit-MLX}},
  note         = {Unofficial quantization of XingChen-AGI/Xing4.0-29B-A4B}
}

@misc{liu2025trainingreporttelechat3moe,
  title         = {Training Report of TeleChat3-MoE},
  author        = {Xinzhang Liu and others},
  year          = {2025},
  eprint        = {2512.24157},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2512.24157}
}

Acknowledgements

• China Telecom AI (XingChen-AGI) for Xing4.0-29B-A4B and the mHC architecture.
• Apple MLX team for mlx and mlx-lm, whose DeepSeek-V3 implementation this architecture builds on directly.