mlx-community/XVERSE-7B-Chat-4bit

This model was converted to MLX format from xverse/XVERSE-7B-Chat using mlx-lm 0.32.0, quantized to 4 bits (group size 64, 4.500 bits per weight).

XVERSE-7B-Chat is a multilingual chat model from Shenzhen Yuanxiang Technology. It uses a standard LLaMA-style decoder-only architecture, so mlx-lm loads it through mlx_lm/models/llama.py with a "xverse": "llama" entry in MODEL_REMAPPING.

Use with mlx

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/XVERSE-7B-Chat-4bit")

messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

response = generate(model, tokenizer, prompt=prompt, verbose=True)

Conversion notes

Three changes were needed on top of the upstream repository. Each one is reproducible from the source repo.

1. Weights repacked to safetensors

xverse/XVERSE-7B-Chat publishes only pytorch_model-*.bin shards. mlx-lm loads model*.safetensors only, so the 8 shards were loaded with PyTorch and re-saved as safetensors. The 32 self_attn.rotary_emb.inv_freq buffers were dropped, because the rope layer recomputes them. 291 of the original 323 tensors remain, and no other tensor was renamed or modified.

2. tokenizer.json patched for tokenizers >= 0.23

The upstream tokenizer.json declares its Metaspace pre-tokenizer and decoder with the legacy add_prefix_space field and no prepend_scheme. Current tokenizers releases reject this:

Exception: add_prefix_space does not match declared prepend_scheme at line 78 column 3

Both Metaspace entries were rewritten to the equivalent modern form, "prepend_scheme": "never" with "split": true, which preserves the original add_prefix_space: false behaviour. Encoding is unchanged.

3. chat_template added

Upstream ships no chat_template. The prompt format lives in Python, in XverseForCausalLM._build_chat_input, so it is unavailable to any tool that only reads the tokenizer. The format is:

Human: {user}\n\nAssistant: {assistant}<|endoftext|>Human: {user}\n\nAssistant:

A Jinja template implementing exactly that was added to tokenizer_config.json, so apply_chat_template now works and mlx_lm.generate no longer needs --ignore-chat-template.

Only user and assistant roles are supported. The template raises on any other role, matching upstream, which rejects them with message role not supported yet. There is no system-prompt slot.

One known difference from upstream. A chat template renders to a string that is then tokenized in one pass, whereas _build_chat_input encodes "Human: " and the message content separately and concatenates the ids. When the content starts with a Latin word these differ by one token at the boundary: the template yields ▁What where upstream yields ▁ + What. Chinese prompts and multi-turn histories tokenize identically. This is inherent to string-rendering chat templates and cannot be expressed in Jinja. For bit-exact upstream parity, build the token ids with _build_chat_input instead.

Limitations and disclaimer

Inherited from the upstream model. Like all LLMs, XVERSE-7B-Chat may produce inaccurate, biased, or otherwise objectionable content. Developers should run their own safety testing before deploying it. Quantization to 4 bits may further affect output quality relative to the bf16 original.

License

Source code follows Apache-2.0. The model weights follow the XVERSE Model License Agreement. The weights are fully open for academic research and free commercial use; a commercial licence is requested through the application form. Questions go to opensource@xverse.cn.