thomasfchrr/Spark-X2.5-1.7B-CopilotPlus-NPU

Spark-X2.5-1.7B for Copilot+ PCs (Snapdragon X / Hexagon NPU)

Highlights: Hybrid QNN execution (Qualcomm QNN / ONNX Runtime) · OpenAI-compatible local API · ~18.5 tok/s decode · 2.8 GB weights on disk

An experimental INT8 QDQ conversion of XHToken/Spark-X2.5-1.7B, packaged as an installable Python runtime for Copilot+ PCs. Two fixed-shape ONNX graphs share a compact external weights file and run on the Hexagon NPU through the Qualcomm QNN Execution Provider, with an OpenAI-compatible chat API for local use.

Scope: hybrid QNN/CPU execution (a few intended CPU partitions remain), 512-token short context, greedy decoding. This is an engineering release, not a broadly evaluated accuracy baseline.

Quickstart

One-line install (Windows ARM64, native CPython 3.12, Qualcomm NPU drivers required):

pip install git+https://huggingface.co/thomasfchrr/Spark-X2.5-1.7B-CopilotPlus-NPU.git

Then point the runtime at the downloaded models:

spark-npu chat --model-dir .\models --cache

CLI Chat

spark-npu chat --model-dir .\models --cache

• Sessions load once into two persistent worker processes and are reused across prompts — no recompilation between questions.
• Every turn uses the model's native Spark chat template with a default system instruction (concise, same language as the user).
• History is trimmed oldest-first to fit 350 prompt tokens; the generation budget defaults to min(150, 512 - prompt_tokens) unless --max-new-tokens is given. Generation stops at the EOS token when emitted.
• /clear or /reset clears history · /quit or /exit quits.

REST Server (OpenAI-compatible)

spark-npu serve --model-dir .\models --host 127.0.0.1 --port 8000

No authentication or TLS — localhost only. GET /v1/models advertises spark-npu-v2. Requests are serialized on one NPU queue.

cURL with streaming:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "spark-npu-v2",
       "messages": [{"role": "user", "content": "Why is the sky blue?"}],
       "max_tokens": 32, "temperature": 0, "stream": true}'

Python with the openai package:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")
stream = client.chat.completions.create(
    model="spark-npu-v2",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
    max_tokens=32, temperature=0, stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Only greedy decoding (temperature: 0) is supported; sampling parameters are rejected rather than silently ignored.

Architecture

Component	Specification
Base model	Spark-X2.5-1.7B, INT8 QDQ quantized, FP32 scales
Context window	512 tokens, static KV cache (native sliding window preserved)
Graphs	models/prefill.onnx (64-token chunks) + models/decode.onnx (1 token, fixed cache)
Runtime	ONNX Runtime + QNN Execution Provider, HTP/Hexagon, burst mode
QNN options	FP16 precision, graph-I/O quantization offload, RPC control latency 0
Decoding	Greedy argmax, persistent IOBinding buffers, CPU-managed KV copies
Isolation	One worker process per graph (avoids cross-context QNN failures)

Measured Performance (Snapdragon X Plus, Windows 11 ARM64)

Metric	Observed
Cold compile (first launch)	prefill ~126 s, decode ~174 s
Warm startup with local .bin cache	~16 s total, ~12 s pure ORT init
Decode rate	~18.5 tok/s (inference calls only, short smoke runs)
Example	351-token prompt + 32 tokens, coherent length-limited output

These are smoke-test measurements, not steady-state throughput: they exclude compilation, tokenization, IPC, and cache copies. QNN context caches are device-specific, built locally on first use (~15 GB free disk advised), and are not shipped. Expect a one-time ~5-minute compilation on first launch, then ~16-second warm starts.

Files

• spark_npu/ — engine.py, server.py, cli.py, utils.py
• pyproject.toml — packaging (spark-npu console script)
• models/ — prefill.onnx, decode.onnx, weights.data (2,781,749,248 bytes), model.json, tokenizer assets
• README.md, LICENSE

Limitations

• Hybrid execution: two Einsum and one DequantizeLinear partition stay on CPU by design; KV-cache copies are CPU-managed (no zero-copy NPU I/O claimed).
• 512-token capacity only: prompt template + history + output must fit; oversized requests are rejected, never truncated. No long-context support.
• Peak RAM has not been measured; the 2.8 GB figure is weights on disk.
• Text quality: short continuations validated; no multilingual, complete-answer, or safety benchmark established. Do not transfer upstream benchmark claims.
• Native QNN may log benign failed to close queue warnings on shutdown while exiting cleanly with code 0 and no orphan processes.

License and Attribution

Upstream model: SparkLLM Team / XHToken, Apache License 2.0 (see LICENSE). Derived ONNX QDQ/static-shape conversion with a custom runtime — not an upstream official QNN release. Qualcomm drivers and runtime packages have their own terms and are installed separately.