X2-NativeCursor-Qwen3TTS-12Hz

Reading-progress observer for token-level streaming Qwen3-TTS

Token-level streaming TTS starts speaking before the sentence is finished, so a client receives audio without knowing which characters of the source text it carries. X2-NativeCursor answers that question before waveform decoding: a 2M-parameter observer reads the codebook-0 token the Qwen3-TTS Talker emits every 80 ms, scores it against the spoken labels of the text visible so far, and publishes a cursor into the source text that never moves backward. The TTS generator, the speech tokenizer and the vocoder stay as they are; the observer is the only addition.

This repository holds the released observer head. It is integrated in the Qwen3TTS-Streaming engine (dev branch) as the native estimator behind the engine's text_progress events, and is described on the X2Streaming-TTS feature page.

Files

File	Size	SHA-256
qwen3_tts_12hz_la1_seed0.pt	8.17 MB	aa94527fddff97cca0c337529a3b840b141d8301c4a6082557984438a7e584bc

Compatibility

Backbone	Qwen3-TTS 12 Hz custom-1.7b native tokens: codec vocabulary 3072, 80 ms frames
Released against	zehan1/X2Streaming-TTS-1.7B (speaker robot_service_v1), the checkpoint served by the engine during validation
Labels	503 spoken labels: 477 pinyin syllables + 26 English letters (en_unit=letter)
Lookahead	1 native frame (80 ms); left context 29 frames
Parameters	2,036,991 (label embedding tied to the content classifier)
Runs on	CPU by default inside the engine's frontend thread (native_device: auto); CUDA is opt-in
Engine version	Qwen3TTS-Streaming dev branch (feature feat/native-cursor, based on main @ bea54a4)

The head is voice- and backbone-specific. For another checkpoint, voice or codec, re-train and validate the observer before enabling native; the engine refuses a head whose SHA-256 differs from native_expected_head_sha256 when that field is set.

Usage

Download the head and place it under the engine's resources/native_cursor/:

huggingface-cli download zehan1/X2-NativeCursor-Qwen3TTS-12Hz \
  --local-dir ./weights/X2-NativeCursor-Qwen3TTS-12Hz
cp ./weights/X2-NativeCursor-Qwen3TTS-12Hz/qwen3_tts_12hz_la1_seed0.pt \
   <Qwen3TTS-Streaming>/resources/native_cursor/

Switch the estimator in engine.yaml, or set ENGINE_TEXT_PROGRESS_ESTIMATOR=native:

text_progress:
  estimator: native                      # ema (default) | native
  native_head_path: resources/native_cursor/qwen3_tts_12hz_la1_seed0.pt
  native_device: auto                    # auto (= cpu) | cpu | cuda | cuda:N
  native_expected_head_sha256: "aa94527fddff97cca0c337529a3b840b141d8301c4a6082557984438a7e584bc"
  native_fail_open: true                 # a load failure falls back to ema

A successful start logs:

native cursor head loaded: resources/native_cursor/qwen3_tts_12hz_la1_seed0.pt on cpu
  (labels=503, en_unit=letter, offsets=[-2,-1,0,1,2,3,4], left=29 right=1 frames,
   params=2036991, sha256=aa94527fddff)

Cursor anchors are delivered on the engine's existing text_progress events, so existing clients keep working. Each anchor carries progress_basis = native_cursor_v1 and progress_quality = aligned plus native_* diagnostic fields; the built-in ema estimator remains the fallback when the head is absent. The event contract and the design are documented upstream in docs/dev/design/native_cursor_progress.md.

How it works

1. TNPlan turns the committed text into spoken labels (the normalized reading) and records which span of the original text owns each label, so the projection back to the raw text stays exact when reading order differs from writing order (99% → 百分之九十九).
2. A native-token encoder (four dilated convolution blocks, hidden size 256) embeds each codebook-0 token with one frame of lookahead.
3. A local matcher scores the labels at offsets −2 … +4 around the previous position against the token feature and a small location state, and moves a continuous internal position by the expected offset.
4. The published cursor is the high-water mark of that position mapped to the furthest raw-text span end reached; it is monotone by construction.

Training uses onset times from Qwen3-ForcedAligner as supervision, on 20,235 utterances synthesized by the deployed custom voice, for 10 epochs (seed 0). Only the observer is trained.

Evaluation

Research evaluation (800 held-out texts in four groups of 200: plain Chinese, Chinese with numbers, Chinese with symbols, English; text supplied in 2–8 character chunks; reference = Qwen3-ForcedAligner; mean of three seeds):

Method	Online	Lookahead	Params	MAE zh (chars) ↓	MAE en ↓	Onset F1@80 ms ↑
WindowMMS+PersistentCTC (online waveform)	✓	320 ms	315.5 M	1.253	2.226	0.341
Cross-attention readout (native tokens)	✗	320 ms	2.490 M	0.414	1.629	0.828
CodecCTC+skip-DP (native tokens)	✓	320 ms	1.705 M	0.416	1.568	0.823
X2-NativeCursor	✓	80 ms	2.166 M	0.151 ± 0.005	1.247	0.924

Real-time factor 0.0180 against 0.3598 for the online waveform baseline.

Engine acceptance of this exact head (the engine's own audio on 80 held-out utterances, live text_progress anchors, same reference):

Metric	Value
Raw-character MAE	0.215 (95% CI 0.166–0.280)
Onset F1 @ 80 ms	0.928
Backward steps	0 in 6,441 anchors
Observer cost per frame, p50 / p90 at 1 session (CPU)	4.2 / 5.5 ms
Observer cost per frame, p50 / p90 at 16 sessions (CPU)	13.6 / 17.4 ms

One native frame represents 80 ms of speech, so the observer stays inside its budget at these concurrency levels.

Limitations

• Trained for one voice and one backbone; accuracy drops on other voices (mean MAE 0.772 on three voices outside training in the research ablation). Re-train for a new voice.
• English labels are whole words, so cursor updates are coarser in English than in Chinese.
• The reference integration recomputes a 31-frame window per native frame on the frontend thread; cost grows with concurrency. Moving the observer into the engine thread with per-frame batching is the planned production path.

Citation

The observer is described in a manuscript that is currently under review. Until it is public, please cite the X2Streaming-TTS paper and link this repository:

@article{wen2026x2streamingtts,
  title   = {X2Streaming-TTS: Causal Token-Level Text-to-Speech from Streaming Text with Speech-State Inheritance},
  author  = {Wen, Rime and Liu, Zehan and Qin, Shawn and Shi, Lights and Gan, Roy and Wang, Hao and Wang, Qian},
  journal = {arXiv preprint arXiv:2608.18661},
  year    = {2026},
}

License

Apache License 2.0, Copyright (c) 2026 XSquareRobot. The head is used together with Qwen3-TTS models (Apache-2.0, Qwen team) and the Qwen3-ForcedAligner reference (Qwen3-ASR, Apache-2.0) was used only for training supervision.

中文简介

X2-NativeCursor 是一个约 2M 参数的朗读进度观察器：它读取 Qwen3-TTS Talker 每 80 ms 输出的 codebook-0 token，在波形解码之前给出"现在读到原文第几个字"，位置始终向前。 TTS 生成器、语音 tokenizer 与声码器保持原样。本仓库存放发布的观察器头，对应 zehan1/X2Streaming-TTS-1.7B 的音色；放进 Qwen3TTS-Streaming 引擎的 resources/native_cursor/ 并把 text_progress.estimator 设为 native 即可启用， 进度沿用现有的 text_progress 事件下发。换音色或换模型需要重新训练并验证。 许可为 Apache-2.0。