Kiseki-TTS-1.1

A small, fast Japanese TTS model with a Mamba2 state-space decoder, built on top of Qwen/Qwen3-TTS-Tokenizer-12Hz.

Kiseki-TTS generates discrete neural audio codec tokens at 12.5 Hz and decodes them to waveform with the Qwen3 TTS codec. Because the acoustic decoder is a linear-time SSM rather than a self-attention stack, generation cost is constant per frame — memory does not grow with utterance length, and there is no KV cache to manage.

Example

---

Model Details

Model Description

• Developed by: telecomadm1145
• Model type: Encoder–decoder (Transformer encoder + cross-attention/Mamba2 decoder)
• Language: Japanese (ja)
• License: MIT
• Finetuned from: telecomadm1145/Kiseki-1.1-0.3B (a seq2seq translation model)
• Audio codec: Qwen/Qwen3-TTS-Tokenizer-12Hz

Architecture

Component	Spec
Encoder	12 layers, bidirectional self-attention + RoPE + SwiGLU
Decoder	6 layers, cross-attention → Mamba2 SSM (no self-attention)
Hidden size	1024
Attention heads	8
SSM state size / head dim	128 / 64 (32 SSM heads)
Conv kernel / expand	4 / 2
Norm	RMSNorm (pre-norm), gated RMSNorm inside the SSM mixer
Text vocab	65,792 (shared encoder embed / decoder embed / LM head)
Params	~0.4 B

The decoder deliberately has no causal self-attention. Temporal context is carried entirely by the Mamba2 recurrent state; text conditioning enters through cross-attention whose K/V are computed once from the encoder and reused for every frame.

Audio tokenization

Property	Value
Frame rate	12.5 Hz (80 ms per frame)
Quantizer layers (Q)	16
Codebook size	2048 per layer
Effective token vocab	2176 (2048 codes + EOS/BOS/PAD, padded to a multiple of 128)
Reserved IDs	EOS = 2048, BOS = 2049, PAD = 2050
Nominal bitrate	16 × 12.5 × log₂(2048) = 2.2 kbps
Max trained length	512 frames ≈ 41 seconds

Depth modelling (MTP head). Each frame's 16 codebook layers are predicted by a shared "multi-token prediction" head rather than 16 separate decoder passes. Layer q sees the decoder hidden state plus the exclusive prefix sum of the embeddings of layers 0 … q-1:

logits_q = W_q · Block( h_t + (1/√Q) · Σ_{j<q} E_j(c_t^j) ) + b_q

where Block is a small RMSNorm → SwiGLU(×2) → RMSNorm residual body shared across all 16 layers. This means one trunk evaluation per frame and 16 cheap head evaluations, instead of 16 full autoregressive steps.

---

Why it's fast

1. 12.5 Hz is the headline number. One second of speech is 12.5 decoder steps. Codecs running at 50 Hz or 75 Hz need 4–6× more autoregressive steps for the same audio. Concretely:

Audio duration	Decoder trunk steps	Codebook head evals
1 s	12.5	200
5 s	63	1,000
10 s	125	2,000
30 s	375	6,000

A 10-second utterance is 125 recurrent steps. For comparison, a token-level LLM TTS at 50 Hz × 8 codebooks would be pushing ~500 trunk steps for the same clip.

2. O(1) state, not O(T) cache. The Mamba2 decoder carries a fixed (32 heads × 128 state × 64 dim) tensor plus a 3-frame conv window per layer. Generating 40 seconds costs exactly as much per step as generating 1 second — no attention matrix, no KV cache reallocation, no quadratic blowup. Long-form synthesis degrades gracefully instead of falling off a memory cliff.

3. Cross-attention K/V is computed once. Encoder output is projected to per-layer K/V a single time during prefill. Every subsequent frame does one small Q·Kᵀ against a fixed-length text sequence.

4. A shallow decoder. Only 6 decoder layers sit in the autoregressive loop. The 12-layer encoder runs exactly once, fully parallel over the input text.

5. The depth loop is cheap. The 16 codebook layers are resolved sequentially (layer q conditions on layers <q), but each step is one ×2 SwiGLU block at d=1024 — small enough that batch-1 generation is memory-bandwidth-bound rather than compute-bound.

---

How to Get Started

Setup

import torch
from transformers import AutoModelForSeq2SeqLM, PreTrainedTokenizerFast

repo = "telecomadm1145/Kiseki-TTS-1.1"
tok = PreTrainedTokenizerFast.from_pretrained(repo)
m = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True).eval().cuda()

ids = m.build_tts_input_ids(tok.encode("こんにちは"), device="cuda")
out = m.generate_audio(ids, max_new_frames=250, temperature=0.9, top_k=50)
codes = out["audio_codes"][0][out["valid_mask"][0]]   # (T, Q)

build_tts_input_ids assembles [TTS] <|2ja|> …text… <eos>. max_new_frames=250 ≈ 20 seconds of audio at 12.5 Hz.

Decode codes to waveform

import soundfile as sf
from qwen_tts import Qwen3TTSTokenizer

tokenizer = Qwen3TTSTokenizer.from_pretrained(
    "Qwen/Qwen3-TTS-Tokenizer-12Hz",
    device_map="cuda:0",
)

wavs, sr = tokenizer.decode({"audio_codes": codes})
sf.write("decode_output.wav", wavs[0], sr)

Sampling parameters

Argument	Default	Notes
max_new_frames	512	Divide by 12.5 for seconds
temperature	0.9	Lower → flatter, more monotone prosody
top_k	50
top_p	0.95
temperature_q0 / top_k_q0	inherit	Tune layer 0 separately — it carries most of the semantic content; residual layers tolerate more randomness

Generation stops when layer 0 emits EOS (2048). BOS and PAD are masked out of the logits, so they can never be sampled.