TextPro-SLM-7B-Interleaving

Your speech LLM can be a prosody-aware text LLM.

Built with Qwen.

TextPro-SLM brings speech closer to a text LLM's native input format: transcript tokens carry linguistic content, and compact prosody embeddings carry complementary acoustic information. The paper studies how this input-side factorization preserves semantic capabilities while learning paralinguistic understanding with roughly 1,000 hours of LLM training audio.

Paper · GitHub project · Paired WhisperPro

Checkpoint

This is the original 7B interleaving stage-2 checkpoint (step 1272), paired with the original WhisperPro. It accepts speech clips of at most 30 seconds. The separate 30Plus models provide a segmentation-based long-audio extension.

Component	Specification
Text backbone	Qwen2.5-7B-Instruct, fully fine-tuned
Speech encoder	WhisperPro, decoder-layer-16 prosody features
Projector	1280 → 3584 → 3584, GELU, BF16
Fusion	5:1 text–prosody interleaving
Audio	Mono, 16 kHz, at most 30 seconds
LLM context	32,768 tokens
Output	Text response

The original inference policy inserts a prosody marker before each group of five text tokens. Token-level WhisperPro features are split into the corresponding number of groups and mean-pooled before projection.

The trained prosody marker is <|repo_name|> (ID 151663). Preserve the supplied tokenizer and the fusion settings in textpro_config.json.

Download

pip install "huggingface-hub>=0.36,<1"
hf download qqjz/TextPro-SLM-7B-Interleaving --local-dir TextPro-SLM-7B-Interleaving

This Hub repository contains weights, configurations, tokenizer artifacts, and documentation only. Inference code, dependencies, and usage examples are being prepared separately in the GitHub project. Loading the text backbone alone does not perform prosody-aware speech inference.

Files and verification

llm/ contains the complete fine-tuned language model and tokenizer. projector.safetensors contains the compact trained projector. textpro_config.json records fusion settings and pins the original WhisperPro revision. SHA256SUMS.json records artifact sizes and hashes.

The four projector parameter tensors are unchanged from the original checkpoint. The export was checked locally and on GPU: parameter hashes, synthetic-input projections, injected embeddings, next-token logits, and fixed-seed generated tokens matched the original implementation on two real-audio checks. The original 5:1 interleaving implementation was used for the before/after check.

These are artifact-equivalence checks, not new benchmark measurements. Evaluation results and experimental settings are described in the paper.

Model family

Encoder	Speech LLM	Fusion / audio handling
WhisperPro	TextPro-SLM-7B-Global	Global prepending, original ≤30s
WhisperPro	TextPro-SLM-7B-Interleaving	5:1 interleaving, original ≤30s
WhisperPro	TextPro-SLM-3B-Global	Global prepending, original ≤30s
WhisperPro-30Plus	TextPro-SLM-7B-30Plus	Global prepending, segmented long audio

Limitations

Transcription errors can propagate to the answer. Performance can vary with language, recording conditions, speaker characteristics, and domain. Prosody pooling compresses temporal detail, and generated answers may contain hallucinations. This release returns text rather than synthesized speech. The 30Plus checkpoints are separate artifacts; this model card does not claim unlimited audio duration or new benchmark accuracy.

License and provenance

Released under Apache-2.0; see LICENSE and NOTICE. Built with Qwen2.5-7B-Instruct.

The paired WhisperPro derives from OpenAI Whisper large-v3 and has its own Apache-2.0 license. See MODIFICATIONS.md for the modified artifact list and export changes.

Citation

@inproceedings{cui2026textproslm,
  title={Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM},
  author={Cui, Wenqian and Li, Xiao-Hui and Tan, Daxin and Zheng, Qiyong and King, Irwin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  url={https://openreview.net/forum?id=Jsjmo8eX29}
}