TextPro-SLM-3B-Global

Your speech LLM can be a prosody-aware text LLM.

Built with Qwen.

TextPro-SLM brings speech closer to a text LLM's native input format: transcript tokens carry linguistic content, and compact prosody embeddings carry complementary acoustic information. The paper studies how this input-side factorization preserves semantic capabilities while learning paralinguistic understanding with roughly 1,000 hours of LLM training audio.

Paper · GitHub project · Paired WhisperPro

Checkpoint

This is the original 3B global-prepending stage-2 checkpoint (step 1272), paired with the original WhisperPro. It accepts speech clips of at most 30 seconds. The separate 30Plus models provide a segmentation-based long-audio extension.

Component	Specification
Text backbone	Qwen2.5-3B-Instruct, fully fine-tuned
Speech encoder	WhisperPro, decoder-layer-16 prosody features
Projector	1280 → 2048 → 2048, GELU, BF16
Fusion	Global prepending
Audio	Mono, 16 kHz, at most 30 seconds
LLM context	32,768 tokens
Output	Text response

A single global prosody vector is prepended to transcript tokens.

The trained prosody marker is <|repo_name|> (ID 151663). Preserve the supplied tokenizer and the fusion settings in textpro_config.json.

Download

pip install "huggingface-hub>=0.36,<1"
hf download qqjz/TextPro-SLM-3B-Global --local-dir TextPro-SLM-3B-Global

This Hub repository contains weights, configurations, tokenizer artifacts, and documentation only. Inference code, dependencies, and usage examples are being prepared separately in the GitHub project. Loading the text backbone alone does not perform prosody-aware speech inference.

Files and verification

llm/ contains the complete fine-tuned language model and tokenizer. projector.safetensors contains the compact trained projector. textpro_config.json records fusion settings and pins the original WhisperPro revision. SHA256SUMS.json records artifact sizes and hashes.

The four projector parameter tensors are unchanged from the original checkpoint. The export was checked locally and on GPU: parameter hashes, synthetic-input projections, injected embeddings, next-token logits, and fixed-seed generated tokens matched the original implementation on two real-audio checks. An obsolete total-size statistic in the shard index was corrected; all tensor payloads and parameter-to-shard mappings are unchanged. Original-index and corrected-index model loading were both checked.

These are artifact-equivalence checks, not new benchmark measurements. Evaluation results and experimental settings are described in the paper.

Model family

Encoder	Speech LLM	Fusion / audio handling
WhisperPro	TextPro-SLM-7B-Global	Global prepending, original ≤30s
WhisperPro	TextPro-SLM-7B-Interleaving	5:1 interleaving, original ≤30s
WhisperPro	TextPro-SLM-3B-Global	Global prepending, original ≤30s
WhisperPro-30Plus	TextPro-SLM-7B-30Plus	Global prepending, segmented long audio

Limitations

Transcription errors can propagate to the answer. Performance can vary with language, recording conditions, speaker characteristics, and domain. Prosody pooling compresses temporal detail, and generated answers may contain hallucinations. This release returns text rather than synthesized speech. The 30Plus checkpoints are separate artifacts; this model card does not claim unlimited audio duration or new benchmark accuracy.

License and provenance

Built with Qwen. This checkpoint inherits the Qwen Research License from its Qwen2.5-3B-Instruct backbone. The upstream grant covers non-commercial research/evaluation; commercial use requires a separate license from Alibaba Cloud. See LICENSE and Notice.

The paired WhisperPro derives from OpenAI Whisper large-v3 and has its own Apache-2.0 license. See MODIFICATIONS.md for the modified artifact list and export changes.

Citation

@inproceedings{cui2026textproslm,
  title={Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM},
  author={Cui, Wenqian and Li, Xiao-Hui and Tan, Daxin and Zheng, Qiyong and King, Irwin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  url={https://openreview.net/forum?id=Jsjmo8eX29}
}