Hugging Face's logo

nvidia
/
personaplex-7b-v1

Audio-to-Audio
Moshi
Safetensors
English
personaplex
speech-to-speech
agent
Model card Files
xet
 Community
65

Maximum supported text prompt length and corresponding performance benchmarks

#64
by XiaomingZhou - opened
Discussion
XiaomingZhou

Hi team,

I am evaluating PersonaPlex for my use case and would like to get more technical details regarding prompt scalability and system performance. Specifically, I have the following questions:

  1. Maximum Supported Text Prompt Length:

    • What is the theoretical and recommended upper limit for the input text prompt length (in tokens or characters)?
    • Are there any known truncation behaviors or context window degradation beyond a certain threshold?

  2. Performance Benchmarks:

    • Do you have any benchmark data or test results showing how performance metrics (e.g., latency, TTFT, throughput, or memory consumption) scale as the text prompt length increases?
    • If available, could you share the hardware configuration used for these tests?

Any guidance, existing benchmarks, or recommended configuration practices for handling long prompts would be greatly appreciated.

Thanks!
susun-123

Not from the NVIDIA team, but we serve PersonaPlex at scale in pega-omni (open source), so here is what we measured.

Prompt length

  • The model's context is 3000 steps (12.5 steps/s, i.e. 4 minutes), shared by the prompt and the conversation. A prompt is the voice prompt (51–69 steps for the shipped voices), 6 silence steps, the role prompt at one text token per step, and 6 more silence steps.
  • The text is wrapped as <system> ... <system> and tokenized with the 32k SentencePiece model: roughly 4–5 characters per token for English, and characters outside the vocabulary fall back to one token per UTF-8 byte (an emoji can cost 4 tokens).
  • Hard limit: the whole prompt must fit the 3000 steps, so about 2900 text tokens with the longest shipped voice.
  • Practical limit: the KV cache is a 3000-step ring, so once prompt plus conversation pass 3000 steps, the oldest steps (the prompt) fall out of attention. A 1000-token prompt starts dropping out after about 2.5 minutes of conversation, a 100-token prompt after about 3.7 minutes. Keep the role prompt short if the persona has to hold for the whole call.
  • In pega-omni we refuse role prompts over 711 characters, a bound that fits even if every character falls back to 4 byte tokens with the longest voice.

Performance (single GPU, our server, default-length prompt of about 100 text tokens)

  • Connect to first frame (prompt prefill): about 40 ms p50, 75 ms p99, from 8 to 128 concurrent sessions.
  • One 80 ms frame for all sessions (batched in one CUDA graph): 8.1 ms at 1 session, 11.0 ms at 64, 15.0 ms at 128.
  • Memory: 1.5 GiB of KV per session, fixed, independent of prompt length.
  • 128 concurrent sessions streaming speech at real time ran with no playback underrun.

We have not measured how prefill time scales with prompt length; prefill is one batched pass, so we expect it to stay small next to the 80 ms frame budget, but that is an expectation, not a measurement.

Method and numbers: docs/personaplex.md, docs/bench.md.

Sign up or log in to comment