pipecat-ai/phonellm-max-alpha-2

PhoneLLM Alpha 2: a blazingly fast, open-weights text model designed to be integrated into voice agent pipelines.

This new version is available in two variants: Mini for speed, and Max for accuracy.

Check out our blog post to find out more about the improvements we made, and how we trained the models:

https://www.daily.co/blog/phonellm-alpha-2/

---

Pipecat PhoneLLM Max Alpha 2

PhoneLLM Max Alpha 2 is a full-parameter supervised fine-tune of the language backbone of NVIDIA Nemotron 3.5 Super VL 120B-A12B for phone voice agents with tool calling.

• Architecture: text-only NemotronHForCausalLM (the vision encoder and projector were removed before training); bfloat16 safetensors
• Requirements: transformers >= 5.8 (native NemotronH, no remote code) or vLLM >= 0.30
• Thinking is ON: the model emits one short sentence of reasoning before every response — we refer to this as "microthinking". The model has not been trained to run with thinking disabled.
  • The bundled chat_template.jinja is the stock Nemotron template (thinking on by default).
  • Serve with --reasoning-parser nemotron_v3 and send chat_template_kwargs: {"enable_thinking": true, "truncate_history_thinking": false} on every request — the parser reads the mode from the request, so a request that omits the kwargs has its reply filed under reasoning.
  • The thought comes back in the message's reasoning field; to keep earlier thoughts in the prompt (as in training) re-send each assistant message with its reasoning.
• Recommended settings: temperature=0.

Serving with vLLM

Measured recipe (vLLM 0.30, two B200s with tensor parallelism: ~210 tok/s single-stream decode, ~27 ms TTFT, ~5.8k tok/s aggregate at 128 concurrent requests). The ~240 GB of bf16 weights need two 180 GB+ GPUs; B300:2 is the same speed.

vllm serve pipecat-ai/phonellm-max-alpha-2 \
  --served-model-name phonellm-max-alpha-2 \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --async-scheduling \
  --mamba-ssm-cache-dtype float32 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

• Do not add --enforce-eager: CUDA graphs are ~7x faster decode on this model.
• If CUDA-graph capture runs out of memory, use --max-num-seqs 64 --max-cudagraph-capture-size 64 (bf16 KV cache, also measured) rather than eager mode. fp8 KV only saves memory here (the model has few attention layers); it does not change outputs measurably.
• To make thinking unconditional on a shared endpoint, rewrite requests server-side (we run a small ASGI middleware through vLLM's --middleware) rather than editing the template: the reasoning parser must agree with the template on the mode.
• The weights include the base model's multi-token-prediction (MTP) block; the recipe above does not use it.

License

PhoneLLM Max Alpha 2 is released under the OpenMDW License Agreement, version 1.1, the same licence as its base model, NVIDIA Nemotron 3.5 Super VL 120B-A12B. If you redistribute this model or your own derivatives of it, you must retain (1) a copy of the OpenMDW License Agreement and (2) the copyright notices and other notices of origin that apply to your distribution. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.