Instructions to use pipecat-ai/phonellm-mini-alpha-2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pipecat-ai/phonellm-mini-alpha-2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pipecat-ai/phonellm-mini-alpha-2", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("pipecat-ai/phonellm-mini-alpha-2", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("pipecat-ai/phonellm-mini-alpha-2", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use pipecat-ai/phonellm-mini-alpha-2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pipecat-ai/phonellm-mini-alpha-2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pipecat-ai/phonellm-mini-alpha-2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/pipecat-ai/phonellm-mini-alpha-2

SGLang

How to use pipecat-ai/phonellm-mini-alpha-2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pipecat-ai/phonellm-mini-alpha-2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pipecat-ai/phonellm-mini-alpha-2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pipecat-ai/phonellm-mini-alpha-2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pipecat-ai/phonellm-mini-alpha-2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use pipecat-ai/phonellm-mini-alpha-2 with Docker Model Runner:
```
docker model run hf.co/pipecat-ai/phonellm-mini-alpha-2
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

PhoneLLM Alpha 2: a blazingly fast, open-weights text model designed to be integrated into voice agent pipelines.

This new version is available in two variants: Mini for speed, and Max for accuracy.

Check out our blog post to find out more about the improvements we made, and how we trained the models:

https://www.daily.co/blog/phonellm-alpha-2/

Pipecat PhoneLLM Mini Alpha 2

PhoneLLM Mini Alpha 2 is a full-parameter supervised fine-tune of NVIDIA Nemotron 3 Nano 30B-A3B for phone voice agents with tool calling.

Precision: bfloat16 safetensors
Thinking is disabled by default: the bundled chat_template.jinja defaults enable_thinking to false (the model was trained without reasoning). Serve without a reasoning parser.
Recommended settings: temperature=0.

Serving with vLLM

Measured recipe (vLLM 0.30, one B200: ~420 tok/s single-stream decode, ~17 ms TTFT, ~10k tok/s aggregate at 128 concurrent requests; H200 and RTX PRO 6000 also work at lower throughput):

vllm serve pipecat-ai/phonellm-mini-alpha-2 \
  --served-model-name phonellm-mini-alpha-2 \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --async-scheduling \
  --mamba-ssm-cache-dtype float32 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Do not add --enforce-eager: CUDA graphs are 5-7x faster decode on this model.
No --reasoning-parser: the bundled template keeps thinking off, so replies and tool calls are parsed directly. (With a reasoning parser, a request that omits enable_thinking would have its tool calls filed under reasoning.)
Keep 128-256 requests in flight per GPU for batch workloads; raise --max-model-len only if needed.

License

PhoneLLM Mini Alpha 2 is released under the BSD 2-Clause License.

PhoneLLM is a derivative work of NVIDIA Nemotron 3 Nano 30B-A3B, which is licensed under the NVIDIA Nemotron Open Model License. Under Section 3 (Redistribution) of that license, if you redistribute this model or your own derivatives of it, you must (a) include a copy of the NVIDIA Nemotron Open Model License, and (b) retain the NVIDIA copyright and attribution notices. Our BSD 2-Clause terms apply to our modifications and to the model as a whole, as Section 3 permits; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.

Downloads last month: 552

Safetensors

Model size

32B params

Tensor type

F32

BF16

Model tree for pipecat-ai/phonellm-mini-alpha-2

Base model

nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

Finetuned

(73)

this model

Quantizations

4 models