Instructions to use xanguera/phonellm-alpha-1-int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use xanguera/phonellm-alpha-1-int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="xanguera/phonellm-alpha-1-int8", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("xanguera/phonellm-alpha-1-int8", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("xanguera/phonellm-alpha-1-int8", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use xanguera/phonellm-alpha-1-int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "xanguera/phonellm-alpha-1-int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xanguera/phonellm-alpha-1-int8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/xanguera/phonellm-alpha-1-int8

SGLang

How to use xanguera/phonellm-alpha-1-int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "xanguera/phonellm-alpha-1-int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xanguera/phonellm-alpha-1-int8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "xanguera/phonellm-alpha-1-int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xanguera/phonellm-alpha-1-int8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use xanguera/phonellm-alpha-1-int8 with Docker Model Runner:
```
docker model run hf.co/xanguera/phonellm-alpha-1-int8
```

PhoneLLM Alpha 1 — INT8 (compressed-tensors)

A community INT8 quantization of pipecat-ai/phonellm-alpha-1, built to fit Ampere GPUs (e.g. 3x RTX 3090) that lack NVFP4 support and can't hold the full BF16 checkpoint alongside a usable KV cache.


Base model	`pipecat-ai/phonellm-alpha-1` (Nemotron 3 Nano 30B-A3B hybrid Mamba-Transformer MoE, fine-tuned for voice agents)
Quantization format	compressed-tensors, INT8
Weights	int8, symmetric, per-channel
Activations	int8, symmetric, dynamic per-token
Kept in BF16	attention/Mamba mixer projections, gates, norms, `lm_head`
MoE experts	INT8 via RTN — this architecture's forward pass structurally prevents GPTQ calibration from reaching per-expert modules (a known limitation, not a bug)
Size on disk	~32GB (vs. ~59GB BF16)
Serving	vLLM, `trust_remote_code=True`
Validated on	3x RTX 3090, `tensor_parallel_size=2`

Why this exists

Neither official PhoneLLM Alpha 1 release fits 3x RTX 3090: they're Ampere (no NVFP4 support), and the BF16 checkpoint alone is ~59GB — too tight for KV cache headroom across only 3x24GB cards. This checkpoint quantizes the MoE experts and general Linear layers to INT8 (channel-wise symmetric weights, dynamic per-token symmetric int8 activations) while keeping attention/Mamba projections, norms, gates, and lm_head in BF16, matching NVIDIA's own precedent for this architecture.

Validation

Coherent generations, and correct <tool_call> emission with accurate parameter extraction, confirmed via a say/do test. This checkpoint has not been run against PhoneBench — no formal accuracy comparison against the BF16 or NVFP4 releases is published here.

Serving with vLLM

vllm serve xanguera/phonellm-alpha-1-int8 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --reasoning-parser-plugin nano_v3_reasoning_parser.py \
  --reasoning-parser nano_v3

Important: the reasoning-parser plugin is required. Without it, the model's <think>...</think> reasoning text leaks directly into the visible response content (a literal </think> shows up mid-sentence) instead of landing in a separate reasoning field — setting enable_thinking: false via chat_template_kwargs alone is not sufficient. This only shows up when testing the actual served HTTP endpoint; vLLM's offline batch API doesn't exercise this code path.

Recommended inference settings match the base model: temperature=0, thinking disabled.

License

This model is a quantization of pipecat-ai/phonellm-alpha-1, released under the BSD 2-Clause License, itself a derivative of NVIDIA Nemotron 3 Nano 30B-A3B, licensed under the NVIDIA Nemotron Open Model License.

Per Section 3 (Redistribution) of the NVIDIA Nemotron Open Model License, this repository includes a copy of that license and retains the NVIDIA copyright and attribution notices carried in the bundled model code. This quantization is released under the BSD 2-Clause License, consistent with the terms under which the base model was released; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.

Acknowledgments

Built on PhoneLLM Alpha 1 by the Pipecat team at Daily, itself built on NVIDIA Nemotron 3 Nano 30B-A3B.

Downloads last month: -

Safetensors

Model size

32B params

Tensor type

F32

BF16

Model tree for xanguera/phonellm-alpha-1-int8

Base model

nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

Finetuned

pipecat-ai/phonellm-alpha-1

Quantized

(7)

this model