droyster/phonellm-mini-alpha-2-mlx-4bit

PhoneLLM Mini Alpha 2 — Experimental MLX 4-bit with selective BF16

Community MLX conversion of Pipecat PhoneLLM Mini Alpha 2, a voice-agent and tool-calling text model based on NVIDIA Nemotron 3 Nano 30B-A3B. This is an experimental mixed-precision quantization for Apple Silicon. It is not an official Pipecat release.

Format and conversion

Property	Value
Source	Original BF16 PhoneLLM Mini Alpha 2 weights
Source revision	05553ce82536e8415659f86a341ed2e93e2b9a95
Quantization	MLX affine 4-bit, group size 64
Weight storage	Four sharded safetensors files, 19.32 GB total
Average storage precision	4.896 bits per original parameter, including scales and retained higher-precision parameters
Quantized modules	126
Explicit BF16 exclusions	61, including embeddings
Tested runtime	MLX 0.32.4, MLX-LM 0.32.1, Transformers 5.20.0

The layer-selection policy follows the BF16 exclusions in Pipecat's official NVFP4 recipe: the output head, all attention projections, selected Mamba projections, and convolution layers remain BF16. Embeddings remain BF16 as they do in that export. Router and state-space parameters retain their native handling in MLX-LM. See conversion.json for the complete explicit exclusion list.

Quantization starts from the original BF16 weights. MLX computes new affine scales; this conversion does not reuse NVFP4's packed weights, calibration, or benchmark results. The exclusions are a conservative starting point; their optimality for MLX affine quantization has not been established.

Usage

No custom model implementation or patched MLX installation is required. The architecture is provided by MLX-LM. This model is intended for text generation and tool calling within a voice pipeline; speech recognition and speech generation are separate components.

On an Apple Silicon Mac, install the tested runtime in a Python 3.12 virtual environment. After downloading this repository, the fully pinned environment is available with:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements-lock.txt

The main runtime packages are:

pip install mlx==0.32.4 mlx-lm==0.32.1 transformers==5.20.0

Run the included example (use its full path when outside the repository):

python run_example.py --prompt "Explain what a local voice assistant can help with in one sentence."

Or load it with MLX-LM's Python API (replace the repository placeholder with the published repository ID, or use a local model directory):

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("droyster/phonellm-mini-alpha-2-mlx-4bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello!"}],
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
)
print(generate(model, tokenizer, prompt=prompt,
               max_tokens=128, sampler=make_sampler(temp=0)))

Explicitly set enable_thinking=False and temperature zero. MLX-LM 0.32.1's tokenizer wrapper otherwise enables thinking when it recognizes thinking markers, overriding the upstream template's default. The weights themselves need no patch or special loader.

For a local OpenAI-compatible API:

python -m mlx_lm server \
  --model droyster/phonellm-mini-alpha-2-mlx-4bit \
  --host 127.0.0.1 --port 8766 \
  --temp 0 --max-tokens 512 \
  --chat-template-args '{"enable_thinking":false}' \
  --prefill-step-size 2048 \
  --prompt-concurrency 1 --decode-concurrency 1 --prompt-cache-size 2

Use default_model as the API request's model name. The tested MLX-LM version recognizes the Qwen3-Coder-style tool-call format and returns structured calls. Executing tools requires a separate application. Compatibility with other MLX versions or third-party applications has not been validated. The upstream advertised context limit is retained in the configuration; this conversion was only exercised with prompts up to 8K tokens.

Local measurements and validation

Measured October 15, 2026 on an Apple M5 Pro with 64 GB unified memory:

• Longer warmed-up reply: 77.2 generated tokens/second over 63 output tokens.
• Peak MLX allocation across the initial short generation checks: 20.50 GB. This is not total process RSS or total Mac memory usage.
• Text generation, a two-turn conversation, correction of a tool request, response to a simulated tool result, API streaming, and structured API tool calls passed functional checks. No real tools executed.

API time to first nonempty text chunk, with the model loaded and 2048-token prefill chunks (two trials per case, median):

Prompt tokens	Fresh prompt	Small follow-up with cached prefix
2,048	1.14 s	0.14 s
4,096	2.05 s	0.15 s
8,192	4.31 s	0.17 s

The latency test used synthetic meeting-reference notes. Follow-ups added approximately 35 unprocessed tokens; API usage confirmed prefix reuse. These numbers exclude model loading, transcription, speech synthesis, and tool execution. See the bundled measurements files for raw results. To repeat the latency checks, start the API above on port 8766, then run python benchmark_latency.py. This benchmark script is separate from model inference and does not execute real tools.

Paired public tool-calling evaluation

A fixed 100-case sample of public BFCL V4 data was scored with the pinned upstream Python AST checking functions, comparing this release with an 8-bit MLX reference built from the same original BF16 weights and exact same BF16 exclusions. Inputs were frozen before predictions; both runs used the same native tool template, API, temperature zero, disabled thinking, and 512-token output limit. No tools executed.

Category	This 4-bit release	8-bit reference
Single-tool arguments	38/40	38/40
Choosing among tools	29/30	29/30
All parallel calls in one response	8/20	10/20
Abstaining from irrelevant tools	2/10	3/10
Total	77/100	80/100

All 77 cases passed by 4-bit also passed by the reference. The reference passed three additional cases. Neither run was truncated. This is a small paired diagnostic, not evidence of quality equivalence. The higher-precision reference is itself quantized, not BF16 ground truth or official NVFP4.

Both versions showed weaknesses in abstention and in returning every requested parallel call at once. In two separate post-hoc diagnostics, both completed the second call after receiving the first simulated tool result. Those diagnostic continuations do not change the frozen scores.

This is not an official BFCL leaderboard score or PhoneBench score. Training-set overlap with these public cases is unknown. The upstream scorer and OpenAI schema-mapping functions were used through a small Python-only adapter, not the full BFCL model-handler harness. There has been no direct model-quality comparison against BF16, official NVFP4, GGUF, or alternative layer-selection policies. No fastest-runtime or optimal-quantization claim is made.

See the complete evaluation report, frozen cases, raw outputs, source hashes, and paired disagreements in evaluation/. The BFCL source revision is 6ea57973c7a6097fd7c5915698c54c17c5b1b6c8, with sample seed 42.

To repeat the 4-bit sample, start the API above and run:

python evaluate_tool_calls.py --label my-4bit-run

For the paired reference, reproduce with --bits 8, serve that model with the same settings, and run the evaluator under a new label. Existing run files are never overwritten.

Reproduction

With the tested packages installed:

python reproduce.py --output ./phonellm-mini-alpha-2-mlx-4bit-reproduced

This downloads the pinned original BF16 source (approximately 63.16 GB) and converts it. To use an existing copy of that revision, add --source /path/to/original-bf16-model. The target directory must not exist. The script uses the bundled exclusion recipe and audits saved tensor headers to check BF16 preservation and packed quantized weights. Add --bits 8 to create the higher-precision evaluation reference instead of the 4-bit release. The reference uses approximately 34.57 GB for weights.

The included 4-bit reproduction script was run end to end in a fresh Python 3.12.8 environment using the existing pinned BF16 snapshot. All four saved weight shards, containing 653 tensors, were byte-for-byte identical to the evaluated release. The fresh environment also loaded the release offline from outside the repository and passed a short generation check. These checks used the same M5 Pro Mac; another machine and a fresh source download were not tested. See measurements/release-validation.json for the results.

After downloading, verify the packaged files from the repository directory:

shasum -a 256 -c SHA256SUMS

Attribution and license

PhoneLLM is developed by Daily / Pipecat. The source is a fine-tune of NVIDIA Nemotron 3 Nano 30B-A3B. This conversion performs no additional language-model training.

The upstream release uses BSD 2-Clause terms and includes the underlying NVIDIA Nemotron Open Model License. Both upstream files, LICENSE and LICENSE_NVIDIA.txt, are retained with their attribution notices. Refer to those files and the upstream model card for the applicable terms. Bundled BFCL data and scorer sources carry their upstream Apache 2.0 terms; see evaluation/LICENSE_BFCL and evaluation/benchmark_source/LICENSE.