executorch-community/Phonon-31M-EN-ExecuTorch-WebGPU

Phonon 31M (English) for ExecuTorch WebGPU

This repository contains an ExecuTorch conversion of gradium/phonon-31m-en for the ExecuTorch WebGPU backend, so the model runs entirely in a web browser on the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground.

• Base model: gradium/phonon-31m-en
• Format: ExecuTorch .pte programs delegated to the WebGPU backend
• Quantization: fp32 programs; the text embedding table is stored as bf16 (its values are exactly representable in bf16)
• Total size: 150.7 MB

Note: The voices are from gradium/phonon-voices (Apache-2.0); each is stored here as its 125-frame voice prompt projected into the model.

About the original model

An English text-to-speech model that streams natural 24 kHz speech while it is still generating. A flow-matching transformer predicts an audio latent for each 80 ms frame from the text and a voice prompt, and the Mimi codec's decoder turns the latents into audio.

• Developed by: Gradium (builds on Pocket TTS and the Mimi codec by Kyutai)
• 31M parameters (excluding the audio tokenizer)
• English
• Mono 24 kHz output, one 80 ms frame per step

For training data, evaluation results and full details, see the original model card: gradium/phonon-31m-en.

Intended use and limitations

On-device speech for assistants, screen readers and other accessibility tools, games and interactive apps. Do not use it to impersonate a real person, to deceive listeners about who is speaking, or to clone a voice without the speaker's consent; tell listeners when speech is synthetic where that matters to them.

English only: text in other languages is read with the wrong pronunciation. The official runtime normalizes numbers, currency, dates, emails and so on before synthesis; that step is not part of the model and must be done by the application. Long inputs are spoken sentence by sentence, so prosody does not carry across the whole text.

What changed in this conversion

• Exported as two ExecuTorch programs: the flow-matching transformer with its 1-step sampler and end-of-speech score, and the streaming Mimi decoder
• A few operations are expressed with equivalent ones the WebGPU backend supports (layer norm, mean, 1-D transposed convolution, ELU, SiLU); each was checked against the original model
• Weights stay fp32; the text embedding table is stored as bf16, which is exact for these weights
• Context capped at 512 positions (voice prompt, text and audio frames)

Inputs and outputs

• phonon_flow_lm.pte: embeddings for 1 to 128 positions plus the start position and noise -> the next audio latent and the end-of-speech score; the attention cache lives inside the program
• phonon_mimi.pte: one latent plus 15 streaming state tensors -> 1,920 samples at 24 kHz and the updated state
• phonon.json documents the full contract: text embedding (phonon_text_embed.bf16.bin), the first-step embedding, the voices (voice_*.f32.bin), noise, the end-of-speech rule and the frame budget; tokenize with tokenizer.json

Files

File	Size	SHA-256
phonon.json	8.4 KB	e8fa6604769aeb3b…
phonon_bos_embed.f32.bin	2.0 KB	a4464c50596adea2…
phonon_flow_lm.pte	106.7 MB	178698a6c4539ba4…
phonon_mimi.pte	39.4 MB	d7acdf2d93ba4897…
phonon_text_embed.bf16.bin	3.9 MB	7c6f3394fc86b55e…
tokenizer.json	271.5 KB	691a00068fa7b989…
voice_default.f32.bin	250.0 KB	d8a8ffc97e46c033…
voice_en_us_easton.f32.bin	250.0 KB	1d6ec67cbd3a45b2…

Full checksums are in manifest.json. The playground verifies them after download.

Running it

These programs were exported with the ExecuTorch WebGPU backend (backends/webgpu) and run in a browser (headless Chromium, WebGPU on SwiftShader) with the WebAssembly runtime build used by the ExecuTorch WebGPU Playground, where the full text-to-speech loop matched the CPU programs frame for frame (audio cosine 0.9999997). That runtime build is published in executorch-community/ExecuTorch-WebGPU-Runtime; the playground is the easiest way to run them. It downloads these files once, caches them in the browser, and runs everything locally.

Requirements: a browser with WebGPU enabled.

Provenance

• Upstream: gradium/phonon-31m-en at revision 52433c46975bfa13144c56204c9f4ee1c5b148aa (official)
• Exported with: executorch 1.5.1+cpu, torch 2.14.1+cpu, torchao 0.18.0+cpu, transformers 5.19.0 (executorch git 3b6068392324)
• Reproduce: python recipes/phonon-31m-en.py --out out/phonon-31m-en (recipe in export/)
• Re-running the recipe produces byte-identical files.

License

This is a derivative of the upstream model and is distributed under the same terms (apache-2.0); see LICENSE. Modifications (conversion to the ExecuTorch format) by Meta Platforms, Inc.