A Quantized Native Runtime for On-Device Semantic Audio Generation
Abstract
A dependency-free runtime enables efficient text-to-music generation on embedded devices through quantization and activation steering while maintaining audio quality.
Semantic audio applications increasingly require controllable generation on commodity and embedded hardware rather than through framework-heavy datacenter stacks. We present aria, a dependency-free native runtime that runs the complete text-to-music pipeline of Stable Audio~3 (SA3) on ordinary GPUs, CPU-only machines, and a Raspberry~Pi~5, with no Python or deep-learning framework underneath. Our main contribution is a study of quantization: running the model at lower numerical precision to fit tight memory budgets, saving memory in place rather than adding to it. Because the runtime owns every internal tensor, it also exposes activation steering, a low-cost way to steer what the model generates. We judge the quality cost with three independent measures of the output (prompt adherence, overall audio quality, taste preservation), each compared against the ordinary variation between random seeds. Eight-bit precision shows no measurable quality loss on any measure while sharply cutting memory, and it is the fastest mode on the GPU; four-bit adds a small, bounded cost but shrinks the footprint enough to run the 1.2-billion-parameter model on an 8\,GB Pi. Against the official implementation, aria matches or exceeds generation speed and starts about seven times faster. A case study of the steering interface generates music carrying taste associations (sonic seasoning), with genuine but bounded control for a subset of attributes. These results make a compact, quantized runtime with built-in control a practical basis for on-device semantic audio in Internet-of-Sounds settings. The aria runtime is released at https://github.com/matteospanio/aria.
Community
We release aria, a native inference engine for audio diffusion models, written from scratch in C with no third-party dependencies. It runs Stable Audio 3 (small-music and medium) end to end, from a text prompt to a stereo WAV, with no Python or deep-learning framework underneath.
It is CPU-first by design and runs on ordinary laptops, on old and cheap NVIDIA cards through an optional CUDA backend, and on an 8 GB Raspberry Pi 5. The point is to make capable open-weight audio models usable on hardware people already own, where the usual PyTorch stack is impractical.
Running a full text-to-music pipeline on a Raspberry Pi 5 with no Python underneath is the kind of engineering constraint that actually forces good decisions. Most quantization papers stop at "we quantized it and it still works" — this one owns the whole stack, so the memory savings are real and additive rather than theoretical. The dependency-free native runtime angle matters more than the numbers: if your model needs conda, pip, and a GPU driver to run, it doesn't run on-device, period. I'd be curious how the int8 path holds up under sustained inference on the Pi — thermal throttling tends to be the silent killer in these setups, not the model itself. Either way, this is the kind of paper that makes me want to try it before the weekend is over.
Thanks! I actually tested it under sustained inference on a Raspberry Pi 5 with both passive and active cooling, running sessions of about 20-30 minutes each. With active cooling, the system completed the runs without entering thermal throttling. With passive cooling alone, however, we did observe thermal throttling... it was the first bottleneck we encountered during development. So I completely agree that thermal management is a critical aspect of real on-device deployments.
In my opinion, this is one of those cases where "edge" is still somewhat a proof of concept, but it demonstrates that running these models on extremely constrained hardware is feasible. The main challenge is that the architecture of these models benefits greatly from parallelism, so optimizing the kernels specifically for ARM provided a substantial performance boost on the Raspberry Pi. For truly massive inference workloads, I think a distributed setup with multiple Raspberry Pis would probably be necessary (I would start testing with 4 devices, I suspect it could give promising results).
The main reason I started this project, however, was actually to make inference on an old NVIDIA GT 1030 with 2 GB of VRAM, which is practically unusable for modern models. With this implementation, I can run a state of the art model on hardware I already own, at a very low cost, especially considering the much lower power consumption compared to modern GPUs. I would also love to optimize the implementation for Apple processors, but I currently do not have access to that hardware. Hopefully, I will be able to run some tests on Apple Silicon in the future.
In any case, I would definitely encourage you to try it out and let me know how it performs on your setup!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- DEMON: Diffusion Engine for Musical Orchestrated Noise (2026)
- A 35B Hybrid-Attention Mixture-of-Experts Model on a 6GB 2011 GPU: Hand-Written 4-bit CUDA Inference for Fermi (2026)
- FLASH-MAXSIM: IO-Aware Fused Kernels for Late-Interaction Retrieval (2026)
- Phase Matters: Characterizing Heterogeneous Vision-Language Inference on a Mobile SoC (2026)
- LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation (2026)
- Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs (2026)
- Still: Amortized KV Cache Compaction in a Single Forward Pass (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2607.08526 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper