matbee/LFM2.5-Audio-1.5B-ONNX-cuda-gqa

LFM2.5-Audio-1.5B-ONNX (CUDA-clean GQA patch + sampling-capable unrolled depthformer)

This is a derivative of LiquidAI/LFM2.5-Audio-1.5B-ONNX containing only the depthformer graph files that differ from upstream. It does NOT re-host the large .onnx_data sidecar tensors — clone the upstream repo first and overlay these files.

What's different

The upstream depthformer's GroupQueryAttention (com.microsoft) nodes:

• Wire cos/sin tensors as required inputs
• Don't set do_rotary / local_window_size / softcap attributes
• FP16: routes to CUDA EP, dereferences cos/sin regardless of do_rotary → cudaErrorIllegalAddress at /layers.0/gqa
• Q4: falls back to CPU EP with 372 MemcpyToHost / MemcpyFromHost nodes (slow)

All four files in this repo:

• Clear the cos / sin GQA inputs (they were dead weight — the upstream graph pre-applies the RoPE layout transform via rearrange_interleaved_to_split)
• Explicitly set do_rotary=0, local_window_size=-1, softcap=0.0
• Drop the now-unused gqa_cos / gqa_sin initializers
• Apply to either the rolled (If-subgraph) or unrolled-8-codebook graph

Plus two new variants do in-graph top-k temperature sampling via the Gumbel-max trick — no CPU↔GPU round-trips per codebook step, fully sampled (NOT greedy argmax).

Files

Model artifacts (onnx/)

File	Size	Description	Replaces / supersedes
onnx/vocoder_depthformer_fp16.onnx	46 KB	Patched base FP16 depthformer (logits output)	upstream vocoder_depthformer_fp16.onnx
onnx/vocoder_depthformer_fp16_argmax.onnx	46 KB	Patched FP16 argmax variant (token_id output, greedy)	upstream vocoder_depthformer_fp16_argmax.onnx
onnx/vocoder_depthformer_fp16_unrolled_sample.onnx	391 KB	NEW — 8 codebook steps fused + in-graph top-k Gumbel-max sampling, FP16	(no upstream equivalent)
onnx/vocoder_depthformer_q4.onnx	53 KB	Patched base Q4 depthformer (logits output)	upstream vocoder_depthformer_q4.onnx
onnx/vocoder_depthformer_q4_argmax.onnx	54 KB	Patched Q4 argmax variant	upstream vocoder_depthformer_q4_argmax.onnx
onnx/vocoder_depthformer_q4_unrolled.onnx	380 KB	Patched Q4 unrolled-greedy variant	upstream vocoder_depthformer_q4_unrolled.onnx
onnx/vocoder_depthformer_q4_unrolled_sample.onnx	420 KB	NEW — same as FP16 unrolled+sample, but Q4	(no upstream equivalent)

All files use the same external-data sidecars (.onnx_data) as upstream — drop them in place on top of an upstream clone.

Reproduction scripts (tools/)

File	Description
tools/patch_depthformer_gqa.py	Rewrites GroupQueryAttention nodes to clear cos/sin inputs + set do_rotary=0, local_window_size=-1, softcap=0.0. Applied to produce the patched vocoder_depthformer_fp16{,_argmax}.onnx here. Works on any precision/variant.
tools/build_lfm25_depthformer_unrolled.py	Builds the greedy unrolled variant from an *_argmax.onnx source (8 codebook steps fused). Kept for reference / fallback — superseded by the sample variant for production.
tools/build_lfm25_depthformer_unrolled_sample.py	Builds the sampling unrolled variant from a base (logits-output) graph. Adds in-graph top-k temperature sampling via the Gumbel-max trick. Produced the two _unrolled_sample.onnx files in this repo.

How to rebuild from upstream

# 1. Clone upstream (gets the heavy .onnx_data sidecars)
hf download LiquidAI/LFM2.5-Audio-1.5B-ONNX --local-dir LFM2.5-Audio-1.5B-ONNX

# 2. Clone this repo (gets the tool scripts and the pre-built artifacts)
hf download matbee/LFM2.5-Audio-1.5B-ONNX-cuda-gqa --local-dir cuda-gqa
pip install onnx numpy

# 3. Patch the upstream FP16 graphs (the .onnx_data sidecars are NOT modified)
python cuda-gqa/tools/patch_depthformer_gqa.py \
  --input  LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16.onnx \
  --output LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16.onnx
python cuda-gqa/tools/patch_depthformer_gqa.py \
  --input  LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16_argmax.onnx \
  --output LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16_argmax.onnx

# 4. Build the sampling-capable unrolled variants
python cuda-gqa/tools/build_lfm25_depthformer_unrolled_sample.py \
  --input  LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16.onnx \
  --output LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16_unrolled_sample.onnx \
  --temperature 1.0 --top-k 4
python cuda-gqa/tools/build_lfm25_depthformer_unrolled_sample.py \
  --input  LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_q4.onnx \
  --output LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_q4_unrolled_sample.onnx \
  --temperature 1.0 --top-k 4

Or skip steps 3+4 and just overlay the pre-built .onnx files from this repo onto your upstream clone.

Sampling math (the new _unrolled_sample variants)

For each of the 8 codebook steps:

logits → Mul(1/T)              # temperature scaling
       → TopK(k=4)              # top-k masking, returns (values, indices)
       → RandomUniformLike      # u ~ U(1e-9, 1) on same shape as values
       → Log → Mul(-1) → Log → Mul(-1)   # Gumbel(0,1): g = -log(-log(u))
       → Add(top_k_vals, g)     # perturbed logits
       → ArgMax                 # index in [0, k) — equivalent to categorical sample
       → GatherElements(top_k_idx, argmax)   # recover real token id

This is the Gumbel-max trick: argmax(logits/T + g) where g ~ Gumbel(0,1) is equivalent to sampling from softmax(logits/T). Combined with top-k masking, it gives top-k categorical sampling with no softmax/multinomial ops, all GPU-resident.

Build script: tools/build_lfm25_depthformer_unrolled_sample.py in the remotemedia-sdk project. To rebuild with different defaults:

python tools/build_lfm25_depthformer_unrolled_sample.py \
  --input  vocoder_depthformer_fp16.onnx \
  --output vocoder_depthformer_fp16_unrolled_sample.onnx \
  --temperature 1.0 --top-k 4

(Temperature and top-k are baked into the graph as constants — rebuild to change them.)

Benchmarks

Hardware: NVIDIA RTX 4090, ONNX Runtime via the ort Rust crate (CUDA EP). Workload: woodworks_question.wav (a 4.9s "Can you help me come up with a slogan…" prompt), 5 measured utterances + 2 warmup utterances, firstChunkAudioBatchSize=4, audioBatchSize=12, audioTemperature=1.0, audioTopK=4, maxNewTokens=4096. Quality scored by Whisper-large-v3-turbo coherence (jaccard between expected response set and transcribed response).

Reference: vs Python liquid_audio (LFM2-Audio-1.5B)

Path	TTFA p50	Wall (5 utts)	Whisper jaccard	Avg words/utt	Sample response
Python LFM2AudioNode (reference, Mimi decoder)	614 ms	20.5 s	0.36	5.6	"How about handcrafted excellence every day?"
This repo (full FP16 + _unrolled_sample)	262 ms	26.7 s	0.75	11.8	"Sure thing. Here are a few ideas. Timber talent, room for creativity, strong craftsmanship..."

Same input prompt, same model weights, same conversational target. Both speed (2.3× faster TTFA) and quality (2× higher jaccard) improved.

Three-way precision comparison

Metric	Full Q4	Hybrid (Q4 + FP16 depthformer)	Full FP16
GPU memory (model)	~5.0 GB	~5.1 GB	~7.7 GB
TTFA p50	451 ms	298 ms	1087 ms
TTFA p95	4544 ms	4546 ms	5071 ms
Wall clock (5 utts)	36.9 s	31.1 s	26.7 s
Per-turn total_ms	191–417	168–328	147–388
depthformer_ms	73–181	39–137	36–143
decoder_ms	88–163	92–143	79–200
Whisper jaccard	0.82	0.75	0.75
Avg words/response	15.4	21.4	11.8
Stutter (longest_run)	1.2	1.0	1.0

Memcpy boundaries (ORT graph warnings at session-build time)

Graph	Q4	Hybrid	FP16
decoder	26	26 (Q4)	0
conformer_encoder	34	34 (Q4)	34
audio_detokenizer	12	12 (Q4)	12
audio_istft	4	4	4
vocoder_depthformer_unrolled_sample	372	0 (FP16)	0

The 372 Memcpy nodes in the Q4 depthformer come from Q4-quantized MatMul ops which lack full CUDA kernel coverage in ONNX Runtime — the GQA patch in this repo only fixes the attention boundary. Hybrid mode replaces just the depthformer with FP16 to eliminate those 372 Memcpy from the inner autoregressive loop, while keeping Q4's smaller RAM footprint everywhere else.

Sample responses (same input prompt across all three configs)

Full Q4 (jaccard 0.82, smallest footprint):

• "Sure. Do you prefer a nature-focused vibe or a modern design focus?"
• "How about crafting beauty, one cut at a time?"
• "Create better homes, greener work. Does that capture the vibe you want?"

Hybrid Q4 + FP16 depthformer (best TTFA p50, 21.4 words/utt):

• "Try in Woodcrafted Beauty, where each piece sings from original wood. How does that feel for your brand?"
• "Join the craft lovers with precision and style at Mosaic Basics Woodworking. Does that capture the vibe anyone?"
• "Sure thing. How about something like handcrafted wood, precision made great way..."

Full FP16 (lowest wall clock, cleanest CUDA placement):

• "How about handcrafted woodworking creates your dream? Or handmade woodworking turning home's heart's design?"
• "Sure thing. How about... Handmade with love and power."
• "How about adorn your home with timeless craft?"

Recommendation by VRAM budget

GPU VRAM	Config	Notes
8 GB+	Full FP16 (precision: "fp16")	Lowest wall clock, cleanest CUDA placement
5–7 GB	Hybrid (precision: "q4", depthformerPrecision: "fp16")	Q4's RAM footprint + FP16's depthformer perf; only ~50 MB more than full Q4
<5 GB	Full Q4 (precision: "q4")	Slower per turn (Q4 MatMulNBits CPU fallback) but everything fits

Manifest examples for each config are in the remotemedia-sdk repo under examples/manifests/lfm2_onnx_audio_s2s_bench_*.json.

Usage

Drop these files on top of a fresh LiquidAI/LFM2.5-Audio-1.5B-ONNX clone:

git clone https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B-ONNX
git clone https://huggingface.co/matbee/LFM2.5-Audio-1.5B-ONNX-cuda-gqa
cp -r LFM2.5-Audio-1.5B-ONNX-cuda-gqa/onnx/* LFM2.5-Audio-1.5B-ONNX/onnx/
# now use LFM2.5-Audio-1.5B-ONNX as you would the upstream repo

Or, with hf (Hugging Face Hub CLI):

hf download LiquidAI/LFM2.5-Audio-1.5B-ONNX --local-dir LFM2.5-Audio-1.5B-ONNX
hf download matbee/LFM2.5-Audio-1.5B-ONNX-cuda-gqa --local-dir LFM2.5-Audio-1.5B-ONNX-cuda-gqa
cp -r LFM2.5-Audio-1.5B-ONNX-cuda-gqa/onnx/* LFM2.5-Audio-1.5B-ONNX/onnx/

License

This is a derivative of LiquidAI/LFM2.5-Audio-1.5B-ONNX, distributed under LFM Open License v1.0.

See NOTICE.md for the modification notice required by §4(b).

Acknowledgements

Original model + ONNX export by Liquid AI. Patches & sampling variants by acidhax/remotemedia-sdk.