LFM2.5-Audio Delegate — MLX Streaming Speech Adapter (Experimental)

This English speech-to-speech research release adds learned delegation to LiquidAI/LFM2.5-Audio-1.5B on Apple silicon. A spoken request can receive a local spoken answer or produce delegate(query="…") for a stronger text model. The same speech model then speaks the returned text incrementally, while more text is arriving.

Michael Chen's project contributes the LoRA adapter, MLX inference/runtime work, data preparation and evaluation, built with Claude Code. Liquid AI supplied the pretrained speech architecture and base weights. The measured gain is routing under a narrow protocol; factual answers and room-microphone routing remain weak. This is an experimental half-duplex assistant, with no live-human conversation validation or human voice-quality study.

The runnable release remains v0.9.1 (bb911378fd7015bd5e4d8bc5c06b34851a842738). This main-page revision changes documentation only. The complete earlier card is preserved verbatim as history, including superseded wording and failures; use the qualifications below when interpreting it.

Model details

Liquid AI's pretrained path connects a FastConformer audio encoder through an MLP adapter to the LFM2 shared core, then a depthformer audio head and causal detokenizer producing 24 kHz speech. The shared core handles both local replies and structured tool calls. External answer text is forced into that core's text slots to generate speech through the same audio head.

Only rank-16, alpha-32 LoRA on 92 shared-core linear layers (11.11 million parameters) was trained here, for 500 steps. Base weights, audio encoder, encoder adapter, audio head and detokenizer stayed frozen. This is an adapter to an existing integrated architecture, not pretraining of a new speech model. Cactus Whistle is the ASR evaluator and part of a pipeline baseline; it does not generate this model's voice.

The runtime supplies sentence-aligned segments of roughly 35 words, call-format recovery, cancellation and recent-answer context. These are software behaviors, not learned abilities. Delegations receive the last three spoken-answer text entries. After cancellation, the stopped entry is an even-pacing estimate of spoken words; it is not a transcript. The model's native context can still contain text generated ahead of its played audio.

Release component	Value
Base	LFM2.5-Audio-1.5B at `c362a0625dfe45aa588dce5f0ada28a7e5707628`
Retained checkpoint	E1, `adapter/adapter.safetensors`, 44,454,313 bytes
Adapter SHA-256	`97bc6ef4e7d7375e1c959179f820c84ce13ff7646ba571566d93c35d77d72d10`
Runtime	Python 3.12, MLX, `mlx-audio==0.5.7`; Apple silicon
Measured hardware	One Apple M4 Mac mini, 10-core GPU, 24 GB unified memory

How to get started

Use an Apple silicon Mac with Python 3.12. Download the pinned release, then install its inference environment:

pip install -U huggingface_hub
hf download mchen04/lfm2.5-audio-1.5b-mlx-delegate --revision v0.9.1 --local-dir lfm25-delegate
cd lfm25-delegate
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.lock
pip install -e .

Run this from the downloaded directory. The first base-model download needs network access and about 3 GB; these two independent example turns then run locally. The delegation branch prints a call without contacting a provider.

from huggingface_hub import snapshot_download
import soundfile as sf
from iss.engine import Engine, Conversation, GenConfig, generate_turn, system_prompt

base = snapshot_download(
    "LiquidAI/LFM2.5-Audio-1.5B",
    revision="c362a0625dfe45aa588dce5f0ada28a7e5707628",
)
e = Engine(base, adapter="adapter/adapter.safetensors")
for path in ("data/scenarios/audio/d1_capital.wav", "data/scenarios/audio/d1_trip.wav"):
    conv = Conversation(e, system=system_prompt())
    wav, sr = sf.read(path, dtype="float32")
    conv.add_user_audio(wav, sr)
    res = generate_turn(conv, GenConfig(seed=0))
    if res.route == "delegate":
        print(path, "->", res.tool_call)
    else:
        print(path, "->", e.decode([t for t in res.text_ids if t != 130]))
        sf.write("answer.wav", e.decode_frames(res.frames), 24000)

The input files are synthetic Kokoro development utterances. The local example writes answer.wav; the trip example exercises learned delegation. requirements-full.lock records the broader research environment, including data generation, training and automated judges.

For the accepted open-speaker mode, run the bridge on the control Mac with microphone access and an already signed-in compatible streaming CLI. The published bridge used Claude Code (claude-sonnet-5-5). Replace <mac>, <dir> and <base> with the inference Mac's SSH host, downloaded release directory and base-model directory:

python3 iss/bridge.py --ssh <mac> --dir <dir> --mic device --runtime-args "--model <base> \
  --adapter adapter/adapter.safetensors --mic stdin --out device \
  --half-duplex --guard-ms 1500 --min-voiced-ms 300"

Listening is off during replies and for a 1.5-second guard, then waits for 600 ms of quiet. A 10-second fallback after the guard re-arms even without quiet; continuing speech can then start a turn. Wait for [listening] and repeat anything said while listening was off. Type c + Enter to cancel: playback stops, active delegation closes, and context rolls back to played audio frames. Voice interruption through open speakers is not accepted. See runtime setup and options.

Listen to an existing demonstration

Play or download the R3 demo (WAV, 3 min 0.2 s)

Caption and accessibility summary: this is the recorded output-callback audio from actual Mac mini speaker playback, driven by prerecorded synthetic FILE input and a real external text-model stream. It is not a live-human conversation or an acoustic re-recording of the room. The sequence is a local capital answer, a local Lisbon-history reply cancelled after generation, a delegated family trip plan, and a delegated follow-up about the plan's relaxing day. The history reply contains false claims; the stronger model subsequently corrects them before planning. This demonstrates continuity and streaming, not reliable historical or travel advice. The file includes silence between replies. This summary describes the published R3 report, not a verbatim transcript.

Training data

E1 used 2,000 synthetic spoken conversations: 1,100 delegate and 900 local targets, including 300 two-turn dialogs. Qwen3-8B generated category/topic request texts and follow-ups; GSM8K supplied math problems and MBPP supplied programming tasks. Kokoro-82M voiced text using 18 training voices, with separate development/test voices. Local targets were the base model's own interleaved answers; delegate targets were structured calls with a request or self-contained rewrite. Labels followed source categories, not item-by-item difficulty judgments.

Training topics, voices and evaluation questions were separated. VoiceBench sd-qa human recordings were development/test only for retained E1 (British-accent dev, US-accent test); bbh synthetic speech was test only. E1 contains no real-human training speech. The rejected E2 experiment added human recordings but is not the released checkpoint. See provenance, source revisions and split rules and the frozen manifest. YAML datasets list training sources only.

Training procedure

E1's configuration uses AdamW, effective batch 8 through accumulation, learning rate 1e-4, 20 warm-up steps, cosine decay, zero dropout and seed 0. Two epochs produced 500 steps. Training supervised text and codebook-weighted audio cross-entropy, with the route-decision token weighted ×3.

The retained run took 3,648 seconds, peaking at 23.5 GB memory (swap peaked at 5.2 GB), on the M4 Mini. Energy and carbon were not metered. Process caveat: numeric quality-loss bounds were fixed before substantive E1 training, but several early engine/precision choices preceded those bounds; see LEDGER P3c/P3d. No new training accompanies this documentation revision.

Evaluation

These are historical frozen held-out results, not new evaluations of the latest runtime. Baselines and E1 used the same test items; development data guided selection. Intervals below are 95% item-bootstrap intervals (2,000 resamples). Handoff success requires the correct route and, for delegate items, a valid call with query WER ≤0.3 against the reference. It does not measure general answer correctness.

Measure and sample	Untrained base	E1	Paired E1 − base, 95% CI
Handoff success, n=812	0.537 [0.504, 0.573]	0.9384 [0.921, 0.954]	+0.401 [0.365, 0.437]
QA reference containment, n=200 real-speech questions	0.195 [0.140, 0.250]	0.205 [0.150, 0.265]	+0.010 [−0.020, +0.045]
Speech intelligibility WER, n=250 pool; complete turns only	0.168	0.132	−0.035 [−0.056, −0.013], n=157 pairs
DNSMOS OVRL, n=250	3.347	3.351	+0.004 [−0.011, +0.019]
Context-follow-up handoff, n=101	0.564	0.535 [0.436, 0.634]	−0.030 [−0.119, +0.059]

Adding a routing policy to the untrained prompt still scored 0.537 handoff success. The Cactus Whistle→Needle3 pipeline scored 0.516 [0.483, 0.550] on the same 812 requests; this narrow delegation task is not a general ranking of those components. E1 route-decision accuracy was 0.985 [0.977, 0.993].

WER compares Whistle's recognition of generated speech with the model's decoded text. The displayed per-model means use 179 base and 174 E1 complete turns; the paired difference uses their 157-item intersection. DNSMOS is an automated quality predictor, not a human rating. QA shows no demonstrated factual gain, and context shows no learned improvement. The runtime supplies context separately. See all result tables and evaluation summaries.

Current runtime performance (Mac mini M4, 24 GB)

Latest recorded R3 regression: four FILE-input turns, real API delegation and actual Mini DEVICE playback; 7/7 registered checks passed, with no worker errors, output gaps or underflows. MLX peak was 3.637 GB and process peak RSS 3.945 GB; these are separate measurements, not total-system RAM.

End-of-turn decision → first reply-carrying output callback	R3 values
Local replies, n=2	0.553 / 0.590 seconds
Delegated replies, n=2	8.596 / 3.010 seconds

These callback timings exclude the device's reported approximately 32 ms output latency. The older iss.perf_report speaker-time estimate adds it: corresponding R3 values are approximately 0.585/0.622 and 8.628/3.043 seconds. Neither is a measured acoustic arrival time. End-of-turn silence precedes these intervals; they do not start at the user's last voiced frame. The slow delegated reply included 6.62 seconds awaiting first external text.

R3 cancellation retained 431/485 audio frames; the next delegation received the expected estimated 599/677-character prefix. The trip and follow-up received 31/59 and 17/57 text chunks after playback began. R3 report and criteria explain this limited demonstration.

Physical-microphone evidence (Q/R): a real control-Mac microphone heard prerecorded Kokoro speech from a loudspeaker, transported in 20 ms blocks. R's two sessions yielded 10 scripted requests→10 turns, suppressed four utterances during replies and demonstrated four local typed cancels. These were not live-human tests. Across the four Q/R sessions, the trip request was wrongly answered locally in 3/4.

Historical sustained evidence: final_sustained64_g6 ran 64 turns over 26.9 minutes with FILE input, API and DEVICE playback. It had 63/64 correct routes and 7/7 delegations speaking before text finished, but one local response had gaps totaling 0.04 seconds, with no device underflows. MLX peak was 3.919 GB; RSS 4.062 GB. It was not rerun after the later half-duplex/context changes. Recompute its stored metrics without generating a new benchmark:

python -m iss.perf_report benchmarks/sessions/final_sustained64_g6

Uses

Use this for research into local speech interaction, delegation, streamed speech generation and turn-taking on Apple silicon. A downstream text model can serve the tool call, subject to its own quality, safety and service terms. Production/unattended use, safety-critical advice and reliance on spoken facts are out of scope.

Bias, risks and limitations

Local factual and arithmetic errors can sound fluent. Routing follows a narrow category policy and misses obscure factual uncertainty. Synthetic training speech and limited recorded-accent coverage do not establish performance for other languages, accents, ages, noise or spontaneous speech. Long-context calls can become malformed; runtime recovery is not proof of robust understanding. No added safety training or runtime content filter was provided. No human listening study or live-human conversation validation was run.

Privacy and network boundaries

Audio understanding and generation run locally; the two-Mac microphone path carries audio over SSH. Delegation is not all offline: query text and recent-answer context go through the signed-in CLI to the external provider. Review that boundary before using personal content. Research logs can contain conversation text. Published sessions with unknown room speech are sanitized; raw private recordings and transcripts are excluded. The linked R3 demo uses safe scripted inputs and generated output.

Research status: interruption through open speakers (closed — half-duplex accepted)

Full-duplex experiments remain rejected/off by default. P's augmentation checkpoint was not retained; Q, R and R2 keep their registered failures, including room-routing failures and R2's F7 protocol conflict. R3 does not regrade them. test_feed_log retains an intermittent bf16 tolerance failure: historical identical-source trials passed 3/5, 2/5 and 2/5, with matching argmax predictions; its bound was not relaxed.

Files in this repository

The append-only LEDGER preserves protocols, commands, failed attempts and decisions. ACCEPTANCE maps complementary demonstrations; DECISIONS records architecture choices. Frozen data/sets/ contains source IDs/texts, while most evaluation audio must be regenerated using data_build/; demo/scenario audio is included.

The archived full card retains the earlier detailed history, result appendix and attribution. All existing release tags, artifact paths and the old repository URL remain available.

License and attribution

This derivative uses the LFM Open License v1.0, including its US$10 million annual-revenue commercial-use restriction; consult the license's definitions and conditions. It is not an unrestricted MIT model. Full attribution covers Liquid AI, NVIDIA/NeMo, Kyutai Mimi, MLX/mlx-audio, Cactus, Qwen, Kokoro and dataset contributors. GSM8K is MIT; MBPP is CC-BY-4.0; Qwen, Kokoro and VoiceBench carry Apache-2.0 terms. Component and provider terms still apply.

Citation and contact

@misc{chen2026lfm25delegate,
  author = {Chen, Michael},
  title = {LFM2.5-Audio Delegate: MLX Streaming Speech Adapter},
  year = {2026},
  url = {https://huggingface.co/mchen04/lfm2.5-audio-1.5b-mlx-delegate},
  note = {Experimental LoRA adapter and runtime; release v0.9.1}
}

Maintainer: Michael Chen / mchen04. Report reproducibility issues through this repository's Community tab, including revision and hardware without private audio or credentials.