mlx-community

/

fish-audio-s2-pro-bf16

like 23

Follow

MLX Community 15.4k

Hi everyone,

For those using this base model on Apple Silicon Macs, here is a local training and fine-tuning pipeline for Fish-Speech S2 Pro using the MLX framework. It was co-authored with an AI developer crew (Fable 5, ChatGPT 5, Claude 4, and Gemini 3).

Since the official training suite is CUDA-only, this pipeline allows you to run the training loop (from raw audio to merged weights) locally on macOS Sonoma (14.0+).

Core Architecture Changes

• Dual-AR Causal Alignment: Resolves the off-by-one training alignment issue in the Fast-AR decoder. Applying the causal shift (seq[:-1] predicting seq[1:]) to both slow and fast loss components ensures the training targets match inference behavior, preventing slurring and pacing issues.
• Trainable Slow Transformer: Unlocks LoRA fine-tuning for the 36-layer slow transformer (semantic tokens) so custom voices can adapt their accent, identity, and prosody.

Features

• Automated Setup: The setup_mlx_training.sh script configures a virtual environment, downloads the base model via Hugging Face Hub, and executes a 1-step dummy training smoke test.
• Memory Autotuning: Configures batch sizes and cache limits based on your Mac's unified memory size to prevent swapping.
• Weight Merging: Merges both slow-AR layers and fast-AR layers back into the base safetensors shards.
• Validation: Generates test clips and computes MFCC similarity stats to objectively score voice quality against the training dataset.

Accessing the Code

The code is hosted publicly at: github.com/brickheadbs/mlx-fish-speech-finetuning

Feel free to open issues or check out the code to run it locally on your own voice datasets.