dernet/acestep-captioner-GGUF

ACE-Step Captioner GGUF

Fast, local music descriptions with llama.cpp.

This is Side-Step's GGUF conversion of the ACE-Step Captioner checkpoints: a model that listens to audio and writes a detailed plain-language description of it. It is useful for captioning music libraries, preparing training data, or inspecting a clip without sending it to a hosted service.

The download

Download both files. They are a matched pair.

File	What it does	Size
acestep-captioner-Q4_K_M.gguf	Writes the description	4.7 GB
acestep-captioner-mmproj-Q8_0.gguf	Listens to the audio	1.5 GB

Q4_K_M is a compact version of the main model. It is the smallest llama.cpp pair that passed our checks.

SHA-256 of the published files:

1c6fb97c2599dc259af70bbfc89a65da24360ba55d7b982366ca32f7e2ae8786  acestep-captioner-Q4_K_M.gguf
77f15ee6e123a85deb2e853ef08d791613e2350c3cd24e033e99e6bad027a8b8  acestep-captioner-mmproj-Q8_0.gguf

Quick start

The easiest supported route is Side-Step's command line:

uv sync --extra captioning
uv run sidestep fetch-engine engine
uv run sidestep caption audio --engine-root engine --models-root models

Put the two GGUF files in models/acestep-captioner-gguf/. The command starts the compatible local engine and captions the audio in audio.

Important: use the supplied engine

These files need the Side-Step build of llama.cpp b10796 and its mtmd overlay. The patch source is maintained separately from the checkpoint. Stock llama.cpp can load the files, but it handled this model's audio input incorrectly in our tests. In particular, some 30-second-multiple clips gained an extra silent segment, which could lead to captions about silence that was not there.

The supplied overlay fixes that and matches the reference model's audio processing closely. On Windows, install the Microsoft Visual C++ 2015-2022 Redistributable if mtmd.dll cannot load.

The full patch sequence brings the audio encoder's average agreement with the original from 0.32 to 0.997.

Recommended settings

Use this prompt:

*Task* Describe this audio in detail

For repeatable results, use deterministic generation:

temperature: 0
top_k: 1
max_tokens: 400
cache_prompt: false
stream: false

Inside the download

The main model is compacted to 4-bit where that saves space; the audio projector stays 8-bit to preserve the part that listens. The figure below is a map of the actual GGUF files, not a simplified architecture diagram.

What to expect

On the tested Windows machine, an RTX 4070 Ti SUPER 16 GB produced a one-minute caption in about 1.5 seconds. The captioner itself used about 6.9 GiB of VRAM; with the desktop running, the whole card peaked near 8 GiB.

Smaller cards were also checked, by keeping fewer model layers on the GPU (-ngl) and the rest in system RAM:

Your card	Setup	One-minute clip	300 clips
12 GB+	Default, all 29 model layers on GPU	1.5 s	7 min
8 GB	-ngl 20, 5.6 GiB of VRAM	5.4 s	27 min
6 GB	-ngl 4, 3.5 GiB of VRAM	12 s	60 min

Every setup passed the same checks as the default, so a slower card gets the same captions. Each needs about 6.6 GiB of system RAM for the server, so 16 GiB of system memory is the practical minimum. Your results will vary with your GPU, desktop load, drivers, and audio length.

What was checked

We compared captions from this pair with the original ACE-Step captioner on 23 one-minute music excerpts. The check asked whether a caption still provides a useful training signal, not whether every phrase is perfect. The final Q4_K_M + Q8_0 pair passed every pre-set comparison gate; a small blind read found no clear overall preference for the original captions.

The shipped pair used 27% of the allowed difference from the reference at its worst checked point. Patch 1, the earlier encoder version, is the single failing row.

In the blind read, four pairs were rated the same and the GGUF caption was preferred twice. The original was never preferred. This is a small sanity check, not a measure of absolute caption accuracy.

Captions matter most late in the training process. This map explains the timesteps used for the comparison; it is evidence about training, not a promise that every individual caption is correct.

This is evidence for the tested CUDA setup only. Vulkan, Metal, other GPUs, other operating systems, human-rated caption accuracy, and a full 300-clip run have not been measured.

For the method, limits, and raw measurements, see How these files were checked. The full admission report, with its hashed protocol and per-timestep tables, ships with the Side-Step source repository.

Attribution and license

ACE-Step provides the captioner checkpoints, which are used with Qwen2.5-Omni-7B. This GGUF conversion is MIT-licensed and retains the ACE-Step notice. The underlying architecture is due to the Qwen team.