laion/chatterbox-s3gen-vc-grow

Chatterbox S3Gen VC: score-conditioned adapter and two flow-RL arms

Research release (10 October 2026). This is a Chatterbox audio-to-audio voice-conversion decoder experiment, not Humaneness Voice Small TTS and not a text-to-speech checkpoint. It takes precomputed S3 semantic content tokens from one utterance, a different 5–10-second target-voice reference, and a 99-dimensional emotion/voice-characteristic vector. It generates 24-kHz speech with the upstream Chatterbox S3Gen decoder and HiFT vocoder.

The repository contains the supervised starting point and both completed RL arms. These are alternative branches, not sequential fine-tunes:

File	What it is	Status
checkpoints/base_300h_step0021350.pt	Final rank-128 q/k/v/out LoRA and score projector from a 300.017-hour supervised pilot, including optimizer/RNG state	Complete supervised pilot, not a standalone full model
checkpoints/rl/continue/step-0004000.pt	Continue training the existing rank-128 adapter for 4,000 on-policy groups	Complete, optimizer-bearing RL arm
checkpoints/rl/merged_new/step-0004000.pt	Merge the supervised linear LoRA into its base, then train a fresh rank-128 adapter for 4,000 groups	Complete, optimizer-bearing RL arm; requires --arm merged_new at inference

For safer inference, use the weights-only base_300h_adapter.safetensors, grow_continue_step0004000_adapter.safetensors, or grow_merged_new_step0004000_adapter.safetensors. The older step-750 snapshot remains available for historical comparison. The .pt checkpoints include optimizer/RNG state and use Python pickle; load them only from a trusted source. The SafeTensors files do not contain optimizer state.

Every optimizer-bearing 250-step checkpoint from both completed arms is published under checkpoints/rl/. The machine-readable RL_CHECKPOINTS.json records the verified SHA-256, size, arm and step of each uploaded file. An entry appears only after a complete local checkpoint and verified Hugging Face upload. The publisher source is code/grow_v1/upload_checkpoints.py.

What was trained

The upstream ResembleAI/chatterbox S3Gen weights are required separately, pinned here to revision 5bb1f6ee58e50c3b8d408bc82a6d3740c2db6e18 and file s3gen.safetensors (SHA-256 2b78103c654207393955e4900aac14a12de8ef25f4b09424f1ef91941f161d4e). The upstream model is MIT-licensed. Our checkpoints contain only the trained adapter, score projector and training state, not a copy of upstream weights.

The supervised pilot froze the S3 semantic tokenizer (25 tokens/s, 6,561 symbols), CAMPPlus speaker encoder (192 dimensions), most of S3Gen and the HiFT vocoder. It trained rank-128 LoRA on 224 decoder-attention q/k/v/out projections plus a zero-initialized MLP that injects 40 emotion + 57 VoiceNet + genuineness + vocal-burst-blend = 99 scores into the speaker condition. The 300.017-hour selection has 111,665 unique training UIDs, including about 30.003 hours of synthetic sidecar material. It starts from the S6/tier-2 training selection, not from the public Chatterbox T3 language model. See REPRODUCING.md for the data and schedule.

The second checkpoint starts from that completed supervised adapter and continues it with a GROW-inspired group-relative flow objective. It is not standard discrete-token GRPO. Each of 4,000 selected source/target conditions generates a group of eight on-policy samples with 10 Euler flow steps. A frozen Humaneness Ears Medium model (revision 818506970d93c809a3295a0a02c40dee8ff2bfc3) scores each sample. The reward is 55% AudioBox Production Quality and 45% negative mean-squared error against the source recording's top three Ears emotion axes. The two terms are standardized separately within each group of eight before mixing. Their signed group advantages weight the flow matching loss, with a frozen-policy velocity anchor of 0.025. The 4,000-step schedule uses 200 warm-up steps, 5e-6 peak LR, cosine decay to 1e-6, AdamW and optimizer-bearing checkpoints every 250 steps. Both arms completed step 4,000.

Matched held-out comparison at step 4,000

The evaluation uses 40 UID-disjoint source cases, three matched seeds each (120 outputs per model), the same target voice per case, and the same input conditions. Higher Production Quality and target-speaker cosine are better; lower emotion MSE is better.

Model	Ears AudioBox Production Quality ↑	Top-three source-emotion MSE ↓	Target CAMPPlus cosine ↑
Supervised 300 h baseline	7.2080	0.5151	0.6188
RL continue, step 4,000	7.4549	0.5207	0.6046
RL merged_new, step 4,000	7.4392	0.5258	0.6072

The two RL arms are close. continue leads slightly in Production Quality and emotion error; merged_new leads slightly in speaker cosine. Relative to the supervised baseline, both improve their trained-on quality score but slightly worsen emotion error and speaker similarity. This does not establish an audible improvement. The machine-readable results and protocol are in evaluation/heldout_step4000_summary.json and REPRODUCING.md. WER and an independent naturalness metric were not measured in this comparison.

Crucial interpretation

• There is no demonstrated independent quality gain for either RL arm. The held-out comparison above is complete, but its quality and emotion metrics come from the same Ears model used as the training reward.
• Reward is not speaker similarity, WER, or an independent naturalness measure. AudioBox PQ is one Ears output axis, not a generic combined aesthetics score. The top-three emotion error is computed with the same frozen Ears model used during training.
• The source pool mixes recorded and synthetic clips. The top-100 selection for each of 40 emotion axes is not guaranteed to consist of real recordings. Rare axes can have modest scores even among their strongest 100 examples.
• The supervised and RL holdouts are UID-disjoint from their respective training selections, not proven speaker-disjoint.

Inference from prepared data

This release provides an explicit, checked inference entry point for the prepared Parquet schema. It is not yet a one-command arbitrary-WAV VC pipeline: a source row must contain S3 tokens, and both rows must contain mel, CAMPPlus and 99 scores in score_order_and_preparation.json order. The included code/prepare.py shows how those fields were computed. Audio is never expanded into millions of loose files.

pip install -r requirements.txt  # install a CUDA-enabled PyTorch build first
hf download ResembleAI/chatterbox s3gen.safetensors \
  --revision 5bb1f6ee58e50c3b8d408bc82a6d3740c2db6e18 \
  --local-dir upstream_chatterbox
export CHATTERBOX_S3GEN_WEIGHTS="$PWD/upstream_chatterbox/s3gen.safetensors"
export CHATTERBOX_BASE_ADAPTER="$PWD/checkpoints/base_300h_adapter.safetensors"
export PYTHONPATH="$PWD/code:${PYTHONPATH:-}"
python code/grow_v1/infer_packed.py \
  --pool /path/to/prepared_pool.parquet \
  --score-order-json score_order_and_preparation.json \
  --source-uid YOUR_SOURCE_UID --target-uid YOUR_TARGET_UID \
  --output-wav base.wav --seed 1901
python code/grow_v1/infer_packed.py \
  --pool /path/to/prepared_pool.parquet \
  --score-order-json score_order_and_preparation.json \
  --source-uid YOUR_SOURCE_UID --target-uid YOUR_TARGET_UID \
  --arm continue \
  --rl-checkpoint checkpoints/grow_continue_step0004000_adapter.safetensors \
  --output-wav rl_continue.wav --seed 1901
python code/grow_v1/infer_packed.py \
  --pool /path/to/prepared_pool.parquet \
  --score-order-json score_order_and_preparation.json \
  --source-uid YOUR_SOURCE_UID --target-uid YOUR_TARGET_UID \
  --arm merged_new \
  --rl-checkpoint checkpoints/grow_merged_new_step0004000_adapter.safetensors \
  --output-wav rl_merged_new.wav --seed 1901

The source provides semantic content and time-varying emotion/style scores; the target provides its CAMPPlus speaker vector, acoustic prefix and the explicitly selected static timbre/age/gender score dimensions. Both commands use the same seed and target for a controlled comparison. The training code, resume commands, required containers and known reproduction limits are in REPRODUCING.md.

For merged_new, --arm merged_new is essential: inference first folds the supervised linear LoRA into the upstream base, then loads the new RL adapter. Loading those weights as continue gives the wrong model. Full .pt files also work but contain Python pickle: use trusted downloads only.

Files and licensing

code/ contains source selection, packed preparation, supervised training, score mixing, reward preparation, flow-RL training, inference, evaluation and generic Slurm templates. score_order_and_preparation.json fixes the exact 99-axis order and token/mel rates. Checkpoint SHA-256 hashes and validation commands are in REPRODUCING.md.

New LAION-authored adapter weights, code and documentation in this repository are released under CC BY 4.0. The dependency on upstream Chatterbox retains its separate MIT license and attribution. Use only audio and voices you have rights to process; this research snapshot does not establish identity fidelity or suitability for impersonation.