minori-live/revox-vocoder-1

Revox Vocoder 1.5

Revox Vocoder 1.5 is a pitch correctness aware neural vocoder designed for high-performance inference at 48 kHz. Its architecture, PC-NSF-Vocos, combines pitch correctness awareness (PC), neural source-filter excitation (NSF), and Vocos-style spectral generation. It converts a log-Mel spectrogram and aligned pitch conditions into a complex spectrum for waveform synthesis. The network has 3,413,250 parameters.

Release overview

Version 1.5 exports the checkpoint at 143,000 training steps. It introduces multi-scale Mel reconstruction loss during training and a smaller ConvNeXt feed-forward layer. The 48 kHz acoustic features, pitch inputs, complex-spectrum outputs, and runtime-side iSTFT contract remain compatible with version 1.0.

Setting	Version 1.0	Version 1.5
ConvNeXt feed-forward channels	256 → 768 → 256	256 → 512 → 256
Generator parameters	4,463,874	3,413,250

Multi-scale Mel training in 1.5

The new training run initialized the generator and discriminator afresh and replaced the single-scale Mel reconstruction objective with the equal-weight mean of three log-Mel L1 losses:

FFT / window length	Hop length	Mel bins
1,024	256	128
2,048	512	128
4,096	1,024	128

Each loss compares the generated and target waveforms after magnitude STFT, Slaney Mel filtering, and natural-log compression with a floor of 1e-5. The average has a weight of 45.0 in the generator objective. The different window lengths supervise reconstruction at multiple time and frequency resolutions.

Related multi-scale Mel reconstruction objectives are described in Descript Audio Codec (DAC; Kumar et al., 2023) and used in BigVGAN-v2 (NVIDIA, n.d.). Revox 1.5 uses the three-scale, fixed-128-bin, natural-log configuration above; DAC uses seven scales with scale-dependent Mel-bin counts and base-10 logarithms.

Training retains input-only PSOLA pitch perturbation, RMVPE pitch supervision, multi-resolution STFT loss, adversarial loss, and feature matching. PSOLA changes the audio used to compute input Mel; the target waveform, F0, voicing, and validity remain unchanged. Inference still accepts the single 128-bin Mel input described below, using a 2,048-sample window and a 480-sample hop.

Training was stopped early at 143,000 of 500,000 planned steps after improvements slowed. This is an early-stopped checkpoint, not a claim of full convergence or a general audio-quality improvement over version 1.0.

Files and revisions

File	Purpose
vocoder.onnx	Float32 inference graph with embedded weights
manifest.json	Release version, checksum, configuration, and inference contract
validation.json	Export parity checks and checkpoint evaluation summary

Use revision v1.5 to pin this release. Version 1.0 remains available at 025862deee2003230d9d0fd8f3ad56a5bc09d31c.

Evaluation and limitations

At step 143,000, the mean validation log-Mel L1 was 0.261708 on eight fixed speech clips from one speaker. This metric uses the 2,048-point FFT and 480-sample hop in the feature convention below; it is distinct from the multi-scale training objective.

Pitch control was measured on three of those clips, keeping Mel and masks fixed while shifting F0 and sharing seeded noise across shifts:

F0 shift	Median absolute error	Mean absolute error	Octave error rate
−12 semitones	10.99 cents	158.35 cents	11.89% (61 / 513 frames)
0 semitones	6.71 cents	11.36 cents	0% (0 / 520 frames)

Errors pool jointly reliable voiced frames; an octave error means an absolute error greater than 600 cents. The −6, +6, and +12 semitone checks had no octave errors on this small sample. Lowering F0 by an octave remains a known limitation, despite the low median error. RMVPE is both the training teacher and the pitch evaluator, so these measurements are not an independent pitch assessment.

The release has no singing validation or formal subjective listening evaluation. These fixed speech checks do not establish quality across speakers, languages, singing styles, or large pitch changes. See validation.json for the measured export and checkpoint results.

Design principles: PC-NSF-Vocos

The design brings together three techniques:

1. Frequency-domain generation with Vocos and iSTFT. The network predicts Fourier spectral coefficients at the acoustic frame rate, then uses an inverse short-time Fourier transform to synthesize the waveform. This substantially simplifies the learned waveform-synthesis problem: the backbone predicts compact spectral frames, while iSTFT performs the final waveform reconstruction. This division supports efficient inference.

2. NSF excitation inspired by NSF-HiFiGAN. An F0-driven harmonic-plus-noise source supplies an explicit periodic prior. Its projected features guide the spectral generator toward the requested pitch. This inductive bias is introduced to improve convergence speed by giving the network a structured excitation from which to learn voiced audio.

3. Pitch correctness awareness through perturbed input. The conditioning Mel spectrogram is computed from pitch-perturbed audio, while the F0, voicing, validity masks, and waveform target remain at the original pitch. The model is required to reconstruct the original-pitch audio despite the perturbed Mel input. This encourages it to prioritize the supplied F0 when the pitch implied by Mel disagrees with the requested pitch. At inference, Mel can remain fixed while F0 supplies the desired pitch trajectory. This formulation draws on the information-perturbation principle in Neural Analysis and Synthesis (NANSY), applied here specifically to pitch-perturbed Mel conditioning.

Here, PC describes this pitch correctness awareness; NSF describes the internal excitation prior; and Vocos describes spectral prediction followed by iSTFT synthesis. The combination aims to make pitch following an explicit design objective while keeping the inference network compact.

Network architecture

The generator follows this computation:

log-Mel + log-F0 + voicing + pitch validity
                  │
          frame-rate Conv1d ───────────────┐
                                          │
F0 + voicing + pitch validity              │
                  │                       │
       harmonic-plus-noise source         │
                  │                       │
       strided waveform projection ───────┤
                                          │
                                   sum + LayerNorm
                                          │
                                 8 ConvNeXt blocks
                                          │
                                      LayerNorm
                                          │
                                 linear spectral head
                                          │
                                 log-magnitude + phase
                                          │
                                   real + imaginary
                                          │
                                  runtime-side iSTFT
                                          │
                                    48 kHz waveform

Acoustic conditioning

The input contains 128 Mel channels and three pitch-related channels: log-F0, voicing, and pitch validity. Reliable voiced F0 is encoded as ln(max(F0, 1) / 440); other frames use zero for the log-F0 channel. A one-dimensional convolution with kernel size seven projects these 131 channels to 256 channels.

Harmonic source

The internal source generates eight harmonics, weighted by the reciprocal of their harmonic index and normalized by the sum of those weights. Harmonics at or above the Nyquist frequency are discarded. F0 is interpolated between adjacent reliable voiced frames; voicing and validity use the nearest frame center. A frame contributes periodic excitation only when it is voiced, pitch-valid, and has positive F0.

Periodic excitation has gain 0.1. Explicit unit-variance noise is scaled by 0.003 and added to the source. A waveform convolution with a 2,048-sample kernel and stride 480 projects the source to 256 frame-rate channels. These features are added to the acoustic conditioning projection.

The source waveform is encoded by a learned conditioning projection. The network predicts the complete output spectrum rather than multiplying a source spectrum by a filter.

ConvNeXt backbone

Eight residual blocks operate at the Mel frame rate. Each block contains:

1. A depthwise one-dimensional convolution, kernel size seven, 256 channels.
2. Layer normalization.
3. A linear expansion from 256 to 512 channels, GELU, and a linear projection back to 256 channels.
4. A learned per-channel layer scale and residual addition.

Layer normalization also surrounds the block stack. The convolutions use both past and future context.

Spectral head and synthesis

A linear head predicts log-magnitude and phase for 1,025 one-sided frequency bins. Magnitudes are exponentiated after an upper clamp at ln(100). The real and imaginary outputs are magnitude multiplied by the cosine and sine of phase; the imaginary DC and Nyquist bins are zero.

The ONNX graph ends at these real and imaginary spectra. A runtime reconstructs the waveform with a centered inverse STFT using a periodic Hann window, FFT/window size 2,048, hop size 480, and no STFT normalization. Output length is 480 * T samples, or a shorter requested length for the final partial frame. The network does not apply output amplitude clipping.

ONNX interface

File: vocoder.onnx. ONNX opset: 17. Batch size is fixed to 1; frame and source-noise sample counts are dynamic. T must be nonzero.

Input	Type	Shape	Description
mel	float32	[1, 128, T]	Natural-log magnitude Mel spectrogram
f0_hz	float32	[1, T]	Fundamental frequency in Hz
voiced	bool	[1, T]	Voiced/unvoiced mask
pitch_valid	bool	[1, T]	Reliability mask for pitch decisions
noise	float32	[1, 480 * T]	Unit-variance source noise

Output	Type	Shape	Description
real	float32	[1, 1025, T]	Real component of the predicted spectrum
imag	float32	[1, 1025, T]	Imaginary component of the predicted spectrum

All feature inputs must share the same frame centers. An unreliable pitch estimate uses pitch_valid=False; it is distinct from a reliable unvoiced decision. Supplying fixed noise makes repeated inference reproducible.

Mel feature convention

Input features correspond to mono audio at 48,000 Hz:

Setting	Value
FFT size / window length	2,048 / 2,048 samples
Hop length	480 samples, or 10 ms
Window	Periodic Hann
STFT padding	Centered, constant zero padding
STFT normalization	Disabled
Mel bins / frequency range	128 / 0–24,000 Hz
Mel scale / filter normalization	Slaney / Slaney area normalization
Spectrum used by Mel	Magnitude
Compression	Natural logarithm with floor 1e-5
Frame count	ceil(num_samples / 480)
Frame centers	k * 0.01 seconds

Mel values are ln(max(MelFilterBank @ abs(STFT(audio)), 1e-5)). The centered STFT is truncated to the frame count above. There is no automatic waveform or loudness normalization. Pitch extraction and feature preparation are performed outside the ONNX graph.

Runtime considerations

The network is noncausal. Source phase is continuous within one call and is reset between calls; the graph does not expose streaming state. Independent chunks therefore require runtime handling of context, overlap, and boundaries. Changes in F0 can be supplied per frame while keeping Mel fixed, but large pitch changes or unfamiliar inputs can produce artifacts.

Acknowledgements

This project was inspired by 伊莱莎Ireza.

References

Choi, H.-S., Lee, J., Kim, W., Lee, J., Heo, H., & Lee, K. (2021). Neural analysis and synthesis: Reconstructing speech from self-supervised representations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in neural information processing systems (Vol. 34, pp. 16251–16265). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2021/hash/87682805257e619d49b8e0dfdc14affa-Abstract.html

Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), Advances in neural information processing systems (Vol. 33, pp. 17022–17033). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2020/hash/c5d736809766d46260d816d8dbc9eb44-Abstract.html

Kumar, R., Seetharaman, P., Luebs, A., Kumar, I., & Kumar, K. (2023). High-fidelity audio compression with improved RVQGAN. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in neural information processing systems (Vol. 36, pp. 27980–27993). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2023/hash/58d0e78cf042af5876e12661087bea12-Abstract-Conference.html

NVIDIA. (n.d.). BigVGAN [Computer software]. GitHub. https://github.com/NVIDIA/BigVGAN

OpenVPI. (n.d.). SingingVocoders [Computer software]. GitHub. https://github.com/openvpi/SingingVocoders

Siuzdak, H. (2024). Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, & Y. Sun (Eds.), International Conference on Learning Representations (pp. 25719–25733). https://proceedings.iclr.cc/paper_files/paper/2024/hash/6db0903efdfe9b1bbafb015c10990b78-Abstract-Conference.html

YuFeng926. (n.d.). OpenTune [Computer software]. GitHub. https://github.com/YuFeng926/OpenTune

Attribution and license

The backbone and spectral head follow the Vocos architecture. Revox adds explicit pitch, voicing, validity, and internal harmonic-source conditioning.

The ONNX weights are dual-licensed, at your option, under either:

• Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0); or
• GNU General Public License, version 3 (GPLv3).

You may choose either license and comply with its terms; compliance with both licenses is not required. The SPDX license expression is CC-BY-NC-SA-4.0 OR GPL-3.0-only.

When using the CC BY-NC-SA 4.0 option, attribute Minori Live — Revox Vocoder 1.5 and link to this repository when sharing the weights or adaptations.