Paper: Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (ICLR-2024)

Reviewer comment (Experiments):
Limited explanation of the backbone architecture: I think the way the backbone network is processing the mel-spectrograms is still ambigious. After digging into the code, it seems that the frequency dimension of the spectrogram is used as input channels, and 1D convolution is performed over the time dimension. The receptive field of each timeframe grows by 7 frames in each layer (right?). Therefore, the decoding of one timeframe depends on the adjacent time frames. This is controlled by the receptive field of the last layer. This raises a lot of questions: (a) Why did you choose 7? (b) how increasing the depth of the backbone network (and therefore the receptive field of the last layer over time) affects the performance.

Author response:
You are correct, and we realize that our paper did not clearly state that we employ 1D convolution layers, as mel-spectrograms are often processed as images. We have now made this clearer in the revised manuscript.

These are interesting questions. You are correct about the receptive field expanding with each layer. With a kernel size of 7 and the number of layers in our proposed config, the receptive field extends to approximately half a second. This aligns with other works in the time-domain, such as HiFi-GAN. We argue that increasing the depth, and consequently the receptive field, might not necessarily enhance performance, as vocoders typically model local auditory patterns. However, this should be methodically evaluated, and we believe it might be explored as future work. In this study we focused on comparing Vocos to other popular neural vocoders within the same parameters budget.

Reviewer comment (Novelty):
Low amount of scientific novelty

Author response:
Thank you for your review and for highlighting the relevance of GANSynth to our work, which we regrettably overlooked in our initial submission. We have revised the manuscript accordingly to include this reference. Nevertheless, there are important distinctions between GANSynth and our proposed Vocos model, and we'd like to clarify these in response to your concerns:

While GANSynth indeed addresses the generation of invertible spectral representations, its focus is limited to the NSynth dataset, which consists solely of individual musical notes. This allows the authors to assume a constant instantaneous frequency. However, this is not true for real-life audio signals (which may have time-varying frequencies), like speech. GANSynth cannot be trivially extended to a broader class of audio signals, and this was left as a future work (see authors' conclusion). But actually it never happened, and shortly after, time-domain GANs for audio took off. MelGAN and subsequent works have demonstrated that, with advanced critics, it is possible to directly model coherent audio in the time domain, alleviating the need for challenging phase estimation. This approach has become the de facto standard in neural vocoding.

Vocos revisits phase (and magnitude) estimation with a straightforward formulation that enables implicit phase wrapping. To clarify and further support our proposed method, we present a new ablation study in which we estimate (scaled) phase angles using the tanh nonlinearity. Please refer to the updated paper, where we demonstrate that this approach results in reduced audio quality (see Table 1). This formulation does not give the model an implicit bias regarding the periodic nature of phase. Vocos, on the other hand, proposes to define activation function in terms of a unit circle, which produces meaningful values for any phase angles. We experimentally demonstrate that this simple formulation allows for successful phase estimation in a non-autoregressive setup.

We disagree with the reviewer's view that the major novelty of our paper is the use of ConvNeXt blocks. Vocos, even without ConvNeXt, demonstrates comparable performance to HiFi-GAN, while being an order of magnitude faster. We've made updates to the paper to better clarify the contributions of this work (see Section 1.2).

Reviewer comment (Experiments):
It's not clear to me whether using "ConvNeXt" solely in the time-domain is already bringing all the benefits stated in the paper. The experiment of using ConvNeXt in the time-domain seems missing.

Author response:
The benefit of Vocos is to synthesize audio 70 times faster than state-of-the-art time-domain vocoder (BigVGAN-base), while maintaining comparable performance, as demonstrated in the subjective evaluation in the paper. This is mainly because of the Fast Fourier Transform rather than differences at a layer level. We respectfully argue that applying ConvNeXt to time-domain vocoders is not relevant here.

Reviewer comment (Evaluation):
There are a few overclaims in the audio reconstruction task. PESQ/UV F1/ Periodicity numbers of EnCodec still win over Vocos by a large margin at 12.0kbps. Should we still consider "Vocos notably outperforms EnCodec?

Author response:
The reconstruction artifacts of EnCodec don’t greatly affect PESQ/UV F1/ Periodicity scores, however the subjective quality is severely impacted. To support this claim we decided to perform a crowd-sourced subjective assessment to evaluate the naturalness of these samples. The results have been included in the paper and can be found in Table 5. Vocos consistently outperforms EnCodec across all bandwidths, based on evaluations by human listeners. We also encourage you to listen to the audio samples at https://anonymous5425.github.io/anonymous-submission/

Reviewer comment (Experiments):
Have you run experiments on ConvNeXt in the time domain?

Author response:
The benefit of Vocos is to synthesize audio 70 times faster than state-of-the-art time-domain vocoder (BigVGAN-base), while maintaining comparable performance, as demonstrated in the subjective evaluation in the paper. This is mainly because of the Fast Fourier Transform rather than differences at a layer level. We respectfully argue that applying ConvNeXt to time-domain vocoders is not relevant here.