Paper: Generative Pre-training for Speech with Flow Matching (ICLR-2024)

Reviewer comment (Experiments):
One primary limitation of this work is the relatively limited range of supported task types. It would be beneficial for the authors to expand their support to include a wider variety of tasks, such as speech editing tasks, to further demonstrate the capabilities of their pre-trained models.

Author response:
We thank the reviewer for the constructive and explicit feedback that would definitely improve our work. We have included new results on speech editing in Section A4.1 in the revision, following closely to the prior work A3T [A]. We also note that with the three fundamental tasks supported by our model, there are potentially more variants that can be easily done in the future. E.g., target speaker extraction with audio prompt, text-guided speaker extraction via multi-task model, speech removal via removing enhancement result, etc.

We thank the reviewer again for helping improve this work, and we hope our response has addressed your concern and enhanced the overall contribution.

[A] A3T:  Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing, Bai et al.

Reviewer comment (Experiments):
By incorporating additional task types, the authors can provide a more comprehensive evaluation of the model's abilities and showcase its versatility across various domains. This would enhance the overall contribution and applicability of the proposed pre-training approach.

Author response:
We thank the reviewer for the constructive and explicit feedback that would definitely improve our work. We have included new results on speech editing in Section A4.1 in the revision, following closely to the prior work A3T [A]. We also note that with the three fundamental tasks supported by our model, there are potentially more variants that can be easily done in the future. E.g., target speaker extraction with audio prompt, text-guided speaker extraction via multi-task model, speech removal via removing enhancement result, etc.

We thank the reviewer again for helping improve this work, and we hope our response has addressed your concern and enhanced the overall contribution.

[A] A3T:  Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing, Bai et al.

Reviewer comment (Novelty):
By omitting key references, this paper seems to be suggesting that nobody has ever thought of using self-supervised pre-training for speech synthesis before, and it seems to be suggesting that nobody has ever used generative flow for speech synthesis before. The manuscript should include references to key works in both areas, in order to more clearly articulate what is the actual contribution of the paper.

Author response:
We would like to highlight that the key innovation of our approach: SpeechFlow is, to the best of our knowledge, the first generative speech pre-training method that is not conditioned for any specific task (i.e., not just for speech synthesis, but also other generation tasks). We apologize if the writing is misleading in the submission and will do our best to avoid overstating our contribution.

Reviewer comment (Novelty):
The paper should better describe the history of the use of self-supervised pre-training for speech enhancement and speech synthesis, and the use of generative flow in speech synthesis.

Author response:
We thank the reviewer for pointing out the reference in semi-supervised TTS [Chung et al. 2019], we have updated Section 2 in our revision accordingly. 

For enhancement, the generative tasks are covered in related work (Section 2, under the “Pre-trained Speech Models” section) with references including the one suggested by the reviewer. We would also like to point out that the enhancement results in Superb-SG [A] are far behind recent enhancement models (that we compared against in this paper), results are more for benchmarking speech representation than building actual enhancement systems. This also showcased how SpeechFlow diverges from existing pre-trained ​​speech models – it is the first general-purpose pre-trained speech model for generative tasks.

Reviewer comment (Novelty):
For self-supervised training for TTS, a relevant reference is: Yu-An Chung, Yuxuan Wang, Wei-Ning Hsu, Yu Zhang and RJ Skerry-Ryan, "Semi-supervised training for improving data efficiency in end-to-end speech synthesis," ICASSP 2019, 6940-6944.

Author response:
We thank the reviewer for pointing out the reference in semi-supervised TTS [Chung et al. 2019], we have updated Section 2 in our revision accordingly.

Reviewer comment (Novelty):
For self-supervised training for speech enhancement, relevant references include: Yang, Shu-wen, Po-Han Chi, Yung-Sung Chuang, Cheng-I. Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu et al. "Superb: Speech processing universal performance benchmark." arXiv preprint arXiv:2105.01051 (2021), and other papers that submitted entries to the Superb challenge.

Author response:
For enhancement, the generative tasks are covered in related work (Section 2, under the “Pre-trained Speech Models” section) with references including the one suggested by the reviewer. We would also like to point out that the enhancement results in Superb-SG [A] are far behind recent enhancement models (that we compared against in this paper), results are more for benchmarking speech representation than building actual enhancement systems. This also showcased how SpeechFlow diverges from existing pre-trained ​​speech models – it is the first general-purpose pre-trained speech model for generative tasks.

Reviewer comment (Novelty):
For generative flow for speech synthesis, relevant references include: Ryan Prenger, Rafael Valle and Bryan Catanzaro, "Waveglow: A flow-based generative network for speech synthesis," ICASSP 2019, 3617-3621; and Rafael Valle, Kevin Shih, Ryan Prenger and Bryan Catanzaro, "Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis," arXiv 2020.

Author response:
The use of generative models in TTS has been discussed in Sections 1 and 2.  In particular, flow-based methods such as WaveGlow [Preger et at. 2019] are covered in the 1st paragraph of related work. We thank the reviewer for providing additional reference [Valle et at. 2020] for generative TTS, which is added in the revision.

We would also like to point out that our approach relied on Flow Matching [B], which is different from flow-based methods. Flow matching learns a time-dependent diffeomorphic map, called “flow”, to push samples from one distribution to another in a continuous (as the mapping is defined by ODE function) manner. The neural network can be any model that predicts the trajectory of flow at any given time, and inference is done iteratively similar to diffusion models [C]. This is different from the flow-based generative models [D,E] which rely on strictly invertible networks to maximize the likelihood of data, and inference can be done in a single forward step.

Reviewer comment (Evaluation):
No subjective evaluation is presented which could be useful for the users of this model. Most TTS works present both subjective and objective metrics for the evaluation.

Author response:
We thank the reviewer for the helpful feedback that helps improve our evaluation.  We have updated the human evaluation for TTS in Section A.4.6 and Table 11 in the paper.

Reviewer comment (Experiments):
What dataset are you using for pre-training? You mention 60k hours of English speech.

Author response:
For pre-training, we use 60k hours of unlabeled speech from English audiobooks in the public domain recorded by thousands of speakers, which is a comparable setup as [A].

Reviewer comment (Novelty):
The novelty is limited. It's basically applying flow matching to the speech synthesis problem.

Author response:
While speech synthesis[1] is indeed one of the applications studied here, **this paper focused more on developing pre-training methods (via flow matching) for generative tasks in speech**. Speech synthesis is only part of the experiment to showcase the use of a pre-trained model, alongside speech enhancement and separation.

Our approach is different from existing generative modeling methods for speech. Generative models have been useful in speech generation tasks, such as GAN-based vocoders, flow-based TTS, diffusion-based enhancement, etc. Nevertheless, these generative models are trained with task-specific conditioning (e.g., vocoders are trained to model waveform conditioning on spectrograms), limiting their generalizability. In contrast, we propose to pre-train a generative model without a predefined application and show that the resulting model can be applied to different downstream generation tasks with strong performance.

In short, **pre-training general-purpose generative models for speech have not yet been explored to the best of our knowledge**. (To further justify our novelty, this is indeed listed as the biggest strength of our work according to all other reviewers.) Given the current review is as short as “The novelty is limited. It's basically applying flow matching to the speech synthesis problem”, please kindly provide more detailed feedback:
If the reviewer thinks a better solution has been proposed before, please list it so we can discuss the differences. 
If the reviewer thinks the problem we studied (a pre-training method that benefits multiple speech generative tasks) is irrelevant, please state why so we can discuss it. 

Otherwise, we hope our responses have clarified the contribution of this paper and we respectfully ask the reviewer to re-evaluate its novelty. 

[1] We assume the reviewer refers to text-to-speech synthesis as “speech synthesis” following the convention in the field. It could be possible that the reviewer is referring to a more general set of tasks with speech as output. In that respect, our method is novel as 1) it is the first general-purpose pre-trained generative model as described above; 2) flow matching has not been used on speech generation tasks besides text-to-speech.

Reviewer comment (Reproducibility):
The description of the experiments are severely limited. For example, what datasets were used, and the details on the model architecture and hyperparameters. I have a major concern on reproductivity.

Author response:
(also associated to first 3 questions raised by the reviewer)

- In all sections from 4.1 (pre-training), 4.2 (enhancement), 4.3 (separation), to 4.4 (zero-shot TTS), we have model/training/data paragraphs with bolded header. Besides description in text, the dataset for each downstream task is provided in the caption of the corresponding Table 1,2, and 3.  
- Here we additionally discuss the data used:
    - For the 60k hours of unlabeled English speech, we use audiobooks in the public domain. A setup with the same amount of data can be found in [N; cited Wang et al. 2023 in the paper] where the data they used is publicly available.
    - The 960/360 hours of speech for TTS/separation are also from the same domain, A setup with the same amount of data can be found in [O; cited Ravanelli et al. 2021 in the paper] where the data they used is publicly available. 
- Model architecture and hyper-parameters (learning rate, # of GPUs, etc.) are already provided in Section 4.1, 4.2, 4.3, 4.4 for each task respectively. We understand that this might be hard to track as they are spreaded out in different sections, we have added a summarization in Table 5 in the appendix to make it clear at a glance.
- In addition to Section 4.6 which already provided analysis on hyper-parameter selection, we have added the study on the model’s robustness w.r.t. masking configuration in Section A.4.5.
- Finally, we note that the model architecture (24-layer Transformer) and loss function (flow matching) of SpeechFlow is already publicly available [L]. 

To conclude, we believe we have revealed as many details in the paper as we can to ensure SpeechFlow can be reproduced.

Reviewer comment (Experiments):
Sec 4.1 -- what are the 60k hours of English speech data for training?

Author response:
- In all sections from 4.1 (pre-training), 4.2 (enhancement), 4.3 (separation), to 4.4 (zero-shot TTS), we have model/training/data paragraphs with bolded header. Besides description in text, the dataset for each downstream task is provided in the caption of the corresponding Table 1,2, and 3.  
- Here we additionally discuss the data used:
    - For the 60k hours of unlabeled English speech, we use audiobooks in the public domain. A setup with the same amount of data can be found in [N; cited Wang et al. 2023 in the paper] where the data they used is publicly available.
    - The 960/360 hours of speech for TTS/separation are also from the same domain, A setup with the same amount of data can be found in [O; cited Ravanelli et al. 2021 in the paper] where the data they used is publicly available.

Reviewer comment (Experiments):
Sec 4.4 -- what's the 960 hours of transcribed English speech used for fine-tuning?

Author response:
- In all sections from 4.1 (pre-training), 4.2 (enhancement), 4.3 (separation), to 4.4 (zero-shot TTS), we have model/training/data paragraphs with bolded header. Besides description in text, the dataset for each downstream task is provided in the caption of the corresponding Table 1,2, and 3.  
- Here we additionally discuss the data used:
    - For the 60k hours of unlabeled English speech, we use audiobooks in the public domain. A setup with the same amount of data can be found in [N; cited Wang et al. 2023 in the paper] where the data they used is publicly available.
    - The 960/360 hours of speech for TTS/separation are also from the same domain, A setup with the same amount of data can be found in [O; cited Ravanelli et al. 2021 in the paper] where the data they used is publicly available.