Paper: Generative Pre-Trained Speech Language Model with Efficient Hierarchical Transformer (ICLR-2024)

Reviewer comment (Evaluation):
The primary limitation of the paper is the insufficient evaluation. Due to the absence of a perfect evaluation metric for speech generation tasks, it is necessary to conduct a subjective evaluation in the experiments. Additionally, it would be advantageous to provide a demo page for the reviewers to experience the quality of the pre-trained model firsthand. Even if the authors try their best to do objective evaluations, subjective evaluation is still required. DNSMOS is not enough to measure the speech quality.

Author response:
We do want to conduct the subjective comparison such as MOS for all the models, but sadly AudioLM and VALL-E are not open-sourced, we cannot perform a fair and rigorous test of MOS. We have tried our best to conduct fair and rigorous experiments to compare the baselines with our model, which shares the same protocol for all the models. DNSMOS metric is widely accepted in SPEAR-TTS, VALL-E, and SoundStorm, we believe it can reflect the quality of the audio to some extent. We can provide some cases on the anonymous demo page (https://gpsticlr.github.io/GPST/demo) and the code will be available too.

Reviewer comment (Evaluation):
The primary focus of the paper is on efficiency; however, no specific metric is provided to evaluate its efficiency. I am curious about the potential inference time savings that can be achieved with the HIERARCHICALTRANSFORMER architecture.

Author response:
We must clarify that the main contribution of our work is a unified pre-trained speech language model that supports Hi-Res speech generation and spoken multi-lingual. And the efficient hierarchical transformer is the approach to achieve it. What the reviewer states in Q4 also agrees with it. However, there appear to be conflicting assertions in Q2 and Q3, suggesting that the primary emphasis of our work lies in efficiency, which doesn't accurately reflect our main focus. We provide our generation speed test with the fairseq-generate tool. The inference is run on the NVIDIA 3090 GPU in the semantic to acoustic mode with a batch size of 64. However, we cannot provide the speed test results for other baselines since both AudioLM and VALL-E are not open-source.

 |Model | $N_g + N_l$ | speed | # codec |# params|
|-|-|-|-|-|
 |GPST | 11+4 | 2.31 sentences/s, 9438.75 tokens/s | 8 | 190M |
 |GPST | 10+8 | 1.89 sentences/s, 7643.03 tokens/s | 8 | 190M |
 |GPST | 9+12 | 1.57 sentences/s, 6232.50 tokens/s | 8 | 190M |
 |GPST-Hi-Res | 9+12 | 0.88 sentences/s, 7186.19 tokens/s | 16 | 207M |

Reviewer comment (Novelty):
Given the existence of previous full NAR approaches like SoundStorm, the contribution of the paper seems relatively limited. The efficiency issue has already been largely addressed by these full NAR approaches. Consequently, I have doubts about whether the proposed HIERARCHICALTRANSFORMER structure will inspire further advancements in speech pre-trained models.

Author response:
We emphasize that efficiency is NOT the primary focus of our model GPST, instead, the main contribution is the unified speech LM framework. The design of GPST considers the potential possibility of integrating the speech generation ability into auto-regressive LLMs like LLaMA, which is not possible for NAR models. For the model architecture design, we do not blindly pursue the number of parameters for GPST, instead, we carefully configure the experimental setting of a large global model and a small local model, simulating scenarios where a small model integrates with LLMs.

We provide the reason why SoundStorm is not appropriate for comparison. It utilizes duplicate semantic tokens as a condition to generate acoustic tokens, leading to a leakage of duration information. Moreover, duplicate semantic tokens would cause failures in the generation of semantic tokens [1], which would limit the application in speech generation [2] and resynthesis for speech translation system [3], while our model GPST can easily assimilate into the SOTA multi-lingual speech translation system SeamlessM4T [4]. Although NAR approach such as SoundStorm can accelerate the generation speed, it exhibit performance limitations when compared to AR models [5,6,7,8]. According to their experiment results, the SPK result for SoundStorm is $0.57$ while GPST achieves a better result of $0.605$.

Reviewer comment (Evaluation):
I am curious about the training and inference time of the structure.

Author response:
We must clarify that the main contribution of our work is a unified pre-trained speech language model that supports Hi-Res speech generation and spoken multi-lingual. And the efficient hierarchical transformer is the approach to achieve it. What the reviewer states in Q4 also agrees with it. However, there appear to be conflicting assertions in Q2 and Q3, suggesting that the primary emphasis of our work lies in efficiency, which doesn't accurately reflect our main focus. We provide our generation speed test with the fairseq-generate tool. The inference is run on the NVIDIA 3090 GPU in the semantic to acoustic mode with a batch size of 64. However, we cannot provide the speed test results for other baselines since both AudioLM and VALL-E are not open-source.

 |Model | $N_g + N_l$ | speed | # codec |# params|
|-|-|-|-|-|
 |GPST | 11+4 | 2.31 sentences/s, 9438.75 tokens/s | 8 | 190M |
 |GPST | 10+8 | 1.89 sentences/s, 7643.03 tokens/s | 8 | 190M |
 |GPST | 9+12 | 1.57 sentences/s, 6232.50 tokens/s | 8 | 190M |
 |GPST-Hi-Res | 9+12 | 0.88 sentences/s, 7186.19 tokens/s | 16 | 207M |

Reviewer comment (Experiments):
Since the model directly predicts all encodec tokens, is there any memory issue in long sequence prediction?

Author response:
The memory issue in long sequence generation is one of the motivations for GPST as well. The hierarchical structure effectively reduces the sequence length handled by the global transformer, mitigating the memory pressure with lengthy sequences. By allowing the global transformer to concentrate on high-level or coarse features, GPST delegates fine-grained details to local transformers. In our experiments, GPST notably did not encounter memory issues. The low WER indicates the model's competence in handling lengthy sequences without experiencing memory-related challenges.

Reviewer comment (Experiments):
In addition to the theoretical efficiency analysis, it would be good to see actual measures of speed or latency in Table 3. The real-time efficiency depends on the number of layers among other factors, so it would be good to see this as part of the ablation study.

Author response:
## Q1: The real-time efficiency.

We provide our generation speed test with the fairseq-generate tool. The inference is run on the NVIDIA 3090 GPU in the semantic to acoustic mode with a batch size 64.

|Model | $N_g + N_l$ | speed | # codec | # params|
|----|----|----|----|----|
|GPST | 11+4 | 2.31 sentences/s, 9438.75 tokens/s | 8 | 190M |
|GPST | 10+8 | 1.89 sentences/s, 7643.03 tokens/s | 8 | 190M |
|GPST | 9+12 | 1.57 sentences/s, 6232.50 tokens/s | 8 | 190M |
|GPST-Hi-Res | 9+12 | 0.88 sentences/s, 7186.19 tokens/s | 16 | 207M |

Reviewer comment (Evaluation):
For the semantic-to-acoustic condition in Table 1, why use WER as the evaluation criterion rather than human evaluation of acoustic generation quality? Does this box refer to the unconditional generation? If so, why not compare WER against the human-labeled ground truth, why were Hubert-Large/XLSR-53 used as 'Ground Truth'? The evaluation methodology/terminology is a bit confusing here.

Author response:
## Q2: The reason for choosing WER as the evaluation metric in the semantic-to-acoustic task in Table 1.

WER aims to evaluate the speech synthesis robustness. Neural TTS systems usually suffer from the robustness issue, which sometimes has deletion, insertion, and replacement errors due to wrong attention alignments. We perform ASR on the generated audio and calculate the word error rate (WER) with respect to the original text transcriptions. We have other metrics SPK and DNSMOS to evaluate the speech quality. These metrics focus on different perspectives on the speech generation models. It is hard for us to conduct human evaluation benchmarks since the baselines AudioLM and VALL-E are not open-sourced, we cannot perform a fair and rigorous MOS test for them. 

For the semantic-to-acoustic condition in Table 1, all the models except GPST-TTS are conditioned on the golden semantic tokens to generate acoustic tokens/waveforms, followed by an ASR model to calculate WER with respect to the text transcriptions. Only our GPST-TTS model can condition the text inputs to generate speeches. The GPST-TTS model first infers semantic tokens with texts as input, then generates acoustic tokens with the semantic tokens as inputs.

## Q3: The meaning of GroundTruth in Table 1 row 1.

The results are directly calculated on the ground truth speech in the LibriSpeech test-clean dataset as the upper bound.

Reviewer comment (Experiments):
I don’t think it is fair to compare GPST/AudioLM with YourTTS/SPEAR-TTS/VALL-E on speaker identity transfer. The latter three are zero-shot text-to-speech synthesis models which take text and speaker prompt as input. In contrast, GPST and AudioLM take semantic tokens inferred from speech, which can have speaker information leaked from speech. If the authors follow the VALL-E setup for speaker identity transfer, then the semantic tokens are inferred from utterances from the same speaker as the audio prompt. For a more fair comparison, I would like to see the GPST-TTS setup for speaker identity transfer where semantic tokens are also inferred from text, in order to compare fairly with the zero-shot TTS models.

Author response:
We must clarify that our primary focus lies in developing a unified speech language model, while GPST-TTS is only a toy model for inspiration trained on LibriSpeech 960h, which is not comparable with VALL-E trained on Librilight 60k and SPEAR-TTS employing pre-training and back-translation techniques. However, we are still interested in the speaker information leak problem and give the extra experiment results here. Unlike VALL-E that directly trains with text transcriptions. If AudioLM and GPST use the semantic tokens inferred by a text-to-semantic model, it will cause a distribution mismatch with the semantic prompt provided in the speaker identity transfer stage. Because, as you state, semantic tokens may contain subtle speaker-related information such as prosody. Therefore, we propose a more rigorous inference mode for the text-to-semantic task like in-context learning in speaker identity transfer mode. We transcribe the speech prompt with an ASR model and prepend it to the text inputs in the text-to-semantic models, which is similar to the speaker identity transfer. We can see that GPST can get a significant improvement with text prompts from $0.547$ to $0.568$. We believe the performance can get further improvement if trained on Librilight 60k and using the pre-training and back-translation method proposed in SPEAR-TTS.

In addition, we find that the semantic tokens can not control the speaker identity in the synthesized speeches. For example, a semantic sequence said by a man would still synthesize the speech in a woman voice. We provide some cases in the anonymous demo page (https://gpsticlr.github.io/GPST/demo).

|Model | SPK | # codec | # params|
|-----|-----|-----|-----|
|**Speaker Identity Transfer**|
|GPST-TTS w/o text prompt | 0.547 | 8 | 182M+190M|
|GPST-TTS w/ text prompt | 0.568 | 8 | 182M+190M|

Reviewer comment (Experiments):
There is no direct comparison between hierarchical and non-hierarchical (i.e., flatten the audio tokens and use vanilla LM) models that are trained with the same objective. The comparison with AudioLM is not sufficient as there are other confounding factors.

Author response:
It's a natural thought to train a vanilla decoder-only transformer for speech generation. However, it is impractical since the length of the flattened acoustic tokens is too large (6000 tokens for 10s speech) to feed into the vanilla LM and will cause a cuda OOM error. This limitation stands as one of the key motivations driving the development of GPST, aiming to address and overcome these computational constraints in speech generation models.

Reviewer comment (Evaluation):
Why are DNSMOS not presented in Semantic to Acoustic and Acoustic Continuation tasks?

Author response:
Since both of AudioLM and VALL-E are not open-sourced, we follow AudioLM's instructions and compute DNSMOS with the examples provided on VALL-E’s demo page for fairness, which is described in Section 4.1.4. Unfortunately, VALL-E did not conduct the Semantic to Acoustic task and AudioLM did not conduct the Acoustic Continuation task. Therefore, the Speaker Identity Transfer task is the only one we can test to compare all the models in fair.

Reviewer comment (Evaluation):
GPST-Hi-Res lags behind GPST in most of the metrics. It is unclear why intelligibility (WER) and speaker similarity (SIM) degrades when having better audio quality.

Author response:
The better audio quality means richer harmonic energy in the high-frequency regions, as shown in Figure 3, which focuses on different perspectives of the speech from WER and SIM. WER focuses on the robustness of speech models, which sometimes has deletion, insertion, and replacement errors due to wrong attention alignments. SIM focuses on speaker similarity to evaluate the voice cloning ability. Better speech quality does not guarantee better WER and SIM, which is the reason we choose three different metrics to evaluate the generated speech. As we mentioned in Section 4.2, the Hi-Res speech generation, with much more quantizers (from 8 quantizers to 16 quantizers), is still a tough task for speech-language models and requires a larger model with more parameters and longer training time. In the experiments, we simply use the same setting as non-Hi-Res models, which is not the best setting for Hi-Res model. Additionally, our model did not reach convergence before cessation of training. We believe the results would get improvement with more training steps.

Reviewer comment (Reproducibility):
Is E_a shared between local and global transformers?

Author response:
No. Since the embedding sizes of the local transformer (512) and global transformer (1024) are different, we do not share the embedding table.

Reviewer comment (Reproducibility):
Unclear what local drops means. Each one predicts D codes for a time step and the batch size is b x T2. Is some (h_t, a_t^1, …, a_t^D) dropped to reduce batch size? Does it improve performance or just save memory?

Author response:
The Local-drop method is proposed to save the GPU memory, without which the GPST-Hi-Res model would cause a cuda OOM error. The local transformer's input size is $(b \times T_2, D)$, so we can simply drop some samples in the batch dimension $b \times T_2$ since they are i.i.d. for the local transformer. The Local-drop method is only effective for hierarchical transformer because the unfolding operation in $T_2$ breaks the dependency in tokens to i.i.d., and is safe to drop them to save the memory. However, if we drop some tokens in the non-hierarchical (flattened) vanilla LM, we can not save memory since the sequence length and batch size are not changed.

Reviewer comment (Reproducibility):
How many layers for Ng, Nl? I did not find it in the paper or appendix.

Author response:
We ablate different settings of $N_g$ and $N_l$ in Table 3 with the number of total parameters fixed. In Table 1, we use the setting $N_g=9, N_l=12$ for evaluation. We will clarify it in the experimental setting.

Reviewer comment (Experiments):
There are various unsupported claims in the paper.

Author response:
We will carefully clarify the unsupported claims as follows:

### (1). ``...which limits the performance of fine acoustic token generation..''

We state it in the introduction of VALL-E. As described in the paper, VALL-E conducts non-auto-regressive generation of the acoustic tokens from subsequent quantizers. As investigated in non-auto-regressive generation models [1,2,3,4], the independence assumption in the output space ignores the real dependency between target tokens, and the maximum-likelihood training forces to cover all possible modes, which makes NAR systems perform worse than the AR baselines. VALL-E leverages the vanilla non-auto-regressive generation model for the subsequent acoustic tokens, typically resulting in suboptimal performance compared to auto-regressive generation models.

### (2). ``These multi-stage generative models induce significant error propagation issues, which can negatively impact the overall performance.''

It is established in the literature that cascaded multi-stage systems often exhibit inferior performance in comparison to end-to-end systems due to inherent error propagation issues [5,6]. This phenomenon stems from the discrepancy between the training and inference procedures within cascaded systems. During training, all modules receive training signals derived from ground truth labels. However, during inference, these modules are fed outputs from preceding modules, resulting in error outputs that are out of distribution concerning the training data. Consequently, as elaborated in Section 3.1 and illustrated in Figure 1, both AudioLM and VALL-E adopt cascaded architectures, inherently predisposing them to the issue of error propagation. This discrepancy contributes to the amplification of errors throughout the cascaded system's feed-forward flow, thereby adversely affecting performance.

### (3). ``Additionally, obstructing the information flow among hierarchical quantizers would degrade the model’s performance, especially in Hi-Res speech generation that requires more residual quantizers.''

The acoustic tokens are obtained with residual quantizers, wherein the subsequent quantizers rely heavily on the results from the previous quantizers. In the Hi-Res speech generation setting, the number of residual quantizers (n=16) is much larger than the normal setting (n=8), which poses more challenges for the model. With the two issues described above, the information flow in VALL-E is obstructed by non-auto-regressive model and cascaded systems, resulting in the degradation of the model's performance.

We will add the support evidence and related citations to the revised version, thanks for helping improve our work.

Reviewer comment (Novelty):
Overall this paper seems incremental considering the other modeling approaches such as AudioGen, VALL-E, and AudioLM. It is not clear whether this is enough to pass the bar for ICLR publication.

Author response:
It is true that the hierarchical transformer is not a new architecture in the NLP community. However, our focus lies specifically on the speech generation task, where the utilization of the hierarchical transformer remains unprecedented. Our work pioneers the application of this architecture for speech generation, marking a significant departure from prior endeavors. Notably, our approach involves a novel design of the architecture, encompassing the integration of semantic tokens, acoustic tokens, in-context learning, multilingual aspects, among others, tailored specifically for the speech language model. This task distinctly differs from conventional text models, thereby underscoring its novelty.

While acknowledging the foundational groundwork laid by AudioLM in the realm of speech generation, it is imperative to note that our model, GPST, draws inspiration from AudioLM while innovatively leveraging the hierarchical transformer to address inherent challenges encountered in AudioLM and VALL-E. These preceding models face significant limitations when generating lengthy acoustic sequences, a hurdle meticulously addressed in Section 3.1. Our model not only tackles these challenges but also outperforms them, as substantiated by our empirical findings.

Reviewer comment (Theory):
Regarding the efficiency analysis, the reviewer disagrees with the author's analysis. The size of $D$ is usually 4/8/12, not more than that, and $m_l$ is probably much bigger. So, in case the authors decide to ignore $m_l$ in their analysis, they should also ignore $D$. Overall, the reviewer agrees with the authors that the proposed method is likely to be more efficient, but does not agree with their specific analysis.

Author response:
The efficiency analysis comes from two different perspectives, one is computation complexity and the other is FLOPS. In the computation complexity analysis, if we omit $D$, which implies flattening the acoustic sequence, we will get a cuda OOM error on GPU. Therefore, we keep it in our analysis and ensure a comprehensive evaluation of the computational complexity.

Reviewer comment (Evaluation):
Regarding the results, can the authors provide more details on how they compute the WER using the GSLM method? GSLM [1] is a decoder-only method that operates on units obtained from HuBERT, with no conditioning on text. So it is not clear how the authors compute WER when the model performs continuation / unconditional generation. More details are requested.

Author response:
GSLM uses HuBERT code (semantic tokens) as inputs like AudioLM and reconstructs the waveform with the Tacotron2 model and the WaveGlow vocoder. We run their open-sourced code using the released model and evaluate the results. GSLM can not generate speeches conditioned on texts, and the evaluation is done directly based on golden semantic tokens. In the Semantic to Acoustic part of Table 1, only our GPST-TTS model can generate speeches conditioned on texts.

Reviewer comment (Experiments):
Did the authors try to explore / compare their method to a simple delay pattern as introduced by [2, 3]? It would greatly simplify the modeling approach and remove the need for the local transformer.

Author response:
The delay pattern models in [7,8] are tailored for semantic tokens like GSLM and music generation. Designing a dedicated model for speech LM incorporating acoustic tokens in a delay pattern extends beyond the scope of our work. However, we provide an analysis here to illustrate why our model is better than the delay pattern approach. The delay pattern model predicts multiple tokens in parallel with a simple linear classification head, while GPST predicts them with a local transformer in an auto-regressive manner, which means GPST can better fit the distribution. The auto-regressive mechanism and the utilization of a local transformer within GPST afford it a superior capacity to model intricate dependencies among tokens, thereby establishing its theoretical superiority over the delay pattern approach.

Reviewer comment (Presentation):
If the reviewer is not mistaken, the paper's length should be 9 pages, with an unlimited number of pages for references and supplemental material. Hence, the "REPRODUCIBILITY AND ETHICS STATEMENT" provided by the authors on the 10th page may break the submission guidelines. The reviewer is willing to change the score if mistaken.

Author response:
As ICLR 2024 Author Guide states (https://iclr.cc/Conferences/2024/AuthorGuide), the optional ethic statement and reproducibility statement will not count toward the page limit, but should not be more than 1 page. So we believe we didn't break the submission guidelines.