Paper: Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM (ICLR-2024)

Reviewer comment (Evaluation):
The association between generated text and speech can be further evaluated. For instance, performing a Word Error Rate (WER) test using speech, [xp, xc], and transcriptions [yp, yc] would be advantageous. In addition, replacing the text with a sequence of null tokens of the same length can help showcase the impact of text on semantic quality. These analyses would lead to a better understanding of the role that text plays in the proposed model.

Author response:
When obtaining predicted transcription and continuation y_hat, we cannot split it to the prompt transcription and the continuation transcription. For the predicted text sequence, we do not have a way to know where the switch between transcription and continuation happens. Thus, we cannot compute WER against prompt only or against continuation only. However we measured the ASR performance of the proposed model trained exclusively for ASR without speech continuation, the results were a WER of 3.1 / 4.9 on Librispeech test-clean / test-other.

We have performed this analysis, we replaced the text with a sequence of null tokens, with a sequence of characters (for example “a a a a”) and with some other arbitrary token. We see consistent results, that when doing this the speech continuation of the model contains mumbling or long human voices. We can thus conclude that indeed the text has an impact on the semantic quality of the generated speech.

Reviewer comment (Experiments):
Could you please provide additional information about the training and inference in the Spoken QA task? Does SPECTRON undergo training for the spoken QA task, whereas the baselines, such as AudioLM and GSLM, does not?

Author response:
No specific training has been done on spoken question answering. Spectron was only trained for speech continuation as described in Section 3.2. AudioLM, GSLM and TWIST also haven't been explicitly trained for spoken question answering. SpeechGPT, on the other hand, was trained using a carefully curated multimodal question answering dataset. We have added a column to Table 4.

Reviewer comment (Evaluation):
In the provided Spoken QA demos, SpeechGPT tends to generate lengthier and more detailed answers, while SPECTRON prefers more concise responses. Could the comparable performance of both models be attributed to the nature of the QA task, such as the prevalence of shorter answers in the test set?

Author response:
Spectron hasn't been trained using spoken QA datasets, we only used those datasets for evaluation. SpeechGPT was explicitly trained for question answering using carefully curated spoken QA dataset, therefore its answers are more detailed. Also, note that the evaluation strategy of the spoken QA datasets does prefer short answers. We define that an answer is correct by checking if the correct answer is contained in the model output. For example, if the model output is “The capital of France is Paris” and the correct answer is “Paris” we say that the model answered the question correctly. Thus, long and detailed answers have a higher chance of containing the correct answer inside of their response.

Reviewer comment (Evaluation):
No details are provided on MOS tests - could the authors share some details?

Author response:
In the MOS experiments 30 raters rated 20 randomly sampled examples. We use the MOS experiments to measure the naturalness of the synthesized continuation speech without the prompt. We have added these details to the revised version.

Reviewer comment (Evaluation):
What was the WER of the Librispeech model used to obtain pseudo-labels on LibriLight? Do the authors have any idea about label quality?

Author response:
The Librispeech model that was used to obtain the pseudo-labels is the NST model. It achieves a WER of 1.7/3.4 on Librispeech test-clean/test-other datasets. From manually sampling some of the training data, it seems that the quality of the pseudo labels is high.

Reviewer comment (Experiments):
The paper says "we use TTS to synthesize questions that fit within this duration" - what does this mean? Do you truncate long questions or modify the durations of phones/words to fit within 3s? Or do you drop questions that are longer than 3s?

Author response:
We drop questions that are longer than 3s. Since previous methods (GSLM and AudioLM) use only 3s prompts, we only considered questions whose synthesized spoken version was shorter than 3s.

Reviewer comment (Reproducibility):
The paper lacks the reproducibility and accessibility of the results due to several pre-training models, which are not publicly available or cannot be reproduced due to the inaccessible training data (e.g., WaveFit and state-of-the-art Conformer ASR system (Zhang et al., 2023b)). I appreciate your efforts in mitigating the issue (e.g., the use of Libriright and the release of the SQA test set), but I think the paper still has this issue.

Author response:
The release of these upstream pre-trained models and code built atop them are beyond our control. We tried to mitigate this issue in other aspects; beyond training and (to-be released) test sets from public data, we also provide a full description of the model architectures, novel objectives and hyperparameters used in the training (Appendix A.1). With these, we believe our key contribution is reproducible with our test suite and the many public LLMs, ASR encoders, and vocoders (e.g., LLaMA, wav2vec2 CTC, HiFiGAN), especially as our method’s semantic improvements are a step-function over the other methods (except for the concurrent SpeechGPT work), and our acoustic improvements are a step-function over SpeechGPT. We welcome suggestions on how to further improve reproducibility for this discussion period and camera review.

Reviewer comment (Experiments):
Due to the above issue, it is not clear whether the superiority of the proposed method compared with other speech LMs comes from their novel training schemes or strong pre-trained models. Thus, the effectiveness, especially for the comparisons with other speech LMs, is weak.

Author response:
Thank you for the comment. In order to evaluate the origin of our models improvement we compare the pre-trained models that each method use:

| Detail | Ours | SpeechGPT | TWIST | GSLM | AudioLM |
|---|---|---|---|---|---|
| LL # Params | 1B/350M | 7B | 1B/ 7B | ~200M  | 0.9B (0.3B per stage)  |
| LM Type | Palm 2 | LLama | OPT (1B) / LLama (7B) | None | None |
| Speech Encoder | USM | mHuBert  | T-Hubert | HuBert Large | Wav2Vec  |
| Speech Encoder # Params | 600M | 317M | 317M | 317M | 600M |
| WER of Speech Encoder | 3.1  | 2.94 | 2.94 | 2.94 | 3.1 |
| Speech Encoder Dataset | Web-scale, Librispeech fine-tune | VoxPopuli 100k | Librispeech, VoxPopuli, Common Voic, Spotify, Fisher.  | Librispeech | Librilight |
| Training Datasets | Librilight | Librilight, Speech Instruct | Librispeech, Librilight, Spotify podcasts, People dataset, VoxPopuli | Librilight | Librilight |
| # Training Examples | 60k | 60k + 38k | 150k | 60k | 60k |
| | | | |

LLM pretrained model - Our method is based on the 1B parameters LLM whereas SpeechGPT and TWIST utilize a 7B parameters model. LLM performance is well known to be strongly influenced by the model size. Therefore, we do not believe that the language model used in our method is more powerful than the language models used in SpeechGPT or TWIST methods. In comparison with AudioLM, we use a comparable model size. We acknowledge that the GSLM model is smaller than our model.  Moreover, SpeechGPT is fine-tuned with cross-modal instruction objectives derived from carefully curated speech-text datasets. TWIST must learn its speech capability from scratch, and so they use approximately 150k hours of speech-text datasets. Our method exhibits competitive results for multiple tasks using only the public Libri-Light dataset (60k hours) for adaptation.

ASR / Tokenization - TWIST, SpeechGPT and GSLM all use the HuBERT model for tokenization. On the ASR task, from the SUPERB paper, which uses the same type of representation, the Hubert-Large model achieves a WER of 2.94 on the LibriSpeech test-clean test set. In comparison, the USM model we used has a performance of 3.1 on the test-clean test sets. Thus, we do not think that the speech-encoder we have used is more powerful than the open-source methods or any of the encoders used in the methods we have compared against.

Reviewer comment (Writing):
Clarity: Section 2 requires more improvements. There are many different aspects between the related studies and the proposed method, and it is not clear what is the advantage of this method. I recommend you rewrite Section 2 to categorize the different aspects (e.g., model architecture, training method, pre-training data, etc.) and emphasize the distinction between the proposed method and others.

Author response:
Thank you for the comment. We re-wrote Section 2 based on your suggestions (e.g., model architecture, training method, pre-training data, etc.). The re-written section can be viewed in the updated version of the paper.  Additionally, extra information about the related work can be found in Appendix A.2.

Reviewer comment (Experiments):
Can you evaluate the ASR performance of this method? It would probably not be nice due to the over-prediction function in the ASR prediction phase (it will predict more than what was spoken), but it would be an interesting result to report.

Author response:
For the ASR task, we have computed the WER between the generated text and the original labels. As you state, it is important to note that the text also contains the continuation, and thus, the WER is not ideal. A large portion of the error comes from the continuation since there is a large variance in text continuation. When comparing a text continuation to a transcript, even a deviation of a single token will mean that the whole sequence that comes after the error is wrong with respect to the ground truth transcription. However, we measured the ASR performance of the proposed model trained exclusively for ASR without speech continuation, the results were a WER of 3.1 / 4.9 on Librispeech test-clean / test-other. For TTS, we are currently exploring the transformation of Spectron into TTS and aim to provide an update shortly.

Reviewer comment (Experiments):
I have the same question for TTS. In this case, due to the lack of conditions, it may not work at all (?).

Author response:
We have successfully completed the TTS experiment for Spectron. In this experiment, we utilized the prompt for acoustic conditioning and manipulated the text in the CoT to match the requested text. The N-MOS results for this experiment are 3.40±0.31, which is a fair outcome considering that the system was not specifically trained for TTS.

Reviewer comment (Experiments):
Why is it 3 seconds? Would it robustly work even if we throw more than (or less than) 3-second spoken prompts?

Author response:
In order to be consistent with the previous work (GSLM and AudioLM), we use prompts of 3 seconds or less during testing.

Reviewer comment (Experiments):
How did you deal with sentences that are less than 3 seconds during training? Can you discard them or concatenate neighboring sentences to make it longer than 3 seconds?

Author response:
We discard sentences shorter than 3 seconds during training. Note that in the LibriLight and Librispeech the amount of examples that are less than 3 seconds are only 0.04% and 2.73% respectively. Thus, discarding these examples is a minor change to the datasets.

Reviewer comment (Experiments):
How about combining multiple sentences and using the original first sentence as a chunk instead of a 3-second chunk? In this case, we would also have access to the transcription corresponding to the spoken prompt part easily. We can have more precise control of the text output to perform ASR inside the framework explicitly.

Author response:
Thanks for the suggestion. We focused on continuations of 3-second prefixes to match past work, but we can try this in future work regarding the use of the Spectron framework as an ASR or TTS system.

Reviewer comment (Theory):
Around equation (3), $\mathcal{P}_s$: why does $\mathcal{P}_s$ depend on position $s$?

Author response:
Thanks for finding this typo, we removed the $s$ from the projection layer $\mathcal{P}$.

Reviewer comment (Experiments):
In section 4.1, the last sentence, "For semantic and acoustic quality, Spectron was trained with a model of 350 million parameters, while for the question answering task, Spectron was trained with a model of 1 billion parameters," I could not understand it. Do you mean you use 350M models for Sections 4.2.1 and 4.2.2 and 1 B models for Section 4.2.3? How are they different?

Author response:
Thank you for the comment. Yes, we mean that we use the 350M LM for Sections 4.2.1 and 4.2.2 and 1B LM model for Section 4.2.3. All other configurations of the models are identical in both methods. We did this to better compare Spectron in the audio continuation task since other audio continuation methods use smaller models. We wrote it explicitly in the revision of the paper.

Reviewer comment (Experiments):
Is the question-answering task performed with zero-shot without fine-tuning it? Please clarify it.

Author response:
The question answering task is performed without fine-tuning; it is performed zero-shot. Spectron was only trained for speech continuation as described in Section 3.2. The world knowledge encapsulated in the pre-trained LLM allows Spectron to correctly answer spoken questions.