Paper: LAURAGPT: LISTEN, ATTEND, UNDERSTAND, AND REGENERATE AUDIO WITH GPT (ICLR-2024)

Reviewer comment (Experiments):
By the end of 2023, speech researchers generally believe that GPT-style models can handle various speech tasks, even though certain specific tasks may not achieve state-of-the-art performance when compared to baseline models of the same size. Instead of offering insights beyond the extensive experiments conducted, the authors primarily focus on demonstrating the effectiveness of GPT-style models across multiple speech tasks. I hope the authors will consider demonstrating whether multi-task learning can result in task synergy, where tasks can benefit from each other rather than being treated as separate or even conflicting objectives. For instance, if I have a 1B model that solely focuses on automatic speech recognition (ASR), would it outperform a 1B model capable of performing ASR, text-to-speech (TTS), and speech-to-text (ST) tasks? If the answer is no, then why should we incorporate all these tasks into the same model? It would be valuable for the authors to analyze the relationship between task performance in the context of multi-task learning, as evidenced in Table 2, where LauraGPT lags significantly behind the state-of-the-art LibriSpeech despite its 2B model size.

Author response:
**Response:** The multi-task learned single model of LauraGPT has the following advantages over single-task models: 

1. Multi-task learning could potentially exploit the synergy between related speech tasks and reduce over-fitting, hence LauraGPT could provide quality performance on a diverse set of tasks, and achieve better performance than single-task training, especially for tasks with limited training data.  

2. Since multi-task learning could learn a single model capable of supporting a diverse set of tasks, it greatly simplifies the practical deployment and applications. LauraGPT can provide a diverse set of audio processing capabilities through the unified API and model implementation.  

Please refer to Our Global Response to Q3 for more details. Our Global Response to Q3 clearly demonstrates the significant performance improvements from multi-task learning of LauraGPT over single-task training on tasks with limited training data.

$$ $$

> as evidenced in Table 2, where LauraGPT lags significantly behind the state-of-the-art LibriSpeech despite its 2B model size.

**Response:** In our Global Response to Q1, we provide a summary of the performance comparisons between our multi-task learned LauraGPT with baselines on each task.  As analyzed below, on ASR task, LauraGPT produces competitive performance on  both Chinese and English test sets compared to competitive baselines.

**For ASR task**,  the baselines are competitive Paraformer and Whisper Large V2.  As shown in Table 2,  on the Chinese test sets,  LauraGPT greatly outperforms Whisper by **-3.9** and **-2.3** absolute on CER and performs comparably to Paraformer with a much smaller amount of training data. On the English test sets, LauraGPT achieves comparable performance to Paraformer and performs better on the more noisy test set, but performs worse than Whisper Large V2 as Whisper Large V2 uses much more English training data than LauraGPT. Note that there are other **targeted optimized models** [1] that can achieve better performance than Whisper Large V2 on LibriSpeech test sets. However, they only focus on English speech recognition and benefit from additional technologies such as language model decoding.

[1] C. S, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing.

Reviewer comment (Novelty):
In the realm of fundamental speech models, are there any emerging points of interest similar to those in the field of natural language processing (NLP)? Exploring this aspect could be a valuable research direction. If speech researchers are unable to answer this question, I believe that running multi-task learning experiments alone may not be sufficient to construct the next generation of speech models.

Author response:
**Response:**  Thank you for the valuable question. There are emerging points of interest for fundamental speech models that are similar to those in the field of NLP.  This is a tremendously valuable research direction. Our work in this paper achieves one important milestone for this research question, as we explored and provided promising answers to the following question:
- How to design more efficient and scalable GPT-style models than existing approaches that can leverage large-scale labeled data and achieve state-of-the-art performance on a diverse set of speech tasks including speech recognition, understanding and generation, using a single model?  Note that previous fundamental speech models either focus solely on speech recognition and understanding but neglect generation tasks, or support speech generation but suffer from significant performance degradation on recognition and understanding tasks.

As shown in Appendix B.3 and Section 5.2, multi-task learning and the combination of continuous features and discrete features for audio is crucial for achieving the above goal.

For the next generation of fundamental speech models, we are inspired by the recent advances of large language models (LLMs) in NLP, and we envision that the fundamental speech models should have the following capabilities:
- In-context learning ability like GPT-3, which can learn from few-shot examples and adapt to new tasks, such as predicting the age of the speaker from a speech sample.
- Instruction-following ability like InstructGPT and ChatGPT, which can perform the appropriate speech-related task given a natural language instruction, such as synthesizing a speech with a specific emotion or style.
- General audio modeling ability, i.e., speech, non-speech audio, and music, such as music generation. 

We consider the following steps to progress from LauraGPT into the next generation of fundamental speech model:
1. Self-supervised pre-training exploring large-scale unlabeled data and enhancing cross-modality alignment. We plan to explore tasks such as next token prediction on unlabeled speech and text before the multi-task supervised training in our paper.
2. Extend the task categories for supervised pre-training, such as music understanding and generation, speaker tasks such as speaker diarization and so on. We plan to add more supervised training tasks into LauraGPT.
3. Construct instruction-following data and conduct supervised fine-tuning. We could generate instruction texts for all audio-related tasks using competitive LLMs (e.g., GPT-4) and pair the instruction with corresponding samples.
4. Optionally, we can collect data and conduct RLHF to make the fundamental speech model more harmless, helpful, and honest. 

Please note that our paper demonstrates that our current LauraGPT has made solid progress and reached one important milestone toward a speech foundation model. As demonstrated in the above steps, from LauraGPT to the next-generation speech foundation model we envisioned, **most remaining efforts are in more data collection and more training**.  **There is no need to modify the model architecture**. We hope that our paper can provide insights and inspiration for future research on fundamental speech models.

Reviewer comment (Experiments):
Apart from Speech translation, could you list more complex tasks that should use fundamental model to solve rather than do them one by one?

Author response:
**Response:**  As stated in Section 3.5, with its modular and flexible design, LauraGPT provides an extensible framework to support complex tasks. By breaking a task into sub-tasks among the basic tasks used in training and cascading the raw inputs and model outputs of sub-tasks, LauraGPT can perform more complex tasks than the basic tasks. 

Similar to the speech-to-speech translation (S2ST) example, LauraGPT can perform more complex tasks by chaining together basic tasks as described above. Here are a few examples of other complex tasks that LauraGPT can support rather than doing them one by one:

1. **Rich transcription**: We can extend LauraGPT to simultaneously transcribe audio into content, speaker information (speaker identification, etc), paralinguistic information (emotion, etc.) and high-level semantic information (intent, slots, etc.) by including different task IDs at the generation process. This approach could avoid error accumulation in a pipelined approach and is more efficient than performing these tasks individually.

2. **Rich translation**: We can further chain the above basic tasks of rich transcription with the MT task to perform rich translation of audio input. The audio input can provide additional paralinguistic information to improve the translation accuracy.

3. **Noise-robust ASR**: We can implement noise-robust ASR by chaining tasks and creating the following input sequence: [noisy speech embedding, <SE>, embedding of the enhanced speech,  <ASR>].  Since SE and ASR are jointly trained for LauraGPT, LauraGPT could effectively exploit embeddings of the original noisy speech and enhanced speech for noise-robust ASR.

Reviewer comment (Evaluation):
If the performance is worse than train an ASR model alone, what is the value of the multi-task learning model?

Author response:
**Response:** The multi-task learned single model of LauraGPT has the following advantages over single-task models: 

1. Multi-task learning could potentially exploit the synergy between related speech tasks and reduce over-fitting, hence LauraGPT could provide quality performance on a diverse set of tasks, and achieve better performance than single-task training, especially for tasks with limited training data.  

2. Since multi-task learning could learn a single model capable of supporting a diverse set of tasks, it greatly simplifies the practical deployment and applications. LauraGPT can provide a diverse set of audio processing capabilities through the unified API and model implementation.  

Please refer to Our Global Response to Q3 for more details. Our Global Response to Q3 clearly demonstrates the significant performance improvements from multi-task learning of LauraGPT over single-task training on tasks with limited training data.

Reviewer comment (Experiments):
The ablation regarding discrete tokens vs continuous + discrete tokens feels incomplete without also using the VALL-E style token prediction setup. Currently, the token prediction scheme is similar to that used by SPEAR-TTS wherein each quantizer level token is predicted one-by-one before moving on to the next audio-frame’s tokens.

Author response:
**Response:**  Our response to this question is two-fold:

1. The goal of the current ablation study is to evaluate the impact of combining continuous features and discrete tokens in LauraGPT on **ALL** relevant spoken language processing tasks, where the inputs of GPT are continuous representations and the outputs are discrete tokens. The token prediction scheme of VALL-E can be applied to audio generation tasks, such as TTS and SE. However, it is not straightforward to apply it to **audio recognition and understanding tasks**, such as ASR, S2TT, and AAC. Therefore, we have not included the comparison to the VALL-E style token prediction setup in the paper.

2. On the other hand, taking SE as an example of audio generation tasks, we evaluated the VALL-E style token prediction in our preliminary experiments. In these experiments, we kept the model inputs and architectures the same as LauraGPT and only replaced the token prediction scheme with the VALL-E style. The results are as follows.  We find that the token prediction scheme of LauraGPT significantly outperforms VALL-E in terms of CER and WER and also improves PESQ while obtaining the same STOI. These results indicate the superiority of predicting discrete tokens through GPT and converting them into a continuous format using a codec vocoder. 

|Token prediction scheme|PESQ|STOI (%)|CER|WER|
|------|------|-------|------|-----------------|
|VALL-E|2.55|88.0|10.52|19.32|
|LauraGPT|2.97|88.0|9.05|15.94|

Reviewer comment (Evaluation):
Some statements are not clearly backed up by experiments. For example, one of the main contributions listed is the fact that continuous and discrete representations of audio are used in LauraGPT and that this preserves both fidelity and generality of audio data. Firstly, I am not quite sure what it means to preserve generality of audio. Second, while it is shown in the ablation that the Discrete IO model suffers, it is not clear to me how these results show that fidelity and generality is preserved because of the use of combined representations. All I see is that performance on various task is improved by using the combination. Also, one additional benefit the combined representation model sees is the use of the codec vocoder. Perhaps that is the source of the improvements in LauraGPT?

Author response:
**Response:**  In Section 1, we explain that we use continuous features to represent the input audio to ensure high performance on **audio recognition and comprehension tasks**, and use codec-based discrete features to represent the output audio, which enables joint autoregressive modeling with text features for **audio generation tasks**. We show in our experiments that continuous features for audio (such as Filterbank) have notable advantages over discrete units on audio recognition, understanding, and audio-signal-related tasks, such as ASR, S2TT, and SE, as analyzed in Section 5.2. Therefore, our model achieves a better trade-off between model uniformity (generality) and high performance (fidelity) on diverse categories of audio tasks than existing approaches that use discrete features for both input and output audio. The generality of audio here refers to the ability of our model to handle different types of audio tasks in a unified framework, by using codec-based discrete features to represent the output audio and jointly modeling them with discrete text tokens. We attribute the fidelity of audio mainly to the use of continuous features for input audio. In the ablation experiment using Discrete IO in Section 5.2, we compare the performance of our LauraGPT with a model that uses discrete features for both input and output audio (Discrete IO). We find that Discrete IO significantly degrades ASR, S2TT and SE tasks, while LauraGPT has good performance on these tasks. This demonstrates that our model preserves both the fidelity and generality of audio data by using a combination of continuous and discrete representations.

Reviewer comment (Writing):
Section 3.4 would be well served with some more detail. The reader would benefit from some repeating information that the GPT model only uses the first quantizer. I found it difficult to understand initially and had to read from the start of section 3 again.

Author response:
**Response:** We have clarified the details of only using the first quantizer to tokenize audio signals in the revised Section 3.4. The remaining 31 quantizers are only used at the training stage of the codec vocoder.

Reviewer comment (Writing):
Do I understand correctly that the model uses continuous features from only the input audio, and uses audio token embeddings for previous audio tokens, meaning that the generated audio is always seen as tokens within the GPT model? It would benefit the reader if this is stated in the text explicitly as well.

Author response:
**Response:** Your understanding is correct. As stated in Section 3, for audio inputs, we extract the log-compressed Mel spectrogram features and feed them into the audio encoder, while the audio outputs are discretized into tokens using the audio tokenizer. This means that the generated audio is always seen as tokens within the GPT model, as you correctly understood. We have made this point more explicit in the revised version.

Reviewer comment (Writing):
The task-related token included in the matrices is not explained enough, how is it utilized and how is it embedded to give the information of the types of the tasks. It lacks some details about it in the description.

Author response:
**Response:** The task-related token (also called taskid in the paper) is added as a special symbol directly to the dictionary. Therefore, it can be converted to an embedding through an embedding matrix like the text tokens. For example, in the case of the ASR task, the input audio is passed through an audio encoder to obtain the corresponding audio embedding. The task-related token and the output text are converted to the text embedding through the embedding matrix, where the first embedding corresponds to the <ASR> taskid. These embeddings are then concatenated as [audio embedding, text embedding], which serves as the input for QWen LLM. 

The task-related token can help the model distinguish between different tasks with the same input. We use ASR and S2TT tasks as an example. Given the same input speech utterance, if we add <ASR> task ID after the audio embedding, the model will generate the ASR recognized result. And if we add <S2TT> taskid, the model will output the translated text.

Reviewer comment (Evaluation):
In evaluation part, there’s a lack of adequate analysis of the relationship between the poor performance in some tasks and model size.

Author response:
**Response:** Please refer to our Global Response to Q1 for performance analysis on all tasks, which includes a detailed analysis of reasons causing relatively lower performance of LauraGPT on some metrics on some test sets. Note that none of the relatively lower performance of LauraGPT compared to baselines is caused by model size differences.

Reviewer comment (Presentation):
If the new capability enabled by the proposed approach is audio output, then one of the most relevant systems appears to be AudioGPT, which also supports speech enhancement. The reason that it is not compared or included in Table 1, which shows capabilities of different existing models, is that it integrates an "expert audio model with LLM", which I don't fully understand the meaning of. An expanded explanation of this would be very useful in understanding this key point.

Author response:
We respectfully disagree on considering AudioGPT as one of the most relevant systems to our LauraGPT. As we stated in Section 2.1, our LauraGPT is a **single unified audio-text model that can directly process both audio and text inputs and generate outputs in speech and text modalities**. In contrast,  AudioGPT is an integrated AI system that **integrates and interfaces specialized audio models, such as ASR and TTS models,  for speech input and speech output (i.e., the expert audio model we mentioned in Section 2.1) with LLM**. Note that these specialized audio models add more complexity, consume more resources, and cause AudioGPT to be prone to unavoidable error accumulation problems.  Hence, AudioGPT is drastically different from our LauraGPT and is not compared to or included in Table 1.

Reviewer comment (Presentation):
Regarding Table 1, it should be clarified that it is showing just what the model has been trained/evaluated on, not what it is necessarily capable of. For example, any system that can perform speech to text translation should also be able to perform automatic audio captioning. Similar arguments hold for speech emotion recognition and spoken language understanding.

Author response:
We agree with you that Table 1 only shows the tasks the most related multi-task unified audio-text models are trained and evaluated on, not their capabilities. We have updated the caption of Table 1 to clarify this and also revised the text in Section 1 since VioLA and AudioPaLM should be able to perform AAC. However, it is important to point out that decoder-only models using discrete speech tokens such as VioLA and AudioPaLM may suffer from the information loss caused by quantization of speech signals into discrete tokens, which leads to significant performance degradation over models using continuous speech features, as our ablation study in Section 5.2 shows that by using continuous features for audio input, LauraGPT significantly outperforms the counterpart using discrete features on ASR, S2TT, and SE tasks.

Reviewer comment (Writing):
A definition of the "endless looping problem" and "loop ratio" would be helpful in the appendix.

Author response:
For some test cases of SE task, if the model generates several fixed tokens repeatedly and cannot stop the generation until reaching a pre-defined maximum length, we consider the problem of endless looping decoding occurs and consider this test case to be an "endless decoded case". To evaluate the probability of occurrence of this problem, we define the metric, "loop ratio", which refers to the fraction of "endless decoded cases" among all test cases. We have clarified the definition of "endless looping problem" and "loop ratio" in the revised Appendix B.1.

Reviewer comment (Evaluation):
The statement "These results indicate that LauraGPT tends to generate captions that closely match one of the references..." should be accompanied by an explanation of how this conclusion is reached.

Author response:
As explained in our Global Response to Q1, for the ACC task, SPICE is designed to capture the **specificity** and accuracy of the generated captions in terms of the semantic content, while CIDEr focuses on evaluating the **consensus** or agreement between the generated captions and the reference captions. SPIDEr can be viewed as an average of SPICE and CIDEr. From Table 6, we find that the single model LauraGPT achieves a comparable SPICE score but underperforms on CIDEr and SPIDEr, compared with the Ensemble baseline. Therefore, we think that LauraGPT tends to generate captions that closely match one specific reference rather than the consensus of references. We have clarified this reasoning process in the AAC Evaluation paragraph in Section 5.1 of the revised paper.

Reviewer comment (Writing):
Please define exactly what "clean_codec_syn" is in table 7.

Author response:
In Table 7, "Clean_codec_syn" refers to waveforms that are reconstructed using the first four codec groups extracted from the clean speeches. The results corresponding to  "Clean_codec_syn" can be considered as the upper bound for the Discrete IO models. The results in the "Clean" and "Clean_codec_syn" rows demonstrate that solely utilizing the first four codec groups extracted from the clean speeches to synthesize waveforms leads to degradation in speech quality (PESQ), intelligibility (STOI), and a significant degradation in recognition error rates. These results demonstrate the superiority of our choice of continuous representations for audio inputs. We have added the definition of "Clean_codec_syn" and this clarification in Section 5.2 in the revised paper.

Reviewer comment (Presentation):
In the appendix, prosody includes both tone and speed, so there is no need to list them separately.

Author response:
Thank you for the advice. We have removed the word "prosody" and listed "tone" and "speed" to specify the additional information conveyed in speech but not in the text, in the revised paper.

Reviewer comment (Writing):
In the appendix, the phrase "For the SER task, we collect corpora including..." should clarify whether there are other corpora used. If so, they should be listed; if not, it should be reworded.

Author response:
No other corpora are used for the SER task. We have also modified this description in the revised paper.

Reviewer comment (Novelty):
Can you clarify the difference between the proposed system and AudioGPT?

Author response:
We respectfully disagree on considering AudioGPT as one of the most relevant systems to our LauraGPT. As we stated in Section 2.1, our LauraGPT is a **single unified audio-text model that can directly process both audio and text inputs and generate outputs in speech and text modalities**. In contrast,  AudioGPT is an integrated AI system that **integrates and interfaces specialized audio models, such as ASR and TTS models,  for speech input and speech output (i.e., the expert audio model we mentioned in Section 2.1) with LLM**. Note that these specialized audio models add more complexity, consume more resources, and cause AudioGPT to be prone to unavoidable error accumulation problems.  Hence, AudioGPT is drastically different from our LauraGPT and is not compared to or included in Table 1.