Paper: BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing (ICLR-2024)

Reviewer comment (Experiments):
Speech contains much more information than just its transcription, such as speaker and emotion. Using paired ASR training data for modality alignment forgoes this extra information and simply marginalizes over it, meaning the resulting model may only be good at tasks that work with text inputs (i.e., recognition and semantic tasks). If we are extending an LLM with another modality, a natural requirement is to extend the capabilities of the model; for example, vision-language models can usually achieve image understanding, which is not possible with text-only LLMs. This kind of “extension” is not shown in this setting. It would have been valuable to evaluate the model on a non-semantic task (such as emotion recognition) that is not possible using text-only LLMs. Such a task could be adversarial for this type of training; in Section 3.1, the authors conjecture that “the LLM should behave the same no matter whether it is given the speech segment or its transcripts as input,” but this may not always hold.

Author response:
As discussed in Appendix D and referenced in the conclusion section, one of the limitations of our current method is the loss of speaker and paralinguistic information. Our focus is on aligning the semantic content of speech and text within the context of LLMs, a challenge that remains a focus of active research in the community [1,2,3]. Consequently, in the scope of our study, the statement "the LLM should behave the same, whether given the speech segment or its transcripts as input" holds true when considering semantic content alone.

However, our approach provides a pretrained end-to-end speech-text LLM with potential adaptability to capture speaker and paralinguistic information through fine-tuning on appropriate training data. Our analysis in Table 8 of Section 4.2 demonstrates that our model can be more effectively fine-tuned for downstream tasks (ST in this case) compared to models without pretraining or those pretrained with ASR. We plan to explore ways to capture speaker and paralinguistic information in future work.

\[1\] Zhang, Dong, et al. "Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities." _arXiv preprint arXiv:2305.11000_ (2023).

\[2\] Shu, Yu, et al. "Llasm: Large language and speech model." _arXiv preprint arXiv:2308.15930_ (2023).

\[3\] Wang, Mingqiu, et al. "SLM: Bridge the thin gap between speech and text foundation models." _arXiv preprint arXiv:2310.00230_ (2023).

Reviewer comment (Experiments):
In Section 2 (Figure 1), the authors find that training the modality adapter only on the ASR task is not beneficial since the input speech representations remain the same regardless of the task instruction used, meaning that the corresponding tokens are not being attended to. This is not surprising because the model only learns transcription since that is all it sees in training. The model might learn to attend to the instruction if the authors trained using a combination of different instructions, such as ASR, translation, and SLU, and also paraphrased the instructions themselves. The resulting model would still not be as good as BLSP but might avoid the representation collapse observed in Figure 2.

Author response:
**A. Representation Collapse**

The occurrence of mode collapse in speech features resulting from ASR training is not immediately evident. In computer vision, BLIP2 [4] has shown that training solely on image caption data can enable general instruction-following capabilities for zero-shot image-to-text generation tasks. Similarly, in the field of speech input, pretraining the model using the ASR task has been a common practice in prior work [1,2]. However, the benefits and limitations of this approach were not thoroughly studied.

We have explored different methods to utilize the ASR task for enabling instruction-following capabilities. As described in footnote 2, we used GPT4 to generate 100 diverse prompts for the ASR task, yet representation collapse still occurred. We also experimented with training on both English ASR and English-to-Chinese speech translation (ST) tasks using respective instructions. However, the resulting model still failed to demonstrate instruction-following capabilities for unseen tasks. Generally, training on specific downstream tasks tends to limit the model's abilities to those particular tasks, making it challenging to generalize to unseen instructions. This phenomenon has also been highlighted in a recently released work [5]. Additionally, we want to emphasize that labeled speech-text training data is quite limited outside of ASR and ST.

\[4\] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." _arXiv preprint arXiv:2301.12597_ (2023).

\[5\] Pan, Jing, et al. "COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning." _arXiv preprint arXiv:2311.02248_ (2023).

Reviewer comment (Experiments):
A common concern about joint speech-text training relates to sequence lengths. Because speech models usually need aggressive downsampling due to large sequence lengths and redundancy of semantic information, the speech-text length difference for ASR is usually solved using either cross-attention (e.g., LAS) or alignment-free training (e.g., CTC). For the BLSP model, the modality adapter takes the speech encoder representations and transforms them for input to LLAMA, which is trained on autoregressive text decoding. Therefore, it is expected that the adapter transforms the speech representations into the same space as LLAMA input representations. In the paper, the authors used simple speech inputs without long pauses, so fixed-length subsampling worked well. However, in real-world scenarios, users often speak with pauses or self-corrections, so it might be better to use variable-length subsampling techniques in the modality adapter [1].

Author response:
**B. Choice of Fixed-Length Subsampling**

Thank you for highlighting this important aspect and suggesting the exploration of dynamic downsampling strategies. We recognize that different downsampling methods can impact the model's performance, and there are indeed various design options to consider. In this study, our primary focus was on investigating the instruction-following capability for speech input. For this purpose, we chose a simple yet efficient approach, employing a CNN followed by a bottleneck layer as the adapter structure.

We are currently exploring more sophisticated methods for the adapter, which could potentially enhance the model's performance further. However, this exploration is part of our ongoing research and will be detailed in a future publication.

Reviewer comment (Evaluation):
While the BLSP performance on ST and SLU is reasonable and promising, the degradation for ASR is quite large (Table 3). LLAMA may already have seen the audiobooks from LibriSpeech, or transcripts of the TED talks from TED-LIUM 3, so these WERs might be expected to be much lower. The authors suggest (in Table 4) that this is mainly due to LLAMA’s tendency to produce fluent text output (possibly because of the prompt used). This raises the question of whether WER results can be shown only for rare words to determine if important words are transcribed faithfully, without caring about stop words.

Author response:
**A. WER on Rare Words** 

We acknowledge that a well-aligned speech-text LLM model should ideally perform ASR tasks with a lower WER, and we recognize that a 20% WER might seem high. In our experiments, the WER remains approximately 20% even after removing stop words. This outcome can be attributed to our training approach, which emphasizes semantic alignment over lexical precision. For instance, phrases like "I love cats" and "I like cats" would generate the same continuation in our model, as the adapter isn’t explicitly trained to capture fine-grained lexical details.

In our continuing research, we have found that incorporating a modest amount (approximately 10%) of ASR data alongside our continuation writing data can notably enhance the WER, reducing it to about 4-5%, while simultaneously preserving the model's broad instruction-following capabilities. However, it is important to highlight that this paper primarily concentrates on developing methods to facilitate general instruction-following abilities. Consequently, we have intentionally chosen not to optimize performance for any specific task.

Reviewer comment (Experiments):
Regarding the ASR degradation in BLSP, another question is whether the authors tried different prompts to see what is the lowest WER achievable. From an application perspective, it would be useful to have one model that can generate both verbatim and non-verbatim transcripts simply by using different prompts.

Author response:
**B. Impact of Different Prompts on WER**

You are indeed correct in noting that different instructions can influence WER. As highlighted in footnote 4, employing the prompt _"Please repeat the following words."_ results in lower WER scores—17.0 on LibriSpeech-dev and 17.4 on LibriSpeech-test. However, our current method aligns speech and text at a semantic level. Consequently, the adapter cannot capture detailed lexical information, making it difficult to address this issue merely through changes in prompts.

Reviewer comment (Writing):
Is the term “continuation writing” used before? Why did the authors not simply call it “next token prediction”?

Author response:
The term 'next token prediction' applies broadly to a variety of LLM training scenarios, including those that involve the ASR task. In such cases, the model optimizes the prediction of the reference transcript through 'next token prediction,' utilizing both textual instructions and speech features. However, we have chosen to use the term 'continuation writing' in our work to specifically distinguish our training task from traditional 'ASR' or other similar tasks.

Reviewer comment (Evaluation):
In Section 4.4, the authors show multilingual capabilities of the trained model even though the modality adapter is only trained with English ASR data. This suggests that the transformation from speech representations to corresponding text is the same (or similar) across all languages. Can the authors measure this isomorphism among spaces more carefully, perhaps using some of the techniques in [2]?

Author response:
Thank you for the suggestion to quantitatively measure isomorphism.

We selected 1,000 quadruples (English text, English speech, French text, French speech) from the CVSS test set. We extracted text features at LLM input via word embedding and obtained speech features from the output of either the Whisper encoder or our modality adapter. Sentence embeddings were then computed through average pooling, allowing us to calculate the relational similarity.

|       | **LLM input** | **Whisper Output** | **Adapter Output** |
| ----- | ------------- | ------------------ | ------------------ |
| en-fr | 0.60          | 0.44               | 0.39               |

As indicated in the table, the multilingual space of LLM inputs shows strong isomorphism, as does the space of Whisper outputs. Even when using only English ASR data for learning the transformation, the adapter’s output space retains the isomorphism characteristic of Whisper.

Reviewer comment (Writing):
The goal of the paper is not clear to the reviewer.

Author response:
While we believe that an end-to-end approach can eventually outperform a pipeline approach with expanded capabilities (e.g., using paralinguistic information), that is not the immediate goal of this paper. As explained in the "General Response," our objective is to extend the instruction-following capability of LLMs to speech input to enable zero-shot capabilities for unseen tasks in an end-to-end model.

In Section 4.1, we demonstrate this by using textual instructions to prompt the model for unseen speech-to-text generation tasks. For instance, our model, trained solely on continuation tasks, can be prompted during inference for ASR tasks as follows:

    ###[Human]: Please transcribe the following speech into text. [speech features]
    ###[Assistant]: [response]

Here, the ASR instruction can be substituted with instructions for other tasks, such as ST and SLU tasks evaluated in the paper, or tasks like summarization and question-answering. We emphasize that our goal in this paper is not to outperform the pipelined approach, but to demonstrate the model's ability to follow instructions across modalities in an end-to-end manner. The comparison with the pipeline approach is to both highlight the promising results of the end-to-end approach and the performance gap with the pipeline approach, encouraging future research to further reduce this gap and eventually surpass it. This aspect is discussed in the limitations of the current approach in the Appendix and referenced in the conclusion section.

In Section 4.2, we present analyses to highlight different aspects of our model. However, these should not be viewed as the main experiment. Table 8 illustrates that our method allows for effective fine-tuning on downstream tasks. Figure 3 demonstrates successful speech-text modality alignment, while Figure 4 investigates the impact of data scale on alignment effectiveness.

Reviewer comment (Evaluation):
If the goal is to show the effect of fine-tuning, it provides some marginal gain.

Author response:
As noted in response to Q1, our main focus in this paper is on the instruction-following capabilities for speech input, as presented in quantitative zero-shot evaluations in Section 4. The analyses in Section 4.2 are designed to highlight different aspects of our model. Specifically, Table 8 in Section 4.2 is intended to demonstrate that our pretrained model is effective not only in its instruction-following capabilities but also for fine-tuning on downstream tasks. The improvement over models without pretraining is substantial in both BLEU and COMET metrics, and is significantly better than ASR pretraining, with an average improvement of 0.5 BLEU and 1 COMET score across eight directions. However, we wish to emphasize that this aspect of the model is not the main focus of our paper.

Reviewer comment (Experiments):
Please clarify the goal of the experiments more clearly.

Author response:
While we believe that an end-to-end approach can eventually outperform a pipeline approach with expanded capabilities (e.g., using paralinguistic information), that is not the immediate goal of this paper. As explained in the "General Response," our objective is to extend the instruction-following capability of LLMs to speech input to enable zero-shot capabilities for unseen tasks in an end-to-end model.

In Section 4.1, we demonstrate this by using textual instructions to prompt the model for unseen speech-to-text generation tasks. For instance, our model, trained solely on continuation tasks, can be prompted during inference for ASR tasks as follows:

    ###[Human]: Please transcribe the following speech into text. [speech features]
    ###[Assistant]: [response]

Here, the ASR instruction can be substituted with instructions for other tasks, such as ST and SLU tasks evaluated in the paper, or tasks like summarization and question-answering. We emphasize that our goal in this paper is not to outperform the pipelined approach, but to demonstrate the model's ability to follow instructions across modalities in an end-to-end manner. The comparison with the pipeline approach is to both highlight the promising results of the end-to-end approach and the performance gap with the pipeline approach, encouraging future research to further reduce this gap and eventually surpass it. This aspect is discussed in the limitations of the current approach in the Appendix and referenced in the conclusion section.

In Section 4.2, we present analyses to highlight different aspects of our model. However, these should not be viewed as the main experiment. Table 8 illustrates that our method allows for effective fine-tuning on downstream tasks. Figure 3 demonstrates successful speech-text modality alignment, while Figure 4 investigates the impact of data scale on alignment effectiveness.

Reviewer comment (Evaluation):
Is having a BLEU score over 5 good enough in terms of model quality given that the baseline models achieve over 15? Similarly, a WER over 20% does not show the benefit of the model.

Author response:
As we acknowledged in the paper, there currently exists a performance gap between our proposed end-to-end approach and the pipelined approach. We want to note that the primary contribution of our work is to demonstrate the effectiveness of the training method in transferring instruction-following capabilities from text-based language models (LLMs) to the speech modality. We deliberately compare with the pipelined approach to highlight both the potential of our approach as well as its current limitations.

Developing end-to-end LLMs with the ability to follow cross-modal instructions for speech input is not trivial and has become an active research area [1,2,3,4]. While these studies focused on constructing new speech instruction datasets, our work provides a new approach through behavior alignment and takes advantage of existing ASR training data. In contrast to most recent works that demonstrate their effectiveness through demos, we provided both demonstrations and quantitative evaluations to understand both the strengths and limitations. It is our hope that this transparent approach will provide the research community with useful insights to continue developing new approaches to reduce the performance gap and ultimately surpass pipelined approaches.

We also want to mention the similarity to the development of end-to-end approaches to other problems. The concept of end-to-end speech translation was initially proposed in 2016 with a significant performance gap compared to cascaded systems. It was not until 2020 that end-to-end speech translation achieved comparable performance to cascaded systems and eventually became the mainstream approach. The research efforts and publications during these years have collectively contributed to the advancement of the research field.

Finally, we want to acknowledge again that more research is needed to further improve the performance of the end-to-end approach, and some of our ongoing research has achieved  improvement in this direction, as we also noted in part in responses to other reviewers. However, we believe the approach developed in this paper has its own merit and would be beneficial to the research community.

[1] Zhang, Dong, et al. "Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities." arXiv preprint arXiv:2305.11000 (2023).

[2] Shu, Yu, et al. "Llasm: Large language and speech model." arXiv preprint arXiv:2308.15930 (2023).

[3] Wang, Mingqiu, et al. "SLM: Bridge the thin gap between speech and text foundation models." arXiv preprint arXiv:2310.00230 (2023).

[4] Pan, Jing, et al. "COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning." arXiv preprint arXiv:2311.02248 (2023).

Reviewer comment (Experiments):
How did you choose the structure of the adapter?

Author response:
The modality adapter in our study serves two primary functions. First, it reduces the length of output from the Whisper encoder, which operates at a frequency of 50Hz. Second, it maps the speech feature space to the textual space of the LLM. We acknowledge that various structural options exist for the adapter. For length reduction, methods like CNN downsampling, frame stacking, or q-transformer are potential choices. Similarly, for cross-modality alignment, both MLP and transformer-based methods are feasible.

However, our study's focus was not on exhaustively exploring different adapter structures. Instead, our main contribution lies in proposing a novel training method to connect a frozen speech encoder with an LLM. Consequently, we opted for a simple yet effective approach, employing a CNN followed by a bottleneck layer for the adapter. This decision allowed us to concentrate on the core aspect of our research while maintaining effectiveness in achieving our objectives.

Reviewer comment (Experiments):
What is the performance if you use other speech encoders?

Author response:
We acknowledge that there are various options for the speech encoder, such as the Whisper model's encoder [1], USM [2], or other pre-trained speech models, which could impact the model's performance. However, our work  focuses on the methodology of aligning a frozen speech encoder with a frozen LLM to achieve instruction-following capability for speech input on unseen tasks. As such, an in-depth investigation into different speech encoders is not the primary focus of this work, and we believe that changing the speech encoder would not fundamentally alter the conclusions of our paper.

We chose the encoder of the Whisper model due to its widespread recognition and to facilitate comparison with a strong ASR model in the pipelined approach. This choice mirrors our decision to use Llama2 as the LLM, despite the availability of many open-source LLM options. Our aim was to employ well-known and robust components to showcase the effectiveness of our methodology.

\[1\] Shu, Yu, et al. "Llasm: Large language and speech model." arXiv preprint arXiv:2308.15930 (2023). 

\[2\] Wang, Mingqiu, et al. "SLM: Bridge the thin gap between speech and text foundation models." arXiv preprint arXiv:2310.00230 (2023).