Paper: Acoustic Prompt Tuning: Empowering Large Language Models with Audition Capabilities (ICLR-2024)

Reviewer comment (Theory):
The authors claimed "Diverse audio-related tasks are formulated in a sequence-to-sequence manner without imposing any constraints on input sequences", which is not true since the input sequence could not be speech or music, as claimed in the limitation section. There might also be a maximum input sequence length imposed by the use of the Audio-MAE encoder, and the Q-Former aligner.

Author response:
Although AudioMAE ingests fixed-length waveforms, the APT-LLM supports audio clips of variable length as input. Specifically, we will truncate or pad audio to 10 seconds length and generate acoustic tokens for each 10-second audio clip. After that, we will concatenate the acoustic tokens together so that the APT-LLM can digest varying-length audio clips. 

Regarding “ without imposing any constraints on input sequences”, we replace it with “impose zero constraints on input format” to avoid potential misunderstanding.

Reviewer comment (Novelty):
The authors claimed that one of the key contributions of the paper is: "this is the first attempt to unify fully-supervised learning with in-context learning." It is not clear to me what this means precisely. The authors need to make the motivation and benefits of combining multitask training with in-context learning clear.

Author response:
Thanks for your valuable feedback. We have updated the contribution part for clarity.

In terms of the benefits of the combination, as shown in the Table 3, fewer audio-text pairs and parameters are introduced to the overall network compared to the existing approaches while APT still yielded a comparable performance compared to other audio-language models.

Reviewer comment (Evaluation):
The performance of the proposed approach is not satisfying based on Table 2, in particular on audio tagging on the AudioSet dataset.

Author response:
We notice that compared to the common classifiers, there is a performance drop due to the difficulty for the open-set issue and text matching. For example, when CLAP, a open-set classifier pretrained with contrastive loss, is used for evaluation, it calculates the similarity with 527 classes whereas APT predicts an appropriate token out of 32k tokens. Another issue is how to match the predicted answers with the ground-truth labels. Suppose APT outputs “rat” and we get the class of “mouse” in the ground truth. In that case, a proper text matching approach is required for a fair comparison. During evaluation on audio classification tasks, we used the APT’s checkpoint from audio-text-alignment pretraining as the text encoder. However, we did find that using different text encoders to evaluate the same checkpoint impacts the classification values a lot. We are planning to apply some discriminative text encoders, such as CLIP text encoder and ChatGPT Ada model, for the evaluation, and we will add the discussion on text matching to the Appendix.

Regarding the audio captioning task, training on multiple datasets does not improve the performance on a specific dataset because the model cannot fit the dataset-specific vocabulary. In addition to the general-purpose audio [1], we found the same phenomena in other domains [2,3]. Following [3,4], we have conducted additional experiments to finetune APT on both Clotho and AudioCaps datasets for two epochs only. We compared the finetuned APT with expert models trained on the same datasets. As shown in Table 4, the APT yielded the best scores on both tasks.

Reviewer comment (Theory):
It sounds less sensible to me to use a standard Q-Former to convert audio input sequences into a fixed number of 32 tokens. Generating a fixed number of tokens is a good choice for images with fixed sizes, but not so much for audio input sequences due to their variable input lengths.

Author response:
Regarding “Generating a fixed number of tokens is not so much for audio input sequences due to their variable input lengths”, as we mentioned in “[Maximum input sequence length]”, it is actually 10-second audio clip corresponds to a fixed number of tokens. We have updated the manuscript for a better understanding.

Reviewer comment (Experiments):
It is not clear what are the benefits of having the ATM, ATC, and AGTG multitask training on the Q-Former aligner.

Author response:
Regarding the triplet losses in the audio-text alignment pretraining, we did a few preliminary studies on the pretraining and decided to use the combination for a better alignment.

Reviewer comment (Reproducibility):
What's the size of the Vicuna LLMs used in the paper?

Author response:
We used Vicuna 7B throughout the work. We have clarified this in the paper.

Reviewer comment (Evaluation):
The levels of accuracy achieved by APT-LLM on the audio tagging and audio captioning are unfortunately slightly disappointing (Table 1. Section 4.2). They fall below the accuracy levels of domain-expert models, by a noticeable margin in the case of audio tagging. The authors argue that this has to do with the open-ended nature of APT-LLM's output, which puts APT-LLM at a disadvantage compared to the closed-ended nature of the previous models audio tagging models. However, in the context of visual benchmarks such as ImageNet, open-ended text-visual models such as CLIP have previously outperformed domain expert models such as various types of CNNs. A similar under-performance can be seen in the few-shot audio classification task (Table 2), especially for the 12-way case. The authors try to attribute this poor performance to the length of the input, but they did not explain why the model couldn't be configured and trained with a sufficiently long input context window in order to accommodate this task and thereby address this limitation.

Author response:
For the audio classification, we found that there is a performance drop due to the difficulty for the open-set problem and text matching. For example, when CLAP, a open-set classifier pretrained with contrastive loss, is used for evaluation, it calculates the similarity with 527 classes whereas APT predicts an appropriate token out of 32k tokens. It is harder for generative models to carry out multi-label classification where the threshold is essential to predict the presence of a sound event. 

Another issue is how to match the predicted answers with the ground-truth labels. Suppose APT outputs “rat” and we get the class of “mouse” in the ground truth. In that case, a proper text matching approach is required for a fair comparison. During evaluation on audio classification tasks, we used the APT’s checkpoint from audio-text-alignment pretraining as the text encoder. However, we did find that using different text encoders to evaluate the same checkpoint impacts the classification values a lot. We are planning to apply some discriminative text encoders, such as CLIP text encoder and ChatGPT Ada model, for the evaluation, and we will add the discussion on text matching to the Appendix.

Regarding the audio captioning task, training on multiple datasets does not improve the performance on a specific dataset because the model cannot fit the dataset-specific vocabulary. In addition to the general-purpose audio [1], we found the same phenomena in other domains [2,3]. Following [3,4], we have conducted additional experiments to finetune APT on both Clotho and AudioCaps datasets for two epochs only. We compared the finetuned APT with expert models trained on the same datasets. As shown in Table 4, the APT yielded the best scores on both tasks.

Reviewer comment (Writing):
The section on integration of BLIP-2 and APT-LLM (Section 4.4) is not easy to follow for readers not familiar with the audio-visual tasks. Not enough background information is provided about either the BLIP-2 model architecture or the audio-visual learning task that the author performed evaluation on. This unfortunately makes this part of the claimed contribution less convincing.

Author response:
Thanks for your feedback. To improve the readability, we have added the introduction on both BLIP-2 structure and the audio-visual task to Section 4.4.

Reviewer comment (Presentation):
The diagram in Figure 1 seems to miss a part between the input text and the audio aligner for tokenizing the input text and looking up the embeddings for the input text for the aligner's use. This figure needs clarification.

Author response:
Thanks for pointing that out. e have now updatedFigure 1.

Reviewer comment (Experiments):
How does the frozen LLM receive the audio-text juxtaposed embeddings? Its built-in embedding lookup layer should have been removed so that the mixed audio-text embeddings can be passed directly as the input to the LLM. This begs the question of why the authors decided to freeze the LLM. It seems that by freezing the LLM, the burden and opportunities for learning are limited to the audio encoder and the audio aligner. Is it possible allowing the weights of the LLM to change and adapt during training would lead to better learning outcomes by the entire APT-LLM model? The authors didn't describe any hyperparameter tuning processes.

Author response:
Actually, the LLM’s embedding layer was not removed in APT-LLM. Instead, we truncated the input sequence as per positions of audio indicators and concatenated them again once acoustic/text tokens have been encoded (separately). 

Regarding the reason why we froze the LLM, the motivation of this work is to extend the existing LLMs/VLMs to the audio domain and prevent it from losing its original abilities as an LLM/VLM. To this end, instead of adjusting the LLM, we learn an audio aligner to generate soft prompts, conditioned on audio-text embeddings, for the language model. While finetuning LLMs with PEFT will boost the overall performance, we argue that it may jeopardise the LM performance on original tasks, even causing catastrophic forgetting. By tuning audio aligner only, we build up an audio language model with fewer parameters and fewer training samples while yielding a comparable performance on each understanding task compared to other audio foundational models.

Reviewer comment (Presentation):
Figure 3 in the appendix lacks legends. In addition, the y-axis on the right-hand side is unlabeled. As a result, it is unclear what are plotted by the blue and orange curves, which makes this figure hard to read.

Author response:
Thank you for your comment. We will update the revised one in the latest version of the manuscript.

Reviewer comment (Evaluation):
Performance on Certain Tasks: Despite the novelty of the idea, the performance of APT-LLMs on the close-ended datasets - ESC50 (few shot classification) and AudioSet (captioning) tasks is not competitive compared to state-of-the-art, task-specific models. This indicates a need for improvement in these areas.

Author response:
We notice that compared to the common classifiers, there is a performance gap due to the difficulty for the open-set problem and text matching. For example, when CLAP, a open-set classifier pretrained with contrastive loss, is used for evaluation, it calculates the similarity with 527 classes whereas APT predicts an appropriate token out of 32k tokens. Another issue is how to match the predicted answers with the ground-truth labels. Suppose APT outputs “rat” and we get the class of “mouse” in the ground truth. In that case, a proper text matching approach is required for a fair comparison. During evaluation on audio classification tasks, we used APT’s checkpoint from audio-text-alignment pretraining as the text encoder. However, we did find that using different text encoders to evaluate the same checkpoint impacts the classification values a lot. We are planning to apply some discriminative text encoders, such as CLIP text encoder and ChatGPT Ada model, for the evaluation, and we will add the discussion on text matching to the Appendix.

Regarding the audio captioning task, training on multiple datasets does not improve the performance on a specific dataset because the model cannot fit the dataset-specific vocabulary. In addition to the general-purpose audio [1], we found the same phenomena in other domains [2,3]. Following [3,4], we have conducted additional experiments to finetune APT on both Clotho and AudioCaps datasets for two epochs only. We compared the finetuned APT with expert models trained on the same datasets. As shown in Table 4, the APT yielded the best scores on both tasks.

Reviewer comment (Experiments):
Handling of Errors: It is unclear how the model handles potential discrepancies or errors in the labeled examples used for in-context learning. This raises questions about the robustness of the model in real-world scenarios where such issues are common.

Author response:
We can roughly divide the errors into two types:
a) label error (noisy label): It is actually really interesting to notice in our preliminary study that if we replaced the name of sound event with an abstract but distinguishable one, e.g., “dog” -> “class A”, APT will still figure out the most similar one in terms of acoustic characters and return with “class A”.
b) audio discrepancy: this is really a good question. To be honest, we didn’t set up experiments for that so far. Yet, it is very interesting for future work. We have added our discussion on error handling for the future work at the end of the manuscript.

Reviewer comment (Experiments):
How can APT's performance on tasks like ESC50 and AudioSet be improved to be more competitive with task-specific models?

Author response:
We notice that compared to the common classifiers, there is a performance gap due to the difficulty for the open-set problem and text matching. For example, when CLAP, a open-set classifier pretrained with contrastive loss, is used for evaluation, it calculates the similarity with 527 classes whereas APT predicts an appropriate token out of 32k tokens. Another issue is how to match the predicted answers with the ground-truth labels. Suppose APT outputs “rat” and we get the class of “mouse” in the ground truth. In that case, a proper text matching approach is required for a fair comparison. During evaluation on audio classification tasks, we used APT’s checkpoint from audio-text-alignment pretraining as the text encoder. However, we did find that using different text encoders to evaluate the same checkpoint impacts the classification values a lot. We are planning to apply some discriminative text encoders, such as CLIP text encoder and ChatGPT Ada model, for the evaluation, and we will add the discussion on text matching to the Appendix.

Regarding the audio captioning task, training on multiple datasets does not improve the performance on a specific dataset because the model cannot fit the dataset-specific vocabulary. In addition to the general-purpose audio [1], we found the same phenomena in other domains [2,3]. Following [3,4], we have conducted additional experiments to finetune APT on both Clotho and AudioCaps datasets for two epochs only. We compared the finetuned APT with expert models trained on the same datasets. As shown in Table 4, the APT yielded the best scores on both tasks.

Reviewer comment (Experiments):
How can the model be improved to handle discrepancies or errors in the labeled examples used for in-context learning?

Author response:
We can roughly divide the errors into two types:
a) label error (noisy label): It is actually really interesting to notice in our preliminary study that if we replaced the name of sound event with an abstract but distinguishable one, e.g., “dog” -> “class A”, APT will still figure out the most similar one in terms of acoustic characters and return with “class A”.
b) audio discrepancy: this is really a good question. To be honest, we didn’t set up experiments for that so far. Yet, it is very interesting for future work. We have added our discussion on error handling for the future work at the end of the manuscript.

Reviewer comment (Experiments):
How does APT perform for more specific types of audio (e.g., music, notes...)?

Author response:
Regarding more specific types of audio, we are planning to extend APT to speech and music as well. Since we use AudioMAE, a generative model trained with reconstruction loss, as the audio encoder, APT has the potential to generalise to recognise speech and music.

Reviewer comment (Novelty):
The Figure 2 in this paper is very similar to Figure 2 in BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (https://arxiv.org/pdf/2301.12597.pdf), just replace the word image to audio. Thus, the novelty of this paper is very limited compared with BLIP-2. The Figure 1 in this paper is similar to Figure 3 in BLIP-2. The authors use the exactly three loss in BLIP-2, matching loss, contrastive loss and gounded text generation loss, with the exactly three mask strategies, and the authors use the learnable query.

Author response:
Motivated by BLIP-2’s idea on Qformer, this work applies a BERT-ish transformer to bridge the modality gap. After the preliminary study, we found the combination of contrastive loss, grounded loss, and language generative loss, as depicted in BLIP-2, performed the best to pretrain the transformer. We also adjusted the number of transformer blocks and dimension of hidden states for a better performance of audio alignment. While BLIP-2 pretrained this transformer with the captioning task only, this work combines captioning and tagging tasks to mitigate the lack of high-quality data in the audio domain. 
  
The distinction of this work is to learn a more versatile framework that is capable of learning from more diverse tasks and can easily adapt LLMs/VLMs to the audio modality. As shown in Fig. 1, this work highlights a trainable audio identification token added to each group of acoustic tokens from the audio aligner and the juxtaposition between grouped acoustic tokens and text embeddings. This enables the proposed framework to address multiple diverse tasks, such as few-shot learning and complex audio understanding, in both training and inference stages compared to BLIP-2. Moreover, to evaluate more rigorously how the proposed model benefits from LLMs beyond common audio tasks, we proposed a new task, namely natural language audio reasoning, that tests the logic capacity of models with a series of comparison and summarization tasks.

Reviewer comment (Novelty):
Model Capability: Unlike methods such as SpeechGPT, the approach presented in this article limits the use of speech modality to input only, preventing the synthesis of speech output. This results in a model lacking genuine speech interaction capability.

Author response:
SpeechGPT is a great work that integrates speech recognition and generation within one architecture. However, the focus of SpeechGPT is speech recognition/generation which is quite different from this work where an adapter is studied to extend LLMs/VLMs to general-purpose audio understanding tasks.

We acknowledge that speech processing is essential to AI development, but perceiving the general audio is also challenging. For example, different from speech with well-known phoneme structure, basic units of environmental audio, namely sound events, are distributed more sparsely without a clear structure, and they are more prone to overlap with each other. 

However, inspired by the idea of SpeechGPT, this work can be extended to explore foundational models for joint understanding and generation of general audio in the future.

Reviewer comment (Experiments):
For the audio modality, speech interaction is important, why not the authors focus on this part, which is different from image modality.

Author response:
We acknowledge that speech processing is essential to AI development, but perceiving the general audio is also challenging. For example, different from speech with well-known phoneme structure, basic units of environmental audio, namely sound events, are distributed more sparsely without a clear structure, and they are more prone to overlap with each other. 

However, inspired by the idea of SpeechGPT, this work can be extended to explore foundational models for joint understanding and generation of general audio in the future.