Paper: Listen, Think, and Understand (ICLR-2024)

Reviewer comment (Evaluation):
For the open-ended questions, this work seems to focus mainly on solely LLMs assisted question answer generation. It is not intuitive to understand to what extent the model relies on the input audio versus on the common sense knowledge that is already encoded in the LLMs. It would be great to define and identify beyond current close-ended tasks with new lower level tasks which really require using the audio, such as counting sound events, ordering of events, etc. These type of questions might already exist in the proposed dataset, it would provide more insights to dive deeper into those.

Author response:
We totally agree with the reviewer on this point! And this is the reason why we rigorously evaluate the model performance on closed-ended tasks while the main novelty on open-ended tasks. In the paper, Appendix K, we conduct a probe experiment to check if LTU can really understand the sound concepts rather than associates the acoustic concept with a sound class. In our training data, samples of the ``ambulance siren`` class are always described as ``high pitched``. To check if LTU can disentangle the concept of a high-pitch and sound class of ambulance sirens. We manually lower the pitch (with librosa.pitch_shift) of 53 evaluation audios of the ambulance siren class and check LTU's output to the question ``What is the pitch?`` on these audios. As shown in Figure 2 of the paper, LTU's prediction aligns well with the actual pitch, indicating it indeed learns the concept of pitch rather than just associating it with a specific sound class.

Reviewer comment (Evaluation):
Table 5 on the right for the training curriculum, it would be great to also include the language instruction following rate. And further discuss the correlation between the classification performance and the instruction following rate, if there is any insights that can be drawn.

Author response:
We thank the reviewer for the suggestion. Following the reviewer's suggestion, we add the instruction following rate for Table 5 right. Please see the following Table:

| Curriculum Setting                                 | Audio Classification Performance | Pure Language Task Instruction Following Rate |
|----------------------------------------------------|:--------------------------------:|:---------------------------------------------:|
| LTU (trained w/ default curriculum)                |               50.3               |                      87.6                     |
| No Curriculum (same total tr. iterations)          |               23.0               |                    **67.7**                   |
| No Open-ended Training (Stage 1&2&3 only)          |               37.1               |                      87.8                     |
| No Open-ended Training (same total tr. iterations) |               47.3               |                      86.6                     |
| Use a large learning rate of 1e-3 in Stage 4       |               49.5               |                    **75.2**                   |
| Add half an epoch to the training in Stage 4       |               50.3               |                      86.7                     |
| Train for only one epoch in Stages 1&2             |               49.7               |                      88.5                     |
| Cut tr. iterations by 50% across all stages        |               45.3               |                      87.9                     |

Interestingly, the relationship between the audio classification task (reflecting the model's audio perception ability) and the pure language task instruction following rate (indicative of the model's original language ability) is not linear. We find that the **training setting** significantly impacts the model's language abilities. It's possible for two models with similar audio perception performance to exhibit markedly different capabilities in language tasks. There are two settings that can lead to a lower language task instruction following rate:

- Absence of a training curriculum. This finding reinforces the necessity of a structured training curriculum for optimal model development.
- Using a too large learning rate. Though the audio perception performance does not change much, training a model with a learning rate of 1e-3 can diminish its performance in language tasks compared with a learning rate of 1e-4.

We thank the reviewer once again for raising this insightful question. We will incorporate this finding into the revised version of our paper.

Reviewer comment (Evaluation):
The acoustic features mentioned in 3.1 are also generated from LLMs, how are these verified?

Author response:
We thank the reviewer for the question. We utilize GPT-3.5-Turbo, a strong large language model, to generate acoustic feature descriptions. These descriptions are used to train the LTU model to enhance its capability to recognize acoustic features alongside sound classes and to enable LTU to generalize predictions for unseen sound classes based on their acoustic features. In the following table, we show examples of GPT-3.5-Turbo generated descriptions for 10 major and 10 minor sound classes from AudioSet. For diversity consideration, for each sound class, we generate 10 different descriptions but show only one sample in the Table for each due to space limitations. Our manual evaluation confirms that these feature descriptions are high-quality, being both factually accurate and informative for a range of common and rare sounds.

| Sound Class                   | # Samples in AudioSet | Sample GPT-3.5-Turbo Generated Acoustic Description |
|-------------------------------|:---------------------:|:---------------------------------------------------:|
| Music                         |       1,011,305       |             distinctly pitched and timed            |
| Speech                        |       1,010,480       |             rich in frequency modulation            |
| Vehicle                       |        128,051        |                low, rumbling and loud               |
| Inside, small room            |         76,694        |               intimate and reverberant              |
| Guitar                        |         51,597        |               bright and well-rounded               |
| Plucked string instrument     |         44,565        |      plucked, staccato articulation of a string     |
| Singing                       |         42,493        |                 vibrant and melodic                 |
| Animal                        |         40,758        |            usually high-pitched and short           |
| Electronic music              |         38,958        |             synthesized and exaggerated             |
| Outside, rural or natural     |         35,731        |              generally calm and mellow              |
| Dental drill, dentist's drill |          182          |                  sharp and metallic                 |
| Crushing                      |          176          |                   gritty and harsh                  |
| Hoot                          |          169          |               high pitched and shrill               |
| Finger snapping               |          164          |                 percussive and sharp                |
| Squawk                        |          160          |                  sharp and piercing                 |
| Splinter                      |          153          |                sharp, loud and brief                |
| Pulleys                       |          152          |           a high-pitched, metallic ringing          |
| Creak                         |          149          |            high pitched, harsh and sharp            |
| Gargling                      |          137          |             low pitch, gurgling and wet             |
| Toothbrush                    |          127          |               high-pitched and buzzing              |

One problem related to this is that we apply the same set of descriptions to *all* samples labeled of the sound class. While these descriptions are generally accurate for the majority of samples, there can be exceptions. For instance, the ``vehicle`` class is described as ``low, rumbling, and loud``, but this might not hold true for some samples with quieter vehicle sounds. However, we observed that most generated acoustic features appear in **contrasting pairs**. For example, ``high pitched`` is used in the descriptions of 243 audio classes, whereas its opposite, ``low pitched``, is found in 77 classes. This presence of opposing features allows the LTU model to effectively disentangle the specific acoustic feature from the sound class, even when we provide the same set of features for each sound class.

| The most frequent acoustic features generated based on the 527-class AudioSet labels, frequency shown in parentheses |
|:--------------------------------------------------------------------------------------------------------------------:|
|                      high pitched(243)-low pitched(77), loud(150)-soft(31), bright(99)-dull(10)                      |
|                           sharp(197)-mellow(44), deep(81)-shrill(43), metallic(72)-warm(49)                          |
|                         harsh(60)-soothing(20), clear(33)-muffled(22), intense(35)-gentle(17)                        |

Reviewer comment (Evaluation):
In the temporal analysis paragraph in 3.1, how are the understanding of order of sounds evaluated?

Author response:
We sincerely appreciate the reviewer's suggestion to explore the temporal analysis capability of the model, and in response, we are proceeding with the following new experiments. Specifically, we conduct experiments on the audio event order recognition task (seen in the closed-ended training stage) and audio event counting task (unseen in the closed-ended training stage) with ESC-50, a dataset not being used in training. For both experiments, we only include the cases in which LTU follows the instruction. 

---
**1. Recognizing the Order of Sound Events**

We create a test set where each test sample consists of two audio samples from different categories of the ESC50 dataset, with no temporal overlapping. We then ask LTU to predict the order of the sound events with the prompt ``Which sound begins and ends first?``. We use regular expressions and cosine similarity (based on gpt-text-embedding-ada) to interpret the raw LTU output. For example, for the LTU raw output ``the applause starts first, while the footsteps follow in a rhythmic pattern afterward.``, we first extract ``applause`` and ``footsteps follow in a rhythmic pattern`` using regular expressions. Then we compare the cosine similarity between the text embedding of the ground truth ``clapping`` with the text embeddings of ``applause`` and ``footsteps follow in a rhythmic pattern``, respectively. Since the cosine similarity between the text embedding of the ground truth ``clapping`` and the LTU prediction ``applause`` is higher, we count this sample as correct.

We evaluate LTU on 550 such samples, obtaining 390 correct and 160 incorrect answers, achieving an accuracy of 70.9%. This shows that LTU can recognizes the order of sound events with a reasonable accuracy.

Reviewer comment (Evaluation):
In 5.2.1, LTU can answer follow-up questions about the details, there is an example where LTU shows that the bell rings three times, are there other examples showing the counting capabilities? Is it possible to further quantify this?

Author response:
In Table 6, the first sample shows a demo where LTU recognizes the bell rings *three times*. In this section, we further quantify LTU's capacity in counting the number of appearances of sound events. Note that this task is not in our closed-ended training. One problem in evaluating this task is that the definition of the number of appearances is not clear for all sound classes, e.g., it is hard to define what an appearance of a train sound is as it could consist of multiple click-clack sounds. Thus, we create a test set where each test sample consists of one or two (repeated) audio samples from the ESC50 dataset, with no temporal overlapping. We then ask LTU to count the sounds in the audio clips using the prompt, ``In how many instances is the sound of {:s} heard?``. Since one ESC50 audio sample may contain sounds multiple times (e.g., three dog barking sounds), we calculate the Pearson Correlation Coefficient (PCC) between the LTU output count and the number of ESC50 audio clips in the test sample. 

| Sound Class    | Pearson Correlation Coefficient (PCC) | LTU Output Count Ratio for Double vs. Single ESC50 Audio Samples (Ideal = 2) |
|----------------|:-------------------------------------:|:----------------------------------------------------------------------------:|
| Chainsaw       |                  0.81                 |                                      2.5                                     |
| Brushing Teeth |                  0.79                 |                                      2.6                                     |
| Helicopter     |                  0.77                 |                                      2.1                                     |
| Siren          |                  0.58                 |                                      2.0                                     |
|                |                  ...                  |                                                                              |
| Train          |                  0.02                 |                                     1.01                                     |
| Toilet Flush   |                 -0.03                 |                                     0.98                                     |
| Frog           |                 -0.18                 |                                     0.92                                     |

As shown in the above Table, we find a high correlation for many sound classes. Additionally, we calculate the ratio between LTU output counts for test samples consisting of two ESC50 samples and test samples consisting of one ESC50 sample. Ideally, this value should be 2 because the number of appearances of sounds in two ESC50 samples should be twice the number of appearances of sounds in one ESC50 sample. We find it is indeed close to 2 for many sound classes. However, there are some sound classes that LTU cannot count well, potentially due to an undefined number of sound appearances for these classes. This demonstrates the sound event counting capability of LTU, although it has not been explicitly trained for this task in the closed-ended training stage.

---
**Summary**

To summarize, LTU demonstrates reasonable ability in temporal analysis. However, compared with classification and captioning tasks, its temporal analysis ability is weaker. We hypothesize that this is due to two reasons: 
- First, we aggressively pool the audio tokens to save computation, which may hurt fine-grained tasks.
- Second, LTU has not been trained with a sufficient amount of data for temporal analysis, as reported in [1], such training is crucial for fine-grained tasks.

[1] Peng, Zhiliang, et al. "Kosmos-2: Grounding Multimodal Large Language Models to the World." 2023.

---

We thank the reviewer again for the very valuable comment. In the revised paper, we have added this discussion to **Appendix H LTU Temporal Analysis Experiments**. We hope the new section will provide readers with a deeper understanding of LTU's performance in temporal analysis and its limitations. Furthermore, we hope that these experiments can partly answer the question "To what extent the model rely on the input audio versus on the common sense knowledge that is already encoded in the LLMs?"

Reviewer comment (Experiments):
In 5.2.1, multi-turn conversation is mentioned, are the previous audio tokens and questions also passed in as context or are the multi-turn independent each turn? This can be another interesting property for few-shot in-context learning if possible.

Author response:
We appreciate the reviewer's question. In Table 6, all questions are independent, i.e., they are asked without prior context. Our intention is to avoid providing the model with information that might hint at the answer. By 'follow-up question,' we refer to posing a new question that seeks further details.

However, LTU model inherently supports multi-round dialogue. We have conducted preliminary experiments in this direction. For instance, in the ESC-50 classification, for samples that the LTU initially misclassified, we can prompt it to make a correction

``Input: Is the sound Rooster or Hen? Why?``

``LTU Output: It is a Rooster.``

``Input: No, the sound is actually from a hen. Can you tell me why you are wrong?``

``LTU Output: I apologize for my mistake earlier. The sound in this audio clip is indeed from a hen, not a rooster. This can be inferred based on its lower pitch and shorter duration compared to a rooster's crow.``

Although this sample appears promising, we acknowledge that rigorously quantifying the success rate of such corrections, and examining the impact of model size and training on it, are non-trivial tasks. In the context of in-context learning, the design of the prompt is crucial. Considering the current paper is already quite long, we plan to address these aspects in a future paper.

Reviewer comment (Novelty):
There seems to be a lack of thought about model structure and loss. It is just a combination of the strong pretrained LLM and the existing audio encoder, AST. Utilizing strong pretrained LLM with multimodal inputs has an alignment issue. Thus, while concatenating the audio feature and the text feature can introduce desired performance, there could be some advancements not just combining pretrained audio model and LLM. For example, BLIP-2 [1] solves misalignment between text and image using 3 losses: 1) Image-Text matching, 2) Image-Grounded Text generation, and 3) Image Text Contrastive Learning. The simple combination of the audio model and LLM does not seem to be novel.

Author response:
Dear Reviewer oFtB,

Thank you so much for taking the time to read our paper and providing very valuable comments and suggestions regarding the model architecture and comparing LTU with Pengi. Please see the following for our point-by-point response. Please also note that we have added substantially more materials in the revised paper (please check the updated PDF file).

---
**Question 1. Audio-Text Alignment**

>There seems to be a lack of thought about model structure and loss. It is just a combination of the strong pretrained LLM and the existing audio encoder, AST. Utilizing strong pretrained LLM with multimodal inputs has an alignment issue. Thus, while concatenating the audio feature and the text feature can introduce desired performance, there could be some advancements not just combining pretrained audio model and LLM. For example, BLIP-2 [1] solves misalignment between text and image using 3 losses: 1) Image-Text matching, 2) Image-Grounded Text generation, and 3) Image Text Contrastive Learning. The simple combination of the audio model and LLM does not seem to be novel.

We understand the reviewer’s concern about the architecture and are grateful for the highly constructive suggestion of adopting BLIP-2-like alignment training. Just as the reviewer points out, we connect the audio encoder with the large language model with only a linear layer and before LTU training, this audio encoder has not been trained for any audio-text alignment tasks. 

In our design, the AST's output serves as *soft prompts* for the large language model. According to [1,2], these (multimodal) *soft prompts* do not necessarily need to be in the text embedding space. In practice, some multi-modal large language models in the vision community such as PaLM-E, LLaVA, and Kosmos use a similar architecture and achieve good performance. One reason for not allowing us to directly use Q-Former in BLIP-2 is that the learnable queries are not guaranteed to be temporally aligned, which is important for audio tasks.

We sincerely appreciate the reviewer's suggestion to explore an additional training stage for audio-text alignment, and in response, we are proceeding with the following new experiments.

[1] Lester, Brian, et al. The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP 2021.

[2] Driess, Danny, et al. PaLM-E: An Embodied Multimodal Language Model. 2023.

In the new experiment, we add an additional audio-text alignment training stage before LTU training. We test two types of alignment modules: one is the linear layer used in the original LTU, and the other is a Transformer layer, which allows for more model capacity in the alignment. We train the alignment module on 1.6M audio-text pairs (comprising audio labels or captions) for 10 epochs, using a batch size of 1536 distributed over 4 GPUs, with 384 samples on each GPU. We start with an initial learning rate of 1e-3 and halve it after each epoch. During this stage, we keep the audio and LLaMA text encoders frozen and only train the alignment module. We use the following loss:

$$
 \mathcal{L} = \mathcal{L}_c + \lambda \cdot \mathcal{L}\_{mse}
$$

$$
\mathcal{L}_{mse} = \frac{1}{N} \sum\_{i=1}^N (e^a_i - e^t_i)^2
$$

$$
\mathcal{L}_\mathrm{c} = - \frac{1}{N} \sum\_{i=1}^N {\rm log}  \left[ \frac{ {\rm exp} (s\_{i,i}/\tau)}{\sum\_{k \neq i} {\rm exp} (s\_{i,k}/\tau) + {\rm exp} (s\_{i,i}/\tau)} \right]
$$

where $\lambda = 10$ is used to balance the scale between MSE loss and contrastive loss, $e^t$ and $e^a$ is the text and audio embeddings, respectively, $N = 384$ is the batch size, $s_{i,j}$ is the similarity score calculated as the dot product of normalized text embedding $e^t_i$ and audio embedding $e^a_j$, and $\tau = 0.05$ is the temperature. This loss aligns both the scale and direction of the audio and text embeddings.

Reviewer comment (Evaluation):
The concurrent works show higher performances. Compared to Pengi, the closed-ended audio task performances are lower. I understand that the proposed paper is focusing on the open-ended problem, but it would be better to elaborate more in detail that the proposed paper is competitive compared to the concurrent works.

Author response:
**Question 2. Compare with Pengi**

------
>The concurrent works show higher performances. Compared to Pengi, the closed-ended audio task performances are lower. I understand that the proposed paper is focusing on the open-ended problem, but it would be better to elaborate more in detail that the proposed paper is competitive compared to the concurrent works.

We thank the reviewer for the comment and agree that comparing LTU with concurrent work would better place LTU in the context. The reviewer is correct that Pengi is better on some closed-ended benchmarks (e.g., ESC-50), but as we will show below, on average, LTU is **slightly better** than Pengi on closed-ended tasks and significantly better for open-ended tasks.

We compare LTU with Pengi in three aspects.

------
### 1. Technical Similarity and Difference

**Similarity**:
- Architecture-wise, both Pengi and LTU connects an audio encoder with an autoregressive language model. Both models can generate text from the given audio and text prompt.
- Performance-wise, both Pengi and LTU can do zero-shot predictions on unseen datasets. On closed-ended tasks, Pengi and LTU perform similarly. 

**Difference**
- **Motivation:** Pengi focuses on *transfer learning*, i.e., using a single model for 8 audio perception tasks, while LTU focuses on *unifying audio perception with understanding*. Unlike LTU, Pengi is not designed for audio understanding and answering *free-form open-ended* questions from users.
- **Language Model Scale:** Pengi employs GPT2-Base (124M parameters) as its language model, whereas LTU uses LLaMA (7B parameters), which is over 50x larger than GPT2-Base. Additionally, LLaMA is trained with significantly more data than GPT2. Scaling up the language model substantially enhances LTU's understanding capabilities.
- **Training Data:** Pengi is trained on multiple audio datasets, and directly uses the original text-form labels as answers for AQA. This approach is similar to the closed-ended part of our OpenAQA dataset. However, OpenAQA also includes another 3.7M open-ended AQAs generated with the proposed *audio instruction generation* method (not generated from a template). The open-ended portion of our training data is pivotal in enabling LTU to answer *any free-form open-ended* questions.
- **Training Curriculum:** To better unifying audio perception and generation, LTU adopts a unique perception-to-understanding training curriculum.
- **Open-Ended Evaluation:** LTU undergoes evaluation for both closed-ended and open-ended tasks, incorporating both subjective and objective evaluations. Conversely, Pengi is evaluated solely with closed-ended tasks. Please note that the definition of an open-ended task differs between the Pengi paper and this paper; Pengi considers audio captioning as an open-ended task, while we categorize them as closed-ended tasks.

---
### 2.Closed-Ended Task Performance Comparison

|               |   ESC50  |  FSD50K  |   DCASE  |    TUT   | Beijing Opera | Vocal Sound |  Average |
|---------------|:--------:|:--------:|:--------:|:--------:|:-------------:|:-----------:|:--------:|
| Pengi         | **91.9** | **46.7** |   33.8   | **35.2** |      62.3     |     60.3    |   55.1   |
| LTU (Default) |   83.1   |   46.3   |   45.9   |   32.5   |    **69.9**   |     55.6    |   55.6   |
| LTU (Full FT) |   85.9   |   45.7   | **47.2** |   33.0   |      59.8     |   **69.0**  | **56.7** |

We summarize the performance of LTU and Pengi on closed-ended audio classification tasks in the above Table (using the result reported by the Pengi paper). Pengi and LTU each win on three out of the six tasks. In general, LTU with the default LoRA setting performs similarly to Pengi (average score 55.6 vs 55.1), while the fully finetuned version of LTU performs slightly better, achieving an average score of 56.7.

Reviewer comment (Writing):
I don't understand 64(time) X 16(frequency) in the 5th line of Audio Encoder part in Section 2: LTU Model Architecture. Is it typo (I think it should be 8)? Or what is the meaning of frequency 16?

Author response:
**Question 3. Dimension of the audio input**

------
>I don't understand 64(time) X 16(frequency) in the 5th line of Audio Encoder part in Section 2: LTU Model Architecture. Is it typo(I think it should be 8)? Or what is the meaning of frequency 16?

The reviewer is totally correct on this. It is a typo and should be 8, we have fixed this in the revised paper.

Reviewer comment (Experiments):
I want to know how the dataset is formulated (Q and A) that is generated from GPT.

Author response:
>I want to know how the dataset is formulated (Q and A) that is generated from GPT.

We thank the reviewer for the comment. In Table 2 of the paper, we show the prompt for GPT-3.5-Turbo, but due to the space limitation, we omit the formatting instruction ``Format each QA pair in a single line as a JSON dictionary (key "q'' for question, and "a'' for answer, wrapped with { and }). Do not include any other explanation.``, which may cause confusion, we apologize for this.

The full prompt is 

>Based on the following audio clip, generate 10 different types of complex open-ended questions that require step-by-step thinking, and corresponding step-by-step answers.
The following information is provided: the sound events appear in the audio clip, together with its acoustic features, and corresponding onset and offset time stamps. A description of the content of the audio clip is also provided. 
Questions should be about the audio, e.g., which sound event is recognized and why (e.g., based on its acoustic feature), what can be inferred based on the combination of sound events; the temporal relationship between the sound events and what can be inferred from that; the potential scenario that such an audio clip could happen, if the audio clip is special (e.g., urgent, funny, interesting, abnormal, unique, etc) and why, what mood or atmosphere this audio clip conveys, etc. **Format each QA pair in a single line as a JSON dictionary (key "q'' for question, and "a'' for answer, wrapped with { and }). Do not include any other explanation.**

The output of GPT is a JSON dictionary looks like 
``
[{
  "q": "How would you describe the tone of the sound of the 
        accelerating engine?", 
  "a": "The tone of the sound of the accelerating engine is 
        high-pitched, short and intense."
},
...
{
  "q": "What is the acoustic feature that distinguishes the 
        sound of the ambulance siren from the generic impact sounds?", 
  "a": "The acoustic feature that distinguishes the sound of the 
        ambulance siren from the sound of generic impact sounds is that 
        the former is high-pitched and wailing, while the latter is 
        loud and sharp."
}]
``
We then extract the questions and answers from the JSON dictionary. In the revised paper, we added a new section **Appendix M Full GPT-3.5-Turbo Prompt and Sample Outputs** to show the full prompt and sample JSON output. We also added a note in Table 2 to indicate the prompt is an abbreviated version due to the space limitation. We hope these changes will make the description of our data generation process more clear.

Reviewer comment (Experiments):
It might be valuable to evaluate PENGI on open ended tasks. My hunch is LTU would be far better than Pengi in open ended tasks although Pengi might be better on Close-ended tasks. This would probably highlight LTU significantly as the two approaches are contemporary in many ways and are very similar in the overarching goal. However the approaches taken are different. I realize that PENGI's checkpoint was probably not available when this paper was submitted but it is now. I would encourage the authors to do this comparison to show how LTU could be a more generic model which can be super useful for users to interact with an audio language model through natural text.

Author response:
We thank the reviewer for the comment and for pointing us to the pretrained Pengi checkpoint. We were able to download the checkpoint and run experiments based on it. We agree that comparing LTU with concurrent work would better place LTU in the context. The reviewer is correct that Pengi is better on some closed-ended benchmarks (e.g., ESC-50), but as we will show below, on average, LTU is **slightly better** than Pengi on closed-ended tasks and significantly better for open-ended tasks.

We compare LTU with Pengi in three aspects.

------
### 1. Technical Similarity and Difference

**Similarity**:
- Architecture-wise, both Pengi and LTU connects an audio encoder with an autoregressive language model. Both models can generate text from the given audio and text prompt.
- Performance-wise, both Pengi and LTU can do zero-shot predictions on unseen datasets. On closed-ended tasks, Pengi and LTU perform similarly. 

**Difference**
- **Motivation:** Pengi focuses on *transfer learning*, i.e., using a single model for 8 audio perception tasks, while LTU focuses on *unifying audio perception with understanding*. Unlike LTU, Pengi is not designed for audio understanding and answering *free-form open-ended* questions from users.
- **Language Model Scale:** Pengi employs GPT2-Base (124M parameters) as its language model, whereas LTU uses LLaMA (7B parameters), which is over 50x larger than GPT2-Base. Additionally, LLaMA is trained with significantly more data than GPT2. Scaling up the language model substantially enhances LTU's understanding capabilities.
- **Training Data:** Pengi is trained on multiple audio datasets, and directly uses the original text-form labels as answers for AQA. This approach is similar to the closed-ended part of our OpenAQA dataset. However, OpenAQA also includes another 3.7M open-ended AQAs generated with the proposed *audio instruction generation* method (not generated from a template). The open-ended portion of our training data is pivotal in enabling LTU to answer *any free-form open-ended* questions.
- **Training Curriculum:** To better unifying audio perception and generation, LTU adopts a unique perception-to-understanding training curriculum.
- **Open-Ended Evaluation:** LTU undergoes evaluation for both closed-ended and open-ended tasks, incorporating both subjective and objective evaluations. Conversely, Pengi is evaluated solely with closed-ended tasks. Please note that the definition of an open-ended task differs between the Pengi paper and this paper; Pengi considers audio captioning as an open-ended task, while we categorize them as closed-ended tasks.

---
### 2. Closed-Ended Task Performance Comparison

|               |   ESC50  |  FSD50K  |   DCASE  |    TUT   | Beijing Opera | Vocal Sound |  Average |
|---------------|:--------:|:--------:|:--------:|:--------:|:-------------:|:-----------:|:--------:|
| Pengi         | **91.9** | **46.7** |   33.8   | **35.2** |      62.3     |     60.3    |   55.1   |
| LTU (Default) |   83.1   |   46.3   |   45.9   |   32.5   |    **69.9**   |     55.6    |   55.6   |
| LTU (Full FT) |   85.9   |   45.7   | **47.2** |   33.0   |      59.8     |   **69.0**  | **56.7** |

We summarize the performance of LTU and Pengi on closed-ended audio classification tasks in the above Table (using the result reported by the Pengi paper). Pengi and LTU each win on three out of the six tasks. In general, LTU with the default LoRA setting performs similarly to Pengi (average score 55.6 vs 55.1), while the fully finetuned version of LTU performs slightly better, achieving an average score of 56.7.

---
### 3. Open-Ended Task Performance Comparison

|               | GPT-3.5 Generated Questions | GPT-4 Generated Questions | Human Generated Questions |
|---------------|:---------------------------:|:-------------------------:|:-------------------------:|
| Pengi         |             7.4             |            3.1            |            28.3           |
| LTU (Default) |           **95.9**          |          **96.9**         |          **70.1**         |

We use GPT-4 (with the prompt ``Below is a pair of question and response. Identify if the response answers the question. Return yes or no.``) to evaluate the instruction following rate of LTU and Pengi in response to open-ended questions and present the results in the above Table. With a stronger language model and training with the OpenAQA dataset, LTU dramatically outperforms Pengi. This also answers the reviewer's first question regarding the value of OpenAQA.

However, please note that Pengi is neither designed nor trained for open-ended question answering and understanding, thus this task is not entirely fair to Pengi. We conduct this experiment just to highlight the differences between LTU and Pengi.

---
### Summary

To summarize, while LTU and Pengi share some similarities in architecture, they have different motivations and designs. They perform similarly in closed-ended audio classification tasks, but LTU is more capable of answering open-ended questions and demonstrating audio understanding.

We have added this discussion to **Section G Compare with Pengi** in the revised paper. We thank the reviewer again for this comment and hope the discussion will help the reader understand the difference between LTU and concurrent works.

---
**Other Improvements in the Revision:** In addition to the above-mentioned changes, we have incorporated new experiments in response to the comments from other reviewers. These are detailed in Appendix E, which discusses scaling up the LLM, Appendix F, which discusses audio-text pretraining, and Appendix H, which focuses on evaluating LTU in temporal analysis tasks. We would greatly appreciate it if the reviewer could take the time to review these new sections.

---
We hope we addressed all your concerns, and if not we are more than willing and happy to engage in a discussion with the reviewer to answer further questions and/or provide more clarifications as needed.

Reviewer comment (Evaluation):
The authors may add subjective evaluations to the ablation experiments to better demonstrate that the LoRA fine-tuning strategy mitigates catastrophic forgetting issues.

Author response:
**Question 2 - Evaluating the catastrophic forgetting difference?**
>The authors may add subjective evaluations to the ablation experiments to better demonstrate that the LoRA fine-tuning strategy mitigates catastrophic forgetting issues.

We thank the reviewer for the question. 

In Table 5 of the paper, we show that full finetuning would cause a dramatic instruction following rate drop on pure language tasks.

| Training Setting  | Average Audio Classification Performance | Pure Language Task Instruction Following Rate |
|-------------------|:----------------------------------------:|:---------------------------------------------:|
| LoRA (lr=1e-4)    |                   50.3                   |                      87.6                     |
| Full FT (lr=1e-4) |                   50.4                   |                      35.0                     |
| Full FT (lr=2e-5) |                   51.3                   |                      53.5                     |

The instruction following rate is evaluated by GPT-4 with the prompt ``Below is a pair of question and response. Identify if the response answers the question. Return yes or no.`` While GPT-4 may not be a perfect tool for evaluation, a notable difference of approximately 35% clearly indicates a significant drop in performance. The following example illustrates a case where LTU-Full-FT fails.

``Question: Generate a shopping list for a fast food meal. `` (no audio is provided)

``LTU-LoRA: Here is the shopping list for a fast food meal:\n\n* A burger\n* French fries\n* A drink (such as soda or water)\n* Condiments (such as ketchup, mustard, and mayo)\n* Napkins\n* Straws (if desired)\n\nThis list can be adjusted based on personal preferences and dietary restrictions.``

``LTU-Full-FT: The audio clip does not provide any information about the specific items on the shopping list.``

For open-ended audio tasks, we also observe that the fully-finetuned LTU tends to forget its original knowledge, resulting in shorter and less diverse responses. For audio questions generated by GPT-4, the fully-finetuned LTU's average answer word length is 26 and its vocabulary size is 8,456, whereas the LoRA version shows an average word length of 35 and a vocabulary size of 9,621.

Reviewer comment (Evaluation):
Why is the results of LTU on open-ended problems preferred compared to those of GPT-4 in Section 5.2.2? In terms of the data generation method of open-ended Q-A pairs, LTU's inferential knowledge appears to be transferred from the data generation model, i.e. GPT 3.5 Turbo model. These experimental results seem to indicate that the performance of the student model is superior to that of the teacher model (family).

Author response:
**Question 3 - Why LTU outperforms GPT-4?**

>Why is the results of LTU on open-ended problems preferred compared to those of GPT-4 in Section 5.2.2? In terms of the data generation method of open-ended Q-A pairs, LTU's inferential knowledge appears to be transferred from the data generation model, i.e. GPT 3.5 Turbo model. These experimental results seem to indicate that the performance of the student model is superior to that of the teacher model (family).

We thank the reviewer for the question. The reason why LTU can outperform GPT-4 is because the input to LTU is **raw audio** while the input to GPT-4 is **text-form audio event label, timestamps, and caption**. Raw audio inherently contains richer information compared to its text-form counterpart. The limitation of GPT-4 not being able to process audio directly (which is a key motivation for this work) restricts its use to text-based inputs only.

We summarize the model input and language model for LTU and GPT-4 in the following tables:

|                    |                                                    |                             |
|:------------------:|:--------------------------------------------------:|:---------------------------:|
| **LTU Evaluation** |                **Input: Raw Audio**                |      **LLM: LLaMA-7B**      |
| Advantage          |             Contain richer information             | Trained specially for audio |
| Disadvantage       | No ground truth is provided, LTU may make mistake. |      Smaller and weaker     |

|                    |                                                    |                             |
|:------------------:|:--------------------------------------------------:|:---------------------------:|
| **GPT-4 Evaluation**    | **Input: Text-form Ground Truth Audio Lable, Time Stamps, and Caption** |          **LLM: GPT-4**          |
| Advantage    |               Ground truth information is provided.              |      Most advanced LLM      |
| Disadvantage |             Less information compared with raw audio             | Not trained for audio tasks |

This comparison serves as an important purpose: one trivial solution for audio large language model is first to convert audio to text form, e.g., audio caption, and then feed it to a strong large language model. Our aim is to demonstrate that this method cannot surpass LTU's performance, primarily because audio captions inherently contain limited information. In the experiment, we allowed the baseline model to have an advantage by using the ground truth label and employing one of the strongest large language models, GPT-4. Despite this, the model still could not outperform LTU, thereby supporting our claim.

Reviewer comment (Experiments):
Did the authors investigate whether errors (or hallucinations) in open-ended question generation model (i.e. GPT 3.5 turbo) could impact the performance of the LTU? In other words, is the correctness of the Q&A pairs generated by GPT from audio metadata reliable enough?

Author response:
**Question 4 - Errors caused by GPT generated data**

>Did the authors investigate whether errors (or hallucinations) in open-ended question generation model (i.e. GPT 3.5 turbo) could impact the performance of the LTU? In other words, is the correctness of the Q&A pairs generated by GPT from audio metadata reliable enough?

We appreciate the reviewer's insightful question, which is about the fundamental aspect of our core method audio instruction generation (AIG).

As discussed in Section 3.2, AIG produces a broad spectrum of audio QAs, ranging from low-level tasks (like identifying audio events) to high-level understanding tasks (such as determining the atmosphere conveyed by the audio). And the errors can be broken down for low-level and high-level QAs.

- For low-level QAs, we observe that GPT-3.5-Turbo almost never make an error, e.g., when the provided label is ``ambulance``, GPT-3.5-Turbo will not generate a QA with an answer like ``it is a police car''.

- For high-level QAs, GPT-3.5-Turbo may make error/hallucination, e.g., when the provided label is ``ambulance``, GPT-3.5-Turbo typically associate it with an urgent situation while this may not be true for all audio clips containing ambulance. 

Therefore, we believe that for low-level questions, GPT-3.5-Turbo generated data does not introduce *additional* factual errors, as these QAs are based on ground truth labels. The factual errors in LTU's responses primarily stem from imperfections in its audio encoder. In contrast, the training data's hallucinations do impact LTU's performance on high-level questions.

Given the challenge of automatically measuring errors in GPT-generated data due to its volume, we opted for a direct human evaluation of the LTU model. Table 7 in our paper presents the results of this evaluation, where human raters found 79.3% of LTU's answers to GPT-4 generated open-ended questions and 83.7% to human-generated open-ended questions completely correct. Overall, LTU's performance on open-ended questions is satisfactory.