Paper: CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models (ICLR-2024)

Reviewer comment (Presentation):
It is a bit difficult to get an entire picture of number of different datasets curated and how each of them are curated, a lot of the information are buried in the appendix. Consider in the main paper including a table describing different datasets curated, how many of them for each subset used for training, and evaluation, their source, and methods (leverage ChatGPT, replace the sounding object, swap the orders, etc.), and then refer to the table in the narratives.

Author response:
**Ans:** Thank You for your suggestion. Due to space limitations, despite our best efforts, we could not include such a table in our revised version due to space constraints. However, we have made the following changes to make it clearer:
- We have added Table 7 in the Appendix of our paper. This Table describes all the data used in our experiments, including their source and the stage of training in which they were used. We also show the Table in the Questions section.
- We have made several writing changes in all sections to make it clearer which data is being used in which stage of training. The changes can be seen in the revised version of our paper.

Reviewer comment (Presentation):
The data curation is one of the main contribution for this work, however currently a lot of details are left in the appendix. Consider integrating more information such as the source datasets, and some manipulated examples to the main narratives, this can help the reader to understand this work early on.

Author response:
**Ans:** Thank You for your suggestion. We have made the following changes to incorporate more details about data curation both in the main paper and appendix:
1. We expanded the details about the annotation in Section 2 and included more details about the exact annotation process.
2. In Appendix B3, we provide more details about annotator demographics.
3. In Appendix B3, we list some rules adhered to during the annotation process.

We would also like to point out that both our proposed contrastive learning approaches are novel and also serve as the main contributions of the paper. Compa-CLAP serves as the first step toward solving compositional reasoning in audio-language models. Thus, to accommodate everything within the 9-page limit, we made a division by allocating 3 pages to the dataset description and 4 to Compa-CLAP.

Reviewer comment (Presentation):
A minor suggestion: The figures and tables are far from the actual text in the appendix, consider reorganize them to make it easier to read.

Author response:
**Ans:** Thank You for your suggestion. We have reorganized the Figures and Tables in the appendix of the revised version of our paper to make it easier to read.

Reviewer comment (Presentation):
A minor comment: A lot of the texts in the tables and the figures are very small and difficult to read.

Author response:
**Ans:** Thank You for your suggestion. We have revised the text in our figures in the revised version of our paper to make them clearer.

Reviewer comment (Experiments):
Regarding the data curated for training, there are CompA-661k, and AudioSet-compA, another 110k audio-text pairs for complex compositional audios? How are these different subsets used in different training scenario? It might worth adding a table to show clearly what combination of datasets, training strategies, whether there is hard negative and modular contrastive utilized, and link them directly to your abbreviation in the result table (CLAP, CompA-CLAP, CLAP-CompA-661k, etc.) Currently it is not easy to associate from the narratives to exactly how each variation in the result tables comes from.

Author response:
We apologize for the confusion. Here is a table to make things clearer. We have also added this table to our Appendix as Table 7.

| Dataset                     | Source                                  | Experiment                                                                       |
|--|---|---|
| Pre-training                |                                         |                                                                                  |
| CompA-661K                  | Multiple (refer to Table 6 in Appendix) | Vanilla Contrastive Pre-training (Section 3.2)                                   |
| CompA-AudioSet              | AudioSet Strong [1]                     | Contrastive Pre-training with compositionally aware hard negatives (Section 3.3) |
| Modular Contrastive Dataset | AudioSet Strong [1]  (but synthetic)                   | Modular Contrastive Learning (Section 3.4)                                       |
| Evaluation                  |                                         |                                                                                  |
| CompA-order                 | AudioSet Strong [1]                           | Evaluation Dataset                                                               |
| CompA-Attribute             | AudioSet Strong [1]                            | Evaluation Dataset                                                               |
| CompA-Attribute             | WavJourney [3]                          | Evaluation Dataset                                                               |

Table 1: List of various datasets used in our experiments, their sources, and the specific experiments they are used in.

Note: As also mentioned in the paper, no audio used to create our CompA benchmark was included in the training set at any stage. For modular contrastive learning, we only take individual sliced acoustic events from the AudioSet Strong (not the entire audio), to generate our own synthetic data.

Reviewer comment (Presentation):
In Table 1, what is the difference between CLAP (ours) and CLAP-CompA-661k (ours)?

Author response:
**Ans:** We apologize for the mistake and thank you for pointing this out. CLAP (ours) and CLAP-CompA-661k (ours) are the same models, and CLAP-CompA-661k (ours) in Table 1 (right) is a typo. Since we trained our version of CLAP on CompA-661k, we had initially named it CLAP-CompA-661k but later changed it to CLAP (ours).

Reviewer comment (Evaluation):
For the common mistakes mentioned in the discussion, is it possible to perform some error analysis? Or confusion matrices to show these more quantitatively?

Author response:
**Ans:** Thank You for the suggestion. We have added both quantitative and qualitative comparisons in Appendix C.4 in Tables 11, 12 and 13. In terms of qualitative analysis, we have highlighted specific acoustic compositions in the CompA-order and CompA-attribute benchmarks where we observed significant improvement in CompA-CLAP compared to the other baselines.

| CompA-order (Acoustic Scene)                | CompA-attribute (Acoustic Scene)                     |  |
|---|---|----|
| Frog croaking followed by music playing.    | A man speaks then a woman panics                     | |
| Man speaking before bell ringing.           | A baby laughs while a woman make sounds              | |
| A man speaking followed by gunshot          | A child sneezes and an adult laughs                  | |
| Tiger growling followed by people speaking. | Adult female is speaking and a young child is crying | |

Table 2: Some examples of acoustic compositions in the CompA-order and CompA-attribute benchmarks where we observed significant improvement in CompA-CLAP compared to the other baselines

Reviewer comment (Theory):
As previously stated, my major (and almost only) criticism comes from the fact that the concept of « compositionality » presented on this paper is rather largely focused on « temporal » compositionality. Although I understand that it is mandatory to make choices as this is still a very young research topic, I think the paper would gain in strength if a more clear definition of various types of compositionality would be presented.

Author response:
Thank you for your valuable suggestion. We have added a subsection in the Appendix discussing different types of compositionality observed in audio. The same can be found in Appendix A.

Reviewer comment (Experiments):
With regards to generating examples, although I understand that using prompts and GPT can perform a large array of tasks, your temporal task could be quite easily simulated (collating examples together, such as done with the Mixup augmentations). Could you quantify gains and differences with those more straightforward approaches?

Author response:
Thank You for the question. Below, we have compared the results with traditional text augmentation methods, such as switching nouns and verbs using the Spacy library. We find that GPT-generated negatives outperform traditional text augmentation techniques. We hypothesize that this is due to the lower-quality and incoherent negatives generated by traditional NLP methods. We compare negatives from various LLMs and traditional NLP methods in Table 8 in the Appendix. Additionally, we provide code in the Supplementary material.

|                       | CompA-Order |       |       | CompA-attribute |       |       |
|-----------------------|-------------|-------|-------|-----------------|-------|-------|
| Model                 | Text        | Audio | Group | Text            | Audio | Group |
| CompA-CLAP (GPT-4 - ours)     | 40.70       | 35.60 | 33.85 | 44.28           | 22.52 | 15.13 |
| CompA-CLAP (Traditional NLP) | 35.25       | 30.45 | 17.80 | 39.37           | 17.26 | 12.54 |

Table 1: Comparison of CompA-CLAP with negatives generated by GPT-4 and Traditional NLP methods.

Reviewer comment (Novelty):
The core technical contribution of submission boils down to - "selecting appropriate negatives for the task at hand", an established idea in contrastive learning, thus raising the question of novelty in terms of technical contribution.

Author response:
**Ans:** We agree that hard-negative-based contrastive learning is a well-established idea in contrastive learning. However, as also stated in our paper, we would like to argue that just adding hard negatives is not our primary technical contribution. Our primary contributions are as follows:

1. (Technical) Section 3.3: We modify the formulation of naive contrastive learning with hard negatives such that hard negative captions for a particular audio are ignored by other audios in the batch.
2. (Technical) Section 3.4: We propose an entirely new algorithm called “modular contrastive learning.” Modular Contrastive Learning has two primary innovations:
    - We propose a novel and simple methodology of data generation that overcomes the data scarcity issue of compositional audios: Template-based synthetic creation of audio-caption pairs.
    - Our proposed method promotes fine-grained composition learning. Each audio in the batch has multiple positives that describe compositional relationships between certain acoustic events in the batch. Each audio in the batch also has multiple negatives for each positive. An example of this can be seen in Figure 4 (Figure 6 in revised version).
    - We modify the contrastive learning formulation such that each audio in the batch ignores the positives and negatives for all other audio.
3. (Non-Technical) Section 2: We propose two benchmarks, CompA-order and CompA-attribute, to evaluate compositional reasoning in Audio-Language Models (ALMs). The expert-annotated benchmarks, which are also the first of their kind, are complex to curate and annotate and annotate and represent a significant advancement in an understudied field. By focusing on this niche, our benchmark paves the way for a more nuanced understanding and development of ALMs.

Reviewer comment (Presentation):
The structuring of manuscript could be better.

Author response:
**Ans.:** Thank you for bringing these concerns to our attention! In response, we've incorporated a few introductory lines at the beginning of each section to concisely outline the main theme and facilitate smoother transitions between sections, all while considering space constraints.

Reviewer comment (Presentation):
Transition from section 3.3 to 3.4 is not clear in the first reading.

Author response:
**Ans.:** Thank you for bringing these concerns to our attention! In response, we've incorporated a few introductory lines at the beginning of each section to concisely outline the main theme and facilitate smoother transitions between sections, all while considering space constraints.

Reviewer comment (Presentation):
Every section reads like a mini-paper with its own background, methodology. May be add a paragraph at the beginning of section to prep the reader of upcoming sections and their differences.

Author response:
**Ans.:** Thank you for bringing these concerns to our attention! In response, we've incorporated a few introductory lines at the beginning of each section to concisely outline the main theme and facilitate smoother transitions between sections, all while considering space constraints.

Reviewer comment (Experiments):
Can the author's clarify if the order of hard negative training and modular contrastive training matter. (section 4: 2nd paragraph).

Author response:
**Ans:** The core rationale behind performing hard negative training first and modular contrastive training next is a Coarse-to-Fine Contrastive Learning approach. Hard negative training with an entire caption as a positive or a negative in the batch is an easy task and teaches CLAP to distinguish between entirely different contexts and concepts. On the other hand, modular contrastive training, a bit more difficult task, makes CLAP focus on specific compositional relationships of the entire scene, thereby refining its ability to discern subtle distinctions and nuances within similar contexts.

Table 1 provides results when the order is changed. Empirical results prove that the order matters for training. 

|                            | CompA-Order |       |       | CompA-attribute |       |       |
|----------------------------|-------------|-------|-------|-----------------|-------|-------|
| Model                      | Text        | Audio | Group | Text            | Audio | Group |
| CompA-CLAP (ours)          | 40.70       | 35.60 | 33.85 | 44.28           | 22.52 | 15.13 |
| CompA-CLAP (order changed) | 38.35       | 32.75 | 24.65 | 41.52           | 19.15 | 12.17 |

Table 1: Comparison of CompA-CLAP results with the order of compositionally aware hard negatives and modular contrastive learning changed.