Paper: Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation (ICLR-2024)

Reviewer comment (Experiments):
Apart from the performance, it would be good to also show the acquired network architecture, and run-time efficiency for cross-modal inference.

Author response:
The focus of our work is to learn an effective joint embedding space for audio and text retrieval. We acknowledge the importance of run-time efficiency and leave that to our future work.

Reviewer comment (Writing):
The reference of "noise" is not clear and potentially confusing, even with clear references. When talking about the noise, it can be many things. Especially for speech people who are very likely to refer to this paper, it seems like the definition of "noise" is different from real-world interruption – it is totally fine, but please spend some text on clarifying it in the introduction and methodology (aka Section 3.3).

Author response:
We totally agree with your suggestion and have added a sentence in the third paragraph of the introduction to articulate noisy data in the audio-text matching task.

Reviewer comment (Writing):
Although I did not find significant grammatical flaws, please do a thorough check on the language usage. For example, at the beginning of Section 4, "Cross-modal matching is an essential task....." sounds a bit weird.

Author response:
We have proofread the manuscript and corrected all language usage.

Reviewer comment (Reproducibility):
Do you think if it is possible to open-source the models?

Author response:
We will definitely publish our source code and pretrained models to the community.

Reviewer comment (Experiments):
Do you think your model can be adopted to other sound-related tasks, with rather little amount of data for domain adaptation? Or maybe collection of experts is needed?

Author response:
Our model could be used for other downstream tasks by finetuning the pretrained model, for example, text-to-audio generation. However, we need an appropriate decoder/adapter model for decoding embedding space to data space.

Reviewer comment (Writing):
Would you classify your approach as essentially an "unsupervised" or "supervised" contrastive loss?

Author response:
We would like to classify our approach as a supervised method since the m-LTM needs to use aligned audio-text pairs for training.

Reviewer comment (Novelty):
Do you see any possibility on developing a "parameter-free" variant of your framework?

Author response:
We assume that you are asking about replacing the Mahalanobis distance with a parameter-free one, like kernel distance. We thought about that but left it for future work.

Reviewer comment (Experiments):
Retrieval applications have an expectation of scaling. Ideally a single query would be used to retrieve one or more corresponding examples from an extremely large source. However, in this paper the datasets (particularly the test sets) have a fairly small source to retrieve from (a few thousand examples typically). It would strengthen the work substantially to demonstrate the capabilities of the algorithm to scale to instances where there are orders of magnitude more examples to retrieve from than queries.

Author response:
**A1:** We agree that scalability is important for retrieval applications. An accurate similarity function for ranking is also crucial. In this work, we focus on measuring the similarity of examples in a rich and expressive joint embedding space designed for audio-text retrieval problems. We leave the investigation of the scalability issue to future work.

Reviewer comment (Presentation):
There is a claim that the correspondence in Figure 1 is obviously better in figure 1b than figure 1a. I think this is not as obvious as claimed. I don’t think this image adds particular value to the understanding of the work.

Author response:
**A2:**  Figure. 1 demonstrates the joint embedding space for zero-shot sound event detection setting. We would like to show that the joint embedding space learned from our framework is more expressive than contrastive learning. All sound event labels are described in plain text by the template “this is a sound of {sound event}.” Each text embedding in Figure. 1 is equivalent to a sound event label. The sound event detection is now converted to an audio-text retrieval task, therefore, a compact and clustering embedding space is beneficial to the performance. Figure. 1b shows that the m-LTM framework encourages the compact clustering of audio and event texts in the embedding space compared with contrastive loss shown in Figure. 1a. The claim is supported by the performance of zero-shot sound event detection experiment shown in Table. 3.

Reviewer comment (Experiments):
Partial Optimal Transport appears to be performed by noising some percentage of the data prior to learning. When this noise is applied with a batch size of 2, there is a substantial likelihood that both batch elements will contain incorrect (corrupted) examples. Is there any risk in this from an algorithmic perspective?

Author response:
**A3:** We agree that there is a risk in this when the batch size is very small. The underlying assumption of our algorithm is that there is at least a proportion of matched examples in a batch. If a batch is too small, either there are no matched examples, or all are matched. Hence, small batches are not the focus of this algorithm.  The intuition of utilizing POT for noise correspondence data is to discard as many noisy pairs as possible. In the case of a small batch size, there is a risk of a whole minibatch of training data being noisy, POT regularization fails to handle that circumstance since it always imposes a percentage of matching(the mass parameter for POT) between two sources. The risk of a corrupted minibatch is critical for all existing methods, including my proposed approach. However, with a large enough minibatch, POT regularization is able to mitigate the harm of misaligned training data.

Reviewer comment (Reproducibility):
How does this approach deal with instances where no match is available?

Author response:
**A4:** The POT approach deals with misaligned instances within a minibatch by a parameter, which forces only to match a percentage of audio-text pairs, the total transportation mass between two sources $0 \leq s \leq 1$. The total transportation mass of POT acts as a regularizer to discard the less certain matches, therefore, it is able to mitigate the harmfulness of mismatched training data.

Reviewer comment (Experiments):
The authors describe one of the positive attributes of this work as identifying a joint embedding space for both speech and text through which to perform retrieval. Has this embedding space been used for other downstream tasks? This would strengthen the argument that this is a good joint space rather than solely useful for retrieval.

Author response:
**A5:** We conducted a zero-shot setting for a downstream task, sound event detection, on the ESC-50 test set in the Table. 3 to illustrate the expressiveness of the learned join embedding space.

Reviewer comment (Experiments):
How effectively does the learned representation work across corpus? For example, when training on Audiocaps, how effective is retrieval of Clotho test sets and vice versa? This would be a more effective measure of the robustness of the algorithm than the intra-corpus analyses.

Author response:
**A6:** To answer your question, we compare our method with the baselines in the cross-corpus setting. As shown in the table below, the m-LTM is more robust across audio-text retrieval datasets than triplet and contrastive loss. Regarding training on the AudioCaps and testing on the Clotho dataset, our proposed method acquires the highest performance, 15.01%, in terms of R@1 for the text-to-audio retrieval task, compared with 10.41% and 12.31% for triplet and contrastive loss respectively. The same observation is seen for the setting training on Clotho and then testing on AudioCaps dataset.

|     Train-Test    |  Method |       |  T->A |       |       |  A->T |       |
|:-----------------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|
|                   |         |  R@1  |  R@5  |  R@10 |  R@1  |  R@5  |  R@10 |
|                   | Triplet | 10.41 | 27.38 | 38.04 | 11.39 | 28.42 | 39.80 |
| AudioCaps->Clotho |    CL   | 12.31 | 30.93 | 43.44 | 13.87 | 33.30 | 43.54 |
|                   |  m-LTM  | 15.01 | 35.42 | 47.71 | 19.42 | 37.03 | 48.61 |
|                   | Triplet | 14.33 |   40  | 54.42 | 17.34 | 43.05 | 55.69 |
| Clotho->AudioCaps |    CL   | 14.64 | 38.64 | 53.58 | 17.76 | 42.11 | 55.59 |
|                   |  m-LTM  | 17.38 | 43.78 | 58.87 | 22.04 | 47.85 | 61.65 |

Reviewer comment (Evaluation):
The delta_gap described in table 4 is the difference between the means of the embeddings. First, is this gap an absolute value? It’s remarkable that mean(f(x_i)) is always greater than mean(g(y_i)). Second, is the corpus mean of text and audio embeddings a reasonable measure of performance? It seems like this measure could be easily gamed by normalizing the embedding spaces to have a 0-mean, but this wouldn’t add anything to the transferability of the embedding. Also, distance in the embedding space isn’t well defined. It’s not clear that a unit of distance in the triplet loss space can reasonably be compared to a unit in the m-LTM space.

Author response:
**A7:** The gap value is the length of the discrepancy vector $\vec{\Delta_{gap}} = \frac{1}{n}\sum_{i=1}^n f(x_i) - \frac{1}{n}\sum_{i=1}^n g(y_i)$ which is subtracting vector between the mean of embedding vector of two modalities, we described it in the second paragraph of section 5.1. Therefore, the gap is always positive. The discrepancy between the two modalities proved that it is an effective metric for studying the transferability of a jointed embedding space[1]. We use Euclidean distance on the embedding space to measure the modality metric. To achieve a fair comparison, all embedding is normalized to unit vectors and then used to compute the modality gap.

[1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning.

Reviewer comment (Novelty):
The m-LTM method presented is somewhat lower in novelty as a variation on LTM that investigates the use of minibatch training and deep networks, but 'revisiting' is explicitly called out in the title, and this seems a worthy exploration.

Author response:
**A2:** We agree that adapting the learning-to-match framework for deep learning is a valuable exploration. However, we acknowledge that the title and abstract can not be modified during the discussion stage. We will definitely change our title regarding your comment when submitting the camera-ready version.

Reviewer comment (Writing):
There are a few grammatical errors throughout the paper, but the paper is in general well structured, and adequately well written. A glaring exception is the abstract, which is in really poor shape. Authors, please resolve this.

Author response:
**A3:** We have corrected all the grammatical errors and will definitely restructure the abstract when submitting the camera-ready version. Here is our new abstract:

The Learning-to-match(LTM) framework proves to be an effective inverse optimal transport approach for learning the underlying ground metric between two sources of data, facilitating subsequent matching. However, the conventional LTM framework faces scalability challenges, necessitating the use of the entire dataset each time updating parameters of ground metric. In adapting LTM to the deep learning context, we introduce the mini-batch Learning-to-match (m-LTM) framework for audio-text retrieval problems. This framework leverages mini-batch subsampling and Mahalanobis-enhanced family of ground metrics. Moreover, to cope with misaligned training data in practice, we propose a variant using partial optimal transport to mitigate the harm of misaligned data pairs in training data. We conduct extensive experiments on audio-text matching problems using three datasets: AudioCaps, Clotho, and ESC-50. Results demonstrate that our proposed method is capable of learning rich and expressive joint embedding space, which achieves SOTA performance. Beyond this, the proposed m-LTM framework is able to close the modality gap across audio and text embedding, which surpasses both triplet and contrastive loss in the zero-shot sound event detection task on the ESC-50 dataset. Notably, our strategy of employing partial optimal transport with m-LTM demonstrates greater noise tolerance than contrastive loss, especially under varying noise ratios in training data on the AudioCaps dataset.

Reviewer comment (Experiments):
In table 6, the best results for the ground metric hyperparameter $\epsilon$ are at 0.05, but no values less than it are tested, while the results at 0.5 are very poor, suggesting that results for $\epsilon<0.05$ should be instead included.

Author response:
**A4:** $\epsilon$ is a critical hyperparameter for our framework, $\epsilon$ controls the effect of entropic regularization to compute the optimal plan. As mentioned in the ablation study section, if $\epsilon$ is too high, the optimal plan will be a uniform distribution. In the case of $\lim{\epsilon \to 0}$, the optimal plan will converge to the exact optimal transportation plan. We provide additional experiment on $\epsilon$ value lower than 0.05:

| Epsilon |    |   T->A    |       |    |  A->T     |       |
|:-------:|:-----------:|:-----:|:-----:|:-----------:|:-----:|:-----:|
|         |     R@1     |  R@5  |  R@10 |     R@1     |  R@5  |  R@10 |
|   0.01  |     0.10    |  0.52 |  1.04 |     0.10    |  0.31 |  0.41 |
|   0.02  |     0.17    |  0.77 |  2.06 |     0.45    |  1.31 |  3.78 |
|   0.03  |    38.28    | 74.85 | 85.13 |    49.88    | 79.77 | 89.86 |
|   0.04  |    37.22    | 73.58 | 85.37 |    48.38    | 80.04 | 90.17 |
|   0.05  |    39.10    | 74.06 | 85.78 |    49.94    | 80.77 | 90.49 |

As shown in the above table, if the value of $\epsilon$ is lower than 0.02, the performance of audio-text matching plunges from 38.28 to 0.17 and 49.88 to 0.45 in terms of R@1 metric for text-to-audio and audio-to-text retrieval respectively. The performance of the model at $\epsilon$=[0.3, 0.4] are both lower than the performance of the model at $\epsilon$=0.05. Overall, the above results show that  $\epsilon$=0.05 is indeed the best hyperparameter in our experiments.