Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2022)

Abstract: Machines that can represent and describe environmental soundscapes have practical potential, e.g., for audio tagging and captioning. Prevailing learning paradigms of audio-text connections have been relying on parallel audio-text data, which is, however, scarcely available on the web. We propose VIP-ANT that induces Audio-Text alignment without using any parallel audio-text data. Our key idea is to share the image modality between bi-modal image-text representations and bi-modal image-audio representations; the image modality functions as a pivot and connects audio and text in a tri-modal embedding space implicitly. In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by 2.2% R@1. We further investigate cases of minimal audio-text supervision, finding that, e.g., just a few hundred supervised audio-text pairs increase the zero-shot audio classification accuracy by 8% on US8K. However, to match human parity on some zero-shot tasks, our empirical scaling experiments suggest that we would need about 2^{21} \approx 2\text{M} supervised audio-caption pairs. Our work opens up new avenues for learning audio-text connections with little to no parallel audio-text data.

## introduction environmental sound provides rich perspectives on the physical world. for example, if we hear: joyful laughing, a playful scream, and a splash; we not only can visualize literal objects / actions that might have given rise to the audio scene, but also, we can reason about plausible higher-level facets, e.g., a child speeding down a water slide at a water park, splashing through the water (see figure 1 ). ## model we first formalize tri-modal learning by assuming available co-occurrence data for every pair of modalities ( § 3.1). then we present bi-bi-modal pre-training as an alternative when there is no paired audio-text data, and implement vip-ant via bi-bi-modal pre-training ( § 3.2). finally, we describe model variants for cases of varying at ## audio-text experiments we use two types of tasks to evaluate the quality of the audio-text alignments learned by our model: at retrieval and zero-shot audio classification. at retrieval. we conduct audio-text retrieval on audiocaps and clotho for in-domain evaluation and out-of-domain evaluation, respectively: we study the out-of-domain generalizability of our models by applying them to clotho directly, without further fine-tuning on it. 7 zero-shot audio classification. we consider the following three widely used datasets for audio classification. for each audio clip a, we first compute the cosine similarity between it and every possible textual label in the tri-modal representation space. then we predict the label t with the highest similarity: ## conclusion we have presented vip-ant for unsupervised audio-text alignment induction. based on the pivoting idea, our model learns image-text alignment and image-audio alignment explicitly and separately via bi-modal contrastive pre-training. the image modality is shared between the two and thus pivots audio and text in the tri-modal embedding space implicitly, without using any paired audio-text data. we empirically find that our model achieves strong performance on zero-shot audiotext tasks. we further strengthen the audio-text alignment by using varying kinds of audio-text supervision. experimental results show that even un-aligned audio-caption pairs can help.