Do Audio-Language Models Understand Linguistic Variations? (2025)

Abstract: Open-vocabulary audio language models (ALMs), like Contrastive Language Audio Pretraining (CLAP), represent a promising new paradigm for audio-text retrieval using natural language queries. In this paper, for the first time, we perform controlled experiments on various benchmarks to show that existing ALMs struggle to generalize to linguistic variations in textual queries. To address this issue, we propose RobustCLAP, a novel and compute-efficient technique to learn audio-language representations agnostic to linguistic variations. Specifically, we reformulate the contrastive loss used in CLAP architectures by introducing a multi-view contrastive learning objective, where paraphrases are treated as different views of the same audio scene and use this for training. Our proposed approach improves the text-to-audio retrieval performance of CLAP by 0.8%-13% across benchmarks and enhances robustness to linguistic variation. We make our code publicly available

## introduction as user-generated audio content expands at an unprecedented pace, developing methods to index and search effectively across an ever-growing database becomes crucial. open-vocabulary audio language models (alms) such as clap @xcite have emerged as a promising solution to this problem, achieving state-of-the-art (sota) results in text-based audio retrieval @xcite . in a typical setting, a user would use a natural language query to describe an acoustic scene with various audio events and then use it to retrieve audio files that match the query. natural language offers a powerful and intuitive interface for indexing and searching through audio 1 https://github.com/ramaneswaran/linguistic_ robust_clap * these authors contributed equally to this work. text: a helicopter flying as a man speaks in the background text -p: a chopper is airborne as a male speaks in the background ## experimental setup training dataset: we train our model on a combination of audiocaps @xcite and clotho @xcite , which we augment with our two levels of paraphrased captions. evaluation dataset: for t-a retrieval, we adopt the evaluation setup from previous work @xcite and employ audiocaps, clotho, audioset sl @xcite , @xcite and dcase @xcite . we evaluate for recall@10. baselines: for baselines we use ml-act @xcite , msclap-22 @xcite , msclap-23 @xcite , compa @xcite and laion-clap @xcite . we use laion-clap as the base model for robustclap. ## results and analysis quantitative analysis: ## conclusion this paper shows that current audio language models lack robustness to linguistic variation in natural language inputs. to demonstrate this phenomenon, we extend several audio-text retrieval benchmarks with paraphrased captions generated through a two-step llm-based process. to address this issue, we propose a simple mitigation strategy, training clap models with a multi-view contrastive loss on a small set of paraphrased data. the resulting model, robustclap, improves retrieval recall scores on the original benchmarks and their paraphrased versions while retaining its prior pre-trained knowledge. we hope our work fuels further studies into improving the robustness of audio-language models.