PAT : Parameter-Free Audio-Text Aligner to Boost Zero-Shot Audio Classification (2025)

Abstract: Audio-Language Models (ALMs) have demonstrated remarkable performance in zero-shot audio classification. In this paper, we introduce PAT (Parameter-free Audio-Text aligner), a simple and training-free method aimed at boosting zero-shot audio classification performance of CLAP-like ALMs. To achieve this, we propose to improve the cross-modal interaction between audio and language modalities by enhancing the representations for both modalities using mutual feedback. Precisely, to enhance textual representations, we propose a prompt ensemble algorithm that automatically selects and combines the most relevant prompts from a datastore with a large pool of handcrafted prompts and weighs them according to their relevance to the audio. On the other hand, to enhance audio representations, we reweigh the frame-level audio features based on the enhanced textual information. Our proposed method does not require any additional modules or parameters and can be used with any existing CLAP-like ALM to improve zero-shot audio classification performance. We experiment across 18 diverse benchmark datasets and 6 ALMs and show that the PAT outperforms vanilla zero-shot evaluation with significant margins of 0.42%-27.0%. Additionally, we demonstrate that PAT maintains robust performance even when input audio is degraded by varying levels of noise. We make our code publicly available.

## introduction advancements in multimodal language models (mlms) have significantly improved performance across various modalities and applications @xcite . audio-language models @xcite , specifically audio-language encoders (ales) like clap @xcite , are a distinct type of @xcite and ms clap @xcite ) with and without pat on musdb @xcite (music genre classification). our proposed training-free method significantly enhances zero-shot performance, even in low-resource domains where the ales have limited training data. mlms that learn a shared representation space between the audio and language modalities. trained on large-scale audio-caption pairs, these models acquire diverse audio concepts during pre-training, enabling them to generalize to new, unseen audio categories. this capability allows ales to excel in zero-shot audio classification, accurately categorizing any set of classes described with natural language during inference. such flexibility is essential for ales to effectively adapt to dynamic environments with diverse and unknown sounds. significant progress has been made to enhance the zero-shot performance of ales across a wide range of audio classification tasks. for instance, contrastive language audio pretraining (clap) @xcite was one of the first ale models to achieve notable zero-shot improvements by utilizing descriptive audio captions, as opposed to the class labels used in earlier models like wav2clip @xcite and audio-clip @xcite . additionally, ales such as laion-clap @xcite and ms-clap @xcite employ vast collections of audio-text pairs, while models like compa-clap @xcite leverage complex learning objectives during pretraining to show zero-shot performance gains across various audio classification tasks. while these models show zero-shot improvement, it comes at the additional cost of pre-training them with either more refined learning objectives or by improving both the quality and quantity of audio-text pairs. to overcome these challenges, researchers have attempted to improve ales' audio classification capabilities through parameter-efficient transfer learning. inspired by prompt learning (brown, 2020) and adapter finetuning approaches @xcite , treff adapter @xcite and audio prompt learner @xcite incorporate learnable prompts or lightweight adapters to adapt both textual and audio features. although these methods result in performance improvements, they introduce learnable parameters and require additional training phases with few-shot labelled data. on the other hand, training-free methods can utilize the zero-shot capabilities of ales, making them more scalable and efficient for real-world applications which can contain diverse and unknown sounds. to the best of our knowledge, there exists no trainingfree method to improve the zero-shot classification capabilities of such models. our contribution. to this end, we propose pat: parameter-free audio text aligner, a simple, parameter-free, and training-free approach for boosting zero-shot audio classification performance in ales. we first identify two major issues with current ales: 1) current ales rely on simplistic prompts like "the sound @xmath0, leading to suboptimal performance. 2) during zero-shot transfer, when we align audio and text representations through cosine-similarity, audio representations are typically average pooled, which results in information loss and reduced discriminative power @xcite . inspired by our findings, we propose two novel components: 1) weighted prompt ensemble: to enhance zeroshot performance for ales, we enrich the textual embeddings by creating a task-agnostic prompt datastore comprising 400 unique prompts. these prompts are specifically designed to reduce the distribution shift between the zero-shot setting and the training data, where audio captions typically contain more than a single-word description, unlike previous approaches. each prompt is generated from handcrafted templates, capable of accommodating unique sound labels while ensuring semantic coherence. moreover, we show that naively using all the prompts for various downstream applications may not consistently yield the desired performance improvements. to address this, we introduce weighted prompt ensemble, a method which computes a score for each prompt using the ales' response as feedback without the need for additional labeled data or training. these scores are then used as weights to perform a weighted ensemble of the text embedding associated with a prompt and a class label. 2) cross modal aligner: we introduce a cross-modal aligner that enhances audio representations with text-guided information. specifically, we first compute an attention map using parameter-free attention mechanisms between the frame-level audio representations and the enriched textual representations associated with each label. next, we use this attention map to perform weighted pooling on the frame-level audio representations. finally, we compute the cosine similarity between the enriched audio and text representations to perform zero-shot classification. to summarize, our main contributions are: ## methodology fig. 2 demonstrates our proposed method pat. we propose two simple training-free extensions for existing zero-shot transfer learning in ales. to enhance textual representation, we introduce a weighted prompt ensemble that selectively identifies and scores prompts which are more relevant to an unseen downstream task. to enhance audio representation, we introduce a cross-modal alignment that utilizes a parameter-free attention mechanism to align frame-level audio representations with textual descriptions in a zero-shot setting. in upcoming subsections, we explain pat in detail. ## results and result analysis 6.1 main results we summarize the results of pat applied to 6 different ales across 18 audio classification datasets, including 10 non-verbal speech and non-speech sound datasets and 8 musical datasets, in ## conclusion we introduce pat, a parameter-and training-free approach to improve zero-shot audio classification for audio-language encoders (ales). our studies indicate that current methods that guarantee audio classification improvements are not capable of leveraging the core strength of ales, which is zeroshot transfer and often require extensive training or additional parameters to adapt audio and textual representations for an unseen task. to overcome this limitation, pat offers two key contributions: 1) a novel prompt scoring method, called weight prompt ensemble, which adapts prompt ensembling for unseen audio classification tasks in a trainingfree fashion, and 2) a cross-modal alignment framework, which uses parameter-free attention to better align audio and textual representation. pat significantly boosts zero-shot performance in ales across various audio classification datasets.