PALM : Few-Shot Prompt Learning for Audio Language Models (2024)

Abstract: Audio-Language Models (ALMs) have recently achieved remarkable success in zero-shot audio recognition tasks, which match features of audio waveforms with class-specific text prompt features, inspired by advancements in Vision-Language Models (VLMs). Given the sensitivity of zero-shot performance to the choice of hand-crafted text prompts, many prompt learning techniques have been developed for VLMs. We explore the efficacy of these approaches in ALMs and propose a novel method, Prompt Learning in Audio Language Models (PALM), which optimizes the feature space of the text encoder branch. Unlike existing methods that work in the input space, our approach results in greater training efficiency. We demonstrate the effectiveness of our approach on 11 audio recognition datasets, encompassing a variety of speech-processing tasks, and compare the results with three baselines in a few-shot learning setup. Our method is either on par with or outperforms other approaches while being computationally less demanding. Our code is publicly available at https://asif-hanif.github.io/palm/ .

## introduction inspired by the success of vision-language models (vlms) @xcite , audio-language models (alms) have recently emerged, achieving state-of-the-art performance on various zero-shot audio recognition tasks @xcite @xcite . in zero-shot audio recognition, features of the audio waveform are matched with features of text prompts representing each class, and the highest matching class is assigned to the audio waveform. zero-shot audio recognition offers some advantages by eliminating the need for extensive labeled datasets and allowing for the recognition of new classes without additional † https://asif-hanif.github.io/palm/ zero shot coop cocoop palm methods avg. accuracy 0.397 0.711 0.735 0.766 figure 1: comparison of our proposed approach, palm, with three baselines: zero-shot @xcite , coop @xcite and cocoop @xcite . bar plots show classification accuracy averaged across 11 audio datasets encompassing various speech-processing tasks. the choice of text prompt is crucial for pre-trained vision-language and audio-language models, but it becomes a drawback for zero-shot recognition due to the requirement of hand-crafted prompts. this manual prompt-engineering can result in performance variations @xcite . we confirm this observation, previously noted in vlms, within the context of alms (refer to figure 2 ). to automate the learning of text prompts, various approaches have been introduced for prompt learning in vlms @xcite . the domain of prompt learning in alms remains under-explored, lacking comprehensive studies to evaluate the effectiveness of prompt learning techniques within this context. to bridge contributions: our contributions are as follows: ## related work prompt engineering involves adding task-specific hints, called prompts, to a large pre-trained model to adapt it to new tasks. recently, significant advancements have been made in prompt learning, particularly in the fields of language and vision. below, we outline the recent developments in language, vision, and audio domains. ## conclusion in this study, we investigate the application of prompt learning techniques, originally developed for vision-language models (vlms), in the context of audio-language models (alms). we introduce palm, a novel method that optimizes the feature space of the text encoder branch, enhancing training efficiency compared to existing methods that operate in the input space. evaluated on 11 diverse audio recognition datasets, palm consis-tently matches or surpasses established baselines in a few-shot learning setup while reducing computational demands. palm offers a promising direction for enhancing the performance of alms in zeroshot and few-shot learning scenarios, contributing to the broader field of audio recognition and paving the way for future research in multimodal tasks.