Investigating the Emergent Audio Classification Ability of ASR Foundation Models (2024)

Abstract: Text and vision foundation models can perform many tasks in a zero-shot setting, a desirable property that enables these systems to be applied in general and low-resource settings. There has been far less work, however, on the zero-shot abilities of ASR foundation models, with these systems typically fine-tuned to specific tasks or constrained to applications that match their training criterion and data annotation. In this work we investigate the ability of Whisper and MMS, ASR foundation models trained primarily for speech recognition, to perform zero-shot audio classification. We use simple template-based text prompts at the decoder and use the resulting decoding probabilities to generate zero-shot predictions. Without training the model on extra data or adding any new parameters, we demonstrate that Whisper shows promising zero-shot classification performance on a range of 8 audio-classification datasets, outperforming the accuracy of existing state-of-the-art zero-shot baselines by an average of 9%. One important step to unlock the emergent ability is debiasing, where a simple unsupervised reweighting method of the class probabilities yields consistent significant performance gains. We further show that performance increases with model size, implying that as ASR foundation models scale up, they may exhibit improved zero-shot performance.

## introduction the evolution of large-scale pre-trained foundation models has reshaped the way various complex tasks are approached. large language models (llms) have been trained over massive text corpora @xcite @xcite and can be used out of the box for diverse nlp tasks. similarly, vision-totext models, such as those trained to predict image ## zero-shot classification of asr foundation models this paper investigates the emergent zero-shot audio classification abilities of large-scale asr foundation models. these systems are trained specifically for speech recognition and were not explicitly trained for any of the downstream classification tasks considered in this paper. we question whether one can use prompting to leverage the implicit knowledge learned from pre-training to achieve various audio classification tasks. ## conclusions this paper is the first to examine the emergent ability of foundation asr models on audioclassification tasks, that were not seen in training. over a range of tasks, we show that zero-shot prompting of whisper can yield effective performance. calibration methods can be used to readjust the output distribution for better task alignment, allowing whisper to achieve better performance compared to previous zero-shot works, and demonstrating its potential for cross-task generalization.