A udio B ench: A Universal Benchmark for Audio Large Language Models (2025)

Abstract: We introduce AudioBench, a universal benchmark designed to evaluate Audio Large Language Models (AudioLLMs). It encompasses 8 distinct tasks and 26 datasets, among which, 7 are newly proposed datasets. The evaluation targets three main aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic). Despite recent advancements, there lacks a comprehensive benchmark for AudioLLMs on instruction following capabilities conditioned on audio signals. AudioBench addresses this gap by setting up datasets as well as desired evaluation metrics. Besides, we also evaluated the capabilities of five popular models and found that no single model excels consistently across all tasks. We outline the research outlook for AudioLLMs and anticipate that our open-sourced evaluation toolkit, data, and leaderboard will offer a robust testbed for future model developments.

## introduction foundation models, built upon large language models (llms), have demonstrated strong capability in handling diverse tasks and diverse modalities @xcite @xcite . there have been a series of benchmarks proposed recently to provide a holistic evaluation of llms @xcite @xcite , image-enhanced multimodal llms @xcite @xcite , and even video-enhanced multimodal llms @xcite . however, the audio large language models (audiollms) lag, and the performance of their audio interpretation capability is often intractable and not systematically compared in different tasks. the existing evaluation regime does not cover the breadth of their possible use cases. although instruction-following audio-language models have attracted significant interest from both industry and research communities, their reported evaluation datasets are notably different. for instance, the qwen-audio-chat @xcite model has tested on 12 datasets, whereas the salmonn @xcite model used 15 datasets, with only two datasets in common. although wavllm @xcite ) has been compared with some earlier models, the scope of evaluation tasks remains limited, which makes the comparison less justified. another limitation arises from the evaluation set; similar to textbased llms @xcite , evaluations predominantly rely on previous datasets and metrics @xcite . given that these models are expected to respond flexibly following the instructions, conventional evaluation metrics are not well prepared for this purpose. consequently, it is crucial to develop innovative benchmarks for audiollms. since their ultimate goal is to handle diverse audio inputs and respond accurately to user queries, the evaluation set should include not only traditional speech tasks but also new datasets and metrics that better reflect their particular nature and assess the instruction-following capability of models, such as zero-shot speech question answering, etc. in this study, we introduce audiobench as a comprehensive evaluation benchmark specifically designed for general instruction-following audio large language models, which we refer to as audi-ollms. these models are designed to comprehend general audio inputs and user queries. the tasks should include but are not limited to, understanding speech semantics, interpreting environmental sounds, and recognizing paralinguistic features, as well as combinations of these elements. therefore, as shown in figure 1 , we incorporate 8 tasks and 26 datasets covering these aspects, with 7 of the datasets being newly adapted or collected to address gaps in the existing datasets. for a more comprehensive assessment of the models, our benchmark incorporates a variety of prompt templates to evaluate model compatibility with different instructions. this is because we found that some models are more robust to seen instruction while do not generalize to unseen ones which could lead to robustness issues for deployment. additionally, we vary the input lengths from seconds to minutes to better assess model performance on longer audio sequences. our test suite includes 400+ hours of audio, distributed across 100k+ samples. our audiobench toolkit is available, enabling the evaluation of future audiollms by simply implementing a customized model inference program. we also explored and decided what evaluation metrics to leverage, particularly for open-ended generation of audiollms. inspired by text-based and vision-based large language models (llms), which can be accessed with multiple-choice and open-ended questions, audiollms are designed to manage diverse instruction and need specific metrics to cover the unconstrained generation space. this introduces difficulties in quantitatively assessing their outputs. most practices go to model-asjudge methods @xcite . however, cloud-sourced models, e.g. gpt-4, can be costly and subject to instability introduced by commercial version updates. therefore, we investigate open-source model-as-judge methods and validate the best practices by correlation studies. our findings indicate that llama-3-70b-instruct model demonstrates a higher correlation compared to judgment models like prometheus-2 @xcite which is specifically fine-tuned for evaluation purposes. finally, we evaluated 5 models to offer a comprehensive review of various solutions, including both end-to-end and cascaded models. our benchmark results indicate that none of the models performed exceptionally well across all criteria, highlighting significant opportunities for future advancements in model capabilities. to guide these developments, we also include an in-depth discussion on future challenges and directions. text-based llms present the most fundamental advancements in the past years and serve as the core for developing multimodal systems. their evaluation is also multifaceted covering aspects such as reasoning @xcite @xcite , subject knowledge @xcite @xcite , safety @xcite , multilingual capability @xcite , and more @xcite @xcite . text-based llms have driven fundamental advancements in recent years @xcite @xcite and have significantly contributed to the evolution of vision llms that integrate visual understanding with text queries @xcite . additionally, a series of multimodal benchmarks specifically tailored for vision llms have also been established. the evaluation of vision-llms is multifaceted, encompassing aspects such as multidisciplinary understanding @xcite @xcite , perception tests @xcite , mathematical reasoning @xcite , and video understanding @xcite @xcite . notably, video understanding benchmarks assess modalities including both visual and audio elements. however, these benchmarks cannot be directly applied to audio understanding, as they predominantly rely on visual inputs with audio serving merely as a supplementary feature. consequently, there is an urgent need to standardize benchmarks for audiollms that specifically measure model performance with audio context. such benchmarks are essential for advancing research and improving the capabilities of models that focus on audio processing and understanding. ## conclusion in this work, we introduce the first comprehensive evaluation benchmark for audiollms, named au-diobench, which includes 8 tasks and 26 datasets. our report includes a detailed analysis of data selection, metric design, and the evaluation pipeline. we emphasize the critical factors for benchmark design and model development. we anticipate that our proposal will stimulate further progress in multimodal foundation models.