AIR -Bench: Benchmarking Large Audio-Language Models via Generative Comprehension (2024)

Abstract: Recently, instruction-following audio-language models have received broad attention for human-audio interaction. However, the absence of benchmarks capable of evaluating audio-centric interaction capabilities has impeded advancements in this field. Previous models primarily focus on assessing different fundamental tasks, such as automatic speech recognition, and lack an assessment of the open-ended generative capabilities centered around audio. Thus, it is challenging to track the progression in the Large Audio-Language Models (LALMs) domain and to provide guidance for future improvement.In this paper, we introduce AIR-Bench (Audio InstRuction Benchmark), the first benchmark designed to evaluate the ability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench encompasses two dimensions: foundation and chat benchmarks. The former consists of 19 tasks with approximately 19k single-choice questions, intending to inspect the basic single-task ability of LALMs. The latter one contains 2k instances of open-ended question-and-answer data, directly assessing the comprehension of the model on complex audio and its capacity to follow instructions. Both benchmarks require the model to generate hypotheses directly. We design a unified framework that leverages advanced language models, such as GPT-4, to evaluate the scores of generated hypotheses given the meta-information of the audio. Experimental results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation. By revealing the limitations of existing LALMs through evaluation results, AIR-Bench can provide insights into the direction of future research. Dataset and evaluation code are available at https://github.com/OFA-Sys/AIR-Bench.

## introduction recent advancements in artificial general intelligence have been significantly driven by the emergence of large language models (llms) @xcite @xcite @xcite @xcite . these models exhibit remarkable abilities in retaining knowledge, engaging in intricate reasoning, and solving problems following human intents. motivated by the striking progress in large language models (llms), the domain of large audio-language models (lalms) has undergone a revolutionary transformation. to perceive and comprehend rich audio signals and further generate textual responses following human instructions, many works have been proposed, such as salmonn @xcite , blsp @xcite , speech-llama @xcite , and qwen-audio @xcite , showcasing promising capabilities for audio-central dialogues. however, previous lalms @xcite @xcite @xcite @xcite have predominantly concentrated on evaluation in specific fundamental tasks. the absence of a standardized benchmark for assessing the generative instructionfollowing abilities of these models has resulted in a reliance on showcasing examples or releasing the chat models for public experimentation to demonstrate their conversational skills. this approach poses significant challenges for conducting fair and objective comparisons across different research endeavors. moreover, it tends to obscure the models' existing limitations, impeding the ability to monitor advancements within the domain of lalms. for evaluation in audio domains, the majority of research efforts have concentrated on the creation of benchmarks tailored to individual tasks such as librispeech @xcite and common voice benchmark @xcite for asr. beyond task-specific ones, benchmarks like su-perb @xcite and hear @xcite have been designed to test the versatility of self-supervised learning models in a wide variety of tasks. regarding the assessment of lalms' ability to follow instructions, to the best of our knowledge, dynamic-superb @xcite is the only benchmark devoted to this aspect. nevertheless, dynamic-superb only focuses on human speech processing and does not extend to the assessment of models' capabilities in producing open-ended generations such as dialogues. in this paper, we present air-bench (audio instruction benchmark), a novel benchmark designed to evaluate the ability of lalms to comprehend various audio signals and to interact following instructions. air-bench is characterized by three primary features: 1) comprehensive audio signals coverage. air-bench offers comprehensive coverage of audio signals, including human speech, natural sounds, and music, ensuring a comprehensive evaluation of lalms' capabilities. 2) hierarchical benchmark structure. the benchmark consists of foundation and chat benchmarks. the foundation benchmark comprises 19 distinct audio tasks with over 19,000 single-choice questions, with each question focusing only on a specific foundational ability. @xcite extends the questions and candidate choices using dedicated designed prompts. the chat component consists of over 2,000 audio-prompted open-ended questions. to enhance the complexity of the audio and achieve a closer resemblance to the intricate audio encountered in real-life situations, we propose a novel audio mixing strategy that incorporates loudness control and temporal dislocation. specifically, we adjust the loudness and introduce different temporal offsets during the mixing process of two audio clips. the resulting variations in relative loudness and temporal location are then recorded as additional meta-information, contributing to a more comprehensive textual representation of the audio. the quality of data is upheld through automated filtering by gpt-4, followed by manual verification. 3) unified, objective, and reproducible evaluation framework. models are required to generate hypothesis sequences directly across both benchmarks to align more accurately with practical scenarios. then, we employ gpt-4 to generate reference answers given meta-information through carefully constructed prompts. given references and hypotheses, following @xcite ; @xcite , we use @xcite to judge whether the choice is correct for the foundation benchmark or score hypotheses for the chat benchmark. we further perform a second scoring by swapping their positions to eliminate the position bias. based on comprehensive experiments on 9 lalms, we observe that existing lalms either have limited audio understanding or instructionfollowing capabilities, leaving significant room for improvement in this field. our contribution is summarized below: ## related work benchmarks for audio processing. previous studies have primarily focused on evaluating the specific fundamental capabilities of models. in the field of speech processing, automatic speech recognition is one of the most popular tasks, with representative benchmarks including librispeech @xcite , common voice @xcite , and fleurs @xcite . additionally, there are various benchmarks available for different speech processing tasks such as speech-to-text translation @xcite and emotion recognition @xcite . in the field of sound processing, several benchmarks have emerged such as clotho @xcite and audiocaps @xcite for automatic audio captioning, and avqa @xcite for sound question answering. in the domain of music processing, numerous datasets are available, including musiccaps @xcite for automatic music captioning, and music-avqa @xcite for music question answering. note that most existing question-answering benchmarks, such as clotho-aqa, avqa, and music-avqa, have highly constrained answer formats for ease of close-ended evaluation or conversion into classification tasks, rather than supporting open-ended generation. besides the aforementioned datasets that focus on specific tasks, there are benchmarks like su-perb @xcite and hear @xcite for comprehensive evaluation of selfsupervised learning models. when it comes to assessing the ability of lalms to follow instructions, dynamic-superb is the only benchmark dedicated to this aspect. however, dynamic-superb focuses on human speech processing and does not cover open-ended dialogue generation. in contrast, air-bench is the first large-scale generative evaluation benchmark for large audio-language models, encompassing various audio types such as speech, natural sounds, and music. ## air-bench there exist three unique characteristics that differentiate air-bench from existing benchmarks for audio understanding: i) air-bench is the first work to incorporate task evaluation from all types of audio in a hierarchical taxonomy; ii) air-bench is the first generative evaluation benchmark that handles the free-form output of lalms; iii) air-bench adopts gpt-4-based automatic evaluation yielding trustworthy evaluation results with affordable cost. in sec. 3.1, we present the hierarchical taxonomy of air-bench and discuss the design philosophy behind it. in sec. 3.2 and sec. 3.3, we introduce how we collect the audio-central questionanswer pairs for foundation and chat tasks. in sec. 3.4, we present the evaluation framework. ## conclusion in this paper, we present air-bench, the first generative evaluation benchmark designed specifically for audio-language models. air-bench comprises 19 audio tasks with over 19k single-choice questions in the foundation benchmark, as well as over 2k open-ended audio questions in the chat benchmark. notably, the benchmark covers diverse audio types such as speech, natural sounds, and music. we also propose a novel audio mixing strategy to simulate audio from real-world scenarios more accurately. a standardized, objective, and reproducible evaluation framework is employed to au-tomatically assess the quality of hypotheses generated by lalms. we conduct a thorough evaluation of 9 prominent open-source lalms. additionally, we plan to launch and maintain a leaderboard that will serve as a platform for the community to access and compare model performance consistently over time.