Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models (2024)

Abstract: Various audio-LLMs (ALLMs) have been explored recently for tackling different audio tasks simultaneously using a single, unified model. While existing evaluations of ALLMs primarily focus on single-audio tasks, real-world applications often involve processing multiple audio streams simultaneously. To bridge this gap, we propose the first multi-audio evaluation (MAE) benchmark that consists of 20 datasets from 11 multi-audio tasks encompassing both speech and sound scenarios. Comprehensive experiments on MAE demonstrate that the existing ALLMs, while being powerful in comprehending primary audio elements in individual audio inputs, struggling to handle multi-audio scenarios. To this end, we propose a novel multi-audio-LLM (MALLM) to capture audio context among multiple similar audios using discriminative learning on our proposed synthetic data. The results demonstrate that the proposed MALLM outperforms all baselines and achieves high data efficiency using synthetic data without requiring human annotations. The proposed MALLM opens the door for ALLMs towards multi-audio processing era and brings us closer to replicating human auditory capabilities in machines.

## introduction large language models (llms) have become remarkably powerful, driving advancements in various tasks across the field of natural language processing (nlp) @xcite @xcite . recent advancements in llms have also led to the development of various powerful audio large language models (allms) @xcite @xcite , which have achieved impressive results on a range of audio tasks, e.g., automatic speech recognition @xcite , speech synthesis @xcite , sound event classification @xcite . however, a crucial limitation exists: current allm training and evaluation primarily focus on single audio inputs. this is a significant drawback, as various real-world applications, e.g., virtual assistants, often require processing multiple audio streams simultaneously. additionally, multi-audio processing is essential for effectively implementing few-shot in-context learning, which is a fundamental capability for advanced llms. unlike text-based llms, which excel at handling multiple texts @xcite , and vision llms, which have established benchmarks for processing multiple images @xcite @xcite , the audio field lacks systematic evaluations and benchmarks for multi-audio tasks with allms. although several allms @xcite claim to handle multi-audio contexts, their performance quantification remains unclear. this underscores a significant gap in the audio field, in contrast to the vision and language fields, which already possess established evaluations and benchmarks for their respective tasks. to bridge this gap, we propose the first dedicated multi-audio benchmark for allms, encompassing 11 tasks for both open-ended and closeformed generation across speech and sound domains. comprehensive experiments on 15 allms reveal that existing open-source allms fall short in multi-audio scenarios despite their strong singleaudio processing abilities. for instance, as shown in fig. 1 , while allms can perfectly identify au-dio1 as clapping and audio2 as raining respectively in a single-audio scenario, they struggle to determine whether audio1 and audio2 are the same when tasked with understanding the relationship between two audio inputs. this suggests that current allms are not well-equipped to handle multiaudio tasks, even those that are straightforward for humans. this further underlines the need for advancements in allms to handle multiple audio inputs to enhance human-computer interaction. to this end, we propose an innovative and scalable multi-audio large language model (mallm) that effectively captures audio contexts essential for reasoning over multiple audios. inspired by the success of discriminative learning @xcite @xcite , we define a challenging discriminative task that trains the model to discover the subtle differences between two similar audio samples. furthermore, we introduce a scalable audio pairs synthesis strategy to enable multi-audio processing ability without the need for data collection and human labeling. comprehensive experiments show that mallm significantly outperforms existing open-source allms under multi-audio scenarios while maintaining competitiveness under single-audio scenarios. overall, our contributions include: (1) novel evaluation benchmark: we propose the first multi-audio benchmark (mae) for evaluating the multi-audio processing capabilities of allms, encompassing a diverse range of tasks from both the speech and sound domains. (2) advanced multi-audio llm: beyond focusing on the single audio content, our proposed mallm not only demonstrates remarkable performance across diverse multi-audio tasks but also achieves remarkable data efficiency through an innovative data synthesis strategy. (3) comprehensive evaluation: we conduct a comprehensive evaluation on 15 allms across various tasks, providing a solid foundation for future research. ## related works recently, numerous studies have integrated audio encoders with pre-trained llms to develop multimodal allms, serving as general-purpose task solvers for various audio-input tasks across speech and sound domains. most allms focus on specific audio types, like speech @xcite @xcite , or sound @xcite @xcite . a few recent models handle multiple types of audio, showing strong capabilities in universal audio understanding @xcite @xcite . meanwhile, to effectively benchmark the advancements in allms, new evaluation benchmarks such as air-bench @xcite and dynamic-superb @xcite ) have been introduced. however, these benchmarks and existing allms primarily focus on tasks involving single audio inputs, largely overlooking scenarios with multiple audio streams. in contrast, our work introduces the mae benchmark, specifically designed to assess multi-audio processing capabilities. notably, mae includes a variety of tasks across diverse scenarios, encompassing both audio and speech, making it suitable for benchmarking a broad spectrum of allms. additionally, we develop the mallm, the first allm specifically tailored for multi-audio tasks, demonstrating significant improvements in processing multiple audio streams while maintaining competitive performance on single-audio tasks. ## mae benchmark we propose the first multi-audio benchmark (mae) for evaluating allms, as illustrated in fig. 2 . the mae comprehensively benchmarks the multiaudio processing capabilities of allms by including a wide variety of generation tasks from different fields and scenarios. it comprises six speech tasks and five sound tasks, covering both open-ended and close-form generation tasks. open-ended questions allow models to produce free-form responses without predefined constraints. conversely, closedform questions restrict models to a predetermined set of possible outcomes. the mae is automatically constructed using an advanced text-only llm from various existing single-audio datasets without the need for further human annotation. each sample in the mae includes a combination of two audio contexts and a task instruction. speech identification speech1: how came you to leave the key in the door [coughing] speech2: how came you to leave the key in the door [sneezing] are the contents of speech1 and speech2 same? ## mallm in this section, we introduce the specifics of our proposed mallm. fig. 3 depicts our discriminative allm fine-tuning strategy. specifically, mallm is trained to describe the subtle distinctions between two similar audios, enhancing its capability to handle multi-audio scenarios. to effectively tackle the challenging discriminative problem, the mallm is expected to possess a deep comprehension of all input audios and infer both intra-audio and inter-audio relationships. in this way, mallm can be effectively augmented with reasoning abilities across multiple audios. furthermore, to improve the data efficiency and scalability, we propose the automatic construction of a synthetic training dataset without additional human intervention. this dataset includes pairs of both synthetic speech and sound pairs with subtle differences, which are used to fine-tune mallm, expanding its multi-audio processing capabilities. ## conclusion in this paper, we introduce the first multi-audio evaluation benchmark, mae, to examine the multiaudio processing ability of allms. extensive evaluations across 15 allms reveal generally unsatisfactory performance among current open-source models in handling multi-audio scenarios. to this end, we develop a simple and effective discriminative training framework that leverages synthetic data. the resulting model, mallm, significantly surpasses all existing open-source allms without incurring additional costs for human annotation. this work lays a robust foundation for future research aimed at enhancing allm capabilities in multi-audio processing. future work includes introducing more complex multi-audio tasks that involve more than two audios and expanding the mallm training dataset to cover more scenarios.