GAMA : A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities (2024)

Abstract: Perceiving and understanding non-speech sounds and non-verbal speech is essential to making decisions that help us interact with our surroundings. In this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model (LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We build GAMA by integrating an LLM with multiple types of audio representations, including features from a custom Audio Q-Former, a multi-layer aggregator that aggregates features from multiple layers of an audio encoder. We fine-tune GAMA on a large-scale audio-language dataset, which augments it with audio understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for Complex Audio Reasoning), a synthetically generated instruction-tuning (IT) dataset with instructions that require the model to perform complex reasoning on the input audio. We instruction-tune GAMA with CompA-R to endow it with complex reasoning abilities, where we further add a soft prompt as input with high-level semantic evidence by leveraging event tags of the input audio. Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for evaluating the capabilities of LALMs on open-ended audio question-answering that requires complex reasoning. Through automated and expert human evaluations, we show that GAMA outperforms all other LALMs in literature on diverse audio understanding tasks by margins of 1%-84% and demonstrates state-of-the-art performance on deductive reasoning and hallucination evaluation benchmarks. Further, GAMA IT-ed on CompA-R proves to be superior in its complex reasoning capabilities.

## introduction large language models (llms) possess impressive abilities to understand and reason about the world through language @xcite . while * co-leads with equal contribution. † co-advisors. audio encoder llm aggregator qformer soft prompt llm ast write an audio caption describing the sound in detail. based on the various sounds, please infer what the man might be announcing. write an audio caption describing the sound in detail. based on the various sounds, please infer what the man might be announcing. a train horn blows and a bell rings. it is not possible to infer what the man might be announcing without additional context. the train is approaching a station, indicated by the increasing volume of the horn and the bell ringing, suggesting it's about to stop or arrive at the station for a passenger pickup or drop off. the wind noise suggests an open outdoor environment near the tracks. a man's voice can be heard in the background, making an announcement. given the context of the train station, the man might be announcing arrival/departure schedules, boarding calls, or safety instructions. ## related work large multi-modal and audio-language models. prior to the exploration of llms as efficient reasoners, encoder-based multi-modal language models, trained to learn a shared space between language and other modalities, have shown great promise. for example, clap, inspired by clip @xcite in vision, showed stateof-the-art performance on audio-language tasks like retrieval, zero-shot classification, etc. ## methodology in the next sub-sections, we first describe the gama architecture and its components in detail, followed by fine-tuning gama on audio-language pairs, compa-r creation, and instruction-tuning gama on compa-r. ## results and analysis quantitative results. for zero-shot classification evaluation on vocalsound (vs) @xcite , tut 2017 (tut) @xcite , beijing opera (bjo) @xcite , gtzan (gtz) @xcite and medley-solos-db (mdb) @xcite , gama outperforms our baselines by 2%-67%. for weak zero-shot evaluation on esc-50 (piczak, 2015) and dcase2017 task 4 (dcase) @xcite , gama outperforms our baselines by 1%-66%. finally, for in-domain evaluation on vggsound (vgg) @xcite , fsd50k (fsd) @xcite , audioset (as) @xcite and nsynth (ns) @xcite gama outperforms our baselines by 1%-84%. gama sees the steepest drop in performance when the ast and aggregator are removed (i.e., only auio q-former is employed). gama achieves the highest overall f1 score, outperforming all other models, with salmonn being its closest competitor. on the right side of question: based on the given audio, infer the type of environment or setting. consider the presence and duration of car sounds. the audio clip suggests an urban or suburban environment with a mix of residential and commercial areas. the presence of car sounds suggests that the environment is a busy one. the given audio suggests that the environment or setting is a busy city street with a lot of car traffic. the audio also indicates that the car sounds are continuous and long-lasting. therefore, it can be inferred that the audio is from a busy city street with a lot of car traffic. it is music. question: considering the correlation of the man's speech and the music, infer his possible connection to the music playing in the scene. the male speaker's speech adds a sense of authority and control to the event's atmosphere, as well as a sense of urgency or importance to the crowd's reactions. the speech also provides a contrast to the music and crowd reactions, creating a dynamic and engaging soundscape that draws the listener's attention and emotions. the male speaker is interspersed with the music and crowd reactions to create a dynamic and engaging atmosphere at the event. his speech adds energy and excitement to the event, keeps the audience engaged and involved, and helps create a memorable and enjoyable experience for all attendees. an lalm with a question about the input audio that requires advanced understanding and complex reasoning regarding the audio and its individual events. gama is able to provide faithful and accurate responses through improved audio perception and reasoning capabilities. he refers to scores assigned by human evaluators. note that the video is only provided for illustration purposes and not provided as input to the lalm. ## conclusion in this paper, we propose gama, an lalm with improved audio perception abilities. we integrate an llm with multiple types of audio representations, which are responsible for providing diverse knowledge about the input audio. gama fine-tuned on a mixture of open-source datasets outperforms prior audio-language models by significant margins on 16 datasets spanning 4 tasks. next, we propose compa-r, an instruction-tuning dataset that we synthesize using a robust pipeline for endowing an lalm with complex reasoning abilities. gama it-ed on compa-r outperforms baselines on complex open-ended aqa and dense captioning.