Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models (2025)

Abstract: The Audio Question Answering (AQA) task includes audio event classification, audio captioning, and open-ended reasoning. Recently, AQA has garnered attention due to the advent of Large Audio Language Models (LALMs). Current literature focuses on constructing LALMs by integrating audio encoders with text-only Large Language Models (LLMs) through a projection module. While LALMs excel in general audio understanding, they are limited in temporal reasoning, which may hinder their commercial applications and on-device deployment. This paper addresses these challenges and limitations in audio temporal reasoning. First, we introduce a data augmentation technique for generating reliable audio temporal questions and answers using an LLM. Second, we perform a further fine-tuning of an existing baseline using curriculum learning strategy to specialize in temporal reasoning without compromising performance on fine-tuned tasks. We demonstrate the performance of our model using state-of-the-art LALMs on public audio benchmark datasets. Third, we implement our AQA model on-device locally and investigate its CPU inference for edge applications.

## introduction multimodal question answering (mqa) involves generating relevant answers for multimedia inputs such as images, audio, and video, in response to user queries @xcite . following the success of large pretrained transformer models for mqa, audio-specialized question answering has gained traction. audio question answering (aqa) is an audio-to-text task where, given an audio file and a question, the model produces an answer by analyzing the audio content. audio question answering: recent literature @xcite @xcite in aqa develops end-to-end pretrained transformer-based architec-tures known as large audio language models (lalms). figure 1 provides a general framework for our aqa model architecture @xcite . it comprises three components: an audio encoder, a projection module, and a text decoder. the audio spectrogram transformer (ast) @xcite encodes the input audio clip into spectrogram feature representations. the projection module converts these audio feature representations into textequivalent embeddings for the text decoder. the llama model serves as the text llm decoder, taking the converted audio feature embedding and the question as input. during training, we add metadata as an optional input that is generated by the proposed data augmentation in section 2.1. it helps provide extra guidance to the llm decoder along with the text projections of the audio clip and aids in the overall audio-text representation learning. the gama model @xcite follows a similar architecture to ltu @xcite , combining multiple types of audio features, including activations from multiple layers of ast, audio q-former, and a soft prompt that provides audio events information. in this paper, we intend to discuss a few problems and limitations that we discovered in the process of developing a lalm for commercial edge devices and explain our proposed techniques to overcome them. we chose ltu as the base model for our experiments over gama due to the ease of on-device implementation. use case motivation: although lalms excel at general audio understanding and have shown good overall performance in audio captioning, classification tasks, and open-ended reasoning tasks, there is a significant gap between lalm research and real-world product requirements. first, lalms fine-tuned end-to-end with millions of audio-text samples do not capture fine-grained audio understanding well. their performance isn't impressive on specialized reasoning tasks that require finegrained understanding, such as temporal reasoning @xcite . audio temporal reasoning is the ability to understand the temporal context and relationship between events in the input media. specialized audio temporal understanding has significant potential across various sectors for commercial adoption. in healthcare, it can be used for continuous monitoring and analysis of heartbeat and respiration over a period of time and provide useful analysis and recommendations to the user. in smart homes, it can enable advanced security monitoring with privacy protection by capturing and analyzing the sequence of events in live stream audio coming from sensors located in multiple areas. @xcite explains that the lack of fine-grained understanding in lalms might be due to performing temporal downsampling at the audio encoder-projection module juncture, which is a trade-off to save computational efficiency and limited training data for temporal analysis. in this paper, we address both these limitations while also keeping in mind the limitations in commercial lalms, including low memory footprint, ease of on-device implementation, reliability, and minimal training compute. due to the difficulty in procuring large amounts of pretraining data, expensive compute power, and time constraints, it is painstakingly difficult to retrain an lalm from scratch for improving a particular skill. on top of that, the large memory requirements of lalms make it difficult to run them on low-compute edge devices. ## methodology we divide our proposed methodology into two sections. first, we explain the data augmentation strategy for generating temporal reasoning data. second, we discuss our temporal fine tuning strategy. ## conclusion in this work, we proposed a novel data augmentation strategy to generate temporal reasoning qa pairs using llms. next, we finetuned a sota aqa model on the generated temporal reasoning data and showcased quantitative improvements across evaluation metrics. finally, we showcased our implementation of the aqa model on-device and studied its performance. in the future, we will reduce the memory footprint of our aqa model to be able to fit into low-powered devices. this will also significantly reduce the active ram usage and boost encoding and decoding speeds. also, we plan to investigate quantization-aware fine-tuning techniques and study the generation quality vs. quantization trade off. we plan to introduce an evaluation metric that can appropriately select the facts from the answer and compare them against the ground truth. we can use the metric as a loss term during fine-tuning of the aqa model to prioritize the learning of specialized skills reliably.