S a SR -Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering (2024)

Abstract: Audio-Visual Question Answering (AVQA) is a challenging task that involves answering questions based on both auditory and visual information in videos. A significant challenge is interpreting complex multi-modal scenes, which include both visual objects and sound sources, and connecting them to the given question. In this paper, we introduce the Source-aware Semantic Representation Network (SaSR-Net), a novel model designed for AVQA. SaSR-Net utilizes source-wise learnable tokens to efficiently capture and align audio-visual elements with the corresponding question. It streamlines the fusion of audio and visual information using spatial and temporal attention mechanisms to identify answers in multi-modal scenes. Extensive experiments on the Music-AVQA and AVQA-Yang datasets show that SaSR-Net outperforms state-of-the-art AVQA methods. We will release our source code and pre-trained models.

## introduction recent contributions to the field of audio-visual question answering (avqa) include the creation of diverse datasets and sophisticated models @xcite @xcite @xcite . for example, the pano-avqa dataset @xcite ) contains 360-degree videos paired with corresponding qa sets, while the avqa-yang dataset @xcite is designed for answering audio-visual questions in real-world scenarios. the music-avqa dataset @xcite further broadened the research scope by focusing on spatio-temporal understanding in audio-visual scenes. this dataset uses a dual attention mechanism, identifying soundproducing areas visually first and then applying attention for spatio-temporal reasoning. more recently, pstp-net @xcite was introduced, which progressively identifies key regions relevant to audio-visual questions using refined attention mechanisms. existing avqa methods typically employ general audio and visual encoders to extract features from videos. however, this strategy often fails to link certain sound-producing objects in the video with the responses. consider questions like what is the instrument on the left of the cello? which necessitates specific type and location awareness, as shown in fig. 1 . current models often find it difficult to associate the cello mentioned in the question with its actual representation in the video scene. to address these challenges, we propose the source-aware semantic representation network @xcite . this model enhances the understanding and integration of individual sound sources and visual objects in avqa by two strategies: the efficacy of sasr-net is demonstrated by its performance on the music-avqa @xcite and avqa-yang @xcite datasets, where it surpasses state-of-the-art avqa approaches. the results highlight the effectiveness of the model's source-aware and semantically driven approach in managing complex audio-visual data. our key contributions are as follows: ## the proposed sasr-net given a video with both visual and audio tracks, along with a question related to the content within the video, the objective of the avqa task is to predict an accurate answer response. to achieve this, we propose a novel sasr-net architecture. this model is designed to generate compact, semanticaware embeddings by identifying salient sounding objects present in the audio-visual input that are relevant to the given query. the overview of our proposed framework is illustrated in figure 2 . ## conclusion in this paper, we present sasr-net, a novel avqa approach that introduces source-aware learnable tokens to effectively capture and integrate semanticaware audio-visual representations. this enhances alignment between audio elements and visual cues, crucial for identifying relevant scene regions and their association with questions. by excelling at extracting and understanding single-source information within complex scenes, sasr-net significantly improves performance on avqa tasks. limitation: while sasr-net achieves remarkable performance on multi-modal tasks, its results on single-modality (audio-only or visual-only) questions are not as outstanding. this may be due to training data bias, as the dataset contains a higher proportion of audio-visual questions, leading the model to be better tuned for multi-modal scenarios. to address this issue, we can fine-tune sasr-net on single-modality tasks, aiming to enhance its performance on audio-only and visual-only questions while maintaining its strong capabilities in multimodal contexts. additionally, sasr-net may still face challenges in handling extremely noisy audio-visual data or scenarios with highly complex and overlapping audio sources. these situations could affect the model's ability to accurately extract and align semantic representations, highlighting areas for future improvement and research.