Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamic Audio-Visual Scenarios (2023)

Abstract: Audio-visual question answering (AVQA) is a challenging task that requires multistep spatio-temporal reasoning over multimodal contexts. Recent works rely on elaborate target-agnostic parsing of audio-visual scenes for spatial grounding while mistreating audio and video as separate entities for temporal grounding. This paper proposes a new target-aware joint spatio-temporal grounding network for AVQA. It consists of two key components: the target-aware spatial grounding module (TSG) and the single-stream joint audio-visual temporal grounding module (JTG). The TSG can focus on audio-visual cues relevant to the query subject by utilizing explicit semantics from the question. Unlike previous two-stream temporal grounding modules that required an additional audio-visual fusion module, JTG incorporates audio-visual fusion and question-aware temporal grounding into one module with a simpler single-stream architecture. The temporal synchronization between audio and video in the JTG is facilitated by our proposed cross-modal synchrony loss (CSL). Extensive experiments verified the effectiveness of our proposed method over existing state-of-the-art methods.

## introduction audio-visual question answering (avqa) has received considerable attention due to its potential applications in many real-world scenarios. it provides avenues to integrate multimodal information to achieve scene understanding ability as humans. as shown in figure 1 , the avqa model aims to answer questions regarding visual objects, sound patterns, and their spatio-temporal associations. compared to traditional video question answering, the avqa task presents specific challenges in the following areas. firstly, it involves effectively fus-figure 1 : an illustration of the audio-visual question answering task. concretely, the question is centered around the "instruments" (i.e., the target) and is broken down into "how many", "did not sound", and "from beginning to end" in terms of visual space, audio, and temporality, respectively. identifying the three instruments that did not produce sound throughout the video may entail a significant time investment for a human viewer. nonetheless, for an ai system with effective spatio-temporal reasoning capabilities, the task can be accomplished much more efficiently. although there have been a number of promising works @xcite @xcite in the audio-visual scene understanding community that attempted to solve the first challenge, they are primarily a targetless parsing of the entire audio-visual scenes. most of them @xcite @xcite obtain untargeted sound-related visual regions by designing attention schemes performing on audio-to-visual while ignoring the questionoriented information from the text modality. however, the understanding of audio-visual scenes in avqa tasks is often target-oriented. for example, as illustrated in figure 1 , our focus lies solely on the subject of inquiry, i.e., instruments, disregarding the singing person or ambient sound. traditional avqa approaches @xcite @xcite , inherited from the audio-visual scene understanding community, rely on aligning all audio-visual elements in the video to answer a question. this results in much irrelevant information and difficulties in identifying the relevant objects in complex scenes. as for the second challenge, most existing methods @xcite @xcite employ a typical attention-based two-stream framework. as shown in figure 2 .a, such a two-stream architecture processes audio and video in each stream separately while overlooking the unity of audio and visual modalities. in particular, the temporal grounding and audio-visual fusion are isolated, with fusion occurring through an additional module. to effectively address these two challenges, we propose a target-aware joint spatio-temporal grounding (tjstg) network for avqa. our proposed approach has two key components. firstly, we introduce the target-aware spatial grounding (tsg) module, which enables the model to focus on audio-visual cues relevant to the query subject, i.e., target, instead of all audio-visual elements. we exploit the explicit semantics of text modality in the question and introduce it during audio-visual alignment. in this way, there will be a noticeable distinction between concepts such as the ukulele and the guitar. accordingly, we propose an attention-based target-aware (ta) module to recognize the query subject in the question sentence first and then focus on the interesting sounding area through spatial grounding. secondly, we propose a cross-modal synchrony loss (csl) and corresponding joint audio-visual temporal grounding (jtg) module. in contrast to the existing prevalent two-stream frameworks that treat audio and video as separate entities (figure 2 .a), the csl enforces the question to have synchronized attention weights on visual and audio modalities during question-aware temporal grounding (figure 2 .b) via the js divergence. furthermore, it presents avenues to incorporate question-aware temporal grounding and audio-visual fusion into a more straightforward single-stream architecture (figure 2 .c), instead of the conventional approach of performing temporal grounding first and fusion later. in this way, the network is forced to jointly capture and fuse audio and visual features that are supposed to be united and temporally synchronized. this simpler architecture facilitates comparable or even better performance. the main contributions of this paper are summarized as follows: ## methodology to solve the avqa problem, we propose a targetaware joint spatio-temporal grounding network and ensure the integration between the audio-visual modalities by observing the natural integrity of the audio-visual cues. the aim is to achieve better audio-visual scene understanding and intentional spatio-temporal reasoning. an overview of the proposed framework is illustrated in figure 3 . ## experiments this section presents the setup details and experimental results on the music-avqa dataset. we also discuss the model's performance and specify the effectiveness of each sub-module in our model through ablation studies and qualitative results. ## conclusions this paper proposes a target-aware spatial grounding and joint audio-visual temporal grounding to better solve the target-oriented audio-visual scene understanding within the avqa task. the targetaware spatial grounding module exploits the explicit semantics of the question, enabling the model to focus on the query subjects when parsing the audio-visual scenes. also, the joint audio-visual temporal grounding module treats audio and video as a whole through a single-stream structure and encourages the temporal association between audio and video with the proposed cross-modal synchrony loss. extensive experiments have verified the superiority and robustness of the proposed module. our work offers an inspiring new direction for audio-visual scene understanding and spatiotemporal reasoning in question answering.