Learning Musical Representations for Music Performance Question Answering (2024)

Abstract: Music performances are representative scenarios for audio-visual modeling. Unlike common scenarios with sparse audio, music performances continuously involve dense audio signals throughout. While existing multimodal learning methods on the audio-video QA demonstrate impressive capabilities on general scenarios, they are incapable of dealing with fundamental problems within the music performances: they underexplore the interaction between the multimodal signals in performance, and fail to consider the distinctive characteristics of instruments and music. Therefore, existing methods tend to inaccurately answer questions regarding musical performances. To bridge the above research gaps, first, given the intricate multimodal interconnectivity inherent to music data, our primary backbone is designed to incorporate multimodal interactions within the context of music; second, to enable the model to learn music characteristics, we annotate and release rhythmic and music sources in the current music datasets; third, for time-aware audio-visual modelling, we align the model’s music predictions with the temporal dimension. Our experiments show state-of-the-art effects on the Music AVQA datasets. Our code is available at: https://github.com/xid32/Amuse.

## introduction imagine standing in a crowd at a live music concert, where every chord strummed, every beat of the drum, and every visual cue from the performers are in perfect harmony, creating an immersive spectacle of sound and sight. unlike common scenarios with sparse audio signals, music performances offer a tight but fluid fusion of audio and visual elements, ideal for exploring effective audio-visual scene understanding and reasoning @xcite , due to the dense and continuous audio signals throughout. this makes modeling music performances one of the most representative tasks in audio-visual learning. @xcite . the right is a sample from the music avqa, showing the dense and continuous audio signals of music performances @xcite . recently, @xcite introduce music avqa, an audio-visual question answering (avqa) dataset for music performances. @xcite later extend this dataset, refining a more challenging and balanced version named music avqa-v2. music avqa datasets require models to leverage both audio and visual representations to answer questions that closely correspond to the presented audio-visual content. existing deep learning methods for avqa have made notable progress: earlier research by @xcite showcase cross-modality feature learning to boost audio-visual speech recognition, while @xcite use a multimodal deep boltzmann machine for multimodal learning. more recent strategies in multimodal fusion @xcite and creating positive-negative pairs @xcite have been introduced to improve audio-visual correlation learning. as a later advancement, lav-ish @xcite is introduced as an adapter to extend the pretrained vit @xcite to audio-visual data for cross-modal learning. additionally, @xcite present dg-sct, which uses audio and visual modalities as prompts in frozen pretrained encoders. however, prior avqa methods, designed for general purposes, face significant challenges when applied to specific music performances, which involve continuously dense audio and interconnected multimodal music signals (as shown in fig. 1 ). these challenges result in sub-optimal accuracy: (i) they often overlook the musical characteristics (rhythm and music source) of instruments and players, as their designs are not originally specified for music performances; (ii) they lack sufficient explicit supervision for modeling music rhythm and music sources over time, which is essential for answering temporal-related questions. most methods rely on attention mechanisms but fail to directly align music rhythm and music sources to the temporal dimension for audio-visual qa. (iii) more essentially, these methods often use either unimodal encoders or parallel non-interactive encoders. this results in influence by irrelevant noise when processing other modalities and specific questions, or a lack of early-stage fusion on intricate interconnected multimodal representations for music avqa data. to overcome these challenges, we propose amuse framework, differing from prior works as shown in tab.1, specially designed for audio-visual question answering in musical performance: state-of-the-art (sota) methods include lavish @xcite , an adapter that generalizes pretrained vit @xcite to audiovisual data for cross-modal learning, and dg-sct @xcite , which leverages audio and visual modalities as prompts in pretrained frozen encoders. despite advances, fundamental problems persist, including temporal inconsistency between audio-visual modalities and inaccurate question answering. to address these challenges, we propose amuse, which identifies musical specialties like rhythm and music sources from both vision and audio, and further aligns them with the temporal dimension. ## experiments amuse addresses unique music characteristics not considered by prior state-of-the-art avqa methods, as tab.1 summarized. thus, first, we compare amuse with other sota avqa methods to evaluate its performance. second, we iteratively verify the effectiveness of our multimodal interactive encoder's early fusion on overall performance. we then assess the impact of incorporating music rhythm and source, their alignment with the temporal dimension, and the contribution of musical rois to the framework's ability to accurately capture music elements for final predictions. ## conclusion we introduce amuse, the first framework specifically designed for audio-visual question answering in music. amuse effectively addresses the main challenges in this field: (i) the lack of early-stage fusion for interconnected multimodal representation, (ii) the under-utilization of musical characteristics along the temporal dimension, and (iii) insufficient supervision for modeling rhythm and sources. by overcoming these challenges, amuse provides valuable insights for future research on musical performance and multimodal understanding with continuous and dense audio.