Unsupervised Sounding Pixel Learning (2023)

Abstract: Sounding source localization is a challenging cross-modal task due to the difficulty of cross-modal alignment. Although supervised cross-modal methods achieve encouraging performance, heavy manual annotations are expensive and inefficient. Thus it is valuable and meaningful to develop unsupervised solutions. In this paper, we propose an **U**nsupervised **S**ounding **P**ixel **L**earning (USPL) approach which enables a pixel-level sounding source localization in unsupervised paradigm. We first design a mask augmentation based multi-instance contrastive learning to realize unsupervised cross-modal coarse localization, which aligns audio-visual features to obtain coarse sounding maps. Secondly, we present an *Unsupervised Sounding Map Refinement (SMR)* module which employs the visual semantic affinity learning to explore inter-pixel relations of adjacent coordinate features. It contributes to recovering the boundary of coarse sounding maps and obtaining fine sounding maps. Finally, a *Sounding Pixel Segmentation (SPS)* module is presented to realize audio-supervised semantic segmentation. Extensive experiments are performed on the AVSBench-S4 and VGGSound datasets, exhibiting encouraging results compared with previous SOTA methods.

## introduction audio-visual localization tasks have attracted much attention in recent years. various works have made great achievements, including sound source localization (ssl) @xcite , audio-visual event localization (ave), audio-visual video parsing (avvp) @xcite , and audio-visual segmentation (avs) @xcite . in this work, we focus on a variant of ssl task : unsupervised pixellevel sound source localization (pixel ssl), which aims at accurately localizing the sounding objects cross attention (a) unsup ssl. fusion refine (c) our uspl. @xcite calculates audiovisual cross attention to obtain patch-level coarse results. (b) supervised sound source localization (sup ssl) @xcite provides a supervised label for model training. (c) our unsupervised sounding pixel learning (uspl) aims to obtain pixel-level localization results in an unsupervised paradigm. despite recent advancements in ssl, some challenges remain to be addressed. one of the primary challenges is to learn accurate pixel-level sounding maps rather than approximate patch-level maps. as shown in fig. 1(a) , unsup ssl methods @xcite always learn the audio-visual co-occurrence by contrastive learning @xcite , then directly calculate the audio-visual attention to estimate the sounding maps. however, these approaches often output coarse and unreliable-sounding masks containing only localization information but lack object shape information. another challenge is doing fine-grained sound source localization in a totally unsupervised paradigm. as depicted in fig. 1(b) , to precisely identify regions of sounding, sup ssl methods @xcite build a pixel-level labeled dataset and utilize semantic segmentation models to produce pixel-level results. however, sup ssl methods are limited by the requirement for intensive annotations, which can be a burdensome task and hard to be feasible in real environments. therefore, in unsupervised sounding pixel learning (uspl, fig. 1(c )), our motivation is to produce precise pixel-level sounding object maps in videos by a totally unsupervised paradigm. to address the challenge of pixel ssl, we first conduct mask augmentation based multi-instance contrastive learning (mmicl) to align the paired audio-visual features. we highlight that the mask augmentation layer greatly alleviates the overfit and sub-optimization problem proposed in previous work @xcite . after completing the mmicl, we can use audiovisual attention to obtain patch-level sounding object results which are considered as coarse sounding maps. however, these coarse sounding maps fall short of our pixel ssl requirements. therefore, we perform an unsupervised sounding map refinement (smr) module to obtain fine sounding maps containing shape information. smr predicts semantic affinities of paired adjacent coordinates in the image by inter-pixel relation mining. then, smr produces fine sounding maps by revising the coarse sounding maps according to the affinity matrix. after these processes, smr significantly refines the coarse sounding maps, enabling object boundary information to be recovered. although smr produces fine sounding maps, it is too heavy for evaluation. thus we propose sounding pixel segmentation (sps) module. sps is an encoder-decoder architecture that includes an audio signal supervised. we use the fine sounding maps generated by smr to train the sps. we also find that sps can improve the training of smr to obtain better convergence. therefore, sps makes our methods more stable and accurate in sounding map generation. our major contributions are summarized below: ## methodology in this section, we introduce the proposed method unsupervised sounding pixel learning (uspl) to conduct the pixel ssl task. in uspl, we first align the audio and visual features in section 3.1 to obtain the coarse sounding maps. then, we propose a refined module named sounding map refinement (smr) to recover the boundary from the coarse sounding maps in section 3.2. next, we apply the fine sounding maps generated by smr to train the sounding pixel segmentation (sps) module in section 3.3. ## conclusion in this work, we propose uspl, a novel unsupervised approach for pixel ssl. specifically, we first use the mmicl module to align audio-visual features and obtain coarse sounding maps. then, we propose an smr module to refine the coarse sounding maps by the visual semantic affinity. next, we propose sps to produce final segmentation results. our approach shows promising performance on pixel ssl tasks, especially achieving new sotas on the avsbench-s4 test and vggss benchmark.