Original: Tackling Overfitting and Silence inUnsupervised Audio-Visual Source Localization
Revised: Tackling Overfitting and Silence in Audio-Visual Source Localization

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Audio-visual source localization is a challenging task that aims to predict the location of visual sound sources in a video.
Revised: Audio-visual source localization is a challenging task that aims to predict the location of visual sound sources in a video.

Original: Since collecting ground-truth annotationsof sounding objects can be costly, a plethora of unsupervised localization methodshave been proposed in recent years.
Revised: Since collecting ground-truth annotationsof sounding objects can be costly, a plethora of unsupervised localization methodsthat can learn from audio-visual correspondence alone have been proposed inrecent years.

Original: Despite significant interest, current evaluationprotocols have two major flaws.
Revised: Despite significant interest, current evaluation protocols have twomajor flaws.

Original: First, they allow for the use of a fully annotateddataset to perform early stopping, thus defeating the purpose of unsupervisedsource localization.
Revised: First, they allow for the use of a fully annotated dataset to performearly stopping, thus defeating the purpose of unsupervised source localization.

Original: Second, current evaluation metrics assume the presence ofsound sources at all times.
Revised: Second, current evaluation metrics assume the presence of sound sources at alltimes.

Original: This is of course an unrealistic assumption, and thusbetter metrics are necessary to capture the model’s performance on (negative)samples with no visible sound sources.
Revised: This is of course an unrealistic assumption, and thus better metrics arenecessary to capture the model’s performance on (negative) samples with no visiblesound sources.

Original: To accomplish this, we extend the test set ofpopular benchmarks, Flickr SoundNet and VGG-Sound Sources, in order to includenegative samples, and measure performance using metrics that balance precisionand recall.
Revised: To accomplish this, we extend the test set of popular benchmarks, Flickr SoundNet and VGG-Sound Sources, in order to include negative samples,and measure performance using metrics that balance precision and recall.

Original: Using the new protocol, we conducted an extensive evaluation of priormethods, and found that most prior works are not capable of identifying negativesand suffer from significant overfitting problems (rely heavily on early stopping forbest results).
Revised: Usingthe new protocol, we conducted an extensive evaluation of prior methods, andfound that most prior works are not capable of identifying negatives and sufferfrom significant overfitting problems (rely heavily on early stopping for best results).

Original: We also propose a new approach for visual sound source localizationthat addresses both these problems.
Revised: We also propose a new approach for visual sound source localization thataddresses both these problems.

Original: In particular, we found that, through extremevisual dropout and the use of momentum encoders, the proposed approach combatsoverfitting effectively, and establishes a new state-of-the-art performance on both
Revised: In particular, we found that, through extremevisual dropout and the use of momentum encoders, the proposed approach combatsoverfitting effectively, and establishes a new state-of-the-art performance on both

Original: Flickr SoundNet and VGG-Sound Source.
Revised: Flickr SoundNet and VGG-Sound Source.

Original:  Introduction
Revised:  Introduction

Original: From the meow of a cat to the roar of a lion to the revving of a motorcycle, humans have evolved tolocalize sources of sound in their environment to increase chances of survival.
Revised: From the meow of a cat to the roar of a lion to the revving of a motorcycle, humans have evolved tolocalize sources of sound in their environment to increase chances of survival.

Original: This ability relies inpart on the uniqueness of sounds made by different entities – distinct sounds can be associated withtheir sources, which in turn can be visually localized in the environment.
Revised: This ability relies inpart on the uniqueness of sounds made by different entities – distinct sounds can be associated withtheir sources, which in turn can be visually localized in the environment.

Original: Due to the fast progress inaudio and visual perception several efforts have been made to develop multi-modal systems capableof mimicking our ability to visually localize sound sources.
Revised: Due to the fast progress inaudio and visual perception, several efforts have been made to develop multi-modal systems capableof mimicking our ability to visually localize sound sources.

Original: One promising direction is to leverage the co-occurrence between sounds and the corresponding(visible) sources in video data.
Revised: One promising direction is to leverage the co-occurrence between sounds and the corresponding(visible) sources in video data.

Original: Since audio-visual co-occurrence arises naturally, algorithms canscale to very large datasets, without requiring costly human annotations.
Revised: Since audio-visual co-occurrence arises naturally, algorithms canscale to very large datasets, without requiring costly human annotations.

Original: However, despite thesignificant efforts made to develop self-supervised localization methods [1, 2, 3, 4, 5], currentlyaccepted evaluation protocols hide two critical limitations of current approaches: (1) current methodsoverfit easily even when scaled up to large datasets, and (2) current methods assume that visiblesound sources are always present in the video and thus are unreliable when deployed on realistic datawhere this assumption does not hold.
Revised: However, despite thesignificant efforts made to develop self-supervised localization methods [1, 2, 3, 4, 5], currentlyaccepted evaluation protocols hide two critical limitations of current approaches: (1) current methodsoverfit easily even when scaled up to large datasets, and (2) current methods assume that visiblesound sources are always present in the video and thus are unreliable when deployed on realistic datawhere this assumption does not hold.

Original: The first limitation remained hidden as prior works [2, 4, 6, 5] rely heavily on early stopping foroptimal performance ( i.e ., by continuously validating the model during training using a humanannotated test set).
Revised: The first limitation remained hidden as prior works [2, 4, 6, 5] rely heavily on early stopping foroptimal performance ( i.e ., by continuously validating the model during training using a humanannotated test set).

Original: However, this practice not only violates the unsupervised assumption but is alsoat odds with the major promise of unsupervised learning – i.e ., enhanced capabilities when trainingon larger amounts of data for longer.
Revised: However, this practice not only violates the unsupervised assumption but is alsoat odds with the major promise of unsupervised learning – i.e ., enhanced capabilities when trainingon larger amounts of data for longer.

Original: The second limitation remained hidden as current benchmark datasets [1, 6] and evaluation metricsonly assess the model’s ability to localize a sound source when one is present in the video.
Revised: The second limitation remained hidden as current benchmark datasets [1, 6] and evaluation metricsonly assess the model’s ability to localize a sound source when one is present in the video.

Original: Importantly,it ignores a model’s ability to correctly predict the absence of visual sound sources.
Revised: Importantly,it ignores a model’s ability to correctly predict the absence of visual sound sources.

Original: This has led to abias towards precision with a complete disregard for false positives.
Revised: This has led to abias towards precision with a complete disregard for false positives.

Original: However, effective deploymentof source localization in realistic settings requires both low type I and type II error rates.
Revised: However, effective deploymentof source localization in realistic settings requires both low type I and type II error rates.

Original: The two limitations described above highlight the need for a more balanced and complete evaluationprotocol for visual sound source localization.
Revised: The two limitations described above highlight the need for a more balanced and complete evaluationprotocol for visual sound source localization.

Original: To achieve this, we extend popular benchmark test
Revised: To achieve this, we extend popular benchmark test

Original: sets(Flickr SoundNet [1] and VGG-Sound Source [6]) to include ’negative’ samples without any visiblesound sources.
Revised: sets(Flickr SoundNet [1] and VGG-Sound Source [6]) to include ’negative’ samples without any visiblesound sources.

Original: We then conduct an extensive evaluation of existing methods [2, 3, 4, 6, 7], where inaddition to overall precision, we also assess methods based on their ability to predict negative samples(through average precision and F1 scores).
Revised: We then conduct an extensive evaluation of existing methods [2, 3, 4, 6, 7], where inaddition to overall precision, we also assess methods based on their ability to predict negative samples(through average precision and F1 scores).

Original: We observe that all prior work suffers from significantoverfitting problems, and thus leans heavily upon early stopping to achieve the best performance.
Revised: We observe that all prior work suffers from significantoverfitting problems, and thus leans heavily upon early stopping to achieve the best performance.

Original: Wealso found that previous approaches fail to identify the frames where no visible sound sources arepresent.
Revised: Wealso found that previous approaches fail to identify the frames where no visible sound sources arepresent.

Original: To address these issues, we further propose a new algorithm - Momentum Audio-Visual Matching for
Revised: To address these issues, we further propose a new algorithm - Momentum Audio-Visual Matching for

Original: Visual Sound Localization ( MoVSL ).
Revised: Visual Sound Localization ( MoVSL ).

Original: First, to combat overfitting, we adopt slow-moving momentumaudio and visual encoders and extreme visual dropout.
Revised: First, to combat overfitting, we adopt slow-moving momentumaudio and visual encoders and extreme visual dropout.

Original: Second, we localize visual sources bysimultaneously performing sounding region selection and cross-modal instance discrimination.
Revised: Second, we localize visual sources bysimultaneously performing sounding region selection and cross-modal instance discrimination.

Original: Thefirst term highlights which regions within an image are most associated with particular audio, whilethe second term downplays regions that can be better described by the audio of other samples.
Revised: Thefirst term highlights which regions within an image are most associated with particular audio, whilethe second term downplays regions that can be better described by the audio of other samples.

Original: Bycombining these two terms, the model is able to both accurately localize visible sources and identifywhen no visible sources are present.
Revised: Bycombining these two terms, the model is able to both accurately localize visible sources and identifywhen no visible sources are present.

Original: Using the newly developed evaluation protocol, we show that,unlike all prior work, MoVSL does not overfit and thus can be trained without relying on early stopping(which requires human annotations).
Revised: Using the newly developed evaluation protocol, we show that,unlike all prior work, MoVSL does not overfit and thus can be trained without relying on early stopping(which requires human annotations).

Original: MoVSL achieves state-of-the-art performance on multiple datasetsand is more accurate at identifying samples without any visible sound sources.
Revised: MoVSL achieves state-of-the-art performance on multiple datasetsand is more accurate at identifying samples without any visible sound sources.

Original:  Related Work
Revised:  Related Work

Original: Audio-Visual Joint Learning.
Revised: Audio-Visual Joint Learning.

Original: Audio-visual joint learning has been explored in many works [8,9, 10, 11, 12, 1, 13, 14, 15, 16, 17, 18, 19, 20] to learn the temporal correlation between audioand visual information from videos.
Revised: Audio-visual joint learning has been explored in many works [8,9 , 10, 11, 12, 1, 13, 14, 15, 16, 17, 18, 19, 20] to learn the temporal correlation between audioand visual information from videos.

Original: Given a sequence of videos, the main idea is to close thedistance of audio and visual features from the same timestamp while pushing away those fromdifferent timestamps.
Revised: Given a sequence of videos, the main idea is to close thedistance of audio and visual features from the same timestamp while pushing away those fromdifferent timestamps.

Original: Such audio-visual temporal synchronization is beneficial to several tasks,such as audio separation [17, 21, 22, 14, 15, 23, 24, 25], audio-visual spatialization [13, 26, 27, 18],visual sound source localization [1, 2, 3, 4, 6, 28].
Revised: Such audio-visual temporal synchronization is beneficial to several tasks,such as audio separation [17, 21, 22, 14, 15, 23, 24, 25], audio-visual spatialization [13, 26, 27, 18],visual sound source localization [1, 2, 3, 4, 6, 28].

Original: In this work, we mainly focus on audio-visualsource localization on videos without annotations involved during training, which requires us to learnmeaningful audio-visual representations between a clip of audio and the frame centering around it.
Revised: In this work, we mainly focus on audio-visualsource localization on videos without annotations involved during training, which requires us to learnmeaningful audio-visual representations between a clip of audio and the frame centering around it.

Original: Audio-Visual Source Localization.
Revised: Audio-Visual Source Localization.

Original: Audio-visual source localization aims at predicting the locationof the sounding object in a video.
Revised: Audio-visual source localization aims at predicting the locationof the sounding object in a video.

Original: Early machine learning and deep models [29, 30, 31] were usedto capture the correspondences of audio and visual features.
Revised: Early machine learning and deep models [29, 30, 31] were usedto capture the correspondences of audio and visual features.

Original: Recently, contrast learning basedapproaches [32, 2, 6, 7] have been proposed to learn the audio-visual alignment between featuresfrom two stream networks.
Revised: Recently, contrast learning basedapproaches [32, 2, 6, 7] have been proposed to learn the audio-visual alignment between featuresfrom two stream networks.

Original: For instance, LVS [6] leveraged a contrastive loss with hard negativesmining to learn the audio-visual co-occurrence map discriminatively.
Revised: For instance, LVS [6] leveraged a contrastive loss with hard negativesmining to learn the audio-visual co-occurrence map discriminatively.

Original: Contrastive learning withhard positives was introduced in HardPos [7] to learn audio-visual alignment with negative samples.
Revised: Contrastive learning withhard positives was introduced in HardPos [7] to learn audio-visual alignment with negative samples.

Original: More recently, EZ-VSL [5] proposed the multiple instance contrastive learning without positiveand negative regions for training to match the most similar location in the corresponding image anddistinguish all locations in different images. 
Revised: More recently, EZ-VSL [5] proposed the multiple instance contrastive learning without positiveand negative regions for training to match the most similar location in the corresponding image anddistinguish all locations in different images. However, these baselines usually performed the model validation on the test set during trainingto select the best model as the final checkpoint, which caused the model to overfit. Furthermore,

Original: AudioEncoder
Revised: AudioEncoder

Original: P r o j e c t
Revised: P r o j e c t

Original: P oo l
Revised: P oo l

Original: C
Revised: C

Original: D r o p o u t
Revised: D r o p o u t

Original: VisualEncoder
Revised: VisualEncoder

Original: P r o j e c t
Revised: P r o j e c t

Original: C
Revised: C

Original: H
Revised: H

Original: W D r o p o u t
Revised: W D r o p o u t

Original: P r o j e c t img img2 C
Revised: P r o j e c t img img2 C

Original: H
Revised: H

Original: W
Revised: W

Original: P r o j e c t
Revised: P r o j e c t

Original: C aud aud
Revised: C aud aud

Original: Visual Branch Audio Branch
Revised: Visual Branch Audio Branch

Original: [Figure]
Revised: [Figure]

Original: Theaudio-visual correspondence of pixel-wise similarity between momentum branches and base branches is aligned by the cross-modal multiple instance contrastive learning objective. However, these baselines usually performed the model validation on the test set during training toselect the best model as the final checkpoint, which caused the model to overfit.
Revised: The is they only focus on localizing sound sources that are visible, ignoring non-visible or non-audiblesamples in real videos. Although understanding when sources are not visible is important to severalaudio-visual tasks, like in open-domain audio-visual source separation [33, 34], this issue still posessignificant challenges. DSOL [35] and IEr [36] propose a method to suppress localization of silentobjects, but they require the collection of a clean dataset containing a single source per video.

Original: Furthermore, theyonly focus on localizing sound sources that are visible, ignoring non-visible or non-audible samplesin real videos.
Revised: Incontrast, we tackle visual source localization without assuming knowledge of the number of sources.

Original: To address these challenges, we propose a novel momentum audio-visual matchingapproach for learning the audio-visual correspondence between momentum and base features.
Revised: Specifically, we propose a novel momentum audio-visual matching approach for learning the audiovisual correspondence between momentum and base features.

Original: Wealso explore both sounding and non-sounding frames in this work and propose new metrics forcomprehensive evaluation.
Revised: We also explore both sounding andnon-sounding frames in this work and propose new metrics for comprehensive evaluation.

Original:  Method
Revised:  Method

Original: We propose a new Momentum-based approach for Visual Sound Localization, denoted MoVSL , byminimizing the image-to-audio/audio-to-image pixel-wise similarity between momentum audio/visualfeatures and base visual/audio embeddings.
Revised: We propose a new Momentum-based approach for Visual Sound Localization, denoted MoVSL , byminimizing the image-to-audio/audio-to-image pixel-wise similarity between momentum audio/visualfeatures and base visual/audio embeddings.

Original: In section 3.1, we define the problem and notationsused in this paper and revisit multiple-instance contrastive learning proposed in EZ-VSL [5].
Revised: In section 3.1, we define the problem and notationsused in this paper and revisit multiple-instance contrastive learning proposed in EZ-VSL [5].

Original: Insection 3.2, we detail the proposed Momentum Audio-Visual Matching strategy.
Revised: Insection 3.2, we detail the proposed Momentum Audio-Visual Matching strategy.

Original: Finally, we detail thenew evaluation protocol including the extended test sets for Flickr-SoundNet/VGG-SS and evaluationmetrics in sections 3.3 and 3.3.
Revised: Finally, we detail thenew evaluation protocol including the extended test sets for Flickr-SoundNet/VGG-SS and evaluationmetrics in sections 3.3 and 3.3.

Original: 3.1 Preliminaries: Visual Source Localization by Localized Audio-Visual Matching [5]
Revised: 3.1 Preliminaries: Visual Source Localization by Localized Audio-Visual Matching [5]

Original: Given an audio-visual paired dataset D = { ( v i , a i ) : i = 1 , . . . , N } , visual source localization(VSL) aims at predicting the location of the sources present in sound a i within the visual framev i .
Revised: Given an audio-visual paired dataset D = { ( v i , a i ) : i = 1 , . . . , N } , visual source localization(VSL) aims at predicting the location of the sources present in sound a i within the visual framev i .

Original: This is accomplished by matching the sound a i to each region of v i in a learned commonspace.
Revised: This is accomplished by matching the sound a i to each region of v i in a learned commonspace.

Original: To this end, an audio encoder f a ( · ) is first used to extract global audio features, and avisual encoder f v ( · ) to extract local visual features at each ( x, y ) location.
Revised: To this end, an audio encoder f a ( · ) is first used to extract global audio features, and avisual encoder f v ( · ) to extract local visual features at each ( x, y ) location.

Original: Both features are thenmapped into a common space using two projection heads g a ( · ) and g v ( · ) for the audio and visualfeatures, respectively.
Revised: Both features are thenmapped into a common space using two projection heads g a ( · ) and g v ( · ) for the audio and visualfeatures, respectively.

Original: This results in a set of visual embeddings spanning all locations in an image
Revised: This results in a set of visual embeddings spanning all locations in an image

Original: V i
Revised: V i

Original: =
Revised: =

Original: { v xyi :
Revised: { v xyi :

Original: ∀ x, y } , ∀ i ∈ D , x ∈ [1 , W ] , y ∈ [1 , H ] and one global audio embedding a i .
Revised: ∀ x, y } , ∀ i ∈ D , x ∈ [1 , W ] , y ∈ [1 , H ] and one global audio embedding a i .

Original: All componentsof the model are then trained to align the audio embedding a i with the visualembeddings v xyi at the locations ( x, y ) where the sound sources are located.
Revised: All components are then trained to align the audio embedding a i with the visual embeddings v xyiat the locations ( x, y ) where the sound sources are located.

Original: If trained successfully,the model can be used for VSL by computing the audio-visual similarity at each location.
Revised: If trained successfully, the model can be used for VSL by computing the audio-visual similarity at each location.

Original: However,for unsupervised VSL, the ground-truth locations of sound sources are not known during training.
Revised: However, in the case ofunsupervised VSL, the ground-truth locations of sound sources are not known during training.

Original: To bypass the lack of annotations, prior work [5] proposed a multiple-instance contrastive learningobjective, in which the model is trained to match audio representations a i with at least one locationin the positive set V i , while pushing away from the visual features in all negative bags in the samemini-batch V j ∀ j ̸
Revised: To bypass the lack of annotations, prior work [5] proposed a multiple-instance contrastive learningobjective, in which the model is trained to match audio representations a i with at least one locationin the positive set V i , while pushing away from the visual features in all negative bags in the samemini-batch V j ∀ j ̸

Original: = i (at all locations).
Revised: = i (at all locations).

Original: In particular, the similarity between an audio feature a iand a bag of visual features V j
Revised: In particular, the similarity between an audio feature a iand a bag of visual features V j

Original: = { v xyj : ∀ x, y } is computed by max-pooling audio-visual cosinesimilarities across spatial locations sim ( a i , V j )
Revised: = { v xyj : ∀ x, y } is computed by max-pooling audio-visual cosinesimilarities across spatial locations sim ( a i , V j )

Original: =
Revised: =

Original: max xy s ( a i , v xyj ) .
Revised: max xy s ( a i , v xyj ) .

Original: The model is then trained tominimizing the  lossfunction
Revised: The model is then trained tominimize the average per sample loss

Original: [Equation]
Revised: [Equation]

Original: where τ is a temperature hyper-parameter.
Revised: where τ is a temperature hyper-parameter, and B is the batch-size.

Original: 3.2 Momentum Audio-visual Matching
Revised: 3.2 Momentum Audio-visual Matching

Original: We now introduce the proposed method, MoVSL , for visual sound localization.
Revised: We now introduce the proposed method, MoVSL , for visual sound localization.

Original: As we will show, mostprior work, including the current state-of-the-art EZ-VSL [5], suffer from two critical limitations: (1)models can easily overfit to the self-supervised objective; (2) models are not capable of identifyingnegative samples ( i.e ., samples with no visible sound sources).
Revised: As we will show, mostprior work, including the current state-of-the-art EZ-VSL [5], suffer from two critical limitations: (1)models can easily overfit to the self-supervised objective; (2) models are not capable of identifyingnegative samples ( i.e ., samples with no visible sound sources).

Original: The proposed method, illustrated in Figure 1, is designed to address these two limitations.
Revised: The proposed method, illustrated in Figure 1, is designed to address these two limitations.

Original: To preventoverfitting, we apply two regularization techniques: dropout [33] and (slow-moving) momentumencoders [34, 35].
Revised: To preventoverfitting, we apply two regularization techniques: dropout [37] and (slow-moving) momentumencoders [38, 39].

Original: To enhance the identification of negatives, we propose to conduct VSL bysimultaneously performing sounding region selection and instance discrimination.
Revised: To enhance the identification of negatives, we propose to conduct VSL bysimultaneously performing sounding region selection and instance discrimination.

Original: Dropout We apply dropout [33] on the output of the f v and f a encoders.
Revised: Dropout We apply dropout [37] on the output of the f v and f a encoders.

Original: Interestingly, very heavyvisual dropout is required to prevent overfitting (with a dropout probability as large as 0 . 9 ).
Revised: Interestingly, very heavyvisual dropout is required to prevent overfitting (with a dropout probability as large as 0 . 9 ).

Original: Sincemultiple-instance contrastive learning (Eq. 1) matches the audio to only one of the H × W locationsin the image, spurious alignment becomes more likely.
Revised: Sincemultiple-instance contrastive learning (Eq. 1) matches the audio to only one of the H × W locationsin the image, spurious alignment becomes more likely.

Original: Heavy visual dropout reduces the likelihoodof spurious audio-visual alignments, as visual features become more spatially redundant 

Revised: Heavy visual dropout reduces the likelihoodof spurious audio-visual alignments, as visual features become more spatially redundant

Original: Momentum encoders We use momentum audio-visual encoders [34, 35]
Revised: Momentum encoders We use momentum audio-visual encoders [38, 39]

Original: ˆ f a ( · ) , ˆ f v ( · ) to compute(slow-moving) representations ˆ a i and ˆ v xyi .
Revised: ˆ f a ( · ) , ˆ f v ( · ) to compute(slow-moving) representations ˆ a i and ˆ v xyi .

Original: These momentum features are then used as more stabletargets for audio-visual matching.
Revised: These momentum features are then used as more stabletargets for audio-visual matching.

Original: Following [34, 35], the momentum encoders are updated usingan exponential moving average of the corresponding online encoders with coefficient m .
Revised: Following [38, 39], the momentum encoders are updated usingan exponential moving average of the corresponding online encoders with coefficient m .

Original: 2 Sinceupdates to the audio-visual encoders are slowly incorporated into the momentum encoders, the targetrepresentations display a smoother behavior during the training process.
Revised: 3 Sinceupdates to the audio-visual encoders are slowly incorporated into the momentum encoders, the targetrepresentations display a smoother behavior during the training process.

Original: Enhancing negative detection with simultaneous localization and instance discrimination (SLID)
Revised: Enhancing negative detection with simultaneous localization and instance discrimination (SLID)

Original: Inspired by prior work on weakly-supervised detection [36], we factorize VSL into two probabilityterms.
Revised: Inspired by prior work on weakly-supervised detection [40], we factorize VSL into two probabilityterms.

Original: The first is a localization term P loc that seeks to distinguish within an image the regionsthat likely depict the sound source from the regions that likely do not.
Revised: The first is a localization term P loc that seeks to distinguish within an image the regionsthat likely depict the sound source from the regions that likely do not.

Original: The second is a cross-modalinstance discrimination term P xid that seeks to highlight regions of an image that are likely associatedwith the audio, while suppressing regions that would be better explained by sound sources from othersamples.
Revised: The second is a cross-modalinstance discrimination term P xid that seeks to highlight regions of an image that are likely associatedwith the audio, while suppressing regions that would be better explained by sound sources from othersamples.

Original: The key insight is that to enhance negative detection, VSL should only select the regionsthat are simultaneously highlighted in both terms.
Revised: The key insight is that to enhance negative detection, VSL should only select the regionsthat are simultaneously highlighted in both terms.

Original: While the first term is responsible for sound sourcelocalization, and the second prevents VSL to be overly confident when sound sources are not visiblein the image.
Revised: While the first term is responsible for sound sourcelocalization, and the second prevents VSL to be overly confident when sound sources are not visiblein the image.

Original: To enable different interactions, the localization and instance discrimination terms are defined as audiovisual similarities computed on different spaces.
Revised: To enable different interactions, the localization and instance discrimination terms are defined as audiovisual similarities computed on different spaces.

Original: In particular, let a loc
Revised: In particular, let a loc

Original: i = g loc a ( a i ) and a xid i =
Revised: i = g loc a ( a i ) and a xid i =

Original: g xid a ( a i )be the audio representation of a i in the two separate spaces, and V loc i = { g loc v ( v xyi ) :
Revised: g xid a ( a i )be the audio representation of a i in the two separate spaces, and V loc i = { g loc v ( v xyi ) :

Original: ∀ x, y } and
Revised: ∀ x, y } and

Original: V xid i
Revised: V xid i

Original: = { g xid v ( v xy i ) : ∀ x, y } be the two bags of visual features for v i .
Revised: = { g xid v ( v xyi ) : ∀ x, y } be the two bags of visual features for v i .

Original: Here, the various functions g ( ·  simple linear projections.
Revised: Here, the various functions g ( · ) denote simple linear projections.

Original: The localization term is computed as the softmax ρ xy ( · ) of localized audio-visual similarities over the spatial dimensions x, y
Revised: The localization term is computed as the softmax ρ xy ( · ) oflocalized audio-visual similarities over the spatial dimensions x

Original: [Equation]
Revised: [Equation]

Revised: y

Original: [Equation]
Revised: [Equation]

Original: and the instance discrimination term as the softmax ρ i ( · ) over instances
Revised: and the instance discrimination term as the softmax ρ i ( · ) over instances

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: To select the regions in which both P loc and P xid are active, the two terms are combined into a singleprediction map P xyij = 
Revised: To select the regions in which both P loc and P xid are active, the two terms are combined into a singleprediction map P xyij = P loc [Equation]

Original: P loc ( ai , v xyj ) · P xid ( a i , v xyj ) , and the model is trained to optimize
Revised: a

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: xyj )

Revised: · P xid

Original:  Finally, to combine SLID momentum encoders, audio and visual target representations are used whencomputing Eq.
Revised: [Equation] a i , v xyj ) , and the model is trained to optimize [Equation] Finally, to combine SLID momentum encoders, audio and visual target representations are used whencomputing Eq.

Original: 2 and Eq.
Revised: 2 and Eq.

Original: During inference, both localization and instance discrimination terms are used for source localization.
Revised: During inference, both localization and instance discrimination terms are used for source localization.

Original: We also observed that the (more stable) momentum features lead to improved visual source localization.
Revised: We also observed that the (more stable) momentum features lead to improved visual source localization.

Original: Thus, given an audio-visual pair ( v, a ) , the audio-visual localization map is computed as
Revised: Thus, given an audio-visual pair ( v, a ) , the audio-visual localization map is computed as

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Furthermore, this audio-visual localization map can be easily combined with object-guided localization proposed in [5] for improved performance.
Revised: Furthermore, this audio-visual localization map can be easily combined with object-guided localization proposed in [5] for improved performance.

Original: 3.3 Benchmarking Visual Source Localization
Revised: 3.3 Benchmarking Visual Source Localization

Original: We introduce a new evaluation protocol for VSL.
Revised: We introduce a new evaluation protocol for VSL.

Original: Extended Flickr-SoundNet/VGG-SS Non-visible sounds or frames with silent objects can oftenbe found in videos.
Revised: Extended Flickr-SoundNet/VGG-SS Non-visible sounds or frames with silent objects can oftenbe found in videos.

Original: To tackle these cases, we present a new evaluation protocol.
Revised: To tackle these cases, we present a new evaluation protocol.

Original: We extended
Revised: We extended

Original: VGG-SS/Flickr-SoundNet
Revised: VGG-SS/Flickr-SoundNet

Original: by merging clips with no sounding objects to the original VGG-SS/FlickrSoundNet test set.
Revised: by merging clips with no sounding objects to the original VGG-SS/FlickrSoundNet test set.

Original: Specifically, we download 1k YouTube videos from the large-scale audio-visualdataset, VGG-Sound [37], and manually select 5-second clips with non-audible frames and/ornon-visible sound sources.
Revised: Specifically, we download 1k YouTube videos from the large-scale audio-visualdataset, VGG-Sound [41], and manually select 5-second clips with non-audible frames and/ornon-visible sound sources.

Original: This resulted in 379 non-sounding (negative) samples.
Revised: This resulted in 379 non-sounding (negative) samples.

Original: Beyond the manually identified negative pairs, we further generate negative samples by pairing audioand videos that do not belong together.
Revised: Beyond the manually identified negative pairs, we further generate negative samples by pairing audioand videos that do not belong together.

Original: We control the difficulty of these negative pairs, by sampling25% of pairs from audio and video that are from the same class (hard negative), and 75% fromdifferent classes.
Revised: We control the difficulty of these negative pairs, by sampling25% of pairs from audio and video that are from the same class (hard negative), and 75% fromdifferent classes.

Original: We merge all negatives with the VGG-SS [6] and Flickr-SoundNet [1] test setwhich only contain positive pairs.
Revised: We merge all negatives with the VGG-SS [6] and Flickr-SoundNet [1] test setwhich only contain positive pairs.

Original: Table 1 shows the final statistics of the final test sets.
Revised: Table 1 shows the final statistics of the final test sets.

Original: We alsosplit sounding samples into four groups (Small, Medium, Large, and Huge), and show the shows thedistribution over sources of different sizes of the  ground-truth bounding box.
Revised: We also splittest samples into four groups (Small, Medium, Large, and Huge) according to the size of the soundsources, as measured by the image area (in pixels) occupied by the ground-truth bounding boxes.

Original: Evaluation metrics
Revised: Evaluation metrics

Original: Prior work [1] introduced consensus intersection over union (cIoU, denoted asu ) for evaluating predicted localization maps, by comparing them to a group of human annotations.
Revised: Prior work [1] introduced consensus intersection over union (cIoU, denoted asu ) for evaluating predicted localization maps, by comparing them to a group of human annotations.

Original: Given a set of predictions with cIoUs U = { u i } Ni =1 , each prediction is considered to be correct if itscIoU is above the cIoU threshold γ (usually set at γ  ).
Revised: Given a set of predictions with cIoUs U = { u i } Ni =1 , each prediction is considered to be correct ifits cIoU is above the cIoU threshold γ (usually set at γ = 0 . 5 ).

Original: Thus, the Precision over all positive samples( i.e . with visible sounding objects) is computed asPrecision =
Revised: Thus, the Precision over all positivesamples ( i.e . with visible sounding objects) is computed as

Original: [Equation]
Revised: [Equation]

Original: where T P = { i [Equation] u i 
Revised: where T P = { i | u i > γ } ,

Original: [Equation]
Revised: [Equation]

Original: γ } ,

Original: FP =

Original: { i

Original: [Equation]

Original: u [Equation] γ } (6) Existing work [2, 3, 4, 6, 7, 5] usually reports a metric (misguidingly) called “CIoU”, which is the
Revised: Existing work [2, 3, 4, 6, 7, 5] usually reports a metric (misguidingly) called “CIoU”, which is the

Original: Precision at γ = 0 . 5 .
Revised: Precision at γ = 0 . 5 .

Original: In this work, we also consider samples with no visible sounding sources.
Revised: In this work, we also consider samples with no visible sounding sources.

Original: Thus, we consider adataset D
Revised: Thus, we consider adataset D

Original: =
Revised: =

Original: { ( v i , a i , c i ) :
Revised: { ( v i , a i , c i ) :

Original: i = 1 , . .
Revised: i = 1 , . .

Original: . , N } , where c i is 1 for samples with visible sources (positivesamples) and 0 for samples with no visible sources (negative samples).
Revised: . , N } , where c i is 1 for samples with visible sources (positivesamples) and 0 for samples with no visible sources (negative samples).

Original: Furthermore, in addition
Revised: Furthermore, in addition

Original: [Table]
Revised: [Table]

Original: to the localization map, the model is required to predict whether the current pair contains a visiblesource or not.
Revised: to the localization map, the model is required to predict whether the current pair contains a visiblesource or not.

Original: This is accomplished by computing a confidence score d i for each sample i which wedefine as the maximum value in the predicted audio-visual similarity map.
Revised: This is accomplished by computing a confidence score d i for each sample i which wedefine as the maximum value in the predicted audio-visual similarity map.

Original: Taking the confidencescore into account, we formulate true positives as T P ′
Revised: Taking the confidencescore into account, we formulate true positives as T P ′

Original: = { i | c i = 1 , d i > δ, u i > γ } , falsepositives as FP ′
Revised: = { i | c i = 1 , d i > δ, u i > γ } , falsepositives as FP ′

Original: =
Revised: =

Original: { i | c i = 1 , d i > δ, u i ≤ γ } ∪ { i | c i = 0 , d i > δ } and false negatives as
Revised: { i | c i = 1 , d i > δ, u i ≤ γ } ∪ { i | c i = 0 , d i > δ } and false negatives as

Original: FN ′
Revised: FN ′

Original: = { i | c i = 1 , d i ≤ δ } .
Revised: = { i | c i = 1 , d i ≤ δ } .

Original: Then, to comprehensively evaluate both sounding and non-soundingsamples, we use Average Precision (AP) and maximum F1 (max-F1) score obtained by sweepingthe confidence threshold δ ( i.e . max δ F 1( δ ) ).
Revised: Then, to comprehensively evaluate both sounding and non-soundingsamples, we use Average Precision (AP) and maximum F1 (max-F1) score obtained by sweepingthe confidence threshold δ ( i.e . max δ F 1( δ ) ).

Original: The detailed formula can be seen in Appendix.
Revised: The detailed formula can be seen in Appendix.

Original: Inexperiments, we use γ = 0 .
Revised: Inexperiments, we use γ = 0 .

Original: 5 for evaluation.
Revised: 5 for evaluation.

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate the proposed MoVSL on unsupervised audio-visual source localization.
Revised: In this section, we evaluate the proposed MoVSL on unsupervised audio-visual source localization.

Original: Following previous work [6, 7, 5], we use the VGG-Sound dataset [37] for training without annotations.
Revised: Following previous work [6, 7, 5], we use the VGG-Sound dataset [41] for training without annotations.

Original: Then our trained models are tested on four datasets: Flickr-SoundNet testset [1], VGG-Sound
Revised: Then our trained models are tested on four datasets: Flickr-SoundNet testset [1], VGG-Sound

Original: Source [6], the proposed Extended Flickr-SoundNet and Extended VGG-SS.
Revised: Source [6], the proposed Extended Flickr-SoundNet and Extended VGG-SS.

Original: We report the quantitiveresults using Precision-Visible on datasets with only sounding samples.
Revised: We report the quantitiveresults using Precision-Visible on datasets with only sounding samples.

Original: AP and max-F1 are reportedfor benchmarks with both sounding and non-sounding samples.
Revised: AP and max-F1 are reportedfor benchmarks with both sounding and non-sounding samples.

Original: 4.1 Experimental setup
Revised: 4.1 Experimental setup

Original: Datasets.
Revised: Datasets.

Original: VGG-Sound [37] contains 200k videos of 309 sound categories from YouTube.
Revised: VGG-Sound [41] contains 200k videos of 309 sound categories from YouTube.

Original: Forunsupervised learning, we extract audio-visual pairs with a single frame and its corresponding audioof 20 seconds long.
Revised: Weextract audio-visual pairs with a single frame and its corresponding audio of 20 seconds long.

Original: Following commonly-used settings [6, 7, 5], we use subset of 144k pairsfor training.
Revised: Following commonly-used settings [6, 7, 5], we use subset of 144k pairs for training.

Original: We test our model on two types of datasets: 1) with only sounding samples: FlickrSoundNet testset [1] and VGG-Sound Source [6]; 2) with both sounding and non-sounding samples:the proposed Extended Flickr-SoundNet and Extended VGG-SS.
Revised: We test ourmodel on two types of datasets: 1) with only sounding samples: Flickr-SoundNet testset [1] and VGG-Sound Source [6]; 2) with both sounding and non-sounding samples: the proposed Extended Flickr-SoundNet and Extended VGG-SS.

Original: The Flickr-SoundNet includesonly 250 audio-visual pairs of sounding objects, while VGG-Sound Source contains 5158 pairs ofsounding objects from 220 categories.
Revised: The Flickr-SoundNet includes only 250 audio-visual pairsof sounding objects, while VGG-Sound Source contains 5158 pairs of sounding objects from 220categories.

Original: Extended Flickr-SoundNet totally includes 500 samples, 250sounding and 250 non-sounding.
Revised: Extended Flickr-SoundNet totally includes 500 samples, 250 sounding and 250 nonsounding.

Original: Extended VGG-SS spans 5158 pairs of sounding objects and 5158pairs of non-sounding objects, 10,316 in total.
Revised: Extended VGG-SS spans 5158 pairs of sounding objects and 5158 pairs of non-soundingobjects, 10,316 in total.

Original: Implementation.
Revised: Implementation.

Original: For audio and visual encoders, we use ResNet-18 [38] as the backbone, andinitialize the visual encoder with ImageNet [39] pre-trained weights, as previous work [6, 7, 5] did.
Revised: For audio and visual encoders, we use ResNet-18 [42] as the backbone, andinitialize the visual encoder with ImageNet [43] pre-trained weights, as previous work [6, 7, 5] did.

Original: The dimension of audio-visual features in the projected share space is 512.
Revised: The dimension of audio-visual features in the projected share space is 512.

Original: For visual input, wefirst resize frames of 224 × 224 to 246 along the shortest edge, and then apply random cropping withrandom horizontal flipping on them at training time, We do not use any data augmentation and useresized frames of 224 × 224 for inference.
Revised: For visual input, wefirst resize frames of 224 × 224 to 246 along the shortest edge, and then apply random cropping withrandom horizontal flipping on them at training time, We do not use any data augmentation and useresized frames of 224 × 224 for inference.

Original: For audio encoder, we extract the log spectrograms with257 frequency bands over 300 timesteps from 3s of audio as input.
Revised: For audio encoder, we extract the log spectrograms with257 frequency bands over 300 timesteps from 3s of audio at 22050 kHz. The underlying short-term

Revised: Fourier transform is computed on approximately 25ms windows with a step size of 10ms.

Original: Following the prior work [5],we train the model with a batch size of 128 on 2  GPUs for 20 epochs.
Revised: Followingthe prior work [5], we train the model with a batch size of 128 on 2 NVIDIA RTX 3090 GPUs for 20epochs.

Original: The Adam [40] optimizerwith β 1 =0.9, β 2 =0.999 is applied with an initial learning rate of 1e-4 and weight decay of 1e-4.
Revised: The Adam [44] optimizer with β 1 =0.9, β 2 =0.999 is applied with an initial learning rate of1e-4 and weight decay of 1e-4.

Original: Theprobability of dropout for visual branch is 0.9.
Revised: The probability of dropout for visual branch is 0.9.

Original: The momentum coefficients m a =0.999, m v =0.999.
Revised: The momentumcoefficients m a =0.999, m v =0.999.

Revised: Our implementation is based on PyTorch [45] deep learning tool.

Original: 4.2 Comparison of overfitting to prior work
Revised: 4.2 Comparison of overfitting to prior work

Original: To demonstrate the effectiveness of our model in alleviating the overfitting issue, we compare bothbest and latest checkpoint with prior work [1, 2, 3, 4, 6, 7, 5] using the common metric PrecisionVisible, that is, “CIoU”.
Revised: To demonstrate the effectiveness of our model in alleviating the overfitting issue, we compare bothbest and latest checkpoint with prior work [1, 2, 3, 4, 6, 7, 5] using the common metric Precision,that is, “CIoU”.

Original: Table 2 shows the comparison results on Flickr SoundNet and VGG-SStestsets.
Revised: Table 2 shows the comparison results on Flickr SoundNet and VGG-SS testsets.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We can observe that our MoVSL superiorly outperforms previous work by using the best checkpointfor comparison, demonstrating the state-of-the-art results of our approach in this setting.
Revised: We can observe that our MoVSL superiorly outperforms previous work by using the best checkpointfor comparison, demonstrating the state-of-the-art results of our approach in this setting.

Original: Withoutthe OGL (object-guided localization) proposed in EZ-VSL [5], we achieve the performance gainsof 3.60% and 2.94% on the Flickr and VGG-SS tests sets.
Revised: Withoutthe OGL (object-guided localization) proposed in EZ-VSL [5], we achieve the performance gainsof 3.60% and 2.94% on the Flickr and VGG-SS tests sets.

Original: When the OGL is involved at inferencetime, our MoVSL further improves the SOTA performance by 2.46% and 0.82%.
Revised: When the OGL is involved at inferencetime, our MoVSL further improves the SOTA performance by 2.46% and 0.82%.

Original: Significant gainscan be observed in the setting of using the latest checkpoint for comparison.
Revised: Significant gainscan be observed in the setting of using the latest checkpoint for comparison.

Original: On the Flickr testset, our framework outperforms the state-of-the-art work by large margins, 17.20% and 13.20% forw/o OGL and w OGL.
Revised: On the Flickr testset, our framework outperforms the state-of-the-art work by large margins, 17.20% and 13.20% forw/o OGL and w OGL.

Original: The performance gains of 6.21% and 1.94% on the challenging VGG-SSdataset are achieved, which shows the importance of our momentum audio-visual matching strategyin preventing the model from overfitting.
Revised: The performance gains of 6.21% and 1.94% on the challenging VGG-SSdataset are achieved, which shows the importance of our momentum audio-visual matching strategyin preventing the model from overfitting.

Original: 4.3 Benchmarking on Extended Flickr-SoundNet and VGG
Revised: 4.3 Benchmarking on Extended Flickr-SoundNet and VGG-SS

Original: -SS

Original: With the proposed metrics (AP, max-F1, Precision-Visible) in Sec 3.3, we benchmark state-of-the-artwork [6, 5] and our model on two datasets with both sounding and non-sounding samples: Extended
Revised: With the proposed metrics (AP, max-F1, Precision-Visible) in Sec 3.3, we benchmark state-of-the-artwork [6, 5] and our model on two datasets with both sounding and non-sounding samples: Extended

Original: Flickr-SoundNet, and Extended VGG-SS.
Revised: Flickr-SoundNet, and Extended VGG-SS.

Original: Note that the Center Prior is to generate a random Gaussianmap in the middle of frames for predictions.
Revised: Note that the Center Prior is to generate a random Gaussianmap in the middle of frames for predictions.

Original: The experimental results are reported in Table 3.
Revised: The experimental results are reported in Table 3.

Original: As canbe seen, the proposed MoVSL establishes new state-of-the-art performance in terms of all metrics.
Revised: As canbe seen, the proposed MoVSL establishes new state-of-the-art performance in terms of all metrics.

Original: On the Extended Flickr-SoundNet with 250 additional non-sounding samples, we achieve 51.63% APand 59.10% max-F1, but a high precision of 83.60% on visible objects.
Revised: On the Extended Flickr-SoundNet with 250 additional non-sounding samples, we achieve 51.63% APand 59.10% max-F1, but a high precision of 83.60% on visible objects.

Original: This shows the importanceof taking false negatives into account for non-sounding samples.
Revised: This shows the importanceof taking false negatives into account for non-sounding samples.

Original: Due to the large amount (5158)ofnegatives in the Extended VGG-SS, AP and max-F1 are decreased to 32.95% and 40.00%, indicatingthe advantage ofthe proposed AP and max-F1 in evaluating non-sounding objects over the original
Revised: Due to the large amount (5158) of

Original: [Table] [Table]
Revised: [Table] [Table]

Original: metric, Precision-Visible. In the meanwhile, quantitative results of AP and max-F1 are positivelycorrelated to those of Precision-Visible, which further implies the rationality of the proposed metricsin scoring both sounding and non-sounding samples.
Revised: negatives in the Extended VGG-SS, AP and max-F1 are decreased to 32.95% and 40.00%, indicatingthe advantage of the proposed AP and max-F1 in evaluating non-sounding objects over the originalmetric, Precision-Visible. In the meanwhile, quantitative results of AP and max-F1 are positivelycorrelated to those of Precision-Visible, which further implies the rationality of the proposed metricsin scoring both sounding and non-sounding samples.

Original: 4.4 Experimental analysis
Revised: 4.4 Experimental analysis

Original: In this section, we conduct an extensive benchmarking study of prior work.
Revised: In this section, we conduct an extensive benchmarking study of prior work.

Original: We train severalmethods [1, 4, 2, 3, 6, 7, 5] to evaluate them using the newly proposed evaluation protocol.
Revised: We train severalmethods [1, 4, 2, 3, 6, 7, 5] to evaluate them using the newly proposed evaluation protocol.

Original: Whenimplementing prior work, we used authors implementation with default parameters.
Revised: Whenimplementing prior work, we used authors implementation with default parameters.

Original: When notavailable, we carefully followed the paper description.
Revised: When notavailable, we carefully followed the paper description.

Original: We report both our reproduced results, and theresults reported in the original paper, whenever possible.
Revised: We report both our reproduced results, and theresults reported in the original paper, whenever possible.

Original: To demonstrate the benefits of MoVSL , we compare our method to prior work using the new evaluationprotocol.
Revised: To demonstrate the benefits of MoVSL , we compare our method to prior work using the new evaluationprotocol.

Original: All models were trained on the VGG-Sound 144k subset and the latest checkpoints weretested on the challenging Extended VGG-SS with both sounding and non-sounding objects.
Revised: All models were trained on the VGG-Sound 144k subset and the latest checkpoints weretested on the challenging Extended VGG-SS with both sounding and non-sounding objects.

Original: SLID localization strategy and regularization.
Revised: SLID localization strategy and regularization.

Original: In order to demonstrate the effectiveness of MoVSL ,we ablated the use of momentum encoders and simultaneous localization and instance discrimination(SLID) strategies during training.
Revised: In order to demonstrate the effectiveness of MoVSL ,we ablated the use of momentum encoders and simultaneous localization and instance discrimination(SLID) strategies during training.

Original: Table 4 reports the experimental results.
Revised: Table 4 reports the experimental results.

Original: Adding momentumencoders to the baseline improves AP and max-F1 by large margins, 3.64% and 4.70%.
Revised: Adding momentumencoders to the baseline improves AP and max-F1 by large margins, 3.64% and 4.70%.

Original: This showsthe effectiveness of the proposed momentum audio-visual matching approach in learning the crossmodal alignment between momentum and base representations.
Revised: This showsthe effectiveness of the proposed momentum audio-visual matching approach in learning the crossmodal alignment between momentum and base representations.

Original: When SLID is used, we furtherincrease all metrics by significant margins, 4.76% AP, 4.40% max-F1, and 5.04% Precision.
Revised: When SLID is used, we furtherincrease all metrics by significant margins, 4.76% AP, 4.40% max-F1, and 5.04% Precision.

Original: Furthermore, we ablated audio-visual matching strategies at inference time for the proposed MoVSLtrained with the SLID objective.
Revised: Furthermore, we ablated audio-visual matching strategies at inference time for the proposed MoVSLtrained with the SLID objective.

Original: We evaluate the localization ability of each SLID branch separately,i.e .
Revised: We evaluate the localization ability of each SLID branch separately,i.e .

Original: using the similarities between the global audio feature ˆ a loc ( ˆ a xid ) and the bag of localized visualfeatures ˆ v loc xy ( ˆ v xid xy ).
Revised: using the similarities between the global audio feature ˆ a loc ( ˆ a xid ) and the bag of localized visualfeatures ˆ v loc xy ( ˆ v xid xy ).

Original: We compare these with the localization performance using both SLID branches asin Equation 5.
Revised: We compare these with the localization performance using both SLID branches asin Equation 5.

Original: Results are shown in Table 4.
Revised: Results are shown in Table 4.

Original: Both alternative approaches achieve worse results thanour MoVSL which uses both SLID branches for localization.
Revised: Both alternative approaches achieve worse results thanour MoVSL which uses both SLID branches for localization.

Original: This further indicates the effectiveness ofthe proposed approach in capturing the cross-modal pixel-wise correspondence between momentumand base features.
Revised: This further indicates the effectiveness ofthe proposed approach in capturing the cross-modal pixel-wise correspondence between momentumand base features.

Original: We also ablate the regularization techniques (weight decay, visual dropout,momentum) in Table 5.
Revised: We also ablate the regularization techniques (weight decay, visual dropout,momentum) in Table 5.

Original: It can be seen that dropout in the visual feature is very important to preventoverfitting.
Revised: It can be seen that dropout in the visual feature is very important to preventoverfitting.

Original: The optimal momentum is 0.999 for both visual and audio models.
Revised: The optimal momentum is 0.999 for both visual and audio models.

Original: Effect of Precision, F1 score, and AP.
Revised: Effect of Precision, F1 score, and AP.

Original: To explore the effect of various prediction sizes on the
Revised: To explore the effect of various prediction sizes on the

Original: Precision of visible objects within each subset (Small, Medium, Large, Huge), we vary the size from10% to 100% and compare the performance using the CPL, AVL, OGL, AV+OGL for inference.
Revised: Precision of visible objects within each subset (Small, Medium, Large, Huge), we vary the size from10% to 100% and compare the performance using the CPL, AVL, OGL, AV+OGL for inference.

Original: Theresults are shown in Figure 2.
Revised: Theresults are shown in Figure 2.

Original: All methods achieve the peak of Precision for medium, large, and hugeobjects at 10%, 30%, 70%.
Revised: All methods achieve the peak of Precision for medium, large, and hugeobjects at 10%, 30%, 70%.

Original: Due to the difficulty of identifying small objects in the whole frame,
Revised: Due to the difficulty of identifying small objects in the whole frame,

Original: Precision for different thresholds is always 0.
Revised: Precision for different thresholds is always 0.

Original: This highlights one important limitation of currentunsupervised visual sound source localization methods.
Revised: This highlights one important limitation of currentvisual sound source localization methods.

Original: We also evaluated the F1 score and AP withdifferent confidence and IoU thresholds.
Revised: We also evaluated the F1 score and AP with differentconfidence and IoU thresholds.

Original: Figure 3 shows the comparison results of the proposed MoVSL and previous work.
Revised: Figure 3 shows the comparison results of the proposed MoVSL andprevious work.

Original: When the confidence is 0.5, our method achieves the highest F1 scoreagainst existing methods.
Revised: When the confidence is 0.5, our method achieves the highest F1 score against existingmethods.

Revised: We highlight that MoVSL and EZVSL are the only approaches whose F1 scores increase withhigher confidence thresholds.

Revised: This shows that unlike most of other prior work, MoVSL can effectivelyidentify negative pairs, i.e . reduce false positive rate without sacrificing true positive rate.

Original: Our method also outperforms all prior works in terms of AP using all IoUthresholds.
Revised: Our methodalso outperforms all prior works in terms of AP using all IoU thresholds.

Original: 4.5 Limitations
Revised: 4.5 Limitations

Original: Although we achieve superior precision on large and huge objects in the image, the performance ofsmall and medium objects is not satisfactory, as shown in Figure 2.
Revised: Although we achieve superior precision on large and huge objects in the image, the performance ofsmall and medium objects is not satisfactory, as shown in Figure 2.

Original: When it comes to small objects,the proposed method performs badly as the prior work did.
Revised: When it comes to small objects,the proposed method performs badly as the prior work did.

Original: The possible reason is that all methodsuse the lightweight ResNet-18 as the visual backbone to extract features, which limits the qualityof visual representations for audio-visual matching.
Revised: The possible reason is that all methodsuse the lightweight ResNet-18 as the visual backbone to extract features, which limits the qualityof visual representations for audio-visual matching.

Original: In the meanwhile, we notice that the predictionmap is generated by the bilinear interpolation of an audio-visual response map with a low resolution.
Revised: In the meanwhile, we notice that the predictionmap is generated by the bilinear interpolation of an audio-visual response map with a low resolution.

Original: Increasing the resolution of such a map could be beneficial to small objects in the raw image withhigh resolution.
Revised: Increasing the resolution of such a map could be beneficial to small objects in the raw image withhigh resolution.

Original: Therefore, applying a strong backbone would be the future direction for improvingthe precision of small objects.
Revised: Therefore, applying a strong backbone would be the future direction for improvingthe precision of small objects.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we explore two main issues in common early stopping settings and the visible soundsource assumption for unsupervised audio-visual source localization.
Revised: In this work, we explore two main issues in common early stopping settings and the visible soundsource assumption for unsupervised audio-visual source localization.

Original: To capture the model’s performance on samples with no visible sound sources, we extend the test sets of popular benchmarks,
Revised: To capture the model’s performance on samples with no visible sound sources , we extend the test sets of popular benchmarks

Original: Flickr SoundNet and VGG-Sound Source, to cover negative samples, and use average precisionand F1 scores as metrics to balance precision and recall.
Revised: Flickr SoundNet and VGG-Sound Source, to cover negative samples, and use average precision and F scores as metrics to balance precision and recall

Original: Extensive evaluations of prior work areperformed under the new proposed evaluation protocol.
Revised: Extensive evaluations of prior work areperformed under the new proposed evaluation protocol.

Original: We observe that most previous work suffersfrom significant overfitting problems, and thus relies heavily on early stopping for best results.
Revised: We observe that most previous work suffersfrom significant overfitting problems, and thus relies heavily on early stopping for best results.

Original: Toeliminate this, we propose a novel Momentum audio-visual matching approach for Visual Sound
Revised: Toeliminate this, we propose a novel Momentum audio -visual matching approach for Visual Sound

Original: Localization by applying momentum features as targets of base embeddings for each modality, whichwe denote MoVSL.
Revised: Localization by applying momentum features as targets of base embeddings for each modality , whichwe denote MoVSL

Original: Thorough experiments demonstrate the effectiveness of the proposed approachin combating overfitting, and the state-of-the- art performance on two popular benchmarks
Revised: Thorough experiments demonstrate the effectiveness of the proposed approach in combating overfitting, and the state-of-the-art performance on two popular benchmarks

Original: , Flickr
Revised: , Flickr

Original: SoundNet and VGG-Sound Source.
Revised: SoundNet and VGG-Sound Source.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] See Section 1.
Revised: [Yes] See Section 1.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.5.
Revised: [Yes] See Section 4.5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection ?? .(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] SeeSection G in the appendix.(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] See Supplementary material.
Revised: [Yes] See Supplementary material.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes] See thesupplemental material.(b)
Revised: [Yes] See the supplemental material.

Original:  Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 4.1.(c)
Revised: [Yes] See Section 4.1.(c)

Original: Did you report error bars (e
Revised: Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [No] Training and test sets are too large to run each experiment multiple times.
Revised: [No] Training and test sets are too large to run each experimentmultiple times.

Original: However, given their sizes, error bars are expected to be small.(d)
Revised: However, given their sizes, error bars are expected to be small.(d)

Original: Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4.1.
Revised: [Yes] See Section 4.1.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4.1.
Revised: [Yes] See Section 4.1.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [No]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original:  (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: Appendix

Revised: We start by releasing code and demo, detailing the baselines and metrics used for benchmarking visualsound localization (Precision, F1 score, and Average Precision).

Revised: We demonstrate the effectiveness ofthe proposed MoVSL in other commonly used training datasets (Flickr 10k) and open set visual soundsource localization.

Revised: Finally, we compare the qualitative results of our approach with existing methodsin terms of various sizes of sounding objects.

Revised: A Code & Demo

Revised: Code is available at:https://anonymous.4open.science/r/MoVSL-1021 .

Revised: Demo is available at:https://drive.google.com/file/d/1fz5d8inEUXA2WTVqskZj5FkMm9TBRwY8/view

Revised: B Baselines

Revised: In the field of visual sound source localization, we comprehensively consider existing approaches forbenchmarking as follows:

Revised: • Attention 10k [1] (2018’CVPR): the first attention-based work with a two-stream architecturewith each modality for unsupervised sound source localization in an image, and this approachwas extended to semi-supervised settings with ground-truth maps in 5k Flickr set; (code:https://github.com/ardasnck/learning_to_localize_sound_source )• DMC [2] (2019’CVPR): a multi-modal clustering network for learning audiovisual correspondences by using convolutional maps with each modality in different shared spaces,and it was only used for the unsupervised setting; (code: https://github.com/DTaoo/Simplified_DMC , MIT license is available)• CoarsetoFine [4] (2020’ECCV): a two-stage pipeline that aligned the cross-modalfeatures in a coarse-to-fine way, and it was used on both unsupervised and semisupervised with video class annotations; (code: https://

Revised: github

Revised: .com/shvdiwnkozbw/

Revised: Multi-Source-Sound-Localization )

Revised: • LVS [6] (2021’CVPR): a contrastive learning framework with hard negatives min-

Revised: ing to extract the audio-visual co-occurrence map discriminatively, and it was

Revised: only used for the unsupervised setting; (code: https://github.com/hche11/

Revised: Localizing-Visual-Sounds-the-Hard-Way , Apache license is available)

Revised: • HardPos [7] (2022’ICASSP): an improved work based on LVS by adding hard positives for

Revised: aligning audio-visual matching semantics from negative pairs, and it was only used for the

Revised: unsupervised setting;

Revised: • EZ-VSL

Revised: [5] (2022’ECCV): a strong baseline that proposed the multiple instance contrastive

Revised: learning to align locations with high similarity in the image and dispel all locations in

Revised: different images, and it was only used for the unsupervised setting; (code: https://

Revised: github.com/stoneMo/EZ-VSL , Apache license is available)

Revised: • MoVSL (ours): the first approach with momentum audio-visual matching to alleviate over-

Revised: fitting, and our method is used for both unsupervised and semi-supervised settings with

Revised: ground-truth maps;

Revised: • DSOL [35] (2020’NeurIPS): a two-stage training baseline to deal with silence in category-

Revised: aware sound source localization, where the audio event class is involved in the training

Revised: stage 1 and an evaluation metric (NSA) is proposed specifically to measure the abil-

Revised: ity of a model to ignore visible silent objects; (code: https://github.com/DTaoo/

Revised: Discriminative-Sounding-Objects-Localization , MIT license is available)

Revised: C Precision, F1 Score, Average Precision

Revised: Consider a set of samples D

Revised: = { ( v i , a i ) :

Revised: i = 1 , . .

Revised: . , N } , ground-truth maps G = {G i } Ni =1and predicted maps S

Revised: = {S i } Ni =1 .

Revised: In map based localization, each prediction S i is obtainedby first computing pixel-wise localization scores S i ∈

Revised: R H × W , and then applying a thresold α ,

Revised: S i ( α ) =

Revised: { ( x, y ) | S i ( x, y ) > α } .

Revised: To evaluate each prediction S i , prior work [1] relies on intersectionover union between S i and G i .

Revised: IoU is calculated as

Revised: [Equation]

Revised: (7) where G = { ( x, y ) | g xy > 0 } denotes the the set of pixels that represent sounding objects, andg xy ∈ [0 , 1] the ground-truth evidence that a sounding objects lies at location ( x, y ) .

Revised: Since in somedatasets ground-truth is collected from multiple annotators, consensus IoU (cIoU) is used instead.

Revised: Refer to [1] for details on how multiple annotations are merged to compute g xy .

Revised: Since we’re interested in assessing the model’s performance both when sounding objects are presentor not, we allow the model to make no predictions.

Revised: To do this, in addition to the localization predictionmap S i , the model is asked to output a confidence score

Revised: c i .

Revised: If the confidence score c i is too low (belowa threshold δ ), the model predicts an empty set ( i.e ., the original S i is considered invalid).

Revised: Predictions

Revised: S i are considered correct if i are above a pre-specified threshold β .

Revised: Under these definitions, truepositives, false positives and false negatives are computed as

Revised: [Equation]

Revised: Precision To evaluate the localization accuracy of our models (given that sound sources exist), weuse precision at a threshold of β

Revised: [Equation]

Revised: It should be noted, that most existing work [2, 3, 4, 6, 7, 5] already report Precision at β = 0 .

Revised: 5 underthe name of “CIoU”.

Revised: We thus use this metric to compare to results reported in the original papers.

Revised: Inour work, we also use a threshold of β = 0 .

Revised: 5 for most experiments, and conduct an evaluation athigher values β to assess which methods can achieve higher quality predictions.

Revised: While precision provides a good metric to evaluate how accurate predictions are for samples withsounding objects, it does not assess how accurately models can ignore samples with NO soundingobjects.

Revised: Thus, to comprehensively evaluate both sounding and non-sounding samples, we alsoevaluate our models using F1 score and average precision (AP).

Revised: F1 score balances precision and recall,

Revised: [Equation]

Revised: where Recall ( β, δ ) =

Revised: |T P

Revised: [Equation]

Revised: β,

Revised: [Equation]

Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: However, F1 ( β, δ ) depends on how strict the confidence threshold δ is set (the highest δ is set, themore samples are predicted as non-sounding).

Revised: To find the optimal balance, we compute F1 ( β, δ ) forall values of δ and report the max-F1 score

Revised: [Equation]

Revised: Average precision (AP) is another metric often used in object detection.

Revised: To compute AP, we closelyfollow [46].

Revised: The only difference is that, when computing the Precision-Recall curve, we do notperform 11 point interpolation.

Revised: We compute the full curve (without interpolation).

Revised: Refer to [46] fordetails on AP computation.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Attention10k

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: D Results on Flickr-10k

Revised: In order to demonstrate the effectiveness of the proposed MoVSL in mitigating overfitting issueson unsupervised visual sound source localization, we train the model on a small dataset with 10ksamples, Flickr-10k [1], following existing methods [1, 2, 6, 5].

Revised: The comparison results of usingbest and latest checkpoints on Flickr SoundNet testset are reported in Table 6.

Revised: As can be seen, our

Revised: MoVSL achieves the state-of-the-art results compared to existing methods in both settings.

Revised: Using thebest checkpoint outperforms the current best results without OGL by 11.46%, 13.80%, and 19.35%in terms of AP, max-F1, and Precision.

Revised: Adding the OGL to the proposed approach also achievessignificant improvements.

Revised: More important, there is no gap between the latest and best checkpoints forthe proposed MoVSL .

Revised: These excellent results demonstrate the effectiveness of our method in mitigatingoverfitting issues in unsupervised visual sound source localization.

Revised: E Open Set Results

Revised: To evaluate the generalization of our model beyond sound sources heard during training, we followprevious work [6, 5] and train the model on 70k data with 110 heard categories in VGG-Sounddataset [41].

Revised: Table 7 reports the comparison results on heard and unheard 110 classes.

Revised: We achievesignificant improvements against previous methods.

Revised: For instance, using the latest checkpoint without

Revised: OGL for testing heard 110 classes increases the baseline by 5.71%, 5.90%, and 5.22% in terms of

Revised: [Figure]

Revised: AP, max-F1, and Precision.

Revised: When it comes to unheard 110 classes, the proposed MoVSL without

Revised: OGL outperforms the current state-of-the-art approach by 4.10%, 3.80%, and 3.42% in terms of AP,max-F1, and Precision.

Revised: New state-of-the-art results are achieved in all settings, demonstrating thegeneralization of our approach to unheard sounding categories in the training set.

Revised: F Qualitative Results

Revised: In order to qualitatively demonstrate the effectiveness of the proposed MoVSL , we compare visualizeattention maps from existing work and our model on various sizes of sounding objects in VGG-SS testset.

Revised: The qualitative results of of Attention10k [1], DMC [2], LVS [6], EZ-VSL [5], and the proposed

Revised: MoVSL on huge/large/medium/small sounding objects are shown in Figure 4, 5, 6, and 7.

Revised: Note thatresults of our MoVSL are shown on the last column.

Revised: We can observe that the proposed MoVSL achievesdecent localization maps compared to previous approaches in all settings, such as the barking dog onrow 3 in Figure 4, the sounding cello on row 1 in Figure 5, and the crying baby on row 1 in Figure 6.

Revised: For small objects in Figure 7, we also predict better maps for sounding objects than existing work.

Revised: While our predictions seem to focus more on the actual sources, they are very large compared to theactual objects.

Revised: G Broader Impact

Revised: This paper seeks to establish a more balanced evaluation protocol for visual sound source localization,that considers both cases with sounding and non-sounding objects.

Revised: We hope the proposed metricswill lead to more balanced localization algorithms in the future.

Revised: Improving sound source localizationmethods can lead to interesting applications, for example, interfacing with blind or low-vision individuals as they navigate the world.

Revised: However, our work still relies on data collected from internet sources,and thus is likely to hold biases that have not been identified.

Revised: While we believe current datasets andevaluation protocols are valuable for the development of sound source localization procedure, bettercuration of the datasets against nefarious bias should be conducted before deployment in real worldsettings.

Revised: Advances in audio-visual localization can also be leveraged in surveillance applications,which can potentially have a negative societal impact.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]