Original: Tackling Overfitting and Silence in Weakly-Supervised Audio-Visual Source Localization
Revised: Tackling Overfitting and Silence in Unsupervised Audio-Visual Source Localization

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Audio-visual source localization is a challenging task that aims to predict the location of visual sound sources in a video.
Revised: Audio-visual source localization is a challenging task that aims to predict the location of visual sound sources in a video.

Original: Since collecting ground-truth annotationsof sounding objects can be costly, a plethora of weakly-supervised localizationmethods that can learn from datasets with no bounding-box annotations have beenproposed in recent years, by leveraging the natural co-occurrence of audio andvisual signals.
Revised: Since collecting ground-truth annotationsof sounding objects can be costly, a plethora of unsupervised localization methodshave been proposed in recent years.

Original: Despite significant interest, popular evaluation protocols have twomajor flaws.
Revised: Despite significant interest, current evaluationprotocols have two major flaws.

Original: First, they allow for the use of a fully annotated dataset to performearly stopping, thus significantly increasing the annotation effort required for training.
Revised: First, they allow for the use of a fully annotateddataset to perform early stopping, thus defeating the purpose of unsupervisedsource localization.

Original: Second, current evaluation metrics assume the presence of sound sources atall times.
Revised: Second, current evaluation metrics assume the presence ofsound sources at all times.

Original: This is of course an unrealistic assumption, and thus better metrics arenecessary to capture the model’s performance on (negative) samples with no visiblesound sources.
Revised: This is of course an unrealistic assumption, and thusbetter metrics are necessary to capture the model’s performance on (negative)samples with no visible sound sources.

Original: To accomplish this, we extend the test set of popular benchmarks, Flickr SoundNet and VGG-Sound Sources, in order to include negative samples,and measure performance using metrics that balance precision and recall.
Revised: To accomplish this, we extend the test set ofpopular benchmarks, Flickr SoundNet and VGG-Sound Sources, in order to includenegative samples, and measure performance using metrics that balance precisionand recall.

Original: Usingthe new protocol, we conducted an extensive evaluation of prior methods, andfound that most prior works are not capable of identifying negatives and sufferfrom significant overfitting problems (rely heavily on early stopping for best results).
Revised: Using the new protocol, we conducted an extensive evaluation of priormethods, and found that most prior works are not capable of identifying negativesand suffer from significant overfitting problems (rely heavily on early stopping forbest results).

Original: We also propose a new approach for visual sound source localization thataddresses both these problems.
Revised: We also propose a new approach for visual sound source localizationthat addresses both these problems.

Original: In particular, we found that, through extremevisual dropout and the use of momentum encoders, the proposed approach combatsoverfitting effectively, and establishes a new state-of-the-art performance on both
Revised: In particular, we found that, through extremevisual dropout and the use of momentum encoders, the proposed approach combatsoverfitting effectively, and establishes a new state-of-the-art performance on both

Original: Flickr SoundNet and VGG-Sound Source.
Revised: Flickr SoundNet and VGG-Sound Source.

Original:  Introduction
Revised:  Introduction

Original: From the meow of a cat to the roar of a lion to the revving of a motorcycle, humans have evolved tolocalize sources of sound in their environment to increase chances of survival.
Revised: From the meow of a cat to the roar of a lion to the revving of a motorcycle, humans have evolved tolocalize sources of sound in their environment to increase chances of survival.

Original: This ability relies inpart on the uniqueness of sounds made by different entities – distinct sounds can be associated withtheir sources, which in turn can be visually localized in the environment.
Revised: This ability relies inpart on the uniqueness of sounds made by different entities – distinct sounds can be associated withtheir sources, which in turn can be visually localized in the environment.

Original: Due to the fast progress inaudio and visual perception, several efforts have been made to develop multi-modal systems capableof mimicking our ability to visually localize sound sources.
Revised: Due to the fast progress inaudio and visual perception several efforts have been made to develop multi-modal systems capableof mimicking our ability to visually localize sound sources.

Original: One promising direction is to leverage the co-occurrence between sounds and the corresponding(visible) sources in video data.
Revised: One promising direction is to leverage the co-occurrence between sounds and the corresponding(visible) sources in video data.

Original: Since audio-visual co-occurrence arises naturally, algorithms canscale to very large datasets, without requiring costly human annotations.
Revised: Since audio-visual co-occurrence arises naturally, algorithms canscale to very large datasets, without requiring costly human annotations.

Original: However, despite thesignificant efforts made to develop self-supervised localization methods [1, 2, 3, 4, 5], currentlyaccepted evaluation protocols hide two critical limitations of current approaches: (1) current methodsoverfit easily even when scaled up to large datasets, and (2) current methods assume that visible sound sources are always present in the video and thus are unreliable when deployed on realistic datawhere this assumption does not hold.
Revised: However, despite thesignificant efforts made to develop self-supervised localization methods [1, 2, 3, 4, 5], currentlyaccepted evaluation protocols hide two critical limitations of current approaches: (1) current methodsoverfit easily even when scaled up to large datasets, and (2) current methods assume that visiblesound sources are always present in the video and thus are unreliable when deployed on realistic datawhere this assumption does not hold.

Original: The first limitation remained hidden as prior works [2, 4, 6, 5] rely heavily on early stopping foroptimal performance ( i.e ., by continuously validating the model during training using a humanannotated test set).
Revised: The first limitation remained hidden as prior works [2, 4, 6, 5] rely heavily on early stopping foroptimal performance ( i.e ., by continuously validating the model during training using a humanannotated test set).

Original: However, this practice  violates the main assumption  of weakly supervised visualsource localization, i.e ., the requirement for bounding box annotations during training.
Revised: However, this practice not only violates the unsupervised assumption but is alsoat odds with the major promise of unsupervised learning – i.e ., enhanced capabilities when trainingon larger amounts of data for longer.

Original: The second limitation remained hidden as most prominent benchmark datasets [1, 6] and evaluationmetrics only assess the model’s ability to localize a sound source when one is present in the video.
Revised: The second limitation remained hidden as current benchmark datasets [1, 6] and evaluation metricsonly assess the model’s ability to localize a sound source when one is present in the video.

Original: Importantly, they ignore a model’s ability to correctly predict the absence of visual sound sources,leading to a bias towards precision with  disregard for false positives.
Revised: Importantly,it ignores a model’s ability to correctly predict the absence of visual sound sources. This has led to abias towards precision with a complete disregard for false positives.

Original: It should be noted that falsepositive detection is closely related to the silent object detection problem highlighted in recentworks [7, 8]. Although these works introduce methods to suppress the localization of silent objects,they require the collection of a clean dataset containing a single source per video. Instead, we aim totackle this issue without assuming knowledge of the number of sources.
Revised: However, effective deploymentof source localization in realistic settings requires both low type I and type II error rates.

Original: The two limitations described above highlight the need for a more balanced and complete evaluationprotocol for visual sound source localization.
Revised: The two limitations described above highlight the need for a more balanced and complete evaluationprotocol for visual sound source localization.

Original: To achieve this, we extend popular benchmark test
Revised: To achieve this, we extend popular benchmark test

Original: sets(Flickr SoundNet [1] and VGG-Sound Source [6]) to include ’negative’ samples without any visiblesound sources.
Revised: sets(Flickr SoundNet [1] and VGG-Sound Source [6]) to include ’negative’ samples without any visiblesound sources.

Original: We then conduct an extensive evaluation of existing methods [2, 3, 4, 6,9, 7], where inaddition to overall precision, we also assess methods based on their ability to predict negative samples(through average precision and F1 scores).
Revised: We then conduct an extensive evaluation of existing methods [2, 3, 4, 6, 7], where inaddition to overall precision, we also assess methods based on their ability to predict negative samples(through average precision and F1 scores).

Original: We observe that all prior work suffers from significantoverfitting problems, and thus leans heavily on early stopping to achieve the best performance.
Revised: We observe that all prior work suffers from significantoverfitting problems, and thus leans heavily upon early stopping to achieve the best performance.

Original: Wealso found that previous approaches fail to identify  frames where no visible sound sources are present.
Revised: Wealso found that previous approaches fail to identify the frames where no visible sound sources arepresent.

Original: To address these issues, we further propose a new algorithm - Momentum Audio-Visual Matching for
Revised: To address these issues, we further propose a new algorithm - Momentum Audio-Visual Matching for

Original: Visual Sound Localization ( MoVSL ).
Revised: Visual Sound Localization ( MoVSL ).

Original: First, to combat overfitting, we adopt slow-moving momentumaudio and visual encoders and extreme visual dropout.
Revised: First, to combat overfitting, we adopt slow-moving momentumaudio and visual encoders and extreme visual dropout.

Original: Second, we localize visual sources bysimultaneously performing sounding region selection and cross-modal instance discrimination.
Revised: Second, we localize visual sources bysimultaneously performing sounding region selection and cross-modal instance discrimination.

Original: Thefirst term highlights which regions within an image are most associated with particular audio, whilethe second term downplays regions that can be better described by the audio of other samples.
Revised: Thefirst term highlights which regions within an image are most associated with particular audio, whilethe second term downplays regions that can be better described by the audio of other samples.

Original: Bycombining these two terms, the model is able to both accurately localize visible sources and identifywhen no visible sources are present.
Revised: Bycombining these two terms, the model is able to both accurately localize visible sources and identifywhen no visible sources are present.

Original: Using the newly developed evaluation protocol, we show that,unlike all prior work, MoVSL does not overfit and thus can be trained without relying on early stopping(which requires human annotations).
Revised: Using the newly developed evaluation protocol, we show that,unlike all prior work, MoVSL does not overfit and thus can be trained without relying on early stopping(which requires human annotations).

Original: MoVSL achieves state-of-the-art performance on multiple datasetsand is more accurate at identifying samples without any visible sound sources.
Revised: MoVSL achieves state-of-the-art performance on multiple datasetsand is more accurate at identifying samples without any visible sound sources.

Original:  Related Work
Revised:  Related Work

Original: Audio-Visual Joint Learning.
Revised: Audio-Visual Joint Learning.

Original: Audio-visual joint learning has been explored in many works [10,11 , 12,  13, 14,1, 15, 16, 17, 18, 19, 20, 21, 22] to learn the temporal correlation between audioand visual information from videos.
Revised: Audio-visual joint learning has been explored in many works [8,9, 10, 11, 12, 1, 13, 14, 15, 16, 17, 18, 19, 20] to learn the temporal correlation between audioand visual information from videos.

Original: Given a sequence of videos, the main idea is to close thedistance of audio and visual features from the same timestamp while pushing away those fromdifferent timestamps.
Revised: Given a sequence of videos, the main idea is to close thedistance of audio and visual features from the same timestamp while pushing away those fromdifferent timestamps.

Original: Such audio-visual temporal synchronization is beneficial to several tasks,such as audio separation [19, 23, 24, 16, 17, 25, 26, 27], audio-visual spatialization [15, 28, 29, 20],visual sound source localization [1, 2, 3, 4, 6, 30].
Revised: Such audio-visual temporal synchronization is beneficial to several tasks,such as audio separation [17, 21, 22, 14, 15, 23, 24, 25], audio-visual spatialization [13, 26, 27, 18],visual sound source localization [1, 2, 3, 4, 6, 28].

Original: In this work, we mainly focus on audio-visualsource localization on videos without annotations involved during training, which requires us to learnmeaningful audio-visual representations between a clip of audio and the frame centering around it.
Revised: In this work, we mainly focus on audio-visualsource localization on videos without annotations involved during training, which requires us to learnmeaningful audio-visual representations between a clip of audio and the frame centering around it.

Original: Audio-Visual Source Localization.
Revised: Audio-Visual Source Localization.

Original: Audio-visual source localization aims at predicting the locationof the sounding object in a video.
Revised: Audio-visual source localization aims at predicting the locationof the sounding object in a video.

Original: Early machine learning and deep models [31, 32, 33] were usedto capture the correspondences of audio and visual features.
Revised: Early machine learning and deep models [29, 30, 31] were usedto capture the correspondences of audio and visual features.

Original: Recently, contrast learning basedapproaches [34, 2, 6, 9] have been proposed to learn the audio-visual alignment between featuresfrom two stream networks.
Revised: Recently, contrast learning basedapproaches [32, 2, 6, 7] have been proposed to learn the audio-visual alignment between featuresfrom two stream networks.

Original: For instance, LVS [6] leveraged a contrastive loss with hard negativesmining to learn the audio-visual co-occurrence map discriminatively.
Revised: For instance, LVS [6] leveraged a contrastive loss with hard negativesmining to learn the audio-visual co-occurrence map discriminatively.

Original: Contrastive learning withhard positives was introduced in HardPos [9] to learn audio-visual alignment with negative samples.
Revised: Contrastive learning withhard positives was introduced in HardPos [7] to learn audio-visual alignment with negative samples.

Original: More recently, EZ-VSL [5] proposed the multiple instance contrastive learning without positive
Revised: More recently, EZ-VSL [5] proposed the multiple instance contrastive learning without positiveand negative regions for training to match the most similar location in the corresponding image anddistinguish all locations in different images.

Original: AudioEncoder
Revised: AudioEncoder

Original: P r o j e c t
Revised: P r o j e c t

Original: P oo l
Revised: P oo l

Original: C
Revised: C

Original: D r o p o u t
Revised: D r o p o u t

Original: VisualEncoder
Revised: VisualEncoder

Original: P r o j e c t
Revised: P r o j e c t

Original: C
Revised: C

Original: H
Revised: H

Original: W D r o p o u t
Revised: W D r o p o u t

Original: P r o j e c t img img2 C
Revised: P r o j e c t img img2 C

Original: H
Revised: H

Original: W
Revised: W

Original: P r o j e c t
Revised: P r o j e c t

Original: C aud aud
Revised: C aud aud

Original: Visual 
Revised: Visual Branch Audio Branch

Original: [Figure]
Revised: [Figure]

Original:  and negative regions for training to match the most similar location in the corresponding image anddistinguish all locations in different images.
Revised: The audio-visual correspondence of pixel-wise similarity between momentum branches and base branches is aligned by the cross-modal multiple instance contrastive learning objective.

Original: However, these baselines usually rely on early stopping, validating the model  during training toselect the most accurate one.
Revised: However, these baselines usually performed the model validation on the test set during training toselect the best model as the final checkpoint, which caused the model to overfit.

Original: Furthermore, with the exception of [7, 8], they focus on localizingsound sources that are visible, ignoring non-visible or non-audible samples in real videos.
Revised: Furthermore, theyonly focus on localizing sound sources that are visible, ignoring non-visible or non-audible samplesin real videos.

Original: Althoughunderstanding when sources are not visible is important to several audio-visual tasks, like in opendomain audio-visual source separation [35, 36], this issue still poses significant challenges.

Original: DSOL [7]and IEr [8] proposed a method to suppress localization of silent objects, and an evaluation metricthat measures the ability of a model to ignore visible silent objects. These methods, however, relyon the knowledge of the number of sound sources, not available in most large scale datasets. Incontrast, we tackle visual source localization without assuming knowledge of the number of sources. Specifically, we propose a novel Simultaneous Instance Discrimination and Localization frameworkfor weakly-supervised visual source localization with clear benefits, as it demonstrably improvesboth precision and false positive rates.
Revised: To address these challenges, we propose a novel momentum audio-visual matchingapproach for learning the audio-visual correspondence between momentum and base features.

Original: We also annotate real world test samples that may lead to falsepositives (beyond silent objects), and propose new metrics for comprehensive evaluation.
Revised: Wealso explore both sounding and non-sounding frames in this work and propose new metrics forcomprehensive evaluation.

Original:  Method
Revised:  Method

Original: We propose a new Momentum-based approach for Visual Sound Localization, denoted MoVSL , byminimizing the image-to-audio/audio-to-image pixel-wise similarity between momentum audio/visualfeatures and base visual/audio embeddings.
Revised: We propose a new Momentum-based approach for Visual Sound Localization, denoted MoVSL , byminimizing the image-to-audio/audio-to-image pixel-wise similarity between momentum audio/visualfeatures and base visual/audio embeddings.

Original: In section 3.1, we define the problem and notationsused in this paper and revisit multiple-instance contrastive learning proposed in EZ-VSL [5].
Revised: In section 3.1, we define the problem and notationsused in this paper and revisit multiple-instance contrastive learning proposed in EZ-VSL [5].

Original: Insection 3.2, we detail the proposed Momentum Audio-Visual Matching strategy.
Revised: Insection 3.2, we detail the proposed Momentum Audio-Visual Matching strategy.

Original: Finally, we detail thenew evaluation protocol including the extended test sets for Flickr-SoundNet/VGG-SS and evaluationmetrics in sections 3.3 and 3.3.
Revised: Finally, we detail thenew evaluation protocol including the extended test sets for Flickr-SoundNet/VGG-SS and evaluationmetrics in sections 3.3 and 3.3.

Original: 3.1 Preliminaries:Weakly Supervised Visual Source Localization 
Revised: 3.1 Preliminaries: Visual Source Localization by Localized Audio-Visual Matching [5]

Original: Given an audio-visual paired dataset D = { ( v i , a i ) : i = 1 , . . . , N } , visual source localization (VSL)aims at predicting the location of the sources present in sound a i within the visual frame v i .VSLmodels can be learned from various levels of supervision:
Revised: Given an audio-visual paired dataset D = { ( v i , a i ) : i = 1 , . . . , N } , visual source localization(VSL) aims at predicting the location of the sources present in sound a i within the visual framev i .

Original: Unsupervised VSL learn to localize withoutany form of human annotations; Weakly-supervised VSL forgo bounding-box supervision but mayleverage categorical information. This categorical information can be in the form of object labels forvisual encoder pre-training, or audio event labels for audio encoder pre-training; Semi-supervised VSL learns from a small number of bounding-box annotations, together with a large unsupervised orweakly-supervised dataset; Finally, Fully supervised VSL models can learn from large amounts offully annotated datasets. In this paper, we tackle the Weakly-Supervised VSL problem.
Revised: This is accomplished by matching the sound a i to each region of v i in a learned commonspace.

Original: We note thatequivalent prior work, such as [1, 4, 2, 3, 7, 6, 9, 5], often refer to this problem as unsupervised VSL,despite actually addressing the weakly supervised problem, as they use vision models pre-trained on

Original: ImageNet for object recognition. Visual source localization is accomplished by matching the sound a i to each region of v i in a learnedcommon space. To this end, an audio encoder f a ( · ) is first used to extract global audio features, anda visual encoder f v ( · ) to extract local visual features at each ( x, y ) location.
Revised: To this end, an audio encoder f a ( · ) is first used to extract global audio features, and avisual encoder f v ( · ) to extract local visual features at each ( x, y ) location.

Original: Both features are thenmapped into a common space using two projection heads g a ( · ) and g v ( · ) for the audio and visualfeatures, respectively.
Revised: Both features are thenmapped into a common space using two projection heads g a ( · ) and g v ( · ) for the audio and visualfeatures, respectively.

Original: This results in a set of visual embeddings spanning all locations in an image
Revised: This results in a set of visual embeddings spanning all locations in an image

Original: V i
Revised: V i

Original: =
Revised: =

Original: { v xyi :
Revised: { v xyi :

Original: ∀ x, y } , ∀ i ∈ D , x ∈ [1 , W ] , y ∈ [1 , H ] and one global audio embedding a i .
Revised: ∀ x, y } , ∀ i ∈ D , x ∈ [1 , W ] , y ∈ [1 , H ] and one global audio embedding a i .

Original: All components  are then trained to align the audio embedding a i with the visual embeddings v xyi atthe locations ( x, y ) where the sound sources are located.
Revised: All components of the model are then trained to align the audio embedding a i with the visualembeddings v xyi at the locations ( x, y ) where the sound sources are located.

Original: If trained successfully, the model can be usedfor VSL by computing the audio-visual similarity at each location.
Revised: If trained successfully,the model can be used for VSL by computing the audio-visual similarity at each location.

Original: However, in weakly-supervised VSL, the ground-truth locations of sound sources are not known during training.
Revised: However,for unsupervised VSL, the ground-truth locations of sound sources are not known during training.

Original: To bypass the lack of annotations, prior work [5] proposed a multiple-instance contrastive learningobjective, in which the model is trained to match audio representations a i with at least one locationin the positive set V i , while pushing away from the visual features in all negative bags in the samemini-batch V j ∀ j ̸
Revised: To bypass the lack of annotations, prior work [5] proposed a multiple-instance contrastive learningobjective, in which the model is trained to match audio representations a i with at least one locationin the positive set V i , while pushing away from the visual features in all negative bags in the samemini-batch V j ∀ j ̸

Original: = i (at all locations).
Revised: = i (at all locations).

Original: In particular, the similarity between an audio feature a iand a bag of visual features V j
Revised: In particular, the similarity between an audio feature a iand a bag of visual features V j

Original: = { v xyj : ∀ x, y } is computed by max-pooling audio-visual cosinesimilarities across spatial locations sim ( a i , V j )
Revised: = { v xyj : ∀ x, y } is computed by max-pooling audio-visual cosinesimilarities across spatial locations sim ( a i , V j )

Original: =
Revised: =

Original: max xy s ( a i , v xyj ) .
Revised: max xy s ( a i , v xyj ) .

Original: The model is then trained tominimize theaverage per sample loss 
Revised: The model is then trained tominimizing the loss function

Original: [Equation]
Revised: [Equation]

Original: where τ is a temperature hyper-parameter, and B is the batch-size.
Revised: where τ is a temperature hyper-parameter.

Original: 3.2 Momentum Audio-visual Matching
Revised: 3.2 Momentum Audio-visual Matching

Original: We now introduce the proposed method, MoVSL , for visual sound localization.
Revised: We now introduce the proposed method, MoVSL , for visual sound localization.

Original: As we will show, mostprior work, including the current state-of-the-art EZ-VSL [5], suffer from two critical limitations: (1)models can easily overfit to the self-supervised objective; (2) models are not capable of identifyingnegative samples ( i.e ., samples with no visible sound sources).
Revised: As we will show, mostprior work, including the current state-of-the-art EZ-VSL [5], suffer from two critical limitations: (1)models can easily overfit to the self-supervised objective; (2) models are not capable of identifyingnegative samples ( i.e ., samples with no visible sound sources).

Original: The proposed method, illustrated in Figure 1, is designed to address these two limitations.
Revised: The proposed method, illustrated in Figure 1, is designed to address these two limitations.

Original: To preventoverfitting, we apply two regularization techniques: dropout [37] and (slow-moving) momentumencoders [38, 39].
Revised: To preventoverfitting, we apply two regularization techniques: dropout [33] and (slow-moving) momentumencoders [34, 35].

Original: To enhance the identification of negatives, we propose to conduct VSL bysimultaneously performing sounding region selection and instance discrimination.
Revised: To enhance the identification of negatives, we propose to conduct VSL bysimultaneously performing sounding region selection and instance discrimination.

Original: Dropout We apply dropout [37] on the output of the f v and f a encoders.
Revised: Dropout We apply dropout [33] on the output of the f v and f a encoders.

Original: Interestingly, very heavyvisual dropout is required to prevent overfitting (with a dropout probability as large as 0 . 9 ).
Revised: Interestingly, very heavyvisual dropout is required to prevent overfitting (with a dropout probability as large as 0 . 9 ).

Original: Sincemultiple-instance contrastive learning (Eq. 1) matches the audio to only one of the H × W locationsin the image, spurious alignment becomes more likely.
Revised: Sincemultiple-instance contrastive learning (Eq. 1) matches the audio to only one of the H × W locationsin the image, spurious alignment becomes more likely.

Original: Heavy visual dropout reduces the likelihoodof spurious audio-visual alignments, as visual features become more spatially redundant 

Revised: Heavy visual dropout reduces the likelihoodof spurious audio-visual alignments, as visual features become more spatially redundant

Original: Momentum encoders We use momentum audio-visual encoders [38, 39]
Revised: Momentum encoders We use momentum audio-visual encoders [34, 35]

Original: ˆ f a ( · ) , ˆ f v ( · ) to compute(slow-moving) representations ˆ a i and ˆ v xyi .
Revised: ˆ f a ( · ) , ˆ f v ( · ) to compute(slow-moving) representations ˆ a i and ˆ v xyi .

Original: These momentum features are then used as more stabletargets for audio-visual matching.
Revised: These momentum features are then used as more stabletargets for audio-visual matching.

Original: Following [38, 39], the momentum encoders are updated usingan exponential moving average of the corresponding online encoders with coefficient m .
Revised: Following [34, 35], the momentum encoders are updated usingan exponential moving average of the corresponding online encoders with coefficient m .

Original: 2 Sinceupdates to the audio-visual encoders are slowly incorporated into the momentum encoders, the targetrepresentations display a smoother behavior during the training process.
Revised: 2 Sinceupdates to the audio-visual encoders are slowly incorporated into the momentum encoders, the targetrepresentations display a smoother behavior during the training process.

Original: Enhancing negative detection with simultaneous localization and instance discrimination (SLID)
Revised: Enhancing negative detection with simultaneous localization and instance discrimination (SLID)

Original: Inspired by prior work on weakly-supervised detection [40], we factorize VSL into two probabilityterms.
Revised: Inspired by prior work on weakly-supervised detection [36], we factorize VSL into two probabilityterms.

Original: The first is a localization term P loc that distinguishes within an image the regions that likelydepict the sound source from the regions that likely do not.
Revised: The first is a localization term P loc that seeks to distinguish within an image the regionsthat likely depict the sound source from the regions that likely do not.

Original: The second is a cross-modal instancediscrimination term P xid that highlights regions  that are associated with the audio, while suppressingregions that would be better explained by sound sources from other samples.
Revised: The second is a cross-modalinstance discrimination term P xid that seeks to highlight regions of an image that are likely associatedwith the audio, while suppressing regions that would be better explained by sound sources from othersamples.

Original: The key insight is that toenhance negative detection, VSL should only select the regions that are simultaneously highlighted inboth terms.
Revised: The key insight is that to enhance negative detection, VSL should only select the regionsthat are simultaneously highlighted in both terms.

Original: While the first term is responsible for sound source localization, and the second prevents VSL to be overly confident when sound sources are not visible in the image.
Revised: While the first term is responsible for sound sourcelocalization, and the second prevents VSL to be overly confident when sound sources are not visiblein the image.

Original: To enable different interactions, the localization and instance discrimination terms are defined as audiovisual similarities computed on different spaces.
Revised: To enable different interactions, the localization and instance discrimination terms are defined as audiovisual similarities computed on different spaces.

Original: In particular, let a loc
Revised: In particular, let a loc

Original: i = g loc a ( a i ) and a xid i =
Revised: i = g loc a ( a i ) and a xid i =

Original: g xid a ( a i )be the audio representation of a i in the two separate spaces, and V loc i = { g loc v ( v xyi ) :
Revised: g xid a ( a i )be the audio representation of a i in the two separate spaces, and V loc i = { g loc v ( v xyi ) :

Original: ∀ x, y } and
Revised: ∀ x, y } and

Original: V xid i
Revised: V xid i

Original: = { g xid v ( v xyi ) : ∀ x, y } be the two bags of visual features for v i .
Revised: = { g xid v ( v xy i ) : ∀ x, y } be the two bags of visual features for v i .

Original: Here, the various functionsg ( ·) denote simple linear projections.
Revised: Here, the various functions g ( · simple linear projections.

Original: The localization term is computed as the softmax ρ xy ( · ) oflocalized audio-visual similarities over the spatial dimensions x, y 
Revised: The localization term is computed as the softmax ρ xy ( · ) of localized audio-visual similarities over the spatial dimensions x, y [Equation]

Original: [Equation]
Revised: [Equation]

Original: and the instance discrimination term as the softmax ρ i ( · ) over instances
Revised: and the instance discrimination term as the softmax ρ i ( · ) over instances

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: To select the regions in which both P loc and P xid are active, the two terms are combined into a singleprediction map[Equation] = P loc [Equation]
Revised: To select the regions in which both P loc and P xid are active, the two terms are combined into a singleprediction map P xyij =

Original:  a i , v [Equation] ) · P xid ( a i , v xyj ) , and the model is trained to optimize
Revised: P loc ( a i , v xyj ) · P xid ( a i , v xyj ) , and the model is trained to optimize

Original: [Equation]
Revised: [Equation]

Original: Finally, to combine SLID momentum encoders, audio and visual target representations are used whencomputing Eq.
Revised: Finally, to combine SLID momentum encoders, audio and visual target representations are used whencomputing Eq.

Original: 2 and Eq.
Revised: 2 and Eq.

Original: During inference, both localization and instance discrimination terms are used for source localization.
Revised: During inference, both localization and instance discrimination terms are used for source localization.

Original: We also observed that the (more stable) momentum features lead to improved visual source localization.
Revised: We also observed that the (more stable) momentum features lead to improved visual source localization.

Original: Thus, given an audio-visual pair ( v, a ) , the audio-visual localization map is computed as
Revised: Thus, given an audio-visual pair ( v, a ) , the audio-visual localization map is computed as

Original: [Equation]
Revised: [Equation]

Original: Furthermore, this audio-visual localization map can be easily combined with object-guided localization proposed in [5] for improved performance.
Revised: Furthermore, this audio-visual localization map can be easily combined with object-guided localization proposed in [5] for improved performance.

Original: 3.3 Benchmarking Visual Source Localization
Revised: 3.3 Benchmarking Visual Source Localization

Original: We introduce a new evaluation protocol for VSL.
Revised: We introduce a new evaluation protocol for VSL.

Original: Extended Flickr-SoundNet/VGG-SS Non-visible sounds or frames with silent objects can oftenbe found in videos.
Revised: Extended Flickr-SoundNet/VGG-SS Non-visible sounds or frames with silent objects can oftenbe found in videos.

Original: To tackle these cases, we present a new evaluation protocol.
Revised: To tackle these cases, we present a new evaluation protocol.

Original: We extended
Revised: We extended

Original: VGG-SS/Flickr-SoundNet
Revised: VGG-SS/Flickr-SoundNet

Original: by merging clips with no sounding objects to the original VGG-SS/FlickrSoundNet test set.
Revised: by merging clips with no sounding objects to the original VGG-SS/FlickrSoundNet test set.

Original: Specifically, we download 1k YouTube videos from the large-scale audio-visualdataset, VGG-Sound [41], and manually select 5-second clips with non-audible frames and/ornon-visible sound sources.
Revised: Specifically, we download 1k YouTube videos from the large-scale audio-visualdataset, VGG-Sound [37], and manually select 5-second clips with non-audible frames and/ornon-visible sound sources.

Original: This resulted in 379 non-sounding (negative) samples.
Revised: This resulted in 379 non-sounding (negative) samples.

Original: Beyond the manually identified negative pairs, we further generate negative samples by pairing audioand videos that do not belong together.
Revised: Beyond the manually identified negative pairs, we further generate negative samples by pairing audioand videos that do not belong together.

Original: We control the difficulty of these negative pairs, by sampling25% of pairs from audio and video that are from the same class (hard negative), and 75% fromdifferent classes.
Revised: We control the difficulty of these negative pairs, by sampling25% of pairs from audio and video that are from the same class (hard negative), and 75% fromdifferent classes.

Original: We merge all negatives with the VGG-SS [6] and Flickr-SoundNet [1] test setwhich only contain positive pairs.
Revised: We merge all negatives with the VGG-SS [6] and Flickr-SoundNet [1] test setwhich only contain positive pairs.

Original: Table 1 shows the final statistics of the final test sets.
Revised: Table 1 shows the final statistics of the final test sets.

Original: We also splittest samples into four groups (Small, Medium, Large, and Huge) according to the size ofthe soundsources, as measured by the image area (in pixels) occupied by the ground-truth bounding boxes.
Revised: We alsosplit sounding samples into four groups (Small, Medium, Large, and Huge), and show the shows thedistribution over sources of different sizes of the ground-truth bounding box.

Original: Evaluation metrics
Revised: Evaluation metrics

Original: Prior work [1] introduced consensus intersection over union (cIoU, denoted asu ) for evaluating predicted localization maps, by comparing them to a group of human annotations.[Table]
Revised: Prior work [1] introduced consensus intersection over union (cIoU, denoted asu ) for evaluating predicted localization maps, by comparing them to a group of human annotations.

Original: Given a set of predictions with cIoUs U = { u i } Ni =1 , each prediction is considered to be correct ifits cIoU is above the cIoU threshold γ (usually set at γ= 0 . 5 ).
Revised: Given a set of predictions with cIoUs U = { u i } Ni =1 , each prediction is considered to be correct if itscIoU is above the cIoU threshold γ (usually set at γ ).

Original: Thus, the Precision over all positivesamples ( i.e . with visible sounding objects) is computed as
Revised: Thus, the Precision over all positive samples( i.e . with visible sounding objects) is computed as

Original: Precision =|T
Revised: Precision =

Original: P| |T
Revised: [Equation]

Original: P| + |FP| where T P = { i | u i> γ } ,
Revised: where T P = { i [Equation] u i

Revised: [Equation]

Revised: γ } ,

Original: FP =
Revised: FP =

Revised: { i

Revised: [Equation]

Original: { i | u i ≤ γ } (6)
Revised: u [Equation] γ } (6)

Original: Existing work [2, 3, 4, 6, 9, 5] usually reports a metric (misguidingly) called “CIoU”, which is the
Revised: Existing work [2, 3, 4, 6, 7, 5] usually reports a metric (misguidingly) called “CIoU”, which is the

Original: Precision at γ = 0 . 5 .
Revised: Precision at γ = 0 . 5 .

Original: In this work, we also consider samples with no visible sounding sources.
Revised: In this work, we also consider samples with no visible sounding sources.

Original: Thus, we consider adataset D
Revised: Thus, we consider adataset D

Original: =
Revised: =

Original: { ( v i , a i , c i ) :
Revised: { ( v i , a i , c i ) :

Original: i = 1 , . .
Revised: i = 1 , . .

Original: . , N } , where c i is 1 for samples with visible sources (positivesamples) and 0 for samples with no visible sources (negative samples).
Revised: . , N } , where c i is 1 for samples with visible sources (positivesamples) and 0 for samples with no visible sources (negative samples).

Original: Furthermore, in additionto the localization map, the model is required to predict whether the current pair contains a visiblesource or not.
Revised: Furthermore, in addition [Table] to the localization map, the model is required to predict whether the current pair contains a visiblesource or not.

Original: This is accomplished by computing a confidence score d i for each sample i which wedefine as the maximum value in the predicted audio-visual similarity map.
Revised: This is accomplished by computing a confidence score d i for each sample i which wedefine as the maximum value in the predicted audio-visual similarity map.

Original: Taking the confidencescore into account, we formulate true positives as T P ′
Revised: Taking the confidencescore into account, we formulate true positives as T P ′

Original: = { i | c i = 1 , d i > δ, u i > γ } , falsepositives as FP ′
Revised: = { i | c i = 1 , d i > δ, u i > γ } , falsepositives as FP ′

Original: =
Revised: =

Original: { i | c i = 1 , d i > δ, u i ≤ γ } ∪ { i | c i = 0 , d i > δ } and false negatives as
Revised: { i | c i = 1 , d i > δ, u i ≤ γ } ∪ { i | c i = 0 , d i > δ } and false negatives as

Original: FN ′
Revised: FN ′

Original: = { i | c i = 1 , d i ≤ δ } .
Revised: = { i | c i = 1 , d i ≤ δ } .

Original: Then, to comprehensively evaluate both sounding and non-soundingsamples, we use Average Precision (AP) and maximum F1 (max-F1) score obtained by sweepingthe confidence threshold δ ( i.e . max δ F 1( δ ) ).
Revised: Then, to comprehensively evaluate both sounding and non-soundingsamples, we use Average Precision (AP) and maximum F1 (max-F1) score obtained by sweepingthe confidence threshold δ ( i.e . max δ F 1( δ ) ).

Original: The detailed formula can be seen in Appendix.
Revised: The detailed formula can be seen in Appendix.

Original: Inexperiments, we use γ = 0 .
Revised: Inexperiments, we use γ = 0 .

Original: 5 for evaluation.
Revised: 5 for evaluation.

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate the proposed MoVSL on weakly-supervised audio-visual source localization.
Revised: In this section, we evaluate the proposed MoVSL on unsupervised audio-visual source localization.

Original: Following previous work [6, 9, 5], we use the VGG-Sound dataset [41] for training withoutbounding-box annotations.
Revised: Following previous work [6, 7, 5], we use the VGG-Sound dataset [37] for training without annotations.

Original: Then our trained models are tested on four datasets: Flickr-SoundNettestset [1], VGG-Sound Source [6], the proposed Extended Flickr-SoundNet and Extended VGG-SS.
Revised: Then our trained models are tested on four datasets: Flickr-SoundNet testset [1], VGG-Sound Source [6], the proposed Extended Flickr-SoundNet and Extended VGG-SS.

Original: We report the quantitive results using Precision-Visible on datasets with only sounding samples.
Revised: We report the quantitiveresults using Precision-Visible on datasets with only sounding samples.

Original: APand max-F1 are reported for benchmarks with both sounding and non-sounding samples.
Revised: AP and max-F1 are reportedfor benchmarks with both sounding and non-sounding samples.

Original: 4.1 Experimental setup
Revised: 4.1 Experimental setup

Original: Datasets.
Revised: Datasets.

Original: VGG-Sound [41] contains 200k videos of 309 sound categories from YouTube.
Revised: VGG-Sound [37] contains 200k videos of 309 sound categories from YouTube.

Original: Weextract audio-visual pairs with a single frame and its corresponding audio of 20 seconds long.
Revised: Forunsupervised learning, we extract audio-visual pairs with a single frame and its corresponding audioof 20 seconds long.

Original: Following commonly-used settings [6, 9, 5], we use subset of 144k pairs for training.
Revised: Following commonly-used settings [6, 7, 5], we use subset of 144k pairsfor training.

Original: We test ourmodel on two types of datasets: 1) with only sounding samples: Flickr-SoundNet testset [1] and VGG-Sound Source [6]; 2) with both sounding and non-sounding samples: the proposed Extended Flickr-SoundNet and Extended VGG-SS.
Revised: We test our model on two types of datasets: 1) with only sounding samples: FlickrSoundNet testset [1] and VGG-Sound Source [6]; 2) with both sounding and non-sounding samples:the proposed Extended Flickr-SoundNet and Extended VGG-SS.

Original: The Flickr-SoundNet includes only 250 audio-visual pairsof sounding objects, while VGG-Sound Source contains 5158 pairs of sounding objects from 220categories.
Revised: The Flickr-SoundNet includesonly 250 audio-visual pairs of sounding objects, while VGG-Sound Source contains 5158 pairs ofsounding objects from 220 categories.

Original: Extended Flickr-SoundNet totally includes 500 samples, 250 sounding and 250 nonsounding.
Revised: Extended Flickr-SoundNet totally includes 500 samples, 250sounding and 250 non-sounding.

Original: Extended VGG-SS spans 5158 pairs of sounding objects and 5158 pairs of non-soundingobjects, 10,316 in total.
Revised: Extended VGG-SS spans 5158 pairs of sounding objects and 5158pairs of non-sounding objects, 10,316 in total.

Original: Implementation.
Revised: Implementation.

Original: For audio and visual encoders, we use ResNet-18 [42] as the backbone, andinitialize the visual encoder with ImageNet [43] pre-trained weights, as previous work [6, 9, 5] did.
Revised: For audio and visual encoders, we use ResNet-18 [38] as the backbone, andinitialize the visual encoder with ImageNet [39] pre-trained weights, as previous work [6, 7, 5] did.

Original: The dimension of audio-visual features in the projected share space is 512.
Revised: The dimension of audio-visual features in the projected share space is 512.

Original: For visual input, wefirst resize frames of 224 × 224 to 246 along the shortest edge, and then apply random cropping withrandom horizontal flipping on them at training time, We do not use any data augmentation and useresized frames of 224 × 224 for inference.
Revised: For visual input, wefirst resize frames of 224 × 224 to 246 along the shortest edge, and then apply random cropping withrandom horizontal flipping on them at training time, We do not use any data augmentation and useresized frames of 224 × 224 for inference.

Original: For audio encoder, we extract the log spectrograms with257 frequency bands over 300 timesteps from 3s of audio at 22050 kHz. The underlying short-term
Revised: For audio encoder, we extract the log spectrograms with257 frequency bands over 300 timesteps from 3s of audio as input.

Original: [Table] [Table] Fourier transform is computed on approximately 25ms windows with a step size of 10ms. Followingthe prior work [5], we train the model with a batch size of 128 on 2NVIDIA RTX 3090 GPUs for 20epochs.
Revised: Following the prior work [5],we train the model with a batch size of 128 on 2 GPUs for 20 epochs.

Original: The Adam [44] optimizer with β 1 =0.9, β 2 =0.999 is applied with an initial learning rate of1e-4 and weight decay of 1e-4.
Revised: The Adam [40] optimizerwith β 1 =0.9, β 2 =0.999 is applied with an initial learning rate of 1e-4 and weight decay of 1e-4.

Original: The probability of dropout for visual branch is 0.9.
Revised: Theprobability of dropout for visual branch is 0.9.

Original: The momentumcoefficients m a =0.999, m v =0.999.
Revised: The momentum coefficients m a =0.999, m v =0.999.

Original: Our implementation is based on PyTorch [45] deep learning tool.

Original: Prior work.

Original: In this paper, we conduct an extensive benchmarking study of prior work.

Original: We trainseveral methods [1, 4, 2, 3, 6, 5, 7] to evaluate them using the newly proposed evaluation protocol.

Original: When reproducing prior work, we used authors implementation with default parameters.

Original: When notavailable, we carefully followed the paper description.

Original: We also implement a Center Prior baselinethat always predicts a circle in the middle of the frame occupying 50% of the image.

Original: 4.2 Comparison of overfitting to prior work
Revised: 4.2 Comparison of overfitting to prior work

Original: To demonstrate the effectiveness of our model in alleviating the overfitting issue, we compare bothbest and latest checkpoint with prior work  using the common metric Precision, that is, “CIoU”.
Revised: To demonstrate the effectiveness of our model in alleviating the overfitting issue, we compare bothbest and latest checkpoint with prior work [1, 2, 3, 4, 6, 7, 5] using the common metric PrecisionVisible, that is, “CIoU”.

Original: Table 2 shows the comparison results on Flickr SoundNet and VGG-SS testsets.
Revised: Table 2 shows the comparison results on Flickr SoundNet and VGG-SStestsets.

Original: We report both ourreproduced results, and the results reported in the original paper, whenever possible.
Revised: [Table] [Table]

Original: We can observe that our MoVSL superiorly outperforms previous work by using the best checkpointfor comparison, demonstrating the state-of-the-art results of our approach in this setting.
Revised: We can observe that our MoVSL superiorly outperforms previous work by using the best checkpointfor comparison, demonstrating the state-of-the-art results of our approach in this setting.

Original: Withoutthe OGL (object-guided localization) proposed in EZ-VSL [5], we achieve the performance gainsof 3.60% and 2.94% on the Flickr and VGG-SS tests sets.
Revised: Withoutthe OGL (object-guided localization) proposed in EZ-VSL [5], we achieve the performance gainsof 3.60% and 2.94% on the Flickr and VGG-SS tests sets.

Original: When the OGL is involved at inferencetime, our MoVSL further improves the SOTA performance by 2.46% and 0.82%.
Revised: When the OGL is involved at inferencetime, our MoVSL further improves the SOTA performance by 2.46% and 0.82%.

Original: Significant gainscan be observed in the setting of using the latest checkpoint for comparison.
Revised: Significant gainscan be observed in the setting of using the latest checkpoint for comparison.

Original: On the Flickr testset, our framework outperforms the state-of-the-art work by large margins, 17.20% and 13.20% forw/o OGL and w OGL.
Revised: On the Flickr testset, our framework outperforms the state-of-the-art work by large margins, 17.20% and 13.20% forw/o OGL and w OGL.

Original: The performance gains of 6.21% and 1.94% on the challenging VGG-SS [Table] [Table] dataset are achieved, which shows the importance of our momentum audio-visual matching strategyin preventing the model from overfitting.
Revised: The performance gains of 6.21% and 1.94% on the challenging VGG-SSdataset are achieved, which shows the importance of our momentum audio-visual matching strategyin preventing the model from overfitting.

Original: 4.3 Benchmarking on Extended Flickr-SoundNet and VGG-SS
Revised: 4.3 Benchmarking on Extended Flickr-SoundNet and VGG

Revised: -SS

Original: With the proposed metrics (AP, max-F1, Precision-Visible) in Sec 3.3, we compare our modelto priorwork on two datasets with both sounding and non-sounding samples: ExtendedFlickr-SoundNet, and
Revised: With the proposed metrics (AP, max-F1, Precision-Visible) in Sec 3.3, we benchmark state-of-the-artwork [6, 5] and our model on two datasets with both sounding and non-sounding samples: Extended

Original:  Extended VGG-SS.
Revised: Flickr-SoundNet, and Extended VGG-SS.

Revised: Note that the Center Prior is to generate a random Gaussianmap in the middle of frames for predictions.

Original: The experimental results are reported in Table 3.
Revised: The experimental results are reported in Table 3.

Original: As can be seen, the proposed MoVSL establishes new state-of-the-art performance in terms of all metrics.
Revised: As canbe seen, the proposed MoVSL establishes new state-of-the-art performance in terms of all metrics.

Original: On the Extended Flickr-SoundNet with 250 additional non-sounding samples, we achieve 51.63% AP and 59.10%max-F1, but a high precision of 83.60% on visible objects.
Revised: On the Extended Flickr-SoundNet with 250 additional non-sounding samples, we achieve 51.63% APand 59.10% max-F1, but a high precision of 83.60% on visible objects.

Original: This shows the importance of takingfalse negatives into account for non-sounding samples.
Revised: This shows the importanceof taking false negatives into account for non-sounding samples.

Original: Due to the large amount (5158) of negativesin the Extended VGG-SS, AP and max-F1 are decreased to 32.95% and 40.00%, indicating theadvantage of the proposed AP and max-F1 in evaluating non-sounding objects over the originalmetric, Precision-Visible.
Revised: Due to the large amount (5158) ofnegatives in the Extended VGG-SS, AP and max-F1 are decreased to 32.95% and 40.00%, indicatingthe advantage of the proposed AP and max-F1 in evaluating non-sounding objects over the original [Table]

Original:  In the meanwhile, quantitative results of AP and max-F1 are positivelycorrelated to those of Precision-Visible, which further implies the rationality of the proposed metricsin scoring both sounding and non-sounding samples.
Revised: [Table] metric, Precision-Visible. In the meanwhile, quantitative results of AP and max-F1 are positivelycorrelated to those of Precision-Visible, which further implies the rationality of the proposed metricsin scoring both sounding and non-sounding samples.

Original: 4.4 Experimental analysis
Revised: 4.4 Experimental analysis

Revised: In this section, we conduct an extensive benchmarking study of prior work.

Revised: We train severalmethods [1, 4, 2, 3, 6, 7, 5] to evaluate them using the newly proposed evaluation protocol.

Revised: Whenimplementing prior work, we used authors implementation with default parameters.

Revised: When notavailable, we carefully followed the paper description.

Revised: We report both our reproduced results, and theresults reported in the original paper, whenever possible.

Original: To demonstrate the benefits of MoVSL , we compare our method to prior work using the new evaluationprotocol.
Revised: To demonstrate the benefits of MoVSL , we compare our method to prior work using the new evaluationprotocol.

Original: All models were trained on the VGG-Sound 144k subset and the latest checkpoints weretested on the challenging Extended VGG-SS with both sounding and non-sounding objects.
Revised: All models were trained on the VGG-Sound 144k subset and the latest checkpoints weretested on the challenging Extended VGG-SS with both sounding and non-sounding objects.

Original: SLID localization strategy and regularization.
Revised: SLID localization strategy and regularization.

Original: In order to demonstrate the effectiveness of MoVSL ,we ablated the use of momentum encoders and simultaneous localization and instance discrimination(SLID) strategies during training.
Revised: In order to demonstrate the effectiveness of MoVSL ,we ablated the use of momentum encoders and simultaneous localization and instance discrimination(SLID) strategies during training.

Original: Table 4 reports the experimental results.
Revised: Table 4 reports the experimental results.

Original: Adding momentumencoders to the baseline improves AP and max-F1 by large margins, 3.64% and 4.70%.
Revised: Adding momentumencoders to the baseline improves AP and max-F1 by large margins, 3.64% and 4.70%.

Original: This showsthe effectiveness of the proposed momentum audio-visual matching approach in learning the crossmodal alignment between momentum and base representations.
Revised: This showsthe effectiveness of the proposed momentum audio-visual matching approach in learning the crossmodal alignment between momentum and base representations.

Original: When SLID is used, we furtherincrease all metrics by significant margins, 4.76% AP, 4.40% max-F1, and 5.04% Precision.
Revised: When SLID is used, we furtherincrease all metrics by significant margins, 4.76% AP, 4.40% max-F1, and 5.04% Precision.

Original: Furthermore, we ablated audio-visual matching strategies at inference time for the proposed MoVSLtrained with the SLID objective.
Revised: Furthermore, we ablated audio-visual matching strategies at inference time for the proposed MoVSLtrained with the SLID objective.

Original: We evaluate the localization ability of each SLID branch separately,i.e .
Revised: We evaluate the localization ability of each SLID branch separately,i.e .

Original: using the similarities between the global audio feature ˆ a loc ( ˆ a xid ) and the bag of localized visualfeatures ˆ v loc xy ( ˆ v xid xy ).
Revised: using the similarities between the global audio feature ˆ a loc ( ˆ a xid ) and the bag of localized visualfeatures ˆ v loc xy ( ˆ v xid xy ).

Original: We compare these with the localization performance using both SLID branches asin Equation 5.
Revised: We compare these with the localization performance using both SLID branches asin Equation 5.

Original: Results are shown in Table 4.
Revised: Results are shown in Table 4.

Original: Both alternative approaches achieve worse results than our MoVSL which uses both SLID branches for localization.
Revised: Both alternative approaches achieve worse results thanour MoVSL which uses both SLID branches for localization.

Original: This further indicates the effectiveness ofthe proposed approach in capturing the cross-modal pixel-wise correspondence between momentumand base features.
Revised: This further indicates the effectiveness ofthe proposed approach in capturing the cross-modal pixel-wise correspondence between momentumand base features.

Original: We also ablate the regularization techniques (weight decay, visual dropout,momentum) in Table 5.
Revised: We also ablate the regularization techniques (weight decay, visual dropout,momentum) in Table 5.

Original: It can be seen that dropout in the visual feature is very important to preventoverfitting.
Revised: It can be seen that dropout in the visual feature is very important to preventoverfitting.

Original: The optimal momentum is 0.999 for both visual and audio models.
Revised: The optimal momentum is 0.999 for both visual and audio models.

Original: Effect of Precision, F1 score, and AP.
Revised: Effect of Precision, F1 score, and AP.

Original: To explore the effect of various prediction sizes on the
Revised: To explore the effect of various prediction sizes on the

Original: Precision of visible objects within each subset (Small, Medium, Large, Huge), we vary the size from10% to 100% and compare the performance using the CPL, AVL, OGL, AV+OGL for inference.
Revised: Precision of visible objects within each subset (Small, Medium, Large, Huge), we vary the size from10% to 100% and compare the performance using the CPL, AVL, OGL, AV+OGL for inference.

Original: Theresults are shown in Figure 2.
Revised: Theresults are shown in Figure 2.

Original: All methods achieve the peak of Precision for medium, large, and hugeobjects at 10%, 30%, 70%.
Revised: All methods achieve the peak of Precision for medium, large, and hugeobjects at 10%, 30%, 70%.

Original: Due to the difficulty of identifying small objects in the whole frame,
Revised: Due to the difficulty of identifying small objects in the whole frame,

Original: Precision for different thresholds is always 0.
Revised: Precision for different thresholds is always 0.

Original: This highlights one important limitation of currentvisual sound source localization methods.
Revised: This highlights one important limitation of currentunsupervised visual sound source localization methods.

Original: We also evaluated the F1 score and AP with differentconfidence and IoU thresholds.
Revised: We also evaluated the F1 score and AP withdifferent confidence and IoU thresholds.

Original: Figure 3 shows the comparison results of the proposed MoVSL andprevious work.
Revised: Figure 3 shows the comparison results of the proposed MoVSL and previous work.

Original: When the confidence is 0.5, our method achieves the highest F1 score against existingmethods.
Revised: When the confidence is 0.5, our method achieves the highest F1 scoreagainst existing methods.

Original: We highlight that MoVSL and EZVSL are the only approaches whose F1 scores increase withhigher confidence thresholds.

Original: This shows that unlike most of other prior work, MoVSL can effectivelyidentify negative pairs, i.e . reduce false positive rate without sacrificing true positive rate.

Original: Our methodalso outperforms all prior works in terms of AP using all IoU thresholds.
Revised: Our method also outperforms all prior works in terms of AP using all IoUthresholds.

Original: 4.5 Limitations
Revised: 4.5 Limitations

Original: Although we achieve superior precision on large and huge objects in the image, the performance ofsmall and medium objects is not satisfactory, as shown in Figure 2.
Revised: Although we achieve superior precision on large and huge objects in the image, the performance ofsmall and medium objects is not satisfactory, as shown in Figure 2.

Original: When it comes to small objects,the proposed method performs badly as the prior work did.
Revised: When it comes to small objects,the proposed method performs badly as the prior work did.

Original: The possible reason is that all methodsuse the lightweight ResNet-18 as the visual backbone to extract features, which limits the qualityof visual representations for audio-visual matching.
Revised: The possible reason is that all methodsuse the lightweight ResNet-18 as the visual backbone to extract features, which limits the qualityof visual representations for audio-visual matching.

Original: In the meanwhile, we notice that the predictionmap is generated by the bilinear interpolation of an audio-visual response map with a low resolution.
Revised: In the meanwhile, we notice that the predictionmap is generated by the bilinear interpolation of an audio-visual response map with a low resolution.

Original: Increasing the resolution of such a map could be beneficial to small objects in the raw image withhigh resolution.
Revised: Increasing the resolution of such a map could be beneficial to small objects in the raw image withhigh resolution.

Original: Therefore, applying a strong backbone would be the future direction for improvingthe precision of small objects.
Revised: Therefore, applying a strong backbone would be the future direction for improvingthe precision of small objects.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we explore two main issues in common early stopping settings and the visible soundsource assumption for weakly-supervised audio-visual source localization.
Revised: In this work, we explore two main issues in common early stopping settings and the visible soundsource assumption for unsupervised audio-visual source localization.

Original: To capture the model’sperformance on samples with no visible sound sources, we extend the test sets of popular benchmarks,
Revised: To capture the model’s performance on samples with no visible sound sources, we extend the test sets of popular benchmarks,

Original: Flickr SoundNet and VGG-Sound Source, to cover negative samples, and use average precisionand F1 scores as metrics to balance precision and recall.
Revised: Flickr SoundNet and VGG-Sound Source, to cover negative samples, and use average precisionand F1 scores as metrics to balance precision and recall.

Original: Extensive evaluations of prior work areperformed under the new proposed evaluation protocol.
Revised: Extensive evaluations of prior work areperformed under the new proposed evaluation protocol.

Original: We observe that most previous work suffersfrom significant overfitting problems, and thus relies heavily on early stopping for best results.
Revised: We observe that most previous work suffersfrom significant overfitting problems, and thus relies heavily on early stopping for best results.

Original: Toeliminate this, we propose a novel Momentum audio-visual matching approach for Visual Sound
Revised: Toeliminate this, we propose a novel Momentum audio-visual matching approach for Visual Sound

Original: Localization by applying momentum features as targets of base embeddings for each modality, whichwe denote MoVSL.
Revised: Localization by applying momentum features as targets of base embeddings for each modality, whichwe denote MoVSL.

Original: Thorough experiments demonstrate the effectiveness of the proposed approachin combating overfitting, and the state-of-the-art performance on two popular benchmarks, Flickr
Revised: Thorough experiments demonstrate the effectiveness of the proposed approachin combating overfitting, and the state-of-the- art performance on two popular benchmarks , Flickr

Original: SoundNet and VGG-Sound Source.
Revised: SoundNet and VGG-Sound Source.

Original: Appendix
Revised: Checklist

Original: We start by releasing code and demo, detailing the baselines and metrics used for benchmarking visualsound localization (Precision, F1 score, and Average Precision).
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: We demonstrate the effectiveness ofthe proposed MoVSL in other commonly used training datasets (Flickr 10k) and open set visual soundsource localization.
Revised: [Yes] See Section 1. (b) Did you describe the limitations of your work? [Yes] See Section 4.5.

Original: Finally, we compare the qualitative results of our approach with existing methodsin terms of various sizes of sounding objects.
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: A Code & Demo
Revised: [Yes] SeeSection ?? .(d)

Original: Code is available at:https://anonymous.4open.science/r/MoVSL-1021 .
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes] See Supplementary material. If you are including theoretical results...

Original: Demo is available at:https://drive.google.com/file/d/1SDJj1JoZxRIJBcTJL4pKbmcHs3T1DRdm/view B Baselines
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: In the field of visual sound source localization, we comprehensively consider existing approaches forbenchmarking as follows: • Attention 10k [1] (2018’CVPR): the first attention-based work with a two-stream architecturewith each modality for sound source localization in an image; (code: https://github.com/ardasnck/learning_to_localize_sound_source ) • DMC [2] (2019’CVPR): a multi-modal clustering network for learning audiovisual corre-
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: spondences by using convolutional maps with each modality in different shared spaces;(code: https://github.com/DTaoo/Simplified_DMC , MIT License) • CoarsetoFine [4] (2020’ECCV): a two-stage pipeline that aligned the cross-modalfeatures in a coarse-to-fine way; (code: https://github.com/shvdiwnkozbw/ Multi-Source-Sound-Localization ) • LVS [6] (2021’CVPR): a contrastive learning framework with hard negatives mining to
Revised: [N/A]

Original: extract the audio-visual co-occurrence map discriminatively; (code: https://github.com/hche11/Localizing-Visual-Sounds-the-Hard-Way , Apache License 2.0) • HardPos [9] (2022’ICASSP): an improved work based on LVS by adding hard positives for aligning audio-visual matching semantics from negative pairs; • EZ-VSL
Revised: If you ran experiments...

Original: [5] (2022’ECCV): a strong baseline that proposed the multiple instance contrastive learning to align locations with high similarity in the image and dispel all locations in
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: different images; (code: https://github.com/stoneMo/EZ-VSL , Apache License 2.0) • DSOL [7] (2020’NeurIPS): a two-stage training baseline to deal with silence in category- aware sound source localization, where the audio event class is involved in the training stage 1 and an evaluation metric (NSA) is proposed specifically to measure the abil-
Revised: perimental results (either in the supplemental material or as a URL)?

Original: ity of a model to ignore visible silent objects; (code: https://github.com/DTaoo/ Discriminative-Sounding-Objects-Localization , MIT License); • MoVSL (ours): the first approach with momentum audio-visual matching to alleviate overfit- ting and general silence in visual sounds localization;
Revised: [Yes] See thesupplemental material.(b)

Original: Consider a set of samples D = { ( v i , a i ) :
Revised: Did you specify all the training details (e

Original: i = 1 , . . . , N } , ground-truth maps G = {G i } Ni =1and predicted maps S
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: = {S i } Ni =1 .
Revised: [Yes] See Section 4.1.(c)

Original: In map based localization, each prediction S i is obtainedby first computing pixel-wise localization scores S i ∈ R H × W , and then applying a thresold α , S i ( α ) = { ( x, y ) | S i ( x, y ) > α } .
Revised: Did you report error bars (e

Original: To evaluate each prediction S i , prior work [1] relies on intersectionover union between S i and G i . IoU is calculated as [Equation] where G = { ( x, y ) | g xy > 0 } denotes the the set of pixels that represent sounding objects, andg xy ∈ [0 , 1] the ground-truth evidence that a sounding objects lies at location ( x, y ) .
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: Since in somedatasets ground-truth is collected from multiple annotators, consensus IoU (cIoU) is used instead. Refer to [1] for details on how multiple annotations are merged to compute g xy . Since we’re interested in assessing the model’s performance both when sounding objects are presentor not, we allow the model to make no predictions. To do this, in addition to the localization predictionmap S i , the model is asked to output a confidence score
Revised: [No] Training and test sets are too large to run each experiment

Original: c i . If the confidence score c i is too low(below a threshold δ ), the model predicts an empty set ( i.e ., the original S i is considered invalid). Predictions S i are considered correct if CIoU i are above a pre-specified threshold β . Under thesedefinitions, true positives, false positives and false negatives are computed as
Revised: multiple times.

Original: [Equation] [Equation] Precision To evaluate the localization accuracy of our models (given that sound sources exist), weuse precision at a CIoU threshold of β Precision ( β,
Revised: However, given their sizes, error bars are expected to be small.(d)

Original: [Equation] = [Equation] It should be noted, that most existing work [2, 3, 4, 6, 9, 5] already report Precision at β = 0 .
Revised: Did you include the total amount of compute and the type of resources used (e

Original: 5 underthe name of “CIoU”. We thus use this metric to compare to results reported in the original papers. Inour work, we also use a CIoU threshold of β = 0 . 5 for most experiments, and conduct an evaluationat higher values β to assess which methods can achieve higher quality predictions.
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: While precision provides a good metric to evaluate how accurate predictions are for samples withsounding objects, it does not assess how accurately models can ignore samples with NO soundingobjects. Thus, to comprehensively evaluate both sounding and non-sounding samples, we alsoevaluate our models using F1 score and average precision (AP). F1 score balances precision and recall, F
Revised: [Yes] See Section 4.1.

Original: [Equation] where [Equation] [Equation]
Revised: If you are using existing assets (e

Original: However, F1 ( β, δ ) depends on how strict the confidence threshold δ is set (the highest δ is set, themore samples are predicted as non-sounding).

Original: To find the optimal balance, we compute F1 ( β, δ ) forall values of δ and report the max-F1 score [Equation] Average precision (AP) is another metric often used in object detection. To compute AP, we closelyfollow [46].
Revised: g., code, data, models) or curating/releasing new assets...

Original: The only difference is that, when computing the Precision-Recall curve, we do notperform 11 point interpolation. We compute the full curve (without interpolation). Refer to [46] fordetails on AP computation. D Results on Flickr-10k
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: In order to demonstrate the effectiveness of the proposed MoVSL in mitigating overfitting issues onweakly-supervised visual sound source localization, we train the model on a small dataset with 10ksamples, Flickr-10k [1], following existing methods [1, 2, 6, 5]. The comparison results of usingbest and latest checkpoints on Flickr SoundNet testset are reported in Table 6. As can be seen, our MoVSL achieves the state-of-the-art results compared to existing methods in both settings.
Revised: [Yes] See Section 4.1.

Original: Using thebest checkpoint outperforms the current best results without OGL by 11.46%, 13.80%, and 19.35%in terms ofAP, max-F1, and Precision. Adding the OGL to the proposed approach also achievessignificant improvements. More important, there is no gap between the latest and best checkpoints forthe proposed MoVSL . These excellent results demonstrate the effectiveness of our method in mitigatingoverfitting issues in weakly-supervised visual sound source localization.
Revised: (b) Did you mention the license of the assets?

Original: [Table]
Revised: [N/A]

Original: [Table] [Table]
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Table]
Revised: [N/A]

Original: [Table] [Table] [Table] [Table]
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table] [Table] [Table] Attention10k
Revised: using/curating?

Original: [Table]
Revised: [N/A]

Original: [Table] [Table] [Table] [Table]
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table] [Table] [Table] [Table]
Revised: information or offensive content?

Original: [Table]
Revised: [N/A]

Original: To evaluate the generalization of our model beyond sound sources heard during training, we followprevious work [6, 5] and train the model on 70k data with 110 heard categories in VGG-Sounddataset [41]. Table 7 reports the comparison results on heard and unheard 110 classes. We achievesignificant improvements against previous methods.
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: For instance, using the latest checkpoint without OGL for testing heard 110 classes increases the baseline by 5.71%, 5.90%, and 5.22% in terms of AP, max-F1, and Precision. When it comes to unheard 110 classes, the proposed MoVSL without
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: OGL outperforms the current state-of-the-art approach by 4.10%, 3.80%, and 3.42% in terms of AP,max-F1, and Precision. New state-of-the-art results are achieved in all settings, demonstrating thegeneralization of our approach to unheard sounding categories in the training set. F Qualitative Results In order to qualitatively demonstrate the effectiveness of the proposed MoVSL , we compare visualizeattention maps from existing work and our model on various sizes of sounding objects in VGG-SS testset.
Revised: applicable?

Original: The qualitative results of of Attention10k [1], DMC [2], LVS [6], EZ-VSL [5], and the proposed MoVSL on huge/large/medium/small sounding objects are shown in Figure 4, 5, 6, and 7. Note thatresults of our MoVSL are shown on the last column. We can observe that the proposed MoVSL achievesdecent localization maps compared to previous approaches in all settings, such as the barking dog onrow 3 in Figure 4, the sounding cello on row 1 in Figure 5, and the crying baby on row 1 in Figure 6.
Revised: [N/A]

Original: For small objects in Figure 7, we also predict better maps for sounding objects than existing work.

Original: [Figure] [Figure] While our predictions seem to focus more on the actual sources, they are very large compared to theactual objects. G Broader Impact
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: This paper seeks to establish a more balanced evaluation protocol for visual sound source localization,that considers both cases with sounding and non-sounding objects. We hope the proposed metricswill lead to more balanced localization algorithms in the future. Improving sound source localization [Figure]
Revised: Board (IRB) approvals, if applicable?

Original: [Figure] methods can lead to interesting applications, for example, interfacing with blind or low-vision individuals as they navigate the world. However, our work still relies on data collected from internet sources,and thus is likely to hold biases that have not been identified.
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: While we believe current datasets andevaluation protocols are valuable for the development of sound source localization procedure, bettercuration of the datasets against nefarious bias should be conducted before deployment in real worldsettings. Advances in audio-visual localization can also be leveraged in surveillance applications,which can potentially have a negative societal impact.
Revised: spent on participant compensation? [N/A]