Original: A Closer Look at Weakly-Supervised Audio-Visual Source Localization
Revised: A Closer Look at Weakly-Supervised Audio-Visual Source Localization

Original: Abstract
Revised: Abstract

Original: Audio-visual source localization is a challenging task that aims to predict the location of visual sound sources in a video.
Revised: Audio-visual source localization is a challenging task that aims to predict the location of visual sound sources in a video.

Original: Since collecting ground-truth annotations of sounding objects can be costly, a plethora of weakly-supervised localization methods that can learn from datasets with no bounding-box annotations have been proposed in recent years, by leveraging the natural co-occurrence of audio and visual signals.
Revised: Since collecting ground-truth annotations of sounding objects can be costly, a plethora of weakly-supervised localization methods that can learn from datasets with no bounding-box annotations have been proposed in recent years, by leveraging the natural co-occurrence of audio and visual signals.

Original: Despite significant interest, popular evaluation protocols have two major flaws.
Revised: Despite significant interest, popular evaluation protocols have two major flaws.

Original: First, they allow for the use of a fully annotated dataset to perform early stopping, thus significantly increasing the annotation effort required for training.
Revised: First, they allow for the use of a fully annotated dataset to perform early stopping, thus significantly increasing the annotation effort required for training.

Original: Second, current evaluation metrics assume the presence of sound sources at all times.
Revised: Second, current evaluation metrics assume the presence of sound sources at all times.

Original: This is of course an unrealistic assumption, and thus better metrics are necessary to capture the model’s performance on (negative) samples with no visible sound sources.
Revised: This is of course an unrealistic assumption, and thus better metrics are necessary to capture the model’s performance on (negative) samples with no visible sound sources.

Original: To accomplish this, we extend the test set of popular benchmarks,
Revised: To accomplish this, we extend the test set of popular benchmarks,

Original: Flickr SoundNet and VGG-Sound Sources, in order to include negative samples, and measure performance using metrics that balance localization accuracy and recall.
Revised: Flickr SoundNet and VGG-Sound Sources, in order to include negative samples, and measure performance using metrics that balance localization accuracy and recall.

Original: Using the new protocol, we conducted an extensive evaluation of prior methods, and found that most prior works are not capable of identifying negatives and suffer from significant overfitting problems (rely heavily on early stopping for best results).
Revised: Using the new protocol, we conducted an extensive evaluation of prior methods, and found that most prior works are not capable of identifying negatives and suffer from significant overfitting problems (rely heavily on early stopping for best results).

Original: We also propose a new approach for visual sound source localization that addresses both these problems.
Revised: We also propose a new approach for visual sound source localization that addresses both these problems.

Original: In particular, we found that, through extreme visual dropout and the use of momentum encoders, the proposed approach combats overfitting effectively, and establishes a new state-of-the-art performance on both Flickr SoundNet and VGG-Sound Source.
Revised: In particular, we found that, through extreme visual dropout and the use of momentum encoders, the proposed approach combats overfitting effectively, and establishes a new state-of-the-art performance on both Flickr SoundNet and VGG-Sound Source.

Original: Code and pre-trained models are available at https://github.com/stoneMo/SLAVC .
Revised: Code and pre-trained models are available at https://github.com/stoneMo/SLAVC .

Original:  Introduction
Revised:  Introduction

Original: Humans and most other animals have evolved to localize sources of sound in their environment.
Revised: Humans and most other animals have evolved to localize sources of sound in their environment.

Original: This remarkable ability relies in part on the uniqueness of different sound sources, which allows us to recognize the sounds we hear and visually localize them in our environment.
Revised: This remarkable ability relies in part on the uniqueness of different sound sources, which allows us to recognize the sounds we hear and visually localize them in our environment.

Original: Given recent advances in audio and visual perception research, there is broad interest in developing multi-modal systems capable of mimicking our ability to visually localize sound sources.
Revised: Given recent advances in audio and visual perception research, there is broad interest in developing multi-modal systems capable of mimicking our ability to visually localize sound sources.

Original: One promising direction is to leverage the co-occurrence between sounds and the corresponding sources in video data.
Revised: One promising direction is to leverage the co-occurrence between sounds and the corresponding sources in video data.

Original: Since audio-visual co-occurrence arises naturally, algorithms can scale to very large datasets without requiring costly human annotations.
Revised: Since audio-visual co-occurrence arises naturally, algorithms can scale to very large datasets without requiring costly human annotations.

Original: However, despite encouraging recent progress [1, 2, 3, 4, 5], currently accepted evaluation protocols hide two critical limitations of current methods: (1) current methods overfit easily even when scaled up to large datasets, and (2) current methods assume that visible sound sources are always present in the video and thus are unreliable when deployed on realistic data where this assumption does not hold.
Revised: However, despite encouraging recent progress [1, 2, 3, 4, 5], currently accepted evaluation protocols hide two critical limitations of current methods: (1) current methods overfit easily even when scaled up to large datasets, and (2) current methods assume that visible sound sources are always present in the video and thus are unreliable when deployed on realistic data where this assumption does not hold.

Original: The first limitation remained hidden as prior works [2, 4, 6, 5] rely heavily on early stopping for optimal performance ( i.e ., by continuously validating the model during training using a human- annotated set).
Revised: The first limitation remained hidden as prior works [2, 4, 6, 5] rely heavily on early stopping for optimal performance ( i.e ., by continuously validating the model during training using a human- annotated set).

Original: However, this practice violates the main assumption of weakly supervised visual source localization, i.e ., the requirement for no bounding box annotations during training.
Revised: However, this practice violates the main assumption of weakly supervised visual source localization, i.e ., the requirement for no bounding box annotations during training.

Original: The second limitation remained hidden as most prominent benchmark datasets [1, 6] and evaluation metrics only assess the ability to localize sound sources when one is present in the video.
Revised: The second limitation remained hidden as most prominent benchmark datasets [1, 6] and evaluation metrics only assess the ability to localize sound sources when one is present in the video.

Original: Importantly, it ignores the ability to correctly predict the absence of visual sound sources.
Revised: Importantly, it ignores the ability to correctly predict the absence of visual sound sources.

Original: This has led to a bias towards localization accuracy with disregard for false positive detection.
Revised: This has led to a bias towards localization accuracy with disregard for false positive detection.

Original: False positive detection is closely related to the silent object detection problem highlighted in recent works [7, 8].
Revised: False positive detection is closely related to the silent object detection problem highlighted in recent works [7, 8].

Original: However, although these works introduce methods to suppress the localization of silent objects, they require the collection of a clean dataset containing a single source per video.
Revised: However, although these works introduce methods to suppress the localization of silent objects, they require the collection of a clean dataset containing a single source per video.

Original: Instead, we aim to tackle this issue without assuming knowledge of the number of sources.
Revised: Instead, we aim to tackle this issue without assuming knowledge of the number of sources.

Original: The two limitations above highlight the need for a more balanced and complete evaluation protocol for visual sound source localization.
Revised: The two limitations above highlight the need for a more balanced and complete evaluation protocol for visual sound source localization.

Original: To achieve this, we extend popular benchmark test sets (Flickr
Revised: To achieve this, we extend popular benchmark test sets (Flickr

Original: SoundNet [1] and VGG-Sound Sources [6]) to include ’negative’ samples without any visible sound sources.
Revised: SoundNet [1] and VGG-Sound Sources [6]) to include ’negative’ samples without any visible sound sources.

Original: We conduct an extensive evaluation of existing methods [2, 3, 4, 6, 9, 7], where in addition to overall localization accuracy, we also assess methods based on their ability to predict negative samples.
Revised: We conduct an extensive evaluation of existing methods [2, 3, 4, 6, 9, 7], where in addition to overall localization accuracy, we also assess methods based on their ability to predict negative samples.

Original: We observe that all prior work suffers from significant overfitting problems, relying on early stopping for optimal performance.
Revised: We observe that all prior work suffers from significant overfitting problems, relying on early stopping for optimal performance.

Original: We also found that previous approaches struggle to strike a good balance between false positive and false negative rates.
Revised: We also found that previous approaches struggle to strike a good balance between false positive and false negative rates.

Original: We also propose a novel procedure - Simultaneous Localization and Audio-Visual Correspondence ( SLAVC ) - which provably tackles these two issues.
Revised: We also propose a novel procedure - Simultaneous Localization and Audio-Visual Correspondence ( SLAVC ) - which provably tackles these two issues.

Original: First, to combat overfitting, we adopt slow-moving momentum target encoders and extreme visual dropout.
Revised: First, to combat overfitting, we adopt slow-moving momentum target encoders and extreme visual dropout.

Original: Second, to reduce false positives, we localize visual sources by forcing the model to explicitly perform audio visual correspondence in addition to localization.
Revised: Second, to reduce false positives, we localize visual sources by forcing the model to explicitly perform audio visual correspondence in addition to localization.

Original: The latter term highlights which regions within an image are most associated with a particular audio, while the former downplays regions that can be better described by the audio of other samples.
Revised: The latter term highlights which regions within an image are most associated with a particular audio, while the former downplays regions that can be better described by the audio of other samples.

Original: By combining these two terms, the model is able to both accurately localize visible sources and identify when no visible sources are present.
Revised: By combining these two terms, the model is able to both accurately localize visible sources and identify when no visible sources are present.

Original: Using the newly developed evaluation protocol, we show that, unlike all prior work, SLAVC does not overfit and thus can be trained without relying on early stopping.
Revised: Using the newly developed evaluation protocol, we show that, unlike all prior work, SLAVC does not overfit and thus can be trained without relying on early stopping.

Original: SLAVC achieves state-of-the-art performance on multiple datasets and is more accurate at identifying samples without any visible sound sources.
Revised: SLAVC achieves state-of-the-art performance on multiple datasets and is more accurate at identifying samples without any visible sound sources.

Original:  Related Work
Revised:  Related Work

Original: Audio-Visual Self-Supervised Learning.
Revised: Audio-Visual Self-Supervised Learning.

Original: The natural audio-visual alignment is a rich source of supervision for self-supervised learning [10].
Revised: The natural audio-visual alignment is a rich source of supervision for self-supervised learning [10].

Original: Recently, it has been explored to learn a wide variety of deep learning models [11, 12, 13, 1, 14, 15, 16, 17, 18, 19].
Revised: Recently, it has been explored to learn a wide variety of deep learning models [11, 12, 13, 1, 14, 15, 16, 17, 18, 19].

Original: Given a database of videos, the main idea is to close the distance between audio and visual features from the same video while pushing away those from different videos [20, 19, 21], from the same video but different timestamp [22, 23], or from the same video but difference spatial location [24].
Revised: Given a database of videos, the main idea is to close the distance between audio and visual features from the same video while pushing away those from different videos [20, 19, 21], from the same video but different timestamp [22, 23], or from the same video but difference spatial location [24].

Original: Such audio-visual alignment is beneficial to several tasks, such as audio separation [18, 25, 26, 15, 16, 27, 28, 29], audio-visual spatialization [14, 30, 31, 24], visual sound source localization [1, 2, 3, 4, 6, 32].
Revised: Such audio-visual alignment is beneficial to several tasks, such as audio separation [18, 25, 26, 15, 16, 27, 28, 29], audio-visual spatialization [14, 30, 31, 24], visual sound source localization [1, 2, 3, 4, 6, 32].

Original: In this work, we mainly focus on visual source localization, which requires learning fine-grained and high-resolution representations that are discriminative of the various sound sources.
Revised: In this work, we mainly focus on visual source localization, which requires learning fine-grained and high-resolution representations that are discriminative of the various sound sources.

Original: Audio-Visual Source Localization.
Revised: Audio-Visual Source Localization.

Original: Audio-visual source localization aims at predicting the location of sounding objects in a video.
Revised: Audio-visual source localization aims at predicting the location of sounding objects in a video.

Original: Early models [33, 34, 35] learn to capture the low-level correspondences between audio and visual features.
Revised: Early models [33, 34, 35] learn to capture the low-level correspondences between audio and visual features.

Original: Recently, contrastive approaches [1, 23, 2, 6, 9] seek to localize objects by aligning audio and visual representation spaces.
Revised: Recently, contrastive approaches [1, 23, 2, 6, 9] seek to localize objects by aligning audio and visual representation spaces.

Original: For instance, LVS [6] leveraged a contrastive loss with hard negative mining to learn the audio-visual co-occurrence map discriminatively.
Revised: For instance, LVS [6] leveraged a contrastive loss with hard negative mining to learn the audio-visual co-occurrence map discriminatively.

Original: Contrastive learning with hard positives was introduced in HardPos [9] to learn audio-visual alignment with negative samples.
Revised: Contrastive learning with hard positives was introduced in HardPos [9] to learn audio-visual alignment with negative samples.

Original: EZ-VSL [5] introduced a multiple instance contrastive learning framework that focus only on the most aligned regions when matching the audio to the video.
Revised: EZ-VSL [5] introduced a multiple instance contrastive learning framework that focus only on the most aligned regions when matching the audio to the video.

Original: However, we show that these methods can easily overfit in the source localization task, and heavily rely on early stopping for best performance.
Revised: However, we show that these methods can easily overfit in the source localization task, and heavily rely on early stopping for best performance.

Original: Furthermore, with the exception of [7, 8], they focus on localizing sound sources that are visible, and struggle to identify negatives (when there are no visible sources).
Revised: Furthermore, with the exception of [7, 8], they focus on localizing sound sources that are visible, and struggle to identify negatives (when there are no visible sources).

Original: Although understanding when sources are not visible is important to several audio-visual tasks, like in open-domain audio-visual source separation [36, 37], this issue still poses significant challenges.
Revised: Although understanding when sources are not visible is important to several audio-visual tasks, like in open-domain audio-visual source separation [36, 37], this issue still poses significant challenges.

Original: DSOL [7] and IEr [8] proposed a method to suppress localization of silent objects, and
Revised: DSOL [7] and IEr [8] proposed a method to suppress localization of silent objects, and

Original: Simultaneous Localization and Audio-Visual Correspondence (Ours)
Revised: Simultaneous Localization and Audio-Visual Correspondence (Ours)

Original: [Figure]
Revised: [Figure]

Original: l 𝐸𝑀𝐴(𝑓 ! )
Revised: l 𝐸𝑀𝐴(𝑓 ! )

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: Projections Projections
Revised: Projections Projections

Original: [Figure]
Revised: [Figure]

Original: The audio-visual correspondence of pixel-wise similarity between momentum branches and base branches is aligned by the cross-modal multiple instance contrastive learning objective.
Revised: The audio-visual correspondence of pixel-wise similarity between momentum branches and base branches is aligned by the cross-modal multiple instance contrastive learning objective.

Original: an evaluation metric that measures the ability to ignore them.
Revised: an evaluation metric that measures the ability to ignore them.

Original: These methods, however, rely on the knowledge of the number of sound sources, not available in most large scale datasets.
Revised: These methods, however, rely on the knowledge of the number of sound sources, not available in most large scale datasets.

Original: In this work, we do not assume such knowledge.
Revised: In this work, we do not assume such knowledge.

Original: We propose a novel Simultaneous Instance Discrimination and
Revised: We propose a novel Simultaneous Instance Discrimination and

Original: Localization framework for weakly-supervised visual source localization with clear benefits, as it demonstrably improves both localization accuracy and false positive rates.
Revised: Localization framework for weakly-supervised visual source localization with clear benefits, as it demonstrably improves both localization accuracy and false positive rates.

Original: We also annotate real world test samples that may lead to false positives (beyond silent objects), and propose new metrics for a more comprehensive evaluation.
Revised: We also annotate real world test samples that may lead to false positives (beyond silent objects), and propose new metrics for a more comprehensive evaluation.

Original: Weakly-supervised Object Detection (WSOD).
Revised: Weakly-supervised Object Detection (WSOD).

Original: Weakly-supervised object detection [38, 39, 40, 41] is closely related to visual sound localization, since both aim at localizing object regions by learning from image/frame level supervision (corresponding audio in the case of VSL, and object classes for
Revised: Weakly-supervised object detection [38, 39, 40, 41] is closely related to visual sound localization, since both aim at localizing object regions by learning from image/frame level supervision (corresponding audio in the case of VSL, and object classes for

Original: WSOD).
Revised: WSOD).

Original: However, different from WSOD, we assume no access to class information during training.
Revised: However, different from WSOD, we assume no access to class information during training.

Original: Instead, SLAVC aims at learning from audio-visual correspondence alone.
Revised: Instead, SLAVC aims at learning from audio-visual correspondence alone.

Original:  Visual Source Localization
Revised:  Visual Source Localization

Original: We propose a self-supervised learning approach for Visual Sound Localization (VSL).
Revised: We propose a self-supervised learning approach for Visual Sound Localization (VSL).

Original: This method builds on the current state-of-the-art based on multiple-instance contrastive learning [5].
Revised: This method builds on the current state-of-the-art based on multiple-instance contrastive learning [5].

Original: The proposed approach addresses two critical problems: 1) severe overfitting even when trained on large datasets and 2) the tendency to hallucinate sound sources when none are visible in the video.
Revised: The proposed approach addresses two critical problems: 1) severe overfitting even when trained on large datasets and 2) the tendency to hallucinate sound sources when none are visible in the video.

Original: For clear exposition, we begin by defining the problem and briefly revisit the current state-of-the-art [5] (Sec. 3.1).
Revised: For clear exposition, we begin by defining the problem and briefly revisit the current state-of-the-art [5] (Sec. 3.1).

Original: We then detail the proposed approach in Sec. 3.2, and highlight the main differences to prior work.
Revised: We then detail the proposed approach in Sec. 3.2, and highlight the main differences to prior work.

Original: Finally, we present in 4 an evaluation protocol that is sensitive to these issues.
Revised: Finally, we present in 4 an evaluation protocol that is sensitive to these issues.

Original: 3. Preliminaries: Weakly-Supervised Visual Source Localization
Revised: 3. Preliminaries: Weakly-Supervised Visual Source Localization

Original: Given an audio-visual dataset D = { ( v i , a i ) : i = 1 , . . . , N } , VSL seeks to train a system that can predict the location of the sources present in sound a i within the visual frame v i .
Revised: Given an audio-visual dataset D = { ( v i , a i ) : i = 1 , . . . , N } , VSL seeks to train a system that can predict the location of the sources present in sound a i within the visual frame v i .

Original: VSL models can be learned from various levels of supervision:
Revised: VSL models can be learned from various levels of supervision:

Original: Unsupervised VSL learns to localize without any form of human annotations; Weakly-supervised VSL forgo bounding-box supervision but may leverage categorical information.
Revised: Unsupervised VSL learns to localize without any form of human annotations; Weakly-supervised VSL forgo bounding-box supervision but may leverage categorical information.

Original: This categorical information can be in the form of object labels for visual encoder pre-training, or audio event labels for audio encoder pre-training; Semi-supervised
Revised: This categorical information can be in the form of object labels for visual encoder pre-training, or audio event labels for audio encoder pre-training; Semi-supervised

Original: VSL learns from a small number of bounding-box annotations, together with a large unsupervised or weakly-supervised dataset; Finally, Fully supervised VSL models rely on large amounts of fully annotated datasets.
Revised: VSL learns from a small number of bounding-box annotations, together with a large unsupervised or weakly-supervised dataset; Finally, Fully supervised VSL models rely on large amounts of fully annotated datasets.

Original: In this paper, we tackle the Weakly-Supervised VSL problem.
Revised: In this paper, we tackle the Weakly-Supervised VSL problem.

Original: We note that equivalent prior work, such as [1, 4, 2, 3, 7, 6, 9, 5], often refer to this problem as unsupervised VSL, despite actually addressing the weakly supervised problem, as they use vision models pre-trained on
Revised: We note that equivalent prior work, such as [1, 4, 2, 3, 7, 6, 9, 5], often refer to this problem as unsupervised VSL, despite actually addressing the weakly supervised problem, as they use vision models pre-trained on

Original: ImageNet for object recognition.
Revised: ImageNet for object recognition.

Original: To train the system, the current state-of-the-art method, EZ-VSL [5] maps both audio and visual features into a common space using projection heads g a ( · ) and g v ( · ) , resulting in a set of visual features spanning all locations in an image
Revised: To train the system, the current state-of-the-art method, EZ-VSL [5] maps both audio and visual features into a common space using projection heads g a ( · ) and g v ( · ) , resulting in a set of visual features spanning all locations in an image

Original: V i
Revised: V i

Original: = { g v ( v xyi ) : ∀ x, y } and one global audio feature g a ( a i ) .
Revised: = { g v ( v xyi ) : ∀ x, y } and one global audio feature g a ( a i ) .

Original: Ideally, the model should be trained to align the audio and visual features at locations ( x, y ) , where sound sources are located.
Revised: Ideally, the model should be trained to align the audio and visual features at locations ( x, y ) , where sound sources are located.

Original: However, since these locations are unknown during training, EZVSL [5] optimizes a multiple-instance contrastive learning objective, in which the audio representation g a ( a i ) is matched with at least one location in the corresponding bag of visual features V i , while pushing away from the visual features in all "negative" bags in the same mini-batch V j ∀ j ̸
Revised: However, since these locations are unknown during training, EZVSL [5] optimizes a multiple-instance contrastive learning objective, in which the audio representation g a ( a i ) is matched with at least one location in the corresponding bag of visual features V i , while pushing away from the visual features in all "negative" bags in the same mini-batch V j ∀ j ̸

Original: = i (at all spatial locations).
Revised: = i (at all spatial locations).

Original: Specifically, the model is trained to minimize the average per sample loss
Revised: Specifically, the model is trained to minimize the average per sample loss

Original: [Equation]
Revised: [Equation]

Original: MICL
Revised: MICL

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: where τ is a temperature hyper-parameter, B is the batch-size, and the similarity sim ( a i , V j ) between an audio feature a i and a bag of visual features V j is computed by max-pooling audio-visual cosine similarities s () across spatial locations x, y
Revised: where τ is a temperature hyper-parameter, B is the batch-size, and the similarity sim ( a i , V j ) between an audio feature a i and a bag of visual features V j is computed by max-pooling audio-visual cosine similarities s () across spatial locations x, y

Original: [Equation]
Revised: [Equation]

Original: 3. Simultaneous Localization and Audio-Visual Correspondence ( SLAVC )
Revised: 3. Simultaneous Localization and Audio-Visual Correspondence ( SLAVC )

Original: Previous methods, including EZ-VSL [5], suffer from two critical limitations: (1) models easily overfit to the self-supervised objective; (2) models are not capable of identifying negative samples ( i.e ., samples with no visible sound sources).
Revised: Previous methods, including EZ-VSL [5], suffer from two critical limitations: (1) models easily overfit to the self-supervised objective; (2) models are not capable of identifying negative samples ( i.e ., samples with no visible sound sources).

Original: The method proposed in this work is designed to address these two limitations.
Revised: The method proposed in this work is designed to address these two limitations.

Original: To combat overfitting, we apply two regularization techniques: dropout [42] and (slow-moving) momentum encoders [43, 44].
Revised: To combat overfitting, we apply two regularization techniques: dropout [42] and (slow-moving) momentum encoders [43, 44].

Original: To better identify negatives, we propose to conduct
Revised: To better identify negatives, we propose to conduct

Original: VSL by simultaneously performing sounding region localization and audio-visual correspondence.
Revised: VSL by simultaneously performing sounding region localization and audio-visual correspondence.

Original: These three components are illustrated in Figure 1, and are now discussed separately.
Revised: These three components are illustrated in Figure 1, and are now discussed separately.

Original: Dropout
Revised: Dropout

Original: [42] is a widely used technique to combat overfitting.
Revised: [42] is a widely used technique to combat overfitting.

Original: We applied dropout on the output of both the f v and f a encoders.
Revised: We applied dropout on the output of both the f v and f a encoders.

Original: Interestingly, we found that heavy visual dropout is required to prevent overfitting (with a dropout probability as large as 0 . 9 ), while audio dropout is not required.
Revised: Interestingly, we found that heavy visual dropout is required to prevent overfitting (with a dropout probability as large as 0 . 9 ), while audio dropout is not required.

Original: Since multiple-instance contrastive learning (Eq. 1) requires the audio to match only one of a large number of locations in the image ( h × w ), spurious alignment becomes more likely.
Revised: Since multiple-instance contrastive learning (Eq. 1) requires the audio to match only one of a large number of locations in the image ( h × w ), spurious alignment becomes more likely.

Original: Visual dropout reduces the likelihood of spurious alignments, as visual features become more spatially redundant.
Revised: Visual dropout reduces the likelihood of spurious alignments, as visual features become more spatially redundant.

Original: Momentum encoders
Revised: Momentum encoders

Original: [43, 44] is a technique often used in self-supervised and semi-supervised learning to obtain slow-moving target representations, leading to more stable self-training and enhanced representations.
Revised: [43, 44] is a technique often used in self-supervised and semi-supervised learning to obtain slow-moving target representations, leading to more stable self-training and enhanced representations.

Original: We apply momentum encoders to both audio and visual inputs ˆ a i
Revised: We apply momentum encoders to both audio and visual inputs ˆ a i

Original: =
Revised: =

Original: ˆ f a ( a i ) , ˆ v i
Revised: ˆ f a ( a i ) , ˆ v i

Original: = ˆ f v ( v i ) , in order to obtain more stable targets.
Revised: = ˆ f v ( v i ) , in order to obtain more stable targets.

Original: Following [43, 44], momentum encoders are updated using an exponential moving average of the corresponding online encoders with coefficient m .
Revised: Following [43, 44], momentum encoders are updated using an exponential moving average of the corresponding online encoders with coefficient m .

Original: 2 Since updates to the audio-visual encoders are slowly incorporated into the momentum encoders, the target representations display a smoother behavior during the training process.
Revised: 2 Since updates to the audio-visual encoders are slowly incorporated into the momentum encoders, the target representations display a smoother behavior during the training process.

Original: Simultaneous localization and audio-visual correspondence To better avoid false predictions, we explicitly force the model to only predict regions of an image that can be used to identify the corresponding sound source.
Revised: Simultaneous localization and audio-visual correspondence To better avoid false predictions, we explicitly force the model to only predict regions of an image that can be used to identify the corresponding sound source.

Original: Specifically, we factorize VSL into two terms.
Revised: Specifically, we factorize VSL into two terms.

Original: The first is a localization term P loc that distinguishes regions within an image that likely depict the sound source from the regions that likely do not.
Revised: The first is a localization term P loc that distinguishes regions within an image that likely depict the sound source from the regions that likely do not.

Original: The second is an audio-visual correspondence term P avc that highlights regions that are likely associated with the corresponding audio, and suppresses regions that would be better explained by other sound sources.
Revised: The second is an audio-visual correspondence term P avc that highlights regions that are likely associated with the corresponding audio, and suppresses regions that would be better explained by other sound sources.

Original: The key insight is that to prevent false detections, VSL should select only the regions that are simultaneously highlighted in both terms.
Revised: The key insight is that to prevent false detections, VSL should select only the regions that are simultaneously highlighted in both terms.

Original: While the first term is responsible for sound source localization, the second prevents VSL to be overly confident when no sound sources are visible in the image.
Revised: While the first term is responsible for sound source localization, the second prevents VSL to be overly confident when no sound sources are visible in the image.

Original: Localization and audio-visual correspondence is conducted on separate subspaces.
Revised: Localization and audio-visual correspondence is conducted on separate subspaces.

Original: In particular, let g loc a ( a i ) and g avc a ( a i ) be the two audio representations of a i , and { g loc v ( v xyi ) :
Revised: In particular, let g loc a ( a i ) and g avc a ( a i ) be the two audio representations of a i , and { g loc v ( v xyi ) :

Original: ∀ x, y } and { g avc v ( v xyi ) : ∀ x, y } the two bags of visual features for v i .
Revised: ∀ x, y } and { g avc v ( v xyi ) : ∀ x, y } the two bags of visual features for v i .

Original: We use linear projections for the various functions g ( · ) .
Revised: We use linear projections for the various functions g ( · ) .

Original: Then, we define the localization term as the softmax ρ xy ( · ) over spatial dimensions x, y
Revised: Then, we define the localization term as the softmax ρ xy ( · ) over spatial dimensions x, y

Original: [Equation]
Revised: [Equation]

Original: and the audio-visual correspondence term as the softmax
Revised: and the audio-visual correspondence term as the softmax

Original: ρ i ( · ) over instances
Revised: ρ i ( · ) over instances

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: To select the regions in which both P loc and P avc are active, the two terms are combined into a single prediction map P ( a i , v xyj )
Revised: To select the regions in which both P loc and P avc are active, the two terms are combined into a single prediction map P ( a i , v xyj )

Original: =
Revised: =

Original: P loc
Revised: P loc

Original: [Equation]
Revised: [Equation]

Original: a i , v xyj ) · P avc ( a i , v xyj ) , and the model is trained to optimize
Revised: a i , v xyj ) · P avc ( a i , v xyj ) , and the model is trained to optimize

Original: [Equation]
Revised: [Equation]

Original: SLAVC
Revised: SLAVC

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Full method While we demonstrate the benefits of the three components separately, they can be combined for optimal performance.
Revised: Full method While we demonstrate the benefits of the three components separately, they can be combined for optimal performance.

Original: The full approach is illustrated in Fig.
Revised: The full approach is illustrated in Fig.

Original: Dropout is always applied on the encoders’ outputs.
Revised: Dropout is always applied on the encoders’ outputs.

Original: To combine the SLAVC factorization with momentum encoders, the model is trained to optimize
Revised: To combine the SLAVC factorization with momentum encoders, the model is trained to optimize

Original: [Equation]
Revised: [Equation]

Original: where ˆ a i and ˆ v xyj are the audio and visual momentum features.
Revised: where ˆ a i and ˆ v xyj are the audio and visual momentum features.

Original: During inference, both localization and audio-visual correspondence terms are used for VSL.
Revised: During inference, both localization and audio-visual correspondence terms are used for VSL.

Original: We observed that matching momentum features at test time lead to improved localization.
Revised: We observed that matching momentum features at test time lead to improved localization.

Original: Thus, given an audio-visual pair ( v, a ) , the audio-visual localization map at location xy is computed as
Revised: Thus, given an audio-visual pair ( v, a ) , the audio-visual localization map at location xy is computed as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Furthermore, this audio-visual localization map can be easily combined with object-guided localization proposed in [5] for improved performance.
Revised: Furthermore, this audio-visual localization map can be easily combined with object-guided localization proposed in [5] for improved performance.

Original:  Benchmarking Visual Source Localization
Revised:  Benchmarking Visual Source Localization

Original: We introduce an evaluation protocol for VSL that is more sensitive to the high false positives and overfitting issues of current approaches.
Revised: We introduce an evaluation protocol for VSL that is more sensitive to the high false positives and overfitting issues of current approaches.

Original: First, to ensure overfitting is not hidden by the evaluation protocol, we suggest to rule out early stopping from weakly-supervised VSL evaluation, and instead always evaluate models after training them to convergence (or a large number of iterations).
Revised: First, to ensure overfitting is not hidden by the evaluation protocol, we suggest to rule out early stopping from weakly-supervised VSL evaluation, and instead always evaluate models after training them to convergence (or a large number of iterations).

Original: Note that early stopping defeats the purpose of weakly-supervised VSL, as it requires a fully annotated evaluation subset for tracking performance.
Revised: Note that early stopping defeats the purpose of weakly-supervised VSL, as it requires a fully annotated evaluation subset for tracking performance.

Original: Second, to assess false detection of non-existing sources, we extend the test sets of both FlickrSoundNet [1] and VGG Sound Sources [45] to include samples without visible sound sources.
Revised: Second, to assess false detection of non-existing sources, we extend the test sets of both FlickrSoundNet [1] and VGG Sound Sources [45] to include samples without visible sound sources.

Original: We also use metrics that measure the balance between high localization accuracy and low false positive rates.
Revised: We also use metrics that measure the balance between high localization accuracy and low false positive rates.

Original: Extended Flickr-SoundNet/VGG-SS Non-visible sounds or frames with silent objects are prevalent in video.
Revised: Extended Flickr-SoundNet/VGG-SS Non-visible sounds or frames with silent objects are prevalent in video.

Original: To tackle these cases, we present a new evaluation protocol.
Revised: To tackle these cases, we present a new evaluation protocol.

Original: We extended VGGSS/Flickr-SoundNet by merging clips with no sounding objects to the original test sets.
Revised: We extended VGGSS/Flickr-SoundNet by merging clips with no sounding objects to the original test sets.

Original: Specifically,
Revised: Specifically,

Original: [Table]
Revised: [Table]

Original: we analyzed 1000 videos from VGG-Sound test set (and 250 from Flickr-SoundNet test set), and manually select 5-second clips with non-audible frames and/or non-visible sound sources.
Revised: we analyzed 1000 videos from VGG-Sound test set (and 250 from Flickr-SoundNet test set), and manually select 5-second clips with non-audible frames and/or non-visible sound sources.

Original: This resulted in 379/42 samples with no sounding sources for VGG-SS/Flickr-SoundNet, respectively.
Revised: This resulted in 379/42 samples with no sounding sources for VGG-SS/Flickr-SoundNet, respectively.

Original: Beyond the manually identified negative pairs, we further generate negative samples by pairing audio and videos that do not belong together.
Revised: Beyond the manually identified negative pairs, we further generate negative samples by pairing audio and videos that do not belong together.

Original: We control the difficulty of these negative pairs, by sampling 25% of pairs from audio and video that are from the same class (hard negative), and 75% from different classes.
Revised: We control the difficulty of these negative pairs, by sampling 25% of pairs from audio and video that are from the same class (hard negative), and 75% from different classes.

Original: We merge all negatives with the VGG-SS [6] and Flickr-SoundNet [1] test set.
Revised: We merge all negatives with the VGG-SS [6] and Flickr-SoundNet [1] test set.

Original: Table 1 shows the statistics of the extended test sets.
Revised: Table 1 shows the statistics of the extended test sets.

Original: We also split test samples into four groups (Small, Medium, Large, and Huge) according to the size of the sound sources, as measured by the area (in pixels) occupied by the ground-truth bounding boxes.
Revised: We also split test samples into four groups (Small, Medium, Large, and Huge) according to the size of the sound sources, as measured by the area (in pixels) occupied by the ground-truth bounding boxes.

Original: Evaluation metrics Localization maps are often evaluated by comparing them to a group of human annotations using consensus intersection over union (cIoU, denoted as u ) [1].
Revised: Evaluation metrics Localization maps are often evaluated by comparing them to a group of human annotations using consensus intersection over union (cIoU, denoted as u ) [1].

Original: Given a set of predictions with cIoUs U = { u i } Ni =1 , prior work [2, 3, 4, 6, 9, 5] measures the localization accuracy (LocAcc) among all samples with visible sounding objects, where each prediction is considered to be correct if its cIoU is above the cIoU threshold γ 3 .
Revised: Given a set of predictions with cIoUs U = { u i } Ni =1 , prior work [2, 3, 4, 6, 9, 5] measures the localization accuracy (LocAcc) among all samples with visible sounding objects, where each prediction is considered to be correct if its cIoU is above the cIoU threshold γ 3 .

Original: The cIoU threshold is set at γ = 0 .
Revised: The cIoU threshold is set at γ = 0 .

Original: 5 unless otherwise specified.
Revised: 5 unless otherwise specified.

Original: Beyond localization error, we also evaluate on samples with no visible sounding sources.
Revised: Beyond localization error, we also evaluate on samples with no visible sounding sources.

Original: Thus, the model cannot assume the presence of a sound source, it is required to predict whether the current video contains a visible source or not.
Revised: Thus, the model cannot assume the presence of a sound source, it is required to predict whether the current video contains a visible source or not.

Original: This is accomplished by computing a confidence score d i , which we define as the maximum value in the predicted audio-visual similarity map, max xy S xy AVL .
Revised: This is accomplished by computing a confidence score d i , which we define as the maximum value in the predicted audio-visual similarity map, max xy S xy AVL .

Original: For evaluation, we define a flag c i to be 1 for samples with visible sources (positive samples) and 0 for samples with no visible sources (negative samples).
Revised: For evaluation, we define a flag c i to be 1 for samples with visible sources (positive samples) and 0 for samples with no visible sources (negative samples).

Original: True positive are then given as T P =
Revised: True positive are then given as T P =

Original: { i | c i = 1 , d i > δ, u i > γ } , false positives as FP =
Revised: { i | c i = 1 , d i > δ, u i > γ } , false positives as FP =

Original: { i | c i = 1 , d i > δ, u i ≤ γ } ∪ { i | c i = 0 , d i > δ } , and false negatives as FN =
Revised: { i | c i = 1 , d i > δ, u i ≤ γ } ∪ { i | c i = 0 , d i > δ } , and false negatives as FN =

Original: { i | c i = 1 , d i ≤ δ } .
Revised: { i | c i = 1 , d i ≤ δ } .

Original: These sets are used to compute the Average
Revised: These sets are used to compute the Average

Original: Precision (AP), and the maximum F1 (max-F1) score obtained by sweeping the confidence threshold ( i.e . max δ F 1( δ ) ).
Revised: Precision (AP), and the maximum F1 (max-F1) score obtained by sweeping the confidence threshold ( i.e . max δ F 1( δ ) ).

Original: Detailed formulas are provided in appendix.
Revised: Detailed formulas are provided in appendix.

Original:  Experiments
Revised:  Experiments

Original: Using the evaluation protocol outlined above, we now show that prior works easily overfit and are prone to high false positive rates.
Revised: Using the evaluation protocol outlined above, we now show that prior works easily overfit and are prone to high false positive rates.

Original: We also show that the proposed SLAVC effectively addresses these issues, outperforming prior state-of-the-art by large margins.
Revised: We also show that the proposed SLAVC effectively addresses these issues, outperforming prior state-of-the-art by large margins.

Original: Next, we conduct a thorough analysis of the various components of the system, and identify the main limitations of both ours and prior work, namely, small objects and high-quality detection.
Revised: Next, we conduct a thorough analysis of the various components of the system, and identify the main limitations of both ours and prior work, namely, small objects and high-quality detection.

Original: 5.1 Experimental setup
Revised: 5.1 Experimental setup

Original: Datasets We evaluate the effectiveness of the proposed method on two datasets - Flickr SoundNet [1] and VGG Sound Sources [45].
Revised: Datasets We evaluate the effectiveness of the proposed method on two datasets - Flickr SoundNet [1] and VGG Sound Sources [45].

Original: Following commonly-used settings [6, 9, 5], we use a subset of 144k samples for training in both cases.
Revised: Following commonly-used settings [6, 9, 5], we use a subset of 144k samples for training in both cases.

Original: We evaluate on the extended test sets described in Sec. 4.
Revised: We evaluate on the extended test sets described in Sec. 4.

Original: Models and optimization We followed prior work and used ResNet-18 [46] for both the audio and visual encoders.
Revised: Models and optimization We followed prior work and used ResNet-18 [46] for both the audio and visual encoders.

Original: The visual encoder is initialized with ImageNet [47] pre-trained weights [6, 9, 5].
Revised: The visual encoder is initialized with ImageNet [47] pre-trained weights [6, 9, 5].

Original: The output dimensions of the audio and visual encoders ( i.e ., the output of projection functions g () ) was kept at 512, the momentum encoders update factor at 0 . 999 , and the visual dropout at 0 .
Revised: The output dimensions of the audio and visual encoders ( i.e ., the output of projection functions g () ) was kept at 512, the momentum encoders update factor at 0 . 999 , and the visual dropout at 0 .

Original: No audio dropout is applied.
Revised: No audio dropout is applied.

Original: Models are trained with a batch size of 128 on 2 GPUs for 20 epochs (which
Revised: Models are trained with a batch size of 128 on 2 GPUs for 20 epochs (which

Original: [Table] [Table]
Revised: [Table] [Table]

Original: we found to be enough to achieve convergence in most cases).
Revised: we found to be enough to achieve convergence in most cases).

Original: We used the Adam [48] optimizer with β 1 = 0 .
Revised: We used the Adam [48] optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , learning rate of 1 e − 4 and weight decay of 1 e − 4 .
Revised: 999 , learning rate of 1 e − 4 and weight decay of 1 e − 4 .

Original: Our implementation, available at https://github.com/stoneMo/SLAVC
Revised: Our implementation, available at https://github.com/stoneMo/SLAVC

Original: [Table]
Revised: [Table]

Original: is based on PyTorch [49] deep learning tool
Revised: is based on PyTorch [49] deep learning tool

Original: [Table]
Revised: [Table]

Original: Audio and visual processing
Revised: Audio and visual processing

Original: We extract audio-visual pairs composed of a single frame and 3 seconds of audio centered around the frame.
Revised: We extract audio-visual pairs composed of a single frame and 3 seconds of audio centered around the frame.

Original: The visual frame is resized into 256 along the shortest edge, followed by random cropping and random horizontal flipping.
Revised: The visual frame is resized into 256 along the shortest edge, followed by random cropping and random horizontal flipping.

Original: During inference, frames are resized into 224 × 224, with no additional data augmentations.
Revised: During inference, frames are resized into 224 × 224, with no additional data augmentations.

Original: For the audio, we extract log spectrograms with 257 frequency bands over 300 timesteps from 3s of audio at 22050 kHz.
Revised: For the audio, we extract log spectrograms with 257 frequency bands over 300 timesteps from 3s of audio at 22050 kHz.

Original: The underlying short-term
Revised: The underlying short-term

Original: Fourier transform is computed on approximately 25ms windows with a step size of 10ms.
Revised: Fourier transform is computed on approximately 25ms windows with a step size of 10ms.

Original: Prior works and baselines We compare the proposed approach to several prior VSL methods.
Revised: Prior works and baselines We compare the proposed approach to several prior VSL methods.

Original: Specifically, we considered Attention 10k [1], CoarsetoFine [4], DMC [2], DSOL [7], LVS [6],
Revised: Specifically, we considered Attention 10k [1], CoarsetoFine [4], DMC [2], DSOL [7], LVS [6],

Original: HardPose [9] and EZ-VSL [5].
Revised: HardPose [9] and EZ-VSL [5].

Original: We used authors’ implementations when available, and our own implementations otherwise.
Revised: We used authors’ implementations when available, and our own implementations otherwise.

Original: To ensure that our results match the original papers, we report both results when possible in Tab. 2.
Revised: To ensure that our results match the original papers, we report both results when possible in Tab. 2.

Original: For the current state-of-the-art EZ-VSL [5], we consider two versions: with and without object guided localization (OGL).
Revised: For the current state-of-the-art EZ-VSL [5], we consider two versions: with and without object guided localization (OGL).

Original: OGL computes an object prior which is merged with the audio-visual localization prediction for improved localization.
Revised: OGL computes an object prior which is merged with the audio-visual localization prediction for improved localization.

Original: We also use OGL together with our approach.
Revised: We also use OGL together with our approach.

Original: As a sanity check, we also evaluate a Center Prior baseline, which always selects a circle centered in the middle of the image.
Revised: As a sanity check, we also evaluate a Center Prior baseline, which always selects a circle centered in the middle of the image.

Original: Since when recording, we tend to put objects in the middle of the frame, this baseline provides a good low bound for localization performance, which
Revised: Since when recording, we tend to put objects in the middle of the frame, this baseline provides a good low bound for localization performance, which

Original: VSL methods should outperform.
Revised: VSL methods should outperform.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 5.2 Main results
Revised: 5.2 Main results

Original: Preventing overfitting To demonstrate that current methods suffer from severe overfitting, we compare several methods trained with and without early stopping [1, 2, 3, 4, 6, 9, 5].
Revised: Preventing overfitting To demonstrate that current methods suffer from severe overfitting, we compare several methods trained with and without early stopping [1, 2, 3, 4, 6, 9, 5].

Original: Table 2 shows the Localization Accuracy (LocAcc) of these models on two datasets: Flickr SoundNet and VGG
Revised: Table 2 shows the Localization Accuracy (LocAcc) of these models on two datasets: Flickr SoundNet and VGG

Original: Sound Sources.
Revised: Sound Sources.

Original: We observe that, despite the large training sets (144k audio-visual pairs in both datasets), early stopping is critical to obtain high LocAcc in all prior works, despite the fact that it should not be used in a weakly-supervised VSL setting.
Revised: We observe that, despite the large training sets (144k audio-visual pairs in both datasets), early stopping is critical to obtain high LocAcc in all prior works, despite the fact that it should not be used in a weakly-supervised VSL setting.

Original: In fact, we observed that in most cases performance peaks within the first 2 to 3 training epochs, and decays rapidly afterwards.
Revised: In fact, we observed that in most cases performance peaks within the first 2 to 3 training epochs, and decays rapidly afterwards.

Original: Training curves are shown in Figure 4.
Revised: Training curves are shown in Figure ?? .

Original: This observation suggests that, due to overfitting, prior methods do not scale well ( i.e ., they cannot take advantage of larger datasets).
Revised: This observation suggests that, due to overfitting, prior methods do not scale well ( i.e ., they cannot take advantage of larger datasets).

Original: In contrast, SLAVC does not show the same signs of overfitting.
Revised: In contrast, SLAVC does not show the same signs of overfitting.

Original: The model improves as it is trained for longer.
Revised: The model improves as it is trained for longer.

Original: As a result, early stopping is not required to obtain a high performing model.
Revised: As a result, early stopping is not required to obtain a high performing model.

Original: Also, since the model can better leverage the large training data, it also significantly outperforms all prior work (even without early stopping).
Revised: Also, since the model can better leverage the large training data, it also significantly outperforms all prior work (even without early stopping).

Original: Without OGL (object-guided localization), we outperform the previous
Revised: Without OGL (object-guided localization), we outperform the previous

Original: SOTA (EZ-VSL [5]) by 3.60% and 2.94% on Flickr and VGG-SS, respectively.
Revised: SOTA (EZ-VSL [5]) by 3.60% and 2.94% on Flickr and VGG-SS, respectively.

Original: If early stopping is ruled out, these gains increase to 17.2% and 6.2% on Flickr and VGG-SS, respectively.
Revised: If early stopping is ruled out, these gains increase to 17.2% and 6.2% on Flickr and VGG-SS, respectively.

Original: Finally, by combining SLAVC with OGL, we achieve a new state-of-the-art on weakly-supervised VSL (86.4%
Revised: Finally, by combining SLAVC with OGL, we achieve a new state-of-the-art on weakly-supervised VSL (86.4%

Original: LocAcc on Flickr and 39.67% on VGG-SS).
Revised: LocAcc on Flickr and 39.67% on VGG-SS).

Original: Preventing false positives Since all prior work rely on LocAcc as the main evaluation metric, models are not penalized for high false positives rates.
Revised: Preventing false positives Since all prior work rely on LocAcc as the main evaluation metric, models are not penalized for high false positives rates.

Original: To better assess prior work, we evaluated several models on the proposed Extended Flickr and VGG-SS datasets with the evaluation protocol described in Sec.
Revised: To better assess prior work, we evaluated several models on the proposed Extended Flickr and VGG-SS datasets with the evaluation protocol described in Sec.

Original: 4 (without early stopping).
Revised: 4 (without early stopping).

Original: As can be seen in Table 3, with the exception of EZ-VSL and SLAVC , AP and max-F1 scores of prior works are very low, as these models struggle to avoid false positives without substantially increasing false negatives.
Revised: As can be seen in Table 3, with the exception of EZ-VSL and SLAVC , AP and max-F1 scores of prior works are very low, as these models struggle to avoid false positives without substantially increasing false negatives.

Original: EZ-VSL was shown to be more effective at preventing false positives.
Revised: EZ-VSL was shown to be more effective at preventing false positives.

Original: However, its overall performance was still lacking in comparison to the proposed SLAVC , which achieves the best results on all metrics.
Revised: However, its overall performance was still lacking in comparison to the proposed SLAVC , which achieves the best results on all metrics.

Original: SLAVC +OGL improves AP by 3.
Revised: SLAVC +OGL improves AP by 3.

Original: and max-F1 by 3.3 on the Extended Flickr-SoundNet test set.
Revised: and max-F1 by 3.3 on the Extended Flickr-SoundNet test set.

Original: On the more challenging extended
Revised: On the more challenging extended

Original: VGG-SS dataset, the gains of SLAVC +OGL were even larger, with an increase in AP of 6.75 and in max-F1 score of 6.9%.
Revised: VGG-SS dataset, the gains of SLAVC +OGL were even larger, with an increase in AP of 6.75 and in max-F1 score of 6.9%.

Original: 5.3 Analysis
Revised: 5.3 Analysis

Original: We now conduct a thorough analysis of SLAVC .
Revised: We now conduct a thorough analysis of SLAVC .

Original: Our goal is to improve audio-visual localization of sound sources.
Revised: Our goal is to improve audio-visual localization of sound sources.

Original: As seen in Tables 2 and Tab. 3, OGL enhances localization.
Revised: As seen in Tables 2 and Tab. 3, OGL enhances localization.

Original: However, since it only provides an object prior, OGL can be applied to any method.
Revised: However, since it only provides an object prior, OGL can be applied to any method.

Original: To better understand the proposed procedure, we conduct the various analysis without OGL (unless stated otherwise).
Revised: To better understand the proposed procedure, we conduct the various analysis without OGL (unless stated otherwise).

Original: Regularization strategies.
Revised: Regularization strategies.

Original: We start by showing in Tables 4a, 4b and 4c the effect of the various regularization techniques used to tackle overfitting (dropout and momentum target encoders).
Revised: We start by showing in Tables 4a, 4b and 4c the effect of the various regularization techniques used to tackle overfitting (dropout and momentum target encoders).

Original: Models are trained and tested on VGG Sound Sources.
Revised: Models are trained and tested on VGG Sound Sources.

Original: As can be seen, heavy dropout of visual features ( i.e . with a dropout rate of 0.9) is critical for localization, while audio dropout is not required.
Revised: As can be seen, heavy dropout of visual features ( i.e . with a dropout rate of 0.9) is critical for localization, while audio dropout is not required.

Original: As for the target encoders, we found that a momentum of 0.999 achieves best results, outperforming models that do not use momentum encoders by significant margins ( 6 . 0% AP, 5 . 2% max-F1, and 2 . 0% LocAcc).
Revised: As for the target encoders, we found that a momentum of 0.999 achieves best results, outperforming models that do not use momentum encoders by significant margins ( 6 . 0% AP, 5 . 2% max-F1, and 2 . 0% LocAcc).

Original: [Table]
Revised: [Table]

Original: SLAVC decomposition.
Revised: SLAVC decomposition.

Original: We also studied the effect of SLAVC decomposition during training and the localization strategy used for inference,
Revised: We also studied the effect of SLAVC decomposition during training and the localization strategy used for inference,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e ., either using audio-visual similarities tuned for localization alone (AVLoc; s ( ˆ v loc xy , ˆ a loc ) ), for AV correspondence (AVC; s ( ˆ v avc xy , ˆ a avc ) ) or both (SLAVC; Eq. 7).
Revised: e ., either using audio-visual similarities tuned for localization alone (AVLoc; s ( ˆ v loc xy , ˆ a loc ) ), for AV correspondence (AVC; s ( ˆ v avc xy , ˆ a avc ) ) or both (SLAVC; Eq. 7).

Original: Table 4d shows the localization performance for the different strategies.
Revised: Table 4d shows the localization performance for the different strategies.

Original: We observe that by training to simultaneously perform both audio-visual localization and correspondence, we can enhance localization by significant margins ( 4 .
Revised: We observe that by training to simultaneously perform both audio-visual localization and correspondence, we can enhance localization by significant margins ( 4 .

Original: 76% AP, 4 .
Revised: 76% AP, 4 .

Original: 40% max-F1, and 5 .
Revised: 40% max-F1, and 5 .

Original: 04% LocAcc).
Revised: 04% LocAcc).

Original: Also, both SLAVC branches, audio-visual localization (AVLoc) and correspondence (AVC), are required during inference for optimal performance.
Revised: Also, both SLAVC branches, audio-visual localization (AVLoc) and correspondence (AVC), are required during inference for optimal performance.

Original: Sound source size.
Revised: Sound source size.

Original: Figure 2 shows the performance of Center Prior (CPL), OGL, SLAVC and
Revised: Figure 2 shows the performance of Center Prior (CPL), OGL, SLAVC and

Original: SLAVC +OGL on sources of different sizes ( i.e . using the Small, Medium, Large and Huge subsets) on the VGG Sound Sources datasets.
Revised: SLAVC +OGL on sources of different sizes ( i.e . using the Small, Medium, Large and Huge subsets) on the VGG Sound Sources datasets.

Original: Each method was forced to output a prediction of constant size, by setting an appropriate threshold in the localization map.
Revised: Each method was forced to output a prediction of constant size, by setting an appropriate threshold in the localization map.

Original: We then plotted the LocAcc obtained for predictions that vary between 10% and 90% of the total image area.
Revised: We then plotted the LocAcc obtained for predictions that vary between 10% and 90% of the total image area.

Original: As can be seen, localization performance degrades significantly for smaller objects, regardless of the method used.
Revised: As can be seen, localization performance degrades significantly for smaller objects, regardless of the method used.

Original: In fact, all methods achieve a peak LocAcc of 0 in the Small subset containing sources of size up to 32 2 in area.
Revised: In fact, all methods achieve a peak LocAcc of 0 in the Small subset containing sources of size up to 32 2 in area.

Original: This corroborates observations in prior work, that detection of small sound sources remains one of the main limitations of current approaches.
Revised: This corroborates observations in prior work, that detection of small sound sources remains one of the main limitations of current approaches.

Original: We also observe that all methods perform well when localizing very large sound sources.
Revised: We also observe that all methods perform well when localizing very large sound sources.

Original: In fact, since the IoU threshold is set relatively low ( γ = 0 . 5 ) given the prediction sizes, even the Center Prior, which always selects the center pixels, can achieve high LocAcc.
Revised: In fact, since the IoU threshold is set relatively low ( γ = 0 . 5 ) given the prediction sizes, even the Center Prior, which always selects the center pixels, can achieve high LocAcc.

Original: The methods differ however when they are required to also identify negative samples, in which case SLAVC significantly outperforms prior methods (as shown in AP scores) regardless of prediction size.
Revised: The methods differ however when they are required to also identify negative samples, in which case SLAVC significantly outperforms prior methods (as shown in AP scores) regardless of prediction size.

Original: Balancing positive and negative detection.
Revised: Balancing positive and negative detection.

Original: To better understand the models’ ability to detect samples with no visible sound sources, we plot in Fig.
Revised: To better understand the models’ ability to detect samples with no visible sound sources, we plot in Fig.

Original: 3 (left) the F1 score for increasing confidence thresholds.
Revised: 3 (left) the F1 score for increasing confidence thresholds.

Original: As the threshold increases, models are more selective in identifying sound sources, and thus avoid making predictions if none is visible.
Revised: As the threshold increases, models are more selective in identifying sound sources, and thus avoid making predictions if none is visible.

Original: However, this can come at the cost of missing visible sound sources.
Revised: However, this can come at the cost of missing visible sound sources.

Original: As can be seen in Fig.
Revised: As can be seen in Fig.

Original: 3 (left), SLAVC and EZVSL are the only approaches whose F scores increase with higher confidence thresholds, with SLAVC achieving the highest F1 score.
Revised: 3 (left), SLAVC and EZVSL are the only approaches whose F scores increase with higher confidence thresholds, with SLAVC achieving the highest F1 score.

Original: This shows that unlike most of prior work, SLAVC can effectively identify negative pairs,
Revised: This shows that unlike most of prior work, SLAVC can effectively identify negative pairs,

Original: i.e . ti reduces false positive rates without sacrificing true positive rates.
Revised: i.e . ti reduces false positive rates without sacrificing true positive rates.

Original: High quality detection.
Revised: High quality detection.

Original: Figure 3 (right) also shows the AP scores at varying IoU thresholds.
Revised: Figure 3 (right) also shows the AP scores at varying IoU thresholds.

Original: We observe that while our method consistently outperforms prior work, the performance of all methods degrade substantially as the IoU threshold increases.
Revised: We observe that while our method consistently outperforms prior work, the performance of all methods degrade substantially as the IoU threshold increases.

Original: In fact, the AP at 0 .
Revised: In fact, the AP at 0 .

Original: 75 IoU is close to 0 for all methods, indicating that current weakly-supervised VSL methods still struggle to achieve high quality detection.
Revised: 75 IoU is close to 0 for all methods, indicating that current weakly-supervised VSL methods still struggle to achieve high quality detection.

Original: Negative type.
Revised: Negative type.

Original: Table 5 studies the max-F1 score for false positives among real and hand selected negatives (easy/hard).
Revised: Table 5 studies the max-F1 score for false positives among real and hand selected negatives (easy/hard).

Original: For each of the three negative subsets, we add a similar number of positives (e.g., there are 1185 hard negatives on Extended VGG-SS, so 1185 random positives are added to this set to get a total of 2370 samples).
Revised: For each of the three negative subsets, we add a similar number of positives (e.g., there are 1185 hard negatives on Extended VGG-SS, so 1185 random positives are added to this set to get a total of 2370 samples).

Original: These results measure how well methods can balance positive and negative detections when given only those types of negatives.
Revised: These results measure how well methods can balance positive and negative detections when given only those types of negatives.

Original: The proposed SLAVC achieves the best performance in terms of all types of negatives, which further demonstrates the robustness and effectiveness in balancing positive and negative detections.
Revised: The proposed SLAVC achieves the best performance in terms of all types of negatives, which further demonstrates the robustness and effectiveness in balancing positive and negative detections.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we identify two critical issues with current weakly-supervised visual sound source localization methods: severe overfitting even when trained with large datasets, and their poor ability to identify when no sound sources are visible ( i.e ., negatives).
Revised: In this work, we identify two critical issues with current weakly-supervised visual sound source localization methods: severe overfitting even when trained with large datasets, and their poor ability to identify when no sound sources are visible ( i.e ., negatives).

Original: Since current evaluation protocols allow for early stopping and always assume the presence of a visible sound sources, they are not sensitive to the aforementioned issues, reason why they remained relatively unknown.
Revised: Since current evaluation protocols allow for early stopping and always assume the presence of a visible sound sources, they are not sensitive to the aforementioned issues, reason why they remained relatively unknown.

Original: To fix these issues, we propose a new evaluation protocol and a novel method for VSL.
Revised: To fix these issues, we propose a new evaluation protocol and a novel method for VSL.

Original: We extend current evaluation datasets to also include negative samples ( i.e ., frames with no visible sound source).
Revised: We extend current evaluation datasets to also include negative samples ( i.e ., frames with no visible sound source).

Original: We show that overfitting can be effectively addressed through commonly used regularization techniques like dropout and momentum target encoders.
Revised: We show that overfitting can be effectively addressed through commonly used regularization techniques like dropout and momentum target encoders.

Original: We also show that forcing the model to explicitly conduct both localization and audio visual correspondence enhances the model’s ability to identify negative samples.Appendix
Revised: We also show that forcing the model to explicitly conduct both localization and audio visual correspondence enhances the model’s ability to identify negative samples.

Original: In this appendix, we detail the baselines and metrics used for benchmarking visual sound localization (LocAcc, F1 score, and Average Precision).

Original: We then demonstrate the effectiveness of the proposed

Original: SLAVC in other commonly used training datasets (Flickr 10k) and other settings like semi-supervised localization and open set visual sound source localization.

Original: Finally, we compare the qualitative results of our approach with existing methods.

Original: Code is available at: https://github.com/stoneMo/SLAVC .

Original: A Baselines

Original: We conducted a comprehensive benchmarking study of existing approaches.

Original: For a fair comparison, we use the same backbone-ResNet18 [46] for all baselines.

Original: Namely, we considered:

Original: • Attention 10k [1] (2018’CVPR): the first attention-based work with a two-stream architecture with each modality for weakly-supervised sound source localization in an image, and this approach was extended to semi-supervised settings with ground-truth maps in 5k Flickr set; (code: https://github.com/ardasnck/learning_to_localize_sound_source ) • DMC [2] (2019’CVPR): a multi-modal clustering network for learning audiovisual correspondences by using convolutional maps with each modality in different shared spaces; (code: https://github.com/DTaoo/Simplified_DMC , MIT License) • CoarsetoFine [4] (2020’ECCV): a two-stage pipeline that aligned the cross-modal features in a coarse-to-fine way; (code: https://github.com/shvdiwnkozbw/ Multi-Source-Sound-Localization )

Original: • LVS [6] (2021’CVPR): a contrastive learning framework with hard negatives mining to extract the audio-visual co-occurrence map discriminatively; (code: https://github. com/hche11/Localizing-Visual-Sounds-the-Hard-Way , Apache License 2.0) • HardPos [9] (2022’ICASSP): an improved work based on LVS by adding hard positives for aligning audio-visual matching semantics from negative pairs; • EZ-VSL [5] (2022’ECCV): a strong baseline that proposed the multiple instance contrastive learning to align locations with high similarity in the image and push away from all locations in different images; (code: https://github.com/stoneMo/EZ-VSL , Apache License 2.0) • DSOL [7] (2020’NeurIPS): a two-stage training baseline to deal with silence in category-aware sound source localization; (code: https://github.com/DTaoo/ Discriminative-Sounding-Objects-Localization , MIT License);

Original: B Localization Accuracy, F1 Score, Average Precision

Original: Consider a set of samples D

Original: = { ( v i , a i ) :

Original: i = 1 , . .

Original: . , N } , ground-truth maps G = {G i } Ni = and predicted maps S

Original: = {S i } Ni =1 .

Original: In map based localization, each prediction S i is obtained by first computing pixel-wise localization scores S i ∈

Original: R H × W , and then applying a thresold α ,

Original: S i ( α ) =

Original: { ( x, y ) | S i ( x, y ) > α } .

Original: To evaluate each prediction S i , prior work [1] relies on intersection over union between S i and G i .

Original: IoU is calculated as

Original: [Equation]

Original: where G = { ( x, y ) | g xy > 0 } denotes the the set of pixels that represent sounding objects, and g xy ∈ [0 , 1] the ground-truth evidence that a sounding objects lies at location ( x, y ) .

Original: Since in some datasets ground-truth is collected from multiple annotators, consensus IoU (cIoU) is used instead.

Original: Refer to [1] for details on how multiple annotations are merged to compute g xy .

Original: Since we’re interested in assessing the model’s performance both when sounding objects are present or not, we allow the model to make no predictions.

Original: To do this, in addition to the localization prediction map S i , the model is asked to output a confidence score

Original: c i .

Original: If the confidence score c i is too low

Original: (below a threshold δ ), the model predicts an empty set ( i.e ., the original S i is considered invalid).

Original: Predictions S i are considered correct if IoU i are above a pre-specified threshold γ .

Original: Under these definitions, true positives, false positives and false negatives are computed as

Original: [Equation]

Original: Localization Accuracy To evaluate the localization accuracy of our models among samples with visible sound sources, we measure the localization accuracy at a predefined IoU threshold γ

Original: [Equation]

Original: Since this metric only measures localization performance, it assumes that all samples contain a visible source to be localized.

Original: We thus ignore the confidence threshold ( i.e ., δ = −∞ ), so as to predict a source location for every sample.

Original: Most existing work [2, 3, 4, 6, 9, 5] report LocAcc at γ = 0 .

Original: 5 under the name of “CIoU”.

Original: We use this metric when comparing to results reported in the original papers.

Original: While the localization accuracy provides a good metric to evaluate how accurate predictions are for samples with sounding objects, it does not assess how accurately models can ignore samples with

Original: NO sounding objects.

Original: Thus, to comprehensively evaluate both sounding and non-sounding samples, we also evaluate our models using F1 score and average precision (AP).

Original: F1 score balances precision and recall,

Original: [Equation]

Original: where

Original: Precision

Original: [Equation]

Original: However, F1 ( γ, δ ) depends on how strict the confidence threshold δ is set (the highest δ is set, the more samples are predicted as non-sounding).

Original: To find the optimal balance, we compute F1 ( γ, δ ) for all values of δ and report the max-F1 score

Original: [Equation]

Original: Average precision (AP) is another metric often used in object detection.

Original: To compute AP, we closely follow [50].

Original: The only difference is that, when computing the Precision-Recall curve, we do not perform 11 point interpolation.

Original: We compute the full curve (without interpolation).

Original: Refer to [50] for details on AP computation.

Original: C Training curves

Original: Prior work, including previous state-of-the-art EZ-VSL [5] overfit to the self-supervised loss.

Original: To better see this, we plotted the training curve of both EZ-VSL and our method SLAVC in Fig.

Original: We

Original: [Table]

Original: plot both LocAcc and AP obtained on the extended VGG-SS test set as the model is trained on the

Original: VGG-SS 144k.

Original: As can be seen, the localization performance of EZ-VSL peaks very early (around epoch 6)

Original: [Table]

Original: and degrades significantly afterwards.

Original: We also observe that EZ-VSL has a more unstable training behavior than SLAVC , which completely avoids overfitting and thus continuously improves localization performance as it is trained

Original: [Table]

Original: D Weakly/Semi Supervised Results on Flickr-10k

Original: In addition to the weakly-supervised setting, some works also explore the use of a small annotated dataset to guide training [1].

Original: To better understand the added value of a small number of bounding boxes, we also train our model in the semi-supervised setting.

Original: Following [1], we use the ground-truth maps of 5000 Flickr images to directly supervise SLAVC ’s predictions through the additional loss

Original: [Equation]

Original: where S AVL denote the model’s prediction of Eq. 7, 1 supi indicates whether sample i has ground-truth annotations, G i represents the ground-truth localization map, and ∥ · ∥ represents the norm over all spatial locations x, y .

Original: To compare the semi-supervised to the weakly-supervised setting, we train the model on the Flickr10k [1] (containing 10k samples from Flickr).

Original: Results are reported in Table 6.

Original: We don’t report results with the latest checkpoint in the semi-supervised case, since the available ground-truth annotations can be used for early stopping.

Original: Nevertheless, our SLAVC achieves the state-of-the-art results compared to existing methods in both settings.

Original: The additional annotations allow our model to achieve higher

Original: LocAcc.

Original: However, the difference between the two settings (weakly and semi supervised) is much smaller in our case, when compared to Attention10k.

Original: This result indicates that the gains achieved by Attentio10k with the additional supervision were mostly due to the overfitting and false positive issues identified in this work.

Original: E Open Set Results

Original: To evaluate the generalization of our model beyond sound sources heard during training, we follow previous work [6, 5] and train the model on 70k data with 110 heard categories in VGG-Sound dataset [45].

Original: Table 7 reports the comparison results on heard and unheard 110 classes.

Original: We achieve significant improvements against previous methods.

Original: For instance, using models without OGL and no early stopping for testing heard 110 classes increases the baseline by 5.71%, 5.90%, and 5.22% in terms of AP, max-F1, and LocAcc.

Original: When it comes to unheard 110 classes, the proposed SLAVC without OGL outperforms the current state-of-the-art approach by 4.10%, 3.80%, and 3.42% in terms

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: of AP, max-F1, and LocAcc.

Original: New state-of-the-art results are achieved in all settings, demonstrating the generalization of our approach to unheard sounding categories in the training set.

Original: F Qualitative Results

Original: In order to qualitatively demonstrate the effectiveness of the proposed SLAVC , we compare visualize attention maps from existing work and our model on various sizes of sounding objects in VGG-SS test set.

Original: The qualitative results of of Attention10k [1], DMC [2], LVS [6], EZ-VSL [5], and the proposed

Original: SLAVC on huge/large/medium/small sounding objects are shown in Figure 5, 6, 7, and 8.

Original: Note that

Original: Attention

Original: [Figure]

Original: 𝚂𝙻𝙰𝚅𝙲 results of our SLAVC are shown on the last column.

Original: We can observe that the proposed SLAVC achieves decent localization maps compared to previous approaches in all settings, such as the barking dog on row 3 in Figure 5, the sounding cello on row 1 in Figure 6, and the crying baby on row 1 in Figure 7.

Original: For small objects in Figure 8, we also predict better maps for sounding objects than existing work.

Original: While our predictions seem to focus more on the actual sources, they are very large compared to the actual objects.

Original: G Broader Impact

Original: This paper seeks to establish a more balanced evaluation protocol for visual sound source localization, that considers both cases with sounding and non-sounding objects.

Original: We hope the proposed metrics will lead to more balanced localization algorithms in the future.

Original: Improving sound source localization methods can lead to interesting applications, for example, interfacing with blind or low-vision individuals as they navigate the world.

Original: However, our work still relies on data collected from internet sources, and thus is likely to hold biases that have not been identified.

Original: While we believe current datasets and evaluation protocols are valuable for the development of sound source localization procedure, better curation of the datasets against nefarious bias should be conducted before deployment in real world settings.

Original: Advances in audio-visual localization can also be leveraged in surveillance applications, which can potentially have a negative societal impact.

Original: [Figure]

Original: 𝚂𝙻𝙰𝚅𝙲

Original: [Figure]