Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing
Abstract
The audio-visual video parsing task aims to parse a video into modality- and category-aware temporal segments.
Previous work mainly focuses on weaklysupervised approaches, which learn from video-level event labels.
During training, they do not know which modality perceives and meanwhile which temporal segment contains the video event.
Since there is no explicit grouping in the existing frameworks, the modality and temporal uncertainties make these methods suffer from false predictions.
For instance, segments in the same category could be predicted in different event classes.
Learning compact and discriminative multi-modal subspaces is essential for mitigating the issue.
To this end, in this paper, we propose a novel Multi-modal Grouping Network, namely MGN, for explicitly semantic-aware grouping.
Specifically, MGN aggregates eventaware unimodal features through unimodal grouping in terms of learnable categorical embedding tokens.
Furthermore, it leverages the cross-modal grouping for modality-aware prediction to match the video-level target.
Our simple framework achieves improving results against previous baselines on weakly-supervised audiovisual video parsing.
In addition, our MGN is much more lightweight, using only 47.2% of the parameters of baselines (17 MB vs. 36 MB).
Code is available at https://github.com/stoneMo/MGN .
Introduction
Humans understand the surrounding environment by integrating signals from different senses.
In our daily life, sound and sight are two of the most commonly used modalities, which have drawn much attention from researchers to explore computational audio-visual scene understanding.
Previous audio-visual work [1, 2] usually assumes audio and visual data are temporally aligned.
However, the alignment does not always exist in real-world videos.
For example, sounding objects in many videos are outside of the field-of-view (FoV).
For these non-aligned cases, audio signals become more reliable in understanding the events of interest.
In this work, we address the audio-visual video parsing (AVVP) task [3] that aims to parse a video into temporal event segments and predict the audible, visible, or audio-visible event categories.
Existing approaches [3, 4, 5] usually focus on learning to leverage the unimodal and cross-modal temporal contexts from weak supervisions.
HAN [3] introduced a simple Multimodal Multiple
Instance Learning framework with cross-modal and self-modal attention layers to utilize the videolevel labels.
Recent state-of-the-art methods usually use the HAN as the baseline and modify it to further improve parsing performance.
Particularly, contrastive learning and label refinement are proposed in Wu and Yang [4], where they adopted a contrastive loss to enforce the temporal alignment between the audio and visual features at the same timestamp and augmented training data with modality-aware event labels generation.
Furthermore, Lin et al.
[5] proposed to leverage
audio-visual class co-occurrence to jointly explore the relationship of different categories among all modality streams.
Our key motivation is to learn compact and discriminative audio and visual representations by explicit multi-modal grouping for mitigating the modality and temporal uncertainties in the weakly-supervised audio-visual video parsing problem.
During training, segment-wise event labels are unavailable for audio and visual temporal segments in videos.
Thus, the multi-modal temporal modeling in the above existing methods might aggregate and implicitly group irrelevant semantic information due to lacking of fine-grained supervisions, which causes false positives for predicting categories of events.
That is, audio or visual temporal segments in the same event category might be far away from the class center in the embedding space since there are no segment-level and modality-wise constraints during training.
In the meanwhile, there is no constraint for modality category prediction to match the video-level target at the end.
Different from past approaches, we propose a new Multi-modal Grouping Network, namely MGN, to explicitly group semantic-aware multi-modal contexts, which enables learning more compact and discriminative audio and visual representations.
Specifically, we first extract event-aware unimodal features through unimodal grouping in terms of learnable categorical embedding tokens for each individual modality.
Then, we introduce a cross-attention layer with a hard attention mechanism to aggregate cross-modal temporal contexts.
Finally, we utilize a cross-modal grouping module to predict the modality category from updated class-aware unimodal embeddings.
Experimental results on the LLP [3] dataset validate that our new audio-visual video parsing framework achieves superior results over previous state-of-the-art methods [1, 2, 3, 4].
Empirical results also demonstrate the generalizability of our approach to contrastive learning and label refinement proposed in MA [4].
In addition, we substantially reduce the parameters of previous work by using only 47.2% parameters of baselines (17 MB vs. 36 MB).
Our main contributions can be summarized as follows:
• We propose a new audio-visual video parsing baseline: Multi-modal Grouping Network (MGN) that enables explicit grouping in a multi-modal network to learn compact and discriminative audio and visual embeddings.
• We introduce class-aware unimodal grouping and modality-aware cross-modal grouping modules to aggregate multi-modal temporal contexts.
• The experiments can demonstrate the superiority of our MGN over state-of-the-art AVVP approaches and its generalizability to contrastive learning and label refinement.
Related Work
Audio-Visual Learning.
Audio-visual learning has addressed in many previous works [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] to learn the audio-visual association between the two distinct yet correlated modalities.
Such audio-visual temporal association is crucial to several tasks, such as audio-visual spatialization [17, 18, 19, 14], speech/audio separation [13, 20, 21, 11, 12, 22, 23, 24], visual sound source localization [10, 25, 26, 27, 28, 29].
A comprehensive survey [30] covers recent advances in audio-visual learning.
In this work, we mainly focus on audio-visual video parsing that aims to parse a video into temporal segments in terms of audio, visual, and audio-visual events with only video-level annotations for training.
In addition, different from existing audio-visual architectures, we firstly introduce explicit multi-modal grouping into audio-visual learning.
Audio-Visual Video Parsing.
Audio-visual video parsing aims at temporally localizing audio, visual, and audio-visual events in videos and predicting their event categories.
Early approaches [1, 2, 31, 32] tried to localize only audio-visual events by aggregating cross-modal information in each local segment of the input video.
Due to exhaustive labeling costs on segment-wise audio, visual, and audio-visual events, recent work [3, 4] explored the weakly-supervised audio-visual video parsing with only video-level activity categories for training.
For example, HAN [3] introduced the multimodal multiple instance learning mechanism with a hybrid attention network to aggregate segmentwise representations and video-level features.
To alleviate the harm brought by the audio-visual asynchrony, MA [4] proposed the audio-visual contrastive learning from different frames and refined modality-aware labels by exchanging audio and visual tracks between unrelated videos.
More recently,
[Figure]
𝒇 !" !
#$% 𝒄 &" &#$’ 𝒈 &" &#$’ 𝒈 &( &#$’ 𝒄 &( &#$’ 𝒇 !
#$%
[Figure]
Lin et al.
[5] leveraged the common semantics shared by videos with replaced audios or frames in a mini batch to detect relevant events.
Different from these baselines based on HAN, we develop a fully novel network architecture to alleviate implicit audio-visual matching and modality category uncertainty in the hybrid attention network.
We are the first to exploit unimodal grouping for learning audio-visual representations with class-aware semantics.
Furthermore, we introduce a modality-aware cross-modal grouping module to match the video-level label, although the given target does not indicate modalities.
Method
Given a video with both audio and visual tracks, our goal is to parse the video into temporal segments associated with audible, visible, and audio-visible event categories.
We propose a novel baseline:
Multi-modal Grouping Network (MGN) to address the task, which mainly consists of two modules: class-aware unimodal grouping (Sec. 3.2) and modality-aware cross-modal grouping (Sec. 3.3).
3.1 Preliminaries
In this section, we first describe the problem setup and notations and revisit Multi-modal Multiple
Instance Learning in HAN [3] for the audio-visual video parsing (AVVP) task.
Problem Setup and Notations.
Given a video with T non-overlapping audio and visual segments, our goal is to temporally localize and recognize audio, visual, and audio-visual events that existed in the video.
For the multi-label events with C event categories at time t , we have audio, visual, and audio-visual event labels for evaluation, which are denoted as: y at , y vt , y avt ∈ R 1 × C .
During training, we do not have the segment- and modality-level annotations.
Therefore, we can only use the video-level label y av ∈ R 1 × C to perform weakly-supervised learning.
Revisit Multimodal Multiple Instance Learning.
To address the weakly-supervised audio-visual video parsing problem, HAN [3] introduced a Multimodal Multiple Instance Learning (MMIL) framework based on transformers [33].
Given a set of audio-visual features F a = { f at } Tt =1 , F v = { f vt } Tt =1 in T segments, HAN applied self-attention and cross-attention layers to aggregate the unimodal and cross-modal information at each timestamp:
[Equation]
where ϕ sa ( · ) , ϕ ca ( · ) denote the self-attention and cross-attention functions:
[Equation]
and f at , f vt ∈ R 1 × d , F a , F v ∈ R T × d , d is the dimension of audio-visual features.
Then, the probability of segment-wise categories for audio and visual events ( p a t , p v t ∈ R 1 × C ) is predicted by a shared fully-connected (FC) layer and sigmoid function.
With the attentive MMIL pooling, the video-level prediction is formulated as:
[Equation]
where W t = { w at , w vt } is the temporal attention weights computed by a FC layer and normalized by a softmax function.
P t = { p at , p vt } is the probability set of audio-visual predictions.
M = 2 denotes audio and visual modalities.
Finally, the model is trained to optimize a weakly-supervised loss of p av and a guided loss of p a , p v with label smoothing:
[Equation]
where O ( · ) is a binary cross-entropy function-based loss term, which summarizes binary cross-entropy for all categories and O ( y , p )
= (cid:80) i BCE ( y i , p i ) .
y a , y v are video-level audio and visual labels generated by smoothing y av to decrease the confidence of positive labels.
However, such a training mechanism will pose two main challenges.
First, these methods without explicit grouping suffer from false predictions due to the modality and temporal uncertainties.
Second, there is no constraint for modality category prediction to match the video-level target at the end.
To address these challenges, inspired by [34], we propose a novel Multi-modal Grouping Network (MGN) with class-aware unimodal grouping and modality-aware cross-modal grouping modules, as shown in Figure 1.
3.2 Class-aware Unimodal Grouping
In order to explicitly group class-aware matching semantics for audio-visual representations, we introduce a novel class-aware unimodal grouping module by incorporating learnable modality-specific class tokens { c ai } Ci =1 , { c vi } Ci =1 to help to group raw input unimodal features { f at } Tt =1 , { f vt } Tt =1 .
We first use self-attention transformers: ϕ asa ( · ) and ϕ vsa ( · ) to temporally aggregate unimodal features from audio and visual inputs and align the features with the categorical token embeddings:
[Equation] [Equation]
where [ ; ] denotes the concatenation operator.
Then
[Equation]
the unimodal grouping blocks g a ( · ) , g v ( · ) take the learned audio and visual event class tokens and aggregated features as inputs to generate class-aware audio and visual embeddings as:
[Equation]
During grouping, we merge all the unimodal features that belong to the same class token into a new unimodal class-aware feature, by computing the similarity matrices
A a , A v between unimodal features and class tokens via a softmax operation formulated as
[Equation]
where W aq , W ak and W vq , W vk are the weights of the learned linear projections for the features and class tokens of audio and visual modalities, respectively.
Based on this similarity, we calculate the class-aware features with the weighted sum of all segment features assigned to that class:
[Equation]
where W ao , W av and W vo , W vv denote the learned weights of linear projections for audio and visual modalities, separately.
Note that, the audio features { ˆ f at } Tt =1 are merged with discriminative visual features { f vt } Tt =1 via a similar grouping block with a hard-softmax operation.
In order to constrain the independence of each class token c ai , c vi , we apply FC layers and softmax operation to generate the category probability e ai , e vi of class tokens for audio and visual modalities with a class-constrained loss as:
[Equation]
[Equation]
where CE( · ) refers to cross-entropy loss; h i is an one-hot encoding vector and only its element for the target class entry i is 1.
After the class-aware unimodal grouping, the video-level category predictions p a , p v of audio and visual events is simply computed by a FC layer and sigmoid operator:
[Equation]
With the help of the proposed class-constrained loss, we generate class-aware representations { g ai } Ci =1 , { g vi } Ci =1 of audio and visual modalities for audio-visual matching.
3.3 Modality-aware Cross-modal Grouping
The second challenge requires us to predict the modality category for matching with the only given video-level target in an explicit way.
To achieve this, we propose a modality-aware cross-modal grouping module composed of cross-modal transformers ϕ ca ( · ) and grouping blocks g av ( · ) to aggregate class-aware representations { g ai } Ci =1 , { g vi } Ci =1 .
Based on the audio-visual similarity in the grouping stage, we combine all the audio features with visual features into new cross-modal modality-aware features { g avi } Ci =1 as:
[Equation]
where g av ( · ) denote the grouping operator similar to g a ( · ) and g v ( · ) in Eq.
Then, we leverage the joint audio-visual representations { g avi } Ci =1 to predict the video-level target of audio and visual events via a FC layer and sigmoid function as:
[Equation]
The whole model can be optimized in an end-to-end manner in terms of the objective function:
[Equation] [Equation]
At inference time, the unimodal class-aware similarity is used to predict the audio, visual, and audio
[Table]
[Equation]
[Table]
Experiments 4.1 Experimental Setup
Dataset.
The Look, Listen and Parse (LLP) Dataset [3] contains 11,849 YouTube video clips of 10-seconds long from 25 different event categories, such as car, music, cheering, speech, etc.
Note that each video includes at least 1s audio or visual events and 7202 video clips are annotated with more than one event categories.
We use 10,000 video clips with only video-level event labels for training.
Following the official splits [3] of validation and test sets, we develop and test the model on the remaining 1879 videos with the segment-level annotations, i.e. , the speech event for audio starts at 1s and ends at 5s.
Evaluation Metrics.
We follow the prior work [3, 4] and use F-scores to evaluate both segment-level and event-level predictions for audio, visual, and audio-visual events.
The segment-level metrics can evaluate snippet-wise event prediction performance.
For the event-level metrics, we concatenate positive consecutive segments in the same events and compute F-score based on mIoU=0.5 as the threshold.
Type@AV and Event@AV are reported for the overall evaluation of audio-visual video parsing performance.
Type@AV is the averaged audio, visual, and audio-visual event evaluation results.
Different from Type@AV directly averaging results from different event types, Event@AV considers audio and visual events for each sample.
Implementation.
Following the data pre-processing in previous work [3], we sample video frames at 8 fps from the 10-second videos with 10 non-overlapping snippets of 1 second.
For low-level visual features, we concatenate 2D and 3D visual features extracted by ResNet-152 [35] pre-trained on ImageNet [36] and 3D ResNet [37] pre-trained on Kinetics-400 [38].
We utilize VGGish [39] pre-trained on AudioSet [40] to extract the audio features.
The model is trained with Adam [41] optimizer with β 1 =0.9, β 2 =0.999 and with an initial learning rate of 3e-4.
We train the model with a batch size of 16 for 40 epochs.
4.2 Comparison to Prior Work
In this work, we propose a novel and effective training framework for weakly-supervised audio-visual video parsing.
To demonstrate the effectiveness of our approach, we first compare it to previous network baselines 2 : 1) AVE [1]: an audio-guided co-attention network with additional audio-visual parsing branches; 2) AVSDN [2]: a sequence-to-sequence-based model with additional audio-visual parsing branches to merge global audio-visual features into local ones; 3) HAN [3]: a hybrid attention network with multi-modal multiple instance learning pooling.
Furthermore, we compare MA [4], a method based on HAN with the audio-visual contrastive learning (C) and the label refinement
[Figure] [Figure]
Vacuumcleaner
[Figure]
(R).
The MA series baselines include MA (w C), MA (w R), and MA (w C+R).
The quantitative comparisons on the LLP dataset are reported in Table 1.
As can be seen, the proposed MGN achieves the overall best results against previous network baselines in terms most of metrics.
For the overall evaluation of segment-level predictions, we achieve significant performance gains of 1.5 Type@AV and 1.8 Event@AV.
When evaluated on segment-level predictions of each sample, our MGN also improves the baseline by large margins, 2.
Visual and 1.5 Audio-Visual.
Meanwhile, our MGN outperforms baselines by 3.5 Visual, 1.4 AudioVisual, and 1.6 Tyep@AV for event-level predictions.
These results demonstrate the effectiveness of our approach in weakly-supervised audio-visual video parsing against prior network architectures.
Furthermore, significant gains can be observed in the setting of using the audio-visual contrastive learning and label refinement.
Adding the contrastive learning to our MGN achieves the segment-level performance gain of 3.6 Visual and 2.8 Audio-Visual, and the event-level gain of 3.8 Visual and 2.6 Audio-Visual.
With the label refinement in MA, we significantly improve MA (w R) by 3.
segment-level and 4.0 event-level for visual predictions.
Our framework with both contrastive learning and label refinement achieves the best segment-level performance in terms of Visual, Audio-Visual,
Type@AV, and Event@AV.
These improvements imply the strong generalizability of the
proposed
MGN to the audio-visual contrastive learning and the label refinement.
In order to qualitatively evaluate the predictions of audio and visual events, we compare the proposed
MGN with HAN [3], MA [4] in Figure 2.
We can observe three main things from previous approaches:
First, existing methods can miss the snippet-wise predictions for some hard segments in the video, e.g., MA does not predict the vacuum cleaner for visual events after 3s.
Second, prior work can miss one class prediction if there are two categories happening in the same modality.
Third, the previous method can miss one modality prediction for the class existing in both modalities.
When it comes to the proposed MGN, we superiorly achieve high F1 scores of segment-level and event-level prediction, benefiting from the well-designed class-aware unimodal grouping and modality-aware cross-modal grouping modules.
4.3 Experimental Analysis
In this section, we conducted ablation studies to validate the benefit of Class-aware Unimodal
Grouping (CUG) and Modality-aware Cross-modal (MCG)
Grouping strategies.
We also performed extensive experiments to investigate the false positive issue in HAN and the learned class tokens.
[Table]
Class-aware Unimodal Grouping & Modality-aware Cross-modal Grouping.
In order to demonstrate the effectiveness of the proposed class-aware unimodal grouping (CUG) and modality-aware cross-modal grouping (MCG), we ablated the necessity and strategy of grouping blocks.
The results of segment-level predictions are reported in Table 2.
We can observe that adding CUG to the vanilla baseline achieves significant gains of 2.4 Visual, indicating the effectiveness of grouping class-aware semantics in predicting snippet-wise categories for visual events.
Incorporating MCG with CUG highly increases Audio-Visual, Tyep@AV, Event@AV by 1.5, 1.5 and 1.8.
These results show the importance of modality-aware grouping on predictions of audio-visual events.
Besides effectiveness, our model is also more efficient.
When the depth of CUG and MCG is 3 and 6, the proposed MGN with only 47.2% parameters of the vanilla baseline performs the best on Type@AV and Event@AV, especially on Audio.
These results further show the advantage of our MGN in real applications with lightweight parameters against the prior work [3, 4].
The detailed results are in Appendix.
False Predictions.
In order to demonstrate the effectiveness of the proposed MGN in mitigating false predictions against baselines, we calculate the total amount of false positives for allclasses in the test set.
The comparison results of event-level audio, visual and audio-visual metrics are shown in Figure 3.
We can observe that our MGN with the class-aware unimodal grouping modules decreases the false positives of audio and visual events by large margins,and 494.
Furthermore, the number of false positives of audio-visual events drops down by 678, which verifies the importance of modality-aware cross-modal grouping in mitigating the modality uncertainty.
Overall, our MGN with explicit grouping mechanisms significantly eliminates false predictions caused by the modality and temporal uncertainties existing in the baseline.
Learned Class-aware Features.
The learned class tokens are essential to grouping class-aware semantics from audio and visual features.
To better evaluate the quality of those learned class-level features, we visualize the learned audio and visual representations of 25 categories by t-SNE [42], as shown in Figure 4.
It is noted that each spot denotes the feature of one audio or visual event, while each color represents each class, such as “Speech” in brown and “Dog” in green.
As can be seen in the last column, features extracted by the proposed MGN are intra-class compact and inter-class separable.
However, there still exists mixtures of multiple categories for audio and visual events among the representations of HAN and MA.
For the sub-figure on the bottom right, we can observe a large cluster of brown spots for the “Speech” class of audio events in the test set, while brown spots in prior work are distributed more sparsely.
These meaningful visualizations further demonstrate that our MGN successfully learns compact and discriminative features for each modality.
4.4 Limitation
Although the proposed MGN achieves superior results on visual events and audio-visual events, the gains of audio events are not significant compared to the visual modality.
We notice that there are 1628 visual instances and 2663 audio instances in the test set, which implies the audio modality is much harder than the visual one in this setting.
One possible solution is to introduce a small number of segment-wise audio-visual parsing annotations as supervision for semi-supervised training.
In the meanwhile, our MGN performs worse with the increase of the depth of transformer layers in grouping modules.
This is caused by such a weakly-supervised setting with only video-level annotations that do not indicate either segments or modalities.
However, the model is expected to parse a video into events with different categories and modalities.
Therefore, the potential future work is to add more grouping stages with learned class-tokens as supervision for each one.
Conclusion
In this work, we present MGN, a fully novel Multi-modal Grouping Network to explicitly group class-aware matching semantics for weakly-supervised audio-visual video parsing.
We introduce the class-aware unimodal grouping module to generate class-aware unimodal representations with learnable tokens by using unimodal grouping blocks for each modality.
Furthermore, we leverage the modality-aware cross-modal grouping to match the video-level target with the cross-modal grouping blocks.
Experimental results demonstrate the effectiveness and superiority of our MGN against previous baselines.
We also show the generalizability of our simple framework to the audio-visual contrastive learning and label refinement.
Broader Impact.
The proposed method detects video events in audio and visual modalities based on the learned statistics of the training dataset.
It could capture internal biases in the data, which may have negative societal impacts.
For example, the model might not be able to discover rare but important events, such as fire warnings.
Thus, before deploying our audio-visual video parsing model into real-world applications, we need to carefully address these issues.
Appendix
In the supplementary material
[Table]
we provide the differences between our MGN and the recent work
[Table]
GroupViT [34]
[Table]
more experiments on model size and grouping strategies.
We finally validate the effectiveness of audio/visual tokens in learning class-aware embeddings.
The code of our implementation is also attached as the supplementary material for review.
A Differences between GroupViT [34] and the proposed MGN
Compared to GroupViT [34], the recent grouping work on image segmentation, there are three main distinctive characteristics of our MGN for solving the weakly-supervised auido-visual video parsing problem, which are listed as follows:
1) Constraint on Class Tokens.
Most important, we have learned 25 category tokens with each class constraint, i.e. , each class token does not have semantic overlapping information among each other and we want each token to learn the embeddings for each category.
To achieve this, we
apply
the crossentropy loss to class tokens for audio-visual modalities as L cls =
CE
p ai
} Ci
=1 ,
+ CE
p
vi }
Ci =1 , 1 ) .
However, the number of
learned
group tokens in GroupViT is a hyper-parameter and there is no constraint on it.
2) Modality-aware Cross-modal Grouping.
We have the modality-aware cross-modal grouping to eliminate the modality uncertainty caused by the weakly-supervised setting in the task, while GroupViT does not incorporate text embeddings into the grouping stage.
The text embeddings is used in a contrastive loss to match with the global visual representations.
In this case, their model was trained on large-scale data with a large batch size for self-supervised training.
3) Class-aware Unimodal Grouping.
We introduce the class-aware unimodal grouping for mitigating the
temporal
uncertainty in each modality
but
GroupViT
only leverages the unimodal grouping on visual patch tokens without class tokens involved.
Therefore, the
unimodal grouping
in
GroupViT can not be used in this task directly.
Furthermore, they apply multiple grouping stages during training and this is also a hyper-parameter.
In our case, one grouping stage is enough for us to learn discriminative representations in the embedding space with the help of meaningful class tokens.
B Depth of Transformer Layers and Model Size
The depth of transformer layers affects the extracted audio-visual representations for grouping and the training parameters as well.
To explore such effect more comprehensively, we varied the depth of CUG and MCG modules from { 1, 2, 3, 4, 6, 8 } .
Table 3 reports the comparison results of the segment-level performance and the number of parameters.
When the depth of CUG and MCG isand 2, the proposed MGN with the least parameters 11 MB increases the baseline with parameters
MB by 1.7 Visual and 1.1 Event@AV.
With the increase of the depth of CUG, we achieve consistently improving performance of Visual and Audio-Visual for segment-level prediction since we extract better visual representations from transformers for class-aware unimodal grouping.
When the depth
[Table]
[Figure]
of CUG and MCG
is 3 and 6, the proposed MGN with only 47.2% parameters of the vanilla baseline performs the best on Type@AV and Event@AV, especially on Audio.
These results further shows the advantage of our MGN in real applications with lightweight parameters against the prior work [3, 4].
C Grouping Strategy
In addition, we ablated the strategy of CUG and MCG using Hard-Softmax and Softmax.
To make Hard-Softmax differentiable during training, we adopted the Gumbel-Softmax [43, 44] as the alternative.
Table 4 reports the ablation results.
With Softmax as both unimodal and cross-modal grouping strategies, the proposed MGN achieves the best performance.
Replacing the Softmax with the hard version in either unimodal or cross-modal grouping deteriorate the segment-level predictions, which also complies to the multi-label property of each segment for both modalities.
D More Analysis on Recall
event-level audio, visual audio visual metrics
Event
A
Event
In order to figure out the reduction of false positives in Figure 3, we compare the recall for allclasses in the test set.
The comparison results of event-level audio, visual and audio-visual metrics are shown in Figure 5.
We can observe that the reduction of false positives by the proposed
MGN over baselines indeed comes at the cost of a decrease in recall for audio, visual or audiovisual events.
E Quantitative Validation on Audio/Visual Class Tokens
In order to quantitatively validate the rationality of learned audio/visual token embeddings, we compute the Precision, Recall, and F1 scores of those tokens during training.
The quantitative results are shown in Figure 6.
We can observe that all metrics arrive to closer to 1 at epoch 10, which means that the learned class tokens are semantically class-aware.
Furthermore, these results demonstrate the effectiveness of class-aware tokens in the class-aware unimodal grouping for alleviating the temporal uncertainty in each modality.