Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing
Anonymous Author(s)
AffiliationAddress email
Abstract
The audio-visual video parsing task aims to parse a video into modality- andcategory-aware temporal segments.
Previous work mainly focuses on weaklysupervised approaches, which learn from video-level event labels.
During training,they do not know which modality perceives and meanwhile which temporal segment contains the video event.
Since there is no explicit grouping in the existingframeworks, the modality and temporal uncertainties make these methods sufferfrom false predictions.
For instance, segments in the same category could be predicted in different event classes.
Learning compact and discriminative multi-modalsubspaces is essential for mitigating the issue.
To this end, in this paper, we proposea novel Multi-modal Grouping Network, namely MGN, for explicitly semanticaware grouping.
Specifically, MGN aggregates event-aware unimodal featuresthrough unimodal grouping in terms of learnable categorical embedding tokens.
Furthermore, it leverages the cross-modal grouping for modality-aware predictionto match the video-level target.
Our simple framework achieves improving resultsagainst previous baselines on weakly-supervised audio-visual video parsing.
Inaddition, our MGN is much more lightweight, using only 47.2% of the parametersof baselines (17 MB vs. 36 MB).
Introduction
Humans understand the surrounding environment by integrating signals from different senses.
In ourdaily life, sound and sight are two of the most commonly used modalities, which have drawn muchattention from researchers to explore computational audio-visual scene understanding.
Previous audio-visual work [1, 2] usually assumes audio and visual data are temporally aligned.
However, the alignment does not always exist in real-world videos.
For example, sounding objects inmany videos are outside of the field-of-view (FoV).
For these non-aligned cases, audio signals becomemore reliable in understanding the events of interest.
In this work, we address the audio-visual videoparsing (AVVP) task [3] that aims to parse a video into temporal event segments and predict theaudible, visible, or audio-visible event categories.
Existing approaches [3, 4, 5] usually focus on learning to leverage the unimodal and cross-modaltemporal contexts from weak supervisions.
HAN [3] introduced a simple Multimodal Multiple
Instance Learning framework with cross-modal and self-modal attention layers to utilize the videolevel labels.
Recent state-of-the-art methods usually use the HAN as the baseline and modify itto further improve parsing performance.
Particularly, contrastive learning and label refinementare proposed in Wu and Yang [4], where they adopted a contrastive loss to enforce the temporal alignment between the audio and visual features at the same timestamp and augmented trainingdata with modality-aware event labels generation.
Furthermore, Lin et al.
[5] proposed to leverageaudio-visual class co-occurrence to jointly explore the relationship of different categories among allmodality streams.
Our key motivation is to learn compact and discriminative audio and visual representations by explicitmulti-modal grouping for mitigating the modality and temporal uncertainties in the weakly-supervisedaudio-visual video parsing problem. During training, segment-wise event labels are unavailable foraudio and visual temporal segments in videos.
Thus, the multi-modal temporal modeling in thesemethods might aggregate and implicitly group irrelevant semantic information due to lacking offine-grained supervisions, which causes false positives for predicting categories of events.
That is,audio or visual temporal segments in the same event category might be far away from the classcenter in the embedding space since there are no segment-level and modality-wise constraints duringtraining.
In the meanwhile, there is no constraint for modality category prediction to match thevideo-level target at the end.
Different from past approaches, we propose a new Multi-modal Grouping Network, namely MGN, toexplicitly group semantic-aware multi-modal contexts, which enables learning more compact anddiscriminative audio and visual representations.
Specifically, we first extract event-aware unimodalfeatures through unimodal grouping in terms of learnable categorical embedding tokens for eachindividual modality.
Then, we introduce a cross-attention layer with a hard attention mechanismto aggregate cross-modal temporal contexts.
Finally, we utilize a cross-modal grouping module topredict the modality category from updated class-aware unimodal embeddings.
Experimental results on the LLP [3] dataset validate that our new audio-visual video parsing framework achieves superior results over previous state-of-the-art methods [1, 2, 3, 4].
Empirical resultsalso demonstrate the generalizability of our approach to contrastive learning and label refinementproposed in MA [4].
In addition, we substantially reduce the parameters of previous work by usingonly 47.2% parameters of baselines (17 MB vs. 36 MB).
Our main contributions can be summarized as follows:
• We propose a new audio-visual video parsing baseline: Multi-modal Grouping Network(MGN) that enables explicit grouping in a multi-modal network to learn compact anddiscriminative audio and visual embeddings.• We introduce class-aware unimodal grouping and modality-aware cross-modal groupingmodules to aggregate multi-modal temporal contexts.•
The experiments can demonstrate the superiority of our MGN over state-of-the-art AVVP approaches and its generalizability to contrastive learning and label refinement.
Related Work
Audio-Visual Learning.
Audio-visual learning has addressed in many previous works [6, 7, 8, 9, 10,11, 12, 13, 14, 15, 16] to learn the audio-visual association between the two distinct yet correlatedmodalities.
Such audio-visual temporal association is crucial to several tasks, such as audio-visualspatialization [17, 18, 19, 14], speech/audio separation [13, 20, 21, 11, 12, 22, 23, 24], visual soundsource localization [10, 25, 26, 27, 28, 29].
In this work, we mainly focus on audio-visual videoparsing that aims to parse a video into temporal segments in terms of audio, visual, and audio-visualevents with only video-level annotations for training.
In addition, different from existing audio-visualarchitectures, we firstly introduce explicit multi-modal grouping into audio-visual learning.
Audio-Visual Video Parsing.
Audio-visual video parsing aims at temporally localizing audio, visual,and audio-visual events in videos and predicting their event categories.
Early approaches [1, 2, 30, 31]tried to localize only audio-visual events by aggregating cross-modal information in each localsegment of the input video.
Due to exhaustive labeling costs on segment-wise audio, visual, andaudio-visual events, recent work [3, 4] explored the weakly-supervised audio-visual video parsing
[Figure]
𝒇 !" !
#$% 𝒄 &" &#$’ 𝒈 &" &#$’ 𝒈 &( &#$’ 𝒄 &( &#$’ 𝒇 !
#$%
[Figure]
with only video-level activity categories for training.
For example, HAN [3] introduced the multimodal multiple instance learning mechanism with a hybrid attention network to aggregate segmentwise representations and video-level features.
To alleviate the harm brought by the audio-visualasynchrony, MA [4] proposed the audio-visual contrastive learning from different frames and refinedmodality-aware labels by exchanging audio and visual tracks between unrelated videos.
More recently,
Lin et al.
[5] leveraged the common semantics shared by videos with replaced audios or frames in amini batch to detect relevant events
Different from these baselines based on HAN, we develop a fully novel network architecture toalleviate implicit audio-visual matching and modality category uncertainty in the hybrid attentionnetwork.
We are the first to exploit unimodal grouping for learning audio-visual representations withclass-aware semantics.
Furthermore, we introduce a modality-aware cross-modal grouping module tomatch the video-level label, although the given target does not indicate modalities.
Method
Given a video with both audio and visual tracks, our goal is to parse the video into temporal segmentsassociated with audible, visible, and audio-visible event categories.
We propose a novel baseline:
Multi-modal Grouping Network (MGN) to address the task, which mainly consists of two modules:class-aware unimodal grouping (Sec. 3.2) and modality-aware cross-modal grouping (Sec. 3.3).
3.1 Preliminaries
In this section, we first describe the problem setup and notations and revisit Multi-modal Multiple
Instance Learning in HAN [3] for the audio-visual video parsing (AVVP) task.
Problem Setup and Notations.
Given a video with T non-overlapping audio and visual segments,our goal is to temporally localize and recognize audio, visual, and audio-visual events that existedin the video.
For the multi-label events with C event categories at time t , we have audio, visual,and audio-visual event labels for evaluation, which are denoted as: y at , y vt , y avt ∈ R 1 × C .
Duringtraining, we do not have the segment- and modality-level annotations.
Therefore, we can only use thevideo-level label y av ∈ R 1 × C to perform weakly-supervised learning.
Revisit Multimodal Multiple Instance Learning.
To address the weakly-supervised audio-visualvideo parsing problem, HAN [3] introduced a Multimodal Multiple Instance Learning (MMIL)framework based on transformers [32].
Given a set of audio-visual features F a = { f at } Tt =1 , F v ={ f vt } Tt =1 in T segments, HAN applied self-attention and cross-attention layers to aggregate theunimodal and cross-modal information at each timestamp:
[Equation]
[Equation]
where ϕ sa ( · ) , ϕ ca ( · ) denote the self-attention and cross-attention functions:
[Equation]
[Equation]
and f at , f vt ∈ R 1 × d , F a , F v ∈ R T × d , d is the dimension of audio-visual features.
Then, the probabilityof segment-wise categories for audio and visual events ( p at , p vt ∈ R 1 × C ) is predicted by a sharedfully-connected (FC) layer and sigmoid function.
With the attentive MMIL pooling, the video-levelprediction is formulated as:
[Equation]
where W t = { w at , w vt } is the temporal attention weights computed by a FC layer and normalized bya softmax function.
P t = { p at , p vt } is the probability set of audio-visual predictions.
M = 2 denotesaudio and visual modalities.
Finally, the model is trained to optimize a weakly-supervised loss ofp av and a guided loss of p a , p v with label smoothing:
[Equation]
where O ( · ) is a binary cross-entropy function-based loss term, which summarizes binary cross-entropyfor all categories and O ( y , p ) = (cid:80) i BCE ( y i , p i ) .
y a , y v are video-level audio and visual labelsgenerated by smoothing y av to decrease the confidence of positive labels.
However, such a training mechanism will pose two main challenges.
First, these methods withoutexplicit grouping suffer from false predictions due to the modality and temporal uncertainties.
Second,there is no constraint for modality category prediction to match the video-level target at the end.
To address these challenges, inspired by [33], we propose a novel Multi-modal Grouping Network(MGN) with class-aware unimodal grouping and modality-aware cross-modal grouping modules, asshown in Figure 1.
3.2 Class-aware Unimodal Grouping
In order to explicitly group class-aware matching semantics for audio-visual representations, weintroduce a novel class-aware unimodal grouping module by incorporating learnable modality-specificclass tokens { c ai } Ci =1 , { c vi } Ci =1 to help to group raw input unimodal features { f at } Tt =1 , { f vt } Tt =1 .
We first use self-attention transformers: ϕ asa ( · ) and ϕ vsa ( · ) to temporally aggregate unimodal featuresfrom audio and visual inputs and align the features with the categorical token embeddings:
[Equation] [Equation] [Equation]
where [ ; ] denotes the concatenation operator.
Then, the unimodal grouping blocks g a ( · ) , g v ( · )take the learned audio and visual event class tokens and aggregated features as inputs to generateclass-aware audio and visual embeddings as:
[Equation]
During grouping, we merge all the unimodal features that belong to the same class token into anew unimodal class-aware feature, by computing the similarity matrices
A a , A v between unimodalfeatures and class tokens via a softmax operation formulated as
[Equation]
where W aq , W ak and W vq , W vk are the weights of the learned linear projections for the features andclass tokens of audio and visual modalities, respectively.
Based on this similarity, we calculate theclass-aware features with the weighted sum of all segment features assigned to that class:
[Equation]
where W ao , W av and W vo , W vv denote the learned weights of linear projections for audio and visualmodalities, separately.
Note that, the audio features { ˆ f at } Tt =1 are merged with discriminative visualfeatures { f vt } Tt =1 via a similar grouping block with a hard-softmax operation.
In order to constrain theindependence of each class token c ai , c vi , we apply FC layers and softmax operation to generate thecategory probability e ai , e vi of class tokens for audio and visual modalities with a class-constrainedloss as: [Equation]
[Equation]
where CE( · ) refers to cross-entropy loss; h i is an one-hot encoding vector and only its element for thetarget class entry i is 1.
After the class-aware unimodal grouping, the video-level category predictionsp a , p v of audio and visual events is simply computed by a FC layer and sigmoid operator: [Equation]
[Equation]
With the help of the proposed class-constrained loss, we generate class-aware representations{ g ai } Ci =1 , { g vi } Ci =1 of audio and visual modalities for audio-visual matching.
3.3 Modality-aware Cross-modal Grouping
The second challenge requires us to predict the modality category for matching with the only givenvideo-level target in an explicit way.
To achieve this, we propose a modality-aware cross-modalgrouping module composed of cross-modal transformers ϕ ca ( · ) and grouping blocks g av ( · ) toaggregate class-aware representations { g ai } Ci =1 , { g vi } Ci =1 .
Based on the audio-visual similarity inthe grouping stage, we combine all the audio features with visual features into new cross-modalmodality-aware features { g avi } Ci =1 as:
[Equation]
[Equation]
where g av ( · ) denote the grouping operator similar to g a ( · ) and g v ( · ) in Eq.
Then, we leveragethe joint audio-visual representations { g avi } Ci =1 to predict the video-level target of audio and visualevents via a FC layer and sigmoid function as:
[Equation]
[Equation]
[Table]
C+R
The whole model can be optimized in an end-to-end manner in terms of the objective function: [Equation]
[Equation]
At inference time, the unimodal class-aware similarity is used to predict the audio, visual, andaudio-visual events for each segment t :
[Equation]
[Equation]
where p a , p v ∈ R 1 × C and A a , A v ∈ R T × C .
Experiments
4.1 Experimental Setup
Dataset.
The Look, Listen and Parse (LLP) Dataset [3] contains 11,849 YouTube video clips of10-seconds long from 25 different event categories, such as car, music, cheering, speech, etc.
Notethat each video includes at least 1s audio or visual events and 7202 video clips are annotated withmore than one event categories.
We use 10,000 video clips with only video-level event labels fortraining.
Following the official splits [3] of validation and test sets, we develop and test the model onthe remaining 1879 videos with the segment-level annotations, i.e. , the speech event for audio startsat 1s and ends at 5s.
Evaluation Metrics.
We follow the prior work [3, 4] and use F-scores to evaluate both segment-leveland event-level predictions for audio, visual, and audio-visual events.
The segment-level metricscan evaluate snippet-wise event prediction performance.
For the event-level metrics, we concatenatepositive consecutive segments in the same events and compute F-score based on mIoU=0.5 as thethreshold.
Type@AV and Event@AV are reported for the overall evaluation of audio-visual videoparsing performance.
Type@AV is the averaged audio, visual, and audio-visual event evaluationresults.
Different from Type@AV directly averaging results from different event types
Implementation.
Following the data pre-processing in previous work [3], we sample video framesat 8 fps from the 10-second videos with 10 non-overlapping snippets of 1 second.
For low-levelvisual features, we concatenate 2D and 3D visual features extracted by ResNet-152 [34] pre-trainedon ImageNet [35] and 3D ResNet [36] pre-trained on Kinetics-400 [37].
We utilize VGGish [38]pre-trained on AudioSet [39] to extract the audio features.
The model is trained with Adam [40]optimizer with β 1 =0.9, β 2 =0.999 and with an initial learning rate of 3e-4.
We train the model with abatch size of 16 for 40 epochs.
[Figure]
10 s
Vacuumcleaner
[Figure]
4.2 Comparison to Prior Work
In this work, we propose a novel and effective training framework for weakly-supervised audio-visualvideo parsing.
To demonstrate the effectiveness of our approach, we first compare it to previousnetwork baselines 1 : 1) AVE [1]: an audio-guided co-attention network with additional audio-visualparsing branches; 2) AVSDN [2]: a sequence-to-sequence-based model with additional audio-visualparsing branches to merge global audio-visual features into local ones; 3) HAN [3]: a hybrid attentionnetwork with multi-modal multiple instance learning pooling.
Furthermore, we compare MA [4],a method based on HAN with the audio-visual contrastive learning (C) and the label refinement(R).
The MA series baselines include MA (w C), MA (w R), and MA (w C+R).
The quantitativecomparisons on the LLP dataset are reported in Table 1.
As can be seen, the proposed MGN achieves the overall best results against previous networkbaselines in terms most of metrics.
For the overall evaluation of segment-level predictions, weachieve significant performance gains of 1.5 Type@AV and 1.8 Event@AV.
When evaluated onsegment-level predictions of each sample, our MGN also improves the baseline by large margins, 2.
Visual and 1.5 Audio-Visual.
Meanwhile, our MGN outperforms baselines by 3.5 Visual, 1.4 AudioVisual, and 1.6 Tyep@AV for event-level predictions.
These results demonstrate the effectiveness ofour approach in weakly-supervised audio-visual video parsing against prior network architectures.
Furthermore, significant gains can be observed in the setting of using the audio-visual contrastivelearning and label refinement.
Adding the contrastive learning to our MGN achieves the segment-levelperformance gain of 3.6 Visual and 2.8 Audio-Visual, and the event-level gain of 3.8 Visual and2.6 Audio-Visual.
With the label refinement in MA, we significantly improve MA (w R) by 3.1segment-level and 4.0 event-level for visual predictions.
Our framework with both contrastive learningand label refinement achieves the best segment-level performance in terms of Visual, Audio-Visual,
Type@AV, and Event@AV.
These improvements imply the strong generalizability of the
proposed
MGN to the audio-visual contrastive learning and the label refinement.
In order to qualitatively evaluate the predictions of audio and visual events, we compare the proposed
MGN with HAN [3], MA [4] in Figure 2.
We can observe three main things from previous approaches:
First, existing methods can miss the snippet-wise predictions for some hard segments in the video,e.g., MA does not predict the vacuum cleaner for visual events after 3s.
Second, prior work can missone class prediction if there are two categories happening in the same modality.
Third, the previous
[Table]
method can miss one modality prediction for the class existing in both modalities.
When it comes tothe proposed MGN, we superiorly achieve high F1 scores of segment-level and event-level prediction,benefiting from the well-designed class-aware unimodal grouping modality- -modal
4.3 Experimental Analysis
In this section, we conducted ablation studies to validate the benefit of Class-aware Unimodal
Grouping (CUG) and Modality-aware Cross-modal (MCG)
Grouping strategies.
We also performedextensive experiments to investigate the false positive issue in HAN and the learned class tokens.
Class-aware Unimodal Grouping & Modality-aware Cross-modal Grouping.
In order to demonstrate the effectiveness of the proposed class-aware unimodal grouping (CUG) and modality-awarecross-modal grouping (MCG), we ablated the necessity and strategy of grouping blocks.
The resultsof segment-level predictions are reported in Table 2.
We can observe that adding CUG to the vanillabaseline achieves significant gains of 2.4 Visual, indicating the effectiveness of grouping class-awaresemantics in predicting snippet-wise categories for visual events.
Incorporating MCG with CUGhighly increases Audio-Visual, Tyep@AV, Event@AV by 1.5, 1.5 and 1.8.
These results show theimportance of modality-aware grouping on predictions of audio-visual events.
Besides effectiveness,our model is also more efficient.
When the depth of CUG and MCG is 3 and 6, the proposed MGNwith only 47.2% parameters of the vanilla baseline performs the best on Type@AV and Event@AV,especially on Audio.
These results further show the advantage of our MGN in real applications
withlightweight parameters against the prior work [3, 4].
The detailed results are in Appendix.
False Predictions.
In order to demonstrate theeffectiveness of the proposed MGN in mitigating false predictions against baselines, we calculate the total amount of false positives for all 25classes in the test set.
The comparison results ofevent-level audio, visual and audio-visual metrics are shown in Figure 3.
We can observethat our MGN with the class-aware unimodalgrouping modules decreases the false positivesof audio and visual events by large margins, 381and 494.
Furthermore, the number of false positives of audio-visual events drops down by 678,which verifies the importance of modality-awarecross-modal grouping in mitigating the modalityuncertainty.
Overall, our MGN with explicit grouping mechanisms significantly eliminates falsepredictions caused by the modality and temporal uncertainties existing in the baseline.
Learned Class-aware Features.
The learned class tokens are essential to grouping class-awaresemantics from audio and visual features.
To better evaluate the quality of those learned class-levelfeatures, we visualize the learned audio and visual representations of 25 categories by t-SNE [41], asshown in Figure 4.
It is noted that each spot denotes the feature of one audio or visual event, whileeach color represents each class, such as “Speech” in brown and “Dog” in green.
As can be seenin the last column, features extracted by the proposed MGN are intra-class compact and inter-classseparable.
However, there still exists mixtures of multiple categories for audio and visual events among the representations of HAN and MA.
For the sub-figure on the bottom right, we can observe alarge cluster of brown spots for the “Speech” class of audio events in the test set, while brown spotsin prior work are distributed more sparsely.
These meaningful visualizations further demonstrate thatour MGN successfully learns compact and discriminative features for each modality.
4.4 Limitation
Although the proposed MGN achieves superior results on visual events and audio-visual events, thegains of audio events are not significant compared to the visual modality.
We notice that there are1628 visual instances and 2663 audio instances in the test set, which implies the audio modality ismuch harder than the visual one in this setting.
One possible solution is to introduce a small numberof segment-wise audio-visual parsing annotations as supervision for semi-supervised training.
In themeanwhile, our MGN performs worse with the increase of the depth of transformer layers in groupingmodules.
This is caused by such a weakly-supervised setting with only video-level annotations thatdo not indicate either segments or modalities.
However, the model is expected to parse a video intoevents with different categories and modalities.
Therefore, the potential future work is to add moregrouping stages with learned class-tokens as supervision for each one.
Conclusion
In this work, we present MGN, a fully novel Multi-modal Grouping Network to explicitly groupclass-aware matching semantics for weakly-supervised audio-visual video parsing.
We introducethe class-aware unimodal grouping module to generate class-aware unimodal representations withlearnable tokens by using unimodal grouping blocks for each modality.
Furthermore, we leverage themodality-aware cross-modal grouping to match the video-level target with the cross-modal groupingblocks.
Experimental results demonstrate the effectiveness and superiority of our MGN againstprevious baselines.
We also show the generalizability of our simple framework to the audio-visualcontrastive learning and label refinement.
Broader Impact.
The proposed method detects video events in audio and visual modalities basedon the learned statistics of the training dataset.
It could capture internal biases in the data, whichmay have negative societal impacts.
For example, the model might not be able to discover rare butimportant events, such as fire warnings.
Thus, before deploying our audio-visual video parsing modelinto real-world applications, we need to carefully address these issues. Appendix
In the supplementary material, we provide the differences between our MGN and the recent work,
GroupViT [33], more experiments on model size and grouping strategies.
We finally validatethe effectiveness of audio/visual tokens in learning class-aware embeddings.
The code of ourimplementation is also attached as the supplementary material for review.
A Differences between GroupViT [33] and the proposed MGN
Compared to GroupViT [33], the recent grouping work on image segmentation, there are three maindistinctive characteristics of our MGN for solving the weakly-supervised auido-visual video parsingproblem, which are listed as follows:
1) Constraint on Class Tokens.
Most important, we have learned 25 category tokens with each class
constraint, i.e. , each class token does not have semantic overlapping information among each other
and we want
each token to
learn the
embeddings
for each category.
To
achieve
this
, we apply the cross-
entropy loss to
class tokens
for audio-visual modalities as L cls = CE
( { p ai } Ci
=
, 1 )+ CE
{ p vi
} Ci =1 , 1 ) .However,
the number of learned group tokens in GroupViT is a hyper-parameter and there is noconstraint
on it.
2) Modality-aware Cross-modal Grouping.
We have the modality-aware cross-modal groupingto eliminate the modality uncertainty caused by the weakly-supervised setting in the task, whileGroupViT does not incorporate text embeddings into the grouping stage.
The text embeddings isused in a contrastive loss to match with the global visual representations.
In this case, their modelwas trained on large-scale data with a large batch size for self-supervised training.
3) Class-aware Unimodal Grouping.
We introduce the class-aware unimodal grouping for mitigating
the temporal uncertainty in each modality, but GroupViT only leverages the unimodal grouping onvisual patch tokens without class tokens involved.
Therefore, the unimodal grouping in GroupViT cannot be used in this task directly.
Furthermore, they apply multiple grouping stages during training andthis is also a hyper-parameter.
In our case, one grouping stage is enough for us to learn discriminativerepresentations in the embedding space with the help of meaningful class tokens.
B Depth of Transformer Layers and Model Size
The depth of transformer layers affects the extracted audio-visual representations for grouping andthe training parameters as well.
To explore such effect more comprehensively, we varied the depthof CUG and MCG modules from { 1, 2, 3, 4, 6, 8 } .
Table 3 reports the comparison results of thesegment-level performance and the number of parameters.
{+When the depth of CUG and MCG is 1and 2, the proposed MGN with the least parameters 11 MB increases the baseline with parameters 
+}
MB by 1.7 Visual and 1.1 Event@AV.
With the increase of the depth of CUG, we achieve consistently
[Table]
improving performance of Visual and Audio-Visual for segment-level prediction since we extractbetter visual representations from transformers for class-aware unimodal grouping.
When the depthof CUG and MCG is 3 and 6, the proposed MGN with only 47.2% parameters of the vanilla baselineperforms the best on Type@AV and Event@AV, especially on Audio.
These results further shows theadvantage of our MGN in real applications with lightweight parameters against the prior work [3, 4].
C Grouping Strategy
In addition, we ablated the strategy of CUG and MCG using Hard-Softmax and Softmax.
Tomake Hard-Softmax differentiable during training, we adopted the Gumbel-Softmax [42, 43] as thealternative.
Table 4 reports the ablation results.
With Softmax as both unimodal and cross-modalgrouping strategies, the proposed MGN achieves the best performance.
Replacing the Softmax withthe hard version in either unimodal or cross-modal grouping deteriorate the segment-level predictions,which also complies to the multi-label property of each segment for both modalities.
AV.
D More Analysis on Recall
In order to figure out the reduction of false positives in Figure 3, we compare the recall for all 25classes in the test set.
The comparison results ofevent-level audio, visual and audio-visual metrics are shown in Figure 5.
We can observe thatthe reduction of false positives by
the proposed
MGN over baselines indeed comes at the costof a decrease in recall for audio, visual or audiovisual events.
E Quantitative Validationon Audio/Visual Class Tokens
In order to quantitatively validate the rationalityof learned audio/visual token embeddings, we compute the Precision, Recall, and F1 scores of thosetokens during training.
The quantitative results are shown in Figure 6.
We can observe that allmetrics arrive to closer to 1 at epoch 10, which means that the learned class tokens are semanticallyclass-aware.
Furthermore, these results demonstrate the effectiveness of class-aware tokens in theclass-aware unimodal grouping for alleviating the temporal uncertainty in each modality.
[Figure]
F
[Figure]