Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing
Anonymous Author(s)
AffiliationAddress email
Abstract
The audio-visual video parsing task aims to parse a video into modality- andcategory-aware temporal segments.
Previous work mainly focuses on weaklysupervised approaches, which learn from video-level event labels.
During training,they do not know which modality perceives and meanwhile which temporal segment contains the video event.
Since there is no explicit grouping in the existingframeworks, the modality and temporal uncertainties make these methods sufferfrom false predictions.
For instance, segments in the same category could be predicted in different event classes.
Learning compact and discriminative multi-modalsubspaces is essential for mitigating the issue.
To this end, in this paper, we proposea novel Multi-modal Grouping Network, namely MGN, for explicitly semanticaware grouping.
Specifically, MGN aggregates event-aware unimodal featuresthrough unimodal grouping in terms of learnable categorical embedding tokens.
Furthermore, it leverages the cross-modal grouping for modality-aware predictionto match the video-level target.
Our simple framework achieves improving resultsagainst previous baselines on weakly-supervised audio-visual video parsing.
Inaddition, our MGN is much more lightweight, using only 47.2% of the parametersof baselines (17 MB vs. 36 MB).
Introduction
Humans understand the surrounding environment by integrating signals from different senses.
In ourdaily life, sound and sight are two of the most commonly used modalities, which have drawn muchattention from researchers to explore computational audio-visual scene understanding.
Previous audio-visual work [1, 2] usually assumes audio and visual data are temporally aligned.
However, the alignment does not always exist in real-world videos.
For example, sounding objects inmany videos are outside of the field-of-view (FoV).
For these non-aligned cases, audio signals becomemore reliable in understanding the events of interest.
In this work, we address the audio-visual videoparsing (AVVP) task [3] that aims to parse a video into temporal event segments and predict theaudible, visible, or audio-visible event categories.
The existing approaches mainly focus on learning to leverage the unimodal and cross-modal temporalcontexts simultaneously with only video-level event labels for training.
HAN [3] introduced a simple
Multimodal Multiple Instance Learning framework with cross-modal and self-modal attention layersto utilize the video-level labels.
Recent state-of-the-art methods usually use the HAN as the baselineand modify it to further improve parsing performance.
Particularly, Wu and Yang [4] proposed toaugment training data with modality-aware event labels generation and adopted a contrastive loss to enforce the temporal alignment between the audio and visual features at the same timestamp.
Furthermore, Lin et al.
[5] proposed to leverage audio-visual class co-occurrence to jointly explorethe relationship of different categories among all modality streams.
Weakly-supervised audio-visual video parsing is essentially challenging. During training, we canonly access video-level event labels, but we do not know which temporal segments contain andwhich modalities perceive these events. To mitigate the modality and temporal uncertainties, compactand discriminative audio and visual embeddings are crucial.
However, without explicitly grouping,previous works are not capable of learning such uni-modal and multi-modal representations, whichcauses false positives for predicting categories of events.
That is, audio or visual temporal segmentsin the same event category might be far away from the class center in the embedding space sincethere are no segment-level and modality-wise constraints during training.
In the meanwhile, there isno constraint for modality category prediction to match the video-level target at the end.
In this paper, we propose a new Multi-modal Grouping Network, namely MGN, for explicitly groupingclass-aware matching semantics.
Specifically, we first extract event-aware unimodal features throughunimodal grouping in terms of learnable categorical embedding tokens for each individual modality.
Then, we introduce a cross-attention layer with a hard attention mechanism to aggregate cross-modaltemporal contexts.
Finally, we utilize a cross-modal grouping module to predict the modality categoryfrom updated class-aware unimodal embeddings.
Experimental results on the LLP [3] dataset validate that our new audio-visual video parsing framework achieves superior results over previous state-of-the-art methods [1, 2, 3, 4].
Empirical resultsalso demonstrate the generalizability of our approach to contrastive learning and label refinementproposed in MA [4].
In addition, we substantially reduce the parameters of previous work by usingonly 47.2% parameters of baselines (17 MB vs. 36 MB).
Our main contributions can be summarized as follows:
• We propose a new audio-visual video parsing baseline: Multi-modal Grouping Network (MGN) that enables explicit grouping in a multi-modal network to learn compact and discriminative audio and visual embeddings. • We introduce class-aware unimodal grouping and modality-aware cross-modal grouping
modules to aggregate multi-modal temporal contexts.•
The experiments can demonstrate the superiority of our MGN over state-of-the-art AVVPapproaches and its generalizability to contrastive learning and label refinement.
Related Work
Audio-Visual Learning.
Audio-visual learning has addressed in many previous works [6, 7, 8, 9, 10,11, 12, 13, 14, 15, 16] to learn the audio-visual association between the two distinct yet correlatedmodalities.
Such audio-visual temporal association is crucial to several tasks, such as audio-visualspatialization [17, 18, 19, 14], speech/audio separation [13, 20, 21, 11, 12, 22, 23, 24], visual soundsource localization [10, 25, 26, 27, 28, 29].
In this work, we mainly focus on audio-visual videoparsing that aims to parse a video into temporal segments in terms of audio, visual, and audio-visualevents with only video-level annotations for training.
In addition, different from existing audio-visualarchitectures, we firstly introduce explicit multi-modal grouping into audio-visual learning.
Audio-Visual Video Parsing.
Audio-visual video parsing aims at temporally localizing audio, visual,and audio-visual events in videos and predicting their event categories.
Early approaches [1, 2, 30, 31]tried to localize only audio-visual events by aggregating cross-modal information in each localsegment of the input video.
Due to exhaustive labeling costs on segment-wise audio, visual, andaudio-visual events, recent work [3, 4] explored the weakly-supervised audio-visual video parsingwith only video-level activity categories for training. For example, HAN [3] introduced the multimodal multiple instance learning mechanism with a hybrid attention network to aggregate segmentwise representations and video-level features. To alleviate the harm brought by the audio-visual
[Figure]
𝒇 !" !
#$% 𝒄 &" &#$’ 𝒈 &" &#$’ 𝒈 &( &#$’ 𝒄 &( &#$’ 𝒇 !
#$%
[Figure]
asynchrony, MA [4] proposed the audio-visual contrastive learning from different frames and refinedmodality-aware labels by exchanging audio and visual tracks between unrelated videos.
More recently,
Lin et al.
[5] leveraged the common semantics shared by videos with replaced audios or frames in amini batch to detect relevant events.
Different from these baselines based on HAN, we develop a fully novel network architecture toalleviate implicit audio-visual matching and modality category uncertainty in the hybrid attentionnetwork.
We are the first to exploit unimodal grouping for learning audio-visual representations withclass-aware semantics.
Furthermore, we introduce a modality-aware cross-modal grouping module tomatch the video-level label, although the given target does not indicate modalities.
Method
Given a video with both audio and visual tracks, our goal is to parse the video into temporal segmentsassociated with audible, visible, and audio-visible event categories.
We propose a novel baseline:
Multi-modal Grouping Network (MGN) to address the task, which mainly consists of two modules:class-aware unimodal grouping (Sec. 3.2) and modality-aware cross-modal grouping (Sec. 3.3).
3.1 Preliminaries
In this section, we first describe the problem setup and notations and revisit Multi-modal Multiple
Instance Learning in HAN [3] for the audio-visual video parsing (AVVP) task.
Problem Setup and Notations.
Given a video with T non-overlapping audio and visual segments,our goal is to temporally localize and recognize audio, visual, and audio-visual events that existed in the video.
For the multi-label events with C event categories at time t , we have audio, visual,and audio-visual event labels for evaluation, which are denoted as: y at , y vt , y avt ∈ R 1 × C .
Duringtraining, we do not have the segment- and modality-level annotations.
Therefore, we can only use thevideo-level label y av ∈ R 1 × C to perform weakly-supervised learning.
Revisit Multimodal Multiple Instance Learning.
To address the weakly-supervised audio-visualvideo parsing problem, HAN [3] introduced a Multimodal Multiple Instance Learning (MMIL)framework based on transformers [32].
Given a set of audio-visual features F a = { f at } Tt =1 , F v ={ f vt } Tt =1 in T segments, HAN applied self-attention and cross-attention layers to aggregate theunimodal and cross-modal information at each timestamp:
[Equation]
[Equation]
where ϕ sa ( · ) , ϕ ca ( · ) denote the self-attention and cross-attention functions:
[Equation]
[Equation]
and f at , f vt ∈ R 1 × d , F a , F v ∈ R T × d , d is the dimension of audio-visual features.
Then, the probabilityof segment-wise categories for audio and visual events ( p at , p vt ∈ R 1 × C ) is predicted by a sharedfully-connected (FC) layer and sigmoid function.
With the attentive MMIL pooling, the video-levelprediction is formulated as:
[Equation]
t [Equation] [Equation] where W t = { w at , w vt } is the temporal attention weights computed by a FC layer and normalized bya softmax function.
P t = { p at , p vt } is the probability set of audio-visual predictions.
M = 2 denotesaudio and visual modalities.
Finally, the model is trained to optimize a weakly-supervised loss ofp av and a guided loss of p a , p v with label smoothing:
[Equation]
where CE ( · ) is the cross-entropy function,
y a , y v are video-level audio and visual labels generatedby smoothing y av to decrease the confidence of positive labels.
However, such a training mechanism will pose two main challenges.
First, these methods withoutexplicit grouping suffer from false predictions due to the modality and temporal uncertainties.
Second,there is no constraint for modality category prediction to match the video-level target at the end.
To address these challenges, inspired by [33], we propose a novel Multi-modal Grouping Network(MGN) with class-aware unimodal grouping and modality-aware cross-modal grouping modules, asshown in Figure 1.
3.2 Class-aware Unimodal Grouping
In order to explicitly grouping class-aware matching semantics for audio-visual representations, weintroduce a novel class-aware unimodal grouping module by incorporating learnable modality-specificclass tokens { c ai } C i =1 , { c vi } C i =1 to help to group raw input unimodal features { f at } T t =1 , { f v t } Tt =1 .
We first use self-attention transformers: ϕ asa ( · ) and ϕ vsa ( · ) to temporally aggregate unimodal featuresfrom audio and visual inputs and align the features with the categorical token embeddings:
[Equation] [Equation]
8) where [ ; ] denotes the concatenation operator.
Then, the unimodal grouping blocks g a ( · ) , g v ( · )take the learned audio and visual event class tokens and aggregated features as inputs to generateclass-aware audio and visual embeddings as:
[Equation]
During grouping, we merge all the unimodal features that belong to the same class token into anew unimodal class-aware feature, by computing the similarity matrices
A a , A v between unimodalfeatures and class tokens via a softmax operation formulated as
[Equation]
where W aq , W ak and W vq , W vk are the weights of the learned linear projections for the features andclass tokens of audio and visual modalities, respectively.
Based on this similarity, we calculate theclass-aware features with the weighted sum of all segment features assigned to that class:
[Equation]
where W ao , W av and W vo , W vv denote the learned weights of linear projections for audio and visualmodalities, separately.
Note that, the audio features { ˆ f at } Tt =1 are merged with discriminative visualfeatures { f vt } Tt =1 via a similar grouping block with a hard-softmax operation.
In order to constrainthe independence of each class token c ai , c vi , we apply FC layers to generate the probability p ai , p vi ofclass tokens for audio and visual modalities with a class-constrained loss as:
[Equation]
where 1 ∈ R 1 × C denotes the target of 1 for each class i .
After the class-aware unimodal grouping,the video-level prediction of audio and visual events is simply computed by a FC layer and sigmoidoperator:
[Equation]
With the help of the proposed class-constrained loss, we generate class-aware representations{ g ai } Ci =1 , { g vi } Ci =1 of audio and visual modalities for audio-visual matching.
3.3 Modality-aware Cross-modal Grouping
The second challenge requires us to predict the modality category for matching with the only givenvideo-level target in an explicit way.
To achieve this, we propose a modality-aware cross-modalgrouping module composed of cross-modal transformers ϕ ca ( · ) and grouping blocks g av ( · ) toaggregate class-aware representations { g ai } Ci =1 , { g vi } Ci =1 .
Based on the audio-visual similarity inthe grouping stage, we combine all the audio features with visual features into new cross-modalmodality-aware features { g avi } Ci =1 as:
[Equation]
[Equation]
15) where g av ( · ) denote the grouping operator similar to g a ( · ) and g v ( · ) in Eq.
Then, we leveragethe joint audio-visual representations { g avi } Ci =1 to predict the video-level target of audio and visualevents via a FC layer and sigmoid function as:
[Equation]
[Table]
C+R
The whole model can be optimized in an end-to-end manner in terms of the objective function:
[Equation]
At inference time, the unimodal class-aware similarity is used to predict the audio, visual, andaudio-visual events for each segment t :
[Equation]
[Equation]
where p a , p v ∈ R 1 × C and A a , A v ∈ R T × C .
Experiments
4.1 Experimental Setup
Dataset.
The Look, Listen and Parse (LLP) Dataset [3] contains 11,849 YouTube video clips of10-seconds long from 25 different event categories, such as car, music, cheering, speech, etc.
Notethat each video includes at least 1s audio or visual events and 7202 video clips are annotated withmore than one event categories.
We use 10,000 video clips with only video-level event labels fortraining.
Following the official splits [3] of validation and test sets, we develop and test the model onthe remaining 1879 videos with the segment-level annotations, i.e. , the speech event for audio startsat 1s and ends at 5s.
Evaluation Metrics.
We follow the prior work [3, 4] and use F-scores to evaluate both segment-leveland event-level predictions for audio, visual, and audio-visual events.
The segment-level metricscan evaluate snippet-wise event prediction performance.
For the event-level metrics, we concatenatepositive consecutive segments in the same events and compute F-score based on mIoU=0.5 as thethreshold.
Type@AV and Event@AV are reported for the overall evaluation of audio-visual videoparsing performance.
Type@AV is the averaged audio, visual, and audio-visual event evaluationresults.
Different from Type@AV directly averaging results from different event types, sample.
Implementation.
Following the data pre-processing in previous work [3], we sample video framesat 8 fps from the 10-second videos with 10 non-overlapping snippets of 1 second.
For low-levelvisual features, we concatenate 2D and 3D visual features extracted by ResNet-152 [34] pre-trainedon ImageNet [35] and 3D ResNet [36] pre-trained on Kinetics-400 [37].
We utilize VGGish [38]pre-trained on AudioSet [39] to extract the audio features.
The model is trained with Adam [40]optimizer with β 1 =0.9, β 2 =0.999 and with an initial learning rate of 3e-4.
We train the model with abatch size of 16 for 40 epochs.
[Figure]
10 s
Vacuumcleaner
[Figure]
4.2 Comparison to Prior Work
In this work, we propose a novel and effective training framework for weakly-supervised audio-visualvideo parsing.
To demonstrate the effectiveness of our approach, we first compare it to previousnetwork baselines 1 : 1) AVE [1]: an audio-guided co-attention network with additional audio-visualparsing branches; 2) AVSDN [2]: a sequence-to-sequence-based model with additional audio-visualparsing branches to merge global audio-visual features into local ones; 3) HAN [3]: a hybrid attentionnetwork with multi-modal multiple instance learning pooling.
Furthermore, we compare MA [4],a method based on HAN with the audio-visual contrastive learning (C) and the label refinement(R).
The MA series baselines include MA (w C), MA (w R), and MA (w C+R).
The quantitativecomparisons on the LLP dataset are reported in Table 1.
As can be seen, the proposed MGN achieves the overall best results against previous networkbaselines in terms most of metrics.
For the overall evaluation of segment-level predictions, weachieve significant performance gains of 1.5 Type@AV and 1.8 Event@AV.
When evaluated onsegment-level predictions of each sample, our MGN also improves the baseline by large margins, 2.
Visual and 1.5 Audio-Visual.
Meanwhile, our MGN outperforms baselines by 3.5 Visual, 1.4 AudioVisual, and 1.6 Tyep@AV for event-level predictions.
These results demonstrate the effectiveness ofour approach in weakly-supervised audio-visual video parsing against prior network architectures.
Furthermore, significant gains can be observed in the setting of using the audio-visual contrastivelearning and label refinement.
Adding the contrastive learning to our MGN achieves the segment-levelperformance gain of 3.6 Visual and 2.8 Audio-Visual, and the event-level gain of 3.8 Visual and2.6 Audio-Visual.
With the label refinement in MA, we significantly improve MA (w R) by 3.1segment-level and 4.0 event-level for visual predictions.
Our framework with both contrastive learningand label refinement achieves the best segment-level performance in terms of Visual, Audio-Visual,
Type@AV, and Event@AV.
These improvements imply the strong generalizability of the
proposed
MGN to the audio-visual contrastive learning and the label refinement.
In order to qualitatively evaluate the predictions of audio and visual events, we compare the proposed
MGN with HAN [3], MA [4] in Figure 2.
We can observe three main things from previous approaches:
First, existing methods can miss the snippet-wise predictions for some hard segments in the video,e.g., MA does not predict the vacuum cleaner for visual events after 3s.
Second, prior work can missone class prediction if there are two categories happening in the same modality.
Third, the previous
[Table]
method can miss one modality prediction for the class existing in both modalities.
When it comes tothe proposed MGN, we superiorly achieve high F1 scores of segment-level and event-level prediction,benefiting from the well-designed class-aware unimodal grouping modality-aware cross-modal
4.3 Experimental Analysis
In this section, we conducted ablation studies to validate the benefit of Class-aware Unimodal
Grouping (CUG) and Modality-aware Cross-modal (MCG)
Grouping strategies.
We also performedextensive experiments to investigate the false positive issue in HAN and the learned class tokens.
Class-aware Unimodal Grouping & Modality-aware Cross-modal Grouping.
In order to demonstrate the effectiveness of the proposed class-aware unimodal grouping (CUG) and modality-awarecross-modal grouping (MCG), we ablated the necessity and strategy of grouping blocks.
The resultsof segment-level predictions are reported in Table 2.
We can observe that adding CUG to the vanillabaseline achieves significant gains of 2.4 Visual, indicating the effectiveness of grouping class-awaresemantics in predicting snippet-wise categories for visual events.
Incorporating MCG with CUGhighly increases Audio-Visual, Tyep@AV, Event@AV by 1.5, 1.5 and 1.8.
These results show theimportance of modality-aware grouping on predictions of audio-visual events.
Besides effectiveness,our model is also more efficient.
When the depth of CUG and MCG is 3 and 6, the proposed MGNwith only 47.2% parameters of the vanilla baseline performs the best on Type@AV and Event@AV,especially on Audio.
These results further show the advantage of our MGN in real applications
withlightweight parameters against the prior work [3, 4].
The detailed results are in Appendix.
False Predictions.
In order to demonstrate theeffectiveness of the proposed MGN in mitigating false predictions against baselines, we calculate the total amount of false positives for all 25classes in the test set.
The comparison results ofevent-level audio, visual and audio-visual metrics are shown in Figure 3.
We can observethat our MGN with the class-aware unimodalgrouping modules decreases the false positivesof audio and visual events by large margins, 381and 494.
Furthermore, the number of false positives of audio-visual events drops down by 678,which verifies the importance of modality-awarecross-modal grouping in mitigating the modalityuncertainty.
Overall, our MGN with explicit grouping mechanisms significantly eliminates falsepredictions caused by the modality and temporal uncertainties existing in the baseline.
Learned Class-aware Features.
The learned class tokens are essential to grouping class-awaresemantics from audio and visual features.
To better evaluate the quality of those learned class-levelfeatures, we visualize the learned audio and visual representations of 25 categories by t-SNE [41], asshown in Figure 4.
It is noted that each spot denotes the feature of one audio or visual event, whileeach color represents each class, such as “Speech” in brown and “Dog” in green.
As can be seenin the last column, features extracted by the proposed MGN are intra-class compact and inter-classseparable.
However, there still exists mixtures of multiple categories for audio and visual events among the representations of HAN and MA.
For the sub-figure on the bottom right, we can observe alarge cluster of brown spots for the “Speech” class of audio events in the test set, while brown spotsin prior work are distributed more sparsely.
These meaningful visualizations further demonstrate thatour MGN successfully learns compact and discriminative features for each modality.
4.4 Limitation
Although the proposed MGN achieves superior results on visual events and audio-visual events, thegains of audio events are not significant compared to the visual modality.
We notice that there are1628 visual instances and 2663 audio instances in the test set, which implies the audio modality ismuch harder than the visual one in this setting.
One possible solution is to introduce a small numberof segment-wise audio-visual parsing annotations as supervision for semi-supervised training.
In themeanwhile, our MGN performs worse with the increase of the depth of transformer layers in groupingmodules.
This is caused by such a weakly-supervised setting with only video-level annotations thatdo not indicate either segments or modalities.
However, the model is expected to parse a video intoevents with different categories and modalities.
Therefore, the potential future work is to add moregrouping stages with learned class-tokens as supervision for each one.
Conclusion
In this work, we present MGN, a fully novel Multi-modal Grouping Network to explicitly groupclass-aware matching semantics for weakly-supervised audio-visual video parsing.
We introducethe class-aware unimodal grouping module to generate class-aware unimodal representations withlearnable tokens by using unimodal grouping blocks for each modality.
Furthermore, we leverage themodality-aware cross-modal grouping to match the video-level target with the cross-modal groupingblocks.
Experimental results demonstrate the effectiveness and superiority of our MGN againstprevious baselines.
We also show the generalizability of our simple framework to the audio-visual contrastive learning and label refinement
Broader Impact.
The proposed method detects video events in audio and visual modalities basedon the learned statistics of the training dataset.
It could capture internal biases in the data, whichmay have negative societal impacts.
For example, the model might not be able to discover rare butimportant events, such as fire warnings.
Thus, before deploying our audio-visual video parsing modelinto real-world applications, we need to carefully address these issues.
[-When the depth of CUG and MCG is 1and 2, the proposed MGN with the least parameters 11 MB increases the baseline with parameters 
-]