C ROSS -A TTENTIONAL A UDIO -V ISUAL F USION FOR W EAKLY -S UPERVISED A CTION L OCALIZATION
A BSTRACT
Temporally localizing actions in videos is one of the key components for video understanding.
Learning from weakly-labeled data is seen as a potential solution towards avoiding expensive frame-level annotations.
Different from other works which only depend on visual-modality, we propose to learn richer audiovisual representation for weakly-supervised action localization.
First, we propose a multi-stage cross-attention mechanism to collaboratively fuse audio and visual features, which preserves the intra-modal characteristics.
Second, to model both foreground and background frames, we construct an open-max classiﬁer which treats the background class as an open-set.
Third, for precise action localization, we design consistency losses to enforce temporal continuity for the actionclass prediction, and also help with foreground-prediction reliability.
Extensive experiments on two publicly available video-datasets (AVE and ActivityNet1.2) show that the proposed method effectively fuses audio and visual modalities, and achieves the state-of-the-art results for weakly-supervised action localization.
I NTRODUCTION
The goal of this paper is to temporally localize actions and events of interest in videos with weaksupervision.
In the weakly-supervised setting, only video-level labels are available during the training phase to avoid expensive and time-consuming frame-level annotation.
This task is of great importance for video analytics and understanding.
Several weakly-supervised methods have been developed for it (Nguyen et al., 2018;
Paul et al., 2018; Narayan et al., 2019; Shi et al., 2020; Jain et al., 2020) and considerable progress has been made.
However, only visual information is exploited for this task and audio modality has been mostly overlooked.
Both, audio and visual data often depict actions from different viewpoints (Guo et al., 2019).
Therefore, we propose to explore the joint audio-visual representation to improve the temporal action localization in videos.
A few existing works (Tian et al., 2018; Lin et al., 2019; Xuan et al., 2020) have attempted to fuse audio and visual modalities to localize audio-visual events .
These methods have shown promising results, however, these audio-visual events are essentially actions that have strong audio cues, such as playing guitar, and dog barking.
Whereas, we aim to localize wider range of actions related to sports, exercises, eating etc.
Such actions can also have weak audio aspect and/or can be devoid of informative audio (e.g. with unrelated background music).
Therefore, it is a key challenge to fuse audio and visual data in a way that leverages the mutually complementary nature while maintaining the modality-speciﬁc information.
In order to address this challenge, we propose a novel multi-stage cross-attention mechanism.
It progressively learns features from each modality over multiple stages.
The inter-modal interaction is allowed at each stage only through cross-attention, and only at the last stage are the visuallyaware audio features and audio-aware visual features concatenated.
Thus, an audio-visual feature representation is obtained for each snippet in videos.
Separating background from actions/events is a common problem in temporal localization.
To this end, we also propose: (a) foreground reliability estimation and classiﬁcation via open-max classiﬁer and (b) temporal continuity losses.
First, for each video snippet, an open-max classiﬁer predicts scores for action and background classes, which is composed of two parallel branches for action classiﬁcation and foreground reliability estimation.
Second, for precise action localization with weak supervision, we design temporal consistency losses to enforce temporal continuity of actionclass prediction and foreground reliability.
We demonstrate the effectiveness of the proposed method for weakly-supervised localization of both audio-visual events and actions.
Extensive experiments are conducted on two video datasets for localizing audio-visual events (AVE 1 ) and actions (ActivityNet1.2 2 ).
To the best of our knowledge, it is the ﬁrst attempt to exploit audio-visual fusion for temporal localization of unconstrained actions in long videos.
R ELATED WORK
Our work relates to the tasks of localizing of actions and events in videos, as well as to the regime of multi-model representation learning.
Weakly-supervised action localization: Wang et al.
(2017) and Nguyen et al.
(2018) employed multiple instance learning (Dietterich et al., 1997) along with attention mechanism to localize actions in videos.
Paul et al.
(2018) introduced a co-activity similarity loss that looks for similar temporal regions in a pair of videos containing a common action class.
Narayan et al.
(2019) proposed center loss for the discriminability of action categories at the global-level and counting loss for separability of instances at the local-level.
To alleviate the confusion due to background (nonaction) segments, Nguyen et al.
(2019) developed the top-down class-guided attention to model background, and (Yu et al., 2019) exploited temporal relations among video segments.
Jain et al.
(2020) segmented a video into interpretable fragments, called ActionBytes, and used them effectively for action proposals.
To distinguish action and context (near-action) snippets, Shi et al.
(2020) designed the class-agnostic frame-wise probability conditioned on the attention using conditional variational auto-encoder.
Luo et al.
(2020) proposed an expectation-maximization multi-instance learning framework where the key instance is modeled as a hidden variable.
All these works have explored various ways to temporally differentiate action instances from the near-action background by exploiting only visual modality, whereas we additionally utilize audio modality for the same objective.
Audio-visual event localization: The task of audio-visual event localization, as deﬁned in the literature, is to classify each time-step into one of the event classes or background.
This is different from action localization, where the goal is to determine the start and the end of each instance of the given action class.
In (Tian et al., 2018), a network with audio-guided attention was proposed, which showed prototypical results for audio-visual event localization, and cross-modality synchronized event localization.
To utilize both global and local cues in event localization, Lin et al.
(2019) conducted audio-visual fusion in both of video-level and snippet-level using multiple LSTMs.
Assuming single event videos, Wu et al.
(2019) detected the event-related snippet by matching the video-level feature of one modality with the snippet-level feature sequence of the other modality.
Contrastingly, our cross-attention is over the temporal sequences from both the modalities and does not assume single-action videos.
In order to address the temporal inconsistency between audio and visual modalities, Xuan et al.
(2020) devised the modality sentinel, which ﬁlters out the eventunrelated modalities.
Encouraging results have been reported, however, the localization capability of these methods has been shown only for the short ﬁxed-length videos with distinct audio cues.
Differently, we aim to fuse audio and visual modalities in order to also localize actions in long, untrimmed and unconstrained videos.
Deep multi-modal representation learning: Multi-modal representation learning methods aim to obtain powerful representation ability from multiple modalities (Guo et al., 2019).
With the advancement of deep-learning, many deep multi-modal representation learning approaches have been developed.
Several methods fused features from different modalities in a joint subspace by outerproduct (Zadeh et al., 2017), bilinear pooling (Fukui et al., 2016), and statistical regularization (Aytar et al., 2017).
The encoder-decoder framework has also been exploited for multi-modal learning for image-to-image translation (Huang et al., 2018) and to produce musical translations (Mor et al.,
Another category of approaches aim to disjointly learn the features of each modality under cross-modality constraints such as cross-modal ranking (Frome et al., 2013; Lazaridou et al., 2015;
Kiros et al., 2014) or feature distance (Pan et al., 2016; Xu et al., 2015; Liong et al., 2016).
Our approach belongs to this category and uses cross-correlation as cross-modality constraint.
Crosscorrelation has been exploited to generate visual features attended by text for visual question answering (Kim et al., 2017; Yu et al., 2017).
It has also been used to obtain cross-attention for few-shot learning (Hou et al., 2019) and image-text matching (Lee et al., 2018; Wei et al., 2020).
In our work we adopt the cross-correlation to generate both of audio and visual features attended by each other.
The most similar to our cross-attention mechanism is the cross-attention module of
Hou et al.
(2019), which computes cross-correlation spatially between features maps of two images (sample and query).
Whereas, our cross-attention is designed for video and is computed between two temporal sequences of different modalities.
M ETHODOLOGY
In this section, we introduce the proposed framework for weakly-supervised action and event localization.
Fig.
1 illustrates the complete framework.
We ﬁrst present the multi-stage cross-attention mechanism to generate the audio-visual features in Sec. 3.1.
Then, we explain open-max classiﬁcation to robustly distinguish the actions 3 from unknown background in 3.2.
Finally, in Sec. 3.3, we describe the training loss including two consistency losses designed to enforce temporal continuity of the actions and background.
Problem statement: We suppose that a set of videos only with the corresponding video-level labels are given for training.
For each video, we uniformly sample L non-overlapping snippets, and then extract the audio features U = ( u l ) Ll =1 ∈ R d u × L with a pre-trained network, where u l is the d u dimensional audio feature representation of the snippet l .
Similarly, the snippet-wise visual features
V =
( v l ) Ll =1 ∈ R d v × L are also extracted.
The video-level label is represented as c ∈ { 0 , 1 , . .
, C } , where C is the number of action classes and 0 denotes the background class.
Starting from the audio and visual features, our approach learns to categorize each snippet into C
+ 1 classes and hence localizes actions in weakly-supervised manner.
3. M ULTI - STAGE CROSS - ATTENTION MECHANISM
While multiple modalities can provide more information than a single one, the modality-speciﬁc information may be reduced while fusing different modalities.
To reliably fuse the two modalities, we develop the multi-stage cross-attention mechanism where features are separately learned for each modality under constraints from the other modality.
In this way, the learned features for each modality encodes the inter-modal information, while preserving the exclusive and meaningful intramodal characteristics.
As illustrated in Fig.
1, we ﬁrst encode the input features U and V to X u
= ( x lu ) Ll
=1 and X v = ( x lv ) Ll
=1 via the modality-speciﬁc fully-connected (FC) layers f u and f v , where x lu and x lv are in R d x .
After that, we compute the cross-correlation of X u and X v to measure inter-modal relevance.
To reduce the gap of the heterogeneity between the two modalities, we use a learnable weight matrix
W ∈ R d x × d x and compute the cross-correlation as
[Equation]
where Λ ∈ R L × L .
Note that x lu and x lv are l 2 -normalized before computing the cross-correlation.
In the cross-correlation matrix, a high correlation coefﬁcient means that the corresponding audio and visual snippet features are highly relevant.
Accordingly, the l th column of Λ corresponds to the relevance of x lv to L audio snippet features.
Based on this, we generate the cross-attention weights A u and A v by the column-wise soft-max of Λ and Λ T , respectively.
Then, for each modality, the attention weights are used to re-weight the snippet features to make them more discriminative given the other modality.
Formally, the attention-weighted features ˜ X u and ˜
X v are obtained by
[Equation]
Note that each modality guides the other one through the attention weights.
This is to ensure the meaningful intra-modal information is well-preserved while applying the cross-attention.
To extensively delve into cross-modal information, we repeatedly apply the cross-attention multiple times.
However, during the multi-stage cross-attention, the original modality-speciﬁc characteristics may be over-suppressed.
To prevent this, we adopt the dense skip connection (Huang et al., 2017).
More speciﬁcally, at stage t , we obtain the attended audio features by
[Equation]
where X (0)att ,u is X u , and tanh( · ) denotes the hyperbolic tangent activation function.
Similar to
X ( t ) att ,u , the attended visual features X ( t ) att ,v are generated for the visual modality.
At the last stage t e , we concatenate the attended audio and visual features to yield audio-visual features,
[Equation]
where t e is empirically set to 2 which will be discussed in the ablation studies in Section 4.3.
Discussion Applying the cross-attention (Eq. 2) brings the audio and visual embeddings closer, while the skip connections (Eq. 3) enforce modality speciﬁc information, more so with dense skip connections.
Using both the cross-attention and the dense skip connections alternatively over multiple stages, we progressively learn optimal embeddings for fusion.
Learning in this way, we aim to achieve right amount of compatibility between the two embeddings while preserving the modality speciﬁc information, in order to optimize for the training objective.
3. O PEN - MAX CLASSIFICATION
Video segments can be dichotomized into foreground actions and background.
For precise action localization, distinguishing the background from the actions is important as well as categorizing the action classes.
However, unlike action classes, the background class comprises of extremely diverse types of non-actions.
Therefore, it is not possible to train for the wide range of background classes that the model may confront at the test time.
To resolve this problem, we address the background as an open set (Dietterich, 2017; Bendale &
Boult, 2016).
As illustrated in Fig.
1, we construct an open-max classiﬁer on top of the multistage cross-attentional feature fusion.
Speciﬁcally, the open-max classiﬁer consists of two parallel
FC layers for action classiﬁcation and foreground reliability estimation.
The attended audio-visual feature x l att , where l = 1 , . . .
, L , is fed snippet-wise into the open-max classiﬁer.
The ﬁrst FC layer outputs a snippet-wise activation vector h l =
[ h l (1) , . . .
, h l ( C )] for C action classes, which is converted to probability scores, p l ac by soft-max function.
Simultaneously, the second FC layer is applied on x l att , followed by a sigmoid function to estimate its foreground reliability , µ l ∈ [0 , 1] .
The foreground reliability, µ l , is the probability of snippet l belonging to any action class.
The low reliability indicates that no action occurs in the snippet.
Therefore, we compute the probability for the background class as the complement of µ l , by p l bg = 1 − µ l .
Lastly, the open-max classiﬁer outputs the probability distribution p l over C + 1 classes including the background and C actions as
[Equation]
3. T RAINING L OSS
Next, we describe the loss functions to train our model.
The actions or foreground do not abruptly change over time.
To impose this constraint, we devise two types of temporal continuity losses.
Foreground continuity loss: Foreground continuity implies two important properties for neighboring snippets: (a) similar foreground reliability in a class-agnostic way, and (b) consistent open-max probabilities for a target foreground class.
The ﬁrst of the two constraints is imposed via class-agnostic foreground continuity:
[Equation]
where G ( i ) is a Gaussian window of width B + 1 to apply temporal smoothing over foreground reliability around l th snippet.
For the second constraint, temporal Gaussian smoothing is applied over open-max probability of video-level ground-truth action class, ˆ c , to obtain class-speciﬁc foreground continuity:
[Equation]
Finally, the foreground continuity loss is deﬁned as:
[Equation]
The foreground continuity loss imposes temporal continuity of foreground, and hence also helps in separating background from the action classes.
[Table]
Pseudo localization loss: Here, we consider the action or background class continuity, which implies that the open-max probabilities, p l , agrees with the classiﬁcation of neighbouring snippets.
This can be used to obtain the pseudo label for snippet l .
We ﬁrst average the open-max prediction of N neighbor snippets and itself, q l = 1 N + [Equation] l + N/ 2 i =
l − N/ 2 p i .
We set ˆ c l = arg max c ( q l ( c )) as the pseudo label, but only retain it when the largest class probability of q l is higher than a predeﬁned threshold τ .
Accordingly, the pseudo localization loss is formulated by
[Equation]
Total loss: Additionally, we employ the multiple instance learning (MIL) and co-activity similarity (CAS) losses (Paul et al., 2018).
The ﬁnal loss L is deﬁned by
[Equation]
where L mil and L cas denote MIL and CAS losses, respectively.
For details see Appendix D.
Figs.
2 (b) and (c) compare the class activation sequences along the temporal axis for the target classes between the models trained without and with the two consistency losses, respectively.
We see that class activations are more continuous in the model with the consistency losses.
E XPERIMENTS
In this section, we provide experimental analysis and comparative evaluation to show the effectiveness of the proposed method.
More experiments and qualitative results are in the Appendix.
4. D ATASETS AND EVALUATION METHOD
Datasets: We evaluate our approach on Audio-Visual Event (AVE) and ActivityNet1.2 datasets.
AVE dataset is constructed for audio-visual event localization, which contains 3,339 training and 804 testing videos, each lasting 10 seconds with event annotation per second.
There are 28 audiovisual event categories covering a wide range of domains, such as animal and human actions, vehicle sounds, and music performance.
Each event category has both audio and visual aspects, e.g. church bell, baby crying, man speaking etc.
ActivityNet1.2 is a temporal action localization dataset with 4,819 train and 2,383 validation videos, which in the literature is used for evaluation.
It has 100 action classes of wider variety than AVE dataset, with on an average 1.5 instances per video.
The average length of the videos in this dataset is 115 seconds, often with weak audio cues, which makes action localization as well as leveraging audio harder.
Evaluation metric: We follow the standard evaluation protocol of each dataset.
For the AVE dataset, we report snippet-wise event prediction accuracy.
For the ActivityNet1.2 dataset, we generate the action segments (start and end time) from snippet-wise prediction (details are described in the following section), and then measure mean average precision (mAP) at different intersection over union (IoU) thresholds.
[Table]
4. F EATURE EXTRACTION AND IMPLEMENTATION D ETAILS
Feature extraction: We use the I3D network (Carreira & Zisserman, 2017) and the ResNet architecture (He et al., 2016) to extract the visual features for ActivityNet1.2 and AVE, respectively.
The I3D network is pre-trained on Kinetics-400 (Kay et al., 2017), and the features consist of two components: RGB and optical ﬂow.
The ResNet 152 is pre-trained on the ImageNet (Russakovsky et al., 2015), and the features are extracted from the last global pooling layer.
To extract the audio features, we use the VGG-like network (Hershey et al., 2017), which is pre-trained on the AudioSet (Gemmeke et al., 2017), for both AVE and ActivityNet1.2 datasets.
Implementation Details: We set d x to 1,024, and the LeakyRelu and hyperbolic tangent functions are respectively used for the activation of modality-speciﬁc layers and cross-attention modules.
In training, the parameters are initialized with Xavier method (Glorot & Bengio, 2010) and updated by
Adam optimizer (Kingma & Ba, 2015) with the learning rate of 10 − 4 and the batch size of 30 .
Also, the dropout with a ratio of 0.7 is applied for the ﬁnal attended audio-visual features.
In the loss, the hyper parameters are set as B = 4 , α = 0 .
8 , β = 0 .
8 and γ = 1 .
Localization at test time: For event localization at test time, i.e. snippet classiﬁcation, each snippet l is classiﬁed into one of event classes (including background) by arg max c p l ( c ) , where p l is the open-max probability of snippet l .
For action localization, we follow the two-stage thresholding scheme of (Paul et al., 2018).
The ﬁrst threshold is applied to ﬁlter out the classes that have videolevel scores less than the average over all the classes.
The second threshold is applied along the temporal axis to obtain the start and the end of each action instance.
4. A BLATION ANALYSIS
Multi-stage cross
attention: To evaluate the effectiveness of the multi-stage cross-attention in audio-visual fusion, we compare two uni-modal methods (audio or visual) and four multi-modal methods with different stages (0-3 stages) on the AVE and ActivityNet1.2 datasets in Table 1.
The pseudo-label losses and the open-max classiﬁers are used in all six cases.
In the uni-modal methods, the input feature is embedded using an FC layer, and then fed into the open-max classiﬁer.
The 0-stage method denotes a naive fusion, where audio and visual features are fused by simple con-
[Table] [Table]
catenation.
Even this naive fusion yields higher performance than the uni-modal methods on the
AVE dataset.
However, that is not the case with more challenging task of the action localization on
ActivityNet1.2 dataset.
Furthermore, all the later stages improve considerably over 0-stage and the uni-modal cases, for the both datasets.
The 2-stage cross-attention achieves the best performance for the both datasets (more in Appendix A).
Interestingly, even with the minimal audio cue in ActivityNet1.2 (avg. mAP of audio only is 7.8%), the proposed audio-visual features improve the avg mAP over visual-only and naive fusion (0-stage) models by 4%.
Fig.
3 shows the qualitative results of the proposed and visual-only models given an example of the
ActivityNet1.2 dataset.
At the beginning of the video, a performer is shown without any activity.
The visual-only model incorrectly predicts the beginning part as a target action while our proposed model correctly predicts it as background.
Also, the visual-only model cannot catch the action at the last part of the video since it is visually similar across the frames and has minimal visual activity.
Whereas, our model correctly recognizes the last part as an action, owing to the multi-stage crossattention of effective fusion of the two modalities.
More qualitative results are in Appendix E.
Consistency losses: We show the ablation over the two proposed losses, L cont and L pseu , while using Open-Max classiﬁer and 2-stage cross-attention, in the lower part of the Table 2.
We denote the method with only L cont loss by O-I and with only L pseu loss by O-II.
The proposed method (OIII) with both of the losses performs the best suggesting the importance of both of the losses.
Further,
O-II outperforms O-I by a big margin on both the datasets, implying that the pseudo localization loss is more critical for the action localization (more in Appendix B.1).
This result demonstrates that guiding temporal continuity is essential in the long untrimmed videos as well as the short ones.
Open-max classiﬁer: We compare the open-max classiﬁer with the soft-max classiﬁer where the last FC layer outputs activations for C + 1 classes are normalized by the soft-max function.
As the background is considered a closed set in the soft-max approach, the foreground continuity loss is not available.
The soft-max is denoted by S-I in Table 2.
Both O-II and O-III versions of the open-max outperform the S-I method with the soft-max.
The O-III method improves the accuracy by 8.6% on the AVE dataset and the avg.
mAP by 2.2% on the ActivityNet1.2 dataset.
For further analysis see
Appendix B.2.
This shows the advantage of modelling background with the open-max classiﬁer.
Dense skip connections: We evaluate the impact of dense skip connections in Table 3 for 2-stage model on the ActivityNet1.2.
Compared to no skip connection, performance is improved with the skip connections, and further boosted with the dense skip connection to avg.
mAP of 26.0%.
This shows preserving the modality speciﬁc information leads to better fusion and action localization.
4. M ODEL EFFICIENCY
Though we successfully leverage the audio modality to improve action localization performance, the added modality leads to increased computational cost.
The trade-off between efﬁciency and performance due to the fusion with audio modality is demonstrated in Table 4.
When using feature dimension, d x = 1024, the fusion increases the computation over visual-only method by about 52% and 74% after 1-stage and 2-stage, respectively.
When we reduce d x to 512 , the visual-only model gets affected while the 2-stage model maintains its performance at 25.9%.
Thanks to the effectiveness of the proposed fusion, even with smaller d x its avg.
mAP is well above that of video-only model with d x = 1024 , while using about 26% less computation (1.7 MFLOPS vs 2.3 MFLOPS).
[Table] [Table]
4. C OMPARISON WITH THE STATE - OF - THE - ART
Audio-visual event localization: In Table [Table] we compare the proposed method with the recent fully and weakly-supervised methods on the AVE dataset for audio-visual event localization task.
In the weakly-supervised setting, our method performs better than all of the existing methods at least by 1.4%.
Note that, even though learned in weak-supervision, our approach achieves a comparable accuracy (77.1%) to the fully-supervised accuracy of the state-of-the-art method (Xuan et al., 2020)
[Table]
Temporal action localization: In Table 6, we apply the proposed method to weakly-supervised action localization in long duration videos of the ActivityNet1.2 dataset.
We report results for our method as well as its efﬁcient version from Section 4.4.
The mAP scores at varying IoU thresholds are compared with the current state-of-the-art methods.
Both our method and its efﬁcient version achieve the highest mAPs for 8 out of 10 IoU thresholds, and outperform all of the previous methods with the avg.
mAP of 26.0%.
We also signiﬁcantly outperform the audio-visual based method of Tian et al.
(2018) by the avg.
mAP of 17.2%.
Additionally, we compare with two naive fusions without the cross-attention (0-stage, SoftMax) with and without the continuity losses (denoted as
CL in the Table), both are bettered comfortably by our method.
This demonstrates that the effective fusion of audio and visual modalities is critical for action localization.
Furthermore, our approach is even comparable to the fully-supervised method in (Zhao et al., 2017).
C ONCLUSION
We presented a novel approach for weakly-supervised temporal action localization in videos.
In contrast to other methods, we leveraged both audio and visual modalities for this task.
This is the ﬁrst attempt at audio-visual localization of unconstrained actions in long videos.
To collaboratively fuse audio and visual features, we developed the multi-stage cross-attention mechanism that also preserves the characteristics speciﬁc to each modality.
We proposed to use the open-max classiﬁer to model the action foreground and background, in absence of temporal annotations.
Our model learns to classify video snippets via two consistency losses that enforce continuity for foreground reliability and open-max probabilities for action classes and the background.
We conducted extensive experiments to analyze each of the proposed components and demonstrate their importance.
Our method outperforms the state-of-the-art results on both AVE and ActivityNet1.2 datasets.
A A NALYSIS ON MULTI - STAGE CROSS - ATTENTION
In this section, we conduct extensive analysis for the impact of the multiple stages and dense skip connection of the proposed cross-attention mechanism.
Tables 7 and 8 show the experimental results.
Training multiple stages of cross-attention: As shown in the Table 1, the 3-stage model suffers from performance drop.
To analyze this, in Table 7, we compare 2- and 3-stage models on each of ‘w/o skip connection’, ‘w/ skip connection’, and ‘w/ dense skip connection’.
Without the skip connection, 3-stage model improves over 2-stage model, which is intuitively expected.
With the skip connection, avg. mAP of 3-stage model drops compared to 2-stage model, from 24.9% to 23.2%.
But, when the third stage is appended to the trained (and now frozen) stages of 2-stage model, the avg.
mAP is maintained at 24.9%.
Similarly, with the dense skip connection, training the entire 3-stage model end-to-end leads to degraded performance.
But, when training the model frozen till the second stage the drop is much less.
The fact that, in 3-stage model, better performance is obtained when training with ﬁrst two stages frozen compared to training end-to-end, shows that the optimization gets hard in the latter.
Therefore, we conclude that though the third stage helps without the skip connections, due to harder optimization with more stages and (dense) skip connections, 2-stage model is the optimal choice.
[Table] [Table]
Need for multi-stage cross-attention:
In Table
[Table]
we experiment with 1-stage model, varying the size of dimensions ( d x,u and d x,v ) of the cross-correlation matrix W on the AVE dataset.
We tried several hyper-parameter settings in 1-stage model
[Table]
but none of them could outperform the default setting ( d x,u = 1024 , d x,v = 1024 ) of 2-stage model even with more parameters.
Instead of increasing the parameters in 1-stage model itself, when an additional stage is added (i.e. a weight matrix learned with a non-linear activation function) better performance is achieved.
Indeed, it is often not trivial to replace a sequence of non-linear functions with another non-linear function as we experimentally observe here.
The intention behind the multi-stage is also to extensively delve into cross-modal information, progressively learning the embeddings for each modality.
B A NALYSIS FOR CONSISTENCY LOSSES AND OPEN - MAX CLASSIFICATION B. A NALYSIS OF CONSISTENCY LOSSES ON DIFFERENT STAGE MODELS
In Table 9, we conduct the analysis for the consistency losses for 0, 1 and 3-stage models as well as the chosen 2-stage model.
Effect of losses on different stage models: The impact of continuity losses is analogous on 1-, 2and 3-stage models.
Each of the two continuity losses help, but the pseudo localization loss ( L pseu ) is more effective.
Also, there is further beneﬁt of using them together for almost all the IoU thresholds and stages.
In 0-stage model, i.e. without the cross-attention, O-II shows the highest snippet-level performance on the AVE dataset, but the lowest temporal action localization performance on the
ActivityNet1.2 dataset.
From this, we understand that L pseu has difﬁculty in achieving continuity when audio and visual features are overly heterogeneous.
Consequently, clear beneﬁt is observed when the cross-attention is used.
Interdependence of cross-attention and pseudo localization loss: When comparing the O-I of all 0-3 stage models, we see that the performance improvement by stacking the cross-attention is marginal, and the pseudo localization is critical to the performance.
This follows from Eq.
9, where
L pseu is only activated at snippet l when classiﬁcation over its neighboring snippets does not strongly agree on the action class or background.
To analyze this, we check how frequently L pseu is activated when cross-attention is not used and when it is used.
For 0-stage model, without continuity losses,
L pseu is activated on 11.1% snippets of the ActivityNet1.2 training set.
The same frequency is 38.2% for 2-stage model, again without the continuity losses.
This shows that when the cross-attention is used, more often the open-max classiﬁcation of a snippet fails to strongly agree with its neighbors.
Therefore, the pseudo localization loss is much needed to enforce the continuity.
[Table]
A NALYSIS OF LOSSES AND OPEN - MAX CLASSIFICATION ON 2- STAGE MODEL
In Table
[Table]
we conduct more extensive analysis for the consistency losses and the open-max classiﬁer.
Speciﬁcally
[Table]
we replace the open-max classiﬁcation approach with soft-max one.
Then, for both classiﬁers with the 2-stage cross-attention
[Table]
we ablate the foreground continuity or pseudo local- ization losses where CAS and MIL losses are commonly used.
First, the performance gap between
S-0 and O-0, where only CAS and MIL losses are used, shows the difﬁculty of learning two parallel branches in weakly-supervised manner.
However, when adding the pseudo localization loss, (S-I and O-II), the open-max classiﬁcation approach is further improved than the soft-max.
Hence, the pseudo labels reduce the fallacious action classiﬁcation of snippets and are more effective on the open-set background modeling than the closed-set modeling.
Next, O-I and O-II shows higher performance than O-0.
Similarly, S-I is superior to S-0.
This indicates that erroneous classiﬁcations are suppressed by the correctly classiﬁed neighbors when using the consistency losses.
Also, comparing O-I and O-II, the pseudo localization loss gives more performance improvement.
This is because the pseudo localization loss addresses the consistency of classiﬁcation scores of all the classes including background, while the foreground continuity loss smoothens foreground reliability being class-agnostic or only for the target class.
For all of the IoU thresholds (except 0.7), O-III, open-max classiﬁcation with both of the consistency losses, yields the highest performance.
Therefore, all of the proposed open-max classiﬁcation and consistency losses are effective to temporal action or event localization in videos.
[Table]
C D ETAILS OF THE PROPOSED ARCHITECTURE
[Table]
f u + LeakyRelu d u × d x
U
X u ∈ R d x × L f v + LeakyRelu
d v × d x
V
X v ∈ R d x × L
W (1) d x × d x
X u , X v ˜ X
(1) u ∈ R d x × L , ˜ X (1) v ∈ R d x × L
Dense Connection + TanhX u , X v , ˜ X (1) u , ˜ X (1) v
X (1) att ,u ∈ R d x × L , X (1) att ,v ∈ R d x × L
W (2) d x × d x
X (1) att ,u , X (1) att ,v ˜ X (2) u ∈ R d x × L , ˜ X (2) v ∈ R d x × L
Dense Connection + TanhX u , X v , X (1) att ,u , X (1) att ,v , ˜ X (2) u , ˜ X (2) v
X (2) att ,u ∈ R d x × L , X (2) att ,v ∈ R d x × L
ConcatenationX (2) att ,u , X (2) att ,v
X att ∈ R 2 d x × L f ac + Soft-Max 2 d x × C
X att
[Table]
D M ULTIPLE INSTANCE LOSS AND C O - ACTIVITY SIMILARITY LOSS
We apply multiple-instance learning loss for classiﬁcation.
The prediction score corresponding to a class is computed as the average of its top k activations over the temporal dimension.
Co-activity similarity loss (CASL) (Paul et al., 2018) is computed over two snippet sequences from a pair of videos, to have higher similarity when the videos have a common class.
E Q UALITATIVE EVALUATION
We provide additional qualitative results for action localization on the ActivityNet1.2 dataset.
Fig.compares the proposed method with the method trained on visual modality (‘Visual-only’).
The
[Figure]
open-max classiﬁer and total loss function are commonly used for both.
In Figs. 4(a) and (b), because the videos are static in visual modality, the background segments in early parts of videos are miss-localized as actions in the visual-only model.
Contrarily, proposed method distinguishes the background based on the action-related audio (cheerleading music and violin sound).
In Fig. 4(c), the brushing sound is overlapped with the loud human narration lasting throughout videos.
Nevertheless, the proposed method effectively extracts the crucial audio cues and fuses them with the visual ones.
In Fig. 4(d), even though the early part of the action is visually occluded by large logos, our method exactly localizes the action.
Also, for all of the class activation sequences, the activations by the proposed method are more consistently high for actions.
This means that our collaboration of audio and visual modalities is more robust in distinguishing foreground from background.
Fig.
5 illustrates the cases where audio degrades the performance.
Fig.
5 (a) shows an example video for action class ‘playing violin’.
The violin sound of the soloist and the band is intermingled in the video.
In the end, the sound of violin continues making our model predict the action but since camera focuses on the band, the ground-truth does not include those frames.
Fig.
5 (b) shows an example of action ‘using parallel bars’.
Here the repeated background music is irrelevant to action,
[Figure]
therefore the class activation is bit off in the last part.
However, thanks to the visual modality, the prediction is still reasonable.