M ARGINALIZED A VERAGE A TTENTIONAL N ETWORK FOR W EAKLY -S UPERVISED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In weakly-supervised temporal action localization, previous works suffer from overestimating the most salient regions and fail to locate dense and integral regions for each entire action.
To alleviate this issue, we propose a marginalized average attentional network (MAAN) to suppress the dominant response of the most salient regions in a principled manner.
The MAAN employs a novel marginalized average aggregation (MAA) module and learns a set of latent discriminative probabilities in an end-to-end fashion.
MAA samples the subsets from the video snippet features based on the latent discriminative probabilities and takes the expectation over all the subset features.
Theoretically, we prove that the learned latent discriminative probabilities reduce the difference of responses between the most salient regions and the others, and thus MAAN generates better class activation sequences to identify more dense and integral action regions in the videos.
Moreover, we propose a fast algorithm to reduce the complexity of constructing MAA from
O (2 T ) to O
( T 2 ) .
Extensive experiments on two large-scale video datasets show that our MAAN achieves superior performance on weakly-supervised temporal action localization task.
I NTRODUCTION
The recent weakly-supervised setting for temporal action localization interested to the community is to train the model with solely the video-level class labels which are more easily collected, and to predict both the action class and the temporal boundary of each action instance at the test time.
The main difﬁculty in solving the weakly-supervised localization task is how to ﬁnd the right way to express and infer the underlying location information with only the video-level class labels.
Traditionally, this is achieved by explicitly densely sampling several possible instances at various locations with various durations (Bilen & Vedaldi, 2016; Kantorov et al., 2016; Zhang et al., 2017), and then training instance-level classiﬁers with multiple instances learning (Cinbis et al., 2017; Yuan et al., 2017a) or curriculum learning (Bengio et al., 2009).
However, the length of action and video varies too much such that the number of instance proposals varies a lot and can be huge.
Recent research, however, has pivoted to acquire the location information by generating the class activation sequence (CAS) directly (Nguyen et al., 2018), which produces the classiﬁcation score sequence of being each action for each snippet over time.
The CAS along the 1D temporal dimension for a video is inspired by the class activation map (CAM)
(Zhou et al., 2016a; 2014; Pinheiro &
Collobert, 2015; Oquab et al., 2015) in weakly-supervised object detection.
The CAM-based models have shown that despite being trained on image-level labels, convolutional neural networks (CNNs) have the remarkable ability to localize objects.
Similar to the object detection, the basic idea behind the CAS-based methods for action localization in the training is to sample the non-overlapped snippets from a video, then to aggregate the snippet-level features into a video-level feature, and ﬁnally to yield a video-level class prediction.
During testing, the model generates a CAS for each class that identiﬁes the discriminative action regions, and a threshold is then applied on the CAS to localize each action instance in terms of the start time and the end time.
In CAS-based methods, the feature aggregator that aggregates several snippet-level features into a video-level feature is the critical building block of weakly-supervised neural networks.
A model’s ability to capture the location information of action is primarily determined by the design of aggregators.
While using the global average pooling over a full image or across the video snippets has shown great promise of identifying the discriminative regions (Zhou et al., 2016a; 2014; Pinheiro & Collobert, 2015; Oquab et al., 2015), treating each pixel or snippet equally loses the opportunity to beneﬁt from several more essential parts.
Some recent works (Nguyen et al., 2018; Zhu et al., 2017) have tried to learn attentional weights for different snippets to compute a weighted sum as the aggregated feature.
However, it suffers the problem of easily dominated by only a few most salient snippets.
In general, models trained with only video-level class labels are easily responsive to small and sparse discriminative regions from the snippets of interest, which deviates from the requirement of the localization task that to locate dense and integral regions for each entire action.
To mitigate this gap and reduce the effect of the domination by the most salient regions, several works propose additional heuristic tricks to the previous models.
For example, (Wei et al., 2017; Zhang et al., 2018b) attempt to heuristically erase the current mined most salient regions predicted by the model, and force the network to attend other salient regions in the remaining regions by forwarding the model several times.
However, the heuristic multiple-run model is not end-to-end trainable.
Moreover, it is the ensemble of multiple-run mined regions but not the single model’s own ability that learns the entire action regions.
“Hide-and-seek”(Singh & Lee, 2017) randomly masks out some regions of the input during training, enforcing the model to localize other salient regions when the most salient regions happened to be masked out.
However, due to the uniform prior, all the input regions are masked out with the same probability.
It is very likely that most of the time it is the background that being masked out.
To this end, we propose the marginalized average attentional network (MAAN) to alleviate the effect of the domination by the most salient region in an end-to-end fashion for weakly-supervised action localization.
Speciﬁcally, MAAN suppresses the action prediction response of the most salient regions by employing marginalized average aggregation (MAA) and learning the latent discriminative probability in a principled manner.
Unlike the previous attentional pooling aggregator, which calculates the weighted sum with attention weights, MAA ﬁrst samples a subset of features according to their latent discriminative probability, and then calculates the average of these sampled features.
Finally, MAA takes the expectation (marginalization) of the average aggregated subset features over all the possible subsets to achieve the ﬁnal aggregation.
It not only alleviates the dominant issues of the most salient regions, but also keeps the scale of the aggregated feature in a stable range.
We theoretically prove that, with the MAA, the learned latent discriminative probability indeed reduces the difference of response between the most salient regions and the others.
Therefore,
MAAN can identify more dense and integral regions for each action.
Moreover, since enumerating all the possible subsets is exponential expensive, we further propose a fast iterative algorithm to reduce the complexity of the expectation calculation and provide the proof of it theoretically.
Furthermore, since all the components of the network are differentiable, MAAN is easy to be trained in an end-toend fashion.
Extensive experiments on two large-scale video datasets show that MAAN consistently outperforms the baseline models and achieves superior performance on weakly-supervised temporal action localization task.
In summary, our main contributions include: (1) a new end-to-end learnable marginalized average attentional network (MAAN) with a marginalized average aggregation (MAA) module in the weaklysupervised setting; (2) theoretical analysis of the properties of MAA and the reasons why MAAN alleviates the dominant issues of the most salient regions in weakly-supervised action localization; (3) a fast iterative algorithm to effectively reduce the computational complexity of the MAA; and (4) the superior performance on two benchmark video datasets, THUMOS14 and ActivityNet1.3, on the weakly-supervised temporal action localization task.
M ARGINALIZED A VERAGE A TTENTIONAL N ETWORK
In this section, we describe our marginalized average attentional network (MAAN) for weaklysupervised temporal action localization.
We ﬁrst derive the formulation of the feature aggregation module in MAAN as a marginalized average aggregation (MAA) procedure in Sec. 2.1.
Then, we analyze the properties of MAA in Sec. 2.2, and present our recurrent fast computation algorithm for MAA construction in Sec. 2.3.
Finally, we describe our network architecture that incorporates this feature aggregation module, and introduce the inference with it on weakly-supervised temporal action localization task in Sec. 2.4.
[Figure]
2. M ARGINALIZED A VERAGE A GGREGATION
Let us denote the snippet-level features to be aggregated as a set of representations { x 1 , x 2 , · · · x T } , where x t ∈ R m is the m dimensional feature representation extracted from a video snippet centered at time t and T is the total number of sampled video snippets.
The conventional attentional weighted sum pooling will aggregate the input snippet-level features into a video-level representation x .
Denote the set of attentional weights corresponding to the snippet-level features as { λ 1 , λ 2 , · · · λ T } , where λ t is a scalar attentional weight for x t .
Then the aggregated video-level representation is calculated as
[Equation]
Different from the conventional aggregation mechanism, the proposed MAA module aggregates the features by ﬁrstly generating a set of binary indicators to determinate whether a snippet to be sampled or not.
We then calculate the average aggregation of these sampled snippet-level representations.
Lastly, we calculate the expectation (marginalization) of the aggregated average feature for all the possible subsets, and obtain the proposed marginalized average aggregated feature.
Formally, in our proposed marginalized average aggregation module, we ﬁrst denote a set of probabilities { p 1 , p 2 , · · · p T } , where each p t ∈ [0 , 1] is a scalar corresponding to x t , similar to the λ t .
We then sample a set of random variables { z 1 , z 2 , · · · z T } , where z t ∼ Bernoulli ( p t ) , i.e., z t ∈ { 0 , 1 } with probability P ( z t = 1) =
p t .
The sampled set is used to represent the subset selection of snippet-level features, in which z t = 1 indicates x t is selected, otherwise not.
Therefore, the average aggregation of the sampled subset snipped-level representations is given by s = (cid:80)
Ti =
z i x i / (cid:80) Ti =1 z i , and our proposed aggregated feature, deﬁned as the expectation of all the possible subset-level average aggregated representations, is given by
[Equation]
2. P ARTIAL ORDER P RESERVATION AND D OMINANT R ESPONSE S UPPRESSION
Direct learning and prediction with the attention weights λ in Eq.(1)
lean to over response to the most salient region in weakly-supervised action localization.
The MAA in Eq.(2) has two properties that are useful to alleviate the domination effect of the most salient regions.
First, partial order preservation property, i.e., the latent discriminative probabilities preserve the partial order with respect to their attention weights.
Second, dominant response suppression property, i.e., the differences of the latent discriminative probabilities between the most salient items and others are smaller than the differences between their attention weights.
The partial order preservation property guarantees that it does not mix up the action and non-action snippets by assigning a high latent discriminative probability to a snippet with low response.
Dominant response suppression property enables to reduce the dominant effect of the most salient regions and encourages to identify dense and more integral action regions.
Formally, we present the two properties in Proposition 1 and Proposition 2, respectively.
Detail proofs can be found in the Appendix.
Proposition 1.
Let z i ∼ Bernoulli ( p i ) for i ∈ { 1 , ..., T } .
Then for T ≥ 2 , Eq.(3) holds true.
And p
i ≥ p j ⇔ c i ≥ c j ⇔ λ
i ≥ λ j .
[Equation]
where c i = E (cid:104) 1 / (1 + (cid:80) Tk =1 ,k (cid:54) =
i
z k ) (cid:105) and c j = E (cid:104) 1 / (1 + (cid:80) Tk =1 ,k
(cid:54) =
j z k ) (cid:105) and λ
i =
c i p i
Proposition 1 shows that the latent discriminative probabilities { p i } preserve the partial order of the attention weights { λ i } .
It means that a large attention weight corresponds to a large discriminative probability.
Thus, it guarantees that the latent discriminative probabilities preserve the ranking of action prediction response.
Eq.
(3) can be seen as a factorization of the attention weight
λ
i into the multiplication of two components, p i and c i , for i ∈ { 1 , ..., T } .
p i is the latent discriminative probability related to the feature of snippet i itself.
The factor c i captures the contextual information of snippet i from the other snippets.
This factorization can be considered as introducing structure information into the aggregation.
The factor c i can be considered as performing a structural regularization for learning the latent discriminative probabilities p i for i ∈ { 1 , ..., T } , as well as for learning the more informative aggregation.
Proposition 2.
Let z i ∼ Bernoulli ( p i ) for i ∈ { 1 , ..., T } .
Denote c i
= E (cid:104) 1 / (1 + (cid:80) Tk =1 ,k
(cid:54) =
i
z k ) (cid:105)
and λ i
=
c i p i for i ∈ { 1 , ..., T } .
Denote I = (cid:110) i (cid:12)(cid:12)(cid:12)
c i ≥ 1 / ( (cid:80) Tt =1 p t ) (cid:111) as an index set.
Then I (cid:54) = ∅ and for ∀ i ∈ I , ∀ j ∈ { 1 , ..., T } inequality (4) holds true
[Equation]
The index set I can be viewed as the most salient feature set.
Proposition 2 shows that the difference between the normalized latent discriminative probabilities of the most salient regions and others is smaller than the difference between their attention weights.
It means that the prediction for each snippet with the latent discriminative probability can reduce the gap between the most salient feature and others compared with employing attention weights.
Thus, MAAN suppresses dominant responses of the most salient feature and encourages to identify dense and more integral action regions.
Directly learning the attention weights λ leans to over response to the most salient region in weaklysupervised temporal localization.
Namely, attention weights for only a few snippets are too large and dominate the others, while attention weights for most of the other snippets that also belong to the true action are underestimated.
Proposition 2 shows that latent discriminative probability is able to reduce the gap between the most salient feature and others compared with attention weights.
Thus, by employing the latent discriminative probabilities for prediction instead of the attention weights, our method can alleviate the dominant effect of the most salient region in weakly-supervised temporal localization.
2. R ECURRENT F AST C OMPUTATION
Given a video containing T snippet-level representations, there are 2 T possible conﬁgurations for the subset selection.
Directly summing up all the 2 T conﬁgurations in the expectation to calculate x has O (2 T ) complexity.
In order to reduce the exponential complexity, we propose an iterative method to calculate x recurrently with O ( T 2 ) complexity.
Let us denote the aggregated feature of { x 1 , x 2 , · · · x t } with length t as h t , and denote Y t =
t (cid:80) i =
1 z i x i and Z t = t (cid:80) i =
1 z i for simplicity, then we have a set of
[Equation]
and the aggregated feature of { x 1 , x 2 , · · · x T } can be obtained as x = h T .
In Eq.(5), Z t is the summation of all the z i , which indicates the number of elements selected in the subset.
Although there are 2 t distinct conﬁgurations for { z 1 , z 2 , · · · z t } , it is only t + 1 distinct values for Z t , i.e.
0 , 1 , · · · , t .
Therefore, we can divide all the 2 t distinct conﬁgurations into t + 1 groups, where the conﬁgurations sharing with the same Z t fall into the same group.
Then the expectation h t can be calculated as the summation of the t + 1 parts.
That is, h t =
E
[Equation]
where the m ti , indicating the i th part of h t for group Z t
= i , is shown in Eq.(6).
[Equation]
In order to calculate h t +1 = (cid:80) t +1 i =
m t
+1 i , given m ti , i ∈ { 0 , · · · , t } , we can calculate m t
+1 i , i ∈ { 0 , 1 , · · · , t + 1 } recurrently.
The key idea here is that m t
+
i comes from two cases: if z t +1 = 0 , then m t +1 i is the same as m t i ; if z t +1 = 1 , then m t +1 i is the weighted average of m t i − 1 and x t +1 .
The latter case is also related to the probability P ( Z t
= i − 1) .
By denoting q ti − 1 = P ( Z t = i − 1) for simplicity, we can obtain m t
+1 i as a function of several elements:
[Equation]
Similarly, the computation of q t +1 i
= P ( Z t +1 = i ) comes from two cases: the probability of selecting i − 1 items from the ﬁrst t items and selecting the ( t + 1) th item, i.e., q ti − 1 p t +1 ; and the probability of selecting i items all from the ﬁrst t items and not selecting the ( t + 1) th item, i.e., q ti (1 − p t +1 ) .
We derive the function of m t
+1 i and q t +1 i in Proposition 3.
Detail proofs can be found in the Appendix.
Proposition 3.
Let z t ∼ Bernoulli ( p t ) , Z t = t (cid:80) i
=
z i and Y t = t (cid:80) i =
z i x i for t ∈ { 1 , ..., T } .
Deﬁne m ti , i ∈ { 0 , · · · , t } as Eq.
(6) and q ti = P
( Z t = i ) , then m t
+1 i i ∈ { 0 , 1 , · · · , t + 1 } can be obtained recurrently by Eq.(8) and Eq.(9).
[Equation]
where b i = i
i +1 , q t − 1 = 0 , q tt +
= 0 , q 00 = 1 , m t 0 = 0 , and m tt +
= 0 .
Proposition 3 provides recurrent formula to calculate m ti .
With this recurrent formula, we can calculate the aggregation h T by iteratively calculating m ti from i = 1 to t and t = 1 to T .
Therefore, we can obtain the aggregated feature of { x 1 , x 2 , · · · x T } as x = h T =
(cid:80) Ti =
m
Ti .
The iterative computation procedure is summarized in Algorithm 1 in the Appendix, where the time complexity is
O ( T 2 ) .
With the fast iterative algorithm in Algorithm 1, the MAA becomes practical for end-to-end training.
A demonstration of the computation graph for q t +1 i in Eq.(9) and m t
+1 i in Eq.(8) is presented in the left and right of Figure.
1, respectively.
From the Figure.
1, we can see it clearly that, to compute m 32 (the big black node in the right), it needs m 21 , m 22 , x 3 , p 3 , and q 21 .
The MAA can be easily implemented as a subnetwork for end-to-end training and can be used to replace the operation of other feature aggregators.
2. N ETWORK A RCHITECTURE AND T EMPORAL A CTION L OCALIZATION
Network Architecture : We now describe the network architecture that employs the MAA module described above for weakly-supervised temporal action localization.
We start from a previous state-ofthe-art base architecture, sparse temporal pooling network (STPN) (Nguyen et al., 2018).
As shown
Snippet Snippet
Snippet T
I3D I3D
I3D
Aggregator … … … 𝐱𝐱𝐱 𝑻 𝐱% 1D Temporal
Cricket Bowling … … W ’
W (
W ) 𝑝 ) 𝑝 ( 𝑝 + 𝑓 ) 𝑓 ( 𝑓𝐱𝐱𝐱 𝑻 𝐱 𝒕 𝑝 / 256
F C R eL U F C S
i g m o i d
Attention Module 𝐱 " 𝐱 # 𝐱 $
[Figure]
% 𝐱& 𝜆 " 𝜆 # 𝜆 $ 𝜆 % 𝐱 " 𝐱 # 𝐱 $ 𝐱 % 𝐱& 𝑝 " 𝑝 # 𝑝 $ 𝑝 % 𝑧 " 𝑧 # 𝑧 $ 𝑧 % h " h # h $ h % h % =
E
[Figure] [Figure]
[Figure]
/" 𝑇 in Figure.
2, it ﬁrst divides the input video into several non-overlapped snippets and extracts the
I3D (Carreira & Zisserman, 2017) feature for each snippet.
Each snippet-level feature is then fed to an attention module to generate attention weight between 0 and 1.
STPN then uses a feature aggregator to calculate a weighted sum of the snippet-level features with these class-agnostic attention weights to create a video-level representation, as shown in the left of Figure.
The video-level representation is then passed through an FC layer followed by a sigmoid layer to obtain class scores.
Our MAAN uses the attention module to generate the latent discriminative probability p t and replaces the feature aggregator from the weighted sum aggregation to the proposed marginalized average aggregation, which is demonstrated in the right of Figure.
Training with video-level class labels : Formally, the architecture ﬁrst performs aggregation of the snippet-level features (i.e. x 1 , x 2 , · · · x T ) to obtain the video-level representation ¯ x ( ¯ x =
E [ (cid:80) Ti =
z
i x i / (cid:80) Ti =
z i ] ).
Then, it followed by a logistic regression layer (FC layer + sigmoid) to output video-level classiﬁcation prediction probability.
Speciﬁcally, the prediction probability for class c ∈ { 1 , 2 , · · · C } is parameterized as σ cj =
σ ( w (cid:62) c x j ) , where x j is the aggregated feature for video j ∈ { 1 , ..., N } .
Suppose each video x j is i.i.d and each action class is independent from each other, then the negative log-likelihood function (cross-entropy loss) is given as follows:
[Equation]
where y cj ∈ { 0 , 1 } is the ground-truth video-level label for class c happens in video j and W =
[ w 1 , ..., w C ] .
Temporal Action Localization : Let s c = w (cid:62) c
x be the video-level action prediction score, and σ ( s c )
=
σ ( w (cid:62) c x ) be the video-level action prediction probability.
In STPN, as ¯ x = (cid:80) Tt =1 λ t x t , the s c can be rewritten as:
[Equation]
In STPN, the prediction score of snippet t for action class c in a video is deﬁned as:
[Equation]
where σ ( · ) denotes the sigmoid function.
In MAAN, as ¯ x = E [ (cid:80) Ti =
z i x i / (cid:80) Ti =1 z
i ] , according to
Proposition 1, the s c can be rewritten as:
[Equation]
The latent discriminative probability p t corresponds to the class-agnostic attention weight for snippet t .
According to Proposition 1 and Proposition 2, c t does not relate to the snippet t , but captures the context of other snippets.
w c corresponds to the class-speciﬁc weights for action class c for all the snippets, and w (cid:62) c x t indicates the relevance of the snippet t to class c .
To generate temporal proposals, we compute the prediction score of snippet t belonging to action class c in a video as:
[Equation]
We denote the s c =
( s c 1 , s c 2 , ..., s cT ) (cid:62) as the class activation sequence (CAS) for class c .
Similar to
STPN, the threshold is applied to the CAS for each class to extract the one-dimensional connected components to generate its temporal proposals, and then we perform non-maximum suppression among temporal proposals of each class independently to remove highly overlapped detections.
Compared to STPN (Eq.
(12)), MAAN (Eq.
(14)) employs the latent discriminative probability p t instead of directly using the attention weight λ t (equivalent to c t p t ) for prediction.
Propositionsuggests that MAAN can suppress the dominant response s ct compared with STPN.
Thus, MAAN is more promising to achieve better performance in weakly-supervised temporal action localization.
E XPERIMENTS 3. E XPERIMENTAL S ETTINGS
Datasets.
We evaluate MAAN on two popular action localization benchmark datasets, THUMOS14 (Jiang et al., 2014) and ActivityNet1.3 (Heilbron et al., 2015).
THUMOS14 contains 20 action classes for the temporal action localization task.
It consists of 200 untrimmed videos (3,027 action instances) in the validation set and 212 untrimmed videos (3,358 action instances) in the test set.
Following the standard practice, we train the models on the validation set without using the temporal annotations and evaluate them on the test set.
ActivityNet1.3 is a large-scale video benchmark for action detection that covering a wide range of complex human activities.
It provides samples from 200 activity classes with an average of 137 untrimmed videos per class and 1.41 activity instances per video, for a total of 849 video hours.
This dataset contains 10,024 training videos, 4,926 validation videos and 5,044 test videos.
In the experiments, we train the models on the training videos and test on the validation videos.
Evaluation Metrics.
We follow the standard evaluation metric by reporting mean average precision (mAP) values at several different levels of intersection over union (IoU) thresholds.
We use the benchmarking code provided by ActivityNet 1 to evaluate the models.
Implementation Details.
We use two-stream I3D networks (Carreira & Zisserman, 2017) pre-trained on the Kinetics dataset (Kay et al., 2017) to extract the snippet-level feature vectors for each video.
All the videos are divided into a set of non-overlapping video snippets.
Each snippet containsconsecutive frames or optical ﬂow maps.
We input each 16 stacked RGB frames or ﬂow maps into the
I3D RGB or ﬂow models to extract the corresponding feature vectors of dimension 1024.
Due to the various lengths of the videos, in the training, we uniformly divide each video into T non-overlapped segments, and randomly sample one snippet from each segment.
Therefore, we sample T snippets for each video as the input of the model for training.
We set T to 20 in our MAAN model.
The attention module in Figure.
2 consists of an FC layer of 1024 × 256 , a LeakyReLU layer, an FC
[Table]
layer of 256 × 1 , and a sigmoid non-linear activation, to generate the latent discriminative probability p t .
We pass the aggregated video-level representation through an FC layer of 1024 × C followed by a sigmoid activation to obtain class scores.
We use the ADAM optimizer (Kingma & Ba, 2014) with an initial learning rate of 5 × 10 − 4 to optimize network parameters.
At test time, we ﬁrst reject classes whose video-level probabilities are below 0 .
We then forward all the snippets of the video to generate the CAS for the remaining classes.
We generate the temporal proposals by cutting the
CAS with a threshold th .
The combination ratio of two-stream modalities is set to 0 .
5 and 0 .
Our algorithm is implemented in PyTorch 2 .
We run all the experiments on a single NVIDIA Tesla M
GPU with 24 GB memory.
3. THUMOS14 DATASET
We ﬁrst compare our MAAN model on the THUMOS14 dataset with several baseline models that use different feature aggregators in Figure.
2 to gain some basic understanding of the behavior of our proposed MAA.
The descriptions of the four baseline models are listed below.
(1) STPN.
It employs the weighed sum aggregation ¯ x = (cid:80)
Tt
=
1 λ t x t to generate the video-level representation.
(2) Dropout.
It
explicitly performs dropout
sampling with
dropout
probability
p = 0 .in
STPN to obtain the video-level representation, ¯ x = (cid:80) Tt =1 r t λ t x t , r t ∼ Bernoulli (0 . 5) .
Normalization.
Denoted as “Norm” in the experiments, it utilizes the weighted average aggregation ¯ x = (cid:80) Tt =
1 λ t x t / (cid:80) Tt =
1 λ t for the video-level representation.
(4) SoftMax Normalization.
Denoted as “SoftMaxNorm” in the experiments, it applies the softmax function as the normalized weights to get the weighted average aggregated video-level feature, ¯ x = (cid:80) Tt
=1 e λ t x t / (cid:80)
Tt =1 e λ t .
We test all the models with the cutting threshold th as 0.2 of the max value of the CAS.
We compare the detection average precision (%) at IoU =
[0.1 : 0.1 : 0.9] and the video-level classiﬁcation mean average precision (%) (denoted as Cls mAP) on the test set in Table 1.
From Table 1, we can observe that although all the methods achieve similar video-level classiﬁcation mAP, their localization performances vary a lot.
It shows that achieving a good video-level classiﬁcation performance cannot guarantee to obtain a good snippet-level localization performance because the former only requires the correct prediction on the existence of an action, while the latter requires the correct prediction on both of its existence and its duration and location.
Moreover, Table 1 demonstrates that MAAN consistently outperforms all the baseline models at different levels of IoUs in the weakly-supervised temporal localization task.
Both of the “Norm” and “SoftmaxNorm” are the normalized weighted average aggregation.
However, the “SoftmaxNorm” performs the worst, because the softmax function over-ampliﬁes the weight of the most salient snippet.
As a result, it tends to identify very few discriminative snippets and obtain sparse and non-integral localization.
The “Norm” also performs worse than our MAAN.
It is the normalized weighted average over the snippet-level representation, while MAAN can be considered as the normalized weighted average (expectation) over the subsetlevel representation.
Therefore, MAAN encourages to identify dense and integral action segments as compared to “Norm” which encourages to identify just several discriminative snippets.
MAAN works better than the “Dropout” because the “Dropout” randomly dropouts the snippets with different attention weights by uniform probabilities.
At each iteration, the scale of the aggregated feature varies a lot, however, MAAN samples with the learnable latent discriminative probability and conducts
[Figure]
the expectation to keep the scale of the aggregated feature stable.
Compared to STPN, MAAN also achieves superior results.
MAAN implicitly factorizes the attention weight into c t p t , where p t learns the latent discriminative probability for the current snippet, and c t captures the contextual information and regularizes the network to learn more informative aggregation.
The properties of MAA enable the predicted class activation sequences not to concentrate on the most salient regions.
The quantitative results demonstrate the effectiveness of the feature aggregation of our proposed MAAN.
Figure.
4 visualize the one-dimensional CASs of the proposed MAAN and all the baseline models.
The temporal CAS generated by MAAN can cover large and dense regions to obtain more accurate action segments.
In the example in Figure.
4, MAAN can discover almost all the actions compared with the ground-truth; however, the STPN have missed several action segments, and also tends to ﬁnd the more salient regions in each action segment.
Other methods are much sparser compared with MAAN.
The ﬁrst row of the Figure.
4 shows several action segments in red and in green, corresponding to action segments that are relatively difﬁcult and easy to be localized, respectively.
We can see that all the easily-localized segments contain the whole person that performing the “HammerThrow” action, while the difﬁcultly-localized segments contain only a part of the person or the action.
Our MAAN can successfully localize the easy segments as well as the difﬁcult segments; however, all the other methods fail on the difﬁcult ones.
It shows that MAAN can identify several dense and integral action regions other than only the most discriminative region identiﬁed by the other methods in comparison.
We also compare our model with the state-of-the-art action localization approaches on THUMOS dataset.
The numerical results are summarized in Table.
We include both fully and weaklysupervised learning, as in (Nguyen et al., 2018).
As shown in Table.
2, our implemented STPN performs slightly better than the results reported in the original paper (Nguyen et al., 2018).
From
Table.
2, our proposed MAAN outperforms the STPN and most of the existing weakly-supervised action localization approaches.
Furthermore, even trained with only video-level labels, our model still presents competitive results compared with several recent fully-supervised approaches.
[Table] [Table]
3. A CTIVITY N ET 1.3 DATASET
We train the MAAN model on the ActivityNet1.3 training set and compare our performance with the recent state-of-the-art approaches on the validation set in Table.
The action segment in ActivityNet is usually much longer than that of THUMOS14 and occupies a larger percent of a video.
We use a set of thresholds, which are [0 . 2 , 0 . 15 , 0 . 1 , 0 .
] of the max value of the CAS, to generate the proposals from the one-dimensional CAS.
As shown in Table.
3, with the set of thresholds, our implemented
STPN performs slightly better than the results reported in the original paper (Nguyen et al., 2018).
With the same threshold and experimental setting, our proposed MAAN model outperforms the STPN approach on the large-scale ActivityNet1.3.
Similar to THUMOS14, our model also achieves good results that are close to some of the fully-supervised approaches.
C ONCLUSION
We have proposed the marginalized average attentional network (MAAN) for weakly-supervised temporal action localization.
MAAN employs a novel marginalized average aggregation (MAA) operation to encourage the network to identify the dense and integral action segments in an end-to-end fashion.
Theoretically, we have proved that MAA decreases the gap between the most discriminant regions in the video to the others, and thus MAAN generates better class activation sequences to infer the action locations.
We have also proposed a fast algorithm to reduce the computation complexity of MAA.
Our proposed MAAN achieves superior performance on both of the THUMOS14 and the
ActivityNet1.3 dataset on weakly-supervised temporal action localization task.
A P ROOF OF P ROPOSITIONA. P ROOF OF E QUATION (3) Proof
[Equation]
In addition,
[Equation]
Thus , we achieve that
[Equation]
A. P ROOF OF p i ≥ p j ⇔ c i ≥ c j ⇔ λ i ≥ λ j
Proof.
Denote S T = (cid:80)
Tk =1 ,k (cid:54)
=
i,k (cid:54) =
j z k , then we have that c
i
[Equation]
=
p j E [1 / (2 + S T )]
+ (1 − p j ) E
[1 / (1 + S T )]
− p i E
[1 / (2 + S T )] − (1 − p i )
E
[1 / (1 + S T )]
= ( p i − p j )
( E [1 / (1 + S T )] − E [1 / (2 + S T )]) (19)
Since E [1 / (1 + S T )] − E [1 / (2 + S T )] > 0 , we achieve that p i ≥ p j ⇔ c i ≥ c j .
Since λ i
= c i p i and λ j = c j p j , and c i , c j , p i , p j ≥ 0 , it follows that p i ≥ p j ⇔
λ
i ≥ λ j
B P ROOF OF P ROPOSITION
Proof.
[Equation]
When p 1 = p 2 = · · · = p T , we have λ 1 = λ 2 = · · · = λ T .
then inequality (4) trivially holds true.
Without loss of generality, assume p 1 ≥ p 2 ≥ · · · ≥ p T and there exists a strict inequality.
Then ∃ k ∈ { 1 , ..., T − 1 } such that c
i ≥ 1 / ( (cid:80)
Tt =1 p t ) for 1 ≤ i ≤ k and c j ≤ 1 / ( (cid:80)
Tt =1 p t ) for k < j ≤ T .
Otherwise, we obtain c
i ≥ 1 / ( (cid:80) Tt =1 p t ) or c i ≤ 1 / ( (cid:80)
Tt =1 p t ) for 1 ≤ i ≤ T and there exists a strict inequality.
It follows that (cid:80) T i =
1 c i p i > 1 or (cid:80) T i =1 c i p i < 1 , which contradicts with (cid:80)
Ti =1 c
i p i = 1 .
Thus, we obtain the set I (cid:54)
= ∅ .
Without loss of generality, for 1 ≤ i ≤ k and i ≤ j ≤ T , we have c i ≥ 1 / ( (cid:80)
Tt =1 p t ) and p
i ≥ p j , then we obtain that c
i ≥ c j .
It follows that
[Equation]
C P ROOF OF P ROPOSITIONC. C OMPUTATION OF h t
[Equation]
where 1 ( · ) denotes the indicator function.
We achieve Eq.(26) by partitioning the summation into t + 1 groups .
Terms belong to group i have (cid:80) tj =
z j = i .
Let m ti =
[Equation]
, and we achieve Eq.(28)
C.
P ROOF OF R ECURRENT FORMULA OF m t +1 i
We now give the proof of recurrent formula of Eq.(29)
[Equation]
Proof.
[Equation]
1 (cid:80) tj =
z j x j + x t +1 i
+
[Equation] [Equation]
1 ,z 2 , ···
[Equation]
t
[Equation]
tj
[Equation] [Equation]
j
[Equation]
i
[Equation]
P
[Equation]
z
[Equation]
z
[Equation] [Equation]
t ) (cid:80) tj
[Equation]
1 z j x j i
[Equation]
33) =
p t +
[Equation]
z 1 ,z 2 , ··· z t
[Equation]
tj
[Equation]
1 z j
[Equation]
− 1 (cid:17) P ( z
[Equation]
z 2 , · · ·
[Equation]
t ) i − 1 i (cid:80) tj =
z j x j + x t +1 i
−+(
[Equation]
ti (34) =
p t +
[Equation]
[Equation] [Equation]
t
[Equation]
t j =1 z j
[Equation]
i −
[Equation] [Equation]
[Equation]
i − 1 i (cid:80) tj =
z j x j
[Equation]
−+ x
[Equation]
+
[Equation]
+(
[Equation]
p
[Equation]
ti (35) =
p t + (cid:80) z 1 ,z 2 , ···
[Equation]
t(cid:16)(cid:80) tj =1 z j = i − 1 (cid:17) P ( z
[Equation]
z 2 , · · · z t ) (cid:104) b i − 1 (cid:80) tj =
z j x j i −
[Equation]
1 − b
[Equation]
−
[Equation]
) x
[Equation]
+1 (cid:105) +(
[Equation]
p t +1 ) m ti (36)
Then, we have
[Equation]
Since q ti − 1 = P (cid:16)(cid:80)
[Equation]
(cid:16)(cid:80)
[Equation]
=
[Equation]
· · z
[Equation]
) we can achieve that
[Equation]
C
[Equation]
P ROOF OF RECURRENT FORMULA OF q t +1 i
We present the proof of Eq.(39)
[Equation]
Proof.
[Equation]
D R ELATED W ORK
Video Action Analysis.
Researchers have developed quite a few deep networks for video action analysis.
Two-stream networks (Simonyan & Zisserman, 2014) and 3D convolutional neural networks (C3D) (Tran et al., 2015) are popular solutions to learn video representations and these techniques, including their variations, are extensively used for video action analysis.
Recently, a combination of two-stream networks and 3D convolutions, referred to as I3D (Carreira & Zisserman, 2017), was proposed as a generic video representation learning method, and served as an effective backbone network in various video analysis tasks such as recognition (Wang et al., 2016), localization (Shou et al., 2016), and weakly-supervised learning (Wang et al., 2017).
Weakly-Supervised Temporal Action Localization.
There are only a few approaches based on weakly-supervised learning that rely solely on video-level class labels to localize actions in the temporal domain.
Wang et al.
(Wang et al., 2017) proposed a UntrimmedNet framework, where two softmax functions are applied across class labels and proposals to perform action classiﬁcation and detect important temporal segments, respectively.
However, using the softmax function across proposals may not be effective to identify multiple instances.
Singh et al.
(Singh & Lee, 2017) designed a Hide-and-Seek model to randomly hide some regions in a video during training and force the network to seek other relevant regions.
However, the randomly hiding operation, as a data augmentation, cannot guarantee whether it is the action region or the background region that be hidden during training, especially when the dropout probabilities for all the regions are the same.
Nguyen et al.
(Nguyen et al., 2018) proposed a sparse temporal pooling network (STPN) to identify a sparse set of key segments associated with the actions through attention-based temporal pooling of video segments.
However, the sparse constraint may force the network to focus on very few segments and lead to incomplete detection segments.
In order to avoid concentrating only on the most salient regions, we are inspired to propose the MAAN model to explicitly take the expectation with respect to average aggregated features of all the sampled subsets from the video in an end-to-end fashion.
Feature Aggregators.
Learning discriminative localization representations with only video-level class labels requires the feature aggregation operation to turn the multiple snippet-level representations into a video-level representation for classiﬁcation.
Feature aggregation mechanism is widely adopted in deep learning literature and various of tasks and scenarios, for example, neural machine translation (Bahdanau et al., 2015), visual question answering (Hermann et al., 2015), and so on.
However, most of these cases are fully-supervised learning where the goal is to learn a model that attends the most relevant features with the supervision information corresponding to the task directly.
Many variant feature aggregators have been proposed, ranging from the non-parametric max pooling and average pooling, to parametric hard attention (Gkioxari et al., 2015), soft attention (Vaswani et al., 2017; Sharma et al., 2015), second-order pooling (Girdhar & Ramanan, 2017; Kong & Fowlkes, 2017), structured attention (Kim et al., 2017; Mensch & Blondel, 2018), graph aggregators (Zhang et al., 2018a;
Hamilton et al., 2017), and so on.
Different from the fully-supervised setting, where the feature aggregator is designed for its speciﬁc task, we develop a feature aggregator that is trained with class labels, and then it is used to predict the dense action locations for test data.
Different from the heuristic approaches (Wei et al., 2017; Zhang et al., 2018b), which can be considered as a kind of hard-code attention by erasing some regions with a hand-crafted threshold, we introduce the end-to-end differentiable marginalized average aggregation which incorporates learnable latent discriminative probabilities into the learning process.
E M ARGINALIZED A VERAGE A GGREGATION
Algorithm
Input: Feature Representations { x 1 , x 2 , · · · x T } , Sampling Probability { p 1 , p 2 , · · · p T } .
Output: Aggregated Representation x
Initialize m 00 = 0 , q 00 = 1 , b i
= i
i +1 ; for t = 1 to T do
Set m t 0 = 0 , and q t − 1 = 0 and q tt +1 = 0 ; for i = 1 to t do q t i =
p t q t − 1 i − 1 + (1 − p t )
q t − 1 i m ti = p t (cid:0) b i − 1 m t − 1 i − 1 + (1 − b i − 1 ) q t − 1 i − 1 x t (cid:1)
+ (1 − p t ) m t − 1 i end for end for
Return x = T (cid:80) i =
m Ti
F E XPERIMENTS ON W EAKLY -S UPERVISED I MAGE O BJECT L OCALIZATION F. M ODELS AND I MPLEMENTATION D ETAILS
We also evaluate the proposed model on the weakly-supervised object localization task.
For weaklysupervised object localization, we are given a set of images in which each image is labeled only with its category label.
The goal is to learn a model to predict both the category label as well as the bounding box for the objects in a new test image.
[Table]
Based on the model in (Zhou et al., 2016a) (denoted as CAM model), we replace the global average pooling feature aggregator with other kinds of feature aggregator, such as the weighted sum pooling and the proposed MAA by extending the original 1D temporal version in temporal action localization into 2D spatial version.
We denote the model with weighted sum pooling as the weighted-CAM model.
For the weighted-CAM model and the proposed MAAN model, we use an attention module to generate the attention weight λ in STPN or the latent discriminative probability p in MAAN.
The attention module consists of a 2D convolutional layer of kernel size 1 × 1 , stride 1 with 256 units, a
LeakyReLU layer, a 2D convolutional layer of kernel size 1 × 1 , stride 1 with 1 unit, and a sigmoid non-linear activation.
F. D ATASET AND E VALUATION M ETRIC
We evaluate the weakly-supervised localization accuracy of the proposed model on the CUB-2002011 dataset (Wah et al., 2011).
The CUB-200-2011 dataset has 11,788 images of 200 categories with 5,994 images for training and 5,794 for testing.
We leverage the localization metric suggested by (Russakovsky et al., 2015) for comparison.
This metric computes the percentage of images that is misclassiﬁed or its bounding boxes have less than 50% IoU with the groundtruth as the localization error.
F. C OMPARISONS
We compare our MAA aggregator (MAAN) with the weighted sum pooling (weighted-CAM) and global average pooling (CAM (Zhou et al., 2016b)).
For MAAN and weighted-CAM, we pool the convolutional feature for aggregation into two different sizes, 4 × 4 and 7 × 7 .
We ﬁx all other factors (e.g. network structure, hyper-parameters, optimizer) the same, except for the feature aggregators, to evaluate the models.
F.3. Q UALITATIVE R ESULTS
The localization errors for different methods are presented in table 4, where the GoogLeNet-GAP is the CAM model.
Our method outperforms GoogLeNet-GAP by 5 .
06% in Top-1 error.
Meanwhile,
MAAN achieves consistently lower localization error than weighted-CAM on the two learning schemes.
It demonstrates that the proposed MAAN can improve the localization performance in the weakly-supervised setting.
Moreover, both MAAN and weighted-CAM obtain smaller localization error when employing the 7 × 7 learning scheme than the 4 × 4 learning scheme.
F.3. V ISUALIZATION
Figure.
5 visualizes the heat maps and localization bounding boxes obtained by all the compared methods.
The object localization heat maps generated by the proposed MAAN can cover larger object regions and obtain more accurate bounding boxes.