T EMPORAL G AUSSIAN M IXTURE L AYER FOR V IDEOS
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer and present how it can be used to efﬁciently capture longer-term temporal information in continuous activity videos.
The TGM layer is a temporal convolutional layer governed by a much smaller set of parameters (e.g., location/variance of Gaussians) that are fully differentiable.
We present our fully convolutional video models with multiple TGM layers for activity detection.
The experiments on multiple datasets including Charades and MultiTHUMOS conﬁrm the effectiveness of TGM layers, outperforming the state-of-the-arts.
I NTRODUCTION
Activity videos are spatio-temporal data: they are image frames with a speciﬁc width/height (XY) concatenated along time axis (T).
Recognition from such videos requires capturing both spatial and temporal information in the videos, desirably using learned convolutional kernels.
Temporal convolution is particularly beneﬁcial in activity ‘detection’ tasks, which require making activity decisions at every frame given a continuous video (Sigurdsson et al., 2016b; Yeung et al., 2015).
Previous methods investigated using 3-D XYT convolutional ﬁlters (Tran et al., 2014; Carreira &
Zisserman, 2017) as well as the models with 2-D XY conv.
layers followed by 1-D temporal conv.
(Tran et al., 2018), pooling or attention layers (Piergiovanni et al., 2017).
Understanding complex multi-activity videos requires capturing information in long-term time intervals.
Different frames contain different information, and the model needs to learn to take advantage of as many frames as possible, while abstracting them efﬁciently.
Previous attempts of simply pooling representations over time or learning temporal conv.
ﬁlters with a small number of frames (e.g., 16 or 64) was thus often insufﬁcient to fully consider rich long-term temporal context.
Simultaneously, bruteforcely increasing the temporal ﬁlter length (to look at more frames) results more learnable parameters, requiring more training data, which can be expensive when activities are rare.
In this paper, we introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer, and present how it can be used to efﬁciently capture longer-term temporal information in activity videos.
Our temporal Gaussian mixture layer is a temporal convolutional layer, whose ﬁlters/kernels are controlled by a set of (temporal) Gaussian distribution parameters.
Each of our temporal Gaussian distributions specify (temporally) ‘where’ the model should look, and our Gaussian mixture layer combines them as multiple convolutional ﬁlters to be applied on top of temporallycontinuous representations.
This layer allows the video representation at each time step to be constructed while focusing on different neighboring temporal regions, instead of only focusing on its local segment.
It is a convolutional layer governed by a much smaller set of parameters (i.e., locations/variances of the Gaussians as well as their mixture weights) that are fully differentiable.
The motivation behind our temporal Gaussian mixture layer is to learn the temporal structure of an activity as a composition of temporal Gaussian regions/attentions.
Such structure allows the model to obtain a compact spatio-temporal representation abstracting each (long-term) time interval, using multiple temporal conv.
layers with far fewer parameters.
It is also related to the previous temporal attention works (Piergiovanni et al., 2017), but our model is designed to be fully convolutional to handle continuous data and it learns more compositional structures with multiple layers.
We present video-CNN models using our TGM layers for activity detection in continuous videos.
Our model stacks TGM layers on top of several state-of-the-art CNNs such as I3D (Carreira &
Zisserman, 2017).
This enables our model to capture longer-term temporal information than what we use as base CNNs, compositionally modeling temporal structure with multiple TGM layers.
Our model was evaluated on multiple public datasets including MultiTHUMOS and Charades, and was able to outperform the best previous activity detection CNNs by a meaningful margin.
R ELATED W ORKS
Learning video representations for human activity recognition has been successful.
CNN methods allow end-to-end learning of video features and representations optimized for the training data, performing superior to traditional works (Aggarwal & Ryoo, 2011) for video understanding.
Two-stream CNN models take a single RGB frame and a small number of optical ﬂow frames as inputs to capture both motion and appearance information in videos (Simonyan & Zisserman, 2014; Feichtenhofer et al., 2016).
Models learning 3-D spatio-temporal (XYT) convolutional ﬁlters were designed and applied to many activity recognition tasks as well (Tran et al., 2014; Carreira &
Zisserman, 2017; Tran et al., 2017; Hara et al., 2017).
Large scale datasets for activity detection, such as THUMOS (Jiang et al., 2014), ActivityNet (Heilbron et al., 2015), Kinetics (Kay et al., 2017), and Charades (Sigurdsson et al., 2016b) provided these approach the necessary training data to learn the models.
Such 3-D XYT CNNs were also used to capture spatio-temporal information for activity detection (Xu et al., 2017; Shou et al., 2016; 2017; Zhao et al., 2017).
However, all these
CNNs were limited to the consideration of a ﬁxed local video segment (e.g., 16 frames in (Tran et al., 2014) and 64-99 frames in (Carreira & Zisserman, 2017)) when making activity decisions.
Some works studied combining representations over longer-term temporal intervals (Karpathy et al., 2014; Ng et al., 2015; Varol et al., 2017), but it was generally done with a temporal pooling of local representations or (spatio-)temporal convolutions with a bit larger ﬁxed intervals.
Recurrent neural networks (RNNs) have also been used to model activity transitions between frames (Yeung et al., 2015; 2016; Escorcia et al., 2016), but they were strictly sequential and had limitations in maintaining temporal information over a longer temporal duration, particularly for videos with multiple complex activities.
Recently, CNN models using temporal attention for activity videos (Piergiovanni et al., 2017; Piergiovanni & Ryoo, 2018b) were studied as well.
However, a fully convolutional model to analyze continuous videos while efﬁciently representing information in long term intervals has been lacking.
Our layer is different from the previous standard (spatio-)temporal convolutional layers in that it relies on signiﬁcantly fewer parameters by forcing ﬁlter shapes to be Gaussian compositions.
Our temporal layer is also different from previous Gaussian Mixture Model layers (Variani et al., 2015) in that our layer is convolutional while they are not.
A PPROACH
In this section, we introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer, and present how it can be used for activity recognition.
Our Temporal Gaussian
Mixture layer is a temporal convolutional layer to be applied on top of a sequence of representations (usually from frame-level or segment-level CNNs), whose ﬁlters/kernels are controlled by a set of (temporal) Gaussian distribution parameters.
The motivation is to make each temporal Gaussian distribution specify (temporally) ‘where to look’ with respect to the activity center, and represent the activity as a collection/mixture of such temporal Gaussians convolved with video features.
Our layer is fully differentiable and trainable using standard backpropagation.
Our TGM layer can be interpreted as a a form of 1-D convolution where the ﬁlters are determined by a mixture of Gaussians.
However, our TGM layer differs from the standard temporal convolutional layers of learning 1-D (time) or 2-D (channel-by-time) ﬁlters in the following aspects:
Our temporal Gaussian mixture layer handles multiple 3-D tensors internally to preserve channels from the frame-level CNN by adding a new temporal channel axis.
Its input is 3-D (channel-by-channel-by-time), where one channel dimension is inherited from the frame-level CNN and this dimension size remains unchanged.
Instead of learning temporal convolution ﬁlters of any arbitrary values, our ﬁlter is forced to have the form of a temporal Gaussian mixture shared across all frame-level channels.
This allows the layer to rely on signiﬁcantly fewer number of (fully differentiable) parameters, while capturing the concept of temporal structure/attention.
[Figure]
C
M M
L
C
L
Gaussians
Temporal Gaussian
Mixture
[Figure]
K C
3. T EMPORAL G AUSSIAN M IXTURE LAYER
Our temporal Gaussian mixture layer takes a 3-D input with the dimensionality of C in × D × T , where C in is the number of input channels, D is the dimensionality of the representations from frame-level (or segment-level) CNNs, and T is the time.
Given such input, the TGM layer convolves it with C out number of 1 × L ﬁlters/kernels, generating a C out × D × T -dim representation as an output.
L is the temporal length of the temporal Gaussian mixture ﬁlter.
D is usually 1K or 4K and T is the number of time steps (frames) in each video (i.e., it varies per video).
C out is the number of different mixtures, corresponding to the number of output channels in standard convolution.
Our layer is composed of a set of M Gaussians.
Each Gaussian has 2 parameters: a center ˆ µ and a width ˆ σ .
Each layer has additional hyper-parameters:
L , the temporal duration and M , the number of Gaussians to learn.
We force the learned center to be between − L 2 and L 2 and σ to be positive:
[Equation]
We use the above µ and σ to construct the temporal Gaussian kernels.
This acts as a strong sparsity constraint on the convolutional kernel as well as a drastic reduction of the number of learnable parameters.
We construct a temporal Gaussian mixture convolutional kernel as:
[Equation]
where Z is a normalization constant such that (cid:80)
Ll ˆ K m,l = 1 , resulting in ˆ K being an M × L matrix.
Instead of making the model learn a separate set of Gaussian distributions per activity class, we take the approach of maintaining multiple Gaussian distributions shared across classes and obtain a
Gaussian ‘mixture’ ﬁlter by learning soft-attention weights.
We learn a set of soft-attention weights per output channel i , ω ∈ R C out × M .
We create the soft-attention weights by applying the softmax function over the M Gaussians, enforcing each input channel weights sum to 1.
[Equation]
Based on temporal Gaussian distributions ˆ K i and attention weights a i,m , the temporal convolution ﬁlters our TGM layer is computed as:
[Equation]
This provides us convolutional ﬁlters having the form of a mixture of temporal Gaussians, controlled based on 2 · M + C in · C out · M parameters (instead of learning D 2 · L parameters without any constraint, as in standard temporal convolution where C << D ).
An overview of this process is shown in Fig.
3.1. S INGLE TGM LAYER - DIRECT PER - CLASS ACTIVITY MODELING
The representation we obtain by applying our base CNNs to each frame (or local segment) has the dimensionality of D , and stacking them along time axis provides us the representation with [Equation]
[Figure]
T
[Figure]
= = ...
[Figure]
T
D
C
1 × D × T -dim.
That is, in the case of using only one TGM layer to capture activity representations, our C in is ﬁxed to 1 and C out is ﬁxed to be the number of activity classes.
This is the simplest case of our model, attaching one TGM layer on top of the 1 × D × T representation.
Our convolutional kernel, K , has a learned Gaussian mixture for each activity class.
Let the video features v be a D × T matrix.
Each K i is a 2-D convolutional ﬁlter with a size of 1 × L , and convolving this with v provides us a representation S with C out
number of D × T responses since
C in is 1 in this case.
This per-class representation can then be used as input to a fully-connected layer for activity classiﬁcation.
For i ∈ { 1 , 2 , . . .
, C out } :
[Equation]
Fig. 7 in the appendix visually illustrates how each TGM ﬁlter is convolved with the input (Fig. 7d), compared to the standard 1-D convolution (Fig. 7a) or other forms of the temporal layers (Fig. 7b-c).
3.1. M ULTIPLE TGM LAYERS - GROUPED CONVOLUTION
We generalize the above formulation to allow the TGM layers to be sequentially applied.
The idea is to enable our model to capture more complex, nonlinear temporal structure by having multiple levels of temporal layers.
In this case, the input for each layer is C in × D × T dimensional (instead of 1 × D × T ), where the input channels are the number of output channels from the previous layer.
Our kernels at each layer, K i , are parameterized and learned as before.
By using grouped convolution with the number of groups set to C in , we can efﬁciently separate the input into per-channel values and convolve each of them with the designated K i kernel, as shown in
Fig.
That is, we learn a ﬁlter K i per channel by setting C in = C out .
For i ∈ [1 , C out ] ,
[Equation]
Here, f is a C in × D × T tensor, where D is the dimensionality of the feature and T is the number of frames.
The result of the per-channel convolution, s i , is a D × T representation.
We concatenate these representations along the channel axis, resulting in S , a C out × D × T representation.
As this convolution results in the same output shape, we can stack these layers.
Each layer is able to capture increasing temporal resolution, allowing the model to capture levels of abstractions.
3.1. M ULTIPLE TGM LAYERS - CHANNEL COMBINATION
In the above subsection, we introduced an approach of stacking multiple TGM layers to model a hierarchical composition of temporal representations.
However, in the grouped convolution case, each output channel of the layer is solely dependent on its corresponding input channel. That is, each kernel only considers information from a single output channel of the previous layer. Therefore, we further generalize our TGM layer so that the layer combines representations from multiple input channels for each output channel while using the learned temporal kernels. We learn a set of convolutional kernels K ∈ R C out × C in × L (i.e., we learn C out · C in Gaussian mixtures).
Given f which is the C in × D × T representation, for each output channel i ∈ [1 , C out ] and each input channel j ∈ [1 , C in ] pair, we convolve the associated ﬁlters with the input.
[Equation]
where each G i,j is a D × T -dim representation.
We then learn a 1x1 convolution followed by a ReLU activation function for each i ∈ [1 , C out ] , which we call w i , that maps from C in channels to 1 channel.
The 1x1 convolution learns to combine
T D
C in
C in
Input
TGMs Representation for each C in ... 1* = C in
T
D
C in 1* =
C in
T D T D
C out 1x1 Convolution to combine C in
Concatenation of C out representations ... ...
1 L K i1 ...
T
D * * ...
= = ...
T
K ij
T
D 1 L K 11 ...
T
D * * ...
= = ...
T
K 1j
T
D the channels from the previous layer.
By design, the TGM kernel is positive and sums to 1.
Adding the unconstrained 1x1 convolution adds non-linearity (using the ReLU activation function) to our layer and only adds C out · C in parameters.
[Equation]
We then stack the s i representations along the channel axis to produce S , the C out × D × T
-dim
representation.
This process is illustrated in Fig.
This method generalizes our approach to allow the layer to take input of C in × D × T and produce output of C out × D × T .
These layers can easily be stacked to learn a hierarchical representation.
3. V IDEO CNN MODELS WITH TGM LAYERS
Our goal is to do activity detection which we deﬁne as making a per-frame (or per-segment) classiﬁcation.
Given a video, at each time step t , we want to make the model decide which activity the frame corresponds to (including no-activity).
As a baseline, we train a fully-connected layer that classiﬁes each per-frame D -dimensional vector, v t .
As multiple activities can occur at the same time, or no activities at all, we treat this as a mutli-label classiﬁcation task.
We minimize binary cross entropy:
[Equation]
where z t,c is the ground truth label, 1 if activity c is occurring at time t and p ( c | v t ) is the output of our model for class c at time t . Fig. 4 shows an example CNN. [Figure]
P e r - F r a m e C l a ss C l a ss i f i c a t i o n
[Figure]
C
C
C
[Figure]
C f xDx C f xT  E XPERIMENTS 4. I MPLEMENTATION AND BASELINES Implementation We used I3D (Carreira & Zisserman, 2017) and the two-stream version of InceptionV3 (Szegedy et al., 2016) pretrained on Imagenet and Kinetics as our base per-frame CNNs.
Our default L setting used for the TGM layers as well as the other baselines was as follows: when using I3D segment features (collected at 3fps), the 1 layer models used L = 15 and the 3 layer models used L = 5 . When using InceptionV3 frame feature (collected at 8fps), the 1 layer models used L = 30 and the 3 layer models used L = 10 .
These layers were attached on top of the base CNN, as described in Subsection 3.2.
Please check the appendix for implementation and training details and results on other datasets.
Baselines In order to conﬁrm the advantages of our TGM layers, particularly against previous temporal models, we implemented several baselines.
The ﬁrst is (i) a standard per-frame classiﬁer in which the prediction at each time-step only depends on a single feature vector with no contextual temporal information.
We also used (ii) LSTMs on top of per-frame representations, which were popularly used to capture temporal information (Donahue et al., 2015).
We train a bi-directional
LSTM with 512 hidden units to make per-frame predictions.
We also tried (iii) the ﬁxed pyramid temporal max-pooling of level 3 (Ryoo et al., 2015).
Finally, we compare our model against (iv) the model with standard temporal convolutional layers (i.e., 1-D convolution with a D × L kernel) on top of per-frame representations.
This is similar to the temporal conv.
used in (Tran et al., 2018).
Temporal lengths (i.e., L ) of the 1-D conv.
ﬁlters and the pooling windows were set to be identical to the TGM ﬁlters.
That is, they capture the same temporal duration as TGMs.
In all our experiments, we follow the standard evaluation setting of computing per-frame mean average precision (mAP) and report those values.
We also compare to different versions of the TGM layer, (v) with a learned mixture of random temporal ﬁlters and (vi) with a learned mixture of ﬁxed Gaussians.
In addition, we also tried the approach of combining our TGM layers with the recent super-event representations (Piergiovanni & Ryoo, 2018b).
We concatenated the learned super-event representation with our representations from TGM layers.
4. M ULTI THUMOS
Dataset MultiTHUMOS (Yeung et al., 2015) is an extended version of the THUMOS (Jiang et al., 2014) dataset that densely annotates the continuous videos.
The dataset consists of 65 different classes, compared to 20 in THUMOS, and contains on average 10.5 activities per video and 1.
labels per frame and up to 25 activity instances in each video.
This is in contrast to many other activity detection dataset such as ActivityNet (Heilbron et al., 2015), which only has on average ∼activity per video.
MultiTHUMOS consists of YouTube videos of various sport activities such as basketball games, volleyball games, weight lifting, and track and ﬁeld.
We followed the standard MultiTHUMOS evaluation setting of measuring mAP based on per-frame annotations.
There are 1010 validation videos and 1574 test videos.
We used these continuous validation videos for the training of our models.
We did not need to take advantage of the separate training set with segmented videos; even without them, we outperformed the state-of-the-arts.
Results We compared baselines as well as multiple different versions of our architectures, shown in Table 1.
The model with our TGM layers consistently outperformed baseline I3D (or InceptionV3) while using the same per-segment representations.
Learning 3 TGM layers further improved the performances.
On the other hand, we found that stacking multiple standard temporal convolutional layers does not improve performance, often performing worse than the baseline.
While a single standard temporal conv.
layer improves over the baseline, having multiple of them signiﬁcantly increases the number of parameters to learn (Table 2) and we suspect that this was causing the overﬁtting with the limited amount of samples in the dataset.
In Table 3, we compare the results of using a LSTM or temporal conv.
with a similar number of parameters.
This was done by making their temporal conv.
ﬁlters to share values across multiple channels.
These models result in nearly random performance, as they were not designed to cope with a small number of parameters.
We also show results with a mixture of random (ﬁxed) temporal ﬁlters and with a mixture of ﬁxed Gaussians.
These results conﬁrm that (i) modeling the temporal structure as a learned Gaussian mixture is beneﬁcial and that (ii) further learning the Gaussian distribution parameters is important.
[Table]
[Table] [Table]
[Table]
Learning multiple TGM layers with channel combination outperforms the grouped convolution version of TGM and all the baselines.
We also experimented with a version using soft-attention weights to combine the TGM layer channels, in addition to our method (Fig. 3) of using 1x1 convolution followed by a ReLU (to gain non-linearity).
We found that the 1x1 convolution performed better.
We tested various number of Gaussian mixtures (i.e., output channels) and found that using 80 for the ﬁrst and second layer and using 65 (i.e., number of classes) for the ﬁnal layer performs best.
Table 4 compares our model using TGM layers with multiple previous state-of-the-art approaches and baselines such as LSTM.
Our approach meaningfully outperforms all previous approaches.
Importantly, we are comparing our approach with different methods of capturing temporal information such as LSTMs and ﬁxed temporal pyramid pooling while making them use the exactly same per-frame representations.
We found that while all these methods capture some temporal information, the TGM layers provide the best performance.
Further, combining the super-event representation (Piergiovanni & Ryoo, 2018b) with our TGM feature also beneﬁted detection, conﬁrming that our TGMs and super-events capture different aspects of the activity videos.
In Fig.
5, we show an example of the various models predictions on a basketball video.
We outperform the previous state-of-the-art performance (mAP) by 10% (36.4 vs. 46.4).
4. C HARADES
Dataset Charades (Sigurdsson et al., 2016b) is a large scale dataset with 9848 videos acrossactivity classes.
These videos were recorded in home environments of the participants based on provided scripts.
Each video contains on an average of 6.8 activity instances, and there are often complex activities co-occurring.
The activities were mainly performed at home.
For example, some activity classes are ‘preparing a meal’, ‘eating’, ‘sitting’, ‘cleaning’, etc.
In our experiments, we follow the original Charades detection setting (i.e., Charades v1 localize evaluation), which is the setting used in many previous approaches (Sigurdsson et al., 2016a; Xu Guard Shot
Block
[Figure] Dunk Ground Truth Baseline
Super-Events
Temporal Conv
TMG
TMG + Super
[Table]
et al., 2017; Piergiovanni & Ryoo, 2018b).
This is the original setting more challenging than the Charades Challenge 2017 setting (whose evaluation server was no longer approving new account access) [Table] in the aspect that it uses less amount of training videos.
Results We compare our results with the state-of-the-arts in Table 5.
To our knowledge, our method is obtaining the best known performance in the original localization setting of the Charades dataset.
Notably, it is performing better than I3D that obtained the best competition performance, while using the same feature.
Our method also outperforms standard temporal convolution, LSTMs, and ﬁxed pyramid pooling, as well as the use of latent super-events.
When setting L = 30 and using 3 TGM layers, our model is able to capture around 800 frames (about ± 15 seconds from each frame) of temporal information, signiﬁcantly more than previous works (e
[Table]
g., I3D only captures ±seconds).
C ONCLUSIONS
We newly introduced the Temporal Gaussian Mixture (TGM) layer and demonstrated its effectiveness for multi-activity detection in continuous videos.
Our layer is fully differentiable and trainable using standard backpropagation, designed to learn temporal structure.
We were able to conﬁrm that our layer performs superior to state-of-the-art methods on activity detection datasets including MultiTHUMOS and Charades, obtaining the best known performance.
We also tested our approach with two more public video datasets, MLB-YouTube (Piergiovanni & Ryoo, 2018a) and
AVA (Gu et al., 2017), and conﬁrmed its advantage over the previous works in Appendix.
A I MPLEMENTATION D ETAILS
As our base per-segment CNN, we use the I3D (Carreira & Zisserman, 2017) network pretrained on the ImageNet and Kinetics (Kay et al., 2017) datasets.
I3D obtained state-of-the-art results on segmented video tasks, and this allows us to obtain reliable v t .
We also use two-stream version of
InceptionV3 (Szegedy et al., 2016) pretrained on Imagenet and Kinetics as our base per-frame CNN, and compared them.
We chose InceptionV3 as it is deeper than previous two-stream CNNs such as (Simonyan & Zisserman, 2014; Feichtenhofer et al., 2016).
We extracted frames from the videos at 25 fps, computed TVL1 (Zach et al., 2007) optical ﬂow, clipped to [ − 20 , 20] .
For InceptionV3, we computed features for every 3 frames (8 fps).
For I3D, every frame was used as the input.
I3D has a temporal stride of 8, resulting in 3 features per second (3 fps).
We implemented our TGM layers as well as other baseline layers in PyTorch.
Our default setting was as follows: for 3-layer models, we set L = 10 for frame-based features (i.e., InceptionV3) and L = 5 for segment-based features (i.e., I3D), as each segment already contains some temporal information. For 1-layer models, we set L = 30 for frame-based features and L = 15 for segmentbased features. We set M = 16 and C out = 80 and C out = 65 for the last TGM layer.
We found these values to work well on a held out portion of the training set of MultiTHUMOS.
In all models, we used one fully-connected layer at the end to make the per-frame or per-segment classiﬁcation.
We trained our models using the Adam (Kingma & Ba, 2014) optimizer with the learning rate set to 0.01.
We decayed the learning rate by a factor of 10 after every 10 training epochs.
We trained our models for 50 epochs.
We plan to make all our source code and trained models publicly available once the paper is published. B H YPERPARAMETER E XPERIMENTS
We conducted a set of experiments to compare the effects of the temporal duration, L , number of
Gaussians, M , and the number of output channels, C out . For these experiments, we only used the one-stream version of I3D with RGB inputs. Effect of L : In Table 6, we compare different values of L .
For these experiments, we use M =and C out = 16 . We ﬁnd that the 3-layer model with L = 5 performs the best. With I3D features, this allows the model to capture up to 8 seconds of information. The average activity in MultiTHUMOS is 3.3 seconds long and the maximum is 14.7 seconds long, and with this setting, the model is able to capture enough temporal context to perform well.
Larger values of L capture too much temporal information, but due to the Gaussian structure, it does not drastically harm performance.
Figureshows that even with longer kernels, the Gaussians learn to focus mostly on the center of the interval and capture the rough duration of the activities.
Thus, having too long intervals does not drastically harm performance, which is in contrast to the standard 1-D convolution.
Note that for Charades, the temporal kernels are learned to capture much longer temporal duration, as the average activity in charades is 12.8 seconds and larger values of L perform better.
Figure 6 illustrates examples of the learned TGM kernels of various lengths.
The ﬁgure shows that the kernels focus on short temporal intervals on MultiTHUMOS even if we make the ﬁlters longer, as the activities are an average of 3.3 seconds long.
On Charades, the TGM kernels learn to capture much longer intervals, as the activities are an average of 12.8 seconds long.
We believe that this suggests TGMs are learning to capture information from the important necessary intervals.
In Table 6, we also report the results of using a standard 1-D conv.
layer with different L values.
The number of parameters in our TGM layer is independent of L , however, with the standard 1-D conv.
layer, the number of parameters increases as L increases.
We ﬁnd that increasing L with 1-D convolution helps for small values of L , but for L > 15 , the performance drastically drops, while
TGM layers only show a small decrease.
Effect of M :
In Table 7, we compare different values of M .
For these experiments, we set L =and C out = 16 .
We ﬁnd that M = 16 performs best, suggesting that smaller values of M restrict the possible temporal kernels too much.
We also observe that larger values of M performs slightly worse than M = 16 (but not much), likely because they introduce more parameters than needed.
When M and L have similar values, it allows the model to learn a sufﬁcient number of Gaussians
[Table]
[Table]
[Table]
C out =35.
C out =36.
C out =36.
C out =36.
C out =36.
and create a diverse range of temporal kernels.
When M is larger than L , it results in learning a kernel similar to standard 1-D convolution.
Effect of C out : In Table 8, we compare different values of C out .
For these experiments, L = 15 , we used 1-layer and M = 16 .
We ﬁnd that C out performs best when set to 16 or larger on these datasets.
Larger values of C out seem to capture redundant information, as it does not lower performance.
[Figure] D
L T
D * =
T 1-D [Figure] Output * = ... (a)
D
L T
D * =
T
TGMs Input
Output * = ...
(b)
D
L T
D * =
T
TGMs Input
Output * = ...
(c) 1 L T D * =
T
D
TGMs Input
Output * = ...
(d)
[Figure]
[Figure]
mixtures, but unconstrained 1-D temporal kernels.
C C OMPARISON OF D IFFERENT L AYER F ORMS
To conﬁrm the various aspects of our design, we conducted experiments comparing different types of temporal convolution.
In Fig.
7a we illustrate the standard 1-D convolution, taking D × T input and producing a C × T output, where D is the number of input channels and C is the number of output channels.
In Fig.
7b, we illustrate the method of applying a Gaussian mixture kernel as 1-D convolution.
Here, the Gaussian mixture kernel is shared by all D input channels and we learn a
C number of such kernels.
In Fig.
7c, we illustrate the approach of applying a Gaussian mixture kernel as 1-D convolution while learning D different Gaussian mixtures.
This is very similar to the standard 1-D convolution, except that the ﬁlter values are constrained to have the shape of Gaussian mixtures.
Fig.
8 illustrates one more baseline.
This is similar to our full TGM layer with the channelcombination described Fig.
However, in this baseline, instead of learning Gaussian mixtures, we learn C in · C out number of 1 × L kernels.
The kernel values are left unconstrained.
While the TGM layer has 2 · M + C in · C out · M + C in · C out parameters, this layer has L · C in · C out · M + C in · C out , which is more than the TGM layer.
In Table 9, we compare the results of the various above-mentioned layers on MultiTHUMOS using
RGB I3D features.
We ﬁnd that the Fig.
7b method performs poorly, while the Fig.
7c method slightly outperforms the standard 1-D convolution.
The Fig.
8 method is slightly better than the standard 1-D convolution, but performs worse than Fig.
7c.
However, none of these layers perform as well as our TGM layer, conﬁrming that both the design of learning Gaussian mixtures and maintaining temporal channel axis are important for activity detection.
MultiTHUMOS
Standard 1-D Convolution (Fig. 7a) 32.
The layer described in Fig.
7b 28.
The layer described in Fig.
7c 33.
The layer described in Fig.32.
Our TGM Layer 36.
[Figure]
D E XPERIMENTS ON A DDITIONAL D ATASETS D. MLB-Y OU T UBE D ATASET D.1. D ATASET
The MLB-YouTube dataset (Piergiovanni & Ryoo, 2018a) consists of 20 baseball games from the 2017 MLB post-season available on YouTube.
This dataset consists of over 42 hours of video.
For these experiments, we used the continuous video setting which have 2,126 1-2 minute long clips.
Each clip is densely annotated with the baseball activities that occur.
There are 8 activity classes: pitch, strike, ball, swing, hit, foul, hit by pitch, and bunt.
Examples of some of these classes are shown in Fig.
Each continuous clip contains on average of 7.2 activities, giving a total of over 15,000 activity instances in the dataset.
What makes this dataset challenging is that the variation between classes is very small.
In ActivityNet (Heilbron et al., 2015), for example, the difference between swimming and brushing hair is drastic.
The background, motion, and even size of the person in the video is different.
However, in broadcast baseball videos, the difference between a ball and a strike, or a swing and a bunt, are small.
All actions are recorded from the same camera angle as we can conﬁrm from Fig.
D.1. R ESULTS
In Table 10, we compare various approaches on this dataset.
Our TGM layers improve over the baseline by ∼ 6% (40.1 vs. 34.2).
Additionally, we compare to methods using the super-event representation (Piergiovanni & Ryoo, 2018b), which previously achieved state-of-the-art performance on several activity detection datasets.
On this dataset, our approach outperforms the super-event representation, and further the concatenation of our TGM representation with such super-event representation performs best by a signiﬁcant margin ( ∼ 13% compared to the baseline).
This suggests that TGMs and super-event capture different temporal information and are both useful to the detection task.
We further ﬁnd that using multiple, standard temporal convolution layers leads to worse performance, likely due to overﬁtting from the large number of parameters.
While using multiple TGM layers improves performance, conﬁrming that the Gaussian structure and sparsity constraint beneﬁts model learning.
[Table] [Table]
D. AVA D.2. D ATASET
AVA (Gu et al., 2017) is a large-scale video dataset containing of 80 atomic action classes in 57k video clips.
These clips are drawn from movies.
Existing datasets, such as Charades, have very speciﬁc actions that depend on objects, such as holding a cup vs. holding a picture.
In AVA, the actions are intentionally generic, such as sit, stand, hold, carry, etc.
Further, the AVA dataset is annotated with both spatial and temporal locations of activities.
Since we are interested in temporal activity detection, we follow the setting of Piergiovanni & Ryoo (2018b) and label each frame with the occurring activities while ignoring the spatial location.
We evaluate performance following the same method as MultiTHUMOS, Charades and MLB-YouTube by measuring per-frame mAP.
D.2. R ESULTS
In Table 11, we present the results of our model.
We again ﬁnd that temporal convolution and LSTMs provide some beneﬁt over the baseline, but TGM layers further improve performance.
Again, combining the TGM, which captures local temporal structure, with super-events which capture global temporal structure, provides the best performance by ∼ 7 .