Paper:
A new model for video captioning is developed, using a deep three-dimensional Convolutional Neural Network (C3D) as an encoder for videos and a Recurrent Neural Network (RNN) as a decoder for captions. A novel attention mechanism with spatiotemporal alignment is employed to adaptively and sequentially focus on different layers of CNN features (levels of feature “abstraction”), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on the YouTube2Text benchmark. Experimental results demonstrate quantitatively the effectiveness of our proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantic structures.
1 INTRODUCTION
Accurately understanding the fast-growing number of videos poses a significant challenge for computer vision and machine learning. An important component of video analyasis involves generating natural-language video descriptions, i.e., video captioning. Inspired by the successful deployment of the encoder-decoder framework used in machine translation (Cho et al., 2014) and image caption generation (Vinyals et al., 2015; Pu et al., 2016; Gan et al., 2017), most recent work on video captioning (Venugopalan et al., 2015; Yu et al., 2016) employs a 2-dimentional (2D) or 3-dimentional (3D) Convolutional Neural Network (CNN) as an encoder, mapping an input video to a compact feature vector representation; a Recurrent Neural Network (RNN) is typically employed as a decoder, unrolling this feature vector to generate a sequence of words of arbitrary length.
Despite achieving encouraging successes in video captioning, previous models suffer from important limitations. First, the rich contents in an input video is often compressed to a single compact feature vector for caption generation; this approach is prone to miss detailed spatiotemporal information. Secondly, the video feature representations are typically extracted from the output of a CNN at a manually-selected fixed layer, which is incapable of modeling rich context-aware semantics that requires focusing on different abstraction levels of features. As investigated in Zeiler & Fergus (2014); Simonyan et al. (2014), the features from layers at or near the top of a CNN tends to focus on global semantic discriminative visual percepts, while low-layer feature provides more local, fine-grained information. It is desirable to select/weight features from different CNN layers
∗Most of this work was done when the author was an intern at NEC Labs America.
adaptively when decoding a caption, selecting different levels of feature abstraction by sequentially emphasizing features from different CNN layers. In addition to focusing on features from different CNN layers, it is also desirable to emphasize local spatiotemporal regions in feature maps at particular layers.
To realize these desiderata, our proposed decoding process for generating a sequence of words dynamically emphasizes different levels (CNN layers) of 3D convolutional features, to model important coarse or fine-grained spatiotemporal structure. Additionally, the model employs different contexts and adaptively attends to different spatiotemporal locations of an input video. While some previous models use 2D CNN features to generate video representations, our model adopts the features from a pre-trained deep 3D convolutional neural network (C3D); such features have been shown to be natural and effective for video representations, action recognition and scene understanding (Tran et al., 2015) by learning the spatiotemporal features that can provide better appearance and motion information. In addition, the proposed model is inspired by the recent success of attention-based models that mimic human perception (Mnih et al., 2014; Xu et al., 2015).
The principal contributions of this paper are as follows: (i) A new video-caption-generation model is developed by dynamically modeling context-dependent feature abstractions; (ii) New attention mechanisms to adaptively and sequentially emphasize different levels of feature abstraction (CNN layers), while also imposing attention within local spatiotemporal regions of the feature maps at each layer are employed; (iii) 3D convolutional transformations are introduced to achieve spatiotemporal and semantic feature consistency across different layers; (iv) The proposed model achieves state-ofthe-art performance on Youtube2Text benchmark. We call the proposed algorithm Adaptive SpatioTemporal representation with dynAmic abstRaction (ASTAR).
2 METHOD
Consider N training videos, the nth of which is denoted X(n), with associated caption Y(n). The length-Tn caption is represented Y(n) = (y (n) 1 , . . . ,y (n) Tn
), with y(n)t a 1-of-V (“one hot”) encoding vector, with V the size of the vocabulary.
For each video, the C3D feature extractor (Tran et al., 2015) produces a set of features A(n) = {a(n)1 , . . . ,a (n) L ,a (n) L+1}, where {a (n) 1 , . . . ,a (n) L } are feature maps extracted from L convolutional layers, and the fully connected layer at the top, responsible for a(n)L+1, assumes that the input video is of the same size for all videos. To account for variable-length videos, we employ mean pooling to the video clips, based on a window of length 16 (as in (Tran et al., 2015)) with an overlap of 8 frames.
2.1 CAPTION MODEL
For notational simplicity, henceforth we omit superscript n. The t-th word in a caption, yt, is mapped to an M -dimensional vector wt = Weyt, where We ∈ RM×V is a learned wordembedding matrix, i.e., wt is a column of We chosen by the one-hot yt. The probability of caption Y = {yt}t=1,T is defined as
p(Y|A) = p(y1|A) ∏T t=2 p(yt|y<t,A) . (1)
Specifically, the first word y1 is drawn from p(y1|A) = softmax(Vh1), where h1 = tanh(CaL+1). Bias terms are omitted for simplicity throughout the paper. All the other words in the caption are then sequentially generated using an RNN, until the end-sentence symbol is generated. Conditional distribution p(yt|y<t,A) is specified as softmax(Vht), where ht is recursively updated as ht = H(wt−1,ht−1, zt). V is a matrix connecting the RNN hidden state to a softmax, for computing a distribution over words. zt = φ(ht−1,a1, . . . ,aL) is the context vector used in the attention mechanism, capturing the relevant visual features associated with the spatiotemporal attention (also weighting level of feature abstraction), as detailed in Sec. 2.2. The transition function H(·) is implemented with Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997). Given the video X (with features A) and associated caption Y, the objective function is the sum of the log-likelihood of the caption conditioned on the video representation:
log p(Y|A) = log p(y1|A) + ∑T t=2 log p(yt|y<t,A) , (2)
Equation (2) is a function of all model parameters to be learned; they are not explicitly depicted in (2) for notational simplicity. Further, (2) corresponds to a single video-caption pair, and when training we sum over all such training pairs.
2.2 ATTENTION MECHANISM
We introduce two attention mechanisms when predicting word yt: (i) spatiotemporal-localization attention, and (ii) abstraction-level attention; these, respectively, measure the relative importance of a particular spatiotemporal location and a particular CNN layer (feature abstraction) for producing yt, based on the word-history information y<t.
To achieve this, we seek to map al → âl, where 4D tensors âl all have the same dimensions, are embedded into same semantic spaces, and are aligned spatialtemporally. Specifically, âl, l = 1, . . . , L−1 are aligned in the above ways with aL. To achieve this, we filter each al, l = 1, . . . , L− 1, and then apply max-pooling; the filters seek semantic alignment of the features (including feature dimension), and the pooling is used to spatiotemporally align the features with aL. Specifically, consider
âl = f( ∑nlF k=1 al(k) ∗Uk,l), (3)
for l = 1, . . . , L − 1, and with âL = aL. al(k) is the 3D feature map (tensor) for dictionary k ∈ {1, . . . , nlF } at layer l, and Uk,l is a 4D tensor. The convolution ∗ in (3) operates in the three shift dimensions, and al(k)∗Uk,l manifests a 4D tensor. Function f(·) is an element-wise nonlinear activation function, followed by max pooling, with the pooling dimensions meant to realize final dimensions consistent with aL. Consequently, âi,l ∈ Rn L F is a feature vector.
With {âl}l=1,L semantically and spatiotemporally aligned, we now seek to jointly quantify the value of a particular spatiotemporal region and a particular feature layer (“abstraction”) for prediction of the next word. For each âi,l, the attention mechanism generates two positive weights, αti and βtl, which measure the relative importance of location i and layer l for producing yt based y<t. Attention weights αti and βtl and context vector zt are computed as
eti = w T α tanh(Waαâi +Whαht−1), αti = softmax({eti}), st = ∑L l=1 αtiâi, (4)
btl = w T β tanh(Wsβstl +Whβht−1), βtl = softmax({btl}), zt = ∑L l=1 βtlstl, (5)
where âi is a vector composed by stacking {âi,l}l=1,L (all features at position i). eti and btl are scalars reflecting the importance of spatiotemporal region i and layer t to predicting yt, while αti and βtl are relative weights of this importance, reflected by the softmax output. In (4) we provide attention in the spatiotemporal dimensions, with that spatiotemporal attention shared across all L (now aligned) CNN layers. In (5) the attention is further refined, focusing attention in the layer dimension.
3 EXPERIMENTS
We present results on Microsoft Research Video Description Corpus (YouTube2Text) (Chen & Dolan, 2011). The Youtube2Text contains 1970 Youtube clips, and each video is annotated with around 40 sentences. For fair comparison, we used the same splits as provided in Yu et al. (2016), with 1200 videos for training, 100 videos for validation, and 670 videos for testing. We convert all captions to lower case and remove the punctuation, yielding vocabulary sizes V = 12594.
We consider the RGB frames of videos as input, and all videos are resized to 112×112 spatially, with 2 frames per second. The C3D (Tran et al., 2015) is pretrained on Sports-1M dataset Karpathy et al. (2014), consisting of 1.1 million sports videos belonging to 487 categories. We extract the features from four convolutional layers and one fully connected layer, named as pool2,
pool3, pool4, pool5 and fc-7 in the C3D (Tran et al., 2015), respectively.
The widely used BLEU (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005) and CIDEr (Vedantam et al., 2015) metrics are employed to quantitatively evaluate the performance of our video caption generation model, and other models in the literature.
Results are summarized in Tables 1, and we outperform the previous state-of-the-art result on Youtube2Text. This demonstrates the importance of leveraging intermediate convolutional layer features. In addition, we achieve these results using a single model, without averaging over an ensemble of such models.
4 CONCLUSION AND FUTURE WORK
We have proposed a novel video captioning model, that adaptively selects/weights the feature abstraction (CNN layer), as well as the location within a layer-dependent feature map. Our model achieves state-of-the-art video caption generation performance on Youtube2Text benchmark.

Review:
The authors apply the image captioning architecture of Xu et al. 2015 to video captioning. The model is extended to have attention over multiple layers of the ConvNet instead of just a single layer. Experiments on YouTube2Text, M-VAD and MSR-VTT show that this works better than only using one of the layers at a time.
I think this is solid work on the level of a well-executed course project or a workshop paper. The model makes sense, it is adequately described, and the experiments show that attending over multiple layers works better than attending over any one layer in isolation. Unfortunately, I don't think there is enough to get excited about here from a technical perspective and it's not clear what value the paper brings to the community. Other aspects of the paper, such as including the hard attention component, don't seem to add to the paper but take up space. 
If the authors want to contribute a detailed, focused exploration of multi-level features this could become a more valuable paper, but in that case I would expect a much more thorough exploration of the choices and tradeoffs of different schemes without too many spurious aspects such as video features, hard attention, etc.