The more ﬁne-grained, the better for transfer learning
Abstract
In this paper, we investigate the correlation between the degree of detail (granularity) in the source task and the quality of the learned features for transfer learning to new tasks.
For this purpose, we design a DNN for action classiﬁcation and video captioning.
The same video encoding architecture is trained to solve multiple tasks with different granularity levels.
In our transfer learning experiments, we ﬁne-tune a network on a target task, while freezing the video encoding learned from the source task.
Experiments reveal that training with more ﬁne-grained tasks tends to produce better features for transfer learning.
We use Something-Something dataset with over 220 , 000 videos, and multiple levels of granularity of the target labels.
With impressive coarse-grained and ﬁne-grained classiﬁcation results, our model introduces a strong baseline on the new Something-Something captioning task.
Introduction
Fine-grained video understanding entails recognition of actions, objects and spatiotemporal relations.
A successful framework needs to discriminate myriad variations of actions and interactions, not unlike the emergence of ﬁne-grained tasks in visual object recognition.
To enable extracting rich features from video, right kinds of tasks are needed to train the framework.
There are various levels at which we can describe actions and these levels of granularity match naturally with compositionality of language.
For example, at a coarse-grained level we have actions like ’putting a pen’ .
Then we can have similar actions that differ in relatively subtle ways, for instance, ’putting a pen beside the cup’ , ’putting the pen in the cup’ , or perhaps ’pretending to put the pen in the cup’ .
Adding prepositions and categories like “pretending to put” gives us ﬁne-grained action.
The complexity of the task at this level requires features that capture spatial relations.
As the complexity of the task begins to match the complexity of the world that we are trying to understand, it necessitates more powerful features in order to discriminate these different scenes.
A two-channel DNN architecture is designed for video encoding.
The same architecture is then used for video classiﬁcation and captioning.
Training is performed on Something-Something dataset [1], with 50 coarse-grained action groups, which are further broken to 174 closely related action categories, and a caption authored by the crowd actor.
These captions mirror the ﬁne-grained action category, but with placeholder Something replaced by the speciﬁc object(s).
The main contributions of this paper include:
Explore the link between label granularity and feature quality : We exploit 3 levels of granuWorkshop on Modeling and Decision-Making in the Spatiotemporal Domain, 32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada. [Figure] larity in Something-Something, namely, action groups, action categories, and captions.
Experiments show that more ﬁne-grained labels yield richer features.
Baselines for captioning on Something-Something data : We note that the captioning task is new for this dataset; the original version did not provide captions.
Captioning as a source task for transfer learning : We show that models trained for captioning learn features that transfer better to other tasks.
To the best of our knowledge, captioning has, to date, been used as a target task.
Our results suggest that captioning is a powerful source task.
20bn-kitchenware : We introduce a new dataset, ostensibly for video transfer learning.
Related Work
Video-based action classiﬁcation dates back to seminal work by Laptev et al [2] with hand-tuned features, while most recent approaches have focused on DNN features.
Existing methods differ in the way they aggregate information through time.
Many approaches rely primarily on spatial features with
CNNs applied to individual frames [3].
Other approaches make use of spatiotemporal information [4, 5].
Video captioning have received signiﬁcant attention since the release of large-scale captioning corpora, notably, Microsoft COCO [6] and MSR-VTT [7].
Captioning tasks, if designed appropriately, could represent extremely detailed scene properties.
Most existing captioning architectures are based on an encoder-decoder framework [8, 9, 10].
The encoder is typically a convolutional or recurrent convolutional network.
Despite the signiﬁcant attention to Video tasks, progress has lagged compared to static images, in part because of the lack of large-scale corpora.
Using web sources and human annotators, larger datasets have been collected in recent years [11, 12].
More recently, crowd-sourced data have emerged, where crowd actors are asked to generate videos depicting template actions [1, 13].
One of the most astonishing properties of neural networks is their ability to learn representations that can be successfully transferred to other tasks[14, 15].
One motivation for studying ﬁne-grained video tasks is to understand and improve the potential for transfer learning on video domain.
Architecture
The video encoder , inspired in part by magno- and parvo-cellular pathways in visual cortex, ﬁrst processes the video through a spatial 2D-CNN and a spatio-temporal 3D-CNN in parallel (Fig. 2).
Our video encoder is most closely related to approaches that perform temporal reasoning via a recurrent convolutional architecture [16, 17, 18].
It is also related to TwoStream architectures [19]; but our model does not explicitly use optical ﬂow, opting instead for generic 3D CNN features.
The basic building block of each channel is a 3 × 3 × 3 ( 3 × 3 in 2D-CNN channel) convolution ﬁlter with batchnorm [20] and ReLU activation.
Feature vectors from two channels are concatenated and then fed to a 2-layer bidirectional LSTM.
We average these features to get an encoding of the entire video, h .
This encoding is used by both the classiﬁer and the captioning decoder(See Fig. 1).
The action classiﬁer applies an FC layer to the encoder output h , followed by a softmax layer.
For training we use a cross-entropy loss over the action categories.
[Equation]
The caption decoder is a two-layer LSTM which generates captions using a softmax over the vocabulary words, conditioned on previously generated words.
The loss used for a caption is the
V i
d e o
E n c o d i n g (
[Figure]
f ea t u r e s )
Convolutional
[Figure]
usual negative log-probability of the word sequence: loss captioning = −
N − 1 (cid:88) i = log p ( w i +1 | w ≤ i , h ; θ ) . (2) where w i denotes the i th word of the caption, h is the video encoding, and θ denotes model parameters.
In order to optimize speed and memory usage during training, the length of captions generated by the decoder is ﬁxed at 14 words.
We train using teacher-forcing [21], however at test time, the input to the decoder at each time-step is the token generated at the previous time-step.
Tasks
We have trained our model end-to-end on 4 different tasks: Coarse-grained classiﬁcation (onaction groups), ﬁne-grained classiﬁcation (on 174 action categories), captioning with simpliﬁed object placeholders and ﬁne-grained captioning with full object placeholders.
Labels with more subtle and ﬁne-grained distinctions expose the ability (or inability) of a network to correctly infer the scene properties encoded in the captions.
Coarse- and ﬁne-grained classiﬁcation Something-Something provides coarse-grained categories called action groups, which comprise disjoint sets of ﬁne-grained actions.
Classiﬁcation accuracy of our model is at 57 .
60% on action groups.
We use the same architecture and train it on ﬁne-grained action categories, and achieve 51 .
Captioning with simpliﬁed object placeholders We consider a captioning task in which we modify the ground truth captions to only contain one word per placeholder.
Table 1 shows an example of the process.
In the spectrum of granularity, captioning with simpliﬁed objects can be considered as a middle ground between ﬁne-grained action classiﬁcation and captioning with full labels.
Fine-grained captioning with full object placeholders
We also train networks on the full object placeholders.
This constitutes the ﬁnest level of action granularity.
Table 2 summarizes the captioning results.
We evaluate the models using standard captioning metrics: BLEU [22], ROUGE-L [23] and
METEOR [24].
The captioning models produce impressive qualitative results with a high degree of approximate action and object accuracy.
For qualitative examples of captioning and classiﬁcation, please refer to the supplementary material.
Transfer Learning to 20bn-kitchenware:
We introduce 20bn-kitchenware , a few-shot video classiﬁcation dataset that contains 390 videos ofaction categories.
This dataset contains video clips of manipulating a kitchen utensil for roughly Video ID [Table]
[Table] seconds(see Fig. For each utensil X ∈ { fork, spoon, knife, tongs } , the target label belongs to 1 of 3 actions, namely, “Using X " , “Pretending to use X " or “Trying but failing to use X " .
We also include a fall-back class of “Doing other things" .
We encourage the model to pay attention to visual details by including unused ‘negative’ objects in the scene.
5.1 Experiments
We explore transfer learning performance on 20bn-kitchenware as a function of source task granularity.
We consider two-channel models that are pre-trained on the four aforementioned tasks.
We also include a VGG16 network pre-trained on ImageNet
[Table]
and an Inﬂated-ResNet34 pre-trained on
Kinetics 1 .
For each pre-trained model, we ﬁne-tune an MLP with 512 units on top of the penultimate features from the frozen encoder, using only 10 samples per class.
We evaluate 1-shot, 5-shot and 10-shot performance, averaging scores obtained over 10 runs.
Figure 4 shows the average scores as well as 95 % conﬁdence intervals.
Our results support the contention that training on ﬁne-grained tasks leads to better features.
The best model on this benchmark is our model trained on full captions.
In all our experiments we use frame rate of 12 fps .
During training we randomly pick 48 consecutive frames.
For videos with less than 48 frames, we replicate the ﬁrst and last frames to achieve the intended length.
We resize the frames to 128 × 128 , and then use random cropping of size 96 × 96 .
For validation and testing, we use 96 × 96 center cropping.
We optimize all models using Adam, with an initial learning rate of 0 . 001 .
Conclusion
Ever since ImageNet became popular as a generic feature extractor, a hypothesis has been that the dataset size, the amount of detail and the variety of labels, drive a network’s capability to learn useful features.
This paper provides further evidence for that hypothesis, showing that task granularity has a strong inﬂuence on the quality of the learned features for transfer learning.
For the new task, given the limited amount of training data, the action granularity and the presence of negative objects, we hypothesize that only models that have some understanding of physical world properties will perform well on this dataset.
Our experiments support that ﬁne-grained tasks generally leads to better features.
6.1 Baseline models for classiﬁcation and captioning
As a classiﬁcation baseline, we use ImageNet-pretrained models on individual frames, to which we then add additional layers.
For the ﬁrst baseline, we use just the middle frame of the video, with a classiﬁer comprising a 2-layer MLP with 1024 hidden units.
We also consider a baseline in which we apply this approach to all 48 frames, after which we average the frame by frame predictions.
Lastly, we aggregate temporal information an LSTM layer with 1024 units.
We report results in
Table 5.
There is a marked improvement with the LSTM, conﬁrming that this task requires some form of temporal analysis.
The number of features for VGG16 and Resnet152 are 4096 andrespectively.
Tothebestof our knowledge there are no baselines for the Something-Something captioning task.
To quantify the performance of our captioning models, we count the percentage of generated captions that match ground truth word by word.
We refer to this as “Exact-Match Accuracy”.
This is a challenging metric as the model is deemed correct only if it generates the entire caption correctly.
If we use the action category predicted by our model trained for classiﬁcation, and replace all occurrences of [something] with the most likely object string conditioned on that action class, the
Exact-Match accuracy is 3 .
The same baseline for simpliﬁed object placeholders is 5 .
We also implemented a conventional encoder-decoder model for captioning 4.
Models
[Table]
[Table]