The more ﬁne-grained, the better for transfer learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this paper, we investigate the correlation between the degree of detail(granularity)in the source task and the quality of the learned features for transfer learning tonew tasks.
For this purpose, we design a DNN for action classiﬁcation and videocaptioning.
The same video encoding architecture is trained to solve differenttasks with different granularity levels.
In our transfer learning experiments, forsolving the new task domain, we ﬁne-tune an MLP, while the encoder part isfrozen.
Experiments reveal that training with more ﬁne-grained tasks tends toproduce better features for transfer learning. We use Something-Something datasetwith over 220 , 000 videos, and multiple levels of granularity of the target labels.
With impressive coarse-grained and ﬁne-grained classiﬁcation results, our modelintroduces a baseline on the new Something-Something captioning task.
For supplementary material, please refer to: https://bit.ly/ 2xHA5EW
Introduction
Fine-grained video understanding entails recognition of actions, objects, and spatiotemporal relations.
A successful framework needs to discriminate myriad variations of actions and interactions, notunlike the emergence of ﬁne-grained tasks in visual object recognition.
To enable extracting richfeatures from video, right kinds of tasks are needed to train the framework.
There are various levels atwhich we can describe actions, and these levels of granularity match naturally with compositionalityof language.
For example, at a coarse-grained level we have actions like ’putting a pen’ .
Then we canhave similar actions that differ in relatively subtle ways, for instance, ’putting a pen beside the cup’ ,’putting the pen in the cup’ , or perhaps ’pretending to put the pen in the cup’ .
Adding prepositionsand categories like “pretending to put” gives us ﬁne-grained action.
The complexity of the task at thislevel requires features that capture spatial relations.
As the complexity of the task begins to match thecomplexity of the world that we are trying to understand, it necessitates more powerful features inorder to discriminate these different scenes.
A two-channel DNN architecture is designed for video encoding.
The same architecture is thenused for video classiﬁcation and captioning.
Training is performed on Something-Something dataset[1], with 50 coarse-grained action groups, which are further broken to 174 closely related actioncategories, and a caption authored by the crowd actor.
These captions mirror the ﬁne-grained actioncategory, but with placeholder Something replaced by the speciﬁc object(s).
The main contributionsof this paper include:
Explore the link between label granularity and feature quality : We exploit 3 levels of granularity in Something-Something, namely, action groups, action categories, and captions.
Experimentsshow that more ﬁne-grained labels yield richer features.2.
Baselines for captioning on Something-Something data : We note that the captioning task isnew for this dataset; the original version did not provide captions.3.
Captioning as a source task for transfer learning : We show that models trained for captioning learn features that transfer better to other tasks.
To the best of our knowledge, captioning has, to date,
[Figure]
been used as a target task.
Our results suggest that captioning is a powerful source
task.4.
20bn-kitchenware : We introduce a new dataset, ostensibly for video transfer learning.
Related Work
Video-based action classiﬁcation dates back to seminal work by Laptev et al [2] with hand-tunedfeatures, while most recent approaches have focused on DNN features.
Existing methods differ in theway they aggregate information through time.
Many approaches rely primarily on spatial features with CNNs applied to individual frames [3].
Other approaches make use of spatiotemporal information[4, 5].
Video captioning have received signiﬁcant attention since the release of large-scale captioningcorpora, notably, Microsoft COCO [6] and MSR-VTT [7].
Captioning tasks, if designed appropriately,could represent extremely detailed scene properties.
Most existing captioning architectures are basedon an encoder-decoder framework [8, 9, 10].
The encoder is typically a convolutional or recurrentconvolutional network.
Despite the signiﬁcant attention to Video tasks, progress has lagged comparedto static images, in part because of the lack of large-scale corpora.
Using web sources and humanannotators, larger datasets have been collected in recent years [11, 12].
More recently, crowd-sourceddata have emerged, where crowd actors are asked to generate videos depicting template actions [1, 13].
One of the most astonishing properties of neural networks is their ability to learn representations thatcan be successfully transferred to other tasks[14, 15].
One motivation for studying ﬁne-grained videotasks is to understand and improve the potential for transfer learning on video domain.
Architecture
The video encoder , inspired in part by magno- and parvo-cellular pathways in visual cortex, ﬁrstprocesses the video through a spatial 2D-CNN and a spatio-temporal 3D-CNN in parallel (Fig. 2).
Our video encoder is most closely related to approaches that perform temporal reasoning via arecurrent convolutional architecture [16, 17, 18].
It is also related to TwoStream architectures [19];but our model does not explicitly use optical ﬂow, opting instead for generic 3D CNN features.
Thebasic building block of each channel is a 3 × 3 × 3 ( 3 × 3 in 2D-CNN channel) convolution ﬁlter withbatchnorm [20] and ReLU activation.
Feature vectors from two channels are concatenated and thenfed to a 2-layer bidirectional LSTM.
We average these features to get an encoding of the entire video,h .
This encoding is used by both the classiﬁer and the captioning decoder(See Fig. 1).
The actionclassiﬁer applies an FC layer to the encoder output h , followed by a softmax layer.
For training weuse a cross-entropy loss over the action categories.
[Equation]
The caption decoder is a two-layer LSTM which generates captions using a softmax over thevocabulary words, conditioned on previously generated words.
The loss used for a caption is theusual negative log-probability of the word sequence:
[Equation]
V i
d e o
E n c o d i n g (f ea t u r e s )
Convolutional
Recurrent
[Figure]
ea t u
[Figure]
x48x48x
T x24x24x
[Figure]
onv3 D ( 3x3x3 ) 1x2x
M a x P oo li ng
[Figure]
onv
[Figure]
1x2x
M a x P oo li ng
T x12x12x
[Figure]
1x2x
[Figure]
[Figure]
[Figure]
) 1x2x
[Figure]
1x2x
[Figure]
x1x1x where w i denotes the i th word of the caption, h is the video encoding, and θ denotes model 73 parameters.
In order to optimize speed and memory usage during training, the length of captionsgenerated by the decoder is ﬁxed at 14 words.
We train using teacher-forcing [21], however at testtime, the input to the decoder at each time-step is the token generated at the previous time-step.
Tasks
We have trained our model end-to-end on 4 different tasks: Coarse-grained classiﬁcation (on 50action groups), ﬁne-grained classiﬁcation (on 174 action categories), captioning with simpliﬁedobject placeholders and ﬁne-grained captioning with full object placeholders.
Labels with moresubtle and ﬁne-grained distinctions expose the ability (or inability) of a network to correctly infer thescene properties encoded in the captions.
Coarse- and ﬁne-grained classiﬁcation Something-Something provides coarse-grained categoriescalled action groups, which comprise disjoint sets of ﬁne-grained actions.
Classiﬁcation accuracy ofour model is at 57 .
60% on action groups.
We use the same architecture and train it on ﬁne-grainedaction categories, and achieve 51 .
Captioning with simpliﬁed object placeholders We consider a captioning task in which wemodify the ground truth captions to only contain one word per placeholder.
Table 1 shows an exampleof the process.
In the spectrum of granularity, captioning with simpliﬁed objects can be considered asa middle ground between ﬁne-grained action classiﬁcation and captioning with full labels.
Fine-grained captioning with full object placeholders
We also train networks on the full objectplaceholders.
This constitutes the ﬁnest level of action granularity.
Table 2 summarizes the captioningresults.
We evaluate the models using standard captioning metrics: BLEU [22], ROUGE-L [23] and
METEOR [24].
The captioning models produce impressive qualitative results with a high degree ofapproximate action and object accuracy.
For qualitative examples of captioning and classiﬁcation,please refer to the supplementary material.
Transfer Learning to 20bn-kitchenware:
We introduce 20bn-kitchenware , a few-shot video classiﬁcation dataset that contains 390 videos of 13action categories.
This dataset contains video clips of manipulating a kitchen utensil for roughly 4seconds(see Fig. 3).
For each utensil X ∈ { fork, spoon, knife, tongs } , the target label belongs to1 of 3 actions, namely, “Using X " , “Pretending to use X " or “Trying but failing to use X " .
We alsoinclude a fall-back class of “Doing other things" .
We encourage the model to pay attention to visualdetails by including unused ‘negative’ objects in the scene.
Video ID
[Table] [Table]
5.1 Experiments
We explore transfer learning performance on 20bn-kitchenware as a function of source task granularity.
We consider two-channel models that are pre-trained on the four aforementioned tasks.
Wealso include a VGG16 network pre-trained on ImageNet
[Table]
and an Inﬂated-ResNet34 pre-trained on
Kinetics 1 .
For each pre-trained model, we ﬁne-tune an MLP with 512 units on top of the penultimatefeatures from the frozen encoder, using only 10 samples per class.
We evaluate 1-shot, 5-shot and10-shot performance, averaging scores obtained over 10 runs.
Figure 4 shows the average scores aswell as 95 % conﬁdence intervals.
Our results support the contention that training on ﬁne-grainedtasks leads to better features.
The best model on this benchmark is our model trained on full captions.
In all our experiments we use frame rate of 12 fps .
During training we randomly pick 48 consecutiveframes.
For videos with less than 48 frames, we replicate the ﬁrst and last frames to achieve theintended length.
We resize the frames to 128 × 128 , and then use random cropping of size 96 × 96 .
For validation and testing, we use 96 × 96 center cropping.
We optimize all models using Adam,with an initial learning rate of 0 . 001 .
Conclusion
Ever since ImageNet became popular as a generic feature extractor, a hypothesis has been that thedataset size, the amount of detail and the variety of labels, drive a network’s capability to learn usefulfeatures.
This paper provides further evidence for that hypothesis, showing that task granularity has astrong inﬂuence on the quality of the learned features for transfer learning.
For the new task, giventhe limited amount of training data, the action granularity and the presence of negative objects, wehypothesize that only models that have some understanding of physical world properties will performwell on this dataset.
Our experiments support that ﬁne-grained tasks generally leads to better features.