The more ﬁne-grained, the better for transfer learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this paper, we investigate the correlation between the degree of detail(granularity)in the source task and the quality of the learned features for transfer learning tonew tasks.
For this purpose, we design a DNN for action classiﬁcation and videocaptioning.
The same video encoding architecture is trained to solve different taskswith different granularity levels.
In our transfer learning experiments, for solvingthe new task domain, we ﬁne-tune an MLP, while the encoder part is frozen.
Experiments reveal that training with more ﬁne-grained tasks tends to producebetter features for transfer learning We use Something-Something dataset withover 220 , 000 videos, and multiple levels of target granularity.
With impressivecoarse-grained and ﬁne-grained classiﬁcation results, our model introduces abaseline on the new Something-Something captioning task.
For supplementary material, please refer to: https
//bit ly/2xHA5EW
Introduction
Fine-grained video understanding entails recognition of actions, objects, and spatial and temporalrelations.
A successful framework needs to discriminate myriad variations of actions and interactions,not unlike the emergence of ﬁne-grained tasks in visual object recognition.
To enable extracting richfeatures from video, right kinds of tasks are needed to train the framework.
There are various levels atwhich we can describe actions, and these levels of granularity match naturally with compositionalityof language.
For example, at a coarse-grained level we have actions like ’putting a pen’ .
Then we canhave similar actions that differ in relatively subtle ways, for instance, ’putting a pen beside the cup’ ,’putting the pen in the cup’ , or perhaps ’pretending to put the pen on the table’ .
Adding prepositionsand categories like “pretending to put” gives us ﬁne-grained action.
The complexity of the task at thislevel requires features that capture spatial relations.
As the complexity of the task begins to match thecomplexity of the world that we are trying to understand, it necessitates more powerful features inorder to discriminate these different scenes.
A two-channel DNN architecture is designed for video encoding.
The same architecture is then usedfor video classiﬁcation and captioning.
Training is performed on Something-Something dataset [1],with multiple levels of granularity for the labels, namely 50 coarse-grained action groups, whichare further broken to 174 closely related action categories, and a caption which is authored by thecrowd actor.
These captions mirror the ﬁne-grained action category, but with the generic placeholder
Something replaced by the speciﬁc object(s).
Explore the link between label granularity and feature quality : We exploit 3 levels of granularity in sth-sth-v2, namely, action groups, action categories, and captions.
Experiments show thatmore ﬁne-grained labels yield richer features.2.
Baselines for captioning on sth-sth-v2 data : We note that the captioning task is new for thisdataset; the original dataset did not provide captions.3.
Captioning as a source task for transfer learning : We show that models trained for captioninglearn features that transfer better to other tasks.
To the best of our knowledge, captioning has, to date,
[Figure]
been used as a target task.
Our results suggest that captioning is a powerful source
task.4.
20bn-kitchenware : We introduce a new dataset, ostensibly for video transfer learning.
Related Work
Video-based action classiﬁcation dates back to seminal work by Laptev et al [2] with hand-tunedfeatures, while most recent approaches have focused on DNN features.
Existing methods differin the way they aggregate information through time.
Many approaches rely primarily on spatialfeatures with CNNs applied to individual frames [3].
Other approaches make use of spatio-temporalinformation [4, 5].
Video captioning have received signiﬁcant attention since the release of largescale captioning corpora, notably, Microsoft COCO [6] and MSR-VTT [7].
Captioning tasks, ifdesigned appropriately, could represent extremely detailed scene properties.
Most existing captioningarchitectures are based on an encoder-decoder framework(e.g.
The encoder is typically aconvolutional or recurrent convolutional network.
Despite the signiﬁcant attention to Video tasks,progress has lagged compared to static images, in part because of the lack of large-scale corpora.
Using web sources (e.g., YouTube) and human annotators, larger datasets have been collected inrecent years [11], [12].
More recently, crowd-sourced data have emerged, where crowd actors areasked to generate videos depicting template actions.
(e.g. Something-Something dataset [1] and
Charades [13].)
One of the most astonishing properties of neural networks is their ability to learnrepresentations that can be successfully transferred to other tasks(e.g.
One motivation forstudying ﬁne-grained video tasks is to understand and improve the potential for transfer learning onvideo domain.
Architecture
The video encoder , inspired in part by magno- and parvo-cellular pathways in visual cortex, ﬁrstprocesses the video through a spatial 2D-CNN and a spatio-temporal 3D-CNN in parallel (Fig. 2).
Our video encoder is most closely related to approaches that perform temporal reasoning via arecurrent convolutional architecture [16, 17, 18].
It is also related to TwoStream architectures [19];but our model does not explicitly use optical ﬂow, opting instead for generic 3D CNN features.
Thebasic building block of each channel is a 3 × 3 × 3 ( 3 × 3 in 2D-CNN channel) convolution ﬁlterwith batchnorm [20] and ReLU activation.
Feature vectors from two channels are concatenated andthen fed to a 2-layer bidirectional LSTM.
We average these features to get an encoding of the entirevideo, h .
This encoding is used by both the classiﬁer and the captioning decoder(See Fig.
1) Theaction classiﬁer applies a fully-connected layer to the encoder output h , followed by a softmax layer.
For training we use a cross-entropy loss over the action categories.
[Equation]
The caption decoder is a two-layer LSTM which generates captions using a softmax over thevocabulary words, conditioned on previously generated words.
The loss used for a caption is theusual negative log-probability of the word sequence:loss captioning
[Equation]
V i
d e o
E n c o d i n g (f ea t u r e s )
Convolutional
Recurrent
[Figure]
onv3 D ( 3x3x3 ) 1x2x
[Figure]
x P
[Figure]
1x2x [Figure] [Figure] where w i denotes the i th word of the caption, h is the video encoding, and θ denotes model 75 parameters.
In order to optimize speed and memory usage during training, the length of captionsgenerated by the decoder is ﬁxed at 14 words.
We train using teacher-forcing [21], however at testtime, the input to the decoder at each time-step is the token generated at the previous time-step.
Tasks
We have trained our model end-to-end on 4 different tasks: Coarse-grained classiﬁcation (on 50action groups), ﬁne-grained classiﬁcation (on 174 action categories), captioning with simpliﬁedobject placeholders and ﬁne-grained captioning with full object placeholders.
Labels with moresubtle and ﬁne-grained distinctions expose the ability (or inability) of a network to correctly infer thescene properties encoded in the captions.
Coarse- and ﬁne-grained Classiﬁcation Sth-Sth provides coarse-grained categories called actiongroups, which comprise disjoint sets of ﬁne-grained actions.
Classiﬁcation accuracy of our model isat 57 .
6% on action groups.
We use the same architecture and train it on ﬁne-grained action categories,and achieve 51 . 62 .
88 Captioning with simpliﬁed object placeholders.
We consider a captioning task in which wemodify the ground truth captions to only contain one word per placeholder.
Table 1 shows an exampleof the process.
In the spectrum of granularity, captioning with simpliﬁed objects can be considered asa middle ground between ﬁne-grained action classiﬁcation and captioning with full labels.
Fine-grained captioning with full object placeholders.
We also train networks on the full objectplaceholders.
This constitutes the ﬁnest level of action granularity.
Table 2 summarizes the captioningresults.
We evaluate the models using standard captioning metrics: BLEU [22], ROUGE-L [23] and
METEOR [24].
The captioning models produce impressive qualitative results with a high degree ofapproximate action and object accuracy.
For qualitative examples of captioning and classiﬁcation,please refer to the supplementary material.
Transfer Learning to 20bn-kitchenware:
We introduce 20bn-kitchenware , a few-shot video classiﬁcation dataset that contains 390 videos of 13action categories(see Fig. 3).
This dataset contains video clips of manipulating a kitchen utensil forroughly 4 seconds.
For each utensil X ∈ { fork, spoon, knife, tongs } , the target label belongs to 1of 3 actions, namely, “Using X " , “Pretending to use X " or “Trying but failing to use X " .
We alsoinclude a fall-back class labeled “Doing other things" .
We encourage the model to pay attention tovisual details by including unused ‘negative’ objects in the scene.
Video ID
[Table] [Table]
5.1 Experiments
We explore transfer learning performance on 20bn-kitchenware as a function of source task granularity.
We consider two-channel models that are pre-trained on the four aforementioned tasks.
Wealso include a VGG16 network pre-trained on ImageNet
[Table]
and an Inﬂated-ResNet34 pre-trained on
Kinetics 1 .
For each pre-trained model, we ﬁne-tune an MLP with 512 units on top of the penultimatefeatures from the frozen encoder, using only 10 samples per class.
We evaluate 1-shot, 5-shot and10-shot performance, averaging scores obtained over 10 runs.
Figure 4 shows the average scores aswell as 95 % conﬁdence intervals.
Our results support the contention that training on ﬁne-grainedtasks leads to better features.
The best model on this benchmark is our model trained on full captions.
In all our experiments we use frame rate of 12 fps .
During training we randomly pick 48 consecutiveframes.
For videos with less than 48 frames, we replicate the ﬁrst and last frames to achieve theintended length.
We resize the frames to 128 × 128 , and then use random cropping of size 96 × 96 .
For validation and testing, we use 96 × 96 center cropping.
We optimize all models using Adam,with an initial learning rate of 0 . 001 .
Conclusion
Ever since ImageNet became popular as a generic feature extractor, a hypothesis been that it is datasetsize, the amount of detail and the variety of labels that drive a network’s capability to learn usefulfeatures.
This paper provides further evidence for that hypothesis, showing that task granularity has astrong inﬂuence on the quality of the learned features for transfer learning.
For the new task, giventhe limited amount of training data, the action granularity and the presence of negative objects, wehypothesize that only models that have some understanding of physical world properties will performwell on this dataset.
Our experiments support the contention that training on ﬁne-grained tasks leadsto better features.