Paper:
We describe a DNN for video classification and captioning, trained end-to-end, with shared features, to solve tasks at different levels of granularity, exploring the link between granularity in a source task and the quality of learned features for transfer learning. For solving the new task domain in transfer learning, we freeze the trained encoder and fine-tune an MLP on the target domain. We train on the Something-Something dataset with over 220, 000 videos, and multiple levels of target granularity, including 50 action groups, 174 fine-grained action categories and captions. Classification and captioning with Something-Something are challenging because of the subtle differences between actions, applied to thousands of different object classes, and the diversity of captions penned by crowd actors. Our model performs better than existing classification baselines for SomethingSomething, with impressive fine-grained results. And it yields a strong baseline on the new Something-Something captioning task. Experiments reveal that training with more fine-grained tasks tends to produce better features for transfer learning.

Review:
Paper Summary - This paper presents an approach for fine-grained action recognition and video captioning. The authors train a model using both classification and captioning tasks and show that this improves performance on transfer learning tasks. The method is evaluated on the Something-Something v2 dataset as well as a new dataset (proposed in this paper). The authors also evaluate the benefit of using fine-grained action categories vs. coarse-grained action categories on transfer learning.
Paper Strengths
-  Comparing fine-grained vs. coarse-grained action categories for transfer learning is well motivated. Evaluating just this aspect in the context of video classification is helpful (Section 5.1). Establishing the baseline using linear classifiers for feature transfer makes the feature transfer result more robust. The authors have also done a good job of evaluating their method in the coarse-grained and fine-grained settings (Table 1, 2).
- The architectural and experimental design in this paper is well illustrated.
- The 20bn kitchen dataset has interesting categories about intention - pretending to use, using, and using & failing.
- The ablation in Table 1 is helpful in understanding the contribution of 3D vs. 2D convolutions.
Paper Weaknesses
- I believe this paper tries to do too much and as a result fails to show results convincingly. There are too many results and not much focus on analyzing them. In my opinion, the experimental setup in the paper is weak to fully support the authors' claims.
- I now analyze the main contributions of this paper as outlined by the authors in Section 1.
    - Label granularity and feature quality: To me this is the most interesting part of this paper and most related to its title. However, this is also the most under-analyzed aspect. The only result that the authors show is in Sec 5.1 and Fig 5. Apart from using the provided fine-grained vs. coarse-grained labels for evaluation, the authors do not perform many experiments in this domain and neither do they analyze these results. For example, the gain using fine-grained labels is not significant in Figure 5 (2Channel - CG vs. 2Channel - FG). The authors do not explain this aspect. Another missing baseline from Figure 5 is "2Channel - Captions & CG actions". This baseline is needed to understand the contribution of FG vs CG actions when also using captioning as additional supervision.
    - Baselines for captioning: The authors do not provide any details for this task. If the intent is to establish baselines there needs to be more effort on analyzing design decisions - e.g. decoding, layers in LSTM. Captioning metrics such as CIDER and SPICE are missing.
    - Captions as a source for transfer learning: This is poorly analyzed in this paper. 1) Can the captions be converted to "tags" and then used for supervision? What is the benefit of producing the full sequential text description over this simple approach? 2) Captions for transfer learning are only analyzed in Figure 5 without much explanation. It is hard to claim that captioning is the reason for performance gains without really analyzing it completely.
    - 20bn-kitchenware dataset - This dataset is explained in just one paragraph in Section 6. What is the motivation behind collecting this dataset as opposed to showing transfer learning on some other dataset?
- Missing references
        - There has been work in understanding the effect of fine-grained categories in ImageNet transfer learning - What makes ImageNet good for transfer learning? Huh et al. What is the insight provided over this work?
- Minor comments
    - Section 1: Figure 4 is referenced in points 1 & 3. I think you mean Figure 5.