Paper:
Unsupervised Learning of View-invariant Action Representations
The recent success in human action recognition with deep learning methods mostly adopt the supervised learning paradigm, which requires significant amount of manually labeled data to achieve good performance. However, label collection is an expensive and time-consuming process. In this work, we propose an unsupervised learning framework, which exploits unlabeled data to learn video representations. Different from previous works in video representation learning, our unsupervised learning task is to predict 3D motion in multiple target views using video representation from a source view. By learning to extrapolate cross-view motions, the representation can capture view-invariant motion dynamics which is discriminative for the action. In addition, we propose a view-adversarial training method to enhance learning of view-invariant features. We demonstrate the effectiveness of the learned representations for action recognition on multiple datasets.

Review:
The paper presents a method for learning view-invariant action representations for flow. The method consists of an encoder (convolutional and bi-directional convolutional LSTMs), which learns feature representations for various views up to T=6 frames, and three additional components: (a) a cross-view decoder, which learns to predict motions for multiple views (different than the ones of the encoder), (b) a reconstruction decoder, which reconstructs the input flow, and (c) a view classifier, which predicts from which view the encoded representation comes and operates under an adversarial setting, where the encoder tries to generate view-invariant representations while the view classifier tries to distinguish their view.   The paper is well-written and easy to follow. It tackles the difficult problem of unsupervised learning of view-invariant representations for flow.   The choice of the architecture is interesting. Nevertheless, the reasons for using the view classifier branch are not clear. Given that the goal is to learn view-invariant representations, the view classifier operates against this goal. This is also in line with the small value of the weight of the decoder beta (line 200 for equation 1), which is 0.05. This shows that the view classifier barely contributes to the final loss. Moreover, the ablation study of Table 1 shows that this branch barely changes the performance (flow input/output for cross view). It would be interesting to know if the authors have an explanation for this choice.   The method seems clean but complicated to implement and run. It would be interesting to see some run times, especially to examine the overload of the view classifier.   The ablation study of Table 1 is performed using the Lxview. It would be beneficial to see the total loss of equation 1 (with and w/o some components).  Moreover, it would be interesting to see how the evolution of the classification accuracy also by removing the reconstruction decoder in Table 2.    In the whole experimental section the authors use one modality (RGB, depth or flow) as input and output. It would be interesting to see if the combination of the modalities could also help, at least as an ablation study (Table 1) or even in Tables 2 and 3.   In the ablation study of Table 1, the authors evaluate the loss with RGB, depth and flow inputs for flow outputs. The cross-view decoder, however, takes as additional input a depth map for a view. Therefore, the columns of the table should display that the input is not one single modality but it includes depth in all cases. The same holds for Tables 2 and 3.   For the cross-view decoder, in line 152, the authors state that they deploy multiple view decoders. Is the number of these decoders fixed, eg. 5 (line 207) or do the authors allow the network to learn various views even if they do not exist at test time?   For action recognition, training from scratch (encoder + action classifier) performs significantly worse than all other cases, where the encoder is learnt using the proposed unsupervised setting. I believe that here another baseline is missing. The authors could have used pre-trained models on  ImageNet and/or UCF-101 and/or Kinetics to initialize the encoder and then fine-tune it. It would have been a stronger baseline.   In the action recognition Section 3.2, the architecture of the action classifier does not seem complete. The authors use only one fully-connected layer for as a classifier. It would be interesting to understand this choice and also to see some results with something more complete, eg. at least another one fully-connected layer.   It would be great if the authors could explain how they ended up in the values for the weights a=0.5 and beta=0.05 (lines 200) for equation (1). Did they use cross-validation?   The results in Figure 2 look nice. It would be interesting to see also some failure cases and some discussion for them.   Some choices of architectural parameters are not really justified. For instance, it is not clear how the authors chose k=64 (line 128), T=6 (line 135).   In Table 4, for the Northwestern-UCLA Multiview Action 3D dataset recent works typically report results in all split combinations (training on two splits and evaluating on the third one) and also include their average.  In Table 3, for NTU RGB+D the average is also missing.   The authors have missed some state of the art citations in Tables 3, 4, and 5. For instance: [42],  - Liu, Mengyuan, Hong Liu, and Chen Chen. "Enhanced skeleton visualization for view invariant human action recognition." In Pattern Recognition 2017 - Baradel, Fabien, Christian Wolf, and Julien Mille. "Human action recognition: Pose-based attention draws focus to hands."  In ICCV Workshop on Hands in Action 2017. - Baradel, Fabien, Christian Wolf, Julien Mille, and Graham W. Taylor. "Glimpse clouds: Human activity recognition from unstructured feature points." In CVPR 2018. - Rahmani, Hossein, and Ajmal Mian. "Learning a non-linear knowledge transfer model for cross-view action recognition." In CVPR 2015. - Gupta, Ankur, Julieta Martinez, James J. Little, and Robert J. Woodham. "3D pose from motion for cross-view action recognition via non-linear circulant temporal encoding." In CVPR 2014.  --after feedback The authors addressed many of the issues raised in the reviews. I am still not convinced about the Lcls: even though its value is sometimes higher than the other losses (without any explanation), using it does not seem so crucial (Table 1 and Table 2). However, the idea of casting the problem as a view-adversarial training is indeed interesting. Including results with pre-training on ImgNet and potentially on other datasets (eg. Kinetics) is important and the rebuttal shows that pre-training on ImgNet is inferior to fix. The authors should also include all related works and results.