Paper:
Trajectory Convolution for Action Recognition
How to leverage the temporal dimension is one major question in video analysis. Recent works [47,36] suggest an efficient approach to video feature learning, i.e., factorizing 3D convolutions into separate components respectively for spatial and temporal convolutions. The temporal convolution, however, comes with an implicit assumption -the feature maps across time steps are well aligned so that the features at the same locations can be aggregated. This assumption can be overly strong in practical applications, especially in action recognition where the motion serves as a crucial cue. In this work, we propose a new CNN architecture TrajectoryNet, which incorporates trajectory convolution, a new operation for integrating features along the temporal dimension, to replace the existing temporal convolution. This operation explicitly takes into account the changes in contents caused by deformation or motion, allowing the visual features to be aggregated along the the motion paths, trajectories. On two large-scale action recognition datasets, Something-Something V1 and Kinetics, the proposed network architecture achieves notable improvement over strong baselines.

Review:
This work proposes a novel convolutional operation and corresponding network architecture for action recognition in videos. Starting from 3D convolutions, the authors introduce a trajectory convolution approach where the convolutional aggregation is performed in a dynamic manner through tracked points, rather than fixed locations. Experiments and ablation studies show the effect of the primary components of the new network, as well as decent overall performance on two large-scale action datasets.  Paper strengths: - The first strength of the paper is that it is well written. The main motivation of the paper is presented in a clear and intuitive manner. Where current 3D convolutional operators perform an aggregation within a fixed cube, this work posits that the temporal aggregation should be dynamic by following points over time. This way, it should be possible to aggregate features based on the primary motion of interest.  Furthermore, from a presentation point of view, the embedding of the proposed trajectory convolution in recent convolutional literature adds to the clarity and insight of the trajectory convolution itself. The embeddings is also unclear at several instance, as discussed later. Positive embedding discussions include: -- The difference and novelty with respect to the Trajectory-Pooled Deep-Convolutional Descriptor (TDD) is clear and fair. -- A link is made with deformable convolutions, which helps the authors with the implementation of the trajectory convolution. -- A link is also made with separable convolutions. This makes sense in isolation, although it is unclear how it all combines with deformable convolutions.  - The second strength of the paper is that care is taken to perform the necessary ablation studies. The trajectory convolution is evaluated, as well as the fusion with motion and the learning trajectory component. This paints a fair picture of the method and where it can best be employed. Such ablation studies are a necessity to gain insight into the method and to understand how it works.  Paper weaknesses: - The first weakness of the paper is that the paper is unclear in its method and missing details. -- While Section 3.1 is clear, the rest of Section 3 is unclear at several occasions. In Section 3.2, a link is made with deformable convolutions with the note that the trajectory convolution can be implemented in this manner (lines 152-153). Section 3.4 in turn discusses the implementation by extending separable 3D-convolutional networks. Does this mean that the proposed trajectory convolution combines both notions? If so, these subsections should be combined and rewritten to better describe how the proposed approach incorporates both. If not, the distinction should be made more clear. -- Section 3.3 discusses the addition of motion information by adding offset coordinates as features. This is also done e.g. in Dense Trajectories themselves and in TDD. From Section 3.3, it is however unclear how this is performed in this context. To what are these offsets added and where in the network are they used? The explanation on lines 159-160 is not intuitive. A figure would also really help here. -- Section 3.5 is not clear at all. The method jumps from using a small 3D CNN to estimate estimate trajectories, to preliminary experiments, to MotionNet, to FlowNet-SD, to replacing optical flow. The same subsection discusses multiple topics in half a page without proper explanation and details, making it unnecessarily hard to read. Please rewrite this subsection and again consider a figure to make things clear to the reader. -- On a general note, it remains unclear from the paper which elements form a trajectory. I.e. which points are tracked, how densely this is done, etc. Without these details, the paper remains abstract and hard to interpret.  - The second weakness of the paper is that the experiments are not fully convincing in three aspects. -- The first aspect is that the proposed trajectory convolution does not add much to the current 3D convolutional network setup. While large improvements are not always a requirement, it is difficult to assess whether the improvements in Tables 1 through 3 are even due to the method, or due to randomness within the networks. Rerunning the baselines and methods multiple times with different seeds followed by significance tests seem like a requirement for these ablation studies given the small improvements. On face value, the current improvements are small.  -- The second aspect is that it remains unknown why the method works and when it fails. Next to the missing details on the trajectories themselves, their impact also remains abstract and unknown. Figure 2 provides a qualitative result, but does not really add to the understanding of the trajectory convolution itself. What do the trajectories focus on? Where and when are the trajectories used within a video? For which actions do they work and why? For which actions to they not work? How different are the learned trajectories from straight lines through the temporal axis? The lack of such analyses limit the impact and insight into the proposed network.  The listed strengths and weaknesses of the paper paint an overall interesting work. The trajectory convolution is a natural extension from the current literature on action recognition. Furthermore, the paper is clearly written. The paper does however have several weaknesses that require attention and warrant discussion. The method itself is not well explained and requires thorough rewriting, while the experiments are not fully convincing due to lack of effect. It should however be noted that the method itself makes sense within the current action recognition literature and the lack of big improvements should not be a direct cause for rejection, as it might still be an interesting result for researchers investigating similar avenues.