We propose a self-supervised approach for learning representations of relationships between humans and their environment, including object interactions, attributes, and body pose, entirely from unlabeled videos recorded from multiple viewpoints (Fig. 2) . We train an embedding with a triplet loss that contrasts a pair of simultaneous frames from different viewpoints with temporally adjacent and visually similar frames (Fig. 1) . We call this model TimeContrastive Networks (TCN). The contrastive signal encourages the model to discover meaningful dimensions and attributes that can explain the changing state of objects and the world from visually similar frames while learning invariance to viewpoint, occlusions, motion blur, lighting, background. The experimental evaluation of our multiviewpoint embedding technique examines its application to reasoning about object interactions, as well as human pose imitation with a real robot. We demonstrate that our model can correctly identify corresponding steps in complex object interactions, such as pouring (Table 1) , between different videos and with different instances. We also show what is, to the best of our knowledge, the first self-supervised results for end-to-end imitation learning of human motions with a real robot ( Table 2) . Results are best visualized in videos available at 1 and the full paper is available at 2 . Unsupervised Object Interactions We compare our multi-view TCN model against the Shuffle & Learn[1] approach, using the exact same architecture and only changing the loss and the last layer. Both models are initialized with ImageNet classification weights, then trained in a self-supervised manner using 15 minutes of multi-view pouring videos (no labels). We test on 5 minutes of unseen pouring videos. An off-the-shelf ImageNetpretrained Inception model is also used as a baseline (no training on pouring data). Finally, we also propose a singleview TCN to compare with. We find that TCN outperforms all baselines on different quantitative metrics ( Figure 1: Time-Contrastive Networks (TCN): 2 frames taken at the same time from different viewpoints are trained to contrast with a random or hard-negative frame from the same temporal neighborhood using a triplet loss. that multi-view outperforms the single-view model. Both metrics use the nearest neighbor of a reference frame in the embedding of each method. The alignment metric measures how well different sequences of a same demonstrations can be semantically aligned using different embeddings. The attributes classification metric measures how well different attributes that are useful to perform a pouring task are modeled by different embeddings. End-to-end Self-Supervised Pose Imitation We apply TCN to the problem of human pose imitation by a robot. With an additional self-supervision signal (Fig. 4) , we are able to produce end-to-end imitation without using any labels (Fig. 5) . The model is able to learn a complex human to robot mapping entirely self-supervised and is quantitatively better than a human-supervised imitation ( Table 2) . The combination of all signals (cheap and exact TC and self-supervision, and expensive and noisy human supervision) performs best.
REF uses a triplet loss to utilize same / not same events in video to learn view independent embeddings of human actions.
Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation
2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Computer Science