  We focus on contrastive methods for self-supervised video representation
learning. A common paradigm in contrastive learning is to construct positive
pairs by sampling different data views for the same instance, with different
data instances as negatives. These methods implicitly assume a set of
representational invariances to the view selection mechanism (eg, sampling
frames with temporal shifts), which may lead to poor performance on downstream
tasks which violate these invariances (fine-grained video action recognition
that would benefit from temporal information). To overcome this limitation, we
propose an 'augmentation aware' contrastive learning framework, where we
explicitly provide a sequence of augmentation parameterisations (such as the
values of the time shifts used to create data views) as composable augmentation
encodings (CATE) to our model when projecting the video representations for
contrastive learning. We show that representations learned by our method encode
valuable information about specified spatial or temporal augmentation, and in
doing so also achieve state-of-the-art performance on a number of video
benchmarks.
