  State-of-the-art methods for self-supervised learning (SSL) build
representations by maximizing the similarity between different augmented
"views" of a sample. Because these approaches try to match views of the same
sample, they can be too myopic and fail to produce meaningful results when
augmentations are not sufficiently rich. This motivates the use of the dataset
itself to find similar, yet distinct, samples to serve as views for one
another. In this paper, we introduce Mine Your Own vieW (MYOW), a new approach
for building across-sample prediction into SSL. The idea behind our approach is
to actively mine views, finding samples that are close in the representation
space of the network, and then predict, from one sample's latent
representation, the representation of a nearby sample. In addition to showing
the promise of MYOW on standard datasets used in computer vision, we highlight
the power of this idea in a novel application in neuroscience where rich
augmentations are not already established. When applied to neural datasets,
MYOW outperforms other self-supervised approaches in all examples (in some
cases by more than 10%), and surpasses the supervised baseline for most
datasets. By learning to predict the latent representation of similar samples,
we show that it is possible to learn good representations in new domains where
augmentations are still limited.
