  Contrastive approaches to self-supervised learning (SSL) learn
representations by minimizing the distance between two augmented views of the
same data point (positive pairs) and maximizing the same from different data
points (negative pairs). However, recent approaches like BYOL and SimSiam, show
remarkable performance {\it without} negative pairs, raising a fundamental
theoretical question: how can SSL with only positive pairs avoid
representational collapse? We study the nonlinear learning dynamics of
non-contrastive SSL in simple linear networks. Our analysis yields conceptual
insights into how non-contrastive SSL methods learn, how they avoid
representational collapse, and how multiple factors, like predictor networks,
stop-gradients, exponential moving averages, and weight decay all come into
play. Our simple theory recapitulates the results of real-world ablation
studies in both STL-10 and ImageNet. Furthermore, motivated by our theory we
propose a novel approach that \emph{directly} sets the predictor based on the
statistics of its inputs. In the case of linear predictors, our approach
outperforms gradient training of the predictor by $5\%$ and on ImageNet it
performs comparably with more complex two-layer non-linear predictors that
employ BatchNorm. Code is released in
https://github.com/facebookresearch/luckmatters/tree/master/ssl.
