Contrastive approaches to self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing the same from different data points (negative pairs) . However, recent approaches like  BYOL and SimSiam , show remarkable performance {\it without} negative pairs, raising a fundamental theoretical question : how can SSL with only positive pairs avoid representational collapse ? We study the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our analysis yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Furthermore, motivated by our theory we propose a novel approach  that directly sets the  predictor based on the statistics of its inputs . In the case of linear predictors, our approach outperforms gradient trainingof the predictor by 5 \\%DIF < and on ImageNet it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm. Code is released in URL\end{abstract} %DIF > in 300-epoch training (and 5\\% in 60-epoch). DirectPred is motivated by our theoretical study of the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our study yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Code is URL
While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data points (negative pairs) , recentnon-contrastive SSL (e.g., BYOL and SimSiam ) show remarkable performance {\it without} negative pairs, with an extra learnable predictor and a stop-gradient operation. A fundamental question arises: why do these methods not collapse into trivial representations? We answer this question via a simple theoretical study and propose a novel approach , DirectPred, that directly sets the linear predictor based on the statistics of its inputs , without gradient training. On ImageNet, it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm and outperforms a linear predictor by 2.5 \\%DIF < and on ImageNet it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm. Code is released in URL\end{abstract} %DIF > in 300-epoch training (and 5\\% in 60-epoch). DirectPred is motivated by our theoretical study of the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our study yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Code is URL
R
Contrastive approaches to
While contrastive approaches of
R
the same
views
R
. However, recent approaches like
, recent
A
non-contrastive
A
SSL (e.g.,
R
,
)
R
raising a fundamental theoretical question : how can SSL with only positive pairs avoid representational collapse ? We study the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our analysis yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Furthermore, motivated by our theory we
with an extra learnable predictor and a stop-gradient operation. A fundamental question arises: why do these methods not collapse into trivial representations? We answer this question via a simple theoretical study and
A
, DirectPred,
A
linear
R
. In the case of linear predictors, our approach outperforms gradient trainingof the predictor by 5
, without gradient training. On ImageNet, it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm and outperforms a linear predictor by 2.5