Understanding Self-supervised Learning with Dual Deep Networks
Authors: Yuandong Tian, Lantao Yu, Xinlei Chen, Surya Ganguli
Keywords: self-supervised learning, teacher-student setting, theoretical analysis, hierarchical models, representation learning
Abstract: We propose a novel theoretical framework to understand self-supervised learning methods that employ dual pairs of deep ReLU networks (e.g., SimCLR, BYOL). First, we prove that in each SGD update of SimCLR, the weights at each layer are updated by a \emph{covariance operator} that specifically amplifies initial random selectivities that vary across data samples but survive averages over data augmentations. We show this leads to the emergence of hierarchical features, if the input data are generated from a hierarchical latent tree model. With the same framework, we also show analytically that in BYOL, the combination of BatchNorm and a predictor network creates an implicit contrastive term, acting as an approximate covariance operator. Additionally, for linear architectures we derive exact solutions for BYOL that provide conceptual insights into how BYOL can learn useful non-collapsed representations without any contrastive terms that separate negative pairs.  Extensive ablation studies justify our theoretical findings.