Pretext-based self-supervised learning aims to learn the semantic representation via a handcrafted pretext task over unlabeled data and then use the learned representation for downstream prediction tasks. \mbox{%DIFAUXCMD lee2020predicting that pretext-based self-supervised learning can effectively reduce the sample complexity of downstream tasks under Conditional Independence (CI) between the components of the pretext task conditional on the downstream label. However, the CI condition rarely holds in practice, and the downstream sample complexity will get much worse if the CI condition does not hold. In this paper, we explore the idea of applying a learnable function to the input to make the CI condition hold . In particular, we first rigorously formulate the criteria that the function needs to satisfy. We then design an ingenious loss function for learningsuch a function and prove that the function minimizing the proposed loss satisfies the above criteria. We theoretically study the number of labeled data required, and give a model-free lower bound showing that taking limited downstream data will hurt the performance of self-supervised learning. Furthermore, we take the model structure into account and give a model-dependent lower bound, which gets higher when the model capacity gets larger . Moreover, we conduct several numerical experiments to verify our theoretical results.
Pretext-based self-supervised learning aims to learn the semantic representation via a handcrafted pretext task over unlabeled data and then use the learned representation for downstream prediction tasks. It is proved that pretext-based self-supervised learning can effectively reduce the sample complexity of downstream tasks under Conditional Independence (CI) between the components of the pretext task conditional on the downstream label. However, the downstream sample complexity will get much worse if the CI condition does not hold. One interesting question is whether we can make the CI condition hold by using downstream data to refine the unlabeled data to boost self-supervised learning. At first glance, one might think that seeing downstream data in advance would always boost the downstream performance. However, we show that it is not intuitively true and point out that in some cases, it will hurt the final performance instead. In particular, we prove both model-free and model-dependent lower bounds of the number of downstream samples used for data refinement . Moreover, we conduct several experiments on both synthetic and real-world datasets to verify our theoretical results.
R
\mbox{%DIFAUXCMD lee2020predicting
It is proved
D
CI condition rarely holds in practice, and the
R
In this paper, we explore the idea of applying a learnable function to the input to
One interesting question is whether we can
R
. In particular, we first rigorously formulate the criteria that the function needs to satisfy. We then design an ingenious loss function for learningsuch a function and prove that the function minimizing the proposed loss satisfies the above criteria. We theoretically study the number of labeled data required, and give a model-free lower bound showing that taking limited downstream data
by using downstream data to refine the unlabeled data to boost self-supervised learning. At first glance, one might think that seeing downstream data in advance would always boost the downstream performance. However, we show that it is not intuitively true and point out that in some cases, it
R
performance of self-supervised learning. Furthermore, we take the model structure into account and give a
final performance instead. In particular, we prove both model-free and
R
bound, which gets higher when the model capacity gets larger
bounds of the number of downstream samples used for data refinement
R
numerical experiments
experiments on both synthetic and real-world datasets