  In supervised learning, obtaining a large set of fully-labeled training data
is expensive. We show that we do not always need full label information on
every single training example to train a competent classifier. Specifically,
inspired by the principle of sufficiency in statistics, we present a statistic
(a summary) of the fully-labeled training set that captures almost all the
relevant information for classification but at the same time is easier to
obtain directly. We call this statistic "sufficiently-labeled data" and prove
its sufficiency and efficiency for finding the optimal hidden representations,
on which competent classifier heads can be trained using as few as a single
randomly-chosen fully-labeled example per class. Sufficiently-labeled data can
be obtained from annotators directly without collecting the fully-labeled data
first. And we prove that it is easier to directly obtain sufficiently-labeled
data than obtaining fully-labeled data. Furthermore, sufficiently-labeled data
naturally preserves user privacy by storing relative, instead of absolute,
information. Extensive experimental results are provided to support our theory.
