On the Informativeness of Supervision Signals
Abstract
Learning transferable representations by training a classifier is a well-established technique in deep learning (e.g. ImageNet pretraining), but there is a lack of theory to explain why this kind of task-specific pre-training should result in ‘good’ representations.
We conduct an information-theoretic analysis of several commonly-used supervision signals to determine how they contribute to representation learning performance and how the dynamics are affected by training parameters like the number of labels, classes, and dimensions in the training dataset.
We confirm these results empirically in a series of simulations and conduct a cost-benefit analysis to establish a tradeoff curve allowing users to optimize the cost of supervising representation learning.
Introduction
The success of modern deep learning methods can perhaps best be understood through the lens of deep representation learning.
Representation learning aims to learn latent embeddings of input stimuli.
Generally, training a neural network means learning successive layers of representations that, typically, will be used in the final layer to perform some sort of task (e.g. classification).
The key decision in implementing a representation learning framework often revolves around designing a supervision signal for the model by quantifying the similarity between stimuli.
Significant work has gone into the design of supervision signals for deep representation learning resulting in a plethora of tasks using contrastive objectives [Chen et al., 2020, Khosla et al., 2020], classification objectives [Huh et al., 2016, Ridnik et al., 2021], reconstruction objectives [Devlin et al., 2018, Kingma and Welling, 2013], and many others [Guo et al., 2019].
In addition, deep representation learning techniques provide exciting new avenues for cognitive science.
Specifically, uncovering the structure of human representations by directly training on behavioral data such as similarity judgments can provide a modern alternative to classic methods such as multi-dimensional scaling [Shepard, 1980].
Contrastive learning has been a particularly productive framework for achieving impressive results in learning useful, semantically-meaningful representations.
Most contrastive learning frameworks involve training a model on pairs of stimuli with a loss function that rewards pushing together embeddings of similar stimuli and pushing apart the embeddings of dissimilar ones.
However, we observe that something akin to contrastive learning also implicitly occurs when a model is trained with a classification objective, since embeddings from the same classes are pushed together in deeper layers while embeddings from different classes are pushed apart.
Recent work has shown that models trained on classification tasks can approximate the structure of human latent representations at a fraction of the cost of exhaustively collecting the pairwise-similarity judgments required for conventional contrastive learning [Peterson et al., 2018, Marjieh et al., 2022].
Thus, different representation learning objectives can be applied to many overlapping use-cases, and it is not clear when one objective should be preferred over another.
In this paper, we perform an information-theoretic analysis of two popular supervision signals from the classification literature – hard labels and soft labels – and quantify their relative (representational) information content by comparing them to similarity triplets (i.e., ‘Is x more similar to y than to z ?’), a popular supervision signal from contrastive learning and cognitive science literature [Jamieson and
Nowak, 2011, Hoffer and Ailon, 2015].
We relate the quantity of information each signal provides to three common features of machine learning datasets: number of labels, number of classes, and dimensionality.
We find that while both hard labels and soft labels provide information about hidden representations, the dynamics in terms of those three variables are very different between the signal types.
We conduct a series of simulations to empirically confirm these results and to determine how additional information provided by each label translates into better representation learning performance.
Finally, we conduct a cost-benefit analysis on these results to establish a tradeoff curve that allows users to optimize the cost of supervising representation learning.
Relative Informativeness of Supervision Signals
Problem definition.
We formalize representation learning as the process of recovering a hidden (low-dimensional) latent structure from a set of (high-dimensional) stimuli (e.g. images).
In particular, we focus on the non-metric setting where we want to recover the correct rank order of pairwise distances between all hidden latent vectors.
Our goal is to determine which supervision signal is most effective (in terms of both performance and cost) for representation learning.
Consider a set of stimuli { x i } ni =1 ∈ R d with some associated latent representations { z i } ni =1 ∈ R h .
The distance between each pair of latent vectors induces a relational order over latent pairs and our goal is to find a function f : R d → R h (where typically h << d ) such that it preserves this relational order, that is, || f
( x i ) − f ( x j ) || ≤ || f
( x i ) − f ( x k )
||
iff ||
z i − z j || ≤
|| z i − z k
|| .
Crucially, the latent vectors are accessible only implicitly via different supervision signals (or queries) such as hard and soft labels.
We operationalize the informativeness of different supervision signals as the number of relational constraints that a naive learner can recover based on them (i.e., a learner that attempts to follow the signals as is without applying other geometric constraints such as triangle inequalities).
Third-order constraints.
Conceptually, when training a neural network for classification, providing a label for a point roughly corresponds to requiring that the network weights should be updated such that the embedding of this point will be closer to one class than to other classes .
For this analysis, we assume that each class can be represented by its centroid (e.g., each class is unimodal), and so classification labels provide information about proximity of latent vectors to these centroids.
When training with batches, providing labels for a batch additionally corresponds to requiring that the centroid of each class be closer to its associated set of embeddings than to the other embeddings .
In both cases, neural networks are optimizing constraints of the form ‘ x is closer to y than to z ’, which we call ‘third-order constraints’.
We now formalize this concept to use it as a measure of information content in labels.
Suppose we have a system with n labels, k classes with centroids C 1 , ..., C k , and stimuli { x i } ni =1 with latent representations { z i } ni =1 .
A third-order constraint is an inequality of the form ||
z a − z b || <
||
z a − z c || .
This can be rewritten as the query r i,j,k = (cid:8) x ∈ R d : ∥ f ( x j ) − f ( x i ) ∥ <
∥ f ( x k ) − f ( x i ) ∥ (cid:9) , and each such query provides at most one bit of information [Jamieson and Nowak, 2011].
For any set of three stimuli, there are three unique queries: r i,j,k , r j,i,k , r k,i,j .
Thus, the total number of unique queries for n stimuli is 3 (cid:0) n 3 (cid:1) .
However, in the case of hard and soft labels, we make queries not only in terms of the n objects but also in terms of the k class centroids.
In other words, we are seeking to recover embeddings not only for the n points of interest, but also k additional reference embeddings.
As a result, the total number of unique queries in these cases is 3 (cid:0) n + k 3 (cid:1) .
Hard labels.
We define the hard label for stimulus x as a vector l of length k , such that l i = 1 if i =
arg min j
( || f ( x ) − f ( C j ) || ) and 0 otherwise .
There are two types of third-order queries that we can extract from hard labels.
The first is a triplet consisting of a class centroid C i , a stimulus x P that is a ‘positive’ example of this class, and a negative example stimulus x N .
The query has the form || f ( x P ) − f ( C i )
|| <
|| f ( x N ) − f
( C i ) || .
The second type is a triplet consisting of a stimulus x i , the class centroid that is closest to it ( C P ), and another class centroid further away ( C N ).
This query has the form
|| f ( C P ) − f ( x i )
|| <
|| f ( C N ) − f ( x i )
|| .
If the hard labels are distributed evenly between k classes, then on average there are n/k stimuli in each class.
Then n hard labels give us n ( k − 1)
[Table]
 
O
[Table]
n
[Table]
k
O
[Table]
n [Table] k
O
[Table]
Soft labels.
To produce a probability distribution over classes, neural networks often have a softmax activation function after the output layer [Bridle, 1989, Martins and Astudillo, 2016, Krizhevsky et al., 2017].
Accordingly, we define the soft label for a stimulus x as a vector l of length k , such that
[Equation]
||
[Equation]
x ) [Equation] f ( Cj )
|| .
There are again two types of third-order queries that we can extract from soft labels.
The first is a triplet of the form || f ( x P ) − f ( C i )
|| <
|| f ( x N ) − f
( C i )
|| where C i is the centroid of class i and x P , x N are two training set points with corresponding soft labels l P , l N such that l Pi > l Ni .
The second is a triplet consisting of the form
|| f ( x ) − f
( C i )
|| <
|| f ( x ) − f ( C j ) || where x is a training set point corresponding to label l and C i , C j are the centroids of classes i, j such that l i > l j .
Our n soft labels thus give us nk ( k − 1) / 2 constraints of the second type and kn ( n − 1) / 2 of the first.
Information ratio.
While we now have a measure of how much information each label provides, it is unclear how much information is actually needed to recover ‘good’ representations.
Intuitively, we would expect that more information is required when more objects are being embedded (i.e. when n + k increases).
We can normalize our results from the previous section to account for this by taking the ratio of the number of constraints we can recover from a set of label to the total number of possible queries.
This ‘information ratio’ may be a proxy for how much information we are recovering about the latent representations.
We present the information ratios for hard and soft labels in Table 1 along with their asymptotic behavior in three regimes: the many-shot case (where there are many more points than classes), the one-shot case (where there is one point per class; [Fei-Fei et al., 2006]), and the less-than-one-shot case (where there are fewer points than classes; [Sucholutsky and Schonlau, 2021]).
Our results predict three scaling phases for soft labels and two scaling phases for hard labels.
Experiments
Simulations.
We have posited that information ratios are a proxy for representation learning performance and have shown that if this is the case, then soft labels should lead to better performance than hard labels, particularly when there are few labels and many classes.
However, we still need to understand how information ratio actually translates to representation learning performance.
To that end, we conduct simulations to see the effect of four variables (and their interactions) on representation learning performance: label type (soft or hard), number of points ( n ), number of classes ( k ), and latent dimension ( d ).
We consider values of n and k in the range of [3 , 90] , and d ∈ { 5 , 25 , 125 } .
For each combination of n, k, d we sample a total of n points from Gaussians centered at k randomly sampled centers C 1 , ..., C k ∈ R d .
We compute hard and soft labels for these points using the equations defined above and then mine all third-order constraints of both types from both sets of labels.
We apply Generalized Nonmetric Multi-Dimensional Scaling (GNMDS; Davenport [2013]) to both sets of queries to find embeddings that best fit the respective third-order constraints.
The Gram matrix outputted by GNMDS can be interpreted as the predicted (unnormalized) pairwise cosine similarities between all n + k points and centroids.
To understand how much information we recover from each of these two sets of queries, we construct a matrix of the true pairwise cosine similarities for the set of all n + k points and class centroids and compute the Spearman rank correlation ( ρ ) between the upper triangle of the Gram matrices and the ground truth matrix.
Thus, a higher ρ corresponds to better recovery of the underlying latent representations.
We visualize the results of the simulations in Figure 1.
The results confirm the theoretical findings from the previous section.
Specifically, the three phases for soft labels and two phases for hard labels match our analytical results, and a higher information ratio translates into better representation learning performance.
Cost-benefit tradeoffs.
We can now construct cost-benefit tradeoff curves to determine when a user would prefer to use one signal over the other.
Suppose we define ρ as above, and subjective utility as U ( ρ ) .
This utility function can take many forms (e.g., U ( ρ )
= bρ or bσ ( ρ ) where σ is the sigmoid function and b > 0 , etc.).
If we assume that the cost of collecting a soft label over k classes is about k times more expensive than collecting a hard label, we can define the subjective cost function as L s = C ( s ) − U ( ρ ) , where C ( s ) =
cn if s ∈ S hard and cnk if s ∈
S soft This is equivalent to optimizing ˆ L = cb
ˆ c ( s ) − ˆ u ( ρ ) which we can re-parametrize to a form reminiscent of the information bottleneck [Tishby et al., 2000]: ˆ L = β ˆ c ( s ) − ˆ u ( ρ ) .
Since we have shown that the information ratio, which we define as ˆ ρ , can provide us with an estimate for ρ , we can also replace U ( ρ ) by U (ˆ ρ ) .
We investigate cost-benefit tradeoffs by varying β, ˆ u and visualize the results for several combinations of these two variables in Figure 1.
While the results depend greatly on choice of ˆ u , a few regularities emerge.
First, in all cases, regardless of β , when the number of classes ( k ) or the number of points ( n ) is low, soft labels are roughly as preferred as, or more preferred, than hard labels.
Second, when there is an emphasis on cost (i.e. high β ), hard labels become preferable as n and k both increase, but when the emphasis is on performance (i.e. low β ), soft labels remain preferable as n and k increase.
Conclusion
In this paper, we have provided theoretical grounding for how hidden representations can be recovered through supervised classification, and we have related the quality of these recovered representations to training parameters like number of labels, classes, and dimensions.
We find that, while hard labels and soft labels provide comparable amounts of information in the many-examples-but-few-classes regime, soft labels become increasingly preferable when the number of classes increases or the number of labels decreases.
Our findings explain why, for example, pre-training a classifier on
ImageNet1K (1,000 classes) or ImageNet21k (21,000 classes) using hard labels may lead to decent transfer learning performance
[Huh et al., 2016, Ridnik et al., 2021] but pre-training with (a form of) soft labels may lead to even better transfer learning performance [Xie et al., 2020].
Finally, we enable researchers to analyze cost-benefit tradeoffs when determining which supervision signals to collect.
A Appendix A.1 Limitations
We note that, in our analysis, we made no assumptions about the data distribution in stimulus space, nor the function f ( x i ) =
z i that maps from stimulus space to hidden representations, but when training neural networks we often assume some level of stability or invariance (i.e. a small perturbation in pixel space does not lead to drastically different perception of the image).
When satisfied, assumptions about stability or invariance, often called ‘inductive biases’, allow learners to extract additional information from training examples, sometimes even in an unsupervised way when no labels are present.
As a result, our analysis here can be considered as a sort of lower-bound on how much information about hidden representations a labeled training dataset can provide.
We also examined each supervision signal in isolation, assuming that only labels of one type are collected.
A promising future direction would be to analyze additional sources of information (inductive biases, other supervision signals, etc.)
as well as the interactions between them.