E VALUATING REPRESENTATIONS BY THE COMPLEXITY OF LEARNING LOW - LOSS PREDICTORS
A BSTRACT
We consider the problem of evaluating representations of data for use in solving a downstream task.
We propose to measure the quality of a representation by the complexity of learning a predictor on top of the representation that achieves low loss on a task of interest.
To this end, we introduce two measures: surplus description length (SDL) and ε sample complexity ( ε SC).
To compare our methods to prior work, we also present a framework based on plotting the validation loss versus evaluation dataset size (the “loss-data” curve).
Existing measures, such as mutual information and minimum description length, correspond to slices and integrals along the data axis of the loss-data curve, while ours correspond to slices and integrals along the loss axis.
This analysis shows that prior methods measure properties of an evaluation dataset of a speciﬁed size, whereas our methods measure properties of a predictor with a speciﬁed loss.
We conclude with experiments on real data to compare the behavior of these methods over datasets of varying size.
I NTRODUCTION
One of the ﬁrst steps in building a machine learning system is selecting a representation of data.
Huge strides in unsupervised learning (Hénaff et al., 2020; Chen et al., 2020; He et al., 2019; van den Oord et al., 2018; Bachman et al., 2019; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2019; Brown et al., 2020) have led to common wisdom now recommending that the design of most systems start from a pretrained representation rather than learning representations from scratch in an end-to-end manner.
With this boom in representation learning techniques practitioners and representation researchers alike have the question: Which representation is best for my task?
We take the position that the best representation is the one which allows for learning a predictor to solve the task using the least label information.
This quality is measured on an evaluation dataset which may be distinct from the set of downstream applications.
This position is motivated by practical concerns; the more labels that are needed to solve a new task, the more expensive to use and the less widely applicable a representation will be.
Recent works (Voita & Titov, 2020) have proposed to use an estimate of the minimum description length (MDL) of a dataset’s labels given a representation as a measure of representation quality.
However, this code length perspective conﬂates the information required for learning a model with the irreducible uncertainty in the data.
We show that this leads MDL-based measures to choose different representations when given evaluation datasets of different sizes, potentially leading to a premature decision about which representation to use.
We propose a new measure of representation quality called surplus description length (SDL), which modiﬁes the MDL to measure the complexity of learning an ε -loss predictor rather than computing the complexity of the labels in the evaluation dataset.
SDL, along with a second measure called ε -sample complexity ( ε SC), provide tools for researchers and practitioners to evaluate the extent to which a learned representation can improve data efﬁciency.
Furthermore, they formalize existing research challenges for learning representations which allow state of the art performance while using as few labels as possible (e.g. Hénaff et al.
To facilitate our analysis, we also propose a
[Figure]
framework called the loss-data framework , illustrated in Figure 1, that plots the validation loss against the evaluation dataset size (Talmor et al., 2019; Yogatama et al., 2019; Voita & Titov, 2020).
T HE LOSS - DATA FRAMEWORK FOR REPRESENTATION EVALUATION
Notation.
We use bold letters to denote random variables.
A supervised learning problem is deﬁned by a joint distribution D over observations and labels ( X , Y ) in the sample space X × Y with density denoted by p .
Let the random variable D n be a sample of n i.i.d.
( X , Y ) pairs, realized by
D n = ( X n , Y n ) = { ( x i , y i ) } ni =1 .
This is the evaluation dataset.
Let R denote a representation space and φ : X → R a representation function.
The methods we consider all use parametric probes, which are neural networks ˆ p θ : R → P ( Y ) parameterized by θ ∈ R d that are trained on D n to estimate the conditional distribution p ( y | x ) .
We often abstract away the details of learning the probe by simply referring to an algorithm A which returns a predictor: ˆ p =
A ( φ ( D n )) .
Abusing notation, we denote the composition of A with φ by A φ .
Deﬁne the population loss and the expected population loss for ˆ p =
A φ ( D n ) , respectively as
[Equation]
The representation evaluation problem.
The representation evaluation problem asks us to deﬁne a real-valued measurement of the quality of a representation φ for solving solving the task deﬁned by ( X , Y ) .
Explicitly, each method deﬁnes a real-valued function m ( φ, D , A , Ψ) of a representation φ , data distribution D , probing algorithm A , and some method-speciﬁc set of hyperparameters Ψ .
By convention, smaller values of the measure m correspond to better representations.
Deﬁning such a measurement allows us to compare different representations.
2. D EFINING THE LOSS - DATA FRAMEWORK
The loss-data framework is a lens through which we contrast different measures of representation quality.
The key idea, demonstrated in Figure 1, is to plot the loss L ( A φ , n ) against the evaluation dataset size n .
Explicitly, at each n , we train a probing algorithm
A using a representation φ to produce a predictor ˆ p , and then plot the loss of ˆ p against n .
We can represent prior measures as points on the curve at ﬁxed x (validation accuracy with a small dataset, VA) or integrals of the curve along the x -axis (MDL). Our measures correspond to evaluating points or integrals on the y -axis
(SDL and ε SC). 2. E XISTING METHODS IN THE LOSS - DATA FRAMEWORK
Validation accuracy and mutual information Two popular methods for evaluating representation quality are the validation accuracy of a model trained on limited data and the mutual information between the labels and a representation.
As we describe in Appendix D, these measures are intimately linked.
In the loss-data framework, we formally deﬁne the validation accuracy and mutual information measures respectively as
[Equation]
Minimum description length.
Recent studies (Yogatama et al., 2019; Voita & Titov, 2020) propose using the Minimum Description Length (MDL) principle (Rissanen, 1978; Grünwald, 2004) to evaluate representations.
These works use an online or prequential code (Blier & Ollivier, 2018) to encode the labels given the representations.
The codelength (cid:96) of Y n given φ ( X n ) is then deﬁned as
[Equation]
where ˆ p i is the output of running a pre-speciﬁed algorithm A on the evaluation dataset up to element i : ˆ p i =
A φ ( X n 1: i , Y n 1: i ) .
This measure can exhibit large variance on small evaluation datasets, especially since it is sensitive to the (random) order in which the examples are presented.
We remove this variance by taking an expectation over the sampled evaluation datasets for each i and deﬁne a population variant of the MDL measure (Voita & Titov, 2020) as
[Equation]
Thus, m MDL measures the area under the loss-data curve on the interval x ∈ [0 , n ] .
S URPLUS DESCRIPTION LENGTH (SDL)
The prequential code for estimating MDL computes the description length of the labels given observations in an evaluation dataset by iteratively creating tighter approximations ˆ p 1 . . .
ˆ p n and integrating the area under the curve.
Examining Equation (4), we see that m MDL ( φ, D , A , n ) ≥ (cid:80) n i =1 H
( Y | φ ( X )) .
If H ( Y | φ ( X )) > 0 , MDL grows without bound as the size of the evaluation dataset n increases.
This poses a problem for representation evaluation, since simply evaluating the same representations on the same task, but using an evaluation dataset of a different size, may change the verdict about which representation is better.
We call this issue, which VA shares, sensitivity to evaluation dataset size .
To derive an improved measure we look to information theory.
Imagine trying to efﬁciently encode a large number of samples of a random variable e which takes values in { 1 . . .
K } with probability p ( e ) .
An optimal code for these events has expected length 1 E [ (cid:96) ( e )]
=
E e [ − log p ( e )]
= H ( e ) .
If this data is instead encoded using a probability distribution ˆ p , the expected length becomes H ( e )+ D KL (cid:0) p
||
ˆ p (cid:1) .
We call D KL (cid:0) p || ˆ p (cid:1) the surplus description length (SDL) from encoding according to ˆ p instead of p :
[Equation]
We propose to measure the complexity of a learned predictor p
( Y | φ ( X )) by computing the surplus description length of encoding an inﬁnite stream of data according to the online code instead of the true conditional distribution; when A converges to the true predictor, this measure is bounded.
Deﬁnition 1 (Surplus description length of online codes) .
Given random variables X , Y ∼ D , a representation function φ , and a learning algorithm A , deﬁne
[Equation]
We generalize this deﬁnition to measure the complexity of learning an approximating conditional distribution with loss ε .
This corresponds to the additional description length incurred by encoding data with the learning algorithm
A rather than using a ﬁxed predictor with loss ε .
Deﬁnition 2 (Surplus description length of online codes with a speciﬁed baseline) .
Take random variables X , Y ∼ D , a representation function φ , a learning algorithm A , and a loss tolerance ε ≥ H
( Y | X ) .
Let [ c ] + denote max(0 , c ) and then we deﬁne
[Equation]
One interpretation of this measure is that it gives the cost (in terms of information) for re-creating an ε -loss predictor when using the representation φ .
In our framework, the surplus description length corresponds to computing the area between the loss-data curve and a baseline set by y = ε .
Whereas
MDL measures the complexity of a sample of n points, SDL measures the complexity of a function which solves the task to ε tolerance.
For guidance about setting ε and computing SDL, please see
Appendix B.
3. ε SAMPLE COMPLEXITY ( ε SC)
We also introduce a second measure called ε sample complexity ( ε SC), which consists of the number of samples needed to reach an expected population loss of ε .
We discuss this measure in Appendix A.
E XPERIMENTS
We empirically show the behavior of VA, MDL, SDL, and ε SC with experiments on MNIST.
The results, shown in Figure 2, demonstrate that the issue of sensitivity to evaluation dataset size in fact occurs in practice: VA and MDL choose different representations when given evaluation sets of different sizes.
Because these measures are a function of the evaluation dataset size, making a decision about which representation to use with a small evaluation dataset would be premature.
By contrast, SDL and ε SC are functions only of the data distribution , not a ﬁnite sample.
Once they measure the complexity of learning an ε -loss function, that measure is invariant to the size of the evaluation dataset.
We also performed experiments on part of speech classiﬁcation with representations from ELMo (Peters et al., 2018).
The results on this large-scale task, which are largely the same as those shown here, are in Appendix C. Details of the experiments, including representation training, probe architectures, and hyperparameters, are available in Appendix H.
Representation
CIFAR
Pixels
VAE n
VA 0.
MDL 122.
SDL, ε =0.
> 116.
> 141.
> 87.
ε SC, ε =0.
> 60.
> 60. > 60.
VA 0.
MDL 2165.
SDL, ε =0.
> 1704.
ε SC, ε =0.
> 31936.
[Figure]
ss CIFARPixelsVAE Representation
A PPENDIX A ε SAMPLE COMPLEXITY ( ε SC)
In addition to surplus description length we introduce a second, conceptually simpler measure of representation quality: ε sample complexity.
Deﬁnition 3 (Sample complexity of an ε -loss predictor) .
Given random variables X , Y ∼ D , a representation function φ , a learning algorithm A , and a loss tolerance ε ≥ H
( Y | φ ( X )) , deﬁne
[Equation]
The ε sample complexity measures the complexity of learning an ε -loss predictor by the number of samples it takes to ﬁnd it.
This measure allows the comparison of two representations by ﬁrst picking a target function to learn (via a setting of ε ), then measuring which representation enables learning that function with less data.
In our framework, sample complexity corresponds to taking a horizontal slice of the loss-data curve at y = ε , analogous to VA’s slice at y = n .
VA makes a statement about the data (by setting n ) and reports the accuracy of some function given that data.
In contrast, ε sample complexity speciﬁes the desired function and determines its complexity by how many samples are needed to learn it.
Estimating the ε SC.
Given an assumption that algorithms are monotonically improving such that L ( A , n + 1) ≤ L ( A , n ) , ε SC can be estimated efﬁciently.
With n ﬁnite samples in the evaluation dataset, an algorithm may estimate ε SC by splitting the data into k uniform-sized bins and estimating L ( A , ik / n ) for i ∈ { 1 . . .
k } .
By recursively performing this search on the interval which contains the transition from L > ε to L < ε , we can rapidly reach a precise estimate or report that m ε SC ( φ, D , A , ε ) > n .
A more detailed examination of the algorithmic considerations of estimating ε SC is in Appendix G, and an implementation is available in the supplement.
Using objectives other than negative log-likelihood.
Our exposition of ε SC uses negative loglikelihood for consistency with other methods, such as MDL, which require it.
However, it is straightforward to extend ε SC to work with whatever objective function is desired under the assumption that said objective is monotone with increasing data when using algorithm A .
A natural choice in many cases would be prediction accuracy, where a practitioner might target e.g. a 95% accurate predictor.
A PPENDIX B S ETTING ε AND ESTIMATING SDL B. S ETTING ε
A value for the threshold ε corresponds to the set of ε -loss predictors that a representation should make easy to learn.
Choices of ε ≥ H ( Y | X ) represent attainable functions, while selecting ε <
H ( Y | X ) leads to unbounded SDL and ε SC for any choice of the algorithm A .
For evaluating representation learning methods in the research community, we recommend using
SDL and establishing benchmarks which specify (1) a downstream task, in the form of an evaluation dataset; (2) a criterion for success, in the form of a setting of ε ; (3) a standard probing algorithm A .
The setting of ε can be done by training a large model on the raw representation of the full evaluation dataset and using its validation loss as ε when evaluating other representations.
This guarantees that ε ≥ H ( Y | X ) and the task is feasible with any representation at least as good as the raw data.
In turn, this ensures that SDL is bounded.
In practical applications, ε should be a part of the design speciﬁcation for a system.
As an example, a practitioner might know that an object detection system with 80% per-frame accuracy is sufﬁcient and labels are expensive.
For this task, the best representation would be one which enables the most sample efﬁcient learning of a predictor with error ε = 0 .
2 using a 0 – 1 loss.
B. E STIMATING SDL
Naively computing SDL would require unbounded data and the estimation of L ( A φ , i ) for every i .
However, any reasonable learning algorithm obtains a better-generalizing predictor when given more i.i.d.
data from the target distribution (Kaplan et al., 2020).
If we assume that algorithms are monotonically improving so that L ( A , i + 1) ≤ L ( A , i ) , SDL only depends on i up to the ﬁrst point where L ( A , n ) ≤ ε .
Approximating this integral can be done efﬁciently by taking a log-uniform partition of the evaluation dataset size and computing the Riemann sum as in Voita & Titov (2020).
Note that evaluating a representation only requires training probes, not the large representation functions themselves, and thus has modest computational requirements.
Crucially, if the tolerance ε is set unrealizeably low or the amount of available data is insufﬁcient, an implementation is able to report that the given complexity estimate is only a lower bound.
In Appendix F we provide a detailed algorithm for estimating SDL and a theorem proving its data requirements, and the supplement includes an implementation.
A PPENDIX C E XPERIMENT ON PART OF SPEECH CLASSIFICATION
For a second experiment, shown in Figure 3, we compare the representations given by different layers of a pretrained ELMo model (Peters et al., 2018).
We use the part-of-speech task introduced by
Hewitt & Liang (2019) and implemented by Voita & Titov (2020) with the same probe architecture and other hyperparameters as those works.
This leads to a large-scale representation evaluation task, with 4096-dimensional representation vectors and an output space of size 48 k for a sentence of k words.
The results demonstrate that SDL and ε SC can scale to tasks of a practically relevant size.
This experiment is of a similar size to the widespread use of BERT (Devlin et al., 2019) or SimCLR (Chen et al., 2020), and evaluating our measures to high precision took about an hour on one GPU.
[Table]
ε SC, ε =0.
>>>
VA 0.
MDL 92403.
SDL, ε =0.
> 40882.
ε SC, ε =0.
>
[Figure]
A PPENDIX D D ERIVING VA AND MI MEASURES
Nonlinear probes with limited data.
A simple strategy for evaluating representations is to choose a probe architecture and train it on a limited amount of data from the task and representation of interest (Hénaff et al., 2020; Zhang & Bowman, 2018); we call this the validation accuracy (VA) measure.
This method can be interpreted in our framework by replacing the validation accuracy with the validation loss and taking an expectation over draws of evaluation datasets of size n .
On the loss-data curve, this measure corresponds to evaluation at x = n :
[Equation]
Mutual information Mutual information (MI) between a representation φ ( X ) and targets Y is an often-proposed metric for learning and evaluating representations (Pimentel et al., 2020; Bachman et al., 2019).
In terms of entropy, mutual information is equivalent to the information gain about Y from knowing φ ( X ) :
[Equation]
In general mutual information is intractable to estimate for high-dimensional or continuous-valued variables (McAllester & Stratos, 2020), and a common approach is to use a very expressive model for ˆ p and maximize a variational lower bound:
[Equation]
Since H ( Y ) is not a function of the parameters, maximizing the lower bound is equivalent to minimizing the negative log-likelihood.
Moreover, if we assume that ˆ p is expressive enough to represent p and take n → ∞ , this inequality becomes tight.
As such, MI estimation can be seen a special case of nonlinear probes as described above, where instead of choosing some particular setting of n we push it to inﬁnity.
We formally deﬁne the mutual information measure of a representation as
[Equation]
A decrease in this measure reﬂects an increase in the mutual information.
On the loss-data curve, this corresponds to evaluation at x = ∞ .
A PPENDIX E R ELATED WORK
Representation evaluation methods.
Until recently, the standard technique for evaluating representation quality was the use of linear probes (Kiros et al., 2015; Hill et al., 2016; van den Oord et al., 2018; Chen et al., 2020).
However, Hénaff et al.
ﬁnd that evaluation with linear probes is largely uncorrelated with the more practically relevant objective of low-data accuracy, and Resnick et al.
(2019) show that linear probe performance does not predict performance for transfer across tasks.
Beyond linear probes, Zhang & Bowman (2018) and Hewitt & Liang (2019) show that restrictions on model capacity or evaluation dataset size are necessary to separate the performance of randomly- and linguistically-pretrained representations.
Voita & Titov (2020) propose using the MDL framework, which measures the description length of the labels given the observations.
An earlier work by Yogatama et al.
(2019) also uses prequential codes to evaluate representations for linguistic tasks.
Talmor et al.
(2019) look at the loss-data curve (called “learning curve” in their work) and use a weighted average of the validation loss at various training set sizes to evaluate representations.
Foundational work.
A fundamental paper by Blier & Ollivier (2018) introduces prequential codes as a measure of the complexity of a deep learning model.
Xu et al.
(2020) introduce predictive
V -information, a theoretical generalization of mutual information which takes into account computational constraints, and is essentially the mutual information lower bound often reported in practice.
Work by Dubois et al.
(2020) describe representations which, in combination with a speciﬁed family of predictive functions, have guarantees on their generalization performance.
A PPENDIX F A LGORITHMIC DETAILS FOR ESTIMATING SURPLUS DESCRIPTION LENGTH
Recall that the SDL is deﬁned as
[Equation]
For simplicity, we assume that L is bounded in [0 , 1] .
Note that this can be achieved by truncating the cross-entropy loss.
In our experiments we replace D kM [1 : n ] of Algorithm 1 with sampled subsets of size n from a single evaluation dataset.
Additionally, we use between 10 and 20 values of n instead of evaluating
L ( A φ , n ) at every integer between 1 and M .
This strategy, also used by Blier & Ollivier (2018) and Voita & Titov (2020), corresponds to the description length under a code which updates only periodically during transmission of the data instead of after every single point.
Theorem 4.
Let the loss function L be bounded in [0 , 1] and assume that it is decreasing in n .
With ( M + 1) K datapoints, if the sample complexity is less than M , the above algorithm returns an estimate ˆ m such that with probability at least 1 − δ
[Equation]
[Equation]
Algorithm
Input: tolerance ε , max iterations M , number of datasets K , representation φ , data distribution
D , algorithm A
Output:
Sample K datasets D kM ∼ D of size M +for n = 1 to M do
For each k ∈ [ K ] , run A on D kM [1 : n ] to produce a predictor ˆ p kn
Take K test samples ( x k , y k )
=
D kM [ M + 1]
Evaluate ˆ L n = 1 K (cid:80)
Kk =1 (cid:96) (ˆ p kn , x k , y k )
Set
ˆ m = (cid:80)
Mn =
[ˆ L n − ε ]
+ if ˆ L M ≤ ε/ 2 then Set
I = tight else
Set I = lower bound ; return ˆ m, I
If K ≥ log(1 /δ ) 2 εand the algorithm returns tight then with probability at least 1 − δ the sample complexity is less than M and the above bound holds.
Proof.
First we apply a Hoeffding bound to show that each ˆ L n is estimated well.
For any n , we have
[Equation]
since each (cid:96) (ˆ p kn , x k , y k ) is an independent variable, bounded in [0,1] with expectation L ( A φ , n ) .
Now when sample complexity is less than M , we use a union bound to translate this to a high probability bound on error of ˆ m , so that with probability at least 1 − δ :
[Equation]
This gives us the ﬁrst part of the claim.
We want to know that when the algorithm returns tight , the estimate can be trusted (i.e. that we set
M large enough).
Under the assumption of large enough K , and by an application of Hoeffding, we have that
[Equation]
If ˆ L M ≤ ε/ 2 , this means that L ( A φ , M ) ≤ ε with probability at least 1 − δ .
By the assumption of decreasing loss, this means the sample complexity is less than M , so the bound on the error of ˆ m holds.
A PPENDIX G A LGORITHMIC DETAILS FOR ESTIMATING SAMPLE COMPLEXITY
Recall that ε sample complexity ( ε SC) is deﬁned as
[Equation]
We estimate m ε SC via recursive grid search.
To be more precise, we ﬁrst deﬁne a search interval [1 , N ] , where N is a large enough number such that L ( A φ , N ) (cid:28) ε .
Then, we partition the search interval in to 10 sub-intervals and estimate risk of hypothesis learned from D n ∼ D n with high conﬁdence for each sub-interval.
We then ﬁnd the leftmost sub-interval that potentially contains m ε SC and proceed recursively.
This procedure is formalized in Algorithm 2 and its guarantee is given by Theorem 5.
Input: Search upper limit N , parameters ε , conﬁdence parameter δ , data distribution D , and learning algorithm A .
Output: Estimate ˆ m such that m ε SC ( φ, D , A , ε ) ≤
ˆ m with probability 1 − δ .
let S
= 2 log(20 k/δ ) /ε 2 , and let [ (cid:96), u ] be the search interval initialized at (cid:96)
= 1 , u = N .
for r = 1 to k do
Partition [ (cid:96), u ] into 10 equispaced bins and let ∆ be the length of each bin.
for j = 1 to 10 do
Set n = (cid:96) + j ∆ .
Compute ˆ L n = 1 S (cid:80) S i =1 (cid:96) ( A ( D n i ) , x i , y i ) for S independent draws of D n and test sample ( x, y ) .
if ˆ L n ≤ ε/ 2 then
Set u = n and (cid:96) = n − ∆ .
break return ˆ m = u , which satisﬁes m ε SC ( φ, D , A , ε ) ≤ ˆ m with probability 1 − δ , where the randomness is over independent draws of D n and test samples ( x, y ) .
Theorem 5.
Let the loss function L be bounded in [0 , 1] and assume that it is decreasing in n .
Then,
Algorithm 2 returns an estimate ˆ m that satisﬁes m ε SC ( φ, D , A , ε ) ≤ ˆ m with probability at least 1 − δ .
Proof.
By Hoeffding, the probability that | ˆ L n − L ( A φ , n ) | ≥ ε/ 2 , where ˆ L is computed with
S = 2 log(20 k/δ ) /ε
2 independent draws of D n ∼ D n and ( x, y ) ∼ D , is less than δ/ (10 k ) .
The algorithm terminates after evaluating ˆ L on at most 10 k different n ’s. By a union bound, the probability that | ˆ L n − L ( A φ , n ) | ≤ ε/ 2 for all n used by the algorithm is at least 1 − δ .
Hence, ˆ L n ≤ ε/implies L ( A φ , n ) ≤ ε with probability at least 1 − δ .
A PPENDIX H E XPERIMENTAL DETAILS
In each experiment we ﬁrst estimate the loss-data curve using a ﬁxed number of dataset sizes n and multiple random seeds, then compute each measure from that curve.
Reported values of SDL correspond to the estimated area between the loss-data curve and the line y =
ε using Riemann sums with the values taken from the left edge of the interval.
This is the same as the chunking procedure of
Voita & Titov (2020) and is equivalent to the code length of transmitting each chunk of data using a ﬁxed model and switching models between intervals.
Reported values of ε SC correspond to the ﬁrst measured n at which the loss is less than ε .
All of the experiments were performed on a single server with 4 NVidia Titan X GPUs, and on this hardware no experiment took longer than an hour.
All of the code for our experiments, as well as that used to generate our plots and tables, is included in the supplement.
H. MNIST EXPERIMENTS
For our experiments on MNIST, we implement a highly-performant vectorized library in JAX to construct loss-data curves.
With this implementation it takes about one minute to estimate the loss-data curve with one sample at each of 20 settings of n .
We approximate the loss-data curves at 20 settings of n log-uniformly spaced on the interval [10 , 50000] and evaluate loss on the test set to approximate the population loss.
At each dataset size n we perform the same number of updates to the model; we experimented with early stopping for smaller n but found that it made no difference on this dataset.
In order to obtain lower-variance estimates of the expected risk at each n , we runrandom seeds for each representation at each dataset size, where each random seed corresponds to a random initialization of the probe network and a random subsample of the evaluation dataset.
Probes consist of two-hidden-layer MLPs with hidden dimension 512 and ReLU activations.
All probes and representations are trained with the Adam optimizer (Kingma & Ba, 2015) with learning rate 10 − 4 .
Each representation is normalized to have zero mean and unit variance before probing to ensure that differences in scaling and centering do not disrupt learning.
The representations of the data we evaluate are implemented as follows.
Raw pixels.
The raw MNIST pixels are provided by the Pytorch datasets library (Paszke et al., 2019).
It has dimension 28 × 28 = 784 .
CIFAR.
The CIFAR representation is given by the last hidden layer of a convolutional neural network trained on the CIFAR-10 dataset.
This representation has dimension 784 to match the size of the raw pixels.
The network architecture is as follows:
nn.
Conv2d(1, 32, 3, 1), nn.ReLU(), nn.MaxPool2d(2), nn.
Conv2d(32,64, 3, 1), nn.ReLU(), nn.MaxPool2d(2), nn.
Flatten(), nn.
Linear(1600,784)
nn.ReLU() nn.Linear(784,10) nn.
LogSoftmax()
VAE.
The VAE (variational autoencoder; Kingma & Welling (2014); Rezende et al.
(2014)) representation is given by a variational autoencoder trained to generate the MNIST digits.
This VAE’s latent variable has dimension 8.
We use the mean output of the encoder as the representation of the data.
The network architecture is as follows: self.encoder_layers = nn.
Sequential( nn.Linear(784, 400), nn.ReLU(), nn.
Linear(400, 400), nn.ReLU(), nn.
Linear(400, 400), nn.ReLU(), ) self.mean = nn.
Linear(400, 8) self.variance =
nn.
Linear(400, 8) self.decoder_layers = nn.
Sequential( nn.Linear(8, 400)
, nn.ReLU(), nn.
Linear(400, 400), nn.ReLU(), nn.
Linear(400, 784), )
H. P ART OF SPEECH EXPERIMENTS
We follow the methodology and use the ofﬁcial code 2 of Voita & Titov (2020) for our part of speech experiments using ELMo (Peters et al., 2018) pretrained representations.
In order to obtain lowervariance estimates of the expected risk at each n , we run 4 random seeds for each representation at each dataset size, where each random seed corresponds to a random initialization of the probe network and a random subsample of the evaluation dataset.
We approximate the loss-data curves at 10 settings of n log-uniformly spaced on the range of the available data n ∈ [10 , 10 6 ] .
To more precisely estimate ε SC, we perform one recursive grid search step: we space 10 settings over the range which in the ﬁrst round saw L ( A φ , n ) transition from above to below ε .
Probes consist of the MLP-2 model of Hewitt & Liang (2019); Voita & Titov (2020) and all training parameters are the same as in those works.