S EQUENTIAL L EARNING OF N EURAL N ETWORKS FOR P REQUENTIAL MDL
Anonymous authors
Paper under double-blind review
A BSTRACT
Minimum Description Length (MDL) provides a framework and an objective for principled model evaluation.
It formalizes Occam’s Razor and can be applied to data from non-stationary sources.
In the prequential formulation of MDL, the objective is to minimize the cumulative next-step log-loss when sequentially going through the data and using previous observations for parameter estimation.
It thus closely resembles a continual- or online-learning problem.
In this study, we evaluate approaches for computing prequential description lengths for image classification datasets with neural networks.
Considering the computational cost, we find that online-learning with rehearsal has favorable performance compared to the previously widely used block-wise estimation.
We propose forward-calibration to better align the models predictions with the empirical observations and introduce replay-streams , a minibatch incremental training technique to efficiently implement approximate random replay while avoiding large in-memory replay buffers.
As a result, we present description lengths for a suite of image classification datasets that improve upon previously reported results by large margins.
I NTRODUCTION
Within the field of deep learning, the paradigm of empirical risk minimization (ERM, Vapnik (1991)) together with model and hyper-parameter selection based on held-out data is the prevailing training and evaluation protocol.
This approach has served the field well, supporting seamless scaling to large model and data set sizes.
The core assumptions of ERM are: a) the existence of fixed but unknown distributions q ( x ) and q ( y | x ) that represent the data-generating process for the problem under consideration; b) the goal is to obtain a function ˆ y = f
( x, θ ∗ ) that minimizes some loss L ( y, ˆ y ) in expectation over the data drawn from q ; and c) that we are given a set of (i.i.d.)
samples from q to use as training and validation data.
Its simplicity and well understood theoretical properties make
ERM an attractive framework when developing learning machines.
However, sometimes we wish to employ deep learning techniques in situations where not all these basic assumptions hold.
For example, if we do not assume a fixed data-generating distribution q we enter the realm of continual learning , life long learning or online learning .
Multiple different terms are used for these scenarios because they operate under different constraints, and because there is some ambiguity about what problem exactly a learning machine is supposed to solve.
A recent survey on continual- and life-long learning for example describes multiple, sometimes conflicting desiderata considered in the literature: forward-transfer, backward-transfer, avoiding catastrophic forgetting and maintaining plasticity (Hadsell et al., 2020).
Another situation in which the ERM framework is not necessarily the best approach is when minimizing the expected loss L is not the only, or maybe not even the primary objective of the learning machine.
For example, recently deep-learning techniques have been used to aid structural and causal inference (Vowels et al., 2021).
In these cases, we are more interested in model selection or some aspect of the learned parameters θ ∗ than the generalization loss L .
Here we have little to gain from the generalization bounds provided by the ERM framework and in fact some of its properties can be harmful.
Independent of ERM, compression based approaches to inference and learning such as Minimum
Description Length (Rissanen, 1984; Grunwald, 2004), Minimum Message Length (Wallace, 2005) and Solomonoffs theory of inductive inference (Solomonoff, 1964; Rathmanner & Hutter, 2011) have been extensively studied.
In practice, however, they have been primarily applied to small scale problems and with very simple models compared to deep neural networks.
These approaches strive to find and train models that can compress the observed data well and rely on the fact that such models have a good chance of generalizing on future data.
It is considered a major benefit that these approaches come with a clear objective and have been studied in detail even for concrete sequence of observations; without assuming stationarity or a probabilistic generative process at all (also called the individual sequence scenario).
The individual sequence setting is problematic for ERM because the fundamental step of creating training and validation splits is not clearly defined.
In fact, creating such splits implies making an assumption about what is equally distributed across training and validation data.
But creating validation splits can be problematic even with (assumed) stationary distributions, and this becomes more pressing as ML research moves from curated academic benchmark data sets to large user provided or web-scraped data.
A discussion and examples can be found in A.3.
In contrast to ERM, compression approaches furthermore include a form of Occam’s Razor, i.e. they prefer simpler models (according to some implicit measure of complexity) as long as the simplicity does not harm the model’s predictive performance.
The literature considers multiple, subtly different approaches to defining and computing description lengths.
Most of them are intractable and known approximations break down for overparameterized model families such as neural networks.
One particular approach however, prequential MDL (Dawid & Vovk, 1999; Poland & Hutter, 2005), turns computing the description
length
L ( D| M )
= − log p ( D| M ) into a specific kind of continual or online learning problem: log p ( D| M ):= (cid:80) Tt =1 log p M ( y t | x t , ˆ θ ( D <t )) , where D = { ( x t , y t ) } T 1 is the sequence of inputs x t and associated prediction targets y t ; p M ( y | x, θ ) denotes a model family M and ˆ θ ( D <t ) an associated parameter estimator given training data D <t .
In contrast to ERM which considers the performance of a learning algorithm on held-out data after training on a fixed dataset, prequential MDL evaluates a learner by its generalization performance on y t after trained on initially short but progressively longer sequences of observations D <t .
This resembles an online learning problem where a learner is sequentially exposed to new data ( x t , y t ) , however a sequential learner is also allowed to revisit old data x t ′ , y t ′ for training and when making a prediction at time t>t ′ .
Contributions.
Previous work on computing prequential description lengths with neural networks relied on a block-wise (chunk-incremental) approximation: at some positions t a model is trained from random initialization to convergence on data D <t and then their prediction losses on the next intervals are combined (Blier & Ollivier, 2018;
Bornschein et al., 2020; Jin et al., 2021; Bornschein et al., 2021; Perez et al., 2021; Whitney et al., 2020).
We investigate alternatives that are inspired by continual-learning (CL) based methods.
In particular, chunk-incremental and mini-batch incremental fine-tuning with rehearsal.
Throughout this empirical study, we consider the computational costs associated with these methods.
We propose two new techniques: forward-calibration and replay streams .
Forward-calibration improves the results by making the model’s predictive uncertainty better match the observed distribution.
Replay streams makes replay for mini-batch incremental learning more scalable by providing approximate random rehearsal while avoiding large in-memory replay-buffers.
We identify exponential moving parameter averages, label-smoothing and weight-standardization as generally useful techniques.
As a result we present description lengths for a suite of popular image classification datasets that improve upon the previously reported results (Blier & Ollivier, 2018; Bornschein et al., 2020) by large margins.
The motivation for studying prequential MDL stems from the desire to apply deep-learning techniques in situations violating the assumptions of ERM.
In this work however we concentrate on established iid.
datasets: stationarity is most challenging for a sequential learner, which has to uniformly learn from all previously examples, and it allows us to evaluate well known architectures and compare their held-out performance against ERM-based training, which has been the focus of the community for decades.
We include additional experimental results on non-stationary data in Appendix C.7.3.
An example: model selection using ERM v.s. MDL.
To showcase the properties of prequential
MDL and its bias towards simpler models, we generate images by randomly drawing 3 MNIST examples and superimposing them as red(R), green(G) and blue(B) channels.
The label is always evidence that should be preferred randomly sampled
See Figure
The models are trained on all the previous examples.
We observe that the models conditioned on the
R, R and G, or R, G and B channels all rapidly improve their predictive performance when trained on which is conditioned on the R channel only.
During the first ≈ 2k examples the models on irrelevant channels accumulate hundreds of excess description associated with the red channel; green and blue are distractors.
We train a LeNet (LeCun et al., 1989) convolutional neural network on 100k examples from this dataset; either within the ERM framework by drawing i.i.d.
examples, training to convergence and validating on held-out data, or within the MDL framework by treating the data as a sequence, training online with replay buffer (see Section 2 for details) and recording the (cumulative) next-step prediction losses and errors.
We compare 3 different models: a model conditioned only on the R channel, on the R and G, or on R, G and B channels.
Figure 1 (middle) shows the validation-set performance of the best performing models after ERM training.
Figure 1 (right) shows description lengths on the same data.
It is evident that ERM does not give a clear indication which conditioning should be preferred.
However, MDL strongly prefers the model conditioned only on the R-channel.
Figure 2 shows a more detailed analysis of the three differently conditioned models in the MDL scenario.
We observe that prequential MDL identifies the minimal conditioning because it takes the small-data performance of the models into account.
Given sufficient training data the three alternatives perform almost indistinguishable (as seen from their almost parallel regret curves).
When we treat the description lengths as evidence for Bayesian model selection to choose between M R , M RG and
M RGB we obtain p ( M R |D )
=
p ( D| M R ) / p ( D| M R ) + p ( D| M RG ) + p ( D| M RGB ) .
With the description length − log p ( D| M R ) about 300 nats smaller than the ones for M RG and M RGB , we have ≈ 1: e − 300 odds in favour of M R .
The Appendix C.1 contains additional experiments with more architectures and the experimental details; demonstrating that this is a robust result regardless of architectures and optimization hyperparameters.
O NLINE L EARNING A PPROACHES TO P REQUENTIAL MDL E STIMATION
In this section, we describe practical strategies for computing prequential description lengths with neural networks.
Conceptually, computing the cumulative log-loss resembles a continual- or onlinelearning problem without constraints such as limiting access to previously observed data.
In practice, however, some naive approaches such as retraining from scratch to convergence whenever a new example arrives are infeasible due to their high resource usage.
In our study, we consider the compute requirements of each approach by counting the number of floating-point operations (FLOPs) until the learner reaches the end of the data sequence.
We do not limit the learner’s ability to access or sample
Require: data ( x t , y t )
Tt =1 ; augmentation Aug ( . ) ; number of replay streams K ; EMA decay α 1: Initialize: parameters θ ; EMA parameters ¯ θ = θ ; softmax temperature calibration β = 1 ;
Replay positions { ρ k = 1 } Kk =1 ; L preq =2: for t = 1 to T do 3:
Compute next-step loss: L t := − log p ( y t | x t , ¯ θ, β ) 4:
Update cumulative loss: L preq ← L preq + L t 5:
Update temperature calibration: β ← β − ∇ β log p ( y t | x t , ¯ θ, β ) 6:
Update parameters: θ ← θ − ∇ θ log p ( y t | ˜ x t , θ, 1 . ) , with ˜ x t =
Aug ( x t ) 7:
Update EMA parameters: ¯ θ ← (1 − α )¯ θ + α θ 8: for k = 1 to K do 9:
Get data from stream k : x ← x ρ k ; y ← y ρ k , advance position ρ k ← ρ k +10:
Update parameters: θ ← θ − ∇ θ log p ( y | ˜ x, θ, 1 . ) with ˜ x
= Aug ( x ) 11:
Update EMA parameters: ¯ θ ← (1 − α )¯ θ + α θ 12:
Maintain replay distribution: with probability p reset reset ρ k ← 1 (see Equation (1)) 13: end for 14: end for 15: return L preq
In practice we perform the algorithm with mini-batches instead of individual examples and use gradient based optimizers such as AdamW(Loshchilov & Hutter, 2019) instead of plain SGD.
previously observed data because storing and reading from large datasets is generally not considered a major technical bottleneck.
However, in-memory (RAM) based implementations of replay-buffers for continual learning (rehearsal) suffer from limited capacity because data sets for training deep neural networks often exceed RAM sizes.
Below we describe replay-streams , a simple approach to approximate random sampling for replay while only utilizing cheap sequential access to data on disk.
With this approach, unlimited approximate random rehearsal from data on permanent storage becomes as easy as sampling from large datasets for ERM based optimization.
We compare the following approaches to compute prequential description lengths:
Chunk Incremental / From-Scratch (CI/FS): Variants of this approach have been used to compute the recently reported description lengths for deep-learning models (Blier & Ollivier, 2018; Bornschein et al., 2020; Jin et al., 2021; Bornschein et al., 2021; Perez et al., 2021).
The data-sequence is partitioned into K non-overlapping intervals; typically of increasing size (i.e. choosing exponentially spaced split points { s k } Kk =1 , with s k ∈ [2 , . . . , n ] , s k <s k +1 and s K = N ).
For each k , a neural network is randomly initialized and trained to convergence on all the examples before the split point s k and evaluated on the data between s k and s k
+1 .
This corresponds to the block-wise approximation of the area under the curve: (cid:80) Ni =1 log p ( y i | ˆ θ ( D <i ))
≈ (cid:80) K − 1 k =1 (cid:80) s k
+1 j = s k log p ( y j | ˆ θ ( D <s k )) , where ˆ θ ( D ) denotes parameters after training on data D .
To ensure that the model produces calibrated predictions even when D <s k is small, we use softmax temperature calibration (Guo et al., 2017): at each stage we split the data D <s k into a 90% training and a 10% calibration data.
Conceptually, we could perform post-calibration by first training the network to convergence and then, with all parameters frozen, replacing the output layer softmax( h ) with the calibrated output layer softmax(softplus( β ) h ) , where β is a scalar parameter chosen to minimize the loss on calibration data.
In practice, we optimize β by gradient descent in parallel with the other model parameters.
We alternate computing gradient steps for θ , calculated from the training set and using the uncalibrated network (with final layer softmax( h ) ), with a gradient step on β , calculated from the validation set using the calibrated network (with final layer softmax(softplus( β ) h ) ).
This simple calibration procedure has proven to be surprisingly effective at avoiding overfitting symptoms when training large neural networks on small datasets (Guo et al., 2017; Bornschein et al., 2020).
Chunk Incremental / Continual Fine-tuning (CI/CF): Similar to CI/FS, the sequence is split into increasingly larger chunks.
But instead of training the model from scratch at each stage, the network is continuously fine-tuned on the now larger dataset D <s k .
We use the same calibration strategy as for CI/FS.
We expect to save compute by avoiding training from scratch.
However, recent research suggests that, when first trained on data D A and then trained on data D A ∪ D B , deep-learning models struggle to converge to a solution that generalizes as well as training on D A ∪ D B from scratch; even when both D A and D B are sampled from the same underlying distribution.
In Ash & Adams (2020) the authors observe that shrinking the model-parameters and adding some noise after training on D A can improve the final generalization performance.
We therefore run an ablation study and perform shrink & perturb operation whenever advancing to the next chunk k .
Mini-batch Incremental / Replay-Buffer (MI/RB)
This approach uses continual online minibatch gradient descent with an additional replay buffer to store previously seen examples.
At each time t , the learner performs a number of learning steps on data in the replay buffer.
This could be all or a subset of the data D <t depending on the capacity of the replay.
We propose forward-calibration to optimize the calibration parameter β : each new batch of examples is first used for evaluation, then used to perform a single gradient step to optimize β , and finally used for training the parameters θ and placed into the replay-buffer.
Calibration is computationally almost free: the forward-pass for the calibration gradient step can be shared with the evaluation forward-pass, and backward-propagation for β ends right after the softmax layer.
Appendix B.1 contains a more detailed description.
If the replay buffer is limited by the memory capacity, we either use a FIFO replacement-strategy or reservoir-sampling (Vitter, 1985) to maintain a fixed maximum buffer size.
Mini-batch Incremental / Replay-Streams (MI/RS)
Implementing large replay buffers can be technically challenging: in-memory (RAM) replay buffers have limited capacity and random access and sampling from permanent storage is often slow.
We propose a simple yet effective alternative to storing replay data in-memory.
We instead assume that the data is stored in its original order on permanent storage and we maintain K replay streams (pointers into the sequence) , each reading the data in-order from the sequence ( x 1 , y 1 ) . . .
( x T , y T ) .
We can think of a replay-stream as a file-handle, currently at position ρ ∈ 1 . .
.T .
A read operation yields ( x ρ , y ρ ) and increments the position to ρ ← ρ + 1 .
We denote the position of the k th replay-stream with ρ k .
Every time the learner steps forward from time t to ˜ t =
t +1 we also read a replay example from each of the K replay-streams and perform gradient steps on them.
By stochastically resetting individual streams with probability p reset to position ρ ← 1 , we can influence the distribution of positions ρ .
If we wish to use uniform replay of previous examples we reset with probability p reset = 1 / ˜ t .
If instead we wish to prioritize recent examples and replay them according to p replay ( a ) , where a =˜ t − i is the age of the example ( x i , y i ) relative to the current position of the learner ˜ t , we use
[Equation]
Note that p replay can be an unnormalized distribution and that by setting p replay ∝ 1 we recover p reset = 1 / ˜ t for uniform replay.
See Algorithm 1 for details.
E XPERIMENTS
We empirically evaluate the approaches from Section 2 on a suite of models and image classification data sets.
We use MNIST (LeCun et al., 2010), EMNIST, CIFAR-10, CIFAR-100 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) and randomly shuffle each into a fixed sequence of examples.
With MDL, we do not require validation or test set and could merge it with the training data.
But because the data is known to be i.i.d.
and we compare results with established ERM-based training, we instead maintain the standard test sets as held-out data.
It must be emphasized that the primary objective throughout this study is to minimize the description length.
All hyperparameters are tuned towards that goal.
When we report test set performance it should be considered a sanity check.
For more detailed analyses we will plot the " regret " a method M
i accumulates relative to some comparator method M c :
[Equation]
) + log p ( D ≤ t | M c ) .
We evaluate the following architectures: MLPs, VGG (Simonyan & Zisserman, 2015),
ResNets (He et al., 2016), WideResNets (Zagoruyko & Komodakis, 2016) and the transformer-based DeiT (Touvron et al., 2021) architecture.
We refer to a variant of the VGG architecture with weight-standardized convolutions (Qiao et al., 2019) and with batch normalization (Ioffe & Szegedy, 2015) as VGG++ .
For the bulk of our experiments, we use the AdamW (Loshchilov & Hutter, 2019) as optimizer; a series of control experiments however suggest that the quantitative results generalize to optimizers such as
RMSProp and simple mini-batch SGD with momentum.
We identified the following techniques that consistently improve the results and use them throughout this paper if not mentioned otherwise:
Exponentially moving average (EMA)
Similar to Polyak averaging (Polyak & Juditsky, 1992),
EMA maintains a moving average of the model parameters θ and is used for next-step evaluations as well as when tuning the softmax temperature β .
Label smoothing (Szegedy et al., 2015)
A reliable regularizer, label smoothing often helps for the continual fine-tuning methods such as MI/RB, MI/RS and CI/CF.
Weight standardization For the VGG architecture, weight standardized convolutions lead to significant improvements - bigger than for conventional ERM-based training.
For other architectures such as ResNets, the results are not conclusive.
3. C OMPARISON OF O NLINE L EARNING A PPROACHES .
To compare the effectiveness of the different approaches listed in Section 2, we run a broad random hyperparameter sweep for EMNIST with LeNet, CIFAR-10 and CIFAR-100 with VGG++ and
ImageNet with ResNet50.
The hyperparameter intervals depend on the data and are detailed in
Appendix B. We sample learning rate, EMA step size, batch size, weight decay; but crucially also number of epochs (or, correspondingly, number of replay streams for MI/RS) and an overall scaling of the model width (number of channels).
We thus cover a large range of computational budgets, spanning about 2 orders of magnitude.
When the capacity of the replay-buffer matches the total length of the sequence, MI/RB and MI/RS are equivalent.
In these cases we omit MI/RB and instead use MI/RS to represent online-learning with unlimited rehearsal.
A study of replay buffer v.s. replay streams can be found in Section 3.1.
We show the overall Pareto front for these experiments in
Figure 3.
Mini-batch incremental learning obtains the lowest description lengths for a wide range of computational budgets.
Only for very large compute budgets on CIFAR-100 we observe that the block-wise approximation with training from scratch (CI/FS) has a better results.
[Table]
4.6e18 ImageNet
DeiT-S 2.73M 588k 1.
1.2e 1.
4.8e
Prior Works:
MNIST(Blier & Ollivier, 2018)
MnistNet 2.8k [Table] .5 CIFAR-10(Blier & Ollivier, 2018)
≈ VGG++ 31.4k [Table] [Table]
ImageNet(Bornschein et al., 2020) ResNet3.32M -
MDL for Deep Neural Networks. We apply MI/RS to a wider range of architectures. For most experiments we again use random hyperparameter sweeps, this time however without scaling the size of the models and with an increased lower bound for the number of replay streams (see
Appendix B).
We present our results for the best run in each sweep and contrast them with previously reported description lengths from the literature in Table 1.
We also show the test set performance of these models after they reached the end of training data.
The rightmost columns show the test set performance when performing the same hyperparameter sweep for conventional ERM training on the full data set.
Note that these runs do not use any kind of learning rate annealing, which is otherwise often employed to improve results.
Based on these experiments we plot a regret comparison for different architectures trained on
ImageNet in Figure 4 (left).
We observe that the additional depth and capacity of a ResNet101 is beneficial from the very beginning of the data-sequence.
Figure 4 (right) shows the effect of scaling the number of channels of a VGG++ architecture on CIFAR-100 and it is evident that decreasing the model width has negative effects throughout the sequence and not just when the data gets larger.
Replay Buffer and Replay Streams are equivalent for short data sequences, they differ when the length of a sequence exceeds the capacity of the replay buffer.
In this section, we study the effects of limiting the replay buffer size.
Figure 5 shows that reducing the capacity has a significant negative effect on the results.
In addition, the strategy of which samples to keep in the buffer plays an important role.
Using reservoir sampling instead of a FIFO replacement policy leads to further severely degraded results.
This is maybe not surprising because with reservoir-sampling insertion to the buffer is stochastic; a fraction of the examples are thus never replayed at all.
Appendix C. contains ablations for the number of replay-streams on CIFAR-100 and ImageNet.
Ablations.
We provide ablations for using forward-calibration, label smoothing and weight standardization in Table 2.
It is evident that each of these techniques helps obtaining better description lengths for the datasets and architectures under consideration.
Regret plots for these ablations can be found in Appendix C.
[Table]
Plasticity and the Warm-starting problem We take a closer look at the large-FLOPs regime for CIFAR-100 where CI/FS obtained better results than MI/RS.
Figure 6 (left) shows the regret comparison between the best runs for each training method.
The split positions s k are clearly visible as kinks in the graph for CI/FS and CI/CF.
For CI/CF the plot shows that right after fine-tuning, the predictive performance is approximately equal to MI/RS.
For CI/FS we see that the regret relative to MI/RS decreases: it has better predictive performance.
This effect persists even after running extensive additional hyperparameter sweeps.
We conjecture that this is a signature of the warmstarting problem described in (Ash & Adams, 2020).
Figure 6 (right) shows the Pareto front for the final test set accuracy of the best models (selected by their description length) over FLOPs.
We observe a large gap between fine-tuning based methods and CI/FS; which is consistent with this hypothesis.
Note that the last split-point s K
= T implies that these models are effectively trained like in a conventional ERM paradigm.
Appendix C shows test set pareto fronts for a range of datasets.
Close inspection of various regret plots sometimes reveals the same signatures, suggesting that the effect is ubiquitous, but often much less pronounced.
It is plausible that this also contributes to the gap in final test set performance in Table 1.
D ISCUSSION & R ELATED W ORK
Compression based approaches to inference and learning, such as Solomonoff induction (Solomonoff, 1964), Minimum Message Length (Wallace, 2005) and Minimum Description Length
(Rissanen, 1984; Grunwald, 2004), have been extensively studied.
The Bayesian and variational Bayesian approaches to estimating code lengths have been popular and, at least outside the field of deep learning, also been very successful.
This line of work includes methods such as AIC and BIC, as well as more sophisticated approximations to the posterior distribution over model parameters for many kinds of models, including neural networks (Hinton & Van Camp, 1993; MacKay, 2003).
The description length depends crucially on the quality of these posterior approximations.
Unfortunately, estimating the parameter posterior for modern neural networks is notoriously hard and an area of active research (Izmailov et al., 2021).
Blier & Ollivier (2018) demonstrated that much shorter code lengths can be achieved by using the prequential approach with standard neural network architectures and optimizers.
The prequential perspective with the block-wise approximation technique have been successively used in the context of image-classification (Bornschein et al., 2020), natural language processing (Voita & Titov, 2020; Jin et al., 2021;
Perez et al., 2021) and causal structure discovery (Bornschein et al., 2021).
With our work, we aim to simplify and improve prequential code length estimation for neural networks with insights from the field of continuous learning (CL).
For CL, fine-tuning with rehearsal is considered a strong baseline and large number of methods have been proposed to maintain its benefits while minimizing the size of the replay buffer (Mnih et al.,
Delange et al., 2021).
We instead propose a method that makes it practical and relatively cheap to work with full rehearsal.
Caccia et al.
(2022) investigate very similar scenario to ours; largely phrased around how long a learner should wait and accumulate data when it is compute constraint; most experiments however consider the case without full rehearsal.
He and Lin (He & Lin, 2020) discuss the close relationship between prequential coding and CL, however focus on compressive replay instead of accessing previous data.
In general, the literature on CL methods and evaluation metrics is too broad to be adequately discussed here (Hadsell et al., 2020).
However, we would not be surprised if some existing
CL methods have the potential to vastly improve prequential MDL computation.
In this sense, we see our work as a step of bringing CL and prequential MDL related works closer together.
Besides of taking inspiration from CL to improve description lengths, we believe that the MDL perspective can help guide future CL research: Much of the CL literature is rooted in an ERM-based perspective and thus considers distinct training, validation and test sets at different stages during learning.
This has potentially played a role in the proliferation of many different evaluation metrics and training protocols used in CL research (Hadsell et al., 2020).
This heterogeneity of the evaluation metrics makes it challenging to compare results.
Forward-transfer is a commonly used evaluation metric and it is closely related to a prequential evaluation.
However, it is often computed on separate held-out sets after training and does not take sample-efficiency into account as prequential MDL does.
Recent work has embraced forward-validation in terms of average prediction-error on future training data as a evaluation metric, thus bringing it closer to a prequential MDL based evaluation but without discussing the connection (Cai et al., 2021; Lin et al., 2021).
MDL has been conceived and analyzed as a theoretically well motivated objective when dealing with non-stationary data; without reference to the practical difficulties of actually performing parameter estimates.
It is a challenging objective even for learners without constraints on compute and (replay-) memory.
C ONCLUSIONS & L IMITATIONS
In this paper, we compare different approaches to computing prequential description lengths while considering their computational cost.
We find that continual learning inspired mini-batch incremental training with full rehearsal is the preferred method over a wide range of computational budgets.
We introduce two techniques that either improve results, or make it practical to apply full rehearsal even for very large data.
As a result we report description lengths for a range of standard data sets that improve significantly over previously reported figures.
Some of our results show that the warmstarting-effect can have a negative effect on the performance of continual fine-tuning.
We also generally observe a gap in the final held-out performance after training models within the prequental framework vs. training models with the ERM-based approach.
We are not aware of any technique to reliably mitigate the impact.
It would be an important topic to tackle for future works.
Furthermore, learning rate schedules such as cosine decay are popular and effective techniques for achieving better performance in ERM-based training.
It is not obvious how to best translate them to the prequential learning scenario as the learner constantly receives new data.
S EQUENTIAL L EARNING OF N EURAL N ETWORKS FOR P REQUENTIAL MDL S UPPLEMENTARY M ATERIAL A T HE M INIMUM D ESCRIPTION L ENGTH P RINCIPLE
In this section we try to give a short overview of the mathematical formulation of the Minimum
Description Length Principle (MDL).
For a proper introduction we refer to (Rissanen, 1984) and (Grunwald & Vitanyi, 2004).
Like other closely related approaches such as Minimum Message Length (Wallace, 2005), MDL is based on the fundamental idea that learning and comprehension correspond to compression (Rathmanner & Hutter, 2011):
Given data D = { y t } Ti and a hypothesis space M = { M 1 , M 2 , . . . } , each referring to a probabilistic model family p ( D| θ, M i ) parameterized with the model parameters θ , the most plausible model M ∈ M is the one that yields the shortest description of the data, i.e. compresses it the most.
To simplify the notation in this section we will use D = { y t } Ti to denote the predicted data only and omit the explicit conditioning on x t .
We use code and model interchangeably, since according to Kraft–McMillan’s inequality, the negative logarithm of probability mass functions corresponds to code-length functions:
L ( D | θ, M i ) =
− log p ( D | θ, M i ) .
A crude way to obtain a description length is to consider L ( D | M )=
L M ( θ )+ L M ( D | θ ) , where
L M ( θ ) is the cost of encoding the parameters θ and L M ( D | θ )
= − log p ( D | θ, M ) is the cost of compressing the data with the parameterized model.
While this approach might be intuitive, it is suboptimal and does not specify how to encode the parameters.
To avoid the ambiguity that comes with choosing a code L M ( θ ) for the parameters, the MDL literature instead suggests to use a universal code p ( D | M ) , which is defined to be the distribution that as closely as possible tracks the code-length of the maximum log-likelihood solution ˆ θ MLE ( D ) = arg max θ p ( D | θ, M ) within M on all possible data z 1: T ∈ Y T :
[Equation]
If it exists, the solution of q in Eq.
(2) is the Normalized Maximum Likelihood (NML) p
[Equation]
( y 1: n | M ) =
p
[Equation]
y
[Equation]
|
[Equation]
[Equation]
[Equation]
z
[Equation]
1: T | ˆ θ MLE ( z 1: T ) ,M ) .
Hence, the NML code-length is
[Equation]
Eq.
(3) has two terms: the log-likelihood term, assessing how well the model fits the data, and the model complexity term, formalising Occam’s Razor.
Intuitively, a model’s complexity is large if it can fit well many possible data z 1: T .
As a result, using the universal code for model selection leads to favoring models that have expressiveness to model the observed data D , but do not waste expressiveness on dissimilar data.
Comparing to ERM, MDL doesn’t need to have separate validation data, which means the entirety of the dataset can be used for training purpose.
In addition, MDL applies for i.i.d data as well as non-stationarity data.
A. P RACTICAL INTERPRETATIONS OF MDL
Although NML is intuitive to understand, it is intractable and even undefined in many cases.
There are two other versions of MDL which are also provably universal codes and, in addition, easier to compute in practice:
Bayesian and variational two-part codes.
In the Bayesian approach, we choose a prior p ( θ | M ) over the parameters.
The Bayesian interpretation of the MDL results in a code-length L Bayes ( D ) = − log (cid:82) θ p ( D | θ ) p ( θ ) dθ .
One can show that the Bayesian code is a universal code (Hutter, 2003) and it is equivalent to Bayesian model selection with marginal-likelihood.
The variational approximation to the Bayesian posterior can be considered a two part-code and makes it feasible to obtain an upper bound on the Bayesian description length.
By sampling the parameters from an approximation q ( θ ) to the posterior distribution p ( θ | D , M ) , the code-length is equivalent to the negated ELBO objective in variational inference:
[Equation]
Prequential code.
We can think of prequential coding as forward validation, where the loss incurs when predicting the datapoint y t +1 based on previously datapoints y 1: t .
The code-length
L preq ( D | M ) =
− (cid:80) Tt =1 log p ( y t | ˆ θ ( y <t ) , M ) , where ˆ θ ( y <t ) is an estimator of the parameters after seeing t − 1 datapoints.
Typically, ˆ θ ( · ) is a regularized maximum likelihood estimator, but for many reasonable choices of ˆ θ ( · ) , the resulting prequential (plug-in) code is a universal model despite not having an explicit model complexity term.
In practice, the prequential code can be shorter than that of the approximated Baysian code, especially for neural networks (Blier & Ollivier, 2018).
Unlike Bayesian MDL which requires a Bayesian neural network, prequential MDL can work with any model architecture and optimizer.
A central property that makes the prequential approach attractive for Neural Networks is that any prediction system that predicts the next-step log-loss given the previous data leads to a valid description length.
There are no constraints on the type or implementation of the system as long as it obeys the sequential nature of the data and is deterministic (given the PRNG seeds).
Even a naive system that predicts a uniform distribution over all outcomes constitutes a valid system.
It leads to a description length of
T log( number-of-classes ) and can be considered an upper bound for all sensible systems.
However, the choice and properties of the prequential estimator are important when considering the optimality of the code-length.
Regularized maximum likelihood estimators for the next-step plug-in distribution generally lead to code-lengths that can be considered optimal for that model family because they form a universal-code (Rissanen, 1987).
If it exists, it coincides with the normalized maximum-likelihood code-length up to neglectable offsets (Rissanen, 1987; Grünwald, 2007; Dawid & Vovk, 1999).
SGD as an estimator for Deep Learning.
Deep Neural Networks (DNN) are typically trained with SGD which does not guarantee convergence to a maximum likelihood solution.
Nonetheless, empirically, for ERM, well-tuned SGD finds solutions with good in-distribution generalization.
Equivalently, although we cannot guarantee that DNN+SGD achieves a universal code when used as a next-step predictor in prequential MDL, it still gives a valid code length that can be used to tune hyperparameters.
Just as different techniques developed for DNN with ERM evaluation, techniques can be developed and compared fairly with prequential MDL.
For example, we can compare code length of the different CL techniques or replay strategies to determine their effectiveness.
A. I NDIVIDUAL S EQUENCES : A LGORITHMIC C OMPLEXITY AND S HANNON I NFORMATION T HEORY
MDL is also closely related and sometimes phrased in terms of Algorithmic Complexity Theory (ACT) (Hutter, 2008; Li & Vitányi, 2008) instead of Shannon’s Information Theory (Shannon, 1948).
With ACT the fundamental concept of information is defined in terms of symbols and computation, instead of random variables and distributions.
Quantities such as the entropy of a distribution, the mutual information between random variables and the cross entropy have analogous quantities in
ACT: Kolmogorov complexity, algorithmic mutual information and description length respectively.
In ACT however they are defined for individual sequences without the notion of an underlying probabilistic process.
Grünwald and Vitanyi (Grunwald & Vitanyi, 2004) discuss this formally:
The
Kolmogorov complexity for example is a quantity associated with a concrete, individual sequence of symbols.
If however we consider a stochastic process defining a distribution over such sequences, then the expected Kolmogorov complexity is equal to the Entropy of that process (up to an additive constant).
Similarly, the expected description length of sequences with respect to a model corresponds to the cross-entropy between an empirical distribution and a model distribution.
A similar relationship exists for mutual information and algorithmic mutual information.
Minimizing the description length for a given sequence of observations thus corresponds to the ACT analogon of minimizing the cross-entropy given only a single sample from the sequence-generating process.
Much like the cross-entropy is lower-bounded by the entropy of the empirical distribution, the description length of a sequence is lower-bounded by its Kolmogorov complexity.
A. P ITFALLS OF USING TEST SPLITS : T RAIN - / TEST SET CONTAMINATION
Test set contamination is an increasingly grave unsolved problem that becomes more pressing as machine learning research moves from curated academic benchmark data to large user provided or web-scraped data.
In the recent GPT-3 paper on large scale language modeling the authors dedicate multiple pages in the Appendix to discuss the issue (Brown et al., 2020).
The paper details the method used to detect training-/test set duplicates and presents experiments and the conclusion that their particular model and evaluation approach is relatively insensitive to contamination.
Indeed, some of the contamination was only discovered after the model had been trained, and the model could not be retrained with decontaminated data due to the prohibitive (computational) cost.
This prompted some follow-up discussion on how validation and test sets should be created and decontaminated.
In (Søgaard et al., 2021) the authors show that depending on the exact splitting procedure significantly different results can be obtained.
The discussion lays out arguments in favour of either using deliberately biased splits, or forward-validating on future data.
This work refers to prior work (Gorman & Bedrick, 2019), which also described inconsistent results between different test sets, and between the random split, but at large came to a different conclusion.
To the best of our knowledge these issues have been primarily discussed in the NLP community.
We mention them to point out that validation on held-out data can not blindly be regarded as the gold-standard for model evaluation when, at times, the procedure to create those splits is subject to intense discussion.
Instead, careful consideration will be necessary in order to avoid misleading conclusions based on inappropriate validation and test sets.
Test set contamination is not only an increasingly important problem but also in a certain sense an unsolvable one, because it is ill-defined.
At what threshold for example should a document that verbatimly cites a training-set document be removed from the test-set?
When are two images scraped from the internet essentially the same?
On the other side: Test set contamination is no problem when using any of the MDL variants for model evaluation.
For NML and Bayes this is true because they use model complexity and marginal predictions for regularization instead of held-out sets to avoid overfitting.
But our limited ability to work with good parameter posterior estimates and the benign overfitting of overparametrized NNs make it largely non-applicable to deep learning.
For prequential MDL it is true, because the model is evaluated and hence has to perform well already the first time a (potentially duplicate) data item appears.
For instance, assume the whole photo data base contains three or more (approximate) copies of each photo.
If we randomly split off 10% as the test set, then the train set contains nearly all test set items.
With heldout-validation a pure memorizer without any generalization capacity will perform nearly perfectly on the test set, but will fail miserably in practice on any newly taken photo.
On the other hand, NML and marginal Bayes punish a memorizer due to its huge complexity, and prequential MDL due to the inability to predict (well) the first occurrence of each item.
Test set evaluation is empirically sound for i.i.d.
data, therefore the failure of this paradigm must be attributed to the non-iid nature of data with duplicates Hutter (2022).
For a more detailed discussion, see the FAQ in Hutter (2006/2020).
A. P ROPERTIES OF P MDL BASED MODEL SELECTION
In the following we aim to build some intuition for why pMDL can intuitively be regarded an adequate metric in the i.i.d. scenario. We will consider two situations separately:
The large data regime, where the amount of training data significantly exceeds the amount necessary to train a "good" model: the pMDL score will be dominated by the large number of datapoints where the model’s performance barely improves while learning from more data.
For comparison purposes, the description length can therefore be approximated by # datapoints * generalization-crossentopy (the area under the long tail).
In this scenario pMDL based model selection will basically choose the model with the best generalization performance, just like ERM+validation would.
Only if two models have approximately the same performance in the large data regime, pMDL will use sample-efficiency as a tie-breaker.
In the small-data regime, where each training example can noticeably improve the model, the considerations are different: With ERM one has to choose the fraction of examples used for validation purposes to tune hyperparameters and architectures.
Choosing a small validation set increases variance and thus the risk of choosing hyperparameters that work only well on the validation set (overfitting to the validation set).
With a large validation set however, fewer examples are available for training.
How should one choose the size of the validation set? Choosing hyperparameters with a large validation set and then re-training on the full data bears the risk that optimal hyperparameters change with increasing training set size. With pMDL on the other hand each example is used for validation first and successively available for training: no decision about the split size must be made and all data points are available for parameter estimation. For specific proofs on the identifiability and convergence of MDL based model selection we refer to
Grünwald (2007).
Even assuming optimal continual learners we can not guarantee that MDL will always lead to better decisions than ERM (or vice-versa), especially when considering i.i.d.
scenarios.
However, the arguments laid out weaken the position that ERM+validation should be considered the gold standard without considering its limitations.
B E XPERIMENTAL D ETAILS B. F ORWARD C ALIBRATION
We train the neural network with standard cross-entropy loss and the categorical prediction-head p = softmax( h ) , where h is a vector of logits generated by the neural network.
When using the model for evaluation on next-step data or on test set examples, we replace it with a calibrated prediction head p
= softmax(softplus( β ) h ) , where β is a scalar temperature parameter (Guo et al., 2017).
We use forward-calibration to optimize the calibration parameter β : each new batch of examples from the training sequence is used first for evaluation, then used to perform a single gradient step to optimize β , and finally used for a training to optimizer θ and placed into the replay-buffer.
At the beginning of training we initialize β to β 0 = log(exp(1) − 1) ≈ 0 .
5416 , such that softplus( β 0 )
= 1 .
We use the same optimizer as for the model parameters θ , however scale the learning rate by a factor of √ K , where K is the number of training steps per evaluation step forward (i.e., number replay-streams plus one).
Calibration is computationally almost free: the forward-pass for the calibration gradient step can be shared with the evaluation forward-pass, and backward-propagation for β ends shortly after the softmax layer.
B. MNIST, EMNIST AND RGB-MNIST H YPERPARAMETERS
We run experiments with various model architectures: a) MLP with 1, 2 or 3 hidden layers, 512 units each, with dropout, and ReLU non-linearities, b) LeNet (LeCun et al., 1989) and c) the VGG inspired architecture used by (Blier & Ollivier, 2018), which we call MnistNet here.
We use the same hyperparameter and sampling intervals for all three model architectures.
For the pareto front experiment we extend the number of replay-streams range to 10 . . .
100 ; otherwise we use the same intervals.
[Table]
[Table]
B. CIFAR-10 AND CIFAR-100 WITH VGG++, AND W IDE R ES N ET H YPERPARAMETERS
We use the same hyperparmater sampling intervals for all model architectures.
Following Zagoruyko & Komodakis (2016) we use only minimal augmentations during training:
Images will be horizontally flipped with a probability of 0.5.
For the pareto-front plots we extend the number of epochs (= number of replay-streams) range to 10 . . .
100 and
Parameter
Distribution
Values / Interval
Number of replay-streams log-uniform 25 ...
Learning rate log-uniform 1e-4 ...
3eAdamW ϵ log-uniform 1e-4 ...
EMA step size (Polyak averaging) log-uniform 1e-3 ...
1eWeight decay log-uniform 1e-4 ...1eBatch size uniform {32, 64, 128}
Label smoothing fixed 0.
B. I MAGE N ET WITH VGG++ AND R ES N ETS H YPERPARAMETERS
We use randaugment for data augmentation and the same hyperparameter intervals for all experiments.
For Pareto front experiments we additionally scale the architecture size (number of channels throughout) from 1 / 4 × to 4 × and extend the number of replay-streams interval to 10 . . .
Parameter
Distribution
Values / Interval
Number of replay-streams log-uniform 25 ...
Learning rate log-uniform 1e-4 ...
3eAdamW ϵ log-uniform 1e-4 ...
EMA step size (Polyak averaging) log-uniform 1e-3 ...
1eWeight decay log-uniform 1e-4 ...1.
Batch size fixed
Label smoothing fixed 0.
We report additional baseline metrics for the standard ResNet-50 model on ImageNet, obtained by running the same hyperparameter-sweep:
• Replay Streams: pMDL score 2.21M • Replay Buffer (500k capacity): pMDL score 2.26M nats • Chunkwise-from-scratch (16 split points, <1e19 FLOPs): pMDL score 2.52M nats,
Note that for chunkwise-from-scratch we use the techniques from Bornschein et al.
(2020), who report 3.32M nats. Using better (wider) hyperparameter search and using more split-points we thus found a significantly better result for their method. Using a 4x wider architecture: • Replay Streams: pMDL score 2.18M nats • Replay Buffer (500k capacity): pMDL score 2.
22M
nats • Chunkwise-from-scratch (16 split points, 1e19 FLOPs): pMDL score 2.18M nats,
B. CLOC WITH R ES N ETS H YPERPARAMETERS
We use randaugment for data augmentation and the same hyperparameter intervals for all experiments.
[Table]
C A DDITIONAL E MPIRICAL R ESULTS C. RGB-MNIST F EATURE S ELECTION
To illustrate that the model-selection property demonstrated in section 1 is a robust property of (prequential) MDL, we here present a suite of results on 3 different model architectures: 1) am MLP with 2 hidden layers, 512 units each; 2) the LeNet architecture from (LeCun et al., 1989); and 3) the much higher capacity and better tuned convnet from (Blier & Ollivier, 2018).
In all cases we run the same hyperprameter sweep detailed in Appendix B. We show the regret plot for the best performing model in each sweep relative to the model conditioned only on the R channel.
We see pMDL reliably and with confidence determines that the model conditioned on R only is the most appropriate one for this data; and we see that the small-data performance at the beginning of the sequence is crucial as the relative generalization performance becomes more similar as the length of the sequence increases: [Equation]
Plot
LeNet Regret Plot
MnistNet Regret Plot
MNIST with LeNet
EMNIST with LeNet
CIFAR-10 with VGG
[Equation]
+
CIFAR-100 with VGG++
ImageNet with ResNet-
C.
M ODEL D EPTH A BLATION
C.
W EIGHT -S TANDARDIZATION A BLATION C. N UMBER OF R EPLAY S TREAMS
We run ablations for different number of replay-streams ( K in Algorithm 1) on CIFAR-100 and
ImageNet.
The results on ImageNet show that performing too much replay (choosing K too large) can be harmful.
C. S CALE AND SHAPE OF THE R EPLAY D ISTRIBUTION
Besides of uniform replay for replay-stream training we experimented with alternative replay distributions.
Exponentially decaying replay ( p replay ( a ) = λ exp( − λa ) ) is popular in continual-learning with reservoir sampled replay buffers, but lead to significant longer description lengths and suboptimal results in our experiments.
As an compromise between uniform replay and exponential decay we experimented with long-tailed
Pareto distributions: p replay ( a ) =
[Equation]
The scale parameter λ determines to what extend old examples are replayed and we use a fixed shape parameter α
= log 4 5 ≈ 1 .
16 for all out experiments:
Our experiments suggest that flat, almost uniform replay distributions lead to the shortest description lengths for the sequences from i.i.d.
sources.
On non-stationary sequences however, long-tailed replay that favours recent data, but maintains finite probability of replaying even the earliest examples works best.
C. N ON - STATIONARY SEQUENCES
In this study we concentrate on data sequences from stationary (i.i.d) data sources to ensure we can compare our results with established ERM based training on heldout-data, and, because stationary data is arguably one of the most challenging scenarios for a sequential learner: there is no benefit in focusing on recenly observed data and instead a globally optimal predictive model must be learned.
Nevertheless, one major benefit of MDL over ERM is that it is well defined for individual and nonstationary sequences.
We therefore present a series of experiments on non-stationary data. We leave it to future work to analyze and to improve online learners on non-stationary data {+more thoroughly. C.7. S PLIT CIFAR-  We derive a non-stationary sequence from the CIFAR-10 dataset by treating it as 5 disjunct binary classification tasks between the CIFAR classes 0 vs. 1 (Task-A), 2 vs. 
+}
(Task-B), . . .
, 9 vs(Task-E).
We train the model sequentially on examples from each task, however instead of evaluating forgetting on held-out data after each new task, which is the standard protocol in continual learaning, we here insert new examples from previous tasks into the non-stationary sequence.
A learner can therefore demonstrate good performance by rapidly re-adapting to a previous task from a few examples.
The problem is task agnostic because we do not supply information about the current task-id to the learner.
We concatenate unique examples from the tasks into a joint sequence: [0, 6000):
Task-A, [6000, 13000): Task-B, [13000, 14000): Task-A, [14000, 22000): Task-C, [22000, 23000):
Task-A, [23000, 24000): Task-B, [24000, 33000): Task
[Table]
[Table]
[Table]
[Table]
Task-
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
We run the broad random hyperparameter sweeps for the VGG++ model, however add the scale of the pareto-distributed replay distribution and the capacity of the replay buffer as an additional random hyperparameter between [1000 · · · 50000) .
We show the next-step and cumulative next-step performance for the best performing replay-stream and replay buffer learners:
The regret plot for the same experiments show the difference more clearly.
We also show the regret for the best replay-stream model with various techniques disabled:
Like in the previous plots, the shaded areas indicate the 99% confidence intervals obtained from bootstrap sampling the experiments in the hyperparameter sweep.
We see that multiple replay-stream learners have significantly better performance than the best replay-buffer learner.
Given that this dataset is small and fits completely into RAM, we can close the gap between replay-streams and replay-buffers by configuring a replay-buffer with 50k capacity and using recency-biased sampling; which is the only difference between these learners.
Finally, we run an ablation study with 3 different scales for pareto-distributed replay; and the regret plot for 3 replay-buffer learners with 3 different capacities of the uniform sampled replay-buffer.
The results were obtained by running the same hyperparameter sweeps as before, however with the replay distribution or replay-capacity fixed to 1k, 3k or 10k respectively.
We observe that the width of the replay distribution and the capacity of the replay-buffer trade-off between faster adopting to completely new tasks, and having better performance on recurring tasks:
C.7. V EHICLE C LASSIFICATION D ATA
We run experiments on the vehicle classification data sequence introduced by Duarte & Hu (2004).
The sequence consists of about 80k observations with 100 features each; the classification task is to predict the vehicle category.
We use a 2 hidden-layer MLP as the model architecture with the same hyperparamter-sweep from the VGG++ experiments.
We show the next-step performance, the corresponding regret plots for replay-stream vs replay-buffer training, and regret plots with label smoothing or forward calibration disabled.
Over the whole sequence we obtain an average forward validation error of 15 . 5% (84.5% accuracy).
This is competitive with the more complex and ensemble-based methods described by Celik et al.
(2022) (85% accuracy), and an significant improvement over the baseline methods described therein ( ≈ 82% accuracy).
We highlight that we here essentially just train a neural network with a fixed order of training examples, with recency biased rehearsal, label smoothing and forward-calibration.
C.7. C ONTINUAL GEO - LOCALIZATION (CLOC)
We present results on the CLOC data introduced by Cai et al.
CLOC is a large scale, chronologically sorted image classification sequence consisting of ≈ 39 M images with their geolocation as a categorical label.
For ≈ 5% of the images we either received download errors, or the downloaded content could not be decoded as image data.
In total we were able to obtain 37,093,769 labeled images.
Figure 2 in (Cai et al., 2021) shows that the data is strongly non-stationary; and that traditional ERM training, which treats the data as i.i.d., results in a held-out error rate between 80 and 90%.
Cai et al. (2021) reserve the first 2M ( 5% ) datapoints for pre-training the model statically before entering the online-phase, split-off 1% of the data throughout for validation purposes and only evaluate the images from the first new album in each mini-batch instead of all images.
Their onlinelearning task and metric are thus in spirit similar to the prequential MDL problem we are tackling here, in detail however sufficiently different that the quantitative results are not directly comparable.
Following (Cai et al., 2021) we use a standard ResNet50 architecture and run experiments with replay buffers and replay streams.
The hyperparameter sampling space is detailed in Appendix B.5.
Note that we use a batch size of 128 because small batch-sizes allow the model to more quickly adapt to changes in the data distribution; and a relatively low number of 10 to 25 replay-steps / replay-streams to limit the total computational cost due to the size of the data.
We observe that heavily recency biased replay leads to significantly improved results.
For replay streams we experimented with uniform replay, replay with exponentially decaying priority and with replay governed by a heavy tailed Pareto distribution (see C.6).
From these the heavy tailed Pareto replay leads to the shortest description lengths.
Even more than in the i.i.d.
case, forward-calibration plays an crucial role to obtain good results.
[-We plot the 5k rolling average next-step performance of the best performing model from our random hyperparameter sweep ( K =
-]