Original: Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks
Revised: Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks

Original: Abstract
Revised: Abstract

Original: We propose an algorithm1 that compresses the critical information of a large dataset into compact addressable memories.
Revised: We propose an algorithm that compresses the critical information of a large dataset into compact addressable memories.

Original: These memories can then be recalled to quickly re-train a neural network and recover the performance (instead of storing and re-training on the full original dataset).
Revised: These memories can then be recalled to quickly re-train a neural network and recover the performance (instead of storing and re-training on the full original dataset).

Original: Building upon the dataset distillation framework, we make a key observation that a shared common representation allows for more efﬁcient and effective distillation.
Revised: Building upon the dataset distillation framework, we make a key observation that a shared common representation allows for more efﬁcient and effective distillation.

Original: Concretely, we learn a set of bases (aka “memories”) which are shared between classes and combined through learned ﬂexible addressing functions to generate a diverse set of training examples.
Revised: Concretely, we learn a set of bases (aka “memories”) which are shared between classes and combined through learned ﬂexible addressing functions to generate a diverse set of training examples.

Original: This leads to several beneﬁts: 1) the size of compressed data does not necessarily grow linearly with the number of classes; 2) an overall higher compression rate with more effective distillation is achieved; and 3) more generalized queries are allowed beyond recalling the original classes.
Revised: This leads to several beneﬁts: 1) the size of compressed data does not necessarily grow linearly with the number of classes; 2) an overall higher compression rate with more effective distillation is achieved; and 3) more generalized queries are allowed beyond recalling the original classes.

Original: We demonstrate state-of-the-art results on the dataset distillation task across ﬁve benchmarks, including up to 16.5% and 9.7% in retained accuracy improvement when distilling CIFAR10 and CIFAR100 respectively.
Revised: We demonstrate state-of-the-art results on the dataset distillation task across ﬁve benchmarks, including up to 16.5% and 9.7% in retained accuracy improvement when distilling CIFAR10 and CIFAR100 respectively.

Original: We then leverage our framework to perform continual learning, achieving state-of-the-art results on four benchmarks, with 23.2% accuracy improvement on MANY. 
Revised: We then leverage our framework to perform continual learning, achieving state-of-the-art results on four benchmarks, with 23.2% accuracy improvement on MANY. The code is released on our project webpage 1 .

Original:  Introduction
Revised:  Introduction

Original: Compressing a large amount of information into a small memory storage space is one of the key components in human intelligence [1–3] – a person can retrieve the memories from the past and quickly recover the corresponding skills.
Revised: Compressing a large amount of information into a small memory storage space is one of the key components in human intelligence [1–3] – a person can retrieve the memories from the past and quickly recover the corresponding skills.

Original: Deep learning methods have made large strides in building task-speciﬁc models, but are shown to easily forget the past knowledge when learning new tasks [4, 5].
Revised: Deep learning methods have made large strides in building task-speciﬁc models, but are shown to easily forget the past knowledge when learning new tasks [4, 5].

Original: To equip neural network learners with the memorizing ability, dataset distillation [6] is proposed as a potential solution.
Revised: To equip neural network learners with the memorizing ability, dataset distillation [6] is proposed as a potential solution.

Original: Concretely, a compressed set of examples (memories) is learned to summarize the key information in a dataset that affects model training; these examples can then be used to quickly retrain models and recover the corresponding skills.
Revised: Concretely, a compressed set of examples (memories) is learned to summarize the key information in a dataset that affects model training; these examples can then be used to quickly retrain models and recover the corresponding skills.

Original: This differs from the standard reconstructionbased compression algorithms [7–9] and shows strong performance [10–13].
Revised: This differs from the standard reconstructionbased compression algorithms [7–9] and shows strong performance [10–13].

Original: A critical question in building powerful compressed memories is: what structures and representations should we use to build the memories?
Revised: A critical question in building powerful compressed memories is: what structures and representations should we use to build the memories?

Original: An effective structure and organization of memories can lead to different fundamental assumptions about data and affect the compression and learning behaviours.
Revised: An effective structure and organization of memories can lead to different fundamental assumptions about data and affect the compression and learning behaviours.

Original: Existing works [10, 11, 14, 12, 15, 13, 6] follow a simple representation, where a set of learnable examples is assigned for each class.
Revised: Existing works [10, 11, 14, 12, 15, 13, 6] follow a simple representation, where a set of learnable examples is assigned for each class.

Original: However, under this assumption, the size of the memories can linearly grow with the number of classes, making the distillation of dataset with large number of classes challenging.
Revised: However, under this assumption, the size of the memories can linearly grow with the number of classes, making the distillation of dataset with large number of classes challenging.

Original: Naturally, this can potentially lead to redundancies in the learned memories, due to the separation of data among classes.
Revised: Naturally, this can potentially lead to redundancies in the learned memories, due to the separation of data among classes.

Original: Furthermore, this representation is less generalizable to continuous label space, where inﬁnite number of label values exists.
Revised: Furthermore, this representation is less generalizable to continuous label space, where inﬁnite number of label values exists.

Original: In our paper, we make the observation that there is information shared between classes, and hypothesize that a common and compact representation exists for all classes.
Revised: In our paper, we make the observation that there is information shared between classes, and hypothesize that a common and compact representation exists for all classes.

Original: Following this hypothesis, we propose to formulate the problem as a memory addressing process, where the memories store a common set of bases shared by all classes, and the recombination of bases is performed through an addressing function.
Revised: Following this hypothesis, we propose to formulate the problem as a memory addressing process, where the memories store a common set of bases shared by all classes, and the recombination of bases is performed through an addressing function.

Original: This decomposition between memories and addressing functions enables the possibility that all common information is stored in one part of the representation, and the accessing of the common information depends on the speciﬁc labels and is handled through an extra function.
Revised: This decomposition between memories and addressing functions enables the possibility that all common information is stored in one part of the representation, and the accessing of the common information depends on the speciﬁc labels and is handled through an extra function.

Original: We ﬁnd that this formulation can signiﬁcantly improve both the compression rate and the performance.
Revised: We ﬁnd that this formulation can signiﬁcantly improve both the compression rate and the performance.

Original: We adopt the back-propagation through time learning framework to train the memories and addressing functions, and identify several critical factors that can improve the performance.
Revised: We adopt the back-propagation through time learning framework to train the memories and addressing functions, and identify several critical factors that can improve the performance.

Original: Speciﬁcally, we ﬁnd that adopting the momentum term, and performing long unrolls in the inner optimization loop are both critical.
Revised: Speciﬁcally, we ﬁnd that adopting the momentum term, and performing long unrolls in the inner optimization loop are both critical.

Original: This differs from the common usage of bi-level optimization algorithm on this task [6, 16], and leads to strong performance outperforming single-step gradient matching methods [10, 11] even with the simple data representation.
Revised: This differs from the common usage of bi-level optimization algorithm on this task [6, 16], and leads to strong performance outperforming single-step gradient matching methods [10, 11] even with the simple data representation.

Original: In the experiments, we extensively evaluate our algorithm on ﬁve benchmarks of the Dataset Distillation task, and show that it consistently outperforms previous state-of-the-art by a signiﬁcant margin.
Revised: In the experiments, we extensively evaluate our algorithm on ﬁve benchmarks of the Dataset Distillation task, and show that it consistently outperforms previous state-of-the-art by a signiﬁcant margin.

Original: For example, we achieve 66.4% accuracy on CIFAR10 with the storage space of 1 image per class, improving over the previous state-of-the-art KIP method [12, 13] by 16.5%.
Revised: For example, we achieve 66.4% accuracy on CIFAR10 with the storage space of 1 image per class, improving over the previous state-of-the-art KIP method [12, 13] by 16.5%.

Original: We further demonstrate our method on the continual learning tasks, and show that a simple “compress-then-recall” method using our framework leads to state-of-the-art results on four datasets.
Revised: We further demonstrate our method on the continual learning tasks, and show that a simple “compress-then-recall” method using our framework leads to state-of-the-art results on four datasets.

Original: For example, we outperform all prior methods by 23.2% in retained accuracy on the challenging MANY [17] benchmark.
Revised: For example, we outperform all prior methods by 23.2% in retained accuracy on the challenging MANY [17] benchmark.

Original: Finally, we demonstrate the generality of our approach by extending to image-based (rather than label-based) memory recall, and synthesizing new classiﬁers (unseen during training) from our distilled memories.
Revised: Finally, we demonstrate the generality of our approach by extending to image-based (rather than label-based) memory recall, and synthesizing new classiﬁers (unseen during training) from our distilled memories.

Original:  Related works
Revised:  Related works

Original: Dataset Distillation.
Revised: Dataset Distillation.

Original: The task of dataset distillation is fundamentally a compression problem, with a different prioritization on the information contained in data.
Revised: The task of dataset distillation is fundamentally a compression problem, with a different prioritization on the information contained in data.

Original: There have been several lines of methods, developed with different criteria to prioritize information.
Revised: There have been several lines of methods, developed with different criteria to prioritize information.

Original: Generalization loss with bi-level optimization framework [18–20] has been widely studied and is used in the early works of dataset distillation [6, 16].
Revised: Generalization loss with bi-level optimization framework [18–20] has been widely studied and is used in the early works of dataset distillation [6, 16].

Original: It emphasizes on the loss at the ﬁnal optimization state.
Revised: It emphasizes on the loss at the ﬁnal optimization state.

Original: Gradient-matching or score-matching methods [10, 11, 15] are adopted to directly match the induced gradients from synthetic data.
Revised: Gradient-matching or score-matching methods [10, 11, 15] are adopted to directly match the induced gradients from synthetic data.

Original: If ideally matched over the gradient ﬁeld, the compressed dataset can naturally lead to the same model parameters with gradient descent.
Revised: If ideally matched over the gradient ﬁeld, the compressed dataset can naturally lead to the same model parameters with gradient descent.

Original: Kernel method [13, 12] shows that with the connection to Gaussian processes, a kernel inducing points method can be used to achieve strong performance, but with large computation costs.
Revised: Kernel method [13, 12] shows that with the connection to Gaussian processes, a kernel inducing points method can be used to achieve strong performance, but with large computation costs.

Original: These are also connected with the recent progress on pragmatic compression methods [21, 22], which compress or match distributions based on a decision process (in dataset distillation’s case, the gradient descent search process).
Revised: These are also connected with the recent progress on pragmatic compression methods [21, 22], which compress or match distributions based on a decision process (in dataset distillation’s case, the gradient descent search process).

Original: Continual learning.
Revised: Continual learning.

Original: Broadening the learning paradigms, continual learning problem aims to build agents that learn through a stream of tasks and accrue knowledge along the process.
Revised: Broadening the learning paradigms, continual learning problem aims to build agents that learn through a stream of tasks and accrue knowledge along the process.

Original: “Catastrophic forgetting” [5, 23, 4] is a well-known phenomenon in this setting, where the neural network forgets previous skills when learning new ones.
Revised: “Catastrophic forgetting” [5, 23, 4] is a well-known phenomenon in this setting, where the neural network forgets previous skills when learning new ones.

Original: Various methods [24–27, 17, 28–35] on regularization, replay, or dynamic model, have been proposed to alleviate the issue and address the “stabilityplasticity dilemma” [36, 37].
Revised: Various methods [24–27, 17, 28–35] on regularization, replay, or dynamic model, have been proposed to alleviate the issue and address the “stabilityplasticity dilemma” [36, 37].

Original: Memory buffer has been a critical component in the past methods [24– 27, 17, 28, 38, 39], but mainly relies on random selection of real samples with different strategies.
Revised: Memory buffer has been a critical component in the past methods [24– 27, 17, 28, 38, 39], but mainly relies on random selection of real samples with different strategies.

Original: Recently, several works extend the usage of memory to storing random basis [39] (online setting) or
Revised: Recently, several works extend the usage of memory to storing random basis [39] (online setting) or

Original: SVD bases [38] (ofﬂine setting).
Revised: SVD bases [38] (ofﬂine setting).

Original:  Background: dataset distillation
Revised:  Background: dataset distillation

Original: The task of Dataset Distillation [6] is proposed to compress the key information of a large-scale training dataset into a small amount of learned data, which can be stored using limited memory space and retrieved through label indices or task information to recover the performance of a model.
Revised: The task of Dataset Distillation [6] is proposed to compress the key information of a large-scale training dataset into a small amount of learned data, which can be stored using limited memory space and retrieved through label indices or task information to recover the performance of a model.

Original: Problem Setting.
Revised: Problem Setting.

Original: Formally, given a large dataset D tr = { ( x i , y i ) } Ni =1 containing N pairs of training data ( x i , y i ) , where x i is an image and y
Revised: Formally, given a large dataset D tr = { ( x i , y i ) } Ni =1 containing N pairs of training data ( x i , y i ) , where x i is an image and y

Original: i is the corresponding label in C classes, a small dataset
Revised: i is the corresponding label in C classes, a small dataset

Original: D s = { ( x (cid:48)
Revised: D s = { ( x (cid:48)

Original: j , y j ) } N (cid:48)
Revised: j , y j ) } N (cid:48)

Original: j =1 , N (cid:48) (cid:28) N, can be synthesized or distilled, such that a model trained on D s can have the same generalization ability with ones trained on D tr :
Revised: j =1 , N (cid:48) (cid:28) N, can be synthesized or distilled, such that a model trained on D s can have the same generalization ability with ones trained on D tr :

Original: [Equation]
Revised: [Equation]

Original: where θ ( ∗ ) and θ (cid:48) ( ∗ ) are the optimized parameters using D tr and D s respectively,
Revised: where θ ( ∗ ) and θ (cid:48) ( ∗ ) are the optimized parameters using D tr and D s respectively,

Original: m is a metric, e.g., accuracy, and D te is the test dataset.
Revised: m is a metric, e.g., accuracy, and D te is the test dataset.

Original: The model is often a neural network classiﬁer f ( · ; θ ) parameterized by θ and trained with a loss function (cid:96) ( f ( x ; θ ) , y ) .
Revised: The model is often a neural network classiﬁer f ( · ; θ ) parameterized by θ and trained with a loss function (cid:96) ( f ( x ; θ ) , y ) .

Original: Synthetic dataset representations.
Revised: Synthetic dataset representations.

Original: The synthetic dataset contains the core information that needs to be learned.
Revised: The synthetic dataset contains the core information that needs to be learned.

Original: The representation of the synthetic data affects the compactness and effectiveness of the distillation process.
Revised: The representation of the synthetic data affects the compactness and effectiveness of the distillation process.

Original: In existing methods [6, 10, 11, 14, 12, 15], the dataset D s is deﬁned as a collection of learnable data samples ( x (cid:48) , y ) , and the number of samples is separately and equally distributed across classes.
Revised: In existing methods [6, 10, 11, 14, 12, 15], the dataset D s is deﬁned as a collection of learnable data samples ( x (cid:48) , y ) , and the number of samples is separately and equally distributed across classes.

Original: This representation has several disadvantages: ﬁrst, the number of synthetic data samples needed for a dataset grows linearly with the number of classes, leading to limited applicability when the number of classes is large or undeﬁned (e.g., language or other continuous labels); second, the potentially shared and common information across classes is ignored – this results in a less compact representation of the distilled information and lower compression rate; lastly, the representation is not able to generalize to new classes or tasks, due to the lack of common representation learned across classes.
Revised: This representation has several disadvantages: ﬁrst, the number of synthetic data samples needed for a dataset grows linearly with the number of classes, leading to limited applicability when the number of classes is large or undeﬁned (e.g., language or other continuous labels); second, the potentially shared and common information across classes is ignored – this results in a less compact representation of the distilled information and lower compression rate; lastly, the representation is not able to generalize to new classes or tasks, due to the lack of common representation learned across classes.

Original:  Model
Revised:  Model

Original: Overview.
Revised: Overview.

Original: In this section, ﬁrst, we present a new perspective of the problem, where the Dataset
Revised: In this section, ﬁrst, we present a new perspective of the problem, where the Dataset

Original: Distillation problem is formulated as a memory addressing process : instead of learning synthetic images separately for each class, we construct and learn a common memory representation that can be accessed through addressing matrices to construct synthetic datasets.
Revised: Distillation problem is formulated as a memory addressing process : instead of learning synthetic images separately for each class, we construct and learn a common memory representation that can be accessed through addressing matrices to construct synthetic datasets.

Original: Under this formulation, the number of synthetic images does not need to grow linearly with the number of classes, the shared information among classes can be exploited to reduce redundancies and improve compression rate, and datasets can be distilled with respect to more generic queries.
Revised: Under this formulation, the number of synthetic images does not need to grow linearly with the number of classes, the shared information among classes can be exploited to reduce redundancies and improve compression rate, and datasets can be distilled with respect to more generic queries.

Original: Second, we further show several critical empirical facets of back-propagation through time framework, which lead to drastic improvements on the performance and outperform the single-step gradient matching methods.
Revised: Second, we further show several critical empirical facets of back-propagation through time framework, which lead to drastic improvements on the performance and outperform the single-step gradient matching methods.

Original: This is in contrast with the current common observation that gradient matching outperforms back-propagation through time framework on dataset distillation tasks.
Revised: This is in contrast with the current common observation that gradient matching outperforms back-propagation through time framework on dataset distillation tasks.

Original: In the following, we present the two core components of our method, (1) the new formulation of dataset distillation, with memories and addressing matrices in Sec. 4.1, and (2) the learning framework under back-propagation through time in Sec.
Revised: In the following, we present the two core components of our method, (1) the new formulation of dataset distillation, with memories and addressing matrices in Sec. 4.1, and (2) the learning framework under back-propagation through time in Sec.

Original: 4.1 Dataset distillation as memory addressing
Revised: 4.1 Dataset Distillation as memory addressing

Original: Problem formulation.
Revised: Problem formulation.

Original: Given a task-speciﬁc dataset D tr = { x i , y i } N i =1 , x ∈ X , y ∈ Y , we aim to learn a single compact and compressed representation M , referred to as memories, that can be accessed through a learned addressing function A ( · ) .
Revised: Given a task-speciﬁc dataset D tr = { x i , y i } N i =1 , x ∈ X , y ∈ Y , we aim to learn a single compact and compressed representation M , referred to as memories, that can be accessed through a learned addressing function A ( · ) .

Original: A ( · ) takes all possible values of y as input and recalls the corresponding synthetic data.
Revised: A ( · ) takes all possible values of y as input and recalls the corresponding synthetic data.

Original: With a set of { y i } , a synthetic dataset recalled using the above process can train a model f θ : X → Y from scratch and obtain the same generalization ability as trained on D tr .
Revised: With a set of { y i } , a synthetic dataset recalled using the above process can train a model f θ : X → Y from scratch and obtain the same generalization ability as trained on D tr .

Original: Ideally, the memories M and the addressing function A can jointly capture the critical information that deﬁnes the task mapping from X to Y , such that the recalled synthetic data given a query y
Revised: Ideally, the memories M and the addressing function A can jointly capture the critical information that deﬁnes the task mapping from X to Y , such that the recalled synthetic data given a query y

Original: i contains distinctive information that deﬁnes y i and the synthetic dataset recalled with { y i } can satisfy eqn.
Revised: i contains distinctive information that deﬁnes y i and the synthetic dataset recalled with { y i } can satisfy eqn.

Original: 1 when used for re-training.
Revised: 1 when used for re-training.

Original: For example, in the standard classiﬁcation tasks, we can enumerate all possible values in the label space (discrete) and address the memories, to construct the synthetic dataset that contains critical information.
Revised: For example, in the standard classiﬁcation tasks, we can enumerate all possible values in the label space (discrete) and address the memories, to construct the synthetic dataset that contains critical information.

Original: Under this formulation, since we are learning a single, shared and accessible representation for all y s, the size of memories can be deﬁned ﬂexibly regardless of the number of classes, removing the linear growth limitation in the standard distillation settings.
Revised: Under this formulation, since we are learning a single, shared and accessible representation for all y s, the size of memories can be deﬁned ﬂexibly regardless of the number of classes, removing the linear growth limitation in the standard distillation settings.

Original: There is also no constraint on the form of y s, which can be either discrete or continuous.
Revised: There is also no constraint on the form of y s, which can be either discrete or continuous.

Original: The storage budget or compression rate is calculated by considering the storage space of parameters in both memories and addressing functions, which should be as compact as possible.
Revised: The storage budget or compression rate is calculated by considering the storage space of parameters in both memories and addressing functions, which should be as compact as possible.

Original: [Figure]
Revised: [Figure]

Original: Memory representation.
Revised: Memory representation.

Original: We use a set of bases to store in the memories M = { b 1 , ..., b K } , where each vector b k ∈ R d has the same dimension as x ∈ R d , and all vectors collectively deﬁne the intrinsic components in a dataset that characterize the task mapping from X to Y .
Revised: We use a set of bases to store in the memories M = { b 1 , ..., b K } , where each vector b k ∈ R d has the same dimension as x ∈ R d , and all vectors collectively deﬁne the intrinsic components in a dataset that characterize the task mapping from X to Y .

Original: Through re-using the bases, we can produce a desired synthetic dataset for model re-training.
Revised: Through re-using the bases, we can produce a desired synthetic dataset for model re-training.

Original: Spatial redundancies in images: note that, as a special case, images can contain redundancies spatially and be stored in a downsampled version to improve the compression rate.
Revised: Spatial redundancies in images: note that, as a special case, images can contain redundancies spatially and be stored in a downsampled version to improve the compression rate.

Original: The downsampled image bases can be passed via a deterministic upsampling process (e.g., bilinear interpolation) to recover the original resolution.
Revised: The downsampled image bases can be passed via a deterministic upsampling process (e.g., bilinear interpolation) to recover the original resolution.

Original: Memory addressing.
Revised: Memory addressing.

Original: For each query y , we use a parameterized function A ( y ) to re-combine the bases in the memories M .
Revised: For each query y , we use a parameterized function A ( y ) to re-combine the bases in the memories M .

Original: Similar to previous methods on accessing memories, we use A to produce a set of coefﬁcients, and linearly combine the bases to produce synthetic data.
Revised: Similar to previous methods on accessing memories, we use A to produce a set of coefﬁcients, and linearly combine the bases to produce synthetic data.

Original: Formally, to retrieve r synthetic examples for each y , we deﬁne a set of matrices { A 1 , ..., A r } , A i ∈ R d y × K , where d y is the dimension size of y and r is the number of data samples that can be retrieved.
Revised: Formally, to retrieve r synthetic examples for each y , we deﬁne a set of matrices { A 1 , ..., A r } , A i ∈ R d y × K , where d y is the dimension size of y and r is the number of data samples that can be retrieved.

Original: With the memories
Revised: With the memories

Original: M = { b 1 , ..., b K } , we deﬁne:
Revised: M = { b 1 , ..., b K } , we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: where y ∈ R d y × 1 is in a vectorized form, such one-hot encoding of categorical labels, and v =
Revised: where y ∈ R d y × 1 is in a vectorized form, such one-hot encoding of categorical labels, and v =

Original: y T A
Revised: y T A

Original: i corresponds to a coefﬁcient vector v that combines the bases.
Revised: i corresponds to a coefﬁcient vector v that combines the bases.

Original: The produced synthetic data x (cid:48) is paired with y as the corresponding label.
Revised: The produced synthetic data x (cid:48) is paired with y as the corresponding label.

Original: Our model is shown on the right of ﬁgure 1.
Revised: Our model is shown on the right of ﬁgure 1.

Original: Constructing a dataset.
Revised: Constructing a dataset.

Original: To construct a synthetic dataset D s for model re-training, we are often given a set of samples { y i } , or can enumerate all possible values of y (if discrete).
Revised: To construct a synthetic dataset D s for model re-training, we are often given a set of samples { y i } , or can enumerate all possible values of y (if discrete).

Original: With the set { y i } , we use eqn.2 to address and retrieve the synthetic dataset D y =
Revised: With the set { y i } , we use eqn.2 to address and retrieve the synthetic dataset D y =

Original: y i s =
Revised: y i s =

Original: { ( x (cid:48)
Revised: { ( x (cid:48)

Original: j , y i ) } rj =1 for each y i .
Revised: j , y i ) } rj =1 for each y i .

Original: The ﬁnal dataset is the union of all D s =
Revised: The ﬁnal dataset is the union of all D s =

Original: (cid:83)
Revised: (cid:83)

Original: D y
Revised: D y

Original: =
Revised: =

Original: y i s .
Revised: y i s .

Original: The dataset D s can be used in either a minibatch form for stochastic gradient descent, or as a whole for batch gradient descent.
Revised: The dataset D s can be used in either a minibatch form for stochastic gradient descent, or as a whole for batch gradient descent.

Original: Generalized possibilities of queries.
Revised: Generalized possibilities of queries.

Original: Another advantage of our formulation, besides a compact and shared representation, is the various possibilities of queries y .
Revised: Another advantage of our formulation, besides a compact and shared representation, is the various possibilities of queries y .

Original: In principle, under this formulation, the label y can be ﬂexibly deﬁned as other forms, such as language or audio, where the representation resides in a continuous space or follows a distribution p ( h ( y )) deﬁned by a feature extractor h ( · ) .
Revised: In principle, under this formulation, the label y can be ﬂexibly deﬁned as other forms, such as language or audio, where the representation resides in a continuous space or follows a distribution p ( h ( y )) deﬁned by a feature extractor h ( · ) .

Original: The set of { y i } then can be sampled from the distribution p instead of having to enumerate all possible values.
Revised: The set of { y i } then can be sampled from the distribution p instead of having to enumerate all possible values.

Original: This provides a general way of compressing or distilling a large dataset without constraint on the forms of labels.
Revised: This provides a general way of compressing or distilling a large dataset without constraint on the forms of labels.

Original: Connection with standard Dataset Distillation.
Revised: Connection with standard Dataset Distillation.

Original: In the standard setting, dataset distillation is deﬁned for classiﬁcation tasks with discrete labels and each label owns its unique set of synthetic data.
Revised: In the standard setting, dataset distillation is deﬁned for classiﬁcation tasks with discrete labels and each label owns its unique set of synthetic data.

Original: We can show that this is a special case of our formulation: if the bases are constructed as the collection of those label-speciﬁc synthetic data ( K = N (cid:48) , N (cid:48) is the total size of the synthetic dataset), the addressing matrices
Revised: We can show that this is a special case of our formulation: if the bases are constructed as the collection of those label-speciﬁc synthetic data ( K = N (cid:48) , N (cid:48) is the total size of the synthetic dataset), the addressing matrices

Original: A i are deﬁned in space { 0 , 1 } C × N (cid:48)
Revised: A i are deﬁned in space { 0 , 1 } C × N (cid:48)

Original: where A i [ m, n ] = 1 if n equals
Revised: where A i [ m, n ] = 1 if n equals

Original: Algorithm
Revised: Algorithm

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: Converge to m ( N (cid:48) /C )
Revised: Converge to m ( N (cid:48) /C )

Original: + i ( i th item of m th class) and A i [ m, n ] = 

Revised: + i ( i th item of m th class) and A i [ m, n ] =

Original: at other positions, then the "retrieval" process can also be deﬁned as eqn.
Revised: at other positions, then the "retrieval" process can also be deﬁned as eqn.

Original: 4.2 Learning framework: back-propagation through time
Revised: 4.2 Learning framework: back-propagation through time

Original: In this section, we build upon the back-propagation through time algorithm and discuss in detail the learning framework that performs the distillation process from a dataset to memories and addressing functions.
Revised: In this section, we build upon the back-propagation through time algorithm and discuss in detail the learning framework that performs the distillation process from a dataset to memories and addressing functions.

Original: Starting from notations, we deﬁne the parameters contained in both memories and addressing functions as φ , which are collectively optimized.
Revised: Starting from notations, we deﬁne the parameters contained in both memories and addressing functions as φ , which are collectively optimized.

Original: A loss function is (cid:96) ( · , · ) is deﬁned on a task-speciﬁc dataset.
Revised: A loss function is (cid:96) ( · , · ) is deﬁned on a task-speciﬁc dataset.

Original: We denote an optimization algorithm as OPT ( · , · ; α, β, (cid:96), T ) , where α and β are the learning rate and the momentum rate, respectively, and T is the number of optimization steps.
Revised: We denote an optimization algorithm as OPT ( · , · ; α, β, (cid:96), T ) , where α and β are the learning rate and the momentum rate, respectively, and T is the number of optimization steps.

Original: For a sinlge step optimization, we denote it as OPT - STEP ( · , · ; α, β ) .
Revised: For a sinlge step optimization, we denote it as OPT - STEP ( · , · ; α, β ) .

Original: To learn the parameters φ = {M , A} , we follow a standard bi-level optimization framework with back-propagation through time (BPTT), where the inner-loop uses the synthetic dataset D s to train a randomly initialized model starting from scratch, and a generalization loss is computed using a minibatch B = { ( x i , y i ) } sampled from D tr .
Revised: To learn the parameters φ = {M , A} , we follow a standard bi-level optimization framework with back-propagation through time (BPTT), where the inner-loop uses the synthetic dataset D s to train a randomly initialized model starting from scratch, and a generalization loss is computed using a minibatch B = { ( x i , y i ) } sampled from D tr .

Original: The parameters φ are implicitly contained in the synthetic dataset D s and optimized when minimizing the generalization loss.
Revised: The parameters φ are implicitly contained in the synthetic dataset D s and optimized when minimizing the generalization loss.

Original: The bi-level optimization deﬁnes:
Revised: The bi-level optimization deﬁnes:

Original: [Equation]
Revised: [Equation]

Original: to
Revised: to

Original: [Equation]
Revised: [Equation]

Original: where θ 0 represents the initializing parameters, θ ∗ is the optimized model parameters in the inner optimization loop, α 0 and β 0 are the learning rate and momentum rate for opt ( · , · ) , and J ( φ ) is the generalization loss on minibatch B .
Revised: where θ 0 represents the initializing parameters, θ ∗ is the optimized model parameters in the inner optimization loop, α 0 and β 0 are the learning rate and momentum rate for opt ( · , · ) , and J ( φ ) is the generalization loss on minibatch B .

Original: In practice, for each inner loop training, we random sample a subset Y (cid:48) from y s and retrieve the corresponding subset D Y (cid:48) s .
Revised: In practice, for each inner loop training, we random sample a subset Y (cid:48) from y s and retrieve the corresponding subset D Y (cid:48) s .

Original: This reduces the computation cost in inner loops.
Revised: This reduces the computation cost in inner loops.

Original: We empirically observe that equivalent results can be achieved with faster runtime.
Revised: We empirically observe that equivalent results can be achieved with faster runtime.

Original: The algorithm is summarized in Alg.
Revised: The algorithm is summarized in Alg.

Original: 1, where lines 8-13 deﬁne the inner loop optimization process
Revised: 1, where lines 8-13 deﬁne the inner loop optimization process

Original: OPT ( · , · ; α, β, (cid:96), T ) , and the gradients of generalization loss (line 15) is back-propagated through the inner loop to update φ .
Revised: OPT ( · , · ; α, β, (cid:96), T ) , and the gradients of generalization loss (line 15) is back-propagated through the inner loop to update φ .

Original: Note that, in principle, the inner loop optimization can be performed using any optimizer.
Revised: Note that, in principle, the inner loop optimization can be performed using any optimizer.

Original: In this paper, we mainly rely on the standard stochastic gradient descent with momentum to train the distilled data.
Revised: In this paper, we mainly rely on the standard stochastic gradient descent with momentum to train the distilled data.

Original: Critical factors in BPTT.
Revised: Critical factors in BPTT.

Original: Although being a natural choice in performing dataset distillation and adopted in the original work [6], the BPTT framework has been shown to underperform other algorithms, such as single-step gradient matching methods [10, 11], on various benchmarks.
Revised: Although being a natural choice in performing dataset distillation and adopted in the original work [6], the BPTT framework has been shown to underperform other algorithms, such as single-step gradient matching methods [10, 11], on various benchmarks.

Original: The underlying causes that hinder the performance of the algorithm are still underexplored.
Revised: The underlying causes that hinder the performance of the algorithm are still underexplored.

Original: In our work, we investigate and identify the factors that can unleash potential of back-propagation through time framework on dataset distillation and lead to signiﬁcant performance boosts.
Revised: In our work, we investigate and identify the factors that can unleash potential of back-propagation through time framework on dataset distillation and lead to signiﬁcant performance boosts.

Original: Momentum term.
Revised: Momentum term.

Original: In previous dataset distillation works [6, 16], the usage of back-propagation through time framework omits the momentum term in the inner loop optimization.
Revised: In previous dataset distillation works [6, 16], the usage of back-propagation through time framework omits the momentum term in the inner loop optimization.

Original: Indeed, this has been a common practice in meta-learning tasks [18, 40].
Revised: Indeed, this has been a common practice in meta-learning tasks [18, 40].

Original: Adding momentum term in meta-learning can potentially even hurt the performance and lead to less gradient diversity [40].
Revised: Adding momentum term in meta-learning can potentially even hurt the performance and lead to less gradient diversity [40].

Original: However, we observe that, in dataset distillation tasks, the momentum term is crucial for making BPTT excel, even in the relatively short inner loop optimization settings (e.g. 10 or 20 steps).
Revised: However, we observe that, in dataset distillation tasks, the momentum term is crucial for making BPTT excel, even in the relatively short inner loop optimization settings (e.g. 10 or 20 steps).

Original: We provide results and analysis in the experiments section.
Revised: We provide results and analysis in the experiments section.

Original: Long unrolled trajectories.
Revised: Long unrolled trajectories.

Original: Another aspect in using BPTT in dataset distilltation is the length of unrolled optimization trajectories in the inner loops.
Revised: Another aspect in using BPTT in dataset distilltation is the length of unrolled optimization trajectories in the inner loops.

Original: The previous usage of BPTT on this task [6, 16] adopts relative short inner loop optimization trajectories (e.g. 10-30 steps).
Revised: The previous usage of BPTT on this task [6, 16] adopts relative short inner loop optimization trajectories (e.g. 10-30 steps).

Original: Instead, we show that unrolling the trajectories long enough (e.g. 200 steps) with momentum terms can potentially produce θ ∗ that better summarizes the information contained in memories and addressing matrices, generating more effective gradients to learn the compressed representation.
Revised: Instead, we show that unrolling the trajectories long enough (e.g. 200 steps) with momentum terms can potentially produce θ ∗ that better summarizes the information contained in memories and addressing matrices, generating more effective gradients to learn the compressed representation.

Original:  Experiments
Revised:  Experiments

Original: We thoroughly evaluate our model and demonstrate the beneﬁts previous methods.
Revised: We thoroughly evaluate our model and demonstrate the beneﬁts previous methods.

Original: In section 5.1, we show that using a shared representation is critical to improving the distillation performance and compression rates.
Revised: In section 5.1, we show that using a shared representation is critical to improving the distillation performance and compression rates.

Original: Speciﬁcally, we observe that there is strong evidence that there is information re-using across classes.
Revised: Speciﬁcally, we observe that there is strong evidence that there is information re-using across classes.

Original: We further show the beneﬁts of our model on standard continual learning tasks in section 5.2.
Revised: We further show the beneﬁts of our model on standard continual learning tasks in section 5.2.

Original: For example, we observe that a simple “compress-then-recall” method can achieve performance outperforming state-of-the-art continual learning models with complex designs.
Revised: For example, we observe that a simple “compress-then-recall” method can achieve performance outperforming state-of-the-art continual learning models with complex designs.

Original: Finally, in section 5.3, we show that storing the compressed data enables synthesizing new classiﬁers (section 5.3.1) and the shared representation formulation allows more general queries (section 5.3.2), which can be continuous (e.g. image features).
Revised: Finally, in section 5.3, we show that storing the compressed data enables synthesizing new classiﬁers (section 5.3.1) and the shared representation formulation allows more general queries (section 5.3.2), which can be continuous (e.g. image features).

Original: 5.1 Dataset Distillation
Revised: 5.1 Dataset Distillation

Original: In this section, we follow the standard setting of dataset distillation, and performs dataset compression that can be recalled with discrete class labels.
Revised: In this section, we follow the standard setting of dataset distillation, and performs dataset compression that can be recalled with discrete class labels.

Original: Datasets.
Revised: Datasets.

Original: We test our models on ﬁve standard dataset distillation benchmarks: MNIST [41], FashionMNIST [42], SVHN [43], CIFAR10 [44], CIFAR100 [44].
Revised: We test our models on six standard dataset distillation benchmarks: MNIST [41], FashionMNIST [42], SVHN [43], CIFAR10 [44], CIFAR100 [44], and TinyImageNet [45].

Original: MNIST contains 10 classes with 60, writing digit images as training and 10,000 as test set.
Revised: MNIST contains 10 classes with 60,000 writing digit images as training and 10,000 as test set.

Original: The images are gray-scale with a shape of 28 × 28 .
Revised: The images are gray-scale with a shape of 28 × 28 .

Original: FashionMNIST is a dataset with clothing and shoe images and consists of a training with size 60,000 and a test set with size 10,000.
Revised: FashionMNIST is a dataset with clothing and shoe images and consists of a training with size 60,000 and a test set with size 10,000.

Original: Each image is 28 × 28 in gray scale, and has a label from 10 classes.
Revised: Each image is 28 × 28 in gray scale, and has a label from 10 classes.

Original: SVHN contains street digit images where each image has shape 32 × 32 × 3 .
Revised: SVHN contains street digit images where each image has shape 32 × 32 × 3 .

Original: The dataset contains 73,257 images for training and 26,032 images for testing.
Revised: The dataset contains 73,257 images for training and 26,032 images for testing.

Original: CIFAR10 and CIFAR100 are color image datasets, with 50,000 training images and 10,000 testing images on each. 
Revised: CIFAR10 and CIFAR100 are color image datasets, with 50,000 training images and 10,000 testing images on each. CIFAR10 has 10 classes with 5,000 images per class, and CIFAR100 has 100 classes with 500 images per class. TinyImageNet

Original: CIFAR10 has 10 classes with5,000 images per class, and CIFAR100 has 100 classes with 500 images per class.
Revised: [45] contains 200 categories with images of resolution 64x64. The training and testing sets have 100,000 and 10,000 images respectively.

Original: Experiment settings.
Revised: Experiment settings.

Original: We evaluate our distillation models under three different memory budgets for each dataset: 1/10/50 images per class.
Revised: We evaluate our distillation models under three different memory budgets for each dataset: 1/10/50 images per class.

Original: We focus on high compression rate scenarios and consider the 1 and 10 settings for CIFAR100.
Revised: We focus on high compression rate scenarios and consider the 1 and 10 settings for CIFAR100.

Original: Following previous works [14, 11, 12, 15], the main network architecture used in experiments is a simple convolutional network (ConvNet) with3 × 3 ﬁlters,
Revised: Following previous works [14, 11, 12, 15], the main network architecture used in experiments is a simple convolutional network (ConvNet) with

Original: InstanceNorm, ReLU and 2 × 2 average pooling.

Original: For the evaluation protocol, each model is evaluated on 20 randomly initialized models, trained for 300 epochs on a synthetic dataset, and tested on a held-out testing dataset.

Original: We use one GPU per experiment run.

Original: Memory budget calculation.
Revised: Memory budget calculation.

Original: Since our model uses memories and addressing matrices to store the compressed information, we treat the total number of images as a memory storage budget.
Revised: Since our model uses memories and addressing matrices to store the compressed information, we treat the total number of images as a memory storage budget.

Original: Whencomparing with N images for C classes, we ensure [Equation]
Revised: When

Original: [Table]
Revised: [Table]

Original: a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h o r s e s h i p t r u c k airplane automobile
Revised: comparing with N images for C classes

Original: [Figure]

Original: [Table]

Original: m

Original: [Table]
Revised: [Table]

Original: p l e t r ee

Original: o a k t r ee p a l m t r ee p i n e t r ee w ill o w t r ee b i

Original: c y c l e b u s m o t

Original: o r c y c l e p i c k u p t
Revised: we ensure

Original: [Figure]

Original: [Figure]
Revised: [Equation]

Original: where images is the total number of images for all classes and size ( · ) is the total size of a tensor.
Revised: where images is the total number of images for all classes and size ( · ) is the total size of a tensor.

Original: We assume all tensors are in the format of ﬂoat numbers.
Revised: We assume all tensors are in the format of ﬂoat numbers.

Original: For a certain number of bases
Revised: For a certain number of bases

Original: [Figure]
Revised: [Table]

Original: we calculate the corresponding maximum number of addressing matrices allowed using eqn.
Revised: we calculate the corresponding maximum number of addressing matrices allowed using eqn.

Original: 4 and use the integer lowerbound as the ﬁnal value.
Revised: 4 and use the integer lowerbound as the ﬁnal value.

Original: [Figure]
Revised: [Figure]

Original: Model details.
Revised: Model details.

Original: We use bases with spatial resolution downsampled by a factor of 2 in both height and width from the standard image size based on datasets.
Revised: We use bases with spatial resolution downsampled by a factor of 2 in both height and width from the standard image size based on datasets.

Original: All models are trained for 50k iterations with SGD optimizer.
Revised: All models are trained for 50k iterations with SGD optimizer.

Original: For the inner loop optimization, we set the momentum rate as 0.9, and use 150 steps for small memory budgets 1 and 10, and 200 steps for budget 50.
Revised: For the inner loop optimization, we set the momentum rate as 0.9, and use 150 steps for small memory budgets 1 and 10, and 200 steps for budget 50.

Original: The selection of number of bases to use is based on a held-out validation set (10% of training set), an example is shown in ﬁgure 2, where different numbers of bases and addressing matrices (calculated with eqn.
Revised: The selection of number of bases to use is based on a held-out validation set (10% of training set), an example is shown in ﬁgure 2, where different numbers of bases and addressing matrices (calculated with eqn.

Original: 4) have impacts on accuracies.
Revised: 4) have impacts on accuracies.

Original: More details can be found in the appendix.
Revised: More details can be found in the appendix.

Original: Result 1: state-of-the-art accuracy.
Revised: Result 1: state-of-the-art accuracy.

Original: We compare our model with previous methods: Dataset Condensation (DC)
Revised: We compare our model with previous methods: Dataset Condensation (DC)

Original: [10], Differentiable Siamese Augmentation (DSA) [11],
Revised: [10], Differentiable Siamese Augmentation (DSA) [11],

Original: Kernel Inducing Points (KIP) [12], Distribution Matching (DM) [14], Aligning Features (CAFE) [45] and Trajectory Matching (TM) [15].
Revised: Kernel Inducing Points (KIP) [12], Distribution Matching (DM) [14], Aligning Features (CAFE) [46] and Trajectory Matching (TM) [15].

Original: The results are summarized in table 1.
Revised: The results are summarized in table 1.

Original: Following previous methods [11, 12, 15], we adopt simple data augmentations and preprocessings: ﬂip and rotation on CIFAR10 and CIFAR datasets, and ZCA on SVHN, CIFAR10 and CIFAR100.
Revised: Following previous methods [11, 12, 15], we adopt simple data augmentations and preprocessings: ﬂip and rotation on CIFAR10 and CIFAR datasets, and ZCA on SVHN, CIFAR10 and CIFAR100.

Original: As shown in the table 1, our model consistently outperforms previous methods, especially under high compression rate cases where only 1 image is allowed per class.
Revised: As shown in the table 1, our model consistently outperforms previous methods, especially under high compression rate cases where only 1 image is allowed per class.

Original: For example, we achieve 87.3% and 66.4% on SVHN and CIFAR with 1 image per class, outperforming prior arts by 30% and 16.5% under the same storage budget, respectively, and even beat the performance of previous methods using 10 images per class.
Revised: For example, we achieve 87.3% and 66.4% on SVHN and CIFAR with 1 image per class, outperforming prior arts by 30% and 16.5% under the same storage budget, respectively, and even beat the performance of previous methods using 10 images per class.

Original: Analysis: information sharing across classes.
Revised: Analysis: information sharing across classes.

Original: The core observation in our method is that a common representation can enable information sharing across classes and reduce redundancies.
Revised: The core observation in our method is that a common representation can enable information sharing across classes and reduce redundancies.

Original: To verify this, we calculate the average coefﬁcients ¯ v = 1 r (cid:80)
Revised: To verify this, we calculate the average coefﬁcients ¯ v = 1 r (cid:80)

Original: r − 1 i =

Revised: r − 1 i =

Original: y T A i for each class y and visualize the cosine similarities of ¯ v from two classes.
Revised: y T A i for each class y and visualize the cosine similarities of ¯ v from two classes.

Original: The visualizations are shown in ﬁg. 3.
Revised: The visualizations are shown in ﬁg. 3.

Original: Higher cosine similarity scores indicate that two classes are utilizing similar bases components in the memories to produce synthetic images.
Revised: Higher cosine similarity scores indicate that two classes are utilizing similar bases components in the memories to produce synthetic images.

Original: For example, in CIFAR100 (right one of ﬁgure 3), classes maple, oak, palm, pine and willow trees have strong sharing, while lawn mower and rocket are distinct from each other.
Revised: For example, in CIFAR100 (right one of ﬁgure 3), classes maple, oak, palm, pine and willow trees have strong sharing, while lawn mower and rocket are distinct from each other.

Original: Similar patterns can be found in CIFAR10 dataset, shown in the left one of ﬁgure 3.[Table] [Table]
Revised: Similar patterns can be found in CIFAR10 dataset, shown in the left one of ﬁgure 3.

Original: Result 2: back-propagation through time is a strong baseline.
Revised: Result 2: back-propagation through time is a strong baseline.

Original: In ﬁgure 4 and table 2, we show that a vanilla BPTT variant is already a strong baseline which outperforms previous single-step gradient methods [10] by 40.4% on SVHN and 20.3% on SVHN and CIFAR10 under 1 image per class.
Revised: In ﬁgure 4 and table 2, we show that a vanilla BPTT variant is already a strong baseline which outperforms previous single-step gradient methods [10] by 40.4% on SVHN and 20.3% on SVHN and CIFAR10 under 1 image a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h o r s e s h i p t r u c k

Revised: [Figure]

Original: Note that the performance on SVHN has doubled the accuracy 31.2% obtained using single-step gradient matching method [10].
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: m

Revised: a p l e t r ee o a k t r ee p a l m t r ee p i n e t r ee w ill

Revised: o w t r ee b i c y c l e b u s m o t

Revised: o r c

Revised: y

Revised: c l e p i

Revised: c k u

Revised: p t

Revised: r u c k t r a i n l a w n m o w e r r o c k e t s t r ee t c a r

Revised: [Figure]

Revised: a n

Revised: [Figure]

Revised: a c

Revised: [Figure]

Revised: o

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: per class.

Revised: Note that the performance on SVHN has doubled the accuracy 31.2% obtained using single-step gradient matching method [10].

Original: In the vanilla BPTT variant, no downsampling (ds) or memory addressing formulation is used.
Revised: In the vanilla BPTT variant, no downsampling (ds) or memory addressing formulation is used.

Original: We also analyze the effects of long unrolls and momentum terms on vanilla BPTT in ﬁgure 4.
Revised: We also analyze the effects of long unrolls and momentum terms on vanilla BPTT in ﬁgure 4.

Original: It is observed that on both short inner loops (10 steps) and long ones (100 steps), adding momentum terms can consistently lead to strong performance boost, e.g.
Revised: It is observed that on both short inner loops (10 steps) and long ones (100 steps), adding momentum terms can consistently lead to strong performance boost, e.g.

Original: 7.0% and 9.2% on CIFAR10.
Revised: 7.0% and 9.2% on CIFAR10.

Original: Using longer inner loop trajectories can also increase the recovered accuracy, e.g. 18.2% and 42.3% on CIFAR10 and SVHN, respectively, compared to 1 step cases.
Revised: Using longer inner loop trajectories can also increase the recovered accuracy, e.g. 18.2% and 42.3% on CIFAR10 and SVHN, respectively, compared to 1 step cases.

Original: Ablation studies.
Revised: Ablation studies.

Original: To further analyze the effects of different components in our algorithm, we perform ablation studies on CIFAR10 and CIFAR100.
Revised: To further analyze the effects of different components in our algorithm, we perform ablation studies on CIFAR10 and CIFAR100.

Original: Besides the vanilla BPTT, the ablation results of components (downsampling, augmentation and memory addressing formulation) are summarized in table 2.
Revised: Besides the vanilla BPTT, the ablation results of components (downsampling, augmentation and memory addressing formulation) are summarized in table 2.

Original: We show that downsampling can indeed reduce spatial redundancies (e.g. improve results from 49.1% to 55.2% on CIFAR10 with 1 image per class), and memory addressing formulation can further increase the recovered accuracy (from 55.2% to 64.2% on CIFAR10 with 1 image per class).
Revised: We show that downsampling can indeed reduce spatial redundancies (e.g. improve results from 49.1% to 55.2% on CIFAR10 with 1 image per class), and memory addressing formulation can further increase the recovered accuracy (from 55.2% to 64.2% on CIFAR10 with 1 image per class).

Original: It is also shown that our model is quite robust to the ablation of data augmentation, which has a small effect (1-2%) on the results.
Revised: It is also shown that our model is quite robust to the ablation of data augmentation, which has a small effect (1-2%) on the results.

Original: The resulting algorithm is a simple and effective framework that uses memory addressing formulation and BPTT with long unrolls to distill datasets.
Revised: The resulting algorithm is a simple and effective framework that uses memory addressing formulation and BPTT with long unrolls to distill datasets.

Original: Cross-architecture generalization.
Revised: Cross-architecture generalization.

Original: Our memories and addressing matrices are also generalizable across various architectures.
Revised: Our memories and addressing matrices are also generalizable across various architectures.

Original: We test our algorithm on ConvNet, ResNet12 and AlexNet for training and testing.
Revised: We test our algorithm on ConvNet, ResNet12 and AlexNet for training and testing.

Original: The results are summarized in the appendix.
Revised: The results are summarized in the appendix, section B.1, table 6.

Original: 5.2 Continual Learning
Revised: 5.2 Continual learning

Original: One of the key usage of memories is to prevent forgetting when a model continually learns through tasks.
Revised: One of the key usage of memories is to prevent forgetting when a model continually learns through tasks.

Original: In this section, we evaluate our algorithm on the standard continual learning benchmarks and show that, due to the strong performance, a simple “compress-then-recall” method with our model can already rival with previous state-of-the-arts with complex designs.
Revised: In this section, we evaluate our algorithm on the standard continual learning benchmarks and show that, due to the strong performance, a simple “compress-then-recall” method with our model can already rival with previous state-of-the-arts with complex designs.

Original: Efﬁcient lifelong learning.
Revised: Efﬁcient lifelong learning.

Original: Following [46], we work with the problem where all tasks are streamed in mini-batches and learned in a single pass .
Revised: Following [47], we work with the problem where all tasks are streamed in mini-batches and learned in a single pass .

Original: A learner is allowed to be equipped with a small memory buffer.
Revised: A learner is allowed to be equipped with a small memory buffer.

Original: The data samples after seen will not be available unless stored in the buffer.
Revised: The data samples after seen will not be available unless stored in the buffer.

Original: We use mini-batch size 10 to stream the data, following previous works [28, 24].
Revised: We use mini-batch size 10 to stream the data, following previous works [28, 24].

Original: Evaluation.
Revised: Evaluation.

Original: The learner’s performance after learning on the task stream is evaluated under two metrics: retained accuracy (RA) and backward-transfer and interference (BTI).
Revised: The learner’s performance after learning on the task stream is evaluated under two metrics: retained accuracy (RA) and backward-transfer and interference (BTI).

Original: RA is the averageaccuracy of the ﬁnal trained model on all tasks, and BTI measures the performance difference between after it was learned and after the full training process.
Revised: RA is the average

Original: [Table]
Revised: [Table]

Original: or obtained from public ofﬁcial repos.
Revised: or obtained from public ofﬁcial repos.

Revised: accuracy of the ﬁnal trained model on all tasks, and BTI measures the performance difference between after it was learned and after the full training process.

Original: Benchmarks.
Revised: Benchmarks.

Original: We evaluate our method on three tasks widely used in previous Continual Learning works.
Revised: We evaluate our method on three tasks widely used in previous Continual Learning works.

Original: MNIST Rotations [24] contains 20 tasks with 1,000 samples in each.
Revised: MNIST Rotations [24] contains 20 tasks with 1,000 samples in each.

Original: Every task consists of images rotated by a ﬁxed angle from 0 to 180 degrees.
Revised: Every task consists of images rotated by a ﬁxed angle from 0 to 180 degrees.

Original: MNIST Permutations [47] has 20 tasks, and each task contains 1,000 images generated through shufﬂing the image pixels by a ﬁxed permutation.
Revised: MNIST Permutations [48] has 20 tasks, and each task contains 1,000 images generated through shufﬂing the image pixels by a ﬁxed permutation.

Original: MANY Permutations [17] is a longer variant with 100 tasks in total and 200 samples in each.
Revised: MANY Permutations [17] is a longer variant with 100 tasks in total and 200 samples in each.

Original: Incremental CIFAR-100 [29, 24] splits the CIFAR100 dataset into 20 5-way classiﬁcation tasks as the task stream for learning.
Revised: Incremental CIFAR-100 [29, 24] splits the CIFAR100 dataset into 20 5-way classiﬁcation tasks as the task stream for learning.

Original: Our model.
Revised: Our model.

Original: Based on our distillation method, we adopt a simple framework to perform continual learning: “compress then recall”.
Revised: Based on our distillation method, we adopt a simple framework to perform continual learning: “compress then recall”.

Original: During the training phase, we do not perform learning on neural networks, instead, the dataset of each task is distilled to memories and the paired addressing matrices.
Revised: During the training phase, we do not perform learning on neural networks, instead, the dataset of each task is distilled to memories and the paired addressing matrices.

Original: During test phase, we simply fetch the corresponding memories and addressing matrices for each task, and train a new model from scratch to perform classiﬁcation.
Revised: During test phase, we simply fetch the corresponding memories and addressing matrices for each task, and train a new model from scratch to perform classiﬁcation.

Original: Memory buffer designs.
Revised: Memory buffer designs.

Original: When a new task starts, we use the full remaining memory buffer to store the samples and perform distillation with both buffer samples and streamed samples.
Revised: When a new task starts, we use the full remaining memory buffer to store the samples and perform distillation with both buffer samples and streamed samples.

Original: After a task ends, the distilled memories and addressing matrices are stored in the buffer, taking 1 /T of the space, where T is the total number of tasks.
Revised: After a task ends, the distilled memories and addressing matrices are stored in the buffer, taking 1 /T of the space, where T is the total number of tasks.

Original: Namely, the buffer size keeps shrinking when more compressed representation of tasks is stored.
Revised: Namely, the buffer size keeps shrinking when more compressed representation of tasks is stored.

Original: Note that we compare our model with previous methods under the exact same memory sizes for fair comparisons .
Revised: Note that we compare our model with previous methods under the exact same memory sizes for fair comparisons .

Original: See appendix for more details on model and memory designs.
Revised: See appendix for more details on model and memory designs.

Original: Results.
Revised: Results.

Original: We show that this simple method is already a strong baseline that outperforms prior arts on four benchmarks, summarized in table 3.
Revised: We show that this simple method is already a strong baseline that outperforms prior arts on four benchmarks, summarized in table 3.

Original: Our method is compared with: Online, EWC [47],
Revised: Our method is compared with: Online, EWC [48],

Original: GEM [24], MER [17], C-MAML [28], La-MAML [28] and Sparse-LaMAML [30].
Revised: GEM [24], MER [17], C-MAML [28], La-MAML [28] and Sparse-LaMAML [30].

Original: For example, we can obtain a 23% boost on MNIST MANY benchmark: from 50.81% to 74.07%.
Revised: For example, we can obtain a 23% boost on MNIST MANY benchmark: from 50.81% to 74.07%.

Original: We further compare our model with previous works Kernel Continual Learning [39] and Stable
Revised: We further compare our model with previous works Kernel Continual Learning [39] and Stable

Original: SGD [37] following their settings, where each task in MNIST Rotation and MNIST Permutation contains 60,000 samples instead of 1,000 samples.
Revised: SGD [37] following their settings, where each task in MNIST Rotation and MNIST Permutation contains 60,000 samples instead of 1,000 samples.

Original: Our model achieves 87.3 ± 0 .
Revised: Our model achieves 87.3 ± 0 .

Original: 92 and 88.3 ± 0 .
Revised: 92 and 88.3 ± 0 .

Original: 58 on
Revised: 58 on

Original: Permutated MNIST and Rotated MNIST under their setting, outperforming both KCL (85.5 ± 0 . 78 and 81.8 ± 0 . 60 ) and Stable SGD (80.1 ± 0 .
Revised: Permutated MNIST and Rotated MNIST under their setting, outperforming both KCL (85.5 ± 0 . 78 and 81.8 ± 0 . 60 ) and Stable SGD (80.1 ± 0 .

Original: 51 and 70.8 ± 0 . 78 ).
Revised: 51 and 70.8 ± 0 . 78 ).

Original: Interestingly, our results also are higher than the multitask upperbound (86.5 ± 0 .
Revised: Interestingly, our results also are higher than the multitask upperbound (86.5 ± 0 .

Original: 21 and 87.3 ± 0 . 47 ), potentially due to that there are task interference in joint training, which can be naturally avoided in our method.
Revised: 21 and 87.3 ± 0 . 47 ), potentially due to that there are task interference in joint training, which can be naturally avoided in our method.

Original: 5.3 Synthesizing new classiﬁers after learning
Revised: 5.3 Synthesizing new classiﬁers after learning

Original: If we want to memorize the past, what is the beneﬁt of storing the compressed representation rather than a trained model?
Revised: If we want to memorize the past, what is the beneﬁt of storing the compressed representation rather than a trained model?

Original: In this section, we show that our compressed representation can enable ﬂexible synthesis on new classiﬁers after the learning.
Revised: In this section, we show that our compressed representation can enable ﬂexible synthesis on new classiﬁers after the learning.

Original: Speciﬁcally, we demonstrate extrapolating between tasks to train new models, and performing memory recall with images instead of labels, showing the generalizability of our framework on other query forms.
Revised: Speciﬁcally, we demonstrate extrapolating between tasks to train new models, and performing memory recall with images instead of labels, showing the generalizability of our framework on other query forms.

Original: 5.3.1 Extrapolating between tasks
Revised: 5.3.1 Extrapolating between tasks

Original: In real world, tasks often do not come together and a learner therefore cannot observe all tasks at once.
Revised: In real world, tasks often do not come together and a learner therefore cannot observe all tasks at once.

Original: In current machine learning paradigms, when models are separately trained for disjoint tasks, it has difﬁculty extrapolating between tasks to build new classiﬁers.
Revised: In current machine learning paradigms, when models are separately trained for disjoint tasks, it has difﬁculty extrapolating between tasks to build new classiﬁers.

Original: This is different from human learning.
Revised: This is different from human learning.

Original: We show that storing our compressed representation enables a learner to extrapolate and synthesize new classiﬁers after learning separately on each task.
Revised: We show that storing our compressed representation enables a learner to extrapolate and synthesize new classiﬁers after learning separately on each task.

Original: Speciﬁcally, we separate CIFAR into 20 disjoint 5-way classiﬁcation tasks as training tasks.
Revised: Speciﬁcally, we separate CIFAR into 20 disjoint 5-way classiﬁcation tasks as training tasks.

Original: For testing, we select classes that are not seen together during training by randomly choosing k tasks and pick 1 class from each selected task.
Revised: For testing, we select classes that are not seen together during training by randomly choosing k tasks and pick 1 class from each selected task.

Original: We use k = 2 and k = 5 to construct 2-way and 5-way classiﬁcation tasks, and sample 1, tasks each for evaluation.
Revised: We use k = 2 and k = 5 to construct 2-way and 5-way classiﬁcation tasks, and sample 1, tasks each for evaluation.

Original: To train our models, we independently distill the datasets for 20 training tasks into corresponding memories.
Revised: To train our models, we independently distill the datasets for 20 training tasks into corresponding memories.

Original: For each testing task, the class labels are used as queries to recall the synthetic data from the corresponding memories.
Revised: For each testing task, the class labels are used as queries to recall the synthetic data from the corresponding memories.

Original: The recalled data for each label, although not seen together during training, are used for re-training a k -way classiﬁer from scratch.
Revised: The recalled data for each label, although not seen together during training, are used for re-training a k -way classiﬁer from scratch.

Original: We ﬁnd that the compressed data can indeed train classiﬁers on new combinations, for example, we can achieve 72.53% ± 8 .
Revised: We ﬁnd that the compressed data can indeed train classiﬁers on new combinations, for example, we can achieve 72.53% ± 8 .

Original: 74 on 2-way classiﬁcation, and 46.54% ± 6 .
Revised: 74 on 2-way classiﬁcation, and 46.54% ± 6 .

Original: 42 on 5-way classiﬁcation, with 1 image per class storage budget.
Revised: 42 on 5-way classiﬁcation, with 1 image per class storage budget.

Original: The upperbound with full real dataset is 92.23% ± 4 .
Revised: The upperbound with full real dataset is 92.23% ± 4 .

Original: 76 and 82.72% ± 4 . 29 .
Revised: 76 and 82.72% ± 4 . 29 .

Original: 5.3.2 Dataset Distillation extension – recall the past with images
Revised: 5.3.2 Dataset Distillation extension – recall the past with images

Original: We extend the standard setting to recall the past with images: when the label information and task scopes are missing, but a few visual observations can be made, we would like to build classiﬁers based on the visual data.
Revised: We extend the standard setting to recall the past with images: when the label information and task scopes are missing, but a few visual observations can be made, we would like to build classiﬁers based on the visual data.

Original: For example, when we see a bear image and a deer image, but cannot recall the exact word or category, can we recall the memories with images and build a classiﬁer?
Revised: For example, when we see a bear image and a deer image, but cannot recall the exact word or category, can we recall the memories with images and build a classiﬁer?

Original: This is possible with our problem formulation, where the forms of queries are not constraint to labels and we can distill a dataset to memories addressable by images .
Revised: This is possible with our problem formulation, where the forms of queries are not constraint to labels and we can distill a dataset to memories addressable by images .

Original: [Table]
Revised: [Table]

Original: We formulate the problem as follows.
Revised: We formulate the problem as follows.

Original: Formally, after observing a training dataset D tr with Y = { 0 , ..., C − 1 } , we would like to ﬂexibly build classiﬁers for a subtask Y g
Revised: Formally, after observing a training dataset D tr with Y = { 0 , ..., C − 1 } , we would like to ﬂexibly build classiﬁers for a subtask Y g

Original: ⊂ Y based on visual observations X g from Y g , when the actual information of Y g is unknown.
Revised: ⊂ Y based on visual observations X g from Y g , when the actual information of Y g is unknown.

Original: We work with 1-shot and 5-shot observation cases.
Revised: We work with 1-shot and 5-shot observation cases.

Original: As a baseline, we build a nearest neighbor classiﬁer, which is pretrained on D tr and takes features of few-shot data to classify test images.
Revised: As a baseline, we build a nearest neighbor classiﬁer, which is pretrained on D tr and takes features of few-shot data to classify test images.

Original: As a model variant, we could also “classify-then-recall”, using a classiﬁer trained on D tr to map the image shots into labels and turning into the standard setup.
Revised: As a model variant, we could also “classify-then-recall”, using a classiﬁer trained on D tr to map the image shots into labels and turning into the standard setup.

Original: Beneﬁting from the general design of our formulation, we show that a model can directly perform “image addressing”, where a feature network can provide query vectors y in ﬁg.1.
Revised: Beneﬁting from the general design of our formulation, we show that a model can directly perform “image addressing”, where a feature network can provide query vectors y in ﬁg.1.

Original: The feature network, memories and addressing matrices can be jointly trained on D tr .
Revised: The feature network, memories and addressing matrices can be jointly trained on D tr .

Original: We evaluate the above models and baselines on CIFAR100 and summarize the results in table 4.
Revised: We evaluate the above models and baselines on CIFAR100 and summarize the results in table 4.

Original: As shown in the table, our model is not only able to successfully perform the continuous query addressing with image feature vectors, but also outperforms two strong baselines on constructing new classiﬁers.
Revised: As shown in the table, our model is not only able to successfully perform the continuous query addressing with image feature vectors, but also outperforms two strong baselines on constructing new classiﬁers.

Original: More analysis is in the appendix.
Revised: More analysis is in the appendix.

Original:  Conclusion and limitations
Revised:  Conclusion and limitations

Original: In this paper, we propose a framework that distills a large dataset into compact addressable memories.
Revised: In this paper, we propose a framework that distills a large dataset into compact addressable memories.

Original: This framework introduces several beneﬁts, including removing the linear growth contraint on the compressed data size, allowing more general queries beside categorical labels, and most importantly, achieving high compression rate with strong re-training performance, outperforming previous stateof-the-arts in dataset distillation.
Revised: This framework introduces several beneﬁts, including removing the linear growth contraint on the compressed data size, allowing more general queries beside categorical labels, and most importantly, achieving high compression rate with strong re-training performance, outperforming previous stateof-the-arts in dataset distillation.

Original: We also demonstrate a “compress-then-recall” method using our framework, leading to new state-of-the-arts in continual learning on four datasets.
Revised: We also demonstrate a “compress-then-recall” method using our framework, leading to new state-of-the-arts in continual learning on four datasets.

Original: Our full model has potential limitation on the costly inner optimization loop, which might be time consuming on larger models or datasets.
Revised: Our full model has potential limitation on the costly inner optimization loop, which might be time consuming on larger models or datasets.

Original: This limitation might be solved through combining the memory formulation with a different learning framework.
Revised: This limitation might be solved through combining the memory formulation with a different learning framework.

Original: We do not observe obvious negative societal impacts in our proposed method.
Revised: We do not observe obvious negative societal impacts in our proposed method.

Original:  Acknowledgements
Revised:  Acknowledgements

Original: This material is based upon work supported by the National Science Foundation under Grants No.
Revised: This material is based upon work supported by the National Science Foundation under Grants No.

Original: 2107048 and 2112562.
Revised: 2107048 and 2112562.

Original: Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the National
Revised: Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the National

Original: Science Foundation.
Revised: Science Foundation.

Original: We would also like to thank Vishvak Murahari, Sunny Cui and Zeyu Wang for discussions. 
Revised: We would also like to thank Vishvak Murahari, Sunny Cui, Ruth Fong, Vikram Ramaswamy, and Zeyu Wang for discussions. Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Revised: [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Revised: [N/A] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] Details are in main paper and more in the supplementary materials.

Revised: (c) Did you report error bars (e

Revised: g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e

Revised: g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] More details are in the supplementary materials.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] (b) Did you mention the license of the assets?

Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [No] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Revised: [N/A] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]

Revised: A Experiment setups

Revised: In this section, we provide detailed experimental setups for all the tasks discussed in the main paper.

Revised: Speciﬁcally, we will explain the datasets, architectures and implementation details for all tasks.

Revised: A.1 Dataset Distillation

Revised: Datasets.

Revised: Our models are tested on ﬁve standard dataset distillation benchmarks:

Revised: • MNIST contains 10 classes with 60,000 writing digit images as the training set and 10, images as the test set.

Revised: The images are gray-scale with shape 28 × 28 and associated with a label from 10 classes (digit 0-9).

Revised: • FashionMNIST is a dataset with clothing and shoe images and consists of a training with size 60,000 and a test set with size 10,000.

Revised: Each image is 28 × 28 in gray scale, and has a label from 10 classes.

Revised: • SVHN street digit images where each image has shape 32 × 32 × 3 .

Revised: The dataset contains 73257 images for training and 26032 images for testing.

Revised: We use the cropped SVHN where the center of the image indicates the number and the rest is background.

Revised: Each image is categorized into 10 classes (digit 0-9).

Revised: • CIFAR10 is a dataset consisting of 32 × 32 RGB images and has 10 classes in total: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck.

Revised: Each class contains 5,000 images for training and 1,000 images for testing, leading to 50,000 images for training and 10, images for testing in total.

Revised: • CIFAR100 contains 60,000 images in total from 100 classes.

Revised: For every class, 500 images are used for training and 100 images are used in testing.

Revised: The 100 classes are associated with 20 superclasses, where each superclass contains 5 classes at a ﬁner level.

Revised: Architectures.

Revised: We mainly work with a three-layer convolutional neural network, denoted as "ConvNet", which contains convolutional layers with 3 × 3 ﬁlters, followed by ReLU activation function and InstanceNorm.

Revised: The network has 128 hidden dimensions and uses an average pooling layer with 2 × 2 kernel size after every Instancenorm operation.

Revised: We also test our models on ResNet-12 with 64, 128, 256, 512 hidden dimensions in each block.

Revised: The ResNet-12 architecture is slightly modiﬁed by replacing BatchNorm with InstanceNorm, and removing the ﬁnal average pooling layer.

Revised: We ﬁnd using the full spatial information in the ﬁnal layer is important for distillation.

Revised: Both ConvNet and

Revised: ResNet-12 are standard architectures for few-shot learning benchmarks.

Revised: Implementation details.

Revised: We use one 24-GB GPU for each experiment run.

Revised: For all our models, we use a SGD optimizer with learning rate 0.1 and momentum rate 0.5.

Revised: Every model is trained for 50,000 iterations.

Revised: For both the inner loop optimization and evaluation, we use learning rate 0.01 and momentum rate 0.9.

Revised: For random initialization of addressing matrices and bases, we use Kaiming uniform initialization.

Revised: To select the number of bases for each setting, we randomly sample 10% of training set as the validation set.

Revised: Data augmentations with rotation and ﬂip are applied on CIFAR and CIFAR100 datasets.

Revised: ZCA preprocessing is used on CIFAR10, CIFAR100 and SVHN datasets.

Revised: No ZCA preprocessing or data augmentations are used on MNIST and FashionMNIST datasets.

Revised: A.2 Continual learning

Revised: Datasets.

Revised: We use six datasets to evaluate our models.

Revised: The details are summarized in table 5.

Revised: Architectures and implementation details.

Revised: For all MNIST-based datasets, we use a multi-layer perceptron (MLP) with 256 hidden units.

Revised: Following La-MAML, we use the ConvNet architecture with 160 hidden dimensions.

Revised: All experiments are run on a 24-GB GPU, using a SGD optimizer with 0.1 learning rate and 0.5 momentum rate.

Revised: The inner loop optimization learning rate is set as 0.

Revised: with momentum rate 0.9.

Revised: During testing phase, the re-training phase use the same setups as the inner loop optimization.

Revised: We use data samples stored in the memory buffer for minibatch replay to perform compressing, summarized in table 5.

Revised: [Table]

Revised: A.3 New classiﬁer synthesis

Revised: Datasets and setups.

Revised: For experiments on both extrapolating between tasks and recall with images, we use CIFAR100 as the dataset.

Revised: For task extrapolation experiments , we split CIFAR100 into5-way classiﬁcation tasks for training, and use 2-way and 5-way classiﬁcation for testing.

Revised: The 2-way and 5-way tasks during testing are obtained through randomly selecting 2 or 5 training tasks then randomly sampling 1 class from each selected task.

Revised: This ensures that every pairs of classes in a testing task have not been used together for training.

Revised: For recall with images experiments , we use all classes in CIFAR100 for training, and use 20 5-way classiﬁcation tasks in testing.

Revised: During evaluation on a 5-way classiﬁcation task, we sample 1 or 5 images per class (depends on 1-shot or 5-shot), and use the sampled images for recall.

Revised: The sampled images are from test set, i.e. we would like to use testing images to perform recall and build a new classiﬁer.

Revised: Architectures and implementation details.

Revised: In the task extrapolation experiments, since our models are performing dataset distillation, we use the exact same hyperparameters and architectures as dataset distillation tasks in Sec.

Revised: A.1.

Revised: For recall with images, we use 64 bases and 16 addressing matrices (i.e. each query can generate 16 synthetic images) in our model.

Revised: For baselines, we pretrain the feature backbone for nearest neighbor classiﬁer and the classiﬁer in "classify-then-recall" for 100 epochs on CIFAR100, using SGD optimizer with 0.01 learning rate and 0.9 momentum rate.

Revised: The visual observations (image shots) we used are from test set.

Revised: B Additional results and discussion B.1 Dataset Distillation

Revised: Back-propagation through time as a strong baseline.

Revised: Besides the main ablation study results on

Revised: CIFAR10 and CIFAR100, table 7 provides the results for all ﬁve benchmarks.

Revised: As shown in the table, back-propagation through time is indeed a strong baseline that consistently outperforms the single-step gradient matching method, and downsampling can reduce the spatial redundancies and improve compression rate, leading to a higher recovery performance.

Revised: [Table]

Revised: storage budgets.

Revised: [Table]

Revised: Transfer across architectures.

Revised: To show that our compressed memories are generalizable across architectures, we also test the training on ResNet-12.

Revised: Speciﬁcally, we learn the memories and addressing matrices on ConvNet and ResNet-12, and test them on ResNet-12 and ConvNet, respectively.

Revised: Results are summarized in table 6.

Revised: We use 10 images per class as the storage budget on CIFAR10.

Revised: Each row is the architecture that our method trains on, and each column is the generalization performance.

Revised: The learned compressed representation is quite robust across ConvNet and ResNet-12.

Revised: Choice of # bases.

Revised: To select the number of bases for each experiment, we evaluate the performance on a separate validation set, which is 10% random samples of the training set.

Revised: The results on the validation set are shown in ﬁg. 5.

Revised: We select the number of bases that leads to highest performance on the validation set for the full training set distillation.

Revised: Further ablations on momentum terms.

Revised: How is the momentum term exactly affecting the backpropagation through time process?

Revised: We analyze the performance of baseline BPTT algorithms on three cases: no-momentum, forward-only momentum, and full momentum.

Revised: No-momentum uses

Revised: BPTT without momentum terms.

Revised: Forward-only momentum uses the momentum term only in the forward BPTT, but blocks the gradients on the momentum term in the backward pass (except for the gradients on the current time step weights) to remove the “bridging effect" of momentum term across multiple steps.

Revised: Full momentum is our full model.

Revised: All the experiments are performed on CIFAR with 200 inner optimization steps.

Revised: [Figure]

Revised: [Table]

Revised: Adam optimizer for inner loop.

Revised: We also experimented with using Adam optimizer to optimize the synthetic data, instead of using stochastic gradient descent with momentum.

Revised: Empirically, we found that Adam optimizer leads to certain instability of gradients (e.g., magnitude) on the inner optimization steps when using the same learning rate magnitude and perfers smaller ones such as 1e-4.

Revised: The end results are similar to the SGD algorithms.

Revised: B.2 Continual learning

Revised: Memory designs in “compress-then-recall”.

Revised: We follow the Reservoir sampling strategy to store samples in the memory buffer.

Revised: When learning through the tasks, our algorithm utilizes all the currently available memory buffer storage space to store samples.

Revised: After the learning on one task is ﬁnished, the algorithm saves the compressed representation to the memory buffer, taking 1 /T the buffer where T is the total number of tasks, and clear the storage space which stores the real samples for the current task.

Revised: This strategy makes sure that the compression algorithm has enough samples to replay, resulting in 1 − ( t − 1) /T

Revised: of the storage to use, where t ∈ { 1 , ..., T } is the current task index.

Revised: B.3 New classiﬁer synthesis

Revised: Designs of “image addressing” model.

Revised: Since our formulation allows ﬂexible query forms, we use an extra ConvNet to take the visual observations (images) as input and treat the output feature vectors as queries.

Revised: The feature vector queries are used for vector matrix product with addressing matrices to compute coefﬁcients for combining bases.

Revised: To train the addressing model: For every training iteration, we randomly subsample a subset of classes from all classes and pick 1 or 5 images (depends on 1-shot or 5-shot), and use the recalled synthetic datasets with the feature vectors of the images to perform inner loop optimizations.

Revised: The generalization loss is computed using other image-label pairs from the subset classes (the same as standard dataset distillation training).

Revised: The ConvNet (feature extractor), bases and image matrices are jointly trained.

Revised: Discussion of “image addressing” results.

Revised: We compare the "image addressing" model with two strong baselines: nearest neighbor classiﬁers and "classify-then-recall" method.

Revised: It’s interesting to see that, having the ability to access the dataset-level information (even compressed) can often lead to better performance when building a new classiﬁer, while nearest neighbor classiﬁers can only utilize image shots to serve as limited information for classiﬁcation.

Revised: Note that the "classify-then-recall" method is also a strong baseline, but can suffer from the classiﬁcation errors on test images, leading to less robust recall.

Revised: The direct usage of feature vectors from image shots can provide a continuous space and potentially lead to more robust behaviours in the addressing and recall processes.

Revised: C Visualization and analysis

Revised: Coefﬁcients similarity map.

Revised: We show the full matrix of cosine similarities on the coefﬁcients that combine the bases from all 100 classes in CIFAR100, as shown in ﬁg. 6.

Revised: The order of classes on x and y axis is organized by superclasses.

Revised: Every 5 classes is under a common superclass on the axis.

Revised: As shown by the matrix, we can clearly see that the classes under the same superclass often have signiﬁcant similarities, indicating strong sharings when combining bases.

Revised: For example, categories bridge, castle and house share similar bases; baby, girl, man and woman also share similar bases, while crab and tulips use very different coefﬁcients to perform addressing.

Revised: Visualization on bases.

Revised: In ﬁgure 7, we visualize the learned 64 bases on CIFAR100.

Revised: The bases contain various colors, shapes and textures, and are used to be combined with coefﬁcients generated from queries and addressing matrices.

Revised: D More visualization comparisons

Revised: In this section, we further compare the visualization of our methods under various settings.

Revised: D.1 Same amount of generated images

Revised: We visualize the synthetic images from the baseline method BPTT and from our proposed memory addressing parameterization.

Revised: For BPTT, we use 100 image per class as the budgets, and for our

Revised: [Table]

Revised: m t r ee p i n e t r ee w ill

Revised: o w t r ee b i c y c l e b u s m o t

Revised: o r c

Revised: y

Revised: c l e p i

Revised: c k u

Revised: p t

Revised: r u c k t r a i n l a w n m o w e r r o c k e t s t r ee t c a r t a n k t r a c t o r beaverdolphinottersealwhale aquarium fish flatfishraysharktroutorchidspoppyroses sunflowerstulipsbottlebowlcanscupsplatesapplemushroomsorangespears sweet peppersclockkeyboardlamptelephonetelevisionbedchaircouchtablewardrobebeebeetlebutterflycaterpillarcockroachbearleopardliontigerwolfbridgecastlehouseroadskyscrapercloudforestmountainplainseacamelcattle chimpanzee

Revised: elephantkangaroofoxporcupinepossumraccoonskunkcrablobstersnailspiderwormbabyboygirlmanwomancrocodiledinosaurlizardsnaketurtlehamstermouserabbitshrewsquirrelmaple tree oak tree palm tree pine tree willow tree

Revised: [Figure]

Revised: motorcyclepickup

Revised: [Figure]

Revised: lawn mowerrocketstreetcartanktractor 0.

Revised: method, we use 10 images per class and 43 bases to generate approximately the same amount of recalled images (99).

Revised: The synthetic images are visualized in ﬁgures below.

Revised: To easily compare with the vanilla version of BPTT, we do not use downsampling in either BPTT or the memory addressing formulation.

Revised: [Figure]

Revised: D.2 Various image per class budgets

Revised: We compare the visualizations of synthetic images under various image per class (I/C) budgets.

Revised: Similar to previous section D.1, we use bases with the same shape, and compare the results under 2, 10 and 50 I/Cs.

Revised: The corresponding number of bases are 8, 43 and 215.

Revised: The visualization results are summarized in ﬁgure 12, ﬁgure 13 and ﬁgure 14,

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]