Original: R ETHINKING C URRICULUM L EARNING WITH I NCREMENTAL L ABELS AND A DAPTIVE C OMPENSATION
Revised: R ETHINKING C URRICULUM L EARNING WITH I NCREMENTAL L ABELS AND A DAPTIVE C OMPENSATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Like humans, deep networks learn better when samples are organized and introduced in a meaningful order or curriculum.
Revised: Like humans, deep networks learn better when samples are organized and introduced in a meaningful order or curriculum (Weinshall et al., 2018).

Original: While conventional approaches to curriculum learning emphasize the difﬁculty of samples as the core incremental strategy, it forces networks to learn from small subsets of data while introducing pre-computation overheads.
Revised: While conventional approaches to curriculum learning emphasize the difﬁculty of samples as the core incremental strategy, it forces networks to learn from small subsets of data while introducing pre-computation overheads.

Original: In this work, we proposeLearning with Incremental
Revised: In this work, we propose

Original:  Labels and Adaptive Compensation (LILAC), which introduces a novel approach to curriculum learning.
Revised: Learning with Incremental Labels and Adaptive Compensation (LILAC), which takes a novel approach to curriculum learning.

Original: LILAC emphasizes incrementally learning labels instead of incrementally learning difﬁcult samples.
Revised: LILAC emphasizes incrementally learning labels instead of incrementally learning difﬁcult samples.

Original: It works in two distinct phases: ﬁrst, in the incremental label introduction phase, we unmask ground-truth labels in ﬁxed increments during training, to improve the starting point from which networks learn.
Revised: It works in two distinct phases: ﬁrst, in the incremental label introduction phase, we recursively reveal ground-truth labels in small installments while using a fake label for the remaining data.

Original: In the adaptive compensation phase, we compensate for failed predictions by adaptively altering the target vector to a smoother distribution.
Revised: In the adaptive compensation phase, we compensate for failed predictions by adaptively altering the target vector to a smoother distribution.

Original: We evaluate LILAC against the closest comparable methods in batch and curriculum learning and label smoothing, across three standard image benchmarks, CIFAR-10, CIFAR-100, and STL-10.
Revised: We evaluate LILAC against the closest comparable methods in batch and curriculum learning and label smoothing, across three standard image benchmarks, CIFAR10, CIFAR-100, and STL-10.

Original: We show that our method outperforms batch learning with higher mean recognition accuracy as well as lower standard deviation in performance consistently across all benchmarks .
Revised: We show that our method outperforms batch learning with higher mean recognition accuracy as well as lower standard deviation in performance consistently across all benchmarks .

Original: We further extend LILAC to show state-of-the-art performance on CIFAR-10  using simple data augmentation while exhibiting label order invariance among otherimportant properties.
Revised: We further extend LILAC to show the highest performance on CIFAR-10 for methods using simple data augmentation while exhibiting label-order invariance among other properties.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep networks have seen rich applications in high-dimensional problems characterized by a large number of labels and a high volume of samples.
Revised: Deep networks have seen rich applications in high-dimensional problems characterized by a large number of labels and a high volume of samples.

Original: However, successfully training deep networks to solve problems under such conditions is mystifyingly hard (Erhan et al.
Revised: However, successfully training deep networks to solve problems under such conditions is mystifyingly hard (Erhan et al.

Original: Larochelle et al.
Revised: Larochelle et al.

Original: The go-to solution in most cases is Stochastic Gradient Descent with mini-batches (simple batch learning) and its derivatives.
Revised: The go-to solution in most cases is Stochastic Gradient Descent with mini-batches (simple batch learning) and its derivatives.

Original: While offering a standardized solution, simple batch learning often fails to ﬁnd solutions that are simultaneously stable, highly generalizable and scalable to large systems (Das et al.
Revised: While offering a standardized solution, simple batch learning often fails to ﬁnd solutions that are simultaneously stable, highly generalizable and scalable to large systems (Das et al.

Original: ; Keskar et al. (2016); Goyal et al.
Revised: ; Keskar et al. (2016); Goyal et al.

Original: ; You et al.
Revised: ; You et al.

Original: This is a by-product of how mini-batches are constructed.
Revised: This is a by-product of how mini-batches are constructed.

Original: For example, the uniform prior assumption over datasets emphasizes equal contributions from each data point regardless of the underlying distribution; small batch sizes help achieve more generalizable solutions, but do not scale as well to vast computational resources as large mini-batches.
Revised: For example, the uniform prior assumption over datasets emphasizes equal contributions from each data point regardless of the underlying distribution; small batch sizes help achieve more generalizable solutions, but do not scale as well to vast computational resources as large mini-batches.

Original: It is hard to construct a solution that is a perfect compromise between all cases.
Revised: It is hard to construct a solution that is a perfect compromise between all cases.

Original: Two lines of work, curriculum learning and label smoothing, offer alternative strategies to improve learning in deep networks.
Revised: Two lines of work, curriculum learning and label smoothing, offer alternative strategies to improve learning in deep networks.

Original: Curriculum learning, inspired by strategies used for humans (Skinner (1958); Avrahami et al.
Revised: Curriculum learning, inspired by strategies used for humans (Skinner (1958); Avrahami et al.

Original: (1997)), works by gradually increasing the conceptual difﬁculty of samples used to train deep networks (Bengio et al.
Revised: (1997)), works by gradually increasing the conceptual difﬁculty of samples used to train deep networks (Bengio et al.

Original: ; Florensa et al.
Revised: ; Florensa et al.

Original: ; Graves et al.
Revised: ; Graves et al.

Original: This has been shown to improve performance on corrupted (Jiang et al.
Revised: This has been shown to improve performance on corrupted (Jiang et al.

Original: and small datasets (Fan et al.
Revised: and small datasets (Fan et al.

Original: More recently, deep networks have been used to categorize samples (Weinshall et al.
Revised: More recently, deep networks have been used to categorize samples (Weinshall et al.

Original: and variations on the pace with which these samples were shown to deep networks were analyzed in-depth (Hacohen & Weinshall (2019)).
Revised: and variations on the pace with which these samples were shown to deep networks were analyzed in-depth (Hacohen & Weinshall (2019)).

Original: To the best of our knowledge, previous works assumed that samples cover a broad spectrum of difﬁculty and hence need to be categorized and presented in a speciﬁc order.
Revised: To the best of our knowledge, previous works assumed that samples cover a broad spectrum of difﬁculty and hence need to be categorized and presented in a speciﬁc order.

Original: This introduces computational overheads e.g. pre-computing the relative difﬁculty of samples, and also reduces the effective amount of data from which a model can learn in early epochs.
Revised: This introduces computational overheads e.g. pre-computing the relative difﬁculty of samples, and also reduces the effective amount of data from which a model can learn in early epochs.

Original: Further, curriculum learning approaches have only been tested using shallow
Revised: Further, curriculum learning approaches have not been shown to compete with simple training strategies at the top end of performance in image benchmarks.

Original: A complementary approach to obtaining generalizable solutions is to avoid over-ﬁtting or getting stuck in local minima.
Revised: A complementary approach to obtaining generalizable solutions is to avoid over-ﬁtting or getting stuck in local minima.

Original: In this regard, label smoothing offers an important solution that is invariant to the underlying architecture.
Revised: In this regard, label smoothing offers an important solution that is invariant to the underlying architecture.

Original: Early works like Xie et al.
Revised: Early works like Xie et al.

Original: (2016) replace ground-truth labels with noise while Reed et al.
Revised: (2016) replace ground-truth labels with noise while Reed et al.

Original: (2014) uses other models’ outputs to prevent over-ﬁtting.
Revised: (2014) uses other models’ outputs to prevent over-ﬁtting.

Original: This idea was extended in
Revised: This idea was extended in

Original: Bagherinezhad et al.
Revised: Bagherinezhad et al.

Original: (2018) to an iterative method which uses logits obtained from previously trained versions of the same deep network.
Revised: (2018) to an iterative method which uses logits obtained from previously trained versions of the same deep network.

Original: While Miyato et al.
Revised: While Miyato et al.

Original: (2015) use local distributional smoothness, based on the robustness of a model’s distribution around a data point, to regularize outcomes, Pereyra et al.
Revised: (2015) use local distributional smoothness, based on the robustness of a model’s distribution around a data point, to regularize outcomes, Pereyra et al.

Original: (2017) penalized highly conﬁdent outputs directly.
Revised: (2017) penalized highly conﬁdent outputs directly.

Original: Closest in spirit to our work is the label smoothing method deﬁned in Szegedy et al.
Revised: Closest in spirit to our work is the label smoothing method deﬁned in Szegedy et al.

Original: (2016), which offers an alternative target distribution for all training samples with no extra data augmentation.
Revised: (2016), which offers an alternative target distribution for all training samples with no extra data augmentation.

Original: In general, label smoothing is applied to all examples regardless of how it affects the network’s understanding of them.
Revised: In general, label smoothing is applied to all examples regardless of how it affects the network’s understanding of them.

Original: Further, in methods which use other models to provide logits/labels, often the parent network used to provide those labels is trained using an alternate objective function or needs to be fully re-trained on the current dataset, both of which introduce additional computation.
Revised: Further, in methods which use other models to provide logits/labels, often the parent network used to provide those labels is trained using an alternate objective function or needs to be fully re-trained on the current dataset, both of which introduce additional computation.

Original: In this work, we propose LILAC, Learning with Incremental Labels and Adaptive Compensation , which emphasizes a label-based curriculum and adaptive compensation, to improve upon previous methods and obtain highly accurate and stable solutions.
Revised: In this work, we propose LILAC, Learning with Incremental Labels and Adaptive Compensation , which emphasizes a label-based curriculum and adaptive compensation, to improve upon previous methods and obtain highly accurate and stable solutions.

Revised: LILAC is conceived as a method to learn strong embeddings by using the recursive training strategy of incremental learning alongside the use of unlabelled/wrongly-labelled data as hard negative examples.

Original: It works in two key phases, 1) incremental label introduction and 2) adaptive compensation .
Revised: It works in two key phases, 1) incremental label introduction and 2) adaptive compensation .

Original: The ﬁrst phase, inspired by incremental learning, trains a deep network in ﬁxed intervals with data where a certain portion ofit has its ground-truth labels masked out.
Revised: In the ﬁrst phase, we incrementally introduce groups of labels in the training process.

Original: The network is trained for a ﬁxed number of epochs with data corresponding tomasked out labels as “negative” examples which are contrasted against samples which have groundtruth labels available.
Revised: Data, corresponding to labels not yet introduced to the model, use a single fake label selected from within the dataset.

Original: After training for a ﬁxed number of epochs with this setup, the ground-truth labels for some previously masked-out classes are revealed to the network and the training process is repeated.
Revised: Once a network has been trained for a ﬁxed number of epochs with this setup, an additional set of ground-truth labels is introduced to the network and the training process continues.

Revised: In recursively revealing labels, LILAC allows the model sufﬁcient time to develop a strong understanding of each class by contrasting against a large and diverse set of negative examples.

Original: Once all ground-truth labels are revealed the adaptive compensation phase of training is initiated.
Revised: Once all ground-truth labels are revealed the adaptive compensation phase of training is initiated.

Original: This phase mirrors conventional batch learning, except we adaptively replace the target one-hot vector of incorrectly classiﬁed samples with a softer distribution.
Revised: This phase mirrors conventional batch learning, except we adaptively replace the target one-hot vector of incorrectly classiﬁed samples with a softer distribution.

Original: Thus, we avoid adjusting labels across the entire dataset, like previous methods, while elevating the stability and average performance of the model.
Revised: Thus, we avoid adjusting labels across the entire dataset, like previous methods, while elevating the stability and average performance of the model.

Original: Further, instead of being pre-computed by an alternative model, these softer distributions are generated on-the-ﬂy from the outputs of the model being trained.
Revised: Further, instead of being pre-computed by an alternative model, these softer distributions are generated on-the-ﬂy from the outputs of the model being trained.

Original: We apply LILACto three standard image benchmarks and compare its performance to the strongest known baselines.
Revised: We apply LILAC to three standard image benchmarks and compare its performance to the strongest known baselines.

Revised: While incremental and continual learning work on evolving data distributions with the addition of memory constraints ((Rebufﬁ et al., 2017; Castro et al., 2018) and derivative works), knowledge distillation ((Schwarz et al., 2018; Rolnick et al., 2018) and similar works) or other requirements, this work is a departure into using negative mining and focused training to improve learning on a fully available dataset.

Original: To summarize, our main contributions in LILAC are as follows,
Revised: In incremental/continual learning works, often the amount of data used to retrain the network is small compared to the original dataset while in LILAC we fully use the entire dataset, distinguished by Seen and Unseen labels. Thus, it avoids data deﬁcient learning. Further, works like Bucher et al. ; Li et al.

Revised: ; Wang & Gupta (2015) emphasize the importance of hard negative mining, both in size and diversity, in improving learning.

Original:  • we propose a model-invariant training strategy that introduces a novel take on curriculum learning by incrementally learning labels as opposed to samples, • we avoid external computational overhead by offering a label-order invariant training strategy that adaptively compensatesfor incorrectly labelled samples that are obtained using an older copy of the model being trained, • we improve average recognition accuracy and decrease the standard deviation of performance across several image classiﬁcation benchmarks compared to batch learning, a property not shared by other curriculum learning and label smoothing methods.Further, in Section 3.3 we empirically demonstrate LILAC’s relative invariance to the order of labels provided while also highlighting the importance of the number of labels introduced in each incremental phase as well as the overall impact of the sharpness of the alternate target distribution used in adaptive compensation.
Revised: Although the original formulation of negative mining was based on imbalanced data, recent object detection works have highlighted its importance in contrasting and improving learning in neural networks. To summarize, our main contributions in LILAC are as follows, • we introduce a new take on curriculum learning by incrementally learning labels as opposed to samples , • our method adaptively compensates incorrectly labelled samples by softening their target distribution which improves performance and removes external computational overheads , • we improve average recognition accuracy and decrease the standard deviation of performance across several image classiﬁcation benchmarks compared to batch learning, a property not shared by other curriculum learning and label smoothing methods.

Original:  LILAC
Revised:  LILAC

Original: In LILAC, our main focus is to induce better learning in deep networks.
Revised: In LILAC, our main focus is to induce better learning in deep networks.

Original: Instead of the conventional curriculum learning approach of ranking samples, we consider all samples equally beneﬁcial.
Revised: Instead of the conventional curriculum learning approach of ranking samples, we consider all samples equally beneﬁcial.

Original: Early on, we focus on learning labels in ﬁxed increments (Section 2.1).
Revised: Early on, we focus on learning labels in ﬁxed increments (Section 2.1).

Original: Once the network has had a chance to learn all the labels, we shift to regularizing the network to prevent over-ﬁtting by providing a softer distribution as the target vector for previously misclassiﬁed samples (Section 2.2).
Revised: Once the network has had a chance to learn all the labels, we shift to regularizing the network to prevent over-ﬁtting by providing a softer distribution as the target vector for previously misclassiﬁed samples (Section 2.2).

Original: An overview of the entire algorithm discussed is available in the appendix as Algorithm 1.
Revised: An overview of the entire algorithm discussed is available in the appendix as Algorithm 1.

Original: 2. I NCREMENTAL L ABEL I NTRODUCTION P HASE
Revised: 2. I NCREMENTAL L ABEL I NTRODUCTION P HASE

Original: In the incremental phase, we initially mask out the ground-truth labels of several classes by replacing them with a constant held-out label, and then gradually unmask them over the coarse of several ﬁxed intervals.
Revised: In the incremental phase, we initially replace the ground-truth labels of several class using a constant held-out label. Gradually, over the course of several ﬁxed intervals of training we reveal the true label.

Original: Within an interval, we keep constant two sets of data, ”seen” and ”unseen”.
Revised: Within a ﬁxed interval of training, we keep constant two sets of data, ”Seen”, whose groundtruth labels are known and ”Unseen”, whose labels are replaced by a fake value.

Original: When 
Revised: When training,

Original: Full Dataset (
Revised: Full Dataset (

Original: Virtual Data Partition in LILAC (  : Seen  : Unseen
Revised: Virtual Data Partition in LILAC (  : Seen  : Unseen

Original: Legend (Data, Label 1) (Data, Label 2) (Data, Label 3) (Data, Forced Label 4)
Revised: Legend (Data, Label 1) (Data, Label 2) (Data, Label 3) (Data, Forced Label 4)

Original: Incremental Step Incremental Step
Revised: Incremental Step Incremental Step

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: LILAC

Original: Uses Ground-truth labels
Revised: Uses Ground-truth labels

Original: Uses a fake label
Revised: Uses a fake label

Original: Incremental Step(Final Incremental Step) (Data, Label 4)  = 4 ) = 1 )   ( ) ( )  −   =
Revised: Incremental [Figure] (Final Incremental Step) (Data, Label 4)  = 4 ) = 1 )  [Equation]

Original:  Illustration of the evolution of data partitions in the incremental label introduction phase for a four label dataset.
Revised: [Equation] −  [Equation] Illustration of the evolution of data partitions in the incremental label introduction phase for a four label dataset.

Original: In the ﬁrst incremental step, only one label is unmasked for training while the remaining data use label 4.
Revised: In the ﬁrst incremental step, only one label is used for training while the remaining data use label 4.

Original: A short period of training is performed with this ﬁxed setup, where data from U is uniformly sampled to match the number of samples from S , in every mini-batch.
Revised: A short period of training is performed with this ﬁxed setup, where data from U is uniformly sampled to match the number of samples from S , in every mini-batch.

Original: The ﬁnal incremental step depicted is equivalent to batch learning since all the labels are available to the network.
Revised: The ﬁnal incremental step depicted is equivalent to batch learning since all the labels are available to the network.

Original: Once all the ground-truth labels are unmasked we begin the adaptive compensation phase described in Sec.2.2.
Revised: Once all the ground-truth labels are revealed we begin the adaptive compensation phase described in Sec.

Original: training, mini-batches are uniformly sampled from the entire training set, but the instances from ”unseen” classes use the held-out label.
Revised: mini-batches are uniformly sampled from the entire training set, but the instances from ”Unseen” classes use the held-out label.

Original: By the end of the ﬁnal interval, we reveal all classes.
Revised: By the end of the ﬁnal interval, we reveal all ground-truth labels.

Original: We now describe the incremental phase in more detail.
Revised: We now describe the incremental phase in more detail.

Original: At the beginning of the incremental label introduction phase, we virtually partition data into two mutually exclusive sets, S :
Revised: At the beginning of the incremental label introduction phase, we virtually partition data into two mutually exclusive sets, S :

Original: Seen and U :
Revised: Seen and U :

Original: Unseen, as shown in Fig.
Revised: Unseen, as shown in Fig.

Original: Data samples in S use their ground-truth labels as target values while those in U use a designated unseen label, which is held constant throughout the entire training process.
Revised: Data samples in S use their ground-truth labels as target values while those in U use a designated unseen label, which is held constant throughout the entire training process.

Original: LILAC assumes a random ordering of labels, Or ( M ) , where M denotes the total number of labels in the dataset.
Revised: LILAC assumes a random ordering of labels, Or ( M ) , where M denotes the total number of labels in the dataset.

Original: Within this ordering, the number of labels and corresponding data initially placed in S is deﬁned by the variable b .
Revised: Within this ordering, the number of labels and corresponding data initially placed in S is deﬁned by the variable b .

Original: The remaining labels, M − b , are initially placed in U and incrementally unmasked in intervals of m labels, a hyper-parameter deﬁned by the user.
Revised: The remaining labels, M − b , are initially placed in U and incrementally revealed in intervals of m labels, a hyper-parameter deﬁned by the user.

Original: Training in the incremental phase happens at ﬁxed intervals of E epochs each.
Revised: Training in the incremental phase happens at ﬁxed intervals of E epochs each.

Original: Within a ﬁxed interval, the virtual data partition is held constant.
Revised: Within a ﬁxed interval, the virtual data partition is held constant.

Original: Every mini-batch of data is sampled uniformly from the entire original dataset and within each mini-batch, labels are masked based on their placement in S or U .
Revised: Every mini-batch of data is sampled uniformly from the entire original dataset and within each mini-batch, labels are obtained based on their placement in S or U .

Original: Then the number of samples from U is reduced or augmented, using a uniform prior, to match the number of samples from S .
Revised: Then the number of samples from U is reduced or augmented, using a uniform prior, to match the number of samples from S .

Original: This is done to ensure no unfair skew in predictions towards U since all data points use the same designated label.
Revised: This is done to ensure no unfair skew in predictions towards U since all data points use the same designated label.

Original: Finally, the curated mini-batches of data are used to train the neural network.
Revised: Finally, the curated mini-batches of data are used to train the neural network.

Original: At the end of each ﬁxed interval, we unmask another set of m ground-truth labels and move samples of those classes from U to S after which the entire data curation and training process is repeated for the next interval.
Revised: At the end of each ﬁxed interval, we reveal another set of m groundtruth labels and move samples of those classes from U to S after which the entire data curation and training process is repeated for the next interval.

Original: 2. A DAPTIVE C OMPENSATION
Revised: 2. A DAPTIVE C OMPENSATION

Original: Once all the ground-truth labels are available to the deep network, we begin the adaptive compensation phase of training.
Revised: Once all the ground-truth labels are available to the deep network, we begin the adaptive compensation phase of training.

Original: The main idea behind adaptive compensation is, if the network is unable to correctly predict a sample’s label even after allowing sufﬁcient training time, then we alter the target vector to a less peaked distribution.
Revised: The main idea behind adaptive compensation is, if the network is unable to correctly predict a sample’s label even after allowing sufﬁcient training time, then we alter the target vector to a less peaked distribution.

Original: Compared to learning one-hot vectors, this softer distribution can more easily be learned by the network.
Revised: Compared to learning one-hot vectors, this softer distribution can more easily be learned by the network.

Original: Unlike prior methods we adaptively modify the target vector only for incorrectly classiﬁed samples on-the-ﬂy.
Revised: Unlike prior methods we adaptively modify the target vector only for incorrectly classiﬁed samples on-the-ﬂy.

Original: In this phase, the network is trained for a small number of epochs using standard batch learning.
Revised: In this phase, the network is trained for a small number of epochs using standard batch learning.

Original: Let T be the total number of training epochs in the incremental phase and batch learning.
Revised: Let T be the total number of training epochs in the incremental phase and batch learning.

Original: During the adaptive compensation phase, we start at epoch e , where e > T .
Revised: During the adaptive compensation phase, we start at epoch e , where e > T .

Original: For a mini-batch of samples in epoch e , predictions from the model at e − 1 are used to determine the ﬁnal target vector used in the objective function; speciﬁcally, we soften the target vector for an instance iff it was misclassiﬁed by the model at the end of epoch
Revised: For a mini-batch of samples in epoch e , predictions from the model at e − 1 are used to determine the ﬁnal target vector used in the objective function; speciﬁcally, we soften the target vector for an instance iff it was misclassiﬁed by the model at the end of epoch

Original: e − 1 .
Revised: e − 1 .

Original: The ﬁnal target vector for the i th instance at epoch e , t e,i , is computed based on the model φ e − 1 using Equation 1.
Revised: The ﬁnal target vector for the i th instance at epoch e , t e,i , is computed based on the model φ e − 1 using Equation 1.

Original: [Equation]
Revised: [Equation]

Original: otherwise . [Equation] Here, ( x i , y i ) denote a training sample and its corresponding ground-truth label for sample index i while δ y i represents the corresponding one-hot vector.
Revised: Here, ( x i , y i ) denote a training sample and its corresponding ground-truth label for sample index i while δ y i represents the corresponding one-hot vector.

Original: 1 is a vector of M dimensions with all entries as 1 and (cid:15) is a scaling hyper-parameter.
Revised: 1 is a vector of M dimensions with all entries as 1 and (cid:15) is a scaling hyper-parameter.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Datasets We use three datasets, CIFAR-10, CIFAR-100 (Krizhevsky et al.
Revised: Datasets We use three datasets, CIFAR-10, CIFAR-100 (Krizhevsky et al.

Original: and STL10 (Coates et al.
Revised: and STL10 (Coates et al.

Original: (2011)), to evaluate our method and validate our claims.
Revised: (2011)), to evaluate our method and validate our claims.

Original: CIFAR-10 and CIFARare 10 and 100 class variants of the popular image benchmark CIFAR.
Revised: CIFAR-10 and CIFARare 10 and 100 class variants of the popular image benchmark CIFAR.

Original: Each of these contains 50, images in the training set and 10,000 images in the testing set.
Revised: Each of these contains 50, images in the training set and 10,000 images in the testing set.

Original: STL-10 is a 10 class subset of ImageNet with 500 and 800 samples per class for training and testing subsets, respectively.
Revised: STL-10 is a 10 class subset of ImageNet with 500 and 800 samples per class for training and testing subsets, respectively.

Original: Metrics
Revised: Metrics

Original: The common metric used to evaluate the performance of all the learning algorithms is average recognition accuracy(%) and standard deviation across 5 trials.
Revised: The common metric used to evaluate the performance of all the learning algorithms is average recognition accuracy(%) and standard deviation across 5 trials.

Original: We also report consistency , which is a binary metric that indicates whether the training strategy results in higher average performance and lower standard deviation compared to standard batch learning across all datasets.
Revised: We also report consistency , which is a binary metric that indicates whether the training strategy results in higher average performance and lower standard deviation compared to standard batch learning across all datasets.

Original: [Table]
Revised: [Table]

Original: Experimental Setup For CIFAR-10/100, we use ResNet18 (He et al.
Revised: Experimental Setup For CIFAR-10/100, we use ResNet18 (He et al.

Original: as the architectural backbone for all methods; for STL-10, we use ResNet34.
Revised: as the architectural backbone for all methods; for STL-10, we use ResNet34.

Original: In each interval of LILAC’s incremental phase, we train the model for 10 epochs each for CIFAR-10/100, and 5 epochs each for STL-10.
Revised: In each interval of LILAC’s incremental phase, we train the model for 10 epochs each for CIFAR-10/100, and 5 epochs each for STL-10.

Original: During these incremental steps, we use a learning rate of 0.1, 0.01 and 0.1 for CIFAR-10, CIFAR100, and STL- 10 respectively.
Revised: During these incremental steps, we use a learning rate of 0.1, 0.01 and 0.1 for CIFAR-10, CIFAR100, and STL-10 respectively.

Original: The standard batch learning settings used across all datasets are listed in the appendix.
Revised: The standard batch learning settings used across all datasets are listed in the appendix.

Original: These settings reﬂect the setup used in LILAC once the incremental portion of training is complete and the algorithm moves into the adaptive compensation phase.
Revised: These settings reﬂect the setup used in LILAC once the incremental portion of training is complete and the algorithm moves into the adaptive compensation phase.

Original: Within this phase epochs 175, 525 and 120 are used as thresholds (epoch T ) for CIFAR-10, 100 and STLrespectively.
Revised: Within this phase epochs 175, 525 and 120 are used as thresholds (epoch T ) for CIFAR-10, 100 and STLrespectively.

Original: Baselines
Revised: [Table]

Original: • Stochastic gradient descent with mini-batches is the baseline against which all methods are compared.
Revised: • Stochastic gradient descent with mini-batches is the baseline against which all methods are compared.

Original: • Curriculum learning (Bengio et al., 2009) forms a family of related works which aim to help models learn faster and optimize to a better minimum .
Revised: • Curriculum learning (Bengio et al., 2009) forms a family of related works which aim to help models learn faster and optimize to a better minimum .

Original: Following the methodology proposed in this work we artiﬁcially create a subset of the dataset called “Simple”, by selecting data that is within a value of 1.1 as predicted by a linear one-vs-all SVR model trained to regress to the ground-truth label.
Revised: Following the methodology proposed in this work we artiﬁcially create a subset of the dataset called “Simple”, by selecting data that is within a value of 1.1 as predicted by a linear one-vs-all SVR model trained to regress to the ground-truth label.

Original: The deep network is trained on the “Simple” dataset for a ﬁxed period of time that mirrors the total number of epochs of the incremental phase of LILAC after which the entire dataset is used to train the network.
Revised: The deep network is trained on the “Simple” dataset for a ﬁxed period of time that mirrors the total number of epochs of the incremental phase of LILAC after which the entire dataset is used to train the network.

Original: • Label Smoothing (Szegedy et al., 2016) is the closest relevant work to use label smoothing as a form of regularization without extra data augmentation.
Revised: • Label Smoothing (Szegedy et al., 2016) is the closest relevant work to use label smoothing as a form of regularization without extra data augmentation.

Original: This non-invasive baseline is used as a measure of the importance of regularization and for its ability to boost performance.
Revised: This non-invasive baseline is used as a measure of the importance of regularization and for its ability to boost performance.

Original: • Dynamic Batch Size (DBS) is a custom baseline used to highlight the importance of variable batch size in training neural networks.
Revised: • Dynamic Batch Size (DBS) is a custom baseline used to highlight the importance of variable batch size in training neural networks.

Original: DBS randomly copies data available within a mini-batch to mimic variable batch size.
Revised: DBS randomly copies data available within a mini-batch to mimic variable batch size.

Original: Further, all ground-truth labels are available to this model throughout the training process.
Revised: Further, all ground-truth labels are available to this model throughout the training process.

Original: • Random Augmentation (RA) is a custom baseline used to highlight the importance of virtual data partitions in LILAC.
Revised: • Random Augmentation (RA) is a custom baseline used to highlight the importance of virtual data partitions in LILAC.

Original: Its implementation closely follows LILAC but excludes the adaptive compensation phase.
Revised: Its implementation closely follows LILAC but excludes the adaptive compensation phase.

Original: The main difference between LILAC and RA is that RA uses data from a one randomly chosen class, in U , within a mini-batch while data from all classes in U is used in LILAC to equalize the number of samples from S and U .
Revised: The main difference between LILAC and RA is that RA uses data from a one randomly chosen class, in U , within a mini-batch while data from all classes in U is used in LILAC to equalize the number of samples from S and U .

Original: 3. S TANDARDIZED C OMPARISON R ESULTS
Revised: 3. S TANDARDIZED C OMPARISON R ESULTS

Original: Table 1 clearly illustrates improvements in average recognition accuracy, decrease in standard deviation and consistency when compared to batch learning.
Revised: Table 1 clearly illustrates improvements in average recognition accuracy, decrease in standard deviation and consistency when compared to batch learning.

Original: While certain setups have the highestperformance on speciﬁc datasets (e.g., Label Smoothing on CIFAR-10/100), they are not consistent
Revised: While certain setups have the highest

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  across all datasets and do not ﬁnd more stable solutions than LILAC (std. of 0.216 compared to 0.127 from LILAC)
Revised: performance on speciﬁc datasets (e.g., Label Smoothing on CIFAR-10/100), they are not consistent across all datasets and do not ﬁnd more stable solutions than LILAC (std. of 0.216 compared to 0.127 from LILAC)

Original: LILAC is able to achieve superior performance without unnecessary overheads such as computating sample difﬁculty or irreversibly altering the ground-truth distribution across all samples.
Revised: LILAC is able to achieve superior performance without unnecessary overheads such as computing sample difﬁculty or irreversibly altering the ground-truth distribution across all samples.

Original: A key takeaway from DBS is the relative drop in standard deviation combined with higher average performance when compared to baselines like ﬁxed curriculum and label smoothing.
Revised: A key takeaway from DBS is the relative drop in standard deviation combined with higher average performance when compared to baselines like ﬁxed curriculum and label smoothing.

Original: RA serves to highlight the importance of harvesting data from all classes in U simultaneously, for “negative” samples.
Revised: RA serves to highlight the importance of harvesting data from all classes in U simultaneously, for “negative” samples.

Original: The variety of data to learn from provides a boost in performance and standard deviation across the board in LILAC w/o AC as opposed to RA.
Revised: The variety of data to learn from provides a boost in performance and standard deviation across the board in LILAC w/o AC as opposed to RA.

Original: DBS and RA underline the importance of variable batch size and data partitioning in the makeup of LILAC.
Revised: DBS and RA underline the importance of variable batch size and data partitioning in the makeup of LILAC.

Original: We further extend LILAC to train the base pyramidnet with shake-drop regularization ( p = 1 . 0 )
Revised: We further extend LILAC to train the base pyramidnet with shake-drop regularization ( p = 1 . 0 )

Original: (Yamada et al.
Revised: (Yamada et al.

Original: From Table 2 we clearly see that LILAC can be extended to provide state-ofthe-art performance on CIFAR-10 with the exact same preprocessing setup as the best performing method.
Revised: From Table 2 we clearly see that LILAC can be extended to provide the highest performance on CIFAR-10 given a standard preprocessing setup.

Revised: To provide a fair comparison we highlight top performing methods with standard preprocessing setups that avoid partial inputs (at the node or image level) since LILAC was developed with fully available inputs in mind.

Original: Across all these learning schemes, LILAC is the only one to consistently increase classiﬁcation accuracy and decrease the standard deviation across all datasets compared to batch learning .
Revised: Across all these learning schemes, LILAC is the only one to consistently increase classiﬁcation accuracy and decrease the standard deviation across all datasets compared to batch learning .

Original: 3. A BLATION : B REAKDOWN OF LILAC’ S PHASES
Revised: 3. A BLATION : B REAKDOWN OF LILAC’ S PHASES

Original: Fig.
Revised: Fig.

Original: 2 illustrates the evolution of the embedding across the span of the incremental phase.
Revised: 2 illustrates the evolution of the embedding across the span of the incremental phase.

Original: This space has more degrees of separation when compared to an equivalent epoch of training with batch learning where all the labels are available.
Revised: This space has more degrees of separation when compared to an equivalent epoch of training with batch learning where all the labels are available.

Original: Table 3 provides a breakdown of the contribution of each phase of LILAC and how they combine to elevate the ﬁnal performance.
Revised: Table 3 provides a breakdown of the contribution of each phase of LILAC and how they combine to elevate the ﬁnal performance.

Original: Here, in LILAC w/o AC we replace the entire AC phase with simple batch learning while in Batch + AC we include adaptive compensation with adjusted thresholds.
Revised: Here, in LILAC w/o AC we replace the entire AC phase with simple batch learning while in Batch + AC we include adaptive compensation with adjusted thresholds.

Original: The ﬁrst half of this table compares the impact of incrementally introducing labels to a deep network against standard batch learning.
Revised: The ﬁrst half of this table compares the impact of incre- [Figure] mentally introducing labels to a deep network against standard batch learning.

Original: We clearly observe that performances across Rows 1 and 2 fall within the indicated standard deviation of each other.[Figure]
Revised: We clearly observe that performances across Rows 1 and 2 fall within the indicated standard deviation of each other.

Original: However, from Fig.
Revised: However, from Fig.

Original: 2 we know that LILAC start from a qualitatively better solution.
Revised: 2 we know that LILAC start from a qualitatively better solution.

Original: Combining these results, we conclude that the emphasis on a lengthy secondary batch learning phase erodes overall performance.
Revised: Combining these results, we conclude that the emphasis on a lengthy secondary batch learning phase erodes overall performance.

Original: The second half of Table 3 shows the impact of adding adaptive compensation on batch learning and LILAC.
Revised: The second half of Table 3 shows the impact of adding adaptive compensation on batch learning and LILAC.

Original: When added to standard batch learning there isn’t a clear and conclusive indicator of improvement across all benchmarks in both average performance and standard deviation.
Revised: When added to standard batch learning there isn’t a clear and conclusive indicator of improvement across all benchmarks in both average performance and standard deviation.

Original: However, in combination with the incremental label introduction phase of LILAC, adaptive compensation improves average performance as well as decreases standard deviation, indicating an improved stability and consistency.
Revised: However, in combination with the incremental label introduction phase of LILAC, adaptive compensation improves average performance as well as decreases standard deviation, indicating an improved stability and consistency.

Original: Given the similarity in learning between the batch setup and LILAC, when all labels have been introduced, we show that the embedding space learned by incrementally introducing labels (Fig. 2) is distinct from standard batch learning and is more amenable to adaptive compensation.
Revised: Given the similarity in learning between the batch setup and LILAC, when all labels have been introduced, we show that the embedding space learned by incrementally introducing labels (Fig. 2) is distinct from standard batch learning and is more amenable to AC.

Original: 3. P ROPERTIES OF LILAC
Revised: 3. P ROPERTIES OF LILAC

Original: Through previous experiments we have established the general applicability of LILAC while contrasting its contributions to that of standard batch learning.
Revised: Through previous experiments we have established the general applicability of LILAC while contrasting its contributions to that of standard batch learning.

Original: In this section we dive deeper to reveal some characteristics of LILAC that further supplement the claim of general applicability.
Revised: In this section we dive deeper to reveal some characteristics of LILAC that further supplement the claim of general applicability.

Original: Speciﬁcally, we characterize the impact of label ordering, smoothness of alternate target vector distribution and injection of larger groups of labels in the incremental phase.
Revised: Speciﬁcally, we characterize the impact of label ordering, smoothness of alternate target vector distribution and injection of larger groups of labels in the incremental phase.

Original: Ordering of Labels Throughout the standard experiments, we assume labels are used in the ascending order of value.
Revised: Ordering of Labels Throughout the standard experiments, we assume labels are used in the ascending order of value.

Original: When this is modiﬁed to a random ordering, results from Table 4 suggest that there is no explicit beneﬁt. Other than the extra impact of continually ﬂuctuating label orders
Revised: When this is modiﬁed to a random ordering or in ascending order of difﬁ-

Original: [Table]
Revised: [Table]

Original: Label Group [Table] m ): 2 (5)
Revised: m )

Original: [Table]
Revised: [Table]

Original: Label Group ( m ): 3 (10)

Original: [Table]
Revised: [Table]

Original:  across trials, there isn’t a large gap in performance.
Revised: culty, results from Table 4 suggest that there is no explicit beneﬁt or pattern. Other than the extra impact of continually ﬂuctuating label orders across trials, there isn’t a large gap in performance.

Original: Thus, we claim LILAC is relatively invariant to the order of label introduction.
Revised: Thus, we claim LILAC is relatively invariant to the order of label introduction.

Original: Smoothness of Target Vector in Adaptive Compensation During adaptive compensation, (cid:15) = 0 .
Revised: Smoothness of Target Vector in Adaptive Compensation During adaptive compensation, (cid:15) = 0 .

Original: 5 is used in the alternate target vector for samples with failed predictions throughout all experiments in Sections 3.1 and 3.2.
Revised: 5 is used in the alternate target vector for samples with failed predictions throughout all experiments in Sections 3.1 and 3.2.

Original: When extended to a variety of (cid:15) values, we observe that most variations of the peak performance still fall within the standard deviation range for each dataset.
Revised: When extended to a variety of (cid:15) values, we observe that most variations of the peak performance still fall within the standard deviation range for each dataset.

Original: However, the peak average performance values usually occur between 0.7 to 0.5.
Revised: However, the peak average performance values usually occur between 0.7 to 0.5.

Original: Injection of Label Groups While LILAC was designed to allow the introduction of multiple labels in a single incremental step, throughout the experiments in Sections 3.1 and 3.2 only 1 label was introduced per step to allow thorough learning while eliminating the chance of conﬂicting decision boundaries from multiple labels.
Revised: Injection of Label Groups While LILAC was designed to allow the introduction of multiple labels in a single incremental step, throughout the experiments in Sections 3.1 and 3.2 only 1 label was introduced per step to allow thorough learning while eliminating the chance of conﬂicting decision boundaries from multiple labels.

Original: Revealing multiple labels instead of 1 label per incremental step has a negative impact on the overall performance of the model.
Revised: Revealing multiple labels instead of 1 label per incremental step has a negative impact on the overall performance of the model.

Original: Table 4 shows that adding large groups of labels force lower performance, which is in line with our hypothesis that revealing fewer labels per incremental step makes the embedding space more amenable to adaptive compensation.
Revised: Table 4 shows that adding large groups of labels force lower performance, which is in line with our hypothesis that revealing fewer labels per incremental step makes the embedding space more amenable to adaptive compensation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we proposed LILAC which rethinks curriculum learning based on incrementally learning labels instead of samples.
Revised: In this work, we proposed LILAC which rethinks curriculum learning based on incrementally learning labels instead of samples.

Original: This approach helps kick-start the learning process from a substantially better starting point while making the learned embedding space amenable to adaptive negative logit compensation.
Revised: This approach helps kick-start the learning process from a substantially better starting point while making the learned embedding space amenable to adaptive negative logit compensation.

Original: Adaptive compensation provides a softer target distribution for mis-classiﬁed samples which helps correct failed predictions and improve overall performance while decreasing the standard deviation across multiple trials.

Original: Both these techniques combine well in LILAC to show state-of-the-art performance on CIFAR-10 for simple data augmentations while easily outperforming batch and curriculum learning and label smoothing on comparable network architectures.
Revised: Both these techniques combine well in LILAC to show the highest performance on CIFAR-10 for simple data augmentations while easily outperforming batch and curriculum learning and label smoothing on comparable network architectures.

Original: The next step in unlocking the full potential of this setup is to extend it to networks with dropout or partial inputs while targeting faster convergence.
Revised: The next step in unlocking the full potential of this setup is to extend this setup to include a conﬁdence measure on the predictions of network so that it can handle the effects of dropout or partial inputs.

Revised: In further expanding LILAC’s ability to handle partial inputs, we aim to explore its effect on standard incremental learning (memory constrained) while also extending it applicability to more complex neural network architectures.

Original: B H YPER - PARAMETER SETUPS
Revised: B H YPER - PARAMETER SETUPS

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: C A PPLICABILITY TO V IDEOS
Revised: C A PPLICABILITY TO V IDEOS

Original: We clearly observe an improvement in over performance when compared to batch learning
Revised: [Table] D P ROPERTY : V ARIATION OF F IXED I NTERVAL S IZE IN I NCREMENTAL L ABEL I NTRODUCTION

Original: [Table]
Revised: [Table]

Revised: From the results in Table 8, we observe that the choice of E is dependent on the dataset.

Revised: There isn’t an explicit pattern that can be used to select the value of E without trial runs.

Revised: Further, the available run-time is an important constraint when select E from a range of values since both m and E affect it.