Original: B RIDGING THE G AP BETWEEN S EMI - SUPERVISED AND S UPERVISED C ONTINUAL L EARNING VIA D ATA P ROGRAMMING
Revised: B RIDGING THE G AP BETWEEN S EMI - SUPERVISED AND S UPERVISED C ONTINUAL L EARNING VIA D ATA P ROGRAMMING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Semi-supervised continual learning (SSCL) has shown its utility in learning cumulative knowledge with partially labeled data per task.
Revised: Semi-supervised continual learning (SSCL) has shown its utility in learning cumulative knowledge with partially labeled data per task.

Original: However, the state-of-the-art has yet to explicitly address how to reduce the performance gap between using partially labeled data and fully labeled.
Revised: However, the state-of-the-art has yet to explicitly address how to reduce the performance gap between using partially labeled data and fully labeled.

Original: In response, we propose a general-purpose
Revised: In response, we propose a general-purpose

Original: SSCL framework, namely DP-SSCL, that uses data programming (DP) to pseudolabel the unlabeled data per task, and then cascades both ground-truth-labeled and pseudo-labeled data to update a downstream supervised continual learning model.
Revised: SSCL framework, namely DP-SSCL, that uses data programming (DP) to pseudolabel the unlabeled data per task, and then cascades both ground-truth-labeled and pseudo-labeled data to update a downstream supervised continual learning model.

Original: The framework includes a feedback loop that brings mutual benefits: On one hand, DP-SSCL inherits guaranteed pseudo-labeling quality from DP techniques to improve continual learning, approaching the performance of using fully supervised data.
Revised: The framework includes a feedback loop that brings mutual benefits: On one hand, DP-SSCL inherits guaranteed pseudo-labeling quality from DP techniques to improve continual learning, approaching the performance of using fully supervised data.

Original: On the other hand, knowledge transfer from previous tasks facilitates training of the DP pseudo-labeler, taking advantage of cumulative information via self-teaching.
Revised: On the other hand, knowledge transfer from previous tasks facilitates training of the DP pseudo-labeler, taking advantage of cumulative information via self-teaching.

Original: Experiments show that (1) DP-SSCL bridges the performance gap, approaching the final accuracy and catastrophic forgetting as using fully labeled data, (2) DP-SSCL outperforms existing SSCL approaches at low cost, by up to 25% higher final accuracy and lower catastrophic forgetting on standard benchmarks, while reducing memory overhead from 100 MB level to 1 MB level at the same time complexity, and (3) DP-SSCL is flexible, maintaining steady performance supporting plug-and-play extensions for a variety of supervised continual learning models.
Revised: Experiments show that (1) DP-SSCL bridges the performance gap, approaching the final accuracy and catastrophic forgetting as using fully labeled data, (2) DP-SSCL outperforms existing SSCL approaches at low cost, by up to 25% higher final accuracy and lower catastrophic forgetting on standard benchmarks, while reducing memory overhead from 100 MB level to 1 MB level at the same time complexity, and (3) DP-SSCL is flexible, maintaining steady performance supporting plug-and-play extensions for a variety of supervised continual learning models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Lifelong machine learning, also known as continual learning (CL), is a machine learning paradigm that accumulates knowledge over sequential tasks (Ruvolo & Eaton, 2013a; Silver et al., 2013;
Revised: Lifelong machine learning, also known as continual learning (CL), is a machine learning paradigm that accumulates knowledge over sequential tasks (Ruvolo & Eaton, 2013a; Silver et al., 2013;

Original: Chen & Liu, 2016; Liu, 2017).
Revised: Chen & Liu, 2016; Liu, 2017).

Original: It empowers machine learning at the application level such that an agent does not need to be trained from scratch with large amounts of data for every new task, as well as enabling the agent’s self-improvement on previously-learned tasks by continuing to learn post-deployment.
Revised: It empowers machine learning at the application level such that an agent does not need to be trained from scratch with large amounts of data for every new task, as well as enabling the agent’s self-improvement on previously-learned tasks by continuing to learn post-deployment.

Original: Nevertheless, researchers have identified that obtaining labeled training data is expensive (Olivier et al., 2006; Settles, 2009), which semi-supervised continual learning (SSCL)
Revised: Nevertheless, researchers have identified that obtaining labeled training data is expensive (Olivier et al., 2006; Settles, 2009), which semi-supervised continual learning (SSCL)

Original: addresses(Baucum et al., 2017; Wang et al., 2021; Smith et al., 2021).
Revised: addresses(Baucum et al., 2017; Wang et al., 2021; Smith et al., 2021).

Original: As the name suggests, SSCL utilizes not only labeled data, but also leverages unlabeled task data to construct a cumulative knowledge base for learning agents, reducing labeling cost in applied machine learning.
Revised: As the name suggests, SSCL utilizes not only labeled data, but also leverages unlabeled task data to construct a cumulative knowledge base for learning agents, reducing labeling cost in applied machine learning.

Original: Despite all the research efforts on SSCL, the state-of-the-art of SSCL (Baucum et al., 2017; Wang et al., 2021; Smith et al., 2021) has yet to address an elephant in the room: closing the performance gap between supervised and semi-supervised CL.
Revised: Despite all the research efforts on SSCL, the state-of-the-art of SSCL (Baucum et al., 2017; Wang et al., 2021; Smith et al., 2021) has yet to address an elephant in the room: closing the performance gap between supervised and semi-supervised CL.

Original: Ideally, learning from n L labeled data and n U unlabeled data per task should provide the same lifelong performance as if all the n L + n U data are labeled, but state-of-the-art SSCL frameworks have not approached this goal, and rarely consider computational cost required to do so.
Revised: Ideally, learning from n L labeled data and n U unlabeled data per task should provide the same lifelong performance as if all the n L + n U data are labeled, but state-of-the-art SSCL frameworks have not approached this goal, and rarely consider computational cost required to do so.

Original: Moreover, multiple supervised CL tools have matured Lee et al.
Revised: Moreover, multiple supervised CL tools have matured Lee et al.

Original: ; Yoon et al.
Revised: ; Yoon et al.

Original: ; Bulat et al.
Revised: ; Bulat et al.

Original: and would likely benefit by extending them to the semi-supervised setting, but current SSCL approaches are architecture-specific and such extension is non-trivial.
Revised: and would likely benefit by extending them to the semi-supervised setting, but current SSCL approaches are architecture-specific and such extension is non-trivial.

Original: Motivated by the challenges above, we propose data programming (DP) (Ratner et al., 2016b) as a solution.
Revised: Motivated by the challenges above, we propose data programming (DP) (Ratner et al., 2016b) as a solution.

Original: DP is an automatic psuedo-labeling approach that collectively generates labels from noisy labeling functions, with some methods providing probabilistic guarantees on pseudo-label accuracy (Ratner et al., 2016a; Varma & R´e, 2016).
Revised: DP is an automatic psuedo-labeling approach that collectively generates labels from noisy labeling functions, with some methods providing probabilistic guarantees on pseudo-label accuracy (Ratner et al., 2016a; Varma & R´e, 2016).

Original: Ideally, the more diverse noisy labelers are sampled, the higher quality pseudo-labels can be produced - approaching perfect labeling accuracy.
Revised: Ideally, the more diverse noisy labelers are sampled, the higher quality pseudo-labels can be produced - approaching perfect labeling accuracy.

Original: Therefore, upon every task in SSCL, by training a pseudo-labeler via DP and then cascading both ground-truthlabeled and pseudo-labeled data into a supervised CL model, we are able to approach high quality pseudo-labeling and decrease the performance gap between semi-supervised and supervised CL.
Revised: Therefore, upon every task in SSCL, by training a pseudo-labeler via DP and then cascading both ground-truthlabeled and pseudo-labeled data into a supervised CL model, we are able to approach high quality pseudo-labeling and decrease the performance gap between semi-supervised and supervised CL.

Original: This procedure is also benefited from the small overhead of DP in terms of both time and memory, lowering resource costs on large amounts of unlabeled data and long task sequences.
Revised: This procedure is also benefited from the small overhead of DP in terms of both time and memory, lowering resource costs on large amounts of unlabeled data and long task sequences.

Original: Furthermore, the cumulative knowledge along CL can assist the pseudo-labeler performance, leveraging transferability analysis metrics (Nguyen et al., 2020; Tan et al., 2021;
Revised: Furthermore, the cumulative knowledge along CL can assist the pseudo-labeler performance, leveraging transferability analysis metrics (Nguyen et al., 2020; Tan et al., 2021;

Original: Pandy et al., 2022; Tran et al., 2019).
Revised: Pandy et al., 2022; Tran et al., 2019).

Original: Intuitively, the more similar two tasks, the more similar ways they should handle the unlabeled data to shrink the gap.
Revised: Intuitively, the more similar two tasks, the more similar ways they should handle the unlabeled data to shrink the gap.

Original: In practice, noisy labeling functions from previous tasks can be retained and transferred to new tasks based on task transferability, utilizing cumulative knowledge to self-teach the pseudo-labeler throughout the lifelong sequence.
Revised: In practice, noisy labeling functions from previous tasks can be retained and transferred to new tasks based on task transferability, utilizing cumulative knowledge to self-teach the pseudo-labeler throughout the lifelong sequence.

Original: This framework design also allows supervised
Revised: This framework design also allows supervised

Original: CL approaches to be extended in a plug-and-play fashion, by decoupling the pseudo-labeling and continual learning modules.
Revised: CL approaches to be extended in a plug-and-play fashion, by decoupling the pseudo-labeling and continual learning modules.

Original: Experiments on standard image classification benchmarks show DP-SSCL achieves final accuracy and catastrophic forgetting comparable to supervised CL on fully labeled data.
Revised: Experiments on standard image classification benchmarks show DP-SSCL achieves final accuracy and catastrophic forgetting comparable to supervised CL on fully labeled data.

Original: Moreover, DPSSCL outperforms existing SSCL tools with up to 25% higher final accuracy and lower catastrophic forgetting, while reducing the memory overhead for unlabeled data processing from the 100 MB to the 1 MB level with the same time complexity.
Revised: Moreover, DPSSCL outperforms existing SSCL tools with up to 25% higher final accuracy and lower catastrophic forgetting, while reducing the memory overhead for unlabeled data processing from the 100 MB to the 1 MB level with the same time complexity.

Original: Additionally, ablation studies show DP-SSCL maintains steady continual performance at increasing sizes of unlabeled data per tasks, over longer task sequences, and using different knowledge transfer mechanisms.
Revised: Additionally, ablation studies show DP-SSCL maintains steady continual performance at increasing sizes of unlabeled data per tasks, over longer task sequences, and using different knowledge transfer mechanisms.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Our problem formulation and proposed solution are related to the following fields.
Revised: Our problem formulation and proposed solution are related to the following fields.

Original: 2. L IFELONG L EARNING /C ONTINUAL L EARNING (CL)
Revised: 2. L IFELONG L EARNING /C ONTINUAL L EARNING (CL)

Original: The primary goal of continual or lifelong learning is to learn tasks consecutively, exploiting forward transfer to facilitate the learning of new tasks while retaining performance on previous tasks without catastrophic forgetting.
Revised: The primary goal of continual or lifelong learning is to learn tasks consecutively, exploiting forward transfer to facilitate the learning of new tasks while retaining performance on previous tasks without catastrophic forgetting.

Original: The vast majority of research focuses on supervised methods, using techniques such as weight importance vectors (Fernando et al., 2017; Aljundi et al., 2019) to cache critical pathways and prevent catastrophic forgetting, factorized transfer to decompose the model parameter space (Ruvolo & Eaton, 2013b;
Revised: The vast majority of research focuses on supervised methods, using techniques such as weight importance vectors (Fernando et al., 2017; Aljundi et al., 2019) to cache critical pathways and prevent catastrophic forgetting, factorized transfer to decompose the model parameter space (Ruvolo & Eaton, 2013b;

Original: Bulat et al., 2020; Lee et al., 2019), deconflicting projections to ensure that new tasks are trained using unused capacity within the deep network (Farajtabar et al., 2019; Zeng et al., 2019; Saha et al., 2021), and dynamically expanding networks that grow to accommodate tasks (Veniat et al., 2021).
Revised: Bulat et al., 2020; Lee et al., 2019), deconflicting projections to ensure that new tasks are trained using unused capacity within the deep network (Farajtabar et al., 2019; Zeng et al., 2019; Saha et al., 2021), and dynamically expanding networks that grow to accommodate tasks (Veniat et al., 2021).

Original: 2. S EMI - SUPERVISED C ONTINUAL L EARNING (SSCL)
Revised: 2. S EMI - SUPERVISED C ONTINUAL L EARNING (SSCL)

Original: Recently, techniques have been developed for CL in semi-supervised settings to take advantage of unlabeled data.
Revised: Recently, techniques have been developed for CL in semi-supervised settings to take advantage of unlabeled data.

Original: A common procedure of SSCL is to pseudo-label these unlabeled data for training set augmentation.
Revised: A common procedure of SSCL is to pseudo-label these unlabeled data for training set augmentation.

Original: For instance, CNNL (Baucum et al., 2017) fine-tunes a lifelong learning model by repeatedly pseudo-labeling unlabeled data using the model itself, and then augments its training set with the newly-labeled data.
Revised: For instance, CNNL (Baucum et al., 2017) fine-tunes a lifelong learning model by repeatedly pseudo-labeling unlabeled data using the model itself, and then augments its training set with the newly-labeled data.

Original: Alternatively, DistillMatch (Smith et al., 2021) identifies unlabeled data points that are possibly seen in previous tasks by an out-of-distribution detector, and pseudo-labels them using distilled accumulated knowledge.
Revised: Alternatively, DistillMatch (Smith et al., 2021) identifies unlabeled data points that are possibly seen in previous tasks by an out-of-distribution detector, and pseudo-labels them using distilled accumulated knowledge.

Original: A third example is ORDisCo (Wang et al., 2021), which trains a GAN-based pseudo-labeler in parallel with a lifelong learning model by using a three-branch network, which enables it to learn the joint distribution of data and labels simultaneously.
Revised: A third example is ORDisCo (Wang et al., 2021), which trains a GAN-based pseudo-labeler in parallel with a lifelong learning model by using a three-branch network, which enables it to learn the joint distribution of data and labels simultaneously.

Original: Similarly,
Revised: Similarly,

Original: Semi-ACGAN (Brahma et al., 2021) utilizes GAN for training task-dependent classifiers while using the unlabeled data only to train the discriminator of GAN for the source of data (real vs fake).
Revised: Semi-ACGAN (Brahma et al., 2021) utilizes GAN for training task-dependent classifiers while using the unlabeled data only to train the discriminator of GAN for the source of data (real vs fake).

Original: The last example (Ho et al., 2022) combines prototypical learning for pseudo-labeling with meta learning to achieve both the label generation on the unlabeled data and fast adaptation to any task in the continual learning scenario.
Revised: The last example (Ho et al., 2022) combines prototypical learning for pseudo-labeling with meta learning to achieve both the label generation on the unlabeled data and fast adaptation to any task in the continual learning scenario.

Original: Under pseudo-labeling, bridging the gap towards supervised CL becomes simple: the higher pseudo-labeling accuracy, the closer performance to CL on fully labeled data.
Revised: Under pseudo-labeling, bridging the gap towards supervised CL becomes simple: the higher pseudo-labeling accuracy, the closer performance to CL on fully labeled data.

Original: Straightforwardly, to shrink the gap is to improve the pseudo-labeler.
Revised: Straightforwardly, to shrink the gap is to improve the pseudo-labeler.

Original: The methods above all generate labels for the unlabeled data by using the classifiers that are trained for task objectives, and their novelty comes from supplementary design decisions that tackle issues of SSCL such as catastrophic forgetting and pseudo-label consistency.
Revised: The methods above all generate labels for the unlabeled data by using the classifiers that are trained for task objectives, and their novelty comes from supplementary design decisions that tackle issues of SSCL such as catastrophic forgetting and pseudo-label consistency.

Original: A major shortcoming of these approaches arises due to the frequently observed tendency of neural network models to have overconfidence in their predictions (Guo et al., 2017a; Nguyen et al., 2015; Hein et al., 2019); as a result neural-net-based labelers may confidently generate incorrect pseudo-labels, especially when encountering novel tasks or out-of-distribution data.
Revised: A major shortcoming of these approaches arises due to the frequently observed tendency of neural network models to have overconfidence in their predictions (Guo et al., 2017a; Nguyen et al., 2015; Hein et al., 2019); as a result neural-net-based labelers may confidently generate incorrect pseudo-labels, especially when encountering novel tasks or out-of-distribution data.

Original: On the other hand, our DP-SSCL method utilizes DP, which provides theoretical guarantees on the quality of labels, making the pseudo-labeling process more robust to new data from the novel tasks that arise in CL settings.
Revised: On the other hand, our DP-SSCL method utilizes DP, which provides theoretical guarantees on the quality of labels, making the pseudo-labeling process more robust to new data from the novel tasks that arise in CL settings.

Original: Moreover, these existing SSCL tools require large additional memory and computation overhead for pseudo-labeling, such as storage of a ResNet-34 backbone (Smith et al., 2021), GAN networks (Wang et al., 2021;
Revised: Moreover, these existing SSCL tools require large additional memory and computation overhead for pseudo-labeling, such as storage of a ResNet-34 backbone (Smith et al., 2021), GAN networks (Wang et al., 2021;

Original: Brahma et al., 2021) or MAML (Ho et al., 2022), while DP-SSCL minimizes this overhead by storing light-weight labeling functions facilitated by data programming, which we describe below.
Revised: Brahma et al., 2021) or MAML (Ho et al., 2022), while DP-SSCL minimizes this overhead by storing light-weight labeling functions facilitated by data programming, which we describe below.

Original: Additionally, techniques that do not depend directly on pseudo-labels have been proposed for stable semi-supervised continual learning.
Revised: Additionally, techniques that do not depend directly on pseudo-labels have been proposed for stable semi-supervised continual learning.

Original: For instance, pseudo-gradient learners (Luo et al., 2022) are trained instead of pseudo-labelers to provide auxiliary gradients for the model update from the unlabeled data.
Revised: For instance, pseudo-gradient learners (Luo et al., 2022) are trained instead of pseudo-labelers to provide auxiliary gradients for the model update from the unlabeled data.

Original: This is useful for the case that the unlabeled data may include instances of unknown classes.
Revised: This is useful for the case that the unlabeled data may include instances of unknown classes.

Original: The other work, CCIC (Smith et al., 2021), adapts MixMatch to use pseudo-labels of the unlabeled data as consistency regularization target rather than training target.
Revised: The other work, CCIC (Smith et al., 2021), adapts MixMatch to use pseudo-labels of the unlabeled data as consistency regularization target rather than training target.

Original: Still, these methods avoid the usage of pseudo-labels as direct training target, but lack theoretical guarantees on the benefit of these approaches as provided by our DP-SSCL.
Revised: Still, these methods avoid the usage of pseudo-labels as direct training target, but lack theoretical guarantees on the benefit of these approaches as provided by our DP-SSCL.

Original: 2. D ATA P ROGRAMMING (DP)
Revised: 2. D ATA P ROGRAMMING (DP)

Original: Data programming (DP) is an approach to automatically produce pseudo-labels on an unlabeled data set X U , given a labeled data set ( X L , y L ) (Ratner et al., 2016b).
Revised: Data programming (DP) is an approach to automatically produce pseudo-labels on an unlabeled data set X U , given a labeled data set ( X L , y L ) (Ratner et al., 2016b).

Original: The idea is to ensemble a set of noisy weak labeling functions (WLFs) with each performing only slightly better than random guessing on their own, such that the ensembled pseudo-labels, or strong labels, achieve high accuracy.
Revised: The idea is to ensemble a set of noisy weak labeling functions (WLFs) with each performing only slightly better than random guessing on their own, such that the ensembled pseudo-labels, or strong labels, achieve high accuracy.

Original: Therefore, DP papers generally discuss two problems in cascading order: (1) how to generate these
Revised: Therefore, DP papers generally discuss two problems in cascading order: (1) how to generate these

Original: WLFs and (2) how to do the ensembling.
Revised: WLFs and (2) how to do the ensembling.

Original: To address the upstream problem on WLF generation, one simple way is to collect manually designed functions by experts (Ratner et al., 2016b).
Revised: To address the upstream problem on WLF generation, one simple way is to collect manually designed functions by experts (Ratner et al., 2016b).

Original: One succeeding tool, namely Snuba (Varma &
Revised: One succeeding tool, namely Snuba (Varma &

Original: R ´ e, 2016), iteratively trains multiple sets of WLFs, and then from every set selects the top k WLFs ranked by a score
Revised: R ´ e, 2016), iteratively trains multiple sets of WLFs, and then from every set selects the top k WLFs ranked by a score

Original: [Equation]
Revised: [Equation]

Original: where ˆ y L and ˆ y U are the predicted label vectors by a WLF on the labeled and unlabeled data respectively.
Revised: where ˆ y L and ˆ y U are the predicted label vectors by a WLF on the labeled and unlabeled data respectively.

Original: The score consists of a performance metric (F1) on the labeled set, as well as a diversity metric (Jaccard distance (Jaccard, 1902)), with a weighting factor w usually = 0 . 5 .
Revised: The score consists of a performance metric (F1) on the labeled set, as well as a diversity metric (Jaccard distance (Jaccard, 1902)), with a weighting factor w usually = 0 . 5 .

Original: The selected
Revised: The selected

Original: WLFs that pass this pruning step form the committed labeler set F .
Revised: WLFs that pass this pruning step form the committed labeler set F .

Original: Then, for the downstream problem on ensembling, multiple techniques can be applied.
Revised: Then, for the downstream problem on ensembling, multiple techniques can be applied.

Original: For example, majority voting is one of the brute force methods.
Revised: For example, majority voting is one of the brute force methods.

Original: Other existing techniques such as repeated labeling could also apply (Ipeirotis et al., 2014).
Revised: Other existing techniques such as repeated labeling could also apply (Ipeirotis et al., 2014).

Original: Among these methods, Snorkel (Ratner et al., 2016a) learns a generative model on top of committed WLFs in form of
Revised: Among these methods, Snorkel (Ratner et al., 2016a) learns a generative model on top of committed WLFs in form of

Original: [Equation]
Revised: [Equation]

Original: where ˆ Y U and Y U are the aggregated label matrices of all committed WLF labels and ground-truth labels, ϕ is the parameters and Z ϕ is a normalization factor.
Revised: where ˆ Y U and Y U are the aggregated label matrices of all committed WLF labels and ground-truth labels, ϕ is the parameters and Z ϕ is a normalization factor.

Original: Snorkel trains this generative model such that it labels y U with high accuracy.
Revised: Snorkel trains this generative model such that it labels y U with high accuracy.

Original: This ensembling requires each WLF to have higher accuracy than random guessing, which is a low requirement for learners.
Revised: This ensembling requires each WLF to have higher accuracy than random guessing, which is a low requirement for learners.

Original: When the downstream ensembling uses a generative model as in equation 2, Snuba provides a probabilistic guarantee that the accuracy of the generative model on labeled data and unlabeled data has a maximum difference of ϵ with probability 1 − δ .
Revised: When the downstream ensembling uses a generative model as in equation 2, Snuba provides a probabilistic guarantee that the accuracy of the generative model on labeled data and unlabeled data has a maximum difference of ϵ with probability 1 − δ .

Original: This guarantee exists because Snuba checks an exit condition on WLFs, such that each committed WLF before termination must have a certain level of confidence on d data points, with
Revised: This guarantee exists because Snuba checks an exit condition on WLFs, such that each committed WLF before termination must have a certain level of confidence on d data points, with

Original: [Equation]
Revised: [Equation]

Original: Here, γ is the measured error, i.e., the difference of accuracy between the generative model and
Revised: Here, γ is the measured error, i.e., the difference of accuracy between the generative model and

Original: WLFs on the labeled data.
Revised: WLFs on the labeled data.

Original: Please refer to the original paper for a more detailed proof.
Revised: Please refer to the original paper for a more detailed proof.

Original: We also extend the proof of this guarantee to the continual learning setting in Appendix A.
Revised: We also extend the proof of this guarantee to the continual learning setting in Appendix A.

Original: 2. T RANSFERABILITY
Revised: 2. T RANSFERABILITY

Original: In transfer learning, transferring a trained model to a target task that shares no common features with the source task typically hurts the performance of the model, even worse than learning only the target task.
Revised: In transfer learning, transferring a trained model to a target task that shares no common features with the source task typically hurts the performance of the model, even worse than learning only the target task.

Original: As such, understanding the similarity of tasks and measuring transferability of a learning model from one task to another is a key aspect of not only transfer learning but also continual learning.
Revised: As such, understanding the similarity of tasks and measuring transferability of a learning model from one task to another is a key aspect of not only transfer learning but also continual learning.

Original: An intuitive metric of transferability is the accuracy of the source model on the target data (Tran et al., 2019; Dhillon et al., 2020): measuring how well the trained model performs on the target task.
Revised: An intuitive metric of transferability is the accuracy of the source model on the target data (Tran et al., 2019; Dhillon et al., 2020): measuring how well the trained model performs on the target task.

Original: LEEP (Nguyen et al., 2020) extends this metric by weighting the likelihood of the model with empirical conditional distribution of the target label given the source label.
Revised: LEEP (Nguyen et al., 2020) extends this metric by weighting the likelihood of the model with empirical conditional distribution of the target label given the source label.

Original: OTCE (Tan et al., 2021) quantifies the distance between two classification tasks as a sum of domain difference and task difference.
Revised: OTCE (Tan et al., 2021) quantifies the distance between two classification tasks as a sum of domain difference and task difference.

Original: The domain difference – the difference of the data distribution – is computed by optimal transport theory with entropic regularization, and the task difference – the difference of the classification objectives such as a set of classes – is derived from conditional entropy using the optimal coupling matrix of the optimal transport problem.
Revised: The domain difference – the difference of the data distribution – is computed by optimal transport theory with entropic regularization, and the task difference – the difference of the classification objectives such as a set of classes – is derived from conditional entropy using the optimal coupling matrix of the optimal transport problem.

Original: GBC (Pandy et al., 2022) adopts the Bhattacharyya coefficient that measures the amount of overlap between two distributions in the feature space of the source model.
Revised: GBC (Pandy et al., 2022) adopts the Bhattacharyya coefficient that measures the amount of overlap between two distributions in the feature space of the source model.

Original: Based on the positive results by using transferability score in transfer learning, this score can help figuring out the reusable knowledge of earlier tasks more effectively.
Revised: Based on the positive results by using transferability score in transfer learning, this score can help figuring out the reusable knowledge of earlier tasks more effectively.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: We aim to solve the following problem: How to design an SSCL framework, such that it is able to (1) minimize the performance gap between using partially labeled data and fully labeled data per task at low cost, (2) allow cumulative knowledge transfer to assist handling of unlabeled data, and (3) extend from arbitrary existing supervised CL frameworks?
Revised: We aim to solve the following problem: How to design an SSCL framework, such that it is able to (1) minimize the performance gap between using partially labeled data and fully labeled data per task at low cost, (2) allow cumulative knowledge transfer to assist handling of unlabeled data, and (3) extend from arbitrary existing supervised CL frameworks?

Original: Formally, in an SSCL problem, a lifelong learner will face a sequence of classification tasks {Z (1) , Z (2) , . . .
Revised: Formally, in an SSCL problem, a lifelong learner will face a sequence of classification tasks {Z (1) , Z (2) , . . .

Original: } , with each task Z ( i ) having data space X ( i )
Revised: } , with each task Z ( i ) having data space X ( i )

Original: ⊆
Revised: ⊆

Original: R d i , label space Y ( i ) = { 1 , . . .
Revised: R d i , label space Y ( i ) = { 1 , . . .

Original: , c i } , and a joint distribution D ( i ) :
Revised: , c i } , and a joint distribution D ( i ) :

Original: X ( i ) × Y ( i ) (cid:55)→ [0 , 1] governing the data-label pairs.
Revised: X ( i ) × Y ( i ) (cid:55)→ [0 , 1] governing the data-label pairs.

Original: For task Z ( i ) , we are given n ( i ) L labeled training data ( X ( i ) L , y
Revised: For task Z ( i ) , we are given n ( i ) L labeled training data ( X ( i ) L , y

Original: ( i ) L )
Revised: ( i ) L )

Original: ∼ D ( i ) , n ( i ) U unlabeled training data X ( i )
Revised: ∼ D ( i ) , n ( i ) U unlabeled training data X ( i )

Original: U ∼ X
Revised: U ∼ X

Original: ( i ) and n ( i ) T testing data ( X ( i ) T , y ( i ) T ) ∼ D ( i ) .
Revised: ( i ) and n ( i ) T testing data ( X ( i ) T , y ( i ) T ) ∼ D ( i ) .

Original: From this setting, we consider a two-part SSCL framework consisting of an upstream pseudo-labeler and a downstream supervised CL module that are isolated from each other’s processes.
Revised: From this setting, we consider a two-part SSCL framework consisting of an upstream pseudo-labeler and a downstream supervised CL module that are isolated from each other’s processes.

Original: Specifically, at task Z ( i ) , a pseudo-labeling function π ( i ) :
Revised: Specifically, at task Z ( i ) , a pseudo-labeling function π ( i ) :

Original: X ( i ) (cid:55)→ Y ( i ) is first trained.
Revised: X ( i ) (cid:55)→ Y ( i ) is first trained.

Original: Then, the unlabeled data
Revised: Then, the unlabeled data

Original: X ( i ) U will be labeled with ˆ y ( i ) U by this function.
Revised: X ( i ) U will be labeled with ˆ y ( i ) U by this function.

Original: Next, labeled data ( X ( i ) L , y ( i ) L ) and pseudo-labeled data ( X ( i ) U , ˆ y ( i ) U ) will be given to a supervised CL module for continual learning.
Revised: Next, labeled data ( X ( i ) L , y ( i ) L ) and pseudo-labeled data ( X ( i ) U , ˆ y ( i ) U ) will be given to a supervised CL module for continual learning.

Original: With this design, users are able to plug-and-play existing well-developed supervised CL tools and augment them into
Revised: With this design, users are able to plug-and-play existing well-developed supervised CL tools and augment them into

Original: SSCL, addressing sub-problem (3).
Revised: SSCL, addressing sub-problem (3).

Original: The design of the pseudo-labeling procedure addresses sub-problems (1) and (2).
Revised: The design of the pseudo-labeling procedure addresses sub-problems (1) and (2).

Original: Intuitively, the higher the pseudo-label quality, the smaller the gap is between using partially labeled and fully labeled data (as if each task Z ( i ) has all n ( i ) L + n ( i ) U labeled).
Revised: Intuitively, the higher the pseudo-label quality, the smaller the gap is between using partially labeled and fully labeled data (as if each task Z ( i ) has all n ( i ) L + n ( i ) U labeled).

Original: Hence, (1) asks for a labeler that can ideally approach perfect labeling by learning every underlying distribution D ( i ) , preferably with guaranteed label quality, and low time and memory overhead.
Revised: Hence, (1) asks for a labeler that can ideally approach perfect labeling by learning every underlying distribution D ( i ) , preferably with guaranteed label quality, and low time and memory overhead.

Original: Furthermore, (2) asks for a cumulative knowledge base and its input/output algorithm specifically for the pseudo-labeler.
Revised: Furthermore, (2) asks for a cumulative knowledge base and its input/output algorithm specifically for the pseudo-labeler.

Original: In the next section, we provide a concrete framework that meets these requirements.
Revised: In the next section, we provide a concrete framework that meets these requirements.

Original:  T HE DP-SSCL F RAMEWORK FOR S EMI -S UPERVISED C ONTINUAL L EARNING
Revised:  T HE DP-SSCL F RAMEWORK FOR S EMI -S UPERVISED C ONTINUAL L EARNING

Original: In this section, we introduce our proposed framework, Data Programming-based SSCL, or DPSSCL.
Revised: In this section, we introduce our proposed framework, Data Programming-based SSCL, or DPSSCL.

Original: The overall workflow is illustrated in Figure 1.
Revised: The overall workflow is illustrated in Figure 1.

Original: As shown, after a meta initialization, upon every new task, four modules are called in cascading order: (1) WLF initialization and training, (2)
Revised: As shown, after a meta initialization, upon every new task, four modules are called in cascading order: (1) WLF initialization and training, (2)

Original: WLF pruning, (3) pseudo-label ensembling and (4) supervised CL model updating.
Revised: WLF pruning, (3) pseudo-label ensembling and (4) supervised CL model updating.

Original: Our framework also includes light-weight storage for all committed WLFs throughout all the tasks.
Revised: Our framework also includes light-weight storage for all committed WLFs throughout all the tasks.

Original: 4. M ETA I NITIALIZATION
Revised: 4. M ETA I NITIALIZATION

Original: In a typical CL setting, the tasks are not available at once, but we are able to assume some general knowledge for the type of tasks (Liu, 2017; Ruvolo & Eaton, 2013a; Silver et al., 2013).
Revised: In a typical CL setting, the tasks are not available at once, but we are able to assume some general knowledge for the type of tasks (Liu, 2017; Ruvolo & Eaton, 2013a; Silver et al., 2013).

Original: For example, what input dimensions and what latent space that encodes representative features.
Revised: For example, what input dimensions and what latent space that encodes representative features.

Original: Therefore, we perform a meta initialization step to allow users to input these prior information before the first task.
Revised: Therefore, we perform a meta initialization step to allow users to input these prior information before the first task.

Original: Such prior knowledge includes a training hyperparameter search space, in which hyperparameters such as learning rate, batch sizes and number of epochs for WLF training will be grid-searched later as new tasks arise.
Revised: Such prior knowledge includes a training hyperparameter search space, in which hyperparameters such as learning rate, batch sizes and number of epochs for WLF training will be grid-searched later as new tasks arise.

Original: Other prior knowledge includes model templates, which define the architectures for WLFs, which are typically decision trees, regressors, or small neural networks not exceedinglayers or 1000 parameters.
Revised: Other prior knowledge includes model templates, which define the architectures for WLFs, which are typically decision trees, regressors, or small neural networks not exceedinglayers or 1000 parameters.

Original: The WLF storage is initialized to be empty.
Revised: The WLF storage is initialized to be empty.

Original: 4. WLF I NITIALIZATION VIA K NOWLEDGE T RANSFER
Revised: 4. WLF I NITIALIZATION VIA K NOWLEDGE T RANSFER

Original: Upon the arrival of a new task, DP-SSCL first acquires a set of WLFs, each in form of f weak :
Revised: Upon the arrival of a new task, DP-SSCL first acquires a set of WLFs, each in form of f weak :

Original: X ( i ) (cid:55)→ Y ( i ) ∪ { 0 } , where an additional 0 label means the confidence of f weak on a data point is lower than a given threshold t such that labeling is abstained.
Revised: X ( i ) (cid:55)→ Y ( i ) ∪ { 0 } , where an additional 0 label means the confidence of f weak on a data point is lower than a given threshold t such that labeling is abstained.

Original: To produce WLFs that can ensemble to an accurate pseudo-labeler, we consider two sources of WLF initialization: (1) random sampling on the pre-defined model templates in meta initialization, leveraging the user’s prior knowledge of the task and (2) transfer from previous tasks, leveraging knowledge accumulated throughout the continual learning process.
Revised: To produce WLFs that can ensemble to an accurate pseudo-labeler, we consider two sources of WLF initialization: (1) random sampling on the pre-defined model templates in meta initialization, leveraging the user’s prior knowledge of the task and (2) transfer from previous tasks, leveraging knowledge accumulated throughout the continual learning process.

Original: Knowledge is represented in the form of previously committed WLFs, that is, WLFs that enters the pseudo-labeler ensembling procedure in previous tasks Z (1) , . . .
Revised: Knowledge is represented in the form of previously committed WLFs, that is, WLFs that enters the pseudo-labeler ensembling procedure in previous tasks Z (1) , . . .

Original: , Z ( i − 1) .
Revised: , Z ( i − 1) .

Original: Upon task Z ( i ) , DP-SSCL evaluates the suitability of the previously committed WLFs for transfer to the current task using transferability measures, such as the LEEP or OTCE transferability score introduced in Section 2.4, and selects from the cached WLFs using the procedure shown in Algorithm 1.
Revised: Upon task Z ( i ) , DP-SSCL evaluates the suitability of the previously committed WLFs for transfer to the current task using transferability measures, such as the LEEP or OTCE transferability score introduced in Section 2.4, and selects from the cached WLFs using the procedure shown in Algorithm 1.

Original: Algorithm 1 returns one of the previously committed WLFs to initialize a new weak labeling function.
Revised: Algorithm 1 returns one of the previously committed WLFs to initialize a new weak labeling function.

Original: In this algorithm, the suitability score and the committed WLF selection criteria can be designed based on prior data or task knowledge.
Revised: In this algorithm, the suitability score and the committed WLF selection criteria can be designed based on prior data or task knowledge.

Original: For example, one can use only the task transferability score to sample tasks relevant to the current task, and then select a weak labeling function at random by treating WLFs of the sampled task uniformly.
Revised: For example, one can use only the task transferability score to sample tasks relevant to the current task, and then select a weak labeling function at random by treating WLFs of the sampled task uniformly.

Original: On the other hand, it is also possible to consider the labeling accuracy of the earlier WLFs in addition to task transferability for the selection of WLFs.
Revised: On the other hand, it is also possible to consider the labeling accuracy of the earlier WLFs in addition to task transferability for the selection of WLFs.

Original: Since there is no guarantee that all previously encountered tasks are closely related to the current task, a suitability score threshold ϕ excludes negatively correlated tasks from the pool of WLFs for transfer.
Revised: Since there is no guarantee that all previously encountered tasks are closely related to the current task, a suitability score threshold ϕ excludes negatively correlated tasks from the pool of WLFs for transfer.

Original: Similarly, we introduce another parameter – probability of initializing a WLF by transfer ρ
Revised: Similarly, we introduce another parameter – probability of initializing a WLF by transfer ρ

Original: Input
Revised: Input

Original: M ,
Revised: M ,

Original: Parameter :
Revised: Parameter :

Original: Suitability score
Revised: Suitability score

Original: – to control the ratio of the transferred WLFs to newly generated WLFs for the current task, which allows for the some WLFs to be initialized from scratch to maintain diversity in the WLF pool.
Revised: – to control the ratio of the transferred WLFs to newly generated WLFs for the current task, which allows for the some WLFs to be initialized from scratch to maintain diversity in the WLF pool.

Original: 4. WLF T RAINING AND P RUNING
Revised: 4. WLF T RAINING AND P RUNING

Original: After initialization, the framework obtains a set of both transferred and randomly generated WLFs, which are fine-tuned by labeled data at current task.
Revised: After initialization, the framework obtains a set of both transferred and randomly generated WLFs, which are fine-tuned by labeled data at current task.

Original: The training hyperparameters such as learning rate, batch size and epochs are selected from within the bounds specified during the meta initialization phase.
Revised: The training hyperparameters such as learning rate, batch size and epochs are selected from within the bounds specified during the meta initialization phase.

Original: The trained WLFs then pass through a Snuba pruner (Varma & R´e, 2016), which commits only the top functions ranked by score computed by Equation equation 1.
Revised: The trained WLFs then pass through a Snuba pruner (Varma & R´e, 2016), which commits only the top functions ranked by score computed by Equation equation 1.

Original: To improve the diversity of each WLF, we adopt bootstrapping on the training data, such that each WLF is trained on a randomly selected subset of the labeled data, with the boostrapped size specified during meta initialization.
Revised: To improve the diversity of each WLF, we adopt bootstrapping on the training data, such that each WLF is trained on a randomly selected subset of the labeled data, with the boostrapped size specified during meta initialization.

Original: The initialization and training procedure is repeated until either a maximum size of committed WLFs is met, or if the condition in equation 3 will become violated in the next iteration.
Revised: The initialization and training procedure is repeated until either a maximum size of committed WLFs is met, or if the condition in equation 3 will become violated in the next iteration.

Original: Consequently, although we include transferred functions in this procedure, our framework still maintains the Snuba guarantee on pseudo-labeler quality.
Revised: Consequently, although we include transferred functions in this procedure, our framework still maintains the Snuba guarantee on pseudo-labeler quality.

Original: A detailed proof of how DP-SSCL maintains this guarantee in the continual learning setting is presented in Appendix A.
Revised: A detailed proof of how DP-SSCL maintains this guarantee in the continual learning setting is presented in Appendix A.

Original: 4. P SEUDO - LABELER E NSEMBLING AND C ONTINUAL M ODEL U PDATE
Revised: 4. P SEUDO - LABELER E NSEMBLING AND C ONTINUAL M ODEL U PDATE

Original: The committed WLFs of task Z ( i ) enter ensembling, where different aggregators can be used.
Revised: The committed WLFs of task Z ( i ) enter ensembling, where different aggregators can be used.

Original: For instance, majority voting is a simple method to combine WLFs, as well as repeated labeling (Ipeirotis et al., 2014).
Revised: For instance, majority voting is a simple method to combine WLFs, as well as repeated labeling (Ipeirotis et al., 2014).

Original: More advanced ensembling methods are available, such as training a Snorkel (Ratner et al., 2016a) generative model π ( i ) in the form of equation 2.
Revised: More advanced ensembling methods are available, such as training a Snorkel (Ratner et al., 2016a) generative model π ( i ) in the form of equation 2.

Original: We empirically evlauate different ensembling techniques in an ablation study in Section 5.2.
Revised: We empirically evlauate different ensembling techniques in an ablation study in Section 5.2.

Original: Via the ensembled labeler, the framework obtains pseudo-labels ˆ y
Revised: Via the ensembled labeler, the framework obtains pseudo-labels ˆ y

Original: ( i ) U .
Revised: ( i ) U .

Original: After ensembling, if the confidence of the pseudo-labels are available, the pseudo-labels can be further adjusted by confidence calibration (Platt, 1999; Guo et al., 2017b).
Revised: After ensembling, if the confidence of the pseudo-labels are available, the pseudo-labels can be further adjusted by confidence calibration (Platt, 1999; Guo et al., 2017b).

Original: Last, together with labeled data ( X ( i ) L , y ( i ) L ) , the pseudo-labeled data ( X ( i ) U , ˆ y
Revised: Last, together with labeled data ( X ( i ) L , y ( i ) L ) , the pseudo-labeled data ( X ( i ) U , ˆ y

Original: ( i ) U ) enters a supervised continual learning module to update its model to learn a function in the form of f : (cid:83) ij =1 X ( j ) (cid:55)→ (cid:83) ij =1 Y ( j ) .
Revised: ( i ) U ) enters a supervised continual learning module to update its model to learn a function in the form of f : (cid:83) ij =1 X ( j ) (cid:55)→ (cid:83) ij =1 Y ( j ) .

Original: For the underlying model, our framework supports many state-of-the-art continual learning approaches, such as DF-CNN (Lee et al., 2019), TF (Bulat et al., 2020), and DEN (Yoon et al., 2018), which we demonstrate below.
Revised: For the underlying model, our framework supports many state-of-the-art continual learning approaches, such as DF-CNN (Lee et al., 2019), TF (Bulat et al., 2020), and DEN (Yoon et al., 2018), which we demonstrate below.

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: In this section, we first explain our instantiation of DP-SSCL from ablation study results on ensembling methods and WLF transfer.
Revised: In this section, we first explain our instantiation of DP-SSCL from ablation study results on ensembling methods and WLF transfer.

Original: Then, we explore how DP-SSCL’s lifelong performance with partially labeled data compares to that of supervised CL using fully labeled data sets, as well as how DP-SSCL performs compared to existing SSCL methods.
Revised: Then, we explore how DP-SSCL’s lifelong performance with partially labeled data compares to that of supervised CL using fully labeled data sets, as well as how DP-SSCL performs compared to existing SSCL methods.

Original: More experiments and discussion are presented in Appendix D.
Revised: More experiments and discussion are presented in Appendix D.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: tasks.
Revised: tasks.

Original: 5. S ETUP
Revised: 5. S ETUP

Original: We show that DP-SSCL is a general framework for SSCL by using different combinations of algorithms for strong label generation (majority voting, repeated labeling, and Snorkel) and supervised
Revised: We show that DP-SSCL is a general framework for SSCL by using different combinations of algorithms for strong label generation (majority voting, repeated labeling, and Snorkel) and supervised

Original: CL (TF Bulat et al.
Revised: CL (TF Bulat et al.

Original: , DEN Yoon et al.
Revised: , DEN Yoon et al.

Original: (2018), and DF-CNN Lee et al.
Revised: (2018), and DF-CNN Lee et al.

Original: Our experiments employ continual learning versions of standard image classification benchmarks, including
Revised: Our experiments employ continual learning versions of standard image classification benchmarks, including

Original: MNIST LeCun & Cortes (2010), CIFAR-10, and CIFAR-100 Krizhevsky (2009).
Revised: MNIST LeCun & Cortes (2010), CIFAR-10, and CIFAR-100 Krizhevsky (2009).

Original: For each task, we hold out 10% of labeled and unlabeled data for validation.
Revised: For each task, we hold out 10% of labeled and unlabeled data for validation.

Original: We evaluate performance using (1) peak per-task performance metrics, (2) final task performance metrics, and (3) forgetting metrics, with forgetting metrics measured as backward-transfer (Ruvolo & Eaton, 2013b; Lopez-Paz & Ranzato, 2017).
Revised: We evaluate performance using (1) peak per-task performance metrics, (2) final task performance metrics, and (3) forgetting metrics, with forgetting metrics measured as backward-transfer (Ruvolo & Eaton, 2013b; Lopez-Paz & Ranzato, 2017).

Original: The detailed experimental setup , including hyperparameter selection and evaluation metrics, is explained in the following subsections as well as in Appendix B.
Revised: The detailed experimental setup , including hyperparameter selection and evaluation metrics, is explained in the following subsections as well as in Appendix B.

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: The following two ablation studies are evaluated on validation data to select components for the main experiments in Section 5.3, which is then evaluated on separate testing data.
Revised: The following two ablation studies are evaluated on validation data to select components for the main experiments in Section 5.3, which is then evaluated on separate testing data.

Original: Ensembling Methods We evaluate three methods of pseudo-labeler ensembling discussed in Section 4.4: majority voting, repeated labeling, and Snorkel.
Revised: Ensembling Methods We evaluate three methods of pseudo-labeler ensembling discussed in Section 4.4: majority voting, repeated labeling, and Snorkel.

Original: As shown in Table 1 (refer Table 7 in
Revised: As shown in Table 1 (refer Table 7 in

Original: Appendix D for details), we see that Snorkel obtains consistently strong performance across continual learning algorithms, and amounts of unlabeled training data.
Revised: Appendix D for details), we see that Snorkel obtains consistently strong performance across continual learning algorithms, and amounts of unlabeled training data.

Original: Consequently, we use Snorkel as the label generator within DP-SSCL for all other experiments.
Revised: Consequently, we use Snorkel as the label generator within DP-SSCL for all other experiments.

Original: WLF Transfer We instantiate our framework with knowledge transfer in WLFs by using DF-CNN (CL model), LEEP score (suitability score) and binary CIFAR10 experiment.
Revised: WLF Transfer We instantiate our framework with knowledge transfer in WLFs by using DF-CNN (CL model), LEEP score (suitability score) and binary CIFAR10 experiment.

Original: As depicted in Figure 2, both peak per-task accuracy and final accuracy slightly increase in comparison to DP-SSCL without WLF transfer (black dashed lines) when approximately a half of WLFs are initialized via transfer.
Revised: As depicted in Figure 2, both peak per-task accuracy and final accuracy slightly increase in comparison to DP-SSCL without WLF transfer (black dashed lines) when approximately a half of WLFs are initialized via transfer.

Original: Due to the fact that the increment is small, we pick a high LEEP score threshold that results in no transfer for all the remaining experiments, without a loss in generality.
Revised: Due to the fact that the increment is small, we pick a high LEEP score threshold that results in no transfer for all the remaining experiments, without a loss in generality.

Original: Though the accuracies are within 95% confidence interval of the baseline, these enhanced CL performances show its positive potential of knowledge transfer in weak labeling functions.
Revised: Though the accuracies are within 95% confidence interval of the baseline, these enhanced CL performances show its positive potential of knowledge transfer in weak labeling functions.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: More ablation studies evaluated on validation data are detailed in Appendix D
Revised: More ablation studies evaluated on validation data are detailed in Appendix D

Original: [Table]
Revised: [Table]

Original: 5. M AIN R ESULTS 5.3. SSCL P ERFORMANCE
Revised: 5. M AIN R ESULTS 5.3. SSCL P ERFORMANCE

Original: Based on the instantiation of Snorkel ensembler and no LEEP transfer, we compare DP-SSCL to three state-of-the-art SSCL algorithms: CNNL (Baucum et al., 2017), ORDisCo (Wang et al., 2021), and DistillMatch (Smith et al., 2021).
Revised: Based on the instantiation of Snorkel ensembler and no LEEP transfer, we compare DP-SSCL to three state-of-the-art SSCL algorithms: CNNL (Baucum et al., 2017), ORDisCo (Wang et al., 2021), and DistillMatch (Smith et al., 2021).

Original: To enable fair comparisons, we replicate the experimental conditions and compare our approach to the best results reported in each algorithm’s original publication.
Revised: To enable fair comparisons, we replicate the experimental conditions and compare our approach to the best results reported in each algorithm’s original publication.

Original: Specifically, we replicate the instance-incremental learning experiments of CNNL on MNIST and CIFAR-10, and the class-incremental learning experiments of ORDisCo and DistillMatch on
Revised: Specifically, we replicate the instance-incremental learning experiments of CNNL on MNIST and CIFAR-10, and the class-incremental learning experiments of ORDisCo and DistillMatch on

Original: CIFAR-10 and CIFAR-100, respectively.
Revised: CIFAR-10 and CIFAR-100, respectively.

Original: The replicated experimental protocols are summarized below; please refer to the original papers for detailed setups.
Revised: The replicated experimental protocols are summarized below; please refer to the original papers for detailed setups.

Original: For instance-incremental experiments, all tasks are present at every epoch, but subsequent epochs contain different batches of unlabeled data.
Revised: For instance-incremental experiments, all tasks are present at every epoch, but subsequent epochs contain different batches of unlabeled data.

Original: Each experiment was run over 10 random seeds.
Revised: Each experiment was run over 10 random seeds.

Original: As shown in Table 2, DP-SSCL achieves 90% of final testing accuracy as if fully labeled data is used, comparable to CNNL in MNIST and higher in CIFAR-10.
Revised: As shown in Table 2, DP-SSCL achieves 90% of final testing accuracy as if fully labeled data is used, comparable to CNNL in MNIST and higher in CIFAR-10.

Original: Moreover, on both benchmarks DP-SSCL has comparable or higher sample efficiency measured in batches to saturation, which means the first epoch where a 3-batch sliding window average meets or exceeds the final accuracy.
Revised: Moreover, on both benchmarks DP-SSCL has comparable or higher sample efficiency measured in batches to saturation, which means the first epoch where a 3-batch sliding window average meets or exceeds the final accuracy.

Original: For class-incremental experiments, the model is sequentially presented with tasks containing new sets of classes.
Revised: For class-incremental experiments, the model is sequentially presented with tasks containing new sets of classes.

Original: Each experiment is run with 10 random seeds.
Revised: Each experiment is run with 10 random seeds.

Original: As shown in Table 3, when equipped with certain supervised module (DF-CNN), DP-SSCL is able to achieve comparable final testing accuracy to supervised CL using fully labeled data, strictly exceeding that of ORDisCo and DistillMatch.
Revised: As shown in Table 3, when equipped with certain supervised module (DF-CNN), DP-SSCL is able to achieve comparable final testing accuracy to supervised CL using fully labeled data, strictly exceeding that of ORDisCo and DistillMatch.

Original: In terms of catastrophic forgetting, we depict backward transfer as a percentage, and so lies in the range [ − 100% , 100%] , and the higher value the less forgetting occurs.
Revised: In terms of catastrophic forgetting, we depict backward transfer as a percentage, and so lies in the range [ − 100% , 100%] , and the higher value the less forgetting occurs.

Original: We can see that DP-SSCL also produces similar forgetting as if fully labeled data is used.
Revised: We can see that DP-SSCL also produces similar forgetting as if fully labeled data is used.

Original: This result shows that DP-SSCL is able to capture the properties of continually shifting data and tasks, and generate appropriate pseudo-labels for the lifelong learners.
Revised: This result shows that DP-SSCL is able to capture the properties of continually shifting data and tasks, and generate appropriate pseudo-labels for the lifelong learners.

Original: 5.3. SSCL C OMPUTATIONAL C
Revised: 5.3. SSCL C OMPUTATIONAL C

Original: OST
Revised: OST

Original: We analyze DP-SSCL’s computational cost required to leverage unlabeled data compared to stateof-the-art SSCL methods, shown in Table 4, where m is the size of one data point, b is the total number of data batches in a task, and r is the number of iterations in CNNL’s pseudo-labeling loop.
Revised: We analyze DP-SSCL’s computational cost required to leverage unlabeled data compared to stateof-the-art SSCL methods, shown in Table 4, where m is the size of one data point, b is the total number of data batches in a task, and r is the number of iterations in CNNL’s pseudo-labeling loop.

Original: Each scalar parameter in a machine learning model is assumed to take a 4B floating point.
Revised: Each scalar parameter in a machine learning model is assumed to take a 4B floating point.

Original: We conclude that our DP-SSCL has small overhead in terms of both memory and time for unlabeled data processing, which we elaborate on below.
Revised: We conclude that our DP-SSCL has small overhead in terms of both memory and time for unlabeled data processing, which we elaborate on below.

Original: For memory, CNNL maintains a constant-length queue of data, such that the queue size is proportional to the size of a data point.
Revised: For memory, CNNL maintains a constant-length queue of data, such that the queue size is proportional to the size of a data point.

Original: A typical queue length is 1000 as used in their experiments, with each MNIST data costing 784 B and CIFAR 3072 B. ORDisCo, DistillMatch and DP-SSCL all have a constant O (1) overhead once the architectures are fixed.
Revised: A typical queue length is 1000 as used in their experiments, with each MNIST data costing 784 B and CIFAR 3072 B. ORDisCo, DistillMatch and DP-SSCL all have a constant O (1) overhead once the architectures are fixed.

Original: Nevertheless, in implementation,
Revised: Nevertheless, in implementation,

Original: ORDisCo requires storage of its generator (G) and discriminator (D), which approximately costs 100 MB for parameters, estimated from the last table in (Li et al., 2017) reporting ORDisCo’s architecture.
Revised: ORDisCo requires storage of its generator (G) and discriminator (D), which approximately costs 100 MB for parameters, estimated from the last table in (Li et al., 2017) reporting ORDisCo’s architecture.

Original: DistillMatch requires storage for its OoD detector, which is 254 MB for a ResNet34-based
Revised: DistillMatch requires storage for its OoD detector, which is 254 MB for a ResNet34-based

Original: DeConf network (Gao et al., 2021).
Revised: DeConf network (Gao et al., 2021).

Original: On the other hand, DP-SSCL utilizes small weak labelers at 0 .
Revised: On the other hand, DP-SSCL utilizes small weak labelers at 0 .

Original: 1 MB-level, with 0 .
Revised: 1 MB-level, with 0 .

Original: 083 MB and 0 .
Revised: 083 MB and 0 .

Original: 108 MB for MNIST and CIFAR10/100 benchmarks, respectively.
Revised: 108 MB for MNIST and CIFAR10/100 benchmarks, respectively.

Original: We set a buffer of maximally 25 weak labelers and reallocate the buffer upon each task, so the overhead is 0 . 

Revised: We set a buffer of maximally 25 weak labelers and reallocate the buffer upon each task, so the overhead is 0 .

Original: × 25 = 2 .
Revised: × 25 = 2 .

Original: 7 MB for MNIST and 0 .
Revised: 7 MB for MNIST and 0 .

Original: 108 × 25 = 2 .
Revised: 108 × 25 = 2 .

Original: 75 MB for CIFAR10/100.
Revised: 75 MB for CIFAR10/100.

Original: Since the existing SSCL papers do not present timing measures, we compare them using time complexity.
Revised: Since the existing SSCL papers do not present timing measures, we compare them using time complexity.

Original: Upon every batch of data, the CNNL algorithm repeatedly labels the unlabeled data for an unbounded number of iterations, denoted as r , until it is confident of the labels.
Revised: Upon every batch of data, the CNNL algorithm repeatedly labels the unlabeled data for an unbounded number of iterations, denoted as r , until it is confident of the labels.

Original: For ORDisCo,
Revised: For ORDisCo,

Original: DistillMatch and DP-SSCL, the G + D, OoD detector and weak labelers are trained on all data batches sequentially, so the time overhead is proportional to the number of batches.
Revised: DistillMatch and DP-SSCL, the G + D, OoD detector and weak labelers are trained on all data batches sequentially, so the time overhead is proportional to the number of batches.

Original: The conclusion is that DP-SSCL has either lower or the same time complexity to utilize unlabeled data for learning compared to existing SSCL tools.
Revised: The conclusion is that DP-SSCL has either lower or the same time complexity to utilize unlabeled data for learning compared to existing SSCL tools.

Original: [Table]
Revised: [Table]

Original: More experiments are run to show that DP-SSCL improves its performance on larger size of unlabeled data per task, maintains stable learning on increasing number of tasks
Revised: More experiments are run to show that DP-SSCL improves its performance on larger size of unlabeled data per task, maintains stable learning on increasing number of tasks

Original: [Table]
Revised: [Table]

Original: and is sensitive to corrupted pseudo-labels.
Revised: and is sensitive to corrupted pseudo-labels.

Original: These studies are detailed in Appendix D.
Revised: These studies are detailed in Appendix D.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We designed an SSCL framework, namely DP-SSCL
Revised: We designed an SSCL framework, namely DP-SSCL

Original: [Table]
Revised: [Table]

Original: that leverages a DP-based pseudo-labeler and a supervised CL module in cascade manner with a feedback loop.
Revised: that leverages a DP-based pseudo-labeler and a supervised CL module in cascade manner with a feedback loop.

Original: This design allows us to obtain high quality pseudo-labels, shrinking the performance gap between SSCL and supervised CL.
Revised: This design allows us to obtain high quality pseudo-labels, shrinking the performance gap between SSCL and supervised CL.

Original: Our framework also shows how CL can improve DP by allowing knowledge transfer from previous tasks to improve pseudo labeling quality based on transferability metrics such as LEEP score.
Revised: Our framework also shows how CL can improve DP by allowing knowledge transfer from previous tasks to improve pseudo labeling quality based on transferability metrics such as LEEP score.

Original: Furthermore, the framework is compatible with many existing mature supervised CL approaches, enabling trivial extension from supervised to semi-supervised CL.
Revised: Furthermore, the framework is compatible with many existing mature supervised CL approaches, enabling trivial extension from supervised to semi-supervised CL.

Original: Our ablation studies show that the framework’s performance depend on component selection, and succeeding research shall focus on exploring different component settings more thoroughly, such as different instantiations of Algorithm 1.
Revised: Our ablation studies show that the framework’s performance depend on component selection, and succeeding research shall focus on exploring different component settings more thoroughly, such as different instantiations of Algorithm 1.

Original: Nevertheless, experiments show DP-SSCL is able to output high lifelong learning accuracy and low forgetting, approaching that of supervised CL on fully labeled data, and outperforming existing SSCL approaches by up to 25% higher final accuracy and lower forgetting, with only 1% of memory overhead at the same time complexity.
Revised: Nevertheless, experiments show DP-SSCL is able to output high lifelong learning accuracy and low forgetting, approaching that of supervised CL on fully labeled data, and outperforming existing SSCL approaches by up to 25% higher final accuracy and lower forgetting, with only 1% of memory overhead at the same time complexity.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Our experiment code is pushed to an anonymous GitHub repository (https://github.com/dpsscl-anon/
Revised: Our experiment code is pushed to an anonymous GitHub repository (https://github.com/dpsscl-anon/

Original: DPSSCL).
Revised: DPSSCL).

Original: All readers are more than welcomed to replicate our experiments.
Revised: All readers are more than welcomed to replicate our experiments.

Original: Appendices A P ROOF OF S NUBA T HEORETICAL G UARANTEE IN A L IFELONG S ETTING
Revised: Appendices A P ROOF OF S NUBA T HEORETICAL G UARANTEE IN A L IFELONG S ETTING

Original: This section extends Snuba’s guaranteed performance, equation 3 in the main paper, to lifelong setting.
Revised: This section extends Snuba’s guaranteed performance, equation 3 in the main paper, to lifelong setting.

Original: For completeness, we start by stating proposition:
Revised: For completeness, we start by stating proposition:

Original: Proposition 1 (Snuba’s guarantee (Varma & R ´ e, 2016), adapted to a lifelong setting) .
Revised: Proposition 1 (Snuba’s guarantee (Varma & R ´ e, 2016), adapted to a lifelong setting) .

Original: Consider consecutive tasks Z (1) , Z (2) , . . .
Revised: Consider consecutive tasks Z (1) , Z (2) , . . .

Original: for which Snuba was used to obtain sets of committed weak labelers
Revised: for which Snuba was used to obtain sets of committed weak labelers

Original: F (1) , F (2) , . . .
Revised: F (1) , F (2) , . . .

Original: for all tasks with corresponding empirical accuracies on X ( i ) L as a vector a ( i ) L .
Revised: for all tasks with corresponding empirical accuracies on X ( i ) L as a vector a ( i ) L .

Original: For each task, before ensembling, a factor graph-based generative model trains a set of fine-tuned weak labelers from F ( i ) , denoted as ˜ F
Revised: For each task, before ensembling, a factor graph-based generative model trains a set of fine-tuned weak labelers from F ( i ) , denoted as ˜ F

Original: ( i ) and | ˜ F ( i )
Revised: ( i ) and | ˜ F ( i )

Original: | = | F ( i ) | , that have accuracies on X ( i ) L as ˜ a ( i ) L and on X ( i ) U as ˜ a ( i ) U , with a ( i ) L , ˜ a ( i ) L , ˜ a ( i ) U ∈ R
Revised: | = | F ( i ) | , that have accuracies on X ( i ) L as ˜ a ( i ) L and on X ( i ) U as ˜ a ( i ) U , with a ( i ) L , ˜ a ( i ) L , ˜ a ( i ) U ∈ R

Original: | F ( i ) | and ˜ a ( i ) U is unknown.
Revised: | F ( i ) | and ˜ a ( i ) U is unknown.

Original: We have a measured error
Revised: We have a measured error

Original: || a ( i ) L − ˜ a ( i ) L || ∞ ≤ ϵ ( i ) .
Revised: || a ( i ) L − ˜ a ( i ) L || ∞ ≤ ϵ ( i ) .

Original: If each labeler labels a minimum of
Revised: If each labeler labels a minimum of

Original: [Equation]
Revised: [Equation]

Original: data points in X ( i ) L with above some given confidence threshold ν for all iterations, we can guarantee that ||
Revised: data points in X ( i ) L with above some given confidence threshold ν for all iterations, we can guarantee that ||

Original: ˜ a ( i ) U − ˜ a ( i ) L || ∞ < γ for all iterations and all tasks with probability 1 − δ .
Revised: ˜ a ( i ) U − ˜ a ( i ) L || ∞ < γ for all iterations and all tasks with probability 1 − δ .

Original: Proof.
Revised: Proof.

Original: We know that Snuba ensures the following bound on the labeling performance of an individual task
Revised: We know that Snuba ensures the following bound on the labeling performance of an individual task

Original: Z ( i ) learned in isolation; the following guarantee is a restatement from the original Snuba publication (Varma & R´e, 2016), modified only to include superscripts for the task index i :
Revised: Z ( i ) learned in isolation; the following guarantee is a restatement from the original Snuba publication (Varma & R´e, 2016), modified only to include superscripts for the task index i :

Original: Proposition 2 (Snuba’s guarantee (Varma & R´e, 2016) for an individual task
Revised: Proposition 2 (Snuba’s guarantee (Varma & R´e, 2016) for an individual task

Original: Z ( i ) ) .
Revised: Z ( i ) ) .

Original: Given a set F ( i ) of committed weak labelers by Snuba for task Z ( i ) , denote their empirical accuracies on X ( i ) L as a vector a ( i ) L .
Revised: Given a set F ( i ) of committed weak labelers by Snuba for task Z ( i ) , denote their empirical accuracies on X ( i ) L as a vector a ( i ) L .

Original: Before ensembling, a factor graph-based generative model trains a set of fine-tuned weak labelers from F ( i ) , denoted as ˜ F
Revised: Before ensembling, a factor graph-based generative model trains a set of fine-tuned weak labelers from F ( i ) , denoted as ˜ F

Original: ( i ) and | ˜ F ( i )
Revised: ( i ) and | ˜ F ( i )

Original: | = | F ( i ) | , that have accuracies on X ( i ) L as ˜ a ( i ) L and on X ( i ) U as ˜ a ( i ) U , with a ( i ) L , ˜ a ( i ) L , ˜ a ( i ) U ∈ R
Revised: | = | F ( i ) | , that have accuracies on X ( i ) L as ˜ a ( i ) L and on X ( i ) U as ˜ a ( i ) U , with a ( i ) L , ˜ a ( i ) L , ˜ a ( i ) U ∈ R

Original: | F ( i ) | and ˜ a ( i ) U is unknown.
Revised: | F ( i ) | and ˜ a ( i ) U is unknown.

Original: We have a measured error
Revised: We have a measured error

Original: || a ( i ) L − ˜ a ( i ) L || ∞ ≤ ϵ .
Revised: || a ( i ) L − ˜ a ( i ) L || ∞ ≤ ϵ .

Original: If each labeler labels a minimum of
Revised: If each labeler labels a minimum of

Original: [Equation]
Revised: [Equation]

Original: data points in X ( i ) L with above some given confidence threshold ν for all iterations, we can guarantee that ||
Revised: data points in X ( i ) L with above some given confidence threshold ν for all iterations, we can guarantee that ||

Original: ˜ a ( i ) U − ˜ a ( i ) L || ∞ < γ for all iterations with probability 1 − δ .
Revised: ˜ a ( i ) U − ˜ a ( i ) L || ∞ < γ for all iterations with probability 1 − δ .

Original: See (Varma & R´e, 2016) for the proof of Proposition 2.
Revised: See (Varma & R´e, 2016) for the proof of Proposition 2.

Original: We now need to show that this same guarantee holds in a lifelong setting.
Revised: We now need to show that this same guarantee holds in a lifelong setting.

Original: Assume we have a sequence of T consecutive tasks Z (1) , Z (2) , . . .
Revised: Assume we have a sequence of T consecutive tasks Z (1) , Z (2) , . . .

Original: , Z ( T ) .
Revised: , Z ( T ) .

Original: Since the first task Z (1) is learned in isolation to produce the set of weak labelers F (1) using Snuba, by Proposition 2 we know that we have a measured error for the first task of
Revised: Since the first task Z (1) is learned in isolation to produce the set of weak labelers F (1) using Snuba, by Proposition 2 we know that we have a measured error for the first task of

Original: [Equation]
Revised: [Equation]

Original: and that
Revised: and that

Original: [Equation]
Revised: [Equation]

Original: for all iterations with probability 1 − δ .
Revised: for all iterations with probability 1 − δ .

Original: Let us assume that the bound holds for each F ( i ) after learning Z (1) , Z (2) , . . .
Revised: Let us assume that the bound holds for each F ( i ) after learning Z (1) , Z (2) , . . .

Original: , Z ( T ) ; we will show that this bound also holds for task
Revised: , Z ( T ) ; we will show that this bound also holds for task

Original: Z ( T +1) .
Revised: Z ( T +1) .

Original: From tasks Z (1) , . . .
Revised: From tasks Z (1) , . . .

Original: , Z ( T ) , Snuba has learned a set of weak labelers F = (cid:83) Ti =1 F ( i ) .
Revised: , Z ( T ) , Snuba has learned a set of weak labelers F = (cid:83) Ti =1 F ( i ) .

Original: When creating the set of weak labelers F ( T +1) incrementally, Snuba has two choices at each iteration j .
Revised: When creating the set of weak labelers F ( T +1) incrementally, Snuba has two choices at each iteration j .

Original: Either it can choose to add to F ( T +1) an existing weak labeler f ∈ F − F ( T +1) or it can add a previously unused weak labeler f ′ ∈ U − F − F ( T +1) , where U is the pool of all candidate weak labelers.
Revised: Either it can choose to add to F ( T +1) an existing weak labeler f ∈ F − F ( T +1) or it can add a previously unused weak labeler f ′ ∈ U − F − F ( T +1) , where U is the pool of all candidate weak labelers.

Original: If Snuba chose f ′ over f , then weak labeler f ′ was assigned a higher score than all other f ∈ F .
Revised: If Snuba chose f ′ over f , then weak labeler f ′ was assigned a higher score than all other f ∈ F .

Original: Similarly, if Snuba chose a particular f ∈ F to add, then that f was assigned a higher score than all others.
Revised: Similarly, if Snuba chose a particular f ∈ F to add, then that f was assigned a higher score than all others.

Original: Let ˆ f j be the weak labeler chosen to add to F ( T +1) at iteration j , either f or f ′ , which we know has maximum score of all weak labelers in U − F ( T +1) .
Revised: Let ˆ f j be the weak labeler chosen to add to F ( T +1) at iteration j , either f or f ′ , which we know has maximum score of all weak labelers in U − F ( T +1) .

Original: Snuba obtains an accuracy on the labeled data set X
Revised: Snuba obtains an accuracy on the labeled data set X

Original: ( T +1) L of
Revised: ( T +1) L of

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˆ X ( T +1) L,j ⊆ X
Revised: where ˆ X ( T +1) L,j ⊆ X

Original: ( T +1) L such that ˆ f j achieves a confidence greater than or equal to the confidence threshold ν on each data point in ˆ X
Revised: ( T +1) L such that ˆ f j achieves a confidence greater than or equal to the confidence threshold ν on each data point in ˆ X

Original: ( T +1)
Revised: ( T +1)

Original: L , y ( T +1)
Revised: L , y ( T +1)

Original: k is the true label, and ˆ y ( T +1) k is the predicted label by the weak labeler
Revised: k is the true label, and ˆ y ( T +1) k is the predicted label by the weak labeler

Original: ˆ f j .
Revised: ˆ f j .

Original: (Varma & R ´ e, 2016) (see their Equation 4) show that the probability of Snuba’s failure to maintain || ˜ a ( T +1) U − ˜ a ( T +
Revised: (Varma & R ´ e, 2016) (see their Equation 4) show that the probability of Snuba’s failure to maintain || ˜ a ( T +1) U − ˜ a ( T +

Original: [Equation]
Revised: [Equation]

Original: || ∞ <
Revised: || ∞ <

Original: γ in one iteration is
Revised: γ in one iteration is

Original: [Equation]
Revised: [Equation]

Original: Following (Varma & R´e, 2016) and applying the union bound over the sequence of iterations to bound the probability of failure over all iterations used to acquire F ( T +1) , we can obtain that
Revised: Following (Varma & R´e, 2016) and applying the union bound over the sequence of iterations to bound the probability of failure over all iterations used to acquire F ( T +1) , we can obtain that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where d ( T +1) =
Revised: where d ( T +1) =

Original: min (cid:16) | ˆ X ( T +1) L,| , . .
Revised: min (cid:16) | ˆ X ( T +1) L,| , . .

Original: , | ˆ X ( T +1)
Revised: , | ˆ X ( T +1)

Original: L,j | (cid:17) , and so consequently ||
Revised: L,j | (cid:17) , and so consequently ||

Original: ˜ a ( T +1)
Revised: ˜ a ( T +1)

Original: U − ˜ a ( T +1)
Revised: U − ˜ a ( T +1)

Original: L || ∞ <
Revised: L || ∞ <

Original: γ similarly holds for Z ( T +1) for all iterations used to obtain F ( T +1) with probability 1 − δ .
Revised: γ similarly holds for Z ( T +1) for all iterations used to obtain F ( T +1) with probability 1 − δ .

Original: By induction, this holds for the entire lifelong sequence.
Revised: By induction, this holds for the entire lifelong sequence.

Original: [Table]
Revised: [Table]

Original: B D ETAILED E XPERIMENTAL S ETUP
Revised: B D ETAILED E XPERIMENTAL S ETUP

Original: This section details the experimental setup used in Section 5 of the main paper.
Revised: This section details the experimental setup used in Section 5 of the main paper.

Original: Our lifelong learning experiments uses the MNIST, CIFAR-10, and CIFAR-100 benchmark datasets (LeCun & Cortes, 2010; Krizhevsky, 2009).
Revised: Our lifelong learning experiments uses the MNIST, CIFAR-10, and CIFAR-100 benchmark datasets (LeCun & Cortes, 2010; Krizhevsky, 2009).

Original: To transform these benchmarks into the lifelong setting, we created tasks as described in Table 5.
Revised: To transform these benchmarks into the lifelong setting, we created tasks as described in Table 5.

Original: For example, to re-create the binary MNIST task sequence, one needs to first arrange the 45 tasks as { 0 vs 1, 0 vs 2, ..., 8 vs 9 } .
Revised: For example, to re-create the binary MNIST task sequence, one needs to first arrange the 45 tasks as { 0 vs 1, 0 vs 2, ..., 8 vs 9 } .

Original: Then, split the labeled training data, unlabeled training data, and testing data into 120/11880/2000 per task, ensuring that all data splits have balanced classes.
Revised: Then, split the labeled training data, unlabeled training data, and testing data into 120/11880/2000 per task, ensuring that all data splits have balanced classes.

Original: We then hold out 10% of training data for validation.
Revised: We then hold out 10% of training data for validation.

Original: That is, only 108 labeled and 10692 unlabeled data participate in training, with 12 labeled and 1188 used for validation.
Revised: That is, only 108 labeled and 10692 unlabeled data participate in training, with 12 labeled and 1188 used for validation.

Original: Finally, perform our procedure described in Section 4 to generate pseudo-labels as well as train a lifelong learner.
Revised: Finally, perform our procedure described in Section 4 to generate pseudo-labels as well as train a lifelong learner.

Original: To train WLFs for data programming, keeps fine-tuning hyperparameters representing the WLF architectures, learning rate, and Snuba configurations in the given search spaces.
Revised: To train WLFs for data programming, keeps fine-tuning hyperparameters representing the WLF architectures, learning rate, and Snuba configurations in the given search spaces.

Original: Since the hyperparameters keep adapting to the current task even throughout a single lifelong sequence, we report these hyperparameters and their constrained search spaces in Table 5 and Figure 4.
Revised: Since the hyperparameters keep adapting to the current task even throughout a single lifelong sequence, we report these hyperparameters and their constrained search spaces in Table 5 and Figure 4.

Original: The search spaces were inspired by various previous works on CNN designs (Lecun et al., 1998;
Revised: The search spaces were inspired by various previous works on CNN designs (Lecun et al., 1998;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: Nonetheless, the final architectures are much smaller.
Revised: Nonetheless, the final architectures are much smaller.

Original: Figure 3 presents the pseudo-labeling accuracy by Snorkel-ensembled pseudo-labeler Ratner et al.
Revised: Figure 3 presents the pseudo-labeling accuracy by Snorkel-ensembled pseudo-labeler Ratner et al.

Original: (2016a), which is the best labeler according to our ablation studies.
Revised: (2016a), which is the best labeler according to our ablation studies.

Original: The mean accuracies of binary
Revised: The mean accuracies of binary

Original: MNIST, binary CIFAR-10, and 5-way MNIST are all around 90% .
Revised: MNIST, binary CIFAR-10, and 5-way MNIST are all around 90% .

Original: Although the accuracies of 10-way CIFAR-10 and 5-way CIFAR-100 are 60 − 65% due to difficulty, we can still outperform existing SSCL frameworks as demonstrated in Section 5.3.
Revised: Although the accuracies of 10-way CIFAR-10 and 5-way CIFAR-100 are 60 − 65% due to difficulty, we can still outperform existing SSCL frameworks as demonstrated in Section 5.3.

Original: We then detail our metric selection for the experiments.
Revised: We then detail our metric selection for the experiments.

Original: The performance of CL can be quantitatively measured by (1) peak per-task performance metrics, (2) final task performance metrics, and (3)
Revised: The performance of CL can be quantitatively measured by (1) peak per-task performance metrics, (2) final task performance metrics, and (3)

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: forgetting metrics.
Revised: forgetting metrics.

Original: After training on task Z ( i ) , we measure the accuracy a ( i ) j of the updated model when evaluated on the testing data of all known tasks {Z ( j ) : j ≤ i } .
Revised: After training on task Z ( i ) , we measure the accuracy a ( i ) j of the updated model when evaluated on the testing data of all known tasks {Z ( j ) : j ≤ i } .

Original: Peak per-task accuracy (Lee et al., 2019) at task Z ( i ) measures the average accuracy upon the first encounter of each task:
Revised: Peak per-task accuracy (Lee et al., 2019) at task Z ( i ) measures the average accuracy upon the first encounter of each task:

Original: ˜ a i = 

Revised: ˜ a i =

Original: i (cid:80) ij =1 a ( j ) j , and final accuracy (Lopez-Paz & Ranzato, 2017) at task i measures the average performance of the current model on all tasks seen so far: ¯ a i = 

Revised: i (cid:80) ij =1 a ( j ) j , and final accuracy (Lopez-Paz & Ranzato, 2017) at task i measures the average performance of the current model on all tasks seen so far: ¯ a i =

Original: i (cid:80) ij =1 a ( i ) j .
Revised: i (cid:80) ij =1 a ( i ) j .

Original: The retention of knowledge is measured by backward transfer (Ruvolo & Eaton, 2013b; Lopez-Paz & Ranzato, 2017) at task Z ( i ) as bt i =
Revised: The retention of knowledge is measured by backward transfer (Ruvolo & Eaton, 2013b; Lopez-Paz & Ranzato, 2017) at task Z ( i ) as bt i =

Original: 1 i − 1 (cid:80) i − 1 j =1 a ( i ) j − a ( j ) j .
Revised: 1 i − 1 (cid:80) i − 1 j =1 a ( i ) j − a ( j ) j .

Original: Note that bt i ∈ [ − 1 , 1] is the negative of the forgetting metric from (Chaudhry et al., 2019), where positive values indicate improvement and negative values indicate forgetting.
Revised: Note that bt i ∈ [ − 1 , 1] is the negative of the forgetting metric from (Chaudhry et al., 2019), where positive values indicate improvement and negative values indicate forgetting.

Original: In addition to the lifelong learning metrics, we measure pseudo-labeling accuracy and system overhead of DP-SSCL in terms for memory cost.
Revised: In addition to the lifelong learning metrics, we measure pseudo-labeling accuracy and system overhead of DP-SSCL in terms for memory cost.

Original: We also report the proportion of weak labelers using knowledge of earlier tasks and the transferability score to analyze the effect of knowledge sharing on data programming.
Revised: We also report the proportion of weak labelers using knowledge of earlier tasks and the transferability score to analyze the effect of knowledge sharing on data programming.

Original: C A DDITIONAL E XPERIMENT A NALYSIS OF S EMI -S UPERVISED C ONTINUAL L EARNING
Revised: C A DDITIONAL E XPERIMENT A NALYSIS OF S EMI -S UPERVISED C ONTINUAL L EARNING

Original: This appendix details and provides additional results that complement Section 5.2 and Section 5.3.1.
Revised: This appendix details and provides additional results that complement Section 5.2 and Section 5.3.1.

Original: We start with the ablation studies that are not covered in the main paper.
Revised: We start with the ablation studies that are not covered in the main paper.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Sizes of Unlabeled Data We measure the performance of continual learners with the groundtruth/generated labels on different supervised methods while varying the quantity of unlabeled training data.
Revised: Sizes of Unlabeled Data We measure the performance of continual learners with the groundtruth/generated labels on different supervised methods while varying the quantity of unlabeled training data.

Original: Table 6 summarizes the empirical results, providing details in addition to Figure 5.
Revised: Table 6 summarizes the empirical results, providing details in addition to Figure 5.

Original: Figure 6 visualizes the learning curves of final accuracy in these experiments, showing stable performance of lifelong learners regardless of the size of unlabeled data, number of tasks, and supervised modules.
Revised: Figure 6 visualizes the learning curves of final accuracy in these experiments, showing stable performance of lifelong learners regardless of the size of unlabeled data, number of tasks, and supervised modules.

Original: Peak per-task accuracy of both continual models increases with more DP-SSCL-labeled data.
Revised: Peak per-task accuracy of both continual models increases with more DP-SSCL-labeled data.

Original: This supports the capability of DP-SSCL to capture the data distribution and assign representative labels.
Revised: This supports the capability of DP-SSCL to capture the data distribution and assign representative labels.

Original: It is well documented that continual models typically have increased negative transfer (i.e., inference or catastrophic forgetting) when trained on more data.
Revised: It is well documented that continual models typically have increased negative transfer (i.e., inference or catastrophic forgetting) when trained on more data.

Original: However, negative transfer increase is not amplified by using DP-SSCL-generated labels in place of true labels - the continual performance with DP-SSCL-generated pseudo-labels achieves at least 96% of the continual performance when models are trained on the same training data but with true labels instead.
Revised: However, negative transfer increase is not amplified by using DP-SSCL-generated labels in place of true labels - the continual performance with DP-SSCL-generated pseudo-labels achieves at least 96% of the continual performance when models are trained on the same training data but with true labels instead.

Original: Consequently, DP-SSCL maintains stable performance at increasing scale of unlabeled data.
Revised: Consequently, DP-SSCL maintains stable performance at increasing scale of unlabeled data.

Original: In real-world deployment, labeled data has limited availability but unlabeled data is easy to be collected.
Revised: In real-world deployment, labeled data has limited availability but unlabeled data is easy to be collected.

Original: Therefore, DP-SSCL’s scalability supports practical continual learning.
Revised: Therefore, DP-SSCL’s scalability supports practical continual learning.

Original: Number of Tasks Figure 6 also demonstrates that DP-SSCL preserves relatively stable continual learning performance on increasing number of tasks under different supervised CL modules and task sequences.
Revised: Number of Tasks Figure 6 also demonstrates that DP-SSCL preserves relatively stable continual learning performance on increasing number of tasks under different supervised CL modules and task sequences.

Original: This result entails that the quality of pseudo-labels generated by DP-SSCL is stable with respect to the number of tasks.
Revised: This result entails that the quality of pseudo-labels generated by DP-SSCL is stable with respect to the number of tasks.

Original: Pseudo-label Noise Levels
Revised: Pseudo-label Noise Levels

Original: We also examined the effect of introducing noise to DP-SSCL’s label generation process: randomly corrupting a portion of the generated labels among the unlabeled data.
Revised: We also examined the effect of introducing noise to DP-SSCL’s label generation process: randomly corrupting a portion of the generated labels among the unlabeled data.

Original: Figure 7 shows an inverse relationship between manually added pseudo-label noises and lifelong performance.
Revised: Figure 7 shows an inverse relationship between manually added pseudo-label noises and lifelong performance.

Original: This shows that CL is sensitive to inaccurate pseudo-labels and using more accurate labeling, such as more diverse WLFs on DP, is necessary.
Revised: This shows that CL is sensitive to inaccurate pseudo-labels and using more accurate labeling, such as more diverse WLFs on DP, is necessary.

Original: We next detail the ablation study on ensembling methods in Section 5.2.
Revised: We next detail the ablation study on ensembling methods in Section 5.2.

Original: Ensembling Methods (More)
Revised: Ensembling Methods (More)

Original: We measure the performance of two continual learners, TF (Bulat et al., 2020) and DF-CNN (Lee et al., 2019), on binary MNIST and binary CIFAR-10 experiments
Revised: We measure the performance of two continual learners, TF (Bulat et al., 2020) and DF-CNN (Lee et al., 2019), on binary MNIST and binary CIFAR-10 experiments

Original: [Table]
Revised: [Table]

Original: (45 binary classification tasks of pairs of classes such as 0 vs 1, 0 vs 2, ..., 8 vs 9).
Revised: (45 binary classification tasks of pairs of classes such as 0 vs 1, 0 vs 2, ..., 8 vs 9).

Original: We ran each
Revised: We ran each

Original: MNIST and CIFAR-10 experiment over 10 and 5 random seeds and task sequences, respectively.
Revised: MNIST and CIFAR-10 experiment over 10 and 5 random seeds and task sequences, respectively.

Original: Table 7 shows all the peak per-task accuracy, final accuracy and backward transfer with respect to the pseudo-labeling methods and the amount of training data.
Revised: Table 7 shows all the peak per-task accuracy, final accuracy and backward transfer with respect to the pseudo-labeling methods and the amount of training data.

Original: We see that Snorkel obtains consistently strong performance across continual learning algorithms, data sets
Revised: We see that Snorkel obtains consistently strong performance across continual learning algorithms, data sets

Original: [Table]
Revised: [Table]

Original: and amounts of unlabeled training data.
Revised: and amounts of unlabeled training data.

Original: Finally, we discuss the learning curves in these experiments.
Revised: Finally, we discuss the learning curves in these experiments.

Original: More detailed experiment result statistics are listed in Tables 8, 9, 11 and 10.
Revised: More detailed experiment result statistics are listed in Tables 8, 9, 11 and 10.

Original: Learning Curve We include learning curves for the instance-incremental semi-supervised lifelong experiments (Figure 8) and the class-incremental semi-supervised lifelong experiments (Figure 9).
Revised: Learning Curve We include learning curves for the instance-incremental semi-supervised lifelong experiments (Figure 8) and the class-incremental semi-supervised lifelong experiments (Figure 9).

Original: All training curves show final accuracy at task Z
Revised: All training curves show final accuracy at task Z

Original: ( i ) as defined in Section ?? , where task Z ( i ) is the current task being trained.
Revised: ( i ) as defined in Appendix B, where task Z ( i ) is the current task being trained.

Original: For easier tasks with high labeling accuracy, such as CIFAR-10,
Revised: For easier tasks with high labeling accuracy, such as CIFAR-10,

Original: the
Revised: the

Original: DP-SSCL-enabled semi-supervised continual learning methods perform similarly to the equivalent
Revised: DP-SSCL-enabled semi-supervised continual learning methods perform similarly to the equivalent

Original: 30  n ( i ) U
Revised: 30  n ( i ) U

Original: F i n a l
Revised: F i n a l

Original: A cc u r a c y 0% Noise 20% Noise 50% Noise (
Revised: A cc u r a c y 0% Noise 20% Noise 50% Noise (

Original: [Figure]
Revised: [Figure]

Original: Binary MNIST    n ( i ) U
Revised: Binary MNIST    n ( i ) U

Original: F i n a l
Revised: F i n a l

Original: A cc u r a c y 0% Noise 20% Noise 50% Noise (
Revised: A cc u r a c y 0% Noise 20% Noise 50% Noise (

Original: [Figure]
Revised: [Figure]

Original: Continual Learner
Revised: Continual Learner

Original: Labeling Method
Revised: Labeling Method

Original: MNIST Final Accuracy n (
Revised: MNIST Final Accuracy n (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: =n ( i ) U
Revised: =n ( i ) U

Original: = 120 n ( i )
Revised: = 120 n ( i )

Original: [Table]
Revised: [Table]

Original: DP-SSCL (TF)
Revised: DP-SSCL (TF)

Original: [Table]
Revised: [Table]

Original: DP-SSCL (DF-CNN)
Revised: DP-SSCL (DF-CNN)

Original: [Table]
Revised: [Table]

Original: Continual
Revised: Continual

Original: [Table]
Revised: [Table]

Original: Labeling Method
Revised: Labeling Method

Original: CIFAR
Revised: CIFAR

Original: [Table]
Revised: [Table]

Original: n ( i ) U
Revised: n ( i ) U

Original: = 400 n ( i ) U =
Revised: = 400 n ( i ) U =

Original: [Table]
Revised: [Table]

Original: supervised fully-labeled task sequence.
Revised: supervised fully-labeled task sequence.

Original: For a further breakdown of results, we include task-specific performance measures for all of the semi-supervised lifelong experiments in Tables 8–
Revised: For a further breakdown of results, we include task-specific performance measures for all of the semi-supervised lifelong experiments in Tables 8–

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D A DDITIONAL E XPERIMENT A NALYSIS OF DP-SSCL WITH K NOWLEDGE T RANSFER IN W EAK L ABELERS
Revised: D A DDITIONAL E XPERIMENT A NALYSIS OF DP-SSCL WITH K NOWLEDGE T RANSFER IN W EAK L ABELERS

Original: This appendix details and provides additional results that complement Section 5.2 with respect to the transfer of weak labeling functions across tasks.
Revised: This appendix details and provides additional results that complement Section 5.2 with respect to the transfer of weak labeling functions across tasks.

Original: As described in Section 4.2, we can utilize the previously trained weak labelers for the current task if the current task has enough similarity with some earlier tasks.
Revised: As described in Section 4.2, we can utilize the previously trained weak labelers for the current task if the current task has enough similarity with some earlier tasks.

Original: The key component is the suitability score in Algorithm 1 determining which earlier tasks or which earlier WLFs are sufficiently related to the current task.
Revised: The key component is the suitability score in Algorithm 1 determining which earlier tasks or which earlier WLFs are sufficiently related to the current task.

Original: The main paper showed experimental results using the LEEP transferability score as the suitability score.
Revised: The main paper showed experimental results using the LEEP transferability score as the suitability score.

Original: In this appendix, we experiment with other suitability scores.
Revised: In this appendix, we experiment with other suitability scores.

Original: Specifically we compare the following similarity measures:
Revised: Specifically we compare the following similarity measures:

Original: • the LEEP transferability score , as used in the original paper.
Revised: • the LEEP transferability score , as used in the original paper.

Original: LEEP provides task-level information on the similarity of earlier tasks to the current one.
Revised: LEEP provides task-level information on the similarity of earlier tasks to the current one.

Original: • LEEP + Snuba score .
Revised: • LEEP + Snuba score .

Original: Instead of only relying on task transferability, this measure incorporates Snuba’s s-score (Equation 1) to measure how earlier WLFs perform on the current task prior to fine-tuning.
Revised: Instead of only relying on task transferability, this measure incorporates Snuba’s s-score (Equation 1) to measure how earlier WLFs perform on the current task prior to fine-tuning.

Original: Specifically, we computed it as s WLF ( w ) =
Revised: Specifically, we computed it as s WLF ( w ) =

Original: α L LEEP( M , X ( i ) L , y ( i ) L )+ α F F1( w, X
Revised: α L LEEP( M , X ( i ) L , y ( i ) L )+ α F F1( w, X

Original: ( i ) L , y
Revised: ( i ) L , y

Original: ( i ) L )
Revised: ( i ) L )

Original: + α J Jaccard( w, X ( i ) L ,
Revised: + α J Jaccard( w, X ( i ) L ,

Original: y
Revised: y

Original: ( i ) L ) where w ∈ W is an individual WLF,
Revised: ( i ) L ) where w ∈ W is an individual WLF,

Original: M is a CL model and { α L , α F , α J } are weights of the sum.
Revised: M is a CL model and { α L , α F , α J } are weights of the sum.

Original: In this experiment, all terms were weighted identically ( α L = α F = α J = 1 / 3 ), and the LEEP score was projected to be in the range [0 , 1] for compatibility with the other terms via the exponential function ( ( − inf , 0)
Revised: In this experiment, all terms were weighted identically ( α L = α F = α J = 1 / 3 ), and the LEEP score was projected to be in the range [0 , 1] for compatibility with the other terms via the exponential function ( ( − inf , 0)

Original: → (0 , 1) ).
Revised: → (0 , 1) ).

Original: We evaluated the different suitability scores in our approach on the CIFAR-10 dataset, usingbinary classification tasks formed by all pairs of two image classes (see Table 5).
Revised: We evaluated the different suitability scores in our approach on the CIFAR-10 dataset, usingbinary classification tasks formed by all pairs of two image classes (see Table 5).

Original: As in the experiments from Section 5.2, DF-CNN is used as the continual learner.
Revised: As in the experiments from Section 5.2, DF-CNN is used as the continual learner.

Original: For both experiments of ( n ( i ) L , n ( i ) U ) = (400 , 200) and ( n ( i ) L , n ( i ) U ) =
Revised: For both experiments of ( n ( i ) L , n ( i ) U ) = (400 , 200) and ( n ( i ) L , n ( i ) U ) =

Original: (50 , 400) , transferring WLFs makes the continual learner achieve equal or better peak per-task accuracy and final accuracy compared to the performance of DP-SSCL without WLF transfer.
Revised: (50 , 400) , transferring WLFs makes the continual learner achieve equal or better peak per-task accuracy and final accuracy compared to the performance of DP-SSCL without WLF transfer.

Original: This experiment shows that this result is robust to the choice of suitability score.
Revised: This experiment shows that this result is robust to the choice of suitability score.

Original: [Table]
Revised: [Table]