Original: M EASURING F ORGETTING OF M EMORIZED T RAINING E XAMPLES
Revised: M EASURING F ORGETTING OF M EMORIZED T RAINING E XAMPLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Machine learning models exhibit two seemingly contradictory phenomena training data memorization
Revised: Machine learning models exhibit two seemingly contradictory phenomena training data memorization

Original: , and various forms of forgetting
Revised: , and various forms of forgetting

Original: In memorization
Revised: In memorization

Original: , models overﬁt speciﬁc training examples and become susceptible to privacy attacks
Revised: , models overﬁt speciﬁc training examples and become susceptible to privacy attacks

Original: In forgetting
Revised: In forgetting

Original: , examples which appeared early in training are forgotten by the end.
Revised: , examples which appeared early in training are forgotten by the end.

Original: In this work, we connect these phenomena.
Revised: In this work, we connect these phenomena.

Original: We propose a technique to measure to what extent models “forget” the speciﬁcs of training examples, becoming less susceptible to privacy attacks on examples they have not seen recently.
Revised: We propose a technique to measure to what extent models “forget” the speciﬁcs of training examples, becoming less susceptible to privacy attacks on examples they have not seen recently.

Original: We show that, while non-convex models can memorize data forever in the worst-case, standard image, speech and language models empirically do forget examples over time.
Revised: We show that, while non-convex models can memorize data forever in the worst-case, standard image, speech and language models empirically do forget examples over time.

Original: We identify nondeterminism as a potential explanation, showing that deterministically trained models do not forget.
Revised: We identify nondeterminism as a potential explanation, showing that deterministically trained models do not forget.

Original: Our results suggest that examples seen early when training with extremely large datasets—for instance those examples used to pre-train a model—may observe privacy beneﬁts at the expense of examples seen later
Revised: Our results suggest that examples seen early when training with extremely large datasets—for instance those examples used to pre-train a model—may observe privacy beneﬁts at the expense of examples seen later

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning models are capable of memorizing information contained in their training data (Shokri et al., 2017; Carlini et al., 2019).
Revised: Machine learning models are capable of memorizing information contained in their training data (Shokri et al., 2017; Carlini et al., 2019).

Original: This is one of the reasons why models are vulnerable to privacy attacks such as membership inference (Homer et al., 2008) and training data extraction (Carlini et al., 2019).
Revised: This is one of the reasons why models are vulnerable to privacy attacks such as membership inference (Homer et al., 2008) and training data extraction (Carlini et al., 2019).

Original: Resulting privacy concerns have led to a variety of techniques for private machine learning, including differentially private training (Abadi et al., 2016;
Revised: Resulting privacy concerns have led to a variety of techniques for private machine learning, including differentially private training (Abadi et al., 2016;

Original: Papernot et al., 2016; 2018b; Ghazi et al., 2021; Malek et al., 2021), machine unlearning (Bourtoule et al., 2021; Neel et al., 2021; Sekhari et al., 2021), and various heuristics like regularization (Nasr et al., 2018), data augmentation (Amid et al., 2022) or gradient clipping (Carlini et al., 2019; Thakkar et al., 2020;
Revised: Papernot et al., 2016; 2018b; Ghazi et al., 2021; Malek et al., 2021), machine unlearning (Bourtoule et al., 2021; Neel et al., 2021; Sekhari et al., 2021), and various heuristics like regularization (Nasr et al., 2018), data augmentation (Amid et al., 2022) or gradient clipping (Carlini et al., 2019; Thakkar et al., 2020;

Original: Huang et al., 2022).
Revised: Huang et al., 2022).

Original: These techniques all modify the learning procedure to actively limit privacy leakage, including leakage that results from memorization.
Revised: These techniques all modify the learning procedure to actively limit privacy leakage, including leakage that results from memorization.

Original: Instead, we observe that the training dynamics inherent to learning algorithms such as stochastic gradient descent may passively afford some forms of privacy.
Revised: Instead, we observe that the training dynamics inherent to learning algorithms such as stochastic gradient descent may passively afford some forms of privacy.

Original: Such dynamics include forgetting: during iterative training, as models see new training examples, they could lose track of the speciﬁcs of earlier examples—as seen in work on catastrophic forgetting (French, 1999; McCloskey & Cohen, 1989; Kemker et al., 2018).
Revised: Such dynamics include forgetting: during iterative training, as models see new training examples, they could lose track of the speciﬁcs of earlier examples—as seen in work on catastrophic forgetting (French, 1999; McCloskey & Cohen, 1989; Kemker et al., 2018).

Original: In this paper, we study to what extent the forgetting exhibited by machine learning models has an impact on privacy.
Revised: In this paper, we study to what extent the forgetting exhibited by machine learning models has an impact on privacy.

Original: Our work is focused at distinguishing between two overarching hypotheses for how privacy interacts with forgetting.
Revised: Our work is focused at distinguishing between two overarching hypotheses for how privacy interacts with forgetting.

Original: For privacy, the pessimistic hypothesis is that memorization is a stronger effect than forgetting: traces of early training examples remain detectable by attacks long after the examples are seen in training, perhaps due to their inﬂuence on the initial decisions made during optimization.
Revised: For privacy, the pessimistic hypothesis is that memorization is a stronger effect than forgetting: traces of early training examples remain detectable by attacks long after the examples are seen in training, perhaps due to their inﬂuence on the initial decisions made during optimization.

Original: The optimistic hypothesis is that forgetting is stronger: early examples will be forgotten due to the many subsequent updates to the model as training progresses.
Revised: The optimistic hypothesis is that forgetting is stronger: early examples will be forgotten due to the many subsequent updates to the model as training progresses.

Original: Studying the impact of forgetting on privacy is most relevant when there is a large variation in how recently an example may be seen during training, allowing earlier examples to experience improved privacy relative to later examples.
Revised: Studying the impact of forgetting on privacy is most relevant when there is a large variation in how recently an example may be seen during training, allowing earlier examples to experience improved privacy relative to later examples.

Original: Indeed, models are increasingly trained on extremely large training sets, so that training consists of only a few epochs (or even a single one).
Revised: Indeed, models are increasingly trained on extremely large training sets, so that training consists of only a few epochs (or even a single one).

Original: Such settings are used when training large image models (Dai et al., 2021; Mahajan et al., 2018), multimodal models (Radford et al., 2021) and language models (Komatsuzaki, 2019; Chowdhery et al., 2022; Hoffmann et al., 2022; Zhang et al., 2022), which have come under signiﬁcant scrutiny due to privacy concerns (Carlini et al., 2021b; Bender et al., 2021).
Revised: Such settings are used when training large image models (Dai et al., 2021; Mahajan et al., 2018), multimodal models (Radford et al., 2021) and language models (Komatsuzaki, 2019; Chowdhery et al., 2022; Hoffmann et al., 2022; Zhang et al., 2022), which have come under signiﬁcant scrutiny due to privacy concerns (Carlini et al., 2021b; Bender et al., 2021).

Original: Similarly, when a model is being ﬁne-tuned, the data that was originally used to pretrain the model is no longer seen in the second stage of training.
Revised: Similarly, when a model is being ﬁne-tuned, the data that was originally used to pretrain the model is no longer seen in the second stage of training.

Original: Fine-tuning is also an ubiquitous technique in many domains, especially in language (Devlin et al., 2018), speech (Chung et al., 2021), and vision (Radford et al., 2021; Kornblith et al., 2019) tasks.
Revised: Fine-tuning is also an ubiquitous technique in many domains, especially in language (Devlin et al., 2018), speech (Chung et al., 2021), and vision (Radford et al., 2021; Kornblith et al., 2019) tasks.

Original: We design a methodology for measuring whether, and how quickly, individual examples are forgotten and become less vulnerable to privacy attacks.
Revised: We design a methodology for measuring whether, and how quickly, individual examples are forgotten and become less vulnerable to privacy attacks.

Original: Our methodology builds on state-of-the-art membership inference attacks (Carlini et al., 2021a;
Revised: Our methodology builds on state-of-the-art membership inference attacks (Carlini et al., 2021a;

Original: Ye et al., 2021), the best known method for testing whether a given point was used in training.
Revised: Ye et al., 2021), the best known method for testing whether a given point was used in training.

Original: We use our methodology to show that, for deep neural networks trained on language, speech, or vision tasks, examples used early in training (and not repeated later on) are indeed forgotten by the model.
Revised: We use our methodology to show that, for deep neural networks trained on language, speech, or vision tasks, examples used early in training (and not repeated later on) are indeed forgotten by the model.

Original: We identify a number of factors which impact the speed at which forgetting happens, such as when examples appear in training, or whether they are duplicated.
Revised: We identify a number of factors which impact the speed at which forgetting happens, such as when examples appear in training, or whether they are duplicated.

Original: We then attempt to understand why forgetting happens by showcasing two settings where forgetting does not happen in the worst-case: non-convex models, such as k -means, and deterministic training algorithms with a large amount of adversary knowledge.
Revised: We then attempt to understand why forgetting happens by showcasing two settings where forgetting does not happen in the worst-case: non-convex models, such as k -means, and deterministic training algorithms with a large amount of adversary knowledge.

Original: Our result on k -means is the ﬁrst instance of privacy leakage due to non-convexity we are aware of.
Revised: Our result on k -means is the ﬁrst instance of privacy leakage due to non-convexity we are aware of.

Original: However, on a mean estimation task, we prove that forgetting does happen as a result of the stochasticity of gradient descent, with similar properties to our empirical results.
Revised: However, on a mean estimation task, we prove that forgetting does happen as a result of the stochasticity of gradient descent, with similar properties to our empirical results.

Original: By using our approach to measuring forgetting, we hope experts training models on large datasets or ﬁne-tuning models can determine whether (and how much) forgetting improves empirical privacy in their training pipelines.
Revised: By using our approach to measuring forgetting, we hope experts training models on large datasets or ﬁne-tuning models can determine whether (and how much) forgetting improves empirical privacy in their training pipelines.

Original: We stress that our approach is complimentary to frameworks that offer worst-case guarantees, like differential privacy, and that it should not be used in lieu of reasoning about privacy within such frameworks.
Revised: We stress that our approach is complimentary to frameworks that offer worst-case guarantees, like differential privacy, and that it should not be used in lieu of reasoning about privacy within such frameworks.

Original:  B ACKGROUND AND R ELATED W ORK 2. D EFINING P RIVACY IN M ACHINE L EARNING
Revised:  B ACKGROUND AND R ELATED W ORK 2. D EFINING P RIVACY IN M ACHINE L EARNING

Original: There are multiple valid privacy guarantees that have been considered for ML algorithms.
Revised: There are multiple valid privacy guarantees that have been considered for ML algorithms.

Original: First, differential privacy ensures that the distribution of the output of the algorithm does not signiﬁcantly change when a single example is changed.
Revised: First, differential privacy ensures that the distribution of the output of the algorithm does not signiﬁcantly change when a single example is changed.

Original: In the context of ML, differential privacy can be obtained through modifying either the training algorithm (Chaudhuri et al., 2011; Abadi et al., 2016) or the inference algorithm (Papernot et al., 2016; Bassily et al., 2018;
Revised: In the context of ML, differential privacy can be obtained through modifying either the training algorithm (Chaudhuri et al., 2011; Abadi et al., 2016) or the inference algorithm (Papernot et al., 2016; Bassily et al., 2018;

Original: Papernot et al., 2018a).
Revised: Papernot et al., 2018a).

Original: Differential privacy provably bounds the success of privacy attacks which leak information about individual training examples (see Section 2.2).
Revised: Differential privacy provably bounds the success of privacy attacks which leak information about individual training examples (see Section 2.2).

Original: More recently, other motivations for privacy have gathered interest.
Revised: More recently, other motivations for privacy have gathered interest.

Original: For instance, in machine unlearning (Cao & Yang, 2015; Ginart et al., 2019), a user may issue a “deletion request”, after which their individual contribution to the model must be erased.
Revised: For instance, in machine unlearning (Cao & Yang, 2015; Ginart et al., 2019), a user may issue a “deletion request”, after which their individual contribution to the model must be erased.

Original: This is different from differential privacy, which requires that the model not learn too much about any of its training examples.
Revised: This is different from differential privacy, which requires that the model not learn too much about any of its training examples.

Original: Algorithms for machine unlearning have been proposed for k -means
Revised: Algorithms for machine unlearning have been proposed for k -means

Original: (Ginart et al., 2019), empirical risk minimization (Guo et al., 2019;
Revised: (Ginart et al., 2019), empirical risk minimization (Guo et al., 2019;

Original: Izzo et al., 2021; Neel et al., 2021;
Revised: Izzo et al., 2021; Neel et al., 2021;

Original: Ullah et al., 2021; Sekhari et al., 2021), and deep learning (Du et al., 2019; Golatkar et al., 2020; 2021; Nguyen et al., 2020; Bourtoule et al., 2021).
Revised: Ullah et al., 2021; Sekhari et al., 2021), and deep learning (Du et al., 2019; Golatkar et al., 2020; 2021; Nguyen et al., 2020; Bourtoule et al., 2021).

Original: If a point is perfectly unlearned, privacy attacks cannot succeed on this point.
Revised: If a point is perfectly unlearned, privacy attacks cannot succeed on this point.

Original: Both of these deﬁnitions of privacy—differential privacy and unlearning—obtain privacy actively: they require that the training algorithm be modiﬁed to obtain privacy.
Revised: Both of these deﬁnitions of privacy—differential privacy and unlearning—obtain privacy actively: they require that the training algorithm be modiﬁed to obtain privacy.

Original: Instead, we propose to capture privacy that is gained passively from dynamics that are inherent to training.
Revised: Instead, we propose to capture privacy that is gained passively from dynamics that are inherent to training.

Original: We deﬁne and measure forgetting as a form of privacy that arises from the decay in the extractable information about an individual training point over the course of training.
Revised: We deﬁne and measure forgetting as a form of privacy that arises from the decay in the extractable information about an individual training point over the course of training.

Original: Our deﬁnition of forgetting is inspired by the widely observed phenomenon of catastrophic forgetting (French, 1999; McCloskey & Cohen, 1989;
Revised: Our deﬁnition of forgetting is inspired by the widely observed phenomenon of catastrophic forgetting (French, 1999; McCloskey & Cohen, 1989;

Original: Kirkpatrick et al., 2017; Kemker et al., 2018; Davidson & Mozer, 2020;
Revised: Kirkpatrick et al., 2017; Kemker et al., 2018; Davidson & Mozer, 2020;

Original: Kaushik et al., 2021), where a model tends to forget previously learned knowledge when training on new data.
Revised: Kaushik et al., 2021), where a model tends to forget previously learned knowledge when training on new data.

Original: More speciﬁcally, catastrophic forgetting is generally formulated in the continual learning setting where the model sequentially learns a number of different tasks, and the performance on previously learned tasks drops signiﬁcantly as the model learns a new task.
Revised: More speciﬁcally, catastrophic forgetting is generally formulated in the continual learning setting where the model sequentially learns a number of different tasks, and the performance on previously learned tasks drops signiﬁcantly as the model learns a new task.

Original: In contrast, our work considers a model trained to solve a single ﬁxed task and measures how it forgets some of its training examples seen earlier in training.
Revised: In contrast, our work considers a model trained to solve a single ﬁxed task and measures how it forgets some of its training examples seen earlier in training.

Original: Our work is also inspired by Feldman et al.
Revised: Our work is also inspired by Feldman et al.

Original: (2018), who show theoretically that iterative, differentially-private algorithms can exhibit forgetting on convex tasks.
Revised: (2018), who show theoretically that iterative, differentially-private algorithms can exhibit forgetting on convex tasks.

Original: This means that the provable bounds on privacy leakage are better for samples seen earlier than those seen later.
Revised: This means that the provable bounds on privacy leakage are better for samples seen earlier than those seen later.

Original: Since provable privacy bounds are often loose (Jagielski et al., 2020;
Revised: Since provable privacy bounds are often loose (Jagielski et al., 2020;

Original: Nasr et al., 2021), it is unknown whether early examples also exhibit better privacy empirically.
Revised: Nasr et al., 2021), it is unknown whether early examples also exhibit better privacy empirically.

Original: We investigate this question here.
Revised: We investigate this question here.

Original: 2. A TTACKS AGAINST P RIVACY IN M ACHINE L EARNING
Revised: 2. A TTACKS AGAINST P RIVACY IN M ACHINE L EARNING

Original: To empirically measure privacy, we use attacks that target individual training examples.
Revised: To empirically measure privacy, we use attacks that target individual training examples.

Original: We consider two types of privacy attacks: membership inference and training data extraction .
Revised: We consider two types of privacy attacks: membership inference and training data extraction .

Original: Both of these privacy attacks have been shown to improve when models are updated repeatedly, if the updated models are released to the adversary (Salem et al., 2020;
Revised: Both of these privacy attacks have been shown to improve when models are updated repeatedly, if the updated models are released to the adversary (Salem et al., 2020;

Original: Zanella-B´eguelin et al., 2020; Jagielski et al., 2022).
Revised: Zanella-B´eguelin et al., 2020; Jagielski et al., 2022).

Original: In our work, we only release the model at the end of all training, so repeated updates do not leak information.
Revised: In our work, we only release the model at the end of all training, so repeated updates do not leak information.

Original: Membership Inference.
Revised: Membership Inference.

Original: In a membership inference attack (Homer et al., 2008; Dwork et al., 2015;
Revised: In a membership inference attack (Homer et al., 2008; Dwork et al., 2015;

Original: Yeom et al., 2018; Shokri et al., 2017), an adversary infers whether or not a target example was contained in a model’s training set.
Revised: Yeom et al., 2018; Shokri et al., 2017), an adversary infers whether or not a target example was contained in a model’s training set.

Original: Most techniques for membership inference predict membership using the loss on the target example: training data is more likely to have low loss than data not in training.
Revised: Most techniques for membership inference predict membership using the loss on the target example: training data is more likely to have low loss than data not in training.

Original: While early techniques applied the same threshold to all examples (Yeom et al., 2018; Shokri et al., 2017), current state-of-the-art membership inference attacks (Long et al., 2020; Sablayrolles et al., 2019; Watson et al., 2021; Carlini et al., 2021a) choose a carefully calibrated threshold for each example, which adjusts to the “hardness” of learning different examples.
Revised: While early techniques applied the same threshold to all examples (Yeom et al., 2018; Shokri et al., 2017), current state-of-the-art membership inference attacks (Long et al., 2020; Sablayrolles et al., 2019; Watson et al., 2021; Carlini et al., 2021a) choose a carefully calibrated threshold for each example, which adjusts to the “hardness” of learning different examples.

Original: We perform calibration and measure both the accuracy and true positive rate (TPR) of attacks at a ﬁxed false positive rate (FPR), both of which are bounded by differential privacy (Kairouz et al., 2015; Thudi et al., 2022).
Revised: We perform calibration and measure both the accuracy and true positive rate (TPR) of attacks at a ﬁxed false positive rate (FPR), both of which are bounded by differential privacy (Kairouz et al., 2015; Thudi et al., 2022).

Original: Training Data Extraction.
Revised: Training Data Extraction.

Original: In training data extraction (Carlini et al., 2021b), the adversary seeks to recover training data from the model.
Revised: In training data extraction (Carlini et al., 2021b), the adversary seeks to recover training data from the model.

Original: One controlled experiment to measure extraction risk is canary extraction (Carlini et al., 2019; Thakkar et al., 2020).
Revised: One controlled experiment to measure extraction risk is canary extraction (Carlini et al., 2019; Thakkar et al., 2020).

Original: In canary extraction, m well-formatted canaries { s i } mi =1 are injected into a model’s training set, chosen uniformly at random from some larger universe of secret canaries S .
Revised: In canary extraction, m well-formatted canaries { s i } mi =1 are injected into a model’s training set, chosen uniformly at random from some larger universe of secret canaries S .

Original: The adversary’s goal is to guess which of the S canaries was in fact inserted.
Revised: The adversary’s goal is to guess which of the S canaries was in fact inserted.

Original: Success is measured with exposure, which computes the reduced entropy in guessing the secret:
Revised: Success is measured with exposure, which computes the reduced entropy in guessing the secret:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term here measures the total number of possible canaries, and the second term measures the number of possible secrets in S which have a smaller loss (cid:96) than the true secret s .
Revised: The ﬁrst term here measures the total number of possible canaries, and the second term measures the number of possible secrets in S which have a smaller loss (cid:96) than the true secret s .

Original: Exposure is thus highest when the injected canary has the lowest loss in the full canary universe.
Revised: Exposure is thus highest when the injected canary has the lowest loss in the full canary universe.

Original: Following Tram ` er et al.
Revised: Following Tram ` er et al.

Original: (2022), we also calibrate the canary losses by subtracting the mean loss over multiple reference models (trained without canaries) before ranking.
Revised: (2022), we also calibrate the canary losses by subtracting the mean loss over multiple reference models (trained without canaries) before ranking.

Original: 2. R ELATED W ORK ON F ORGETTING
Revised: 2. R ELATED W ORK ON F ORGETTING

Original: In concurrent work, Tirumala et al.
Revised: In concurrent work, Tirumala et al.

Original: (2022) show that large language models forget examples seen early in training.
Revised: (2022) show that large language models forget examples seen early in training.

Original: However, their deﬁnition of forgetting only captures a speciﬁc form of privacy leakage: memorization which can be identiﬁed when the model reproduces training examples exactly.
Revised: However, their deﬁnition of forgetting only captures a speciﬁc form of privacy leakage: memorization which can be identiﬁed when the model reproduces training examples exactly.

Original: Instead, our work captures a more general form of privacy leakage because we consider (a) stronger privacy attacks, and (b) multiple strategies to measure worst-case forgetting of the training examples beyond exact regurgitation.
Revised: Instead, our work captures a more general form of privacy leakage because we consider (a) stronger privacy attacks, and (b) multiple strategies to measure worst-case forgetting of the training examples beyond exact regurgitation.

Original: To achieve this, our work leverages recent progress in auditing of privacy in ML (Carlini et al., 2021a; Tram`er et al., 2022).
Revised: To achieve this, our work leverages recent progress in auditing of privacy in ML (Carlini et al., 2021a; Tram`er et al., 2022).

Original: We also perform our experiments on models and datasets representing a wider variety of data domains.
Revised: We also perform our experiments on models and datasets representing a wider variety of data domains.

Original: Additionally, we analyze the theoretical limits of forgetting, and propose an explanation for why forgetting happens.
Revised: Additionally, we analyze the theoretical limits of forgetting, and propose an explanation for why forgetting happens.

Original: Another empirical exploration of forgetting is found in Graves et al.
Revised: Another empirical exploration of forgetting is found in Graves et al.

Original: Their work differs from ours in two ways.
Revised: Their work differs from ours in two ways.

Original: They explore forgetting in the context of machine unlearning (i.e., to determine whether forgetting is sufﬁcient to avoid the need to unlearn altogether).
Revised: They explore forgetting in the context of machine unlearning (i.e., to determine whether forgetting is sufﬁcient to avoid the need to unlearn altogether).

Original: Second, they ﬁnd that continual retraining is too slow to allow for forgetting, but their ﬁndings hold for small datasets only.
Revised: Second, they ﬁnd that continual retraining is too slow to allow for forgetting, but their ﬁndings hold for small datasets only.

Original: Instead, we identify forgetting as especially relevant for large model training.
Revised: Instead, we identify forgetting as especially relevant for large model training.

Original: We also measure precisely how long forgetting takes for such large datasets in a variety of domains, and ﬁnd that it happens quickly enough to be relevant to privacy.
Revised: We also measure precisely how long forgetting takes for such large datasets in a variety of domains, and ﬁnd that it happens quickly enough to be relevant to privacy.

Original: In another related work, Hyland & Tople (2019) used stochasticity to improve differentially private machine learning for models trained on small datasets.
Revised: In another related work, Hyland & Tople (2019) used stochasticity to improve differentially private machine learning for models trained on small datasets.

Original: In this paper, we identify stochasticity in training as a potential cause for forgetting.
Revised: In this paper, we identify stochasticity in training as a potential cause for forgetting.

Original:  F ORGETTING
Revised:  F ORGETTING

Original: Rather than study forgetting through the lens of the model’s accuracy on a speciﬁc example—as done in the catastrophic forgetting literature—we use an instance-speciﬁc notion of forgetting: we attempt to detect a speciﬁc example’s presence in training.
Revised: Rather than study forgetting through the lens of the model’s accuracy on a speciﬁc example—as done in the catastrophic forgetting literature—we use an instance-speciﬁc notion of forgetting: we attempt to detect a speciﬁc example’s presence in training.

Original: In catastrophic forgetting, the model’s accuracy degrades on an entire sub-distribution.
Revised: In catastrophic forgetting, the model’s accuracy degrades on an entire sub-distribution.

Original: This does not necessarily have implications for forgetting of memorized information contained in individual examples.
Revised: This does not necessarily have implications for forgetting of memorized information contained in individual examples.

Original: It is possible that, despite a decrease in accuracy, the model still carries detectable traces of individual examples, which is still harmful for privacy.
Revised: It is possible that, despite a decrease in accuracy, the model still carries detectable traces of individual examples, which is still harmful for privacy.

Original: It is also possible that a high accuracy model, which has not yet forgotten the sub-distribution, generalizes well to the sub-distribution rather than memorizing the speciﬁcs of the training set.
Revised: It is also possible that a high accuracy model, which has not yet forgotten the sub-distribution, generalizes well to the sub-distribution rather than memorizing the speciﬁcs of the training set.

Original: Thus, our deﬁnition of forgetting instead asks a more privacy-motivated question: whether it is possible to detect or extract an example in the training set.
Revised: Thus, our deﬁnition of forgetting instead asks a more privacy-motivated question: whether it is possible to detect or extract an example in the training set.

Original: Hence, we deﬁne forgetting by drawing from the literature on attacking ML models to ﬁnd privacy violations.
Revised: Hence, we deﬁne forgetting by drawing from the literature on attacking ML models to ﬁnd privacy violations.

Original: 3. D EFINING F ORGETTING
Revised: 3. D EFINING F ORGETTING

Original: We measure the rate of forgetting by evaluating the success rate of a privacy attack.
Revised: We measure the rate of forgetting by evaluating the success rate of a privacy attack.

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: A training algorithm T is said to ( A , α , k )- forget a training example z if, k steps after z is last used in T , a privacy attack A achieves no higher than success rate α .
Revised: A training algorithm T is said to ( A , α , k )- forget a training example z if, k steps after z is last used in T , a privacy attack A achieves no higher than success rate α .

Original: For example, consider the case where we let A be a membership inference attack, with success rate measured by the accuracy of the attack, and we set α = 50% as a random guessing baseline.
Revised: For example, consider the case where we let A be a membership inference attack, with success rate measured by the accuracy of the attack, and we set α = 50% as a random guessing baseline.

Original: Then an example is ( A , α, k ) -forgotten if k steps after training on that example, A cannot distinguish between the case where the model was, or was not, trained on that example.
Revised: Then an example is ( A , α, k ) -forgotten if k steps after training on that example, A cannot distinguish between the case where the model was, or was not, trained on that example.

Original: This captures the intuition that a model has “forgotten” a training point if it becomes difﬁcult to detect whether or not that point was used in training.
Revised: This captures the intuition that a model has “forgotten” a training point if it becomes difﬁcult to detect whether or not that point was used in training.

Original: Because we are interested in measuring privacy risk, our deﬁnition allows examples which were never memorized (i.e., A never performs well) to be forgotten immediately (i.e., k = 0 ).
Revised: Because we are interested in measuring privacy risk, our deﬁnition allows examples which were never memorized (i.e., A never performs well) to be forgotten immediately (i.e., k = 0 ).

Original: However, we will focus our analysis on vulnerable examples and state-of-the-art privacy attacks where examples are memorized after being used to train T .
Revised: However, we will focus our analysis on vulnerable examples and state-of-the-art privacy attacks where examples are memorized after being used to train T .

Original: Differentially private systems must satisfy a stronger requirement than what we’ve given: all attacks must perform poorly even k = 0 steps after an example is seen.
Revised: Differentially private systems must satisfy a stronger requirement than what we’ve given: all attacks must perform poorly even k = 0 steps after an example is seen.

Original: Additionally, machine unlearning is an even stronger requirement: all possible privacy attacks must fail (where forgetting allows some small success rate: α ).
Revised: Additionally, machine unlearning is an even stronger requirement: all possible privacy attacks must fail (where forgetting allows some small success rate: α ).

Original: Our notion of forgetting uses known attacks, capturing the common strategy of “empirically testing” privacy guarantees (Jayaraman & Evans, 2019; Jagielski et al., 2020; Thakkar et al., 2020;
Revised: Our notion of forgetting uses known attacks, capturing the common strategy of “empirically testing” privacy guarantees (Jayaraman & Evans, 2019; Jagielski et al., 2020; Thakkar et al., 2020;

Original: Nasr et al., 2021; Huang et al., 2022).
Revised: Nasr et al., 2021; Huang et al., 2022).

Original: Additionally, while machine unlearning is an intentional act, forgetting may apply without intervention after some number of steps k .
Revised: Additionally, while machine unlearning is an intentional act, forgetting may apply without intervention after some number of steps k .

Original: 3. M EASURING F ORGETTING
Revised: 3. M EASURING F ORGETTING

Original: A straightforward way to measure how much a model forgets is to run a membership inference attack on samples seen at various training steps.
Revised: A straightforward way to measure how much a model forgets is to run a membership inference attack on samples seen at various training steps.

Original: However, this only permits an average-case privacy analysis, measuring forgetting for typical examples.
Revised: However, this only permits an average-case privacy analysis, measuring forgetting for typical examples.

Original: Because we want to understand forgetting from a privacy perspective, we instead design a procedure to  measure an algorithm’s worst-case forgetting.
Revised: Because we want to understand forgetting from a privacy perspective, we instead design a procedure to attempt to measure an algorithm’s worst-case forgetting.

Original: We adapt privacy auditing strategies (Jagielski et al., 2020;
Revised: We adapt privacy auditing strategies (Jagielski et al., 2020;

Original: Nasr et al., 2021) which consist of two components: (1) constructing a small set of examples
Revised: Nasr et al., 2021) which consist of two components: (1) constructing a small set of examples

Original: D p (not contained in the standard training set
Revised: D p (not contained in the standard training set

Original: D ) which the model will be forced to memorize in order to classify correctly, and then (2) running a privacy test A to determine whether or not these examples were included during training.
Revised: D ) which the model will be forced to memorize in order to classify correctly, and then (2) running a privacy test A to determine whether or not these examples were included during training.

Original: Our procedure for measuring forgetting trains two models: one model is trained only on D , and one is trained on both D and D p .
Revised: Our procedure for measuring forgetting trains two models: one model is trained only on D , and one is trained on both D and D p .

Original: After training for some number of steps, each model continues training on D , never training on D p again.
Revised: After training for some number of steps, each model continues training on D , never training on D p again.

Original: After starting this second stage of training only on D , we continually measure how well the privacy attack A performs at predicting which model received the injected samples and which did not.
Revised: After starting this second stage of training only on D , we continually measure how well the privacy attack A performs at predicting which model received the injected samples and which did not.

Original: By design, this procedure measures ( A , α, k ) -forgetting of D p , providing an attack success rate α for any number of steps k after removing D p .
Revised: By design, this procedure measures ( A , α, k ) -forgetting of D p , providing an attack success rate α for any number of steps k after removing D p .

Original: We consider two ways to instantiate this procedure, which incorporate D p into training in two distinct ways.
Revised: We consider two ways to instantiate this procedure, which incorporate D p into training in two distinct ways.

Original: The ﬁrst, called P OISON , mimics a ﬁne-tuning setting.
Revised: The ﬁrst, called P OISON , mimics a ﬁne-tuning setting.

Original: The second, called I NJECT , is used to measuring forgetting in extremely large training sets.
Revised: The second, called I NJECT , is used to measuring forgetting in extremely large training sets.

Original: The strategies differ only in how they use D p , and we provide algorithms for both in Appendix A.
Revised: The strategies differ only in how they use D p , and we provide algorithms for both in Appendix A.

Original: P OISON .
Revised: P OISON .

Original: P OISON adds D p into training, and trains on the augmented dataset D ∪ D p .
Revised: P OISON adds D p into training, and trains on the augmented dataset D ∪ D p .

Original: After T I steps, the poisoned examples are removed, and training resumes on the unpoisoned training set D .
Revised: After T I steps, the poisoned examples are removed, and training resumes on the unpoisoned training set D .

Original: This is best suited for small training sets, where shufﬂing data does not heavily impact the position of training examples.
Revised: This is best suited for small training sets, where shufﬂing data does not heavily impact the position of training examples.

Original: This approach reﬂects ﬁne-tuning, where many passes are made over a pre-training dataset containing sensitive examples, but ﬁne-tuning is done on a disjoint dataset.
Revised: This approach reﬂects ﬁne-tuning, where many passes are made over a pre-training dataset containing sensitive examples, but ﬁne-tuning is done on a disjoint dataset.

Original: I NJECT .
Revised: I NJECT .

Original: I NJECT trains a model on the unpoisoned dataset D up until exactly step T I , where it updates with D p , and then resumes training on D thereafter.
Revised: I NJECT trains a model on the unpoisoned dataset D up until exactly step T I , where it updates with D p , and then resumes training on D thereafter.

Original: This approach is preferable when making a small number of passes over a large dataset (as is common in training large language models and speech models), where shufﬂing can heavily impact the position in training.
Revised: This approach is preferable when making a small number of passes over a large dataset (as is common in training large language models and speech models), where shufﬂing can heavily impact the position in training.

Original: I NJECT differs from
Revised: I NJECT differs from

Original: P OISON in that the target examples are not included immediately at the beginning of training.
Revised: P OISON in that the target examples are not included immediately at the beginning of training.

Original: To reﬂect the impact of duplicated training data, or to conservatively estimate forgetting, we can also inject D p multiple times during training.
Revised: To reﬂect the impact of duplicated training data, or to conservatively estimate forgetting, we can also inject D p multiple times during training.

Original:  E MPIRICALLY M EASURING F ORGETTING
Revised:  E MPIRICALLY M EASURING F ORGETTING

Original: In this section, we empirically measure whether machine learning models forget samples over the course of training for large neural network models (and if they do, how quickly do they do so).
Revised: In this section, we empirically measure whether machine learning models forget samples over the course of training for large neural network models (and if they do, how quickly do they do so).

Original: 4. E XPERIMENT S ETUP
Revised: 4. E XPERIMENT S ETUP

Original: We investigate forgetting across a variety of datasets and machine learning models.
Revised: We investigate forgetting across a variety of datasets and machine learning models.

Original: We use canary extraction to evaluate generative models (i.e., for LibriSpeech and LMs) and membership inference for classiﬁcation models (i.e., for ImageNet).
Revised: We use canary extraction to evaluate generative models (i.e., for LibriSpeech and LMs) and membership inference for classiﬁcation models (i.e., for ImageNet).

Original: We provide more detailed experiment setup in Appendix B.
Revised: We provide more detailed experiment setup in Appendix B.

Original: ImageNet.
Revised: ImageNet.

Original: We train ResNet-50 models for 90 epochs (each epoch is roughly 5,000 steps).
Revised: We train ResNet-50 models for 90 epochs (each epoch is roughly 5,000 steps).

Original: To test forgetting, we mainly use the I NJECT strategy, due to the large number of steps made in a single
Revised: To test forgetting, we mainly use the I NJECT strategy, due to the large number of steps made in a single

Original: ImageNet epoch.
Revised: ImageNet epoch.

Original: We also compare with P OISON in Appendix C.1.
Revised: We also compare with P OISON in Appendix C.1.

Original: We start from a ﬁxed checkpoint (with no injected data), and ﬁne-tune this checkpoint for both the injected and baseline models, and perform membership inference to distinguish the injected predictions from the baseline predictions.
Revised: We start from a ﬁxed checkpoint (with no injected data), and ﬁne-tune this checkpoint for both the injected and baseline models, and perform membership inference to distinguish the injected predictions from the baseline predictions.

Original: We calibrate the logit scores based on those produced by the ﬁxed checkpoint, representing a powerful adversary who knows even an intermediate update before the inclusion of sensitive data.
Revised: We calibrate the logit scores based on those produced by the ﬁxed checkpoint, representing a powerful adversary who knows even an intermediate update before the inclusion of sensitive data.

Original: We report results averaged over three trials.
Revised: We report results averaged over three trials.

Original: LibriSpeech.
Revised: LibriSpeech.

Original: We use state-of-the-art Conformer (L) architecture and the training method from
Revised: We use state-of-the-art Conformer (L) architecture and the training method from

Original: Gulati et al.
Revised: Gulati et al.

Original: (2020) to train models over LibriSpeech.
Revised: (2020) to train models over LibriSpeech.

Original: We use the P OISON strategy, as we train on
Revised: We use the P OISON strategy, as we train on

Original: LibriSpeech for many epochs.
Revised: LibriSpeech for many epochs.

Original: We generate 320 unique canaries and insert them with various repeat counts.
Revised: We generate 320 unique canaries and insert them with various repeat counts.

Original: We use loss calibration with 11 reference models.
Revised: We use loss calibration with 11 reference models.

Original: Neural Language Models.
Revised: Neural Language Models.

Original: We train decoder-only, 110M parameter, Transformer-based language models (with the T5 codebase and architecture from Raffel et al.
Revised: We train decoder-only, 110M parameter, Transformer-based language models (with the T5 codebase and architecture from Raffel et al.

Original: for one epoch over a version of C4 (Dodge et al.
Revised: for one epoch over a version of C4 (Dodge et al.

Original: (2021)) that had been deduplicated with MinHash (Lee et al.
Revised: (2021)) that had been deduplicated with MinHash (Lee et al.

Original: We use the
Revised: We use the

Original: I NJECT strategy due to the large training set size and add a single batch of 4096 canaries consisting of random tokens, with between 1 and 100 duplicates for each unique canary.
Revised: I NJECT strategy due to the large training set size and add a single batch of 4096 canaries consisting of random tokens, with between 1 and 100 duplicates for each unique canary.

Original: 4. F ORGETTING H APPENS I N P RACTICE
Revised: 4. F ORGETTING H APPENS I N P RACTICE

Original: In Figure 1, we ﬁnd that the datasets and models we consider exhibit forgetting.
Revised: In Figure 1, we ﬁnd that the datasets and models we consider exhibit forgetting.

Original: Starting with
Revised: Starting with

Original: Figure 1(a), we train a model on the ImageNet training dataset and then, following the I NJECT approach, train the model on a single minibatch containing the poisoned examples repeated 10 times at epoch 50.
Revised: Figure 1(a), we train a model on the ImageNet training dataset and then, following the I NJECT approach, train the model on a single minibatch containing the poisoned examples repeated 10 times at epoch 50.

Original: (The red vertical line here corresponds to the timestep where injection occurs.)
Revised: (The red vertical line here corresponds to the timestep where injection occurs.)

Original: We then plot the membership inference accuracy and precision as a function of time, both before injection and also for ten epochs after.
Revised: We then plot the membership inference accuracy and precision as a function of time, both before injection and also for ten epochs after.

Original: We observe that membership inference is impossible until the injection step, after which precision remains very high for many epochs, taking 10 epochs to decay to roughly 65%,
Revised: We observe that membership inference is impossible until the injection step, after which precision remains very high for many epochs, taking 10 epochs to decay to roughly 65%,

Original: [Figure]
Revised: [Figure]

Original: which is permitted by differential privacy with ε ≈ 0 .
Revised: which is permitted by differential privacy with ε ≈ 0 .

Original: This demonstrates that forgetting does occur for this setting.
Revised: This demonstrates that forgetting does occur for this setting.

Original: Figures 1(b-c) repeat this experiment for two other datasets: LibriSpeech and C4, where we use canary extraction for both.
Revised: Figures 1(b-c) repeat this experiment for two other datasets: LibriSpeech and C4, where we use canary extraction for both.

Original: In Figure 1(b) we plot results from LibriSpeech with the P OISON strategy, with canaries repeated 20 times.
Revised: In Figure 1(b) we plot results from LibriSpeech with the P OISON strategy, with canaries repeated 20 times.

Original: We ﬁnd that the canary exposure decays rapidly after canaries are removed at 50,000 steps, decaying from 14.0 to 5.6 in the ﬁrst 10,000 steps, which indicates the canaries become 330 × harder to guess on average.
Revised: We ﬁnd that the canary exposure decays rapidly after canaries are removed at 50,000 steps, decaying from 14.0 to 5.6 in the ﬁrst 10,000 steps, which indicates the canaries become 330 × harder to guess on average.

Original: After another 40,000 steps, the exposure drops to roughly 2, only slightly higher than the baseline exposure for canaries not seen during training.
Revised: After another 40,000 steps, the exposure drops to roughly 2, only slightly higher than the baseline exposure for canaries not seen during training.

Original: In
Revised: In

Original: Figure 1(c)
Revised: Figure 1(c)

Original: , we show that, for a variety of repeat counts, LMs forget canaries.
Revised: , we show that, for a variety of repeat counts, LMs forget canaries.

Original: They reach an exposure of 2, and decay over the course of at most 10,000 steps to the baseline “random guessing” level of 1.
Revised: They reach an exposure of 2, and decay over the course of at most 10,000 steps to the baseline “random guessing” level of 1.

Original: In Appendix C.1, we show that, on ImageNet, P OISON and I NJECT give similar results.
Revised: In Appendix C.1, we show that, on ImageNet, P OISON and I NJECT give similar results.

Original: We note that in Figure 1, we selected representative parameter settings, but we ﬁnd that the same forgetting behavior qualitatively holds over all settings we tried.
Revised: We note that in Figure 1, we selected representative parameter settings, but we ﬁnd that the same forgetting behavior qualitatively holds over all settings we tried.

Original: However, these parameters can impact, quantitatively, how quickly forgetting happens.
Revised: However, these parameters can impact, quantitatively, how quickly forgetting happens.

Original: 4. I NVESTIGATING F ORGETTING
Revised: 4. I NVESTIGATING F ORGETTING

Original: We investigate the impact of various parameters on the speed of forgetting.
Revised: We investigate the impact of various parameters on the speed of forgetting.

Original: We investigate 1) how many times a point is repeated in training/how hard the point is to learn, and 2) how late the model is seen in training, 3) model size, and 4) optimizer parameters.
Revised: We investigate 1) how many times a point is repeated in training/how hard the point is to learn, and 2) how late the model is seen in training, 3) model size, and 4) optimizer parameters.

Original: While repetitions and hardness are the parameters that we ﬁnd to be most important, each of the other parameters also has some impact.
Revised: While repetitions and hardness are the parameters that we ﬁnd to be most important, each of the other parameters also has some impact.

Original: We analyze these parameters in depth on ImageNet, and a subset of them on LibriSpeech and C4.
Revised: We analyze these parameters in depth on ImageNet, and a subset of them on LibriSpeech and C4.

Original: Repetitions/Hardness.
Revised: Repetitions/Hardness.

Original: Privacy attacks are not equally effective on all examples.
Revised: Privacy attacks are not equally effective on all examples.

Original: Large language models memorize examples which are duplicated many times in their training sets (Lee et al., 2021;
Revised: Large language models memorize examples which are duplicated many times in their training sets (Lee et al., 2021;

Original: Kandpal et al., 2022; Carlini et al., 2022).
Revised: Kandpal et al., 2022; Carlini et al., 2022).

Original: Among examples which are not duplicated, there exist “outliers”, which are more vulnerable to attacks (Kulynych et al., 2019; Carlini et al., 2021a).
Revised: Among examples which are not duplicated, there exist “outliers”, which are more vulnerable to attacks (Kulynych et al., 2019; Carlini et al., 2021a).

Original: Here, we show that these ﬁndings also hold in forgetting: outlier data is forgotten more slowly than inlier data, and data duplicated many times is forgotten more slowly than data duplicated few times.
Revised: Here, we show that these ﬁndings also hold in forgetting: outlier data is forgotten more slowly than inlier data, and data duplicated many times is forgotten more slowly than data duplicated few times.

Original: We use our forgetting deﬁnition to measure this speed, as it reﬂects the privacy risk to examples (although, especially for exposure, this deﬁnition does depend on the initial success rate of the attack).
Revised: We use our forgetting deﬁnition to measure this speed, as it reﬂects the privacy risk to examples (although, especially for exposure, this deﬁnition does depend on the initial success rate of the attack).

Original: We present our results in Figures 2 and 1c.
Revised: We present our results in Figures 2 and 1c.

Original: Our ﬁndings corroborate those made in prior work.
Revised: Our ﬁndings corroborate those made in prior work.

Original: Repetitions have a very clear impact on forgetting: examples repeated more times both start out with a much higher attack success, and the models take much longer to forget them, as well.
Revised: Repetitions have a very clear impact on forgetting: examples repeated more times both start out with a much higher attack success, and the models take much longer to forget them, as well.

Original: On
Revised: On

Original: ImageNet, attack precision takes roughly 20,000 training steps to drop below 70% with 10 repetitions, which is higher than 1 repetition 200 steps after injection.
Revised: ImageNet, attack precision takes roughly 20,000 training steps to drop below 70% with 10 repetitions, which is higher than 1 repetition 200 steps after injection.

Original: On LibriSpeech and C4, canary exposure has a similar trend, where heavily repeated canaries have higher exposures, and decrease to a given exposure more slowly.
Revised: On LibriSpeech and C4, canary exposure has a similar trend, where heavily repeated canaries have higher exposures, and decrease to a given exposure more slowly.

Original: We also validate on ImageNet that hard examples (test points with high loss on a fully-trained model) are forgotten more slowly than easy examples, in Figure 2c.
Revised: We also validate on ImageNet that hard examples (test points with high loss on a fully-trained model) are forgotten more slowly than easy examples, in Figure 2c.

Original: These results reinforce the necessity of considering worst-case samples when measuring privacy.
Revised: These results reinforce the necessity of considering worst-case samples when measuring privacy.

Original: [Figure]
Revised: [Figure]

Original: Recency.
Revised: Recency.

Original: Forgetting inherently considers the recency with which samples have been seen, but we also investigate in Appendix C.2 how the rate of forgetting changes with recency.
Revised: Forgetting inherently considers the recency with which samples have been seen, but we also investigate in Appendix C.2 how the rate of forgetting changes with recency.

Original: We ﬁnd no conclusive effect with the I NJECT strategy
Revised: We ﬁnd no conclusive effect with the I NJECT strategy

Original: [Figure]
Revised: [Figure]

Original: but with P OISON , later removal leads to more training on poisoned data
Revised: but with P OISON , later removal leads to more training on poisoned data

Original: [Figure]
Revised: [Figure]

Original: so forgetting is slower.
Revised: so forgetting is slower.

Original: Ultimately, these results indicate that current attacks struggle to detect an example’s inﬂuence on the early optimization trajectory
Revised: Ultimately, these results indicate that current attacks struggle to detect an example’s inﬂuence on the early optimization trajectory

Original: [Figure]
Revised: [Figure]

Original: near epoch 60.
Revised: near epoch 60.

Original: Examples injected before the decay are forgotten more slowly, as they contribute 10 × as much as examples injected after the decay
Revised: Examples injected before the decay are forgotten more slowly, as they contribute 10 × as much as examples injected after the decay

Original: Learning Rate Decay and Other Parameters.
Revised: Learning Rate Decay and Other Parameters.

Original: On ImageNet, we experiment with learning rate, momentum, and model size, and ﬁnd these mostly have little effect.
Revised: On ImageNet, we experiment with learning rate, momentum, and model size, and ﬁnd these mostly have little effect.

Original: The limited effect of model size on forgetting is somewhat surprising—and possibly due to our smallest tested model being large enough for signiﬁcant memorization to occur.
Revised: The limited effect of model size on forgetting is somewhat surprising—and possibly due to our smallest tested model being large enough for signiﬁcant memorization to occur.

Original: Due to their small effect, we discuss these parameters in
Revised: Due to their small effect, we discuss these parameters in

Original: Appendix C, and focus here on learning rate decay.
Revised: Appendix C, and focus here on learning rate decay.

Original: Our training decays learning rate by a factor of 10 at Epoch 60, and we plot in Figure 3 the precision of membership inference for examples injected near this decay.
Revised: Our training decays learning rate by a factor of 10 at Epoch 60, and we plot in Figure 3 the precision of membership inference for examples injected near this decay.

Original: We ﬁnd that examples injected before the decay are forgotten signiﬁcantly slower than examples injected after.
Revised: We ﬁnd that examples injected before the decay are forgotten signiﬁcantly slower than examples injected after.

Original: An update before the decay will have 10 × the learning rate of subsequent updates, so these will have less ability to inﬂuence forgetting.
Revised: An update before the decay will have 10 × the learning rate of subsequent updates, so these will have less ability to inﬂuence forgetting.

Original:  U NDERSTANDING F ORGETTING
Revised:  U NDERSTANDING F ORGETTING

Original: As we have shown, forgetting empirically occurs in deep learning.
Revised: As we have shown, forgetting empirically occurs in deep learning.

Original: In this section, we investigate why.
Revised: In this section, we investigate why.

Original: We start by showing a setting where k -means, due to its non-convex loss function, does not forget training data, contrasting with our empirical results on large non-convex models, potentially because attacks struggle to take advantage of the optimization trajectory.
Revised: We start by showing a setting where k -means, due to its non-convex loss function, does not forget training data, contrasting with our empirical results on large non-convex models, potentially because attacks struggle to take advantage of the optimization trajectory.

Original: To understand this inconsistency, we offer a possible explanation: randomness in training can lead to forgetting.
Revised: To understand this inconsistency, we offer a possible explanation: randomness in training can lead to forgetting.

Original: To support this explanation, we ﬁrst show that, without nondeterminism, forgetting does not happen in deep learning.
Revised: To support this explanation, we ﬁrst show that, without nondeterminism, forgetting does not happen in deep learning.

Original: We show this empirically as well as formally for SGD on a task of mean estimation.
Revised: We show this empirically as well as formally for SGD on a task of mean estimation.

Original: We then prove that if the training data is randomly shufﬂed, forgetting does happen for SGD on this mean estimation task—thus corroborating our empirical results.
Revised: We then prove that if the training data is randomly shufﬂed, forgetting does happen for SGD on this mean estimation task—thus corroborating our empirical results.

Original: 5. N ON - CONVEXITY C AN P REVENT F ORGETTING
Revised: 5. N ON - CONVEXITY C AN P REVENT F ORGETTING

Original: We show here that there exists a data distribution where k -means clustering will not forget certain examples, due to its non-convex objective.
Revised: We show here that there exists a data distribution where k -means clustering will not forget certain examples, due to its non-convex objective.

Original: The k -means problem can have many local optima, and we will construct a setting where a single example can force the model into one of these optima, where the model will remain even after more examples are added.
Revised: The k -means problem can have many local optima, and we will construct a setting where a single example can force the model into one of these optima, where the model will remain even after more examples are added.

Original: The setting we consider is the one of a one-dimensional clustering task with three clusters in the data, c 1 , c 2 , and c 3 , but where
Revised: The setting we consider is the one of a one-dimensional clustering task with three clusters in the data, c 1 , c 2 , and c 3 , but where

Original: [Figure]
Revised: [Figure]

Original: k -means is conﬁgured to use k = 2 clusters.
Revised: k -means is conﬁgured to use k = 2 clusters.

Original: As a result, k -means needs to choose to “merge” one cluster, c 2 , with the others.
Revised: As a result, k -means needs to choose to “merge” one cluster, c 2 , with the others.

Original: We can construct an outlier from the c 2 cluster which will manipulate this decision, and adding more data does not lead to this choice being forgotten.
Revised: We can construct an outlier from the c 2 cluster which will manipulate this decision, and adding more data does not lead to this choice being forgotten.

Original: The setting is illustrated in Figure 4 (with a two-dimensional clustering task for clarity of exposition).
Revised: The setting is illustrated in Figure 4 (with a two-dimensional clustering task for clarity of exposition).

Original: We provide a more concrete description of the dataset we consider in Appendix D.1, but we verify empirically that this setting results in near-perfect membership inference on the outlier point, with 97% accuracy and perfect precision.
Revised: We provide a more concrete description of the dataset we consider in Appendix D.1, but we verify empirically that this setting results in near-perfect membership inference on the outlier point, with 97% accuracy and perfect precision.

Original: This is the ﬁrst example of privacy leakage we are aware of that relies on non-convexity.
Revised: This is the ﬁrst example of privacy leakage we are aware of that relies on non-convexity.

Original: There is a large body of differentially private algorithms which are speciﬁc to convex models, and do not apply to non-convex settings (Chaudhuri et al., 2011; Talwar et al., 2015;
Revised: There is a large body of differentially private algorithms which are speciﬁc to convex models, and do not apply to non-convex settings (Chaudhuri et al., 2011; Talwar et al., 2015;

Original: Wu et al., 2017; Iyengar et al., 2019).
Revised: Wu et al., 2017; Iyengar et al., 2019).

Original: Our analysis here is the ﬁrst to concretely justify this separation.
Revised: Our analysis here is the ﬁrst to concretely justify this separation.

Revised: 5. D ETERMINISTIC SGD D OES N OT F ORGET

Original: [Figure]
Revised: [Figure]

Original:  FMNIST, when the adversary  knows the exact batch order and training set,forgetting doesn’t happen (using I NJECT ), with membership inference accuracy never decaying below 97%, even 95 epochs after the injection at epoch 5.
Revised: a neural network on FMNIST, an adversary who knows the exact batch order and training set, doesn’t experience forgetting (using I NJECT with 1 repeat), with membership inference accuracy always perfect, even afterepochs of updates. Without this perfect knowledge, forgetting happens steadily.

Original: Having observed a (constructed) setting in which forgetting does not happen, we now turn our attention to why forgetting happens when training a neural network.
Revised: Having observed a (constructed) setting in which forgetting does not happen, we now turn our attention to why forgetting happens when training a neural network.

Original: Earlier, we hypothesized that nondeterminism leads to the forgetting we observed empirically.
Revised: Earlier, we hypothesized that nondeterminism leads to the forgetting we observed empirically.

Original: Here, we support this by showing that some  nondeterminism is required for forgetting, before continuing in Section 5.3 to show a source of nondeterminism that can cause forgetting.
Revised: Here, we support this by showing that some unknown nondeterminism is required for forgetting, before continuing in Section 5.3 to show a source of nondeterminism that can cause forgetting.

Original: The key idea of this section is that an adversary with knowledge of the entire training set and batch order can simulate the training on D and on D ∪ D p , stepping through training in the exact same way as the learner, and receiving the exact same model at each step.
Revised: The key idea of this section is that an adversary with knowledge of the entire training set and batch order can simulate the training on D and on D ∪ D p , stepping through training in the exact same way as the learner, and receiving the exact same model at each step.

Original: Because of the update on D p , these models are never exactly equal, and, because training is deterministic, it is always possible to distinguish them.
Revised: Because of the update on D p , these models are never exactly equal, and, because training is deterministic, it is always possible to distinguish them.

Original: In Figure 5, we see that this attack is possible in practice, on a logistic regression model trained on the FMNIST dataset with a ﬁxed batch order, using the I NJECT strategy.
Revised: In Figure 5, we see that this attack is possible in practice, on a logistic regression model trained on the FMNIST dataset with a ﬁxed batch order, using the I NJECT strategy.

Original: By simulating training, the adversary achieves 97% membership inference accuracy for examples seen in Epoch 5, even afterepochs of training (we expect the 3% drop to be a result of unavoidable GPU nondeterminism).
Revised: By simulating training, the adversary achieves 97% membership inference accuracy for examples seen in Epoch 5, even after 95 epochs of training (we expect the 3% drop to be a result of unavoidable GPU nondeterminism).

Original: In

Original:  Appendix D.2, we prove that this attack results in perfect membership inference on a deterministic version of SGD for mean estimation.
Revised: In Appendix D.2, we prove that this attack results in perfect membership inference on a deterministic version of SGD for mean estimation.

Revised: In this mean estimation setting, adding Gaussian noise to the updates would provably ensure forgetting for D p using the results of Feldman et al.

Revised: so it is the nondeterminism itself which prevents forgetting.

Original: 5. R ANDOM S AMPLING L EADS T O F ORGETTING
Revised: 5. R ANDOM S AMPLING L EADS T O F ORGETTING

Original: Having shown that deterministic training can prevent forgetting, we now show that some amount of  nondeterminism can cause forgetting.
Revised: Having shown that forgetting doesn’t happen when all nondeterminism is known to the adversary, we now show that some amount of unknown nondeterminism can cause forgetting.

Original: While Feldman et al.
Revised: While Feldman et al.

Original: (2018) show that differentially private noise is a sufﬁcient level of nondeterminism to provably cause forgetting, we consider nondeterminism coming from randomized data sampling.
Revised: (2018) show that differentially private noise is a sufﬁcient level of nondeterminism to provably cause forgetting, we consider nondeterminism coming from randomized data sampling.

Original: We consider mean estimation, where our examples are drawn from a d -dimensional Gaussian distribution with mean µ and covariance Σ , i.e., D = N ( µ, Σ) .
Revised: We consider mean estimation, where our examples are drawn from a d -dimensional Gaussian distribution with mean µ and covariance Σ , i.e., D = N ( µ, Σ) .

Original: We are asked to produce a guess for the mean ˜ µ .
Revised: We are asked to produce a guess for the mean ˜ µ .

Original: We estimate the mean with gradient descent with learning rate η , and we ask: does forgetting occur in mean estimation when the data is randomly sampled from this distribution?
Revised: We estimate the mean with gradient descent with learning rate η , and we ask: does forgetting occur in mean estimation when the data is randomly sampled from this distribution?

Original: We show that it does, by considering a simple distinguishing test.
Revised: We show that it does, by considering a simple distinguishing test.

Original: Consider two training runs, both starting SGD from the same ﬁxed initial parameters θ 0 .
Revised: Consider two training runs, both starting SGD from the same ﬁxed initial parameters θ 0 .

Original: At the very ﬁrst timestep, we inject two different arbitrary examples, v and v (cid:48) , into the two models.
Revised: At the very ﬁrst timestep, we inject two different arbitrary examples, v and v (cid:48) , into the two models.

Original: That is, we compute θ +0 by taking a gradient step on v , and θ − 0 by taking a gradient step on v (cid:48) .
Revised: That is, we compute θ +0 by taking a gradient step on v , and θ − 0 by taking a gradient step on v (cid:48) .

Original: We then continue SGD on randomly sampled examples from D .
Revised: We then continue SGD on randomly sampled examples from D .

Original: We write the k th step of each run as θ + k (for those continuing from θ +0 ) and θ − k (respectively).
Revised: We write the k th step of each run as θ + k (for those continuing from θ +0 ) and θ − k (respectively).

Original: In Theorem 1, we bound the R´enyi divergence between the distributions of θ + k and θ − k .
Revised: In Theorem 1, we bound the R´enyi divergence between the distributions of θ + k and θ − k .

Original: R´enyi divergence is the divergence used in R´enyi differential privacy (Mironov, 2017), and is known to bound the success of membership inference attacks and reconstruction attacks (Stock et al., 2022), so a smaller R´enyi divergence implies that these attacks will perform worse, and the model will forget which of v and v (cid:48) was used in training.
Revised: R´enyi divergence is the divergence used in R´enyi differential privacy (Mironov, 2017), and is known to bound the success of membership inference attacks and reconstruction attacks (Stock et al., 2022), so a smaller R ´ enyi divergence implies that these attacks will perform worse, and the model will forget which of v and v (cid:48) was used in training.

Original: To simplify the analysis, we use v (cid:48)
Revised: To simplify the analysis, we use v (cid:48)

Original: = − v , although a similar result can be shown for arbitrary pairs of v and v (cid:48) .
Revised: = − v , although a similar result can be shown for arbitrary pairs of v and v (cid:48) .

Original: Theorem 1.
Revised: Theorem 1.

Original: The distributions of the models θ + k and θ − k produced with learning rate 0 < η < 1 /have
Revised: The distributions of the models θ + k and θ − k produced with learning rate 0 < η < 1 /have

Original: R ´ enyi divergence of order
Revised: R ´ enyi divergence of order

Original: α at most 2 αk v (cid:62) Σ − 1 v .
Revised: α at most 2 αk v (cid:62) Σ − 1 v .

Original: Theorem 1 states that distinguishing between the models θ + k
Revised: Theorem 1 states that distinguishing between the models θ + k

Original: and θ − k becomes harder as training progresses, as well as when v (cid:62)
Revised: and θ − k becomes harder as training progresses, as well as when v (cid:62)

Original: Σ − 1 v is small (when v is more “in-distribution”).
Revised: Σ − 1 v is small (when v is more “in-distribution”).

Original: These agree with observations made in our experiments: forgetting improves as k increases (although with diminishing returns), and more worst-case (or heavily repeated) examples are forgotten more slowly (here, if v (cid:62) Σ − 1 v = 0 , forgetting never happens!).
Revised: These agree with observations made in our experiments: forgetting improves as k increases (although with diminishing returns), and more worst-case (or heavily repeated) examples are forgotten more slowly (here, if v (cid:62) Σ − 1 v = 0 , forgetting never happens!).

Original: We note that this theorem is not operational, as it is impossible to know whether the Gaussian assumption holds in practice (Steinke & Ullman), but our analysis provides intuition for our experiments, suggesting further that this source of nondeterminism may lead to the empirical forgetting we observe.
Revised: We note that this theorem is not operational, as it is impossible to know whether the Gaussian assumption holds in practice (Steinke & Ullman), but our analysis provides intuition for our experiments, suggesting further that this source of nondeterminism may lead to the empirical forgetting we observe.

Revised: Combined with Theorem 2 in Appendix D.2, which also considers mean estimation, Theorem 1 shows that it is the unknown nondeterminism coming from the data which causes forgetting.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In forgetting, we demonstrate a passive form of privacy ampliﬁcation: examples used early in model training may be more robust to privacy attacks.
Revised: In forgetting, we demonstrate a passive form of privacy ampliﬁcation: examples used early in model training may be more robust to privacy attacks.

Original: Our ﬁndings align with the theoretical ﬁndings of Feldman et al.
Revised: Our ﬁndings align with the theoretical ﬁndings of Feldman et al.

Original: (2018) (for convex optimization); improved bounds on privacy leakage for early examples also translate to better empirical privacy.
Revised: (2018) (for convex optimization); improved bounds on privacy leakage for early examples also translate to better empirical privacy.

Original: Our analytical and empirical ﬁndings highlight some previously poorly understood factors underlying forgetting.
Revised: Our analytical and empirical ﬁndings highlight some previously poorly understood factors underlying forgetting.

Original: We ﬁnd the size of the dataset is key: forgetting is more likely to happen when (a) the learning algorithm randomly samples training examples at each step (e.g. as in SGD) and (b) these examples are sampled from a large training set (e.g., when training modern language models).
Revised: We ﬁnd the size of the dataset is key: forgetting is more likely to happen when (a) the learning algorithm randomly samples training examples at each step (e.g. as in SGD) and (b) these examples are sampled from a large training set (e.g., when training modern language models).

Revised: Training sets should be large enough for an example to not be seen for thousands of steps before releasing the model.

Original: These ﬁndings will be useful to practitioners auditing privacy provided by their training algorithm and complement formal analysis of their guarantees (e.g., using differential privacy).
Revised: These ﬁndings will be useful to practitioners auditing privacy provided by their training algorithm and complement formal analysis of their guarantees (e.g., using differential privacy).

Original: Our work suggests the following directions for future investigation.
Revised: Our work suggests the following directions for future investigation.

Original: Protecting the ﬁnal examples.
Revised: Protecting the ﬁnal examples.

Original: Our work suggests that, for large scale training algorithms, the most recently seen examples are most vulnerable to privacy attacks.
Revised: Our work suggests that, for large scale training algorithms, the most recently seen examples are most vulnerable to privacy attacks.

Original: Defenses may take advantage of this to improve empirical privacy by focusing their privacy efforts on the most recently seen examples.
Revised: Defenses may take advantage of this to improve empirical privacy by focusing their privacy efforts on the most recently seen examples.

Original: The number of examples to protect could be determined with an empirical forgetting analysis.
Revised: The number of examples to protect could be determined with an empirical forgetting analysis.

Original: Non-convexity-aware membership inference.
Revised: Non-convexity-aware membership inference.

Original: In route to understanding forgetting, we demonstrated the ﬁrst setting where the optimum that a model ends up at leaks information about training examples.
Revised: In route to understanding forgetting, we demonstrated the ﬁrst setting where the optimum that a model ends up at leaks information about training examples.

Original: While this attack only applies to a speciﬁc k -means setting, it is interesting whether non-convexity can be exploited to design new membership inference attacks on realistic tasks.
Revised: While this attack only applies to a speciﬁc k -means setting, it is interesting whether non-convexity can be exploited to design new membership inference attacks on realistic tasks.

Original: Limitations.
Revised: Limitations.

Original: While our empirical investigation uses state-of-the-art attacks, it remains inherently heuristic, and cannot prove that forgetting happened.
Revised: While our empirical investigation uses state-of-the-art attacks, it remains inherently heuristic, and cannot prove that forgetting happened.

Original: However, we have no reason to expect that the trends we identify do not hold against even stronger attacks.
Revised: However, we have no reason to expect that the trends we identify do not hold against even stronger attacks.

Original: Indeed, in some cases our attacks exploit more knowledge than a realistic adversary can access in practice.
Revised: Indeed, in some cases our attacks exploit more knowledge than a realistic adversary can access in practice.

Original: Further developments in the techniques we rely on here might be taken advantage of to more faithfully measure forgetting.
Revised: Further developments in the techniques we rely on here might be taken advantage of to more faithfully measure forgetting.

Original: A A LGORITHMS FOR M EASURING F ORGETTING
Revised: A A LGORITHMS FOR M EASURING F ORGETTING

Original: We present the algorithm for P OISONin Algorithm 1 and the algorithm for I NJECT in Algorithm 2.
Revised: We present the algorithm for P OISON

Original: Algorithm
Revised: Algorithm

Original: Monitoring Data: Training Algorithm T RAIN , Clean Dataset D , Poison Samples D p , Removal Step T I ,
Revised: Data: Training Algorithm T RAIN , Clean Dataset D , Poison Samples D p , Removal Step T I ,

Original: Initial Model Parameters θ 0 , Total Training Steps T , Privacy Attack A
Revised: Initial Model Parameters θ 0 , Total Training Steps T , Privacy Attack A

Original: Function M EASURE F ORGET P OISON () : 
Revised: Function M EASURE F ORGET P OISON () : θ 0 T I

Original: [Equation]
Revised: = T RAIN ( θ 0 , D, T I ) θ 1 T I =

Original: [Equation] T I )
Revised: P OISON T RAIN ( θ 0 , D, D p , T I )

Original: For i =
Revised: For i =

Original: ( T I + 1) ..
Revised: ( T I + 1) ..

Original: T θ 0 i =
Revised: T θ 0 i =

Original: T RAIN ( θ 

Revised: T RAIN ( θ

Original: i − 1 , D, 1) θ 1 i = T RAIN ( θ 1 i − 1 , D, 1)
Revised: i − 1 , D, 1) θ 1 i = T RAIN ( θ 1 i − 1 , D, 1)

Original: A CC = A ( θ 

Revised: A CC = A ( θ

Original: i , θ 1 i , D p ) (cid:46) Continuously monitor privacy attack success on D p
Revised: i , θ 1 i , D p ) (cid:46) Continuously monitor privacy attack success on D p

Original: Function P OISON T RAIN ( θ, D, D p , T I ) :
Revised: Function P OISON T RAIN ( θ, D, D p , T I ) :

Original: Return T RAIN ( θ, D ∪ D p , T I ) (cid:46) Train with poisoned data over multiple epochs 
Revised: Return T RAIN ( θ, D ∪ D p , T I ) (cid:46) Train with poisoned data over multiple epochs B D ETAILED E XPERIMENT S ETUP

Original: Algorithm
Revised: ImageNet. We train ResNet-50 models for 90 epochs, with a learning rate of 0.1, momentum of 0.9, and batch size of 256 (making each epoch roughly 5,000 steps). To test privacy, we focus mainly on the I NJECT strategy, due to the large number of steps made in a single ImageNet epoch.

Revised: We also brieﬂy compare with P OISON .

Revised: In our experiments, we start from a ﬁxed checkpoint (with no injected data), and ﬁne-tune from this checkpoint for both the injected and baseline models, and perform membership inference attacks to distinguish the injected predictions from the baseline predictions.

Revised: We calibrate the logit scores based on those produced by the ﬁxed checkpoint, representing a powerful adversary who knows even an intermediate update before the inclusion of sensitive data.

Revised: We also experiment with calibrated loss scores, and with not performing calibration, and ﬁnd that these variants of the attack perform worse.

Revised: We inject a single batch of data, and measure the precision at a false positive rate of 10%, averaging over three trials.

Revised: We note that, because both the injected and

Original: Data: Training Algorithm T RAIN , Clean Dataset D , Poison Samples D p , Injection Step T I ,
Revised: Data: Training Algorithm T RAIN , Clean Dataset D , Poison Samples D p , Injection Step T I ,

Original: Injection Count k , Initial Model Parameters θ 0 , Total Training Steps T , Privacy Attack A
Revised: Injection Count k , Initial Model Parameters θ 0 , Total Training Steps T , Privacy Attack A

Original: Function M EASURE F ORGET I NJECT () : θ 0 T I
Revised: Function M EASURE F ORGET I NJECT () : θ 0 T I

Original: =
Revised: =

Original: T RAIN ( θ 0 , D, T I )
Revised: T RAIN ( θ 0 , D, T I )

Original: θ 1 T I =
Revised: θ 1 T I =

Original: I NJECT T RAIN ( θ 0 , D, D p , T I , k )
Revised: I NJECT T RAIN ( θ 0 , D, D p , T I , k )

Original: For i =
Revised: For i =

Original: ( T I + 1) ..
Revised: ( T I + 1) ..

Original: T θ 0 i =
Revised: T θ 0 i =

Original: T RAIN ( θ 

Revised: T RAIN ( θ

Original: i − 1 , D, 1) θ 1 i = T RAIN ( θ 1 i − 1 , D, 1)
Revised: i − 1 , D, 1) θ 1 i = T RAIN ( θ 1 i − 1 , D, 1)

Original: A CC = A ( θ 

Revised: A CC = A ( θ

Original: i , θ 1 i , X I , Y I ) (cid:46) Continuously monitor privacy attack success
Revised: i , θ 1 i , X I , Y I ) (cid:46) Continuously monitor privacy attack success

Original: Function I NJECT T RAIN ( θ, D, D p , T I , k ) :
Revised: Function I NJECT T RAIN ( θ, D, D p , T I , k ) :

Original: θ T
Revised: θ T

Original: I
Revised: I

Original: =
Revised: =

Original: T RAIN ( θ, D, T I )
Revised: T RAIN ( θ, D, T I )

Original: Return T RAIN ( θ T I , D p , k ) (cid:46) Inject after training on clean data B D ETAILED E XPERIMENT S ETUP ImageNet. We train ResNet-50 models for 90 epochs, with a learning rate of 0.1, momentum of 0.9, and batchsize of 256 (making each epoch roughly 5,000 steps).
Revised: Return T RAIN ( θ T I , D p , k ) (cid:46) Inject after training on clean data baseline models are initialized from the same checkpoint, the ﬁrst batch of the injected model will be the injected data, at which point membership inference will reach 100% accuracy/precision for any parameter setting.

Original: To test privacy, we focus mainly on the

Original: I NJECT strategy, due to the large number of steps made in a single ImageNet epoch.

Original: We also brieﬂy compare with P OISON .

Original: In our experiments, we start from a ﬁxed checkpoint (with no injected data), and ﬁne-tune from this checkpoint for both the injected and baseline models, and perform membership inference attacks to distinguish the injected predictions from the baseline predictions.

Original: We calibrate the logit scores based on those produced by the ﬁxed checkpoint, representing a powerful adversary who knows even an intermediate update before the inclusion of sensitive data.

Original: We also experiment with calibrated loss scores, and with not performing calibration, and ﬁnd that these variants of the attack perform worse.

Original: We inject a single batch of data, and measure the precision at a false positive rate of 10%, averaging over three trials.

Original: We note that, because both the injected and baseline models are initialized from the same checkpoint, the ﬁrst batch of the injected model will be the injected data, at which point membership inference will reach 100% accuracy/precision for any parameter setting.

Original: LibriSpeech.
Revised: LibriSpeech.

Original: We use the state-of-the-art Conformer (L) architecture and the training method from
Revised: We use the state-of-the-art Conformer (L) architecture and the training method from

Original: Gulati et al.
Revised: Gulati et al.

Original: (2020) to train models over LibriSpeech.
Revised: (2020) to train models over LibriSpeech.

Original: We train each model with a batch size of 2, utterances for 100,000 steps using the Adam optimizer (Kingma & Ba, 2015) and a transformer learning rate schedule (Vaswani et al., 2017).We use the P OISON strategy, as we make many passes over LibriSpeech.
Revised: We train each model with a batch size of 2, utterances for 100,000 steps using the Adam optimizer (Kingma & Ba, 2015) and a transformer learning rate schedule (Vaswani et al., 2017).We use the P OISON strategy, as we make many passes over LibriSpeech.

Original: To generate canary utterances, we start with a vocabulary consisting of the top 10,000 words in the LibriSpeech test set.
Revised: To generate canary utterances, we start with a vocabulary consisting of the top 10,000 words in the LibriSpeech test set.

Original: For the canary transcripts, we sample each word randomly from the vocabulary, varying the length of the transcript ( { 7, 10 } words), and the number of insertions in the dataset ( { 1, 4, 10, 20 } ).
Revised: For the canary transcripts, we sample each word randomly from the vocabulary, varying the length of the transcript ( { 7, 10 } words), and the number of insertions in the dataset ( { 1, 4, 10, 20 } ).

Original: For the canary audios, we use Text-To-Speech generation, and vary the gender of the speaker ( { Male, Female } ), and the number of speakers per transcript ( { 1, 2 } of the same gender).
Revised: For the canary audios, we use Text-To-Speech generation, and vary the gender of the speaker ( { Male, Female } ), and the number of speakers per transcript ( { 1, 2 } of the same gender).

Original: For each canary conﬁguration, we generate 10 unique canaries.
Revised: For each canary conﬁguration, we generate 10 unique canaries.

Original: As a result, we generate 320 unique canaries that amount to a total of 2,840 utterances ( < 1% of total utterances in
Revised: As a result, we generate 320 unique canaries that amount to a total of 2,840 utterances ( < 1% of total utterances in

Original: LibriSpeech).
Revised: LibriSpeech).

Original: We also use loss calibration, for which we train 11 reference models using a random 80% partition of LibriSpeech for training each model.
Revised: We also use loss calibration, for which we train 11 reference models using a random 80% partition of LibriSpeech for training each model.

Original: Large Language Models.
Revised: Large Language Models.

Original: We train decoder-only, 110M parameter, language models (with the T codebase and architecture from Raffel et al.
Revised: We train decoder-only, 110M parameter, language models (with the T codebase and architecture from Raffel et al.

Original: for one epoch over a version of C4 (Dodge et al.
Revised: for one epoch over a version of C4 (Dodge et al.

Original: In total, this was 83,216 steps with a batch size of 4,096.
Revised: In total, this was 83,216 steps with a batch size of 4,096.

Original: We deduplicated C4 with the
Revised: We deduplicated C4 with the

Original: MinHash strategy introduced by Broder (1997) and the implementation and parameters from Lee et al.
Revised: MinHash strategy introduced by Broder (1997) and the implementation and parameters from Lee et al.

Original: Lee et al.
Revised: Lee et al.

Original: (2021) showed that duplicating examples contributes to memorization, thus we seek to start our experiments with a deduplicated dataset.
Revised: (2021) showed that duplicating examples contributes to memorization, thus we seek to start our experiments with a deduplicated dataset.

Original: We used a threshold of 0.7 for both edit-distance and Jaccard similarity.
Revised: We used a threshold of 0.7 for both edit-distance and Jaccard similarity.

Original: We use the I NJECT strategy due to the large training set.
Revised: We use the I NJECT strategy due to the large training set.

Original: For each batch of canaries, we created 256 unique canaries consisting of random tokens and duplicated them betweenand 100 times to generate a full batch of 4,096 examples (including duplicates).
Revised: For each batch of canaries, we created 256 unique canaries consisting of random tokens and duplicated them betweenand 100 times to generate a full batch of 4,096 examples (including duplicates).

Original: Batches of canaries were inserted 10%, 50%, and 90% of the way into one epoch of training.
Revised: Batches of canaries were inserted 10%, 50%, and 90% of the way into one epoch of training.

Original: Additionally, we another model, holding random seed, architecture, and data order ﬁxed, without any canaries as a baseline.
Revised: Additionally, we another model, holding random seed, architecture, and data order ﬁxed, without any canaries as a baseline.

Original: C E XTENDED E XPERIMENTS C. P OISON ON I MAGE N ET
Revised: C E XTENDED E XPERIMENTS C. P OISON ON I MAGE N ET

Original: We reproduce Figure 1a in Figure 6a, which shows forgetting measured with I NJECT on ImageNet with 10 duplicates, and compare it side-by-side with P OISON in Figure 6b, where each example is added once per ImageNet epoch, for 10 epochs. 
Revised: We reproduce Figure 1a in Figure 7a, which shows forgetting measured with I NJECT on ImageNet with 10 duplicates, and compare it side-by-side with P OISON in Figure 7b, where each example is added once per ImageNet epoch, for 10 epochs. [Figure]

Original: We see forgetting happens here at roughly the same rate, after controlling for how many times an example is seen.
Revised: [Figure] [Figure] [Figure]

Original: C. R ECENCY
Revised: C. R ECENCY

Original: We plot the impact of recency on forgetting in Figure 7.
Revised: We plot the impact of recency on forgetting in Figure 8.

Original: To do so, we inject samples at various points in training, and measure how quickly models return to a small privacy leakage.
Revised: To do so, we inject samples at various points in training, and measure how quickly models return to a small privacy leakage.

Original: We investigate this in
Revised: We investigate this in

Original: Figures 7a, 7b, and 7c for the ImageNet, LibriSpeech, and C4 datasets, respectively.
Revised: Figures 8a, 8b, and 8c for the ImageNet, LibriSpeech, and C4 datasets, respectively.

Original: On ImageNet, it appears that the later a point is seen, the longer it takes to forget, although this effect seems to level[Figure] [Figure] off for ImageNet (from epoch 40 to 80, there is little change).
Revised: On ImageNet, it appears that the later a point is seen, the longer it takes to forget, although this effect seems to level off for ImageNet (from epoch 40 to 80, there is little change).

Original: This is also true for LibriSpeech, where the later a point is seen, the more times it was used in training.
Revised: This is also true for LibriSpeech, where the later a point is seen, the more times it was used in training.

Original: However, there is no clear trend on the
Revised: However, there is no clear trend on the

Original: C4 dataset.
Revised: C4 dataset.

Original: Attacks which better exploit the optimization trajectory may ﬁnd earlier examples to be forgotten more slowly, as they may have a longer impact on this trajectory.
Revised: Attacks which better exploit the optimization trajectory may ﬁnd earlier examples to be forgotten more slowly, as they may have a longer impact on this trajectory.

Original: For now, privacy attacks appear to not be able to take advantage of the trajectory, and the optimization landscape appears to obfuscate rather than reveal memorized examples.
Revised: For now, privacy attacks appear to not be able to take advantage of the trajectory, and the optimization landscape appears to obfuscate rather than reveal memorized examples.

Original: C. M ODEL S IZE
Revised: C. M ODEL S IZE

Original: We show the relationship between model size and forgetting in Figures 8a and 8b, for hard test examples and medium hardness test examples, respectively.
Revised: We show the relationship between model size and forgetting in Figures 9a and 9b, for hard test examples and medium hardness test examples, respectively.

Original: We evaluate forgetting for ResNet-18,
Revised: We evaluate forgetting for ResNet-18,

Original: [Figure]

Original: [Figure]

Original: ResNet-34, ResNet-50, and ResNet-101 models.
Revised: ResNet-34, ResNet-50, and ResNet-101 models.

Original: There seems to be little difference between models for medium hardness test examples, but ResNet-101 models seem to forget hard test examples more slowly.
Revised: There seems to be little difference between models for medium hardness test examples, but ResNet-101 models seem to forget hard test examples more slowly.

Original: Prior work has found that larger models generally memorize more (Carlini et al., 2022), making this effect natural, but it is interesting that there is little difference between most of these models.
Revised: Prior work has found that larger models generally memorize more (Carlini et al., 2022), making this effect natural, but it is interesting that there is little difference between most of these models.

Original: It is possible that all ResNet models we consider are already highly over-parameterized, and forgetting has already nearly saturated.
Revised: It is possible that all ResNet models we consider are already highly over-parameterized, and forgetting has already nearly saturated.

Original: C. O PTIMIZER P ARAMETERS
Revised: C. O PTIMIZER P ARAMETERS

Original: On ImageNet, we vary the momentum and learning rate parameters to see if these result in a signiﬁcant change to forgetting.
Revised: On ImageNet, we vary the momentum and learning rate parameters to see if these result in a signiﬁcant change to forgetting.

Original: For example, it is possible that an increased momentum parameter would lead to less forgetting, as it causes a point’s update to contribute to every subsequent update.
Revised: For example, it is possible that an increased momentum parameter would lead to less forgetting, as it causes a point’s update to contribute to every subsequent update.

Original: However, we  ﬁnd this to generally not be the case in our experiments on ImageNet: learning rate and momentum do not appear to have a signiﬁcant impact on forgetting (Figures 8c and 9a, respectively).
Revised: However, we [Figure] ﬁnd this to generally not be the case in our experiments on ImageNet: learning rate and momentum do not appear to have a signiﬁcant impact on forgetting (Figures 9c and 10a, respectively).

Original: Learning rate likely does not have a large impact, as all examples have the same learning rate, and so the data nondeterminism is equally powerful at causing forgetting in each case.
Revised: Learning rate likely does not have a large impact, as all examples have the same learning rate, and so the data nondeterminism is equally powerful at causing forgetting in each case.

Original: While it is intuitive that larger momentum parameters would lead to longer forgetting, forgetting happens over long enough step counts for momentum to not impact it signiﬁcantly.
Revised: While it is intuitive that larger momentum parameters would lead to longer forgetting, forgetting happens over long enough step counts for momentum to not impact it signiﬁcantly.

Original: However there is one exception: in our ImageNet training, we use a learning rate decay which reduces learning rate by a factor of 10 every 30 epochs (we reproduce Figure 3 in Figure 9b to discuss this effect more here).
Revised: However there is one exception: in our ImageNet training, we use a learning rate decay which reduces learning rate by a factor of 10 every 30 epochs (we reproduce Figure 3 in Figure 10b to discuss this effect more here).

Original: This has two drastic implications on forgetting: examples seen just before the decay are forgotten much more slowly, and there is a period directly after the decay where forgetting happens more quickly.
Revised: This has two drastic implications on forgetting: examples seen just before the decay are forgotten much more slowly, and there is a period directly after the decay where forgetting happens more quickly.

Original: The ﬁrst phenomenon (seen in Epoch 58 and Epoch 59 in Figure 9b) is intuitive: the rate of forgetting depends on the other examples’ contributions, and the point injected before the learning rate decay contributes 10x more than these later examples.
Revised: The ﬁrst phenomenon (seen in Epoch 58 and Epoch 59 in Figure 10b) is intuitive: the rate of forgetting depends on the other examples’ contributions, and the point injected before the learning rate decay contributes 10x more than these later examples.

Original: The other phenomenon is more surprising, and it is likely due to the optimum being more “unstable” after a learning rate decay.
Revised: The other phenomenon is more surprising, and it is likely due to the optimum being more “unstable” after a learning rate decay.

Original: This instability reduces over time, as we see in Figure 9b (at Epochs 60-62); forgetting has begun to increase to a normal rate two epochs after the learning rate decay, at Epoch 62.
Revised: This instability reduces over time, as we see in Figure 10b (at Epochs 60-62); forgetting has begun to increase to a normal rate two epochs after the learning rate decay, at Epoch 62.

Original: D E XTENSION OF U NDERSTANDING F ORGETTING , S ECTIOND. C ONCRETE K - MEANS D ESCRIPTION
Revised: C. C OMPARISON WITH T RADITIONAL F ORGETTING M ETRICS

Revised: We have argued that our approach is more relevant to privacy than traditional metrics for measuring forgetting, such as loss or accuracy (accuracy is standard in the catastrophic forgetting literature and was also used to measure forgetting in Tirumala et al.

Revised: Here, we present additional experimental justiﬁcation.

Revised: First, we see in Figure 11a that accuracy is an overly optimistic metric from a privacy perspective.

Revised: For examples repeated 10 times, accuracy has stabilized by 100 × 200 =steps after injection, suggesting that they are completely forgotten.

Revised: However, compared to Figure 2a, these examples are still at 70% membership inference precision, not invulnerable from privacy attacks.

Revised: Loss (in Figure 11b) takes longer to stabilize, providing more comparable forgetting speed to our approach.

Revised: However, membership inference scores are still more easily interpreted than loss scores for quantifying privacy risk.

Revised: We also note that, to make comparison easier, we have used hard examples to report results with repetitions, but existing work typically considers more average examples, as forgetting is traditionally used to measure average performance on subdistributions.

Revised: For this reason, existing metrics are more similar to our experiments with medium hardness or easy examples in

Revised: Figure 11c, where forgetting happens nearly immediately, while hard examples take signiﬁcantly longer.

Revised: Worst-case analysis is more relevant from a privacy perspective, justifying our approach.

Revised: [Figure]

Revised: [Figure]

Revised: D E XTENSION OF U NDERSTANDING F ORGETTING , S ECTIOND. C ONCRETE K - MEANS D ESCRIPTION

Original: We reproduce here Figure 10, which offers intuition for our setting.
Revised: We reproduce here Figure 12, which offers intuition for our setting.

Original: Concretely, we consider a one-dimensional k -means task with k = 2 , but where three clusters exist in the data, denoted by c 1 , c 2 , c 3 .
Revised: Concretely, we consider a one-dimensional k -means task with k = 2 , but where three clusters exist in the data, denoted by c 1 , c 2 , c 3 .

Original: Samples from cluster c 1 are drawn from N ( − 1 , σ 2 ) , samples from cluster c 2 are drawn from N ( µ, σ 2 ) , and samples from cluster c 3 are drawn from N (1 , σ 2 ) .
Revised: Samples from cluster c 1 are drawn from N ( − 1 , σ 2 ) , samples from cluster c 2 are drawn from N ( µ, σ 2 ) , and samples from cluster c 3 are drawn from N (1 , σ 2 ) .

Original: Here, σ 2 is some ﬁxed variance parameter (we use σ = 0 . 03 ), and µ > 0 controls the distance between c 2 and the other two clusters, and crucially makes c 2 slightly closer to c 3 than to c 1 (we use µ = 0 . 03 ).We consider
Revised: Here, σ 2 is some ﬁxed variance parameter (we use σ = 0 . 03 ), and µ > 0 controls the distance between c 2 and the other two clusters, and crucially makes c 2 slightly closer to c 3 than to c 1 (we use µ = 0 . 03 ).

Original: [Figure] a two-stage learning procedure, where some dataset D 0 is used to learn the clusters initially, and the model is then updated with some new dataset D 1 .
Revised: We consider a two-stage learning procedure, where some dataset D 0 is used to learn the clusters initially, and the model is then updated with some new dataset D 1 .

Original: We ﬁx D 1 to contain an equal number n of samples from each cluster.
Revised: We ﬁx D 1 to contain an equal number n of samples from each cluster.

Original: However, we consider two cases for D 0 : (1) in the IN case, D 0 consists of m samples from c 1 ,
Revised: However, we consider two cases for D 0 : (1) in the IN case, D 0 consists of m samples from c 1 ,

Original: one “outlier” sample
Revised: one “outlier” sample

Original: x
Revised: x

Original: (we use x = − 0 .
Revised: (we use x = − 0 .

Original: 01 , slightly more than one standard deviation from the mean) from c 2 , and m samples from c 3 , and in the OUT case, D 0 consists of m samples from c 1 and m samples from c 3 , without the “outlier”.
Revised: 01 , slightly more than one standard deviation from the mean) from c 2 , and m samples from c 3 , and in the OUT case, D 0 consists of m samples from c 1 and m samples from c 3 , without the “outlier”.

Original: Samples from c 2 are never included in D 0 , perhaps because they are from an emergent subpopulation.
Revised: Samples from c 2 are never included in D 0 , perhaps because they are from an emergent subpopulation.

Original: The single “outlier” sample can result in drastically different clusters, a difference which does not get forgotten by collecting more examples D 1 .
Revised: The single “outlier” sample can result in drastically different clusters, a difference which does not get forgotten by collecting more examples D 1 .

Original: We run an experiment with this setup, using m = 10 , n = 100 , and run 200 trials with fresh samples from each cluster.
Revised: We run an experiment with this setup, using m = 10 , n = 100 , and run 200 trials with fresh samples from each cluster.

Original: Over these trials, membership inference accuracy for x has 97% accuracy, but the attack also has perfect precision.
Revised: Over these trials, membership inference accuracy for x has 97% accuracy, but the attack also has perfect precision.

Original: That is, when x is OUT, c 1 and c 2 are far enough that they are never joined, and there are only examples where the randomness of the data causes cand c 3 to join despite the presence of x .
Revised: That is, when x is OUT, c 1 and c 2 are far enough that they are never joined, and there are only examples where the randomness of the data causes cand c 3 to join despite the presence of x .

Original: We note that differential privacy would prevent this attack, as it would randomize the clustering, preventing membership inference.
Revised: We note that differential privacy would prevent this attack, as it would randomize the clustering, preventing membership inference.

Original: Machine unlearning algorithms designed for k -means (Ginart et al., 2019) would also retrain the entire model upon removing the outlier.
Revised: Machine unlearning algorithms designed for k -means (Ginart et al., 2019) would also retrain the entire model upon removing the outlier.

Original: D. N ONDETERMINISM IS R EQUIRED
Revised: D. N ONDETERMINISM IS R EQUIRED

Original: The analysis in the remainder of this section will use the mean estimation task, a simple task which is frequently used in the theoretical privacy literature to understand the effectiveness of privacy attacks (Homer et al., 2008; Sankararaman et al., 2009; Dwork et al., 2015) or design differentially private algorithms (Bun & Steinke, 2019; Kamath et al., 2019).
Revised: The analysis in the remainder of this section will use the mean estimation task, a simple task which is frequently used in the theoretical privacy literature to understand the effectiveness of privacy attacks (Homer et al., 2008; Sankararaman et al., 2009; Dwork et al., 2015) or design differentially private algorithms (Bun & Steinke, 2019; Kamath et al., 2019).

Original: In mean estimation, a learner wants to ﬁnd the mean of a dataset, which is the minimizer of the squared (cid:96) 2 loss.
Revised: In mean estimation, a learner wants to ﬁnd the mean of a dataset, which is the minimizer of the squared (cid:96) 2 loss.

Original: To understand forgetting, we consider algorithms which perform mean estimation by per-example gradient descent, iterating through samples x from the dataset and updating the model at iteration i , θ i , by a gradient step on the squared (cid:96) 2 loss ( θ i − x ) 2 , with learning rate η .
Revised: To understand forgetting, we consider algorithms which perform mean estimation by per-example gradient descent, iterating through samples x from the dataset and updating the model at iteration i , θ i , by a gradient step on the squared (cid:96) 2 loss ( θ i − x ) 2 , with learning rate η .

Original: To show that determinism prevents forgetting, we consider the deterministic version of per-example gradient descent in Algorithm 3, which iterates through a dataset in a ﬁxed order.
Revised: To show that determinism prevents forgetting, we consider the deterministic version of per-example gradient descent in Algorithm 3, which iterates through a dataset in a ﬁxed order.

Original: In Theorem 2, we prove that this algorithm doesn’t forget, by considering two datasets which differ in only one row, and showing that the two datasets have perpetually different models, so that an adversary can distinguish between them if the adversary knows all other points.
Revised: In Theorem 2, we prove that this algorithm doesn’t forget, by considering two datasets which differ in only one row, and showing that the two datasets have perpetually different models, so that an adversary can distinguish between them if the adversary knows all other points.

Original: In the main body, we also experimentally verify this result, when logistic regression is trained by minimizing the standard cross-entropy loss with ﬁxed-order SGD on the FMNIST dataset. 
Revised: In the main body, we also experimentally verify this result, when logistic regression is trained by minimizing the standard cross-entropy loss with ﬁxed-order SGD on the FMNIST dataset. Algorithm 3:

Revised: Data: Dataset D = { x i } ni =1 , initial parameters θ 0 , learning rate η

Revised: Function T RAIN M EAN D ETERMINISTIC ( D, θ 0 , η ) :

Original:  Theorem 2.
Revised: For i = 1 ..n [Equation] Return θ n Theorem 2.

Original: Consider the described algorithm T in Algorithm 3, with 0 < η < 0 . 5 .
Revised: Consider the described algorithm T in Algorithm 3, with 0 < η < 0 . 5 .

Original: When run on any two distinct datasets D 0 and D 1 differing in only one row, we have T ( D 0 ) (cid:54) =
Revised: When run on any two distinct datasets D 0 and D 1 differing in only one row, we have T ( D 0 ) (cid:54) =

Original: T ( D 1 ) , regardless of which index D 0 and D 1 differ in.
Revised: T ( D 1 ) , regardless of which index D 0 and D 1 differ in.

Original: Proof.
Revised: Proof.

Original: Consider the intermediate points θ 

Revised: Consider the intermediate points θ

Original: i and θ 1 i , which represent the model at the i th step of training on datasets D 0 and D 1 , respectively.
Revised: i and θ 1 i , which represent the model at the i th step of training on datasets D 0 and D 1 , respectively.

Original: Write j for the index where D 0 and D 1 differ.
Revised: Write j for the index where D 0 and D 1 differ.

Original: To

Original: Data: Dataset D = { x i } ni =1 , initial parameters θ 0 , learning rate η

Original: Function T RAIN M EAN D ETERMINISTIC ( D, θ 0 , η ) :

Original: For i = 1 ..n θ i = θ i − 1 − 2 η

Original: ( θ i − 1 − x i )

Original: Return θ n prove the theorem, we ﬁrst note that θ 

Revised: To prove the theorem, we ﬁrst note that θ

Original: j − 1 =
Revised: j − 1 =

Original: θ 1 j − 1 , as the ﬁrst j − 1 steps of training all use identical examples from D 0 , D 1 .
Revised: θ 1 j − 1 , as the ﬁrst j − 1 steps of training all use identical examples from D 0 , D 1 .

Original: Next, we show that θ 0 j (cid:54) = θ 1 j , and then that θ 0 i (cid:54)
Revised: Next, we show that θ 0 j (cid:54) = θ 1 j , and then that θ 0 i (cid:54)

Original: =
Revised: =

Original: θ 1 i for all i > j .
Revised: θ 1 i for all i > j .

Original: To see that θ 0 j (cid:54) =
Revised: To see that θ 0 j (cid:54) =

Original: θ 1 j , we write out the precise gradient updates:
Revised: θ 1 j , we write out the precise gradient updates:

Original: [Equation]
Revised: [Equation]

Original: where the second step holds because θ 0 j − 

Revised: where the second step holds because θ 0 j −

Original: = θ 1 j − 1 , and the third holds because j is the index where
Revised: = θ 1 j − 1 , and the third holds because j is the index where

Original: D 0 and D 1 differ.
Revised: D 0 and D 1 differ.

Original: Now, to show that this implies that θ 0 i (cid:54)
Revised: Now, to show that this implies that θ 0 i (cid:54)

Original: =
Revised: =

Original: θ 1 i for all i > j , we show that, for all x , the gradient update function is one-to-one.
Revised: θ 1 i for all i > j , we show that, for all x , the gradient update function is one-to-one.

Original: That is, for all x , θ (cid:54) = θ (cid:48) , taking a gradient step on x still results in different models:
Revised: That is, for all x , θ (cid:54) = θ (cid:48) , taking a gradient step on x still results in different models:

Original: [Equation]
Revised: [Equation]

Original: From this, θ 0 j (cid:54) = θ 1 j implies that the models are different at step j + 1 , which itself implies that the models are different at step j + 2 , and will be different for the entirety of training.
Revised: From this, θ 0 j (cid:54) = θ 1 j implies that the models are different at step j + 1 , which itself implies that the models are different at step j + 2 , and will be different for the entirety of training.

Original: D. 
Revised: D. D ATA R ANDOMNESS CAN C AUSE F ORGETTING

Original: D ATA R ANDOMNESS CAN C AUSE F ORGETTING

Original: Here, we prove Theorem 1, restated below.
Revised: Here, we prove Theorem 1, restated below.

Original: Algorithm

Original: Data: Injection v , initial parameters θ 0 , training steps k , distribution D = N ( µ, Σ) , learning rate η

Original: Function T RAIN M EAN S AMPLED ( θ 0 , v, k, η, D ) :

Original: [Equation]

Original: For i = 1 , . . .

Original: , k

Original: [Equation]

Original: Return θ + k , θ − k

Original: Theorem 1.
Revised: Theorem 1.

Original: The distributions of the models θ + k and θ − k output by Algorithm 4 with learning rate 0 < η < 1 / 2 have R´enyi divergence of order α at most 2 αk v (cid:62) Σ − 1 v .
Revised: The distributions of the models θ + k and θ − k output by Algorithm 4 with learning rate 0 < η < 1 / 2 have R´enyi divergence of order α at most 2 αk v (cid:62) Σ − 1 v .

Original: Proof.
Revised: Proof.

Original: First, observe that the gradient update on some θ 0 with example x 0 produces θ 1 = θ 0 − 2 η ( θ 0 − x 0 )
Revised: First, observe that the gradient update on some θ 0 with example x 0 produces θ 1 = θ 0 − 2 η ( θ 0 − x 0 )

Original: =
Revised: =

Original: θ 0 (1 − 2 η ) + 2 ηx 0 .
Revised: θ 0 (1 − 2 η ) + 2 ηx 0 .

Original: Another gradient update on x 1 produces θ 2 = θ 0 (1 − 2 η ) [-
-]
Revised: Another gradient update on x 1 produces θ 2 = θ 0 (1 − 2 η ) [-
-]2 + Data: Injection v , initial parameters θ 0 , training steps k , distribution D = N ( µ, Σ) , learning rate η Function T RAIN M EAN S AMPLED ( θ 0 , v, k, η, D ) : [Equation]

Revised: (cid:46) Update with + v

Revised: [Equation]

Original: + 2 η (1 − 2 η ) x 0 + 2 ηx 1  . 
Revised: (cid:46) Update with − v For i = 1 , . . . , k x + i , x − i ← S AMPLE ( D )

Revised: [Equation]

Revised: θ

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: Return θ + k , θ − k 2 η (1 − 2 η ) x 0 + 2 ηx 1 .

Original: In general, we have
Revised: In general, we have

Original: [Equation]
Revised: [Equation]

Original: We write θ +0 = θ 0 (1 − 2 η ) + 2 ηv (a gradient step on + v ) and θ − 

Revised: We write θ +0 = θ 0 (1 − 2 η ) + 2 ηv (a gradient step on + v ) and θ −

Original: = θ 0 (1 − 2 η ) − 2 ηv (a gradient step on − v ).
Revised: = θ 0 (1 − 2 η ) − 2 ηv (a gradient step on − v ).

Original: Then we analyze the distributions θ + k and θ − k .
Revised: Then we analyze the distributions θ + k and θ − k .

Original: Using Equation 1, replacing θ 0 with θ +and θ − 0 , and using that each x i is sampled from N ( µ, Σ) , we obtain the distributions
Revised: Using Equation 1, replacing θ 0 with θ + and θ − 0 , and using that each x i is sampled from N ( µ, Σ) , we obtain the distributions

Original: [Equation]
Revised: [Equation]

Original: Using that the R´enyi divergence of order α between two multivariate Gaussians with means µ 0 , µand equal covariance Σ (cid:48) is α 2 ( µ 0 − µ 1 ) T Σ (cid:48)− 1 ( µ 0 − µ 1 ) , we can compute the divergence for θ + k and θ − k as 8 α η (1 − η )(1 − 2 η ) 2 k 1 − (1 − 2 η ) 2 k v T Σ − 1 v .
Revised: Using that the R´enyi divergence of order α between two multivariate Gaussians with means µ 0 , µand equal covariance Σ (cid:48) is α 2 ( µ 0 − µ 1 ) T Σ (cid:48)− 1 ( µ 0 − µ 1 ) , we can compute the divergence for θ + k and θ − k as 8 α η (1 − η )(1 − 2 η ) 2 k 1 − (1 − 2 η ) 2 k v T Σ − 1 v .

Original: It remains to bound the η term in this product.
Revised: It remains to bound the η term in this product.

Original: The function f ( η ) =
Revised: The function f ( η ) =

Original: η (1 − η )(1 − 2 η ) 2 k 1 − (1 − 2 η ) 2 k has negative derivative on the interval 0 < η < 1 / 2 for any k > 0 .
Revised: η (1 − η )(1 − 2 η ) 2 k 1 − (1 − 2 η ) 2 k has negative derivative on the interval 0 < η < 1 / 2 for any k > 0 .

Original: Then f ( η ) it is maximized on 0 < η < 1 /when η → 0 ; we can compute this limit with L’Hospital’s rule as 14 k , completing the proof.
Revised: Then f ( η ) it is maximized on 0 < η < 1 /when η → 0 ; we can compute this limit with L’Hospital’s rule as 14 k , completing the proof.