Original: E FFICIENT A UGMENTATION VIA D ATA S UBSAMPLING
Revised: E FFICIENT A UGMENTATION VIA D ATA S UBSAMPLING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Data augmentation is commonly used to encode invariances in learning methods.
Revised: Data augmentation is commonly used to encode invariances in learning methods.

Original: However, this process is often performed in an inefﬁcient manner, as artiﬁcial examples are created by applying a number of transformations to all points in the training set.
Revised: However, this process is often performed in an inefﬁcient manner, as artiﬁcial examples are created by applying a number of transformations to all points in the training set.

Original: The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply.
Revised: The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply.

Original: In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance beneﬁts of augmenting the entire dataset.
Revised: In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance beneﬁts of augmenting the entire dataset.

Original: We propose a novel set of subsampling policies, based on model inﬂuence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.
Revised: We propose a novel set of subsampling policies, based on model inﬂuence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Data augmentation is a process in which the training set is expanded by applying class-preserving transformations, such as rotations or crops for images, to the original data points.
Revised: Data augmentation is a process in which the training set is expanded by applying class-preserving transformations, such as rotations or crops for images, to the original data points.

Original: This process has become an instrumental tool in achieving state-of-the-art accuracy in modern machine learning pipelines.
Revised: This process has become an instrumental tool in achieving state-of-the-art accuracy in modern machine learning pipelines.

Original: Indeed, for problems in image recognition, data augmentation is a key component in achieving nearly all state-of-the-art results (Cires¸an et al., 2010; Dosovitskiy et al., 2016; Graham, 2014; Sajjadi et al., 2016).
Revised: Indeed, for problems in image recognition, data augmentation is a key component in achieving nearly all state-of-the-art results (Cires¸an et al., 2010; Dosovitskiy et al., 2016; Graham, 2014; Sajjadi et al., 2016).

Original: Data augmentation is also a popular technique because of its simplicity, particularly in deep learning applications, where applying a set of known invariances to the data is often more straightforward than trying to encode this knowledge directly in the model architecture.
Revised: Data augmentation is also a popular technique because of its simplicity, particularly in deep learning applications, where applying a set of known invariances to the data is often more straightforward than trying to encode this knowledge directly in the model architecture.

Original: However, data augmentation can be an expensive process, as applying a number of transformations to the entire dataset may increase the overall size of the dataset by orders of magnitude.
Revised: However, data augmentation can be an expensive process, as applying a number of transformations to the entire dataset may increase the overall size of the dataset by orders of magnitude.

Original: For example, if applying just 3 sets of augmentations (e.g., translate, rotate, crop), each with 4 possible conﬁgurations, the dataset can easily grow by a factor of 12 (if applied independently), all the way to 64x (if applied in sequence).
Revised: For example, if applying just 3 sets of augmentations (e.g., translate, rotate, crop), each with 4 possible conﬁgurations, the dataset can easily grow by a factor of 12 (if applied independently), all the way to 64x (if applied in sequence).

Original: While this may have some beneﬁts in terms of overﬁtting, augmenting the entire training set can also signiﬁcantly increase data storage costs and training time, which can scale linearly or superlinearly with respect to the training set size.
Revised: While this may have some beneﬁts in terms of overﬁtting, augmenting the entire training set can also signiﬁcantly increase data storage costs and training time, which can scale linearly or superlinearly with respect to the training set size.

Original: Further, selecting the optimal set of transformations to apply to a given data point is often a non-trivial task.
Revised: Further, selecting the optimal set of transformations to apply to a given data point is often a non-trivial task.

Original: Indeed, applying transformations not only takes processing time, but also frequently requires some amount of domain expertise.
Revised: Indeed, applying transformations not only takes processing time, but also frequently requires some amount of domain expertise.

Original: Augmentations are often applied heuristically in practice, and small perturbations are expected (but not proven) to preserve classes.
Revised: Augmentations are often applied heuristically in practice, and small perturbations are expected (but not proven) to preserve classes.

Original: If more complex augmentations are applied to a dataset, they may have to be veriﬁed on a per-sample basis.
Revised: If more complex augmentations are applied to a dataset, they may have to be veriﬁed on a per-sample basis.

Original: In this work, we aim to make data augmentation more efﬁcient and user-friendly by identifying subsamples of the full dataset that are good candidates for augmentation.
Revised: In this work, we aim to make data augmentation more efﬁcient and user-friendly by identifying subsamples of the full dataset that are good candidates for augmentation.

Original: In developing policies for subsampling the data, we draw inspiration from the virtual support vector (VSV) method, which has been used for this purpose in the context of SVMs (Burges & Sch¨olkopf, 1997; Decoste &
Revised: In developing policies for subsampling the data, we draw inspiration from the virtual support vector (VSV) method, which has been used for this purpose in the context of SVMs (Burges & Sch¨olkopf, 1997; Decoste &

Original: Sch¨olkopf, 2002).
Revised: Sch¨olkopf, 2002).

Original: The VSV method attempts to create a more robust decision surface by augmenting only the samples that are close to the margin—i.e., the support vectors.
Revised: The VSV method attempts to create a more robust decision surface by augmenting only the samples that are close to the margin—i.e., the support vectors.

Original: The motivation is intuitive: if a point does not affect the margin, then any small perturbation of that point in data space will likely yield a point that is again too far from the margin to affect it.
Revised: The motivation is intuitive: if a point does not affect the margin, then any small perturbation of that point in data space will likely yield a point that is again too far from the margin to affect it.

Original: The method proceeds by applying classpreserving data augmentations (e.g., small perturbations) to all support vectors in the training set.
Revised: The method proceeds by applying classpreserving data augmentations (e.g., small perturbations) to all support vectors in the training set.

Original: The SVM is then retrained on the support vector dataset concatenated with the augmented dataset, and the end result is a decision surface that has been encoded with transformation invariance while augmenting many fewer samples than found in the full training set.
Revised: The SVM is then retrained on the support vector dataset concatenated with the augmented dataset, and the end result is a decision surface that has been encoded with transformation invariance while augmenting many fewer samples than found in the full training set.

Original: Although proven to be an effective approach for SVMs, methods utilizing support vectors typically do not generalize well to other classiﬁers.
Revised: Although proven to be an effective approach for SVMs, methods utilizing support vectors may not generalize well to other classiﬁers.

Original: Therefore, in this work, we aim to develop policies that can effectively reduce the augmentation set size while applying to a much broader class of models.
Revised: Therefore, in this work, we aim to develop policies that can effectively reduce the augmentation set size while applying to a much broader class of models.

Original: A key step in developing these policies is to determine some metric by which to rank the importance of data points for augmentation.
Revised: A key step in developing these policies is to determine some metric by which to rank the importance of data points for augmentation.

Original: We build policies based on two key metrics.
Revised: We build policies based on two key metrics.

Original: First, we make a natural generalization of the VSV method by measuring the loss induced by a training point.
Revised: First, we make a natural generalization of the VSV method by measuring the loss induced by a training point.

Original: Second, we explore using the inﬂuence of a point as an indicator of augmentation potential.
Revised: Second, we explore using the inﬂuence of a point as an indicator of augmentation potential.

Original: Inﬂuence functions, originating from robust statistics, utilize more information than loss (i.e., residuals) alone, as they take into account both leverage and residual information.
Revised: Inﬂuence functions, originating from robust statistics, utilize more information than loss (i.e., residuals) alone, as they take into account both leverage and residual information.

Original: The contributions of this paper are as follows.
Revised: The contributions of this paper are as follows.

Original: First, we demonstrate that it is typically unnecessary to augment the entire dataset to achieve high accuracy—for example, we can maintain 99 . 8% or more of the full augmentation accuracy while only augmenting 10% of the dataset in the case of translation augmentations, and we observe similar behavior for other augmentations.
Revised: First, we demonstrate that it is typically unnecessary to augment the entire dataset to achieve high accuracy—for example, we can maintain 99 . 86% or more of the full augmentation accuracy while only augmenting 10% of the dataset in the case of translation augmentations, and we observe similar behavior for other augmentations.

Original: Second, we propose several policies to select the subset of points to augment.
Revised: Second, we propose several policies to select the subset of points to augment.

Original: Our results indicate that policies based off of training loss or model inﬂuence are an effective strategy over simple baselines, such as random sampling.
Revised: Our results indicate that policies based off of training loss or model inﬂuence are an effective strategy over simple baselines, such as random sampling.

Original: Finally, we propose several modiﬁcations to these approaches, such as sample reweighting and online learning, that can further improve performance.
Revised: Finally, we propose several modiﬁcations to these approaches, such as sample reweighting and online learning, that can further improve performance.

Original: Our proposed policies are simple and straightforward to implement, requiring only a few lines of code.
Revised: Our proposed policies are simple and straightforward to implement, requiring only a few lines of code.

Original: Throughout, our experiments are performed on common benchmark datasets, such as MNIST, CIFAR10, and NORB.
Revised: Throughout, our experiments are performed on common benchmark datasets, such as MNIST, CIFAR10, and NORB.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In the domain of image classiﬁcation, most state-of-the-art pipelines use some form of data augmentation (Cires¸an et al., 2010; Dosovitskiy et al., 2016; Graham, 2014; Sajjadi et al., 2016).
Revised: In the domain of image classiﬁcation, most state-of-the-art pipelines use some form of data augmentation (Cires¸an et al., 2010; Dosovitskiy et al., 2016; Graham, 2014; Sajjadi et al., 2016).

Original: This typically consists of applying crops, ﬂips, or small afﬁne transformations to all the data points in the training set, with parameters drawn randomly from hand-tuned ranges.
Revised: This typically consists of applying crops, ﬂips, or small afﬁne transformations to all the data points in the training set, with parameters drawn randomly from hand-tuned ranges.

Original: Beyond image classiﬁcation, various studies have applied data augmentation techniques to modalities such as audio (Uhlich et al., 2017) and text (Lu et al., 2006).
Revised: Beyond image classiﬁcation, various studies have applied data augmentation techniques to modalities such as audio (Uhlich et al., 2017) and text (Lu et al., 2006).

Original: The selection of these augmentation strategies can have large performance impacts, and thus can require extensive selection and tuning (Ratner et al., 2017).
Revised: The selection of these augmentation strategies can have large performance impacts, and thus can require extensive selection and tuning (Ratner et al., 2017).

Original: Motivated by the ubiquity of data augmentation and the difﬁculty in selecting augmentations, there has been a signiﬁcant amount of work in selecting and tuning the best transformations to use when performing augmentation.
Revised: Motivated by the ubiquity of data augmentation and the difﬁculty in selecting augmentations, there has been a signiﬁcant amount of work in selecting and tuning the best transformations to use when performing augmentation.

Original: For example, Fawzi et al.
Revised: For example, Fawzi et al.

Original: (2016) use adaptive data augmentation to choose transformations that maximize loss for the classiﬁer; Ratner et al.
Revised: (2016) use adaptive data augmentation to choose transformations that maximize loss for the classiﬁer; Ratner et al.

Original: (2017) propose learning a sequence model of composed transformations; and Cubuk et al.
Revised: (2017) propose learning a sequence model of composed transformations; and Cubuk et al.

Original: (2018) suggest a reinforcement learning approach.
Revised: (2018) suggest a reinforcement learning approach.

Original: In contrast to the above works, our aimin this work is instead to select which data points to augment while holding transformations ﬁxed.
Revised: In contrast to these works, our aim is instead to select which data points to augment while holding transformations ﬁxed.

Original: We note that our subsampling policies are therefore complementary to many of the described approaches, and in fact could be quite beneﬁcial for approaches such as reinforcement learning that can quickly become infeasible for large datasets and transformation spaces.
Revised: Our subsampling policies are therefore complementary to many of the described approaches, and in fact could be quite beneﬁcial for approaches such as reinforcement learning that can quickly become infeasible for large datasets and transformation spaces.

Original: Finally, we note that several recent works have proposed augmentation strategies based on adversarial training approaches, such as robust optimization frameworks or generative adversarial networks (GANs) (Goodfellow et al., 2014; Antoniou et al., 2017; Volpi et al., 2018).
Revised: Finally, we note that several recent works have proposed augmentation strategies based on adversarial training approaches, such as robust optimization frameworks or generative adversarial networks (GANs) (Goodfellow et al., 2014; Antoniou et al., 2017; Volpi et al., 2018).

Original: These approaches generate artiﬁcial points from some target distribution, rather than by directly transforming the original training points.
Revised: These approaches generate artiﬁcial points from some target distribution, rather than by directly transforming the original training points.

Original: We view these works as orthogonal and complementary approaches to the proposed work, which is designed in concert with more traditional data augmentation strategies.
Revised: We view these works as orthogonal and complementary approaches to the proposed work, which is designed in concert with more traditional data augmentation strategies.

Original: The area of work most closely related to our own is that of the Virtual Support Vector (VSV) method (Burges & Sch¨olkopf, 1997; Decoste & Sch¨olkopf, 2002).
Revised: The area of work most closely related to our own is that of the Virtual Support Vector (VSV) method (Burges & Sch¨olkopf, 1997; Decoste & Sch¨olkopf, 2002).

Original: This method was proposedin thecontext of support vector machines as a way to reduce the set ofcandidate points for augmentation by limiting transformations to only support vectors.
Revised: This method was proposed the support vector machine literature as a way to reduce the set of points for augmentation by limiting transformations to only support vectors.

Original: In the context of SVMs, the motivation is straightforward, as points that are far from the margin are unlikely to affect future models if they are transformed via small perturbations.
Revised: In the context of SVMs, the motivation is straightforward, as points that are far from the margin are unlikely to affect future models if they are transformed via small perturbations.

Original: However, to the best of our knowledge, there has been no work extending these ideas to methods beyond SVMs, where the notion of support vectors no longer applies.
Revised: However, to the best of our knowledge, there has been no work extending these ideas to methods beyond SVMs, where the notion of support vectors is not directly applicable.

Original: Inspired by the VSV work, we similarly seek ways to downsample the set of candidate points for augmentation, though through metrics beyond support vectors.
Revised: Inspired by the VSV work, we similarly seek ways to downsample the set of candidate points for augmentation, though through metrics beyond support vectors.

Original: One such metric is that of model inﬂuence, which has been rigorously studied in the ﬁeld of robust statistics as a way to determine which data points are most impactful on the model.
Revised: One such metric is that of model inﬂuence, which has been rigorously studied in the ﬁeld of robust statistics as a way to determine which data points are most impactful on the model.

Original: Model inﬂuence has been studied extensively in the regression literature (Hoaglin & Welsch, 1978; Pregibon et al., 1981; Cook, 1986; Walker & Birch, 1988), and more recently, in non-differentiable (SVMs) and non-convex (deep networks) settings (Koh & Liang, 2017).
Revised: Model inﬂuence has been studied extensively in the regression literature (Hoaglin & Welsch, 1978; Pregibon et al., 1981; Cook, 1986; Walker & Birch, 1988), and more recently, in non-differentiable (SVMs) and non-convex (deep networks) settings (Koh & Liang, 2017).

Original: We also explore policies based off of simpler notions, such as loss; indeed, one of our proposed policies (based on loss) results in the original VSV method as a special case, when the number of augmented points is ﬁxed to be exactly the number of support vectors.
Revised: We also explore policies based off of simpler notions, such as loss; indeed, one of our proposed policies (based on loss) can be seen as a generalization of the original VSV method in the case of training SVMs, as points with non-zero loss are support vectors 1 .

Original: We discuss additional details on these metrics and the resulting policies in Section 4.
Revised: We discuss additional details on these metrics and the resulting policies in Section 4.

Original: Finally, we note that this work is closely related to work in data subsampling methods for general dataset reduction (i.e., not in the context of data augmentation).
Revised: Finally, we note that this work is closely related to work in data subsampling methods for general dataset reduction (i.e., not in the context of data augmentation).

Original: For example, work using gradients (Zhu, 2016), leverage (Drineas et al., 2011), and inﬂuence functions (Ting & Brochu, 2017; Wang et al., 2018) have shown better results than uniform sampling of data samples in the original dataset.
Revised: For example, work using gradients (Zhu, 2016), leverage (Drineas et al., 2011; 2012; Ma et al., 2015), and inﬂuence functions (McWilliams et al., 2014; Ting & Brochu, 2017; Wang et al., 2018) have shown better results than uniform sampling of data samples in the original dataset.

Original: Our scenario differs from the subsampling scenarios in these works as we anticipate ultimately increasing the size of the dataset through augmentation, rather than decreasing it as is the case with subsampling.
Revised: Our scenario differs from the subsampling scenarios in these works as we anticipate ultimately increasing the size of the dataset through augmentation, rather than decreasing it as is the case with subsampling.

Original: Indeed, subsampling methods are motivated by being unable to train models on entire datasets due to the datasets being too large.
Revised: Indeed, subsampling methods are motivated by being unable to train models on entire datasets due to the datasets being too large.

Original: Our motivation is instead that the full augmented dataset may be too large, but the original training set is sufﬁciently small to be handled without special consideration.
Revised: Our motivation is instead that the full augmented dataset may be too large, but the original training set is sufﬁciently small to be handled without special consideration.

Original: We therefore assume it is possible to obtain exact ﬁtting information (e.g., inﬂuence, loss, etc.) by ﬁtting a model to the original data.
Revised: We therefore assume it is possible to obtain exact ﬁtting information (e.g., inﬂuence, loss, etc.) by ﬁtting a model to the original data.

Original: Further, the interpretation of our scenario differs, as the subsampling is performed with the ultimate aim being to retain the accuracy of the some yet-to-be-determined fully augmented dataset, as opposed to the original dataset.
Revised: Further, the interpretation of our scenario differs, as the subsampling is performed with the ultimate aim being to retain the accuracy of the some yet-to-be-determined fully augmented dataset, as opposed to the original dataset.

Original:  M OTIVATION : O N THE E FFECTIVENESS OF S UBSAMPLING
Revised:  M OTIVATION : O N THE E FFECTIVENESS OF S UBSAMPLING

Original: In this work, we seek to make data augmentation more efﬁcient by providing effective policies for subsampling the original training dataset.
Revised: In this work, we seek to make data augmentation more efﬁcient by providing effective policies for subsampling the original training dataset.

Original: To motivate the effect of subsampling prior to augmentation, we begin with a simple example.
Revised: To motivate the effect of subsampling prior to augmentation, we begin with a simple example.

Original: In Table 1, we report the effect that performing translation augmentations has on the ﬁnal test accuracy for several datasets (MNIST, CIFAR10, NORB).
Revised: In Table 1, we report the effect that performing translation augmentations has on the ﬁnal test accuracy for several datasets (MNIST, CIFAR10, NORB).

Original: In the second column, we provide the ﬁnal test accuracy assuming none of the training data points are augmented, and in the last column, the ﬁnal test accuracy after augmenting all of the training data points (i.e., our desired test accuracy).
Revised: In the second column, we provide the ﬁnal test accuracy assuming none of the training data points are augmented, and in the last column, the ﬁnal test accuracy after augmenting all of the training data points (i.e., our desired test accuracy).

Original: Note that the test dataset in these examples has also been augmented with translation so as to highlight the effect of augmentation; we provide full experimental details in
Revised: Note that the test dataset in these examples has also been augmented with translation so as to highlight the effect of augmentation; we provide full experimental details in

Original: Section 5.
Revised: Section 5.

Original: In columns 3-8, we report test accuracies from augmenting 5 , 10 , and 25 percent of the data, where these subsamples are either derived using simple random sampling, or via our proposed policies (to be discussed in Section 4).
Revised: In columns 3–8, we report test accuracies from augmenting 5 , 10 , and 25 percent of the data, where these subsamples are either derived using simple random sampling or via our proposed policies (to be discussed in Section 4).

Original: An immediate take-away from these results is that, even in the case of simple random sampling, it isclear that it is often unnecessary to augment the entire dataset to achieve decent accuracy gains.
Revised: An immediate take-away from these results is that, even in the case of simple random sampling, it is often unnecessary to augment the entire dataset to achieve decent accuracy gains.

Original: For example, augmenting just 25% of the dataset selected at random can yield more than half of the total accuracy gain from full augmentation.
Revised: For example, augmenting just 25% of the dataset selected at random can yield more than half of the total accuracy gain from full augmentation.

Original: However, it is also evident that subsampling can be done more effectively with the appropriate policy.
Revised: However, it is also evident that subsampling can be done more effectively with the appropriate policy.

Original: Indeed, as compared to random sampling, when augmenting just 10% of the data, these optimal policies can achieve almost identical results to full augmentation (within .1-.2% for MNIST andCIFAR, with even higher accuracy  for  NORB).
Revised: Indeed, as compared to random sampling, when augmenting just 10% of the data, these optimal policies can achieve almost identical results to full augmentation (within . 1% for CIFAR10 and higher accuracy than full augmentation for MNIST and NORB).

Original: These results aim to serve as  starting point for the remaining paper: We describe our proposed policies in detail in Section 4, and provide full experiments and experimental details in Section 5.
Revised: These results aim to serve as a starting point for the remaining paper: We describe our proposed policies in detail in Section 4, and provide full experiments and experimental details in Section 5.

Original: [Table]
Revised: [Table]

Original:  A UGMENTATION S ET S ELECTION P OLICIES
Revised:  A UGMENTATION S ET S ELECTION P OLICIES

Original: In this section, we provide details on our augmentation policies, including their general structure (described below), the metrics they utilize (Section 4.1), and improvements such as reweighting or online learning (Section 4.2).
Revised: In this section, we provide details on our augmentation policies, including their general structure (described below), the metrics they utilize (Section 4.1), and improvements such as reweighting or online learning (Section 4.2).

Original: Setup.
Revised: Setup.

Original: The aim in this work is to ﬁnd some subset S :
Revised: The aim in this work is to ﬁnd some subset S :

Original: = { ( x i , y i ) , . . .
Revised: = { ( x i , y i ) , . . .

Original: ( x j , y j ) } of the full training set
Revised: ( x j , y j ) } of the full training set

Original: D := { ( x 1 , y 1 ) , . . .
Revised: D := { ( x 1 , y 1 ) , . . .

Original: ( x n , y n ) } , such that augmenting only the subset S results in similar performance to augmenting the entire dataset D .
Revised: ( x n , y n ) } , such that augmenting only the subset S results in similar performance to augmenting the entire dataset D .

Original: More precisely, the goal is to minimize the size of S , |S| , subject to the constraint that perf ( S aug )
Revised: More precisely, the goal is to minimize the size of S , |S| , subject to the constraint that perf ( S aug )

Original: ≈ perf ( D aug ) , where S aug and D aug represent the dataset after appending augmented examples generated from the original examples in S or D , respectively.
Revised: ≈ perf ( D aug ) , where S aug and D aug represent the dataset after appending augmented examples generated from the original examples in S or D , respectively.

Original: We note that while the performance measure perf( · ) may be broadly construed, we focus in our experiments on measuring speciﬁcally performance based on test accuracy.
Revised: We note that while the performance measure perf ( · ) may be broadly construed, we focus in our experiments on measuring speciﬁcally performance based on test accuracy.

Original: General Policies.
Revised: General Policies.

Original: Our proposed policies consist of two parts: (i) an augmentation score which maps each training point ( x i , y i ) to a value s ∈ R , and (ii) a policy by which to sample points based on these augmentation scores.
Revised: Our proposed policies consist of two parts: (i) an augmentation score which maps each training point ( x i , y i ) to a value s ∈ R , and (ii) a policy by which to sample points based on these augmentation scores.

Original: We describe two metrics by which augmentation scores are generated, including loss and model inﬂuence, in Section 4.1.
Revised: We describe two metrics by which augmentation scores are generated, including loss and model inﬂuence, in Section 4.1.

Original: In terms of policies for subset selection based on these scores, we ﬁrst explore two simple policies—deterministic and random.
Revised: In terms of policies for subset selection based on these scores, we ﬁrst explore two simple policies—deterministic and random.

Original: In particular, given a set of augmentation scores { s 1 , . . .
Revised: In particular, given a set of augmentation scores { s 1 , . . .

Original: , s n } for the n training points, we select a subset S ⊆ D either by ordering the points based on their scores and taking the top k values (in a deterministic fashion), or by converting each augmentation score
Revised: , s n } for the n training points, we select a subset S ⊆ D either by ordering the points based on their scores and taking the top k values (in a deterministic fashion), or by converting each augmentation score

Original: s i to a probability π i ( z ) ∈ [0 , 1] , and then sampling according to this distribution without replacement.
Revised: s i to a probability π i ( z ) ∈ [0 , 1] , and then sampling according to this distribution without replacement.

Original: As the augmentation scores (and resulting policies) may be affected by updates to the model after each augmentation, we additionally explore in Section 4.2 the effect of iteratively updating or re-weighting scores to adjust for shifts in the underlying model.
Revised: As the augmentation scores (and resulting policies) may be affected by updates to the model after each augmentation, we additionally explore in Section 4.2 the effect of iteratively updating or re-weighting scores to adjust for shifts in the underlying model.

Original: A non-exhaustive overview of the various augmentation policies is provided in Table 2. 
Revised: A non-exhaustive overview of the various augmentation policies is provided in Table 2. [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: 4. M ETRICS : L OSS AND I NFLUENCE
Revised: 4. M ETRICS : L OSS AND I NFLUENCE

Original: We propose two metrics to determine our augmentation scores: training loss and model inﬂuence.
Revised: We propose two metrics to determine our augmentation scores: training loss and model inﬂuence.

Original: Training loss.
Revised: Training loss.

Original: One method to obtain augmentation scores is the loss at a point in the training set.
Revised: One method to obtain augmentation scores is the loss at a point in the training set.

Original: This can be viewed as a more direct generalization of the virtual support vector (VSV) method, as support vectors are points with non-zero loss.
Revised: This can be viewed as a more direct generalization of the virtual support vector (VSV) method, as support vectors are points with non-zero loss.

Original: However, studying loss directly will allow us: (i) to extend to methods beyond SVMs [Table] and (ii) to expand the augmented set to data points beyond just the ﬁxed set of support vectors.
Revised: However, studying loss directly will allow us: (i) to extend to methods beyond SVMs, and (ii) to expand the augmented set to data points beyond just the ﬁxed set of support vectors.

Original: Inﬂuence.
Revised: Inﬂuence.

Original: We also explore policies based on Leave-One-Out (LOO) inﬂuence, which measures the inﬂuence that a training data point has against its own loss when it is removed from the training set.
Revised: We also explore policies based on Leave-One-Out (LOO) inﬂuence, which measures the inﬂuence that a training data point has against its own loss when it is removed from the training set.

Original: We follow the notation used in Koh & Liang (2017).
Revised: We follow the notation used in Koh & Liang (2017).

Original: Let ˆ θ be the minimizer of the loss, which is assumed to be twice differentiable and strictly convex in θ .
Revised: Let ˆ θ be the minimizer of the loss, which is assumed to be twice differentiable and strictly convex in θ .

Revised: Let H ˆ θ be the Hessian of the loss with respect to θ evaluated at the minimizer.

Original: We deﬁne the inﬂuence of upweighting a point, z , on the loss at a test point, z test , as I up,loss ( z, z test ) :
Revised: We deﬁne the inﬂuence of upweighting a point, z , on the loss at a test point, z test , as I up,loss ( z, z test ) :

Original: = −∇ θ L ( z test , ˆ θ ) T H − 1 ˆ θ ∇ θ L ( z test , ˆ θ ) .
Revised: = −∇ θ L ( z test , ˆ θ ) (cid:62) H − 1 ˆ θ ∇ θ L ( z, ˆ θ ) .

Original: It follows that if the test point is z , then the LOO inﬂuence can be calculated as: 
Revised: It follows that if the test point is z , then the LOO inﬂuence can be calculated as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For our augmentation scores, we care only about the magnitude of the LOO inﬂuence, so it can be assumed that the sign is dropped.
Revised: For our augmentation scores, we care only about the magnitude of the LOO inﬂuence, so it can be assumed that the sign is dropped.

Original: To understand the potential of using training loss and model inﬂuence for scoring, we provide a histogram of model inﬂuence across the CIFAR and NORB datasets in Figure 1.
Revised: To understand the potential of using training loss and model inﬂuence for scoring, we provide a histogram of model inﬂuence across the CIFAR10 and NORB datasets in Figure 1.

Original: Full results for all datasets and for training loss are provided in Appendix A.
Revised: Full results for all datasets and for training loss are provided in Appendix A.

Original: In Figure 1, we see that while most of the mass is centered around 0 (which we utilize to avoid points), there is sufﬁcient variability to allow for ranking points by preference.
Revised: In Figure 1, we see that while most of the mass is centered around 0 (which we utilize to avoid points), there is sufﬁcient variability to allow for ranking points by preference.

Original: Further, as seen in Figure 2, these values are correlated before and after augmentation, indicating that these metrics are a reliable measure of the future impact of a data point after augmentation.
Revised: Further, as seen in Figure 2, these values are correlated before and after augmentation, indicating that these metrics are a reliable measure of the future impact of a data point after augmentation.

Original: In terms of quantifying correlation, we observe spearman correlations between 0 . 5 and 0 . 97 with p-values less than 0 .
Revised: We observe Spearman’s rank correlations between 0 .and 0 . 97 with p-values less than 0 .

Original: 001 (and usually orders of magnitude less).
Revised: 001 (and usually orders of magnitude less).

Original: [Figure]
Revised: [Figure]

Original: We ﬁnd similar results when measuring training loss (Appendix A).
Revised: We ﬁnd similar results when measuring training loss (Appendix A).

Original: [Figure]
Revised: [Figure]

Original: which that are un-inﬂuential typically remain un-inﬂuential.

Original: 4. R EFINEMENTS : S AMPLE R EWEIGHTING AND S CORE U PDATING
Revised: 4. R EFINEMENTS : S AMPLE R EWEIGHTING AND S CORE U PDATING

Original: Reweighting.
Revised: Reweighting.

Original: To motivate reweighting individual samples, consider an augmentation which is the identity map: f T : z i → { z i } .
Revised: To motivate reweighting individual samples, consider an augmentation which is the identity map: f T : z i → { z i } .

Original: Since we add augmentations back to the training set, our augmentation policy willfor example duplicate selected samples, resulting in a net effect which reweighs samples with twice the original weight.
Revised: Since we add augmentations back to the training set, our augmentation policy will duplicate selected samples, resulting in a net effect which reweights samples with twice the original weight.

Original: Using transformations that result in larger augmentation sets will result in larger weights.
Revised: Using transformations that result in larger augmentation sets will result in larger weights.

Original: One approach is post-processing; Fithian & Hastie (2014), for example, show that the case of class imbalanced sampling can be corrected with a shift of the logistic regressor’s bias.
Revised: One approach is post-processing; Fithian & Hastie (2014), for example, show that the case of class imbalanced sampling can be corrected with a shift of the logistic regressor’s bias.

Original: To normalize for the effect of this implicit reweighing, we divide the weights of the original samples and its augmented samples by the size of that set, | f T ( z i ) | .
Revised: To normalize for the effect of this implicit reweighting, we divide the weights of the original samples and its augmented samples by the size of that set, | f T ( z i ) | .

Original: Under this scheme, we guarantee that we conserve the weight originally assigned to a point (and conserve the ratios of labels).
Revised: Under this scheme, we guarantee that we conserve the weight originally assigned to a point (and conserve the ratios of labels).

Original: More sophisticated polices, such as reweighing samples by a measure of how trustworthy they are (e.g., perhaps some bounds can be derived on the label-preserving properties of an augmentation), remain to be investigated as future work.
Revised: More sophisticated policies, such as reweighting samples by a measure of how trustworthy they are (e.g., perhaps some bounds can be derived on the label-preserving properties of an augmentation), remain to be investigated as future work.

Original: We ﬁnd that in many cases, the performance of reweighting is similar in expectation to the base case, but with lower variance.
Revised: We ﬁnd that in many cases, the performance of reweighting is similar in expectation to the base case, but with lower variance.

Original: However, in some cases, reweighting can in fact have a negative impact, as we discuss in Section 5.2.
Revised: However, in some cases, reweighting can in fact have a negative impact, as we discuss in Section 5.2.

Original: We expect this policy to be more useful in the case of class imbalance, where the duplication of minority class samples may signiﬁcantly alter the distribution over classes.
Revised: We expect this policy to be more useful in the case of class imbalance, where the duplication of minority class samples may signiﬁcantly alter the distribution over classes.

Original: Updating scores.
Revised: Updating scores.

Original: Once we decide to augment a data point, we can either continue to use the same inﬂuence information which we derived for the un-augmented dataset, or we can choose to update it.
Revised: Once we decide to augment a data point, we can either continue to use the same inﬂuence information which we derived for the un-augmented dataset, or we can choose to update it.

Original: The reason for doing this is to account for the drifting model behavior as points are added to the training set and the model is retrained.
Revised: The reason for doing this is to account for the drifting model behavior as points are added to the training set and the model is retrained.

Original: However, if having a single estimate of inﬂuence for the whole lifetime of the model is sufﬁcient, then avoiding repeated inﬂuence calculations will reduce the amount of computation required while also enabling an increased level of parallelism (e.g., minibatching, distributed computations).
Revised: However, if having a single estimate of inﬂuence for the whole lifetime of the model is sufﬁcient, then avoiding repeated inﬂuence calculations will reduce the amount of computation required while also enabling an increased level of parallelism (e.g., minibatching, distributed computations).

Original: We ﬁnd that this modiﬁcation results in similar behavior to that of reweightings, where expected performance of a policy remains similar, but its variance decreases.
Revised: We ﬁnd that this modiﬁcation results in similar behavior to that of reweightings, where expected performance of a policy remains similar, but its variance decreases.

Original: Overall, we haven’t observed a signiﬁcant enough effect to suggest that this technique is justiﬁed given the extra cost it requires.
Revised: Overall, we haven’t observed a signiﬁcant enough effect to suggest that this technique is justiﬁed given the extra cost it requires.

Original: The beneﬁt of this is that it implies that many applications may need to only compute selection metadata once time throughout the augmentation process.
Revised: The beneﬁt of this is that it implies that many applications may need to only compute selection metadata once time throughout the augmentation process.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we provide detailed results on the performance of our proposed policies for data subsampling.
Revised: In this section we provide detailed results on the performance of our proposed policies for data subsampling.

Original: For all experiments, we use a Convolutional Neural Network (CNN) to create bottleneck features, which we then use as input into a linear logistic regression model.
Revised: For all experiments, we use a Convolutional Neural Network (CNN) to create bottleneck features, which we then use as input into a linear logistic regression model.

Original: This is equivalent to freezing the weights of the CNN, or using a  basis function, φ  ( · ) , to transform the inputs (Nasrabadi, 2007), and allows us to quickly calculate training loss and model inﬂuence.
Revised: This is equivalent to freezing the weights of the CNN, or using a set of basis functions, φ i ( · ) , to transform the inputs (Bishop, 2006), and allows us to quickly calculate training loss and model inﬂuence.

Original: We explore the results of our augmentation policies on three datasets:  MNIST, CIFAR, and NORB.
Revised: We explore the results of our augmentation policies on three datasets: binary classiﬁcation variants of MNIST, CIFAR10, and NORB.

Original: For MNIST features, we use a LeNet architecture (LeCun et al., 1998) with ReLu activations, and for CIFAR and NORB, we use ResNet50v2 (He et al., 2016).
Revised: For MNIST features, we use a LeNet architecture (LeCun et al., 1998) with ReLu activations, and for CIFAR10 and NORB, we use ResNet50v2 (He et al., 2016).

Original: While for CIFAR and NORB we generate the bottleneck features once due to cost, for MNIST, we additionally study the effect of re-generating these features as new points are selected and augmented (i.e., training both the features and model from scratch throughout the experiments).
Revised: While for CIFAR10 and NORB we generate the bottleneck features once due to cost, for MNIST, we additionally study the effect of re-generating these features as new points are selected and augmented (i.e., training both the features and model from scratch throughout the experiments).

Original: In terms of augmentations, we consider three examples: translation, rotation, and crop.
Revised: In terms of augmentations, we consider three examples: translation, rotation, and crop.

Original: To control for sources of variance in model performance, all augmentations under consideration are applied exhaustively in a deterministic fashion to any selected samples, and the resulting augmented points are then added back to the training set.
Revised: To control for sources of variance in model performance, all augmentations under consideration are applied exhaustively in a deterministic fashion to any selected samples, and the resulting augmented points are then added back to the training set.

Original: Formally, given a data point, z
Revised: Formally, given a data point, z

Original: = ( x, y ) ∈
Revised: = ( x, y ) ∈

Original: X ×Y , our augmentation is a map from a data point to a ﬁnite set of data points: f T : z → { z 1 , . . .
Revised: X ×Y , our augmentation is a map from a data point to a ﬁnite set of data points: f T : z → { z 1 , . . .

Original: , z n : z i ∈ X ×Y} .
Revised: , z n : z i ∈ X ×Y} .

Original: We controlled for augmentation-induced regularization by performing a simple cross validation sweep for the regularization parameter λ each time the model was re-trained, and we found regularization to have negligible impact in the trends we observed.
Revised: We controlled for augmentation-induced regularization by performing a simple cross validation sweep for the regularization parameter λ each time the model was re-trained, and we found regularization to have negligible impact in the trends we observed.

Original: For all datasets and augmentations, we make the effect of augmentation more apparent by adding augmented test points to the test set.
Revised: For all datasets and augmentations, we make the effect of augmentation more apparent by adding augmented test points to the test set.

Original: For example, in the case of translation, we test the performance of applying translation augmentations to the original training set, and then determine the accuracy using an augmented variant of the test data that has been appended with translated test examples.
Revised: For example, in the case of translation, we test the performance of applying translation augmentations to the original training set, and then determine the accuracy using an augmented variant of the test data that has been appended with translated test examples.

Original: All augmentations are performed using
Revised: All augmentations are performed using

Original: Imgaug 1 , and our code is written in Python using Keras CNN implementations.
Revised: Imgaug 2 , and our code is written in Python using Keras CNN implementations.

Original: Our code will be made publicly available online.
Revised: Our code will be made publicly available online.

Original: Full implementation details are provided in Appendix B.
Revised: Full implementation details are provided in Appendix B.

Original:  G ENERAL P OLICIES : I NFLUENCE AND L OSS
Revised: 5. G ENERAL P OLICIES : I NFLUENCE AND L OSS

Original: In Figure 3, we explore a ﬁrst set of policies in which we randomly sample points for augmentation proportional either to their loss (green) or inﬂuence value (blue).
Revised: In Figure 3, we explore a ﬁrst set of policies in which we randomly sample points for augmentation proportional either to their loss (green) or inﬂuence value (blue).

Original: To calculate the loss and inﬂuence, we incur a one-time cost of training the model on the original dataset.
Revised: To calculate the loss and inﬂuence, we incur a one-time cost of training the model on the original dataset.

Original: As a baseline (red), we compare these methods to a simple strategy in which data points for augmentation are drawn entirely at random (irrespective of loss or inﬂuence).
Revised: As a baseline (red), we compare these methods to a simple strategy in which data points for augmentation are drawn entirely at random (irrespective of loss or inﬂuence).

Original: The red, dotted horizontal line indicates the test accuracy with no augmentation, and the green, dotted line indicates the test accuracy after augmenting the entire training set.
Revised: The red-dotted horizontal line indicates the test accuracy with no augmentation, and the green-dotted line indicates the test accuracy after augmenting the entire training set.

Original: Note that all policies have the same accuracy when the number of points is 0 or k , where k is the number of points in the original training set, which correspond to the un-augmented training set and the fully augmented training set, respectively 2 .
Revised: Note that all policies have the same accuracy when the number of points is 0 or k , where k is the number of points in the original training set, which correspond to the un-augmented training set and the fully augmented training set, respectively 3 .

Original: We observe similar behavior in terms of the deterministic policy, which is provided in Appendix C.
Revised: We observe similar behavior in terms of the deterministic policy, which is provided in Appendix C.

Original: Across the datasets and transformation types, we notice several trends.
Revised: Across the datasets and transformation types, we notice several trends.

Original: First, the policies based on loss and inﬂuence consistently outperform the random baseline.
Revised: First, the policies based on loss and inﬂuence consistently outperform the random baseline.

Original: This is particularly true for the rotation augmentation for all three datasets, where the random-inﬂuence and random-loss policies achieve the full augmentation accuracy with only 5-10% of the data, compared to 90-100% of the data for random sampling.
Revised: This is particularly true for the rotation augmentation for all three datasets, where the random-inﬂuence and random-loss policies achieve the full augmentation accuracy with only 5–10% of the data, compared to 90–100% of the data for random sampling.

Original: Second, we note that the policies based on inﬂuence vs. loss behave very similarly.
Revised: Second, we note that the policies based on inﬂuence vs. loss behave very similarly.

Original: While inﬂuence has slightly better performance (particularly on the NORB dataset), the policies are for the most part equivalent.
Revised: While inﬂuence has slightly better performance (particularly on the NORB dataset), the policies are for the most part equivalent.

Original: A beneﬁt of this is that the loss calculation is slightly simpler than inﬂuence to calculate, as it does not require calculating the inverse Hessian component,
Revised: A beneﬁt of this is that the loss calculation is slightly simpler than inﬂuence to calculate, as it does not require calculating the inverse Hessian component,

Original: H − 1 ˆ θ , as described in 1.
Revised: H − 1 ˆ θ , as described in 1.

Original: Third, we note that it is possible to achieve higher accuracy than full augmentation using only a reduced set of points for augmentation, as observed in several of the  plots (NORB, CIFAR-rotate, MNIST-rotate).
Revised: Third, we note that it is possible to achieve higher accuracy than full augmentation using only a reduced set of points for augmentation, as observed in several of the [Figure] plots (most notably on NORB).

Original: We believe that this higher performance may be due to reduced noise in the dataset as compared to full augmentation.[Figure] s t A
Revised: We believe that this higher performance may be due to reduced noise in the dataset as compared to full augmentation.

Original: [Figure] u r a c y [Figure] Finally, we additionally explore the effect of using support vectors for augmentation, which was proposed in the Virtual Support Vector literature (Burges & Sch¨olkopf, 1997; Decoste & Sch¨olkopf, 2002).
Revised: Finally, we additionally explore the effect of using support vectors for augmentation, which was proposed in the Virtual Support Vector literature (Burges & Sch ¨ olkopf, 1997; Decoste & Sch ¨ olkopf, 2002).

Original: In particular, we ﬁnd VSV points by tuning a linear SVM on the bottleneck features of the original training set, and then using these points as the set of augmentation points for the logistic regression model with bottleneck features.
Revised: In particular, we ﬁnd VSV points by tuning a linear SVM on the bottleneck features of the original training set, and then using these points as the set of augmentation points for the logistic regression model with bottleneck features.

Original: We use search over C ∈ { 0 . 01 , 0 . 1 , 1 , 10 , 100 } via crossvalidation, and the best resulting model is used to obtain support vectors.
Revised: We use search over C ∈ { 0 . 01 , 0 . 1 , 1 , 10 , 100 } via cross-validation, and the best resulting model is used to obtain support vectors.

Original: Interestingly, we note that, though this transfer approach was not originally proposed in the VSV literature, it results in strong performance on a few of our tests (e.g., NORB-translate, NORB-crop, CIFAR-rotate).
Revised: Interestingly, we note that, though this transfer approach was not originally proposed in the VSV literature, it results in strong performance on a few of our tests (e.g., NORB-translate, NORB-crop, CIFAR10-rotate).

Original: However, the approach is not as reliable as the proposed policies in terms of ﬁnding the optimal subset of points for transformation (performing signiﬁcantly below optimal, e.g., for MNIST-crop, MNIST-rotate, and CIFAR-translate), and the major limitation is that the augmentation set size is ﬁxed to exactly equal the number of support vectors, which is more brittle than the proposed policies, which can vary depending on the desired data budget.
Revised: However, the approach is not as reliable as the proposed policies in terms of ﬁnding the optimal subset of points for transformation (performing signiﬁcantly below optimal, e.g., for MNIST and CIFAR10-translate), and the major limitation is that the augmentation set size is ﬁxed to exactly equal the number of support vectors, which is more brittle than the proposed policies, which can vary depending on the desired data budget.

Original: 5. R EFINEMENTS : S AMPLE R EWEIGHTING AND S CORE U PDATING
Revised: 5. R EFINEMENTS : S AMPLE R EWEIGHTING AND S CORE U PDATING

Original: We additionally explore the effect of two reﬁnements on the initial policies: reweighting the samples as they are added back to the training set, and updating the scores as the augmentation proceeds, as described in Section 4.2.
Revised: We additionally explore the effect of two reﬁnements on the initial policies: reweighting the samples as they are added back to the training set, and updating the scores as the augmentation proceeds, as described in Section 4.2.

Original: This latter policy assumes that the method is run in an online fashion, in contrast to the policies described thus far.
Revised: This latter policy assumes that the method is run in an online fashion, in contrast to the policies described thus far.

Original: This add extra expense to the total run time, as the model must be continually updated as new points are augmented.
Revised: This adds extra expense to the total run time, as the model must be continually updated as new points are augmented.

Original: In Figure 4, we observe the effect of these modiﬁcations for all datasets using the rotation augmentation, and using model inﬂuence as the score.
Revised: In Figure 4, we observe the effect of these modiﬁcations for all datasets using the rotation augmentation, and using model inﬂuence as the score.

Original: Full results are provided in Appendix C. Interestingly, while reweighting points seems to have a positive (if negligible) effect for MNIST, we see that it can actually hurt performance in CIFAR and NORB.
Revised: Full results are provided in Appendix C. Interestingly, while reweighting points seems to have a positive (if negligible) effect for MNIST, we see that it can actually hurt performance in CIFAR10 and NORB.

Original: This may indicate that the amplifying effect of augmentation may in fact be beneﬁcial when training the model.
Revised: This may indicate that the amplifying effect of augmentation may in fact be beneﬁcial when training the model.

Original: In terms of the score updating, we see that although updating the score can have a slight positive impact (e [Figure] g., for NORB-rotate), the performance appears to roughly match that of the original policy.
Revised: In terms of the score updating, we see that although updating the score can have a slight positive impact (e.g., for NORB-rotate), the performance appears to roughly match that of the original policy.

Original: Given the extra expense required in model updating, we therefore conclude that the simpler policies are preferable
Revised: Given the extra expense required in model updating, we therefore conclude that the simpler policies are preferable

Original: [Figure]
Revised: [Figure]

Original: 5. U NDERSTANDING P OLICIES
Revised: 5. U NDERSTANDING P OLICIES

Original: Finally, to give insight into the behavior of the proposed polices, we examine the 10 points with highest inﬂuence/loss vs. least inﬂuence/loss, for MNIST.
Revised: Finally, to give insight into the behavior of the proposed polices, we examine thepoints with highest inﬂuence/loss vs. least inﬂuence/loss, for MNIST.

Original: We observe similar results for the other datasets (CIFAR, NORB); additional results are provided in Appendix E.
Revised: We observe similar results for the other datasets (CIFAR10, NORB); additional results are provided in Appendix E.

Original: These examples help to visualize the beneﬁts of downsampling, as it is clear that the bottom set of points are all quite similar.
Revised: These examples help to visualize the beneﬁts of downsampling, as it is clear that the bottom set of points are all quite similar.

Original: The top points, in contrast, appear more diverse—both in terms of class label as well as features (thin lines, thick lines, slanted, straight, etc).
Revised: The top points, in contrast, appear more diverse— both in terms of class label as well as features (thin lines, thick lines, slanted, straight, etc.).

Original: We postulate that promoting this diversity and removing redundancy is key in learning invariances through augmentation more efﬁciently.
Revised: We postulate that promoting this diversity and removing redundancy is key in learning invariances through augmentation more efﬁciently.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we have demonstrated that not all training points are equally useful for augmentation, and we have proposed simple policies that can select the most viable subset of points.
Revised: In this paper, we demonstrate that not all training points are equally useful for augmentation, and we propose simple policies that can select the most viable subset of points.

Original: Our policies, based off of notions of training loss and model inﬂuence, are widely applicable to general machine learning models.
Revised: Our policies, based on notions of training loss and model inﬂuence, are widely applicable to general machine learning models.

Original: Obtaining access to an augmentation score vector can be obtained in only one training cycle on the original data (e.g., a ﬁxed cost), yet the potential improvements in augmented training can scale superlinearly with respect to the original dataset size.
Revised: Obtaining access to an augmentation score vector can be obtained in only one training cycle on the original data (e.g., a ﬁxed cost), yet the potential improvements in augmented training can scale superlinearly with respect to the original dataset size.

Original: With many fewer data points to augment, the augmentations themselves can be applied in a more efﬁcient manner in terms of compute and expert oversight.
Revised: With many fewer data points to augment, the augmentations themselves can be applied in a more efﬁcient manner in terms of compute and expert oversight.

Original: At an extreme, they can be specialized on a per-example basis.
Revised: At an extreme, they can be specialized on a per-example basis.

Original: A natural area of future work is to explore subset selection policies that take the entire subset into account, rather than the greedy policies described.
Revised: A natural area of future work is to explore subset selection policies that take the entire subset into account, rather than the greedy policies described.

Original: For example, even if two samples may independently have large leave-one-out inﬂuence, it may be the case that these points inﬂuence each other and leave-one-out inﬂuence may be an overestimate (e.g., consider the case of two identical samples).
Revised: For example, even if two samples may independently have large leave-one-out inﬂuence, it may be the case that these points inﬂuence each other and leave-one-out inﬂuence may be an overestimate (e.g., consider the case of two identical samples).

Original: Including second-order information or encouraging subset diversity may therefore help to improve performance even further.
Revised: Including second-order information or encouraging subset diversity may therefore help to improve performance even further.

Original: A A DDITIONAL P LOTS : M ETRICS
Revised: A A DDITIONAL P LOTS : M ETRICS

Original: Here we provide histogram plots for loss and inﬂuence for all datasets.
Revised: Here we provide histogram plots for loss and inﬂuence for all datasets.

Original: The key take-away from these results is that the distribution of these metrics indicate that most points have low loss and inﬂuence, and thus (according to our policies) can be augmented with low probability. [Figure] [Figure] [Figure]
Revised: The key take-away from these results is that the distribution of these metrics indicate that most points have low loss and inﬂuence, and thus (according to our policies) can be augmented with low probability

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: B E XPERIMENT D ETAILS
Revised: B E XPERIMENT D ETAILS

Original: Here we provide full implementation details on our experiments throughout the paper.
Revised: Here we provide full implementation details on our experiments throughout the paper.

Original: Setup.
Revised: Setup.

Original: There are a few key architectural ideas in our tests: the data, the augmentations, the selection policy, a featurization preprocessing component, and a logistic regression model.
Revised: There are a few key architectural ideas in our tests: the data, the augmentations, the selection policy, a featurization preprocessing component, and a logistic regression model.

Original: Our implementation is in Python.
Revised: Our implementation is in Python.

Original: The dataset is loaded (possibly via third party libraries) into a numpy array.
Revised: The dataset is loaded (possibly via third party libraries) into a NumPy array.

Original: We can then run this dataset through a trained CNN model, such as ResNet50, to obtain a feature vector.
Revised: We can then run this dataset through a trained CNN model, such as ResNet50, to obtain a feature vector.

Original: The logistic regression model is then trained on this resulting “featurized” dataset and tested on a “featurized” test set.
Revised: The logistic regression model is then trained on this resulting “featurized” dataset and tested on a “featurized” test set.

Original: Once training is complete, both loss and inﬂuence can then be measured for each training point, and can therefore be used as scores.
Revised: Once training is complete, both loss and inﬂuence can then be measured for each training point, and can therefore be used as scores.

Original: Augmentations are then applied exhaustively to the test set.
Revised: Augmentations are then applied exhaustively to the test set.

Original: We refer to this test set as “poisoned”.
Revised: We refer to this test set as “poisoned”.

Original: The test distribution has changed, and therefore a gap has formed between the original test performance and the “poisoned” test performance.
Revised: The test distribution has changed, and therefore a gap has formed between the original test performance and the “poisoned” test performance.

Original: We attempt to close this gap by applying augmentations to the training set.
Revised: We attempt to close this gap by applying augmentations to the training set.

Original: We proceed by initializing a set with the unaugmented training set.
Revised: We proceed by initializing a set with the unaugmented training set.

Original: We augment points in rounds, and the unaugmented training set corresponds to round 0 .
Revised: We augment points in rounds, and the unaugmented training set corresponds to round 0 .

Original: Every round, our policy is given a vector of scores and it selects a point to augment.
Revised: Every round, our policy is given a vector of scores and it selects a point to augment.

Original: This point is featurized and added to the set.
Revised: This point is featurized and added to the set.

Revised: If score updates are enabled and the current round is sampled for testing, the model scores are re-calculated for the original data points.

Original: The CNN can be optionally retrained, but the logistic regression model must be retrained to obtain the current test accuracy. 
Revised: The CNN can be optionally retrained, but the logistic regression model must be retrained to obtain the current test accuracy. Each stochastic policy is tested 5 times.

Revised: Plots show 95% conﬁdence intervals and ﬁx C = 10 for the logistic regression hyperparameter.

Original: Implementation.
Revised: Implementation.

Original: We perform experiments in Python, using Keras (Chollet et al., 2015), ScikitLearn (Pedregosa et al., 2011; Buitinck et al., 2013), AutoGrad (Maclaurin et al.), and Imgaug 3 .
Revised: We perform experiments in Python, using Keras (Chollet et al., 2015), Tensorﬂow (Abadi et al., 2016), Scikit-Learn (Pedregosa et al., 2011; Buitinck et al., 2013), AutoGrad (Maclaurin et al.), and Imgaug 4 .

Original: We wrap Keras implementations of the CNNs in Scikit-Learn transformers, and we create new classes utilizing Scikit-Learn classiﬁers and their corresponding inﬂuence functions calculated with the autograd system.
Revised: We wrap Keras implementations of the CNNs in Scikit-Learn transformers, and we create new classes utilizing Scikit-Learn classiﬁers and their corresponding inﬂuence functions calculated with the autograd system.

Original: This allows us to decouple input data, bottleneck features, and the ﬁnal classiﬁer that calculates inﬂuence.
Revised: This allows us to decouple input data, bottleneck features, and the ﬁnal classiﬁer that calculates inﬂuence.

Original: It also allows us to perform any additional (i.e., cross validation) tuning rather easily.
Revised: It also allows us to perform any additional (i.e., cross validation) tuning rather easily.

Original: Augmentations are performed by Imgaug.
Revised: Augmentations are performed by Imgaug.

Original: Our code will be made publicly available online.
Revised: Our code will be made publicly available online.

Original: Models.
Revised: Models.

Original: For all experiments, we use a CNN to create bottleneck features, which we then use as input into a linear logistic regression model.
Revised: For all experiments, we use a CNN to create bottleneck features, which we then use as input into a linear logistic regression model.

Original: This is equivalent to freezing the weights of the CNN, or using a  basis function, φ  ( · ) , to transform the inputs (Nasrabadi, 2007).
Revised: This is equivalent to freezing the weights of the CNN, or using a set of basis functions, φ i ( · ) , to transform the inputs (Bishop, 2006).

Original: A LeNet archicture (LeCun et al., 1998) with ReLu activations was used for MNIST; however, this model had issues performing well on the augmented sets for CIFAR10 and NORB.
Revised: A LeNet architecture (LeCun et al., 1998) with ReLu activations was used for MNIST; however, this model had issues performing well on the augmented sets for CIFAR10 and NORB.

Original: We had also tried a larger model from the Keras examples on MNIST, which resulted in similar performance to using LeNet 4 .
Revised: We had also tried a larger model from the Keras examples on MNIST, which resulted in similar performance to using LeNet 5 .

Original: Both LeNet and the Keras net were fast to train, so we retrained the models for 40 − 50 epochs with ADAM (Kingma & Ba, 2014) and a minibatch size of 512 , which was enough to obtain convergence.
Revised: Both LeNet and the Keras net were fast to train, so we retrained the models for 40 − 50 epochs with ADAM (Kingma & Ba, 2014) and a minibatch size of 512 , which was enough to obtain convergence.

Original: We used a ResNet50v2 model (He et al., 2016) model trained on the CIFAR10 dataset for the CIFAR tests, and we obtained good performance without using augmentations in the training process.
Revised: We used a ResNet50v2 model (He et al., 2016) model trained on the CIFAR10 dataset for the CIFAR tests, and we obtained good performance without using augmentations in the training process.

Original: Using a pretrained ImageNet ResNet50 model resulted in poor performance (both computationally and in accuracy).
Revised: Using a pretrained ImageNet ResNet50 model resulted in poor performance (both computationally and in accuracy).

Original: For NORB, we were able to get good performance on the translate task without any training augmentations being applied on the NORB dataset.
Revised: For NORB, we were able to get good performance on the translate task without any training augmentations being applied on the NORB dataset.

Original: However, the other augmentations resulted in high prediction degradation, so the ResNet model was retrained with random rotations, shifts, and ﬂips applied to images.
Revised: However, the other augmentations resulted in high prediction degradation, so the ResNet model was retrained with random rotations, shifts, and ﬂips applied to images.

Original: All ResNet models were frozen after the initial training. 
Revised: All ResNet models were frozen after the initial training. Datasets.

Revised: We convert each of the datasets into a binary classiﬁcation task.

Revised: MNIST is 3 vs. 8 ,

Revised: CIFAR10 is airplane vs automobile

Original: Augmentations.
Revised: Augmentations.

Original: Our tests use translate, rotate, and crop.
Revised: Our tests use translate, rotate, and crop.

Original: Each of these augmentations is applied over a range of parameters, which results in multiple augmented images.
Revised: Each of these augmentations is applied over a range of parameters, which results in multiple augmented images.

Original: Translate is applied forpixels in all cardinal directions (e.g., up, down, left, and right) on MNIST and CIFAR10, and 6 pixels for NORB (note: this pixel difference is to account for NORB images being 3 times larger than CIFAR10).
Revised: Translate is applied forpixels in all cardinal directions (e.g., up, down, left, and right) on MNIST, 3 pixels for CIFAR10, and 6 pixels for NORB (note: this pixel difference is to account for NORB images being 3 times larger than CIFAR10).

Original: Rotate is appliedbetween for the 15 ( 14 after removing identity transform) rotations evenly spaced between ± 30 ◦  .
Revised: Rotate is applied for the 15 ( 14 after removing identity transform) rotations evenly spaced between ± 30 ◦ for MNIST. CIFAR10 and NORB use ± 5 ◦ , ± 2 . 5 ◦ .

Original: Crop is applied excluding the outer [1 , 2 , . . .
Revised: For MNIST, crop is applied excluding the outer [1 , 2 , . . .

Original: , 6] pixels on all 4 image sides, and zoom is applied to rescale the resulting image back to its original dimensions.
Revised: , 6] pixels on all 4 image sides, and zoom is applied to rescale the resulting image back to its original dimensions.

Revised: CIFAR10 and NORB exclude the outer 2 pixels.

Original: Usually, augmentations are constructed to preserve labels, but it is possible in principle to construct augmentations that utilize label information for the augmentation itself or perhaps induce a change in label (e.g., an image dataset with segmentation information can segment out all non-background classes to change the label of an image to background).
Revised: Usually, augmentations are constructed to preserve labels, but it is possible in principle to construct augmentations that utilize label information for the augmentation itself or perhaps induce a change in label (e.g., an image dataset with segmentation information can segment out all non-background classes to change the label of an image to background).

Original: Such augmentations are expensive, require domain expertise, and are hard to validate, but they may be viable if the number of total augmentations can be controlled.
Revised: Such augmentations are expensive, require domain expertise, and are hard to validate, but they may be viable if the number of total augmentations can be controlled.

Original: C A DDITIONAL P LOTS : P OLICIES
Revised: C A DDITIONAL P LOTS : P OLICIES

Original: Below we provide full experiments for the randomized (Figure 9) and deterministic (Figure 10) policies using model inﬂuence as the scoring metric, across all datasets and transformations
Revised: Below we provide full experiments for the randomized (Figure 9) and deterministic (Figure 10) policies using model inﬂuence as the scoring metric, across all datasets and transformations. [Figure]

Original: [Figure]
Revised: [Figure]

Original:  r a c y
Revised: T e s t A cc u r a c y

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: D F ULL R ESULTS : P OLICIES
Revised: D F ULL R ESULTS : P OLICIES

Original: We provide below AUC results from the plots provided in Appendix C. The AUC provides a more clear metric by which to determine the best performing policy.
Revised: Below we provide Area Under the Curve (AUC) results from the plots provided in Appendix C as well as the corresponding results for loss. Compared to the plots, the AUC provides a single metric by which to determine the best performing policy.

Original: We separate tables across augmentation (translate/rotate/crop) and score (loss/inﬂuence)
Revised: We separate tables across augmentation (translate/rotate/crop) and score function (loss/inﬂuence). In the tables below

Original: [Figure]
Revised: [Figure]

Revised: we make the following abbreviations corresponding to the policies described in

Revised: Section 4: Det (Deterministic), Prop (Proportional), Down (Downweight), Rand (Random).

Original: D. AUC R ESULTSUSING L OSS
Revised: D. A REA U NDER C URVE (AUC) R ESULTS

Original: [Figure]
Revised: [Figure]

Revised: L

Original: [Figure]  [Figure]
Revised: [Figure] [Table] [Figure]

Original: [Table]  [Table]
Revised: [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: D. AUC R ESULTS USING I NFLUENCE
Revised: D. A REA U NDER C URVE (AUC) R ESULTS USING I NFLUENCE

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Table] [Table] [Table] [Table]

Revised: F SVM M ARGIN AS A S CORE

Revised: Having used the VSV method

Revised: [Figure]

Revised: we can expand on this idea by using an SVM’s margin to score points.

Revised: The idea is that points further away from the margin may also be less important to the logistic regression model.

Revised: However, it is worth noting that the SVM and logistic regression decision surfaces may vary signiﬁcantly

Revised: [Figure]

Revised: and the solution of each model is dependent on a variety of hyperparameters.

Revised: We include results utilizing the absolute value of the margin as well as the inverse of that value.

Revised: We ﬁnd that the model mismatch (i

Revised: [Figure]

Revised: e., SVM vs. logistic regression) combined with a different score (i

Revised: [Figure]

Original: [Table]
Revised: e., margin vs. loss or inﬂuence) results in uniformly worse performance than our proposed inﬂuencebased approach. However, it is worth noting that there is some transfer between models [Figure] which poses potentially interesting questions for future investigation

Original: [Figure]
Revised: [Figure]

Revised: G E FFECTS ON T RAINING P ERFORMANCE

Original: [Table]
Revised: In the context of deep learning, we expect performance to scale linearly with an increased dataset. To highlight this effect [Figure] we train a ResNetv2 network (He et al., 2016) using Tensorﬂow (Abadi et al., 2016) version 1.10.1 with a variable number of training examples obtained from CIFAR10.

Revised: The system which was used for the test has an Intel i7-6700k and an Nvidia GTX 1080 using CUDA 9.2 and CuDNN 7.2.1.

Revised: We show the resulting scaling performance in Figure 15.

Revised: We can see that the scaling is indeed linear (p-value is almost 0).

Revised: If we assume that the number of epochs is ﬁxed, then we can conclude subsampling would result in a linear decrease in time to convergence.

Revised: We observe decreases in training time in our MNIST experiments, where we continously

Original: [Table]
Revised: [Figure]

Revised: retrain our feature-extraction model with variable amounts of augmented data.

Revised: In a more complicated training regime, such as distributed training, we can expect this improvement to be greater.

Original: [Table]
Revised: Although it is true that the initial model training may reduce some of these performance beneﬁts, it’s worth noting that a pretrained model may be used to bootstrap a different model and the cost of the initial training can be amortized over many experiments. For the CIFAR10 translate task, we tried using the SVM support vectors to select points to augment for a ResNetv2 model. Our training used a fraction of the baseline method’s total training data (1296/5000), but achieved within 5% of the test accuracy. We observed similar trends on the other tasks we tried.

Revised: H S TRATIFIED S AMPLING WITH C LUSTERING

Original: [Table]
Revised: Here we present results of performing augmentation with cluster-based stratiﬁed sampling. One motivation of this method is to see how enforced diversity helps the augmentation process, since points with high inﬂuence or loss seem diverse. The subgroups are derived by using k-means clustering, where we ﬁx the number of clusters to be equal to the sample size. We explore two policies: one where a point is selected at random from each cluster (Baseline Clustered), and another where the point is selected based proportional to its inﬂuence value (Random Proportional Inﬂuence Clustered).

Revised: The baseline using clustering outperforms the standard (uniform random sampling) baseline in many tasks.

Revised: We speculate that this is due to the forced increase in diversity.

Revised: However, the standard inﬂuence policy outperforms the clustered baseline policy for all datasets and augmentations, and also outperforms the combined strategy (Random Proportional Inﬂuence Clustered) in most tasks.

Original: [Table]
Revised: This suggests that stratiﬁed sampling can improve over simple random sampling. However, other polices, such as weighting points by inﬂuence/loss, seem to provide greater beneﬁts for augmentation, especially when point quality is aggressively traded off with point diversity. Indeed, as we increase the number of clusters, the combined policies approach random sampling. Selecting the number of clusters also poses practical issues for this sort of approach.

Original: [Table]
Revised: One method of subsampling a dataset is with a deterministic point process (DPP), which is explained further in (Kulesza et al., 2012). A deterministic point process allows for tractable subset selection with both diversity and quality criterion. We use a DPP to select points to augment mirroring the methodology used throughout this paper.

Revised: One exception is the other explored methods are amenable to greedy extensions in the number of points augmented, while the DPP algorithm was re-run for every change in the number of augmented points.

Revised: It is worth noting that our experiment setup is not the intended use of a DPP, as we ﬁx the original training set and concatenate the selected augmented points.

Revised: In contrast, a DPP would typically be used to subsample the original training set.

Revised: For our experiments, we use a publicly available implementation 6 .

Original: [Figure]
Revised: [Figure]

Revised: For our tests, we use a k -DPP, which is a DPP conditioned on the number of points selected being equal to k .

Original: [Table]
Revised: We summarize the notation given in (Kulesza et al., 2012) for our construction of the DPP kernel, L . We set L ij = q i φ (cid:62) i φ j q j 7 , where q i ∈ R + is a quality term and

Original: [Table]
Revised: φ i ∈ R D is a diversity term. In our tests, we use inﬂuence for q i and bottleneck features for φ i . We normalized each sample with || φ

Original: [Table]{+i || 
 = 1 as was suggested in (Kulesza et al., 2012) to make all samples equally likely without taking diversity into account. The DPP results are shown in Figure 17. “Inﬂuence Bottleneck DPP” corresponds to using both inﬂuence and bottleneck features in the kernel.+}
Revised: {+i || 
 = 1 as was suggested in (Kulesza et al., 2012) to make all samples equally likely without taking diversity into account. The DPP results are shown in Figure 17. “Inﬂuence Bottleneck DPP” corresponds to using both inﬂuence and bottleneck features in the kernel.+}

Original: [Table]
Revised: “Bottleneck DPP” corresponds to using only bottleneck features and therefore has no idea of the quality of a point. For the DPP experiments, we ran up to 250 points augmentations and re-ran the experiments 5 times. The other results included in the plot are from previous experiments. As can be seen, the inﬂuence weighted DPP performance is competitive with the inﬂuence driven approach.

Original: E D IAGNOSING I NFLUENCE AND L OSS
Revised: Using solely bottleneck features (i.e., q i = 1 ) for L resulted in poor performance. The reason we chose to only test up to 250 augmented points is because sampling a DPP takes O ( Nk 3 ) , where N is the size of the full set used in the subset selection. We found this computational performance to be limiting in practice, as N = 1000 and k approached 1000 . For low k , it may make sense to use DPP methods, but for larger k , alternative approaches or approximations may be required due to computational budgets.

Revised: [Figure]