Original: A Data-Augmentation Is Worth A Thousand Samples
Revised: A Data-Augmentation Is Worth A Thousand Samples

Original: Author(s)
Revised: Author(s)

Original: Abstract
Revised: Abstract

Original: Data-Augmentation (DA) is known to improve performance across tasks anddatasets.
Revised: Data-Augmentation (DA) is known to improve performance across tasks anddatasets.

Original: We propose a method to theoretically analyze the effect of DA and studyquestions such as: how many augmented samples are needed to correctly estimatethe information encoded by that DA?
Revised: We propose a method to theoretically analyze the effect of DA and studyquestions such as: how many augmented samples are needed to correctly estimatethe information encoded by that DA?

Original: How does the augmentation policy impactthe ﬁnal parameters of a model?
Revised: How does the augmentation policy impactthe ﬁnal parameters of a model?

Original: We derive several quantities in close-form, suchas the expectation and variance of an image, loss, and model’s output under a given
Revised: We derive several quantities in close-form, suchas the expectation and variance of an image, loss, and model’s output under a given

Original: DA distribution.
Revised: DA distribution.

Original: Up to our knowledge, we obtain the ﬁrst explicit regularizer thatcorresponds to using DA during training for non-trivial transformations such asafﬁne transformations, color jittering, or Gaussian blur.
Revised: Up to our knowledge, we obtain the ﬁrst explicit regularizer thatcorresponds to using DA during training for non-trivial transformations such asafﬁne transformations, color jittering, or Gaussian blur.

Original: Those derivations opennew avenues to quantify the beneﬁts and limitations of DA.
Revised: Those derivations opennew avenues to quantify the beneﬁts and limitations of DA.

Original: For example, we showthat common DAs require tens of thousands of samples for the loss at hand to becorrectly estimated and for the model training to converge.
Revised: For example, we showthat common DAs require tens of thousands of samples for the loss at hand to becorrectly estimated and for the model training to converge.

Original: We show that for atraining loss to be stable under DA sampling, the model’s saliency map (gradient ofthe loss with respect to the model’s input) must align with the smallest eigenvectorof the sample variance under the considered DA augmentation, hinting at a possibleexplanation on why models tend to shift their focus from edges to textures.
Revised: We show that for atraining loss to be stable under DA sampling, the model’s saliency map (gradient ofthe loss with respect to the model’s input) must align with the smallest eigenvectorof the sample variance under the considered DA augmentation, hinting at a possibleexplanation on why models tend to shift their focus from edges to textures.

Original:  Introduction
Revised:  Introduction

Original: Data Augmentation (DA) is a prevalent technique in training deep learning models [8, 18].
Revised: Data Augmentation (DA) is a prevalent technique in training deep learning models [8, 18].

Original: These
Revised: These

Original: Deep Networks (DNs) models f γ , governed by some parameters γ ∈ Γ , are trained on the train setand expected to generalize to unseen samples (test set).
Revised: Deep Networks (DNs) models f γ , governed by some parameters γ ∈ Γ , are trained on the train setand expected to generalize to unseen samples (test set).

Original: To combat the tendency of DNs to overﬁt[48], producing a large train-test performance gap, regularization and in particular DA is heavilyemployed, amongst other mechanisms such as weight-decay [28, 21].
Revised: To combat the tendency of DNs to overﬁt[48], producing a large train-test performance gap, regularization and in particular DA is heavilyemployed, amongst other mechanisms such as weight-decay [28, 21].

Original: The beneﬁt of DA over thosealternatives is that deﬁning input transformations that preserve the semantics of their inputs is arelatively simple task, at least in computer vision or acoustic processing [17, 5, 35].
Revised: The beneﬁt of DA over thosealternatives is that deﬁning input transformations that preserve the semantics of their inputs is arelatively simple task, at least in computer vision or acoustic processing [17, 5, 35].

Original: Furthermore, ifthe DA is well designed and rich enough, it can effectively bring the number of training samples closeto the theoretical limits ensuring that any performance gap between the train and test set vanishes[42] as empirically observed in various scenarios [30, 39].
Revised: Furthermore, ifthe DA is well designed and rich enough, it can effectively bring the number of training samples closeto the theoretical limits ensuring that any performance gap between the train and test set vanishes[42] as empirically observed in various scenarios [30, 39].

Original: In fact, DA has proved so useful thatnovel training methods such as self-supervised learning (SSL) entirely rely on DA [24, 47] to learn ameaningful data representation.
Revised: In fact, DA has proved so useful thatnovel training methods such as self-supervised learning (SSL) entirely rely on DA [24, 47] to learn ameaningful data representation.

Original: Despite its empirical effectiveness, our understanding of DA has many open questions, three of whichwe propose to study: (a) how do different DAs impact the model’s parameters during training?;(b) how sample-efﬁcient is the DA sampling, i.e., how many DA samples a model must observeto converge?; and (c) how sensitive is a loss/model to the DA sampling and how this varianceevolves during training as a function of the model’s ability to minimize the loss at hand, and as afunction of the model’s parameters?
Revised: Despite its empirical effectiveness, our understanding of DA has many open questions, three of whichwe propose to study: (a) how do different DAs impact the model’s parameters during training?;(b) how sample-efﬁcient is the DA sampling, i.e., how many DA samples a model must observeto converge?; and (c) how sensitive is a loss/model to the DA sampling and how this varianceevolves during training as a function of the model’s ability to minimize the loss at hand, and as afunction of the model’s parameters?

Original: Our goal is to analytically derive some preliminary answers andinsights into those three areas enabled by a novel image transformation operator that we introducein Section 3.2 coined Data-Space Transform (DST).
Revised: Our goal is to analytically derive some preliminary answers andinsights into those three areas enabled by a novel image transformation operator that we introducein Section 3.2 coined Data-Space Transform (DST).

Original: DST employs a mathematical formulation ofinput transformations akin to the one of Chen et al.
Revised: DST employs a mathematical formulation ofinput transformations akin to the one of Chen et al.

Original: [4] although slightly more general as we do not original image
Revised: [4] although slightly more general as we do not original image

Original: I
Revised: I

Original: [Figure]
Revised: [Figure]

Original: transformed image
Revised: transformed image

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: im require the DA to form a group.
Revised: im require the DA to form a group.

Original: We study the zoom/translation/shearing/rotation DAs in the maintext and Gaussian blur, grayscale, color jitter and random crop in Appendix C.
Revised: We study the zoom/translation/shearing/rotation DAs in the maintext and Gaussian blur, grayscale, color jitter and random crop in Appendix D.

Original: We summarize ourcontributions below:
Revised: We summarize ourcontributions below:

Original: (a) we derive the analytical ﬁrst order moments of augmented samples, and of the losses employingaugmented samples (Section 3.3), effectively providing us with the explicit regularizer inducedby each DA (Section 3.4)(b) we quantify the number of DA samples that are required for a model/loss to obtain a correctestimate of the information conveyed by that DA (Section 4.1)(c)
Revised: (a) we derive the analytical ﬁrst order moments of augmented samples, and of the losses employingaugmented samples (Section 3.3), effectively providing us with the explicit regularizer inducedby each DA (Section 3.4)(b) we quantify the number of DA samples that are required for a model/loss to obtain a correctestimate of the information conveyed by that DA (Section 4.1)(c)

Original: we derive the sensitivity,
Revised: we derive the sensitivity,

Original: i
Revised: i

Original: e. variance, of a given loss and model under a DA policy (Section 4.2)leading us to rediscover from ﬁrst principles a popular deep network regularizer: TangentProp,as being the natural regularization to employ to minimize the loss variance (Section 4.3).
Revised: e. variance, of a given loss and model under a DA policy (Section 4.2)leading us to rediscover from ﬁrst principles a popular deep network regularizer: TangentProp,as being the natural regularization to employ to minimize the loss variance (Section 4.3).

Original: Upshot of gained insights: (a) shows us that the explicit DA regularizer corresponds to a generalized
Revised: Upshot of gained insights: (a) shows us that the explicit DA regularizer corresponds to a generalized

Original: Tikhonov regularizer that depends on each sample’s covariance matrix largest eigenvectors; the kernelspace of the model’s Jacobian matrix aligns with the data manifold tangent space as modeled by the
Revised: Tikhonov regularizer that depends on each sample’s covariance matrix largest eigenvectors; the kernelspace of the model’s Jacobian matrix aligns with the data manifold tangent space as modeled by the

Original: DA (b) shows us that the number of augmented samples required for a loss/model to correctly estimatethe information provided by a DA of a single sample is on the order of 10 4 .
Revised: DA (b) shows us that the number of augmented samples required for a loss/model to correctly estimatethe information provided by a DA of a single sample is on the order of 10 4 .

Original: Even when combining theinformation of thousands of samples, we ﬁnd that the entire train set must be augmented at least 50 ×for a DA policy to be correctly learned by a model (c) quantiﬁes the loss variance as a simple functionof the model’s Jacobian matrix, and the eigenvectors of the augmented sample variance matrix.
Revised: Even when combining theinformation of thousands of samples, we ﬁnd that the entire train set must be augmented at least 50 ×for a DA policy to be correctly learned by a model (c) quantiﬁes the loss variance as a simple functionof the model’s Jacobian matrix, and the eigenvectors of the augmented sample variance matrix.

Original: Regardless of the model or task-at-hand, the loss sensitivity to random DA sampling goes down asthe kernel of a model’s Jacobian matrix aligns with the principal directions of the data manifoldtangent space; as a by-product, we recover TangentProp as the natural regularizer minimizing the losssensitivity to DA sampling.
Revised: Regardless of the model or task-at-hand, the loss sensitivity to random DA sampling goes down asthe kernel of a model’s Jacobian matrix aligns with the principal directions of the data manifoldtangent space; as a by-product, we recover TangentProp as the natural regularizer minimizing the losssensitivity to DA sampling.

Original: All of our image-space results e.g. expectation and variance of images under DA are exact, noapproximation is employed.
Revised: All of our image-space results e.g. expectation and variance of images under DA are exact, noapproximation is employed.

Original: Results on a model’s output/loss are provided both in the case of alinear model and of a nonlinear model.
Revised: Results on a model’s output/loss are provided both in the case of alinear model and of a nonlinear model.

Original: For the former, results are exact, for the latter a third-order
Revised: For the former, results are exact, for the latter a third-order

Original: Taylor approximations is employed which is common and has been shown to provide accurate enoughapproximations to safely rely on the insights/quantities obtained e.g. see Sec.
Revised: Taylor approximations is employed which is common and has been shown to provide accurate enoughapproximations to safely rely on the insights/quantities obtained e.g. see Sec.

Original: A.2 of Wei et al.
Revised: A.2 of Wei et al.

Original: All the proofs and implementation details are provided in the appendix.
Revised: All the proofs and implementation details are provided in the appendix.

Original: Full codebase will be publiclyreleased upon completion of the review process, a sample is provided in the supplementary material.
Revised: Full codebase will be publiclyreleased upon completion of the review process, a sample is provided in the supplementary material.

Original: We emphasize that we are not proposing a tractable alternative to DA but a theoretical analysis onthe type of regularization that DA implicitly puts on your model, and how to visualize that regularizer.
Revised: We emphasize that we are not proposing a tractable alternative to DA but a theoretical analysis onthe type of regularization that DA implicitly puts on your model, and how to visualize that regularizer.

Original:  Background
Revised:  Background

Original: Existing Explicit Regularizers From Data-Augmentation.
Revised: Existing Explicit Regularizers From Data-Augmentation.

Original: It is widely accepted that dataaugmentation (DA) regularizes a model towards the transformations that are modeled [28, 27],and this impacts performances signiﬁcantly, possibly as much as the regularization offered by thechoice of DN architecture [9] and optimizer [37].
Revised: It is widely accepted that dataaugmentation (DA) regularizes a model towards the transformations that are modeled [28, 27],and this impacts performances signiﬁcantly, possibly as much as the regularization offered by thechoice of DN architecture [9] and optimizer [37].

Original: To gain precious insights into the impact of DA onto the learned functional f γ , the most common strategy is to derive the explicit regularizer that directly acts upon f γ in the same manner as if one were to use DA during training.
Revised: To gain precious insights into the impact of DA onto the learned functional f γ , the most common strategy is to derive the explicit regularizer that directly acts upon f γ in the same manner as if one were to use DA during training.

Original: This explicit derivation is however challenging and so far has been limited to DA strategies such as additive white noise or multiplicative binary noise applied identically and independently throughout the image, as with dropout[38, 2].
Revised: This explicit derivation is however challenging and so far has been limited to DA strategies such as additive white noise or multiplicative binary noise applied identically and independently throughout the image, as with dropout[38, 2].

Original: In those settings, various works have studied in the linear regime the relation betweensuch data-augmentation and its equivalence to using Tikhonov [41] or weight decay [49] as inmin W (cid:80)
Revised: In those settings, various works have studied in the linear regime the relation betweensuch data-augmentation and its equivalence to using Tikhonov [41] or weight decay [49] as inmin W (cid:80)

Original: Nn =
Revised: Nn =

Original: [Equation]
Revised: [Equation]

Original: E (cid:15)
Revised: E (cid:15)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) (cid:2) (cid:107) y n − W ( x n + (cid:15) ) (cid:107) 22 (cid:3) =
Revised: ) (cid:2) (cid:107) y n − W ( x n + (cid:15) ) (cid:107) 22 (cid:3) =

Original: min
Revised: min

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: (cid:107) y
Revised: (cid:107) y

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 

Revised: (cid:107)

Original: + λ ( σ ) (cid:107) W (cid:107)
Revised: + λ ( σ ) (cid:107) W (cid:107)

Original: [Equation]
Revised: [Equation]

Original: F[1].
Revised: F[1].

Original: More recently, Wang et al.
Revised: More recently, Wang et al.

Original: [43], LeJeune et al.
Revised: [43], LeJeune et al.

Original: [20] extended the case of additive white noiseto nonlinear models and concluded that this DA corresponds to adding an explicit Frobenius normregularization onto the Jacobian of f θ evaluated at each data sample.
Revised: [20] extended the case of additive white noiseto nonlinear models and concluded that this DA corresponds to adding an explicit Frobenius normregularization onto the Jacobian of f θ evaluated at each data sample.

Original: Going to more involved DA strategies e.g. translations or zooms of the input images is challengingand has so far only been studied from an empirical perspective.
Revised: Going to more involved DA strategies e.g. translations or zooms of the input images is challengingand has so far only been studied from an empirical perspective.

Original: For example, Hernández-Garcíaand König [14, 13, 15] performed thorough ablation studies on the interplay between DAs and acollection of known explicit regularizers to ﬁnd correlations between them.
Revised: For example, Hernández-Garcíaand König [14, 13, 15] performed thorough ablation studies on the interplay between DAs and acollection of known explicit regularizers to ﬁnd correlations between them.

Original: It was concluded thatweight-decay (the explicit regularizer of additive white noise) does not relate to those more advanced
Revised: It was concluded thatweight-decay (the explicit regularizer of additive white noise) does not relate to those more advanced

Original: DAs.
Revised: DAs.

Original: This also led other studies to suggest that norm-based regularization might be insufﬁcientto describe the implicit regularization of DAs involving advanced image transformations [31].
Revised: This also led other studies to suggest that norm-based regularization might be insufﬁcientto describe the implicit regularization of DAs involving advanced image transformations [31].

Original: Wedebunk this last claim in Sec. 3.3.
Revised: Wedebunk this last claim in Sec. 3.3.

Original: Coordinate Space Transformation.
Revised: Coordinate Space Transformation.

Original: Throughout this paper, we will consider a two-dimensionalimage I ( x, y ) to be at least square-integrable
Revised: Throughout this paper, we will consider a two-dimensionalimage I ( x, y ) to be at least square-integrable

Original: I ∈ L 2 ( R 2 ) [22].
Revised: I ∈ L 2 ( R 2 ) [22].

Original: Multi-channel images are dealtwith by treating each channel as its own single-channel image, as commonly done in practice [8].
Revised: Multi-channel images are dealtwith by treating each channel as its own single-channel image, as commonly done in practice [8].

Original: As we are interested in practical cases, we will often assume that I has compact support e.g. hasnonzero values only within a bounded domain such as [0 , 1] 2 .
Revised: As we are interested in practical cases, we will often assume that I has compact support e.g. hasnonzero values only within a bounded domain such as [0 , 1] 2 .

Original: Visualizing this image thus correspondsto displaying the sampled values of I on a regular grid (pixel positions) of [0 , 1] 2 [12].
Revised: Visualizing this image thus correspondsto displaying the sampled values of I on a regular grid (pixel positions) of [0 , 1] 2 [12].

Original: The mostcommon formulation to apply a transformation on the image I to obtain the transformed image Tis to transform the image coordinates [34, 45, 25].
Revised: The mostcommon formulation to apply a transformation on the image I to obtain the transformed image Tis to transform the image coordinates [34, 45, 25].

Original: That is, a mapping t : R 2 (cid:55)→ R 2 describes whatcoordinate t ( u, v ) of I maps to the coordinate u, v of T as in
Revised: That is, a mapping t : R 2 (cid:55)→ R 2 describes whatcoordinate t ( u, v ) of I maps to the coordinate u, v of T as in

Original: [Equation]
Revised: [Equation]

Original: This function t often has some parameter θ governing the underlying transformation as int θ ( x, y )
Revised: This function t often has some parameter θ governing the underlying transformation as int θ ( x, y )

Original: =
Revised: =

Original: [ x −
Revised: [ x −

Original: [Equation]
Revised: [Equation]

Original: 1 , y −
Revised: 1 , y −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: for translation, rotation and zoom respectively.
Revised: for translation, rotation and zoom respectively.

Original: We provide a visual depiction of the zoom transformation applied in coordinate space in Fig.
Revised: We provide a visual depiction of the zoom transformation applied in coordinate space in Fig.

Original: 6, in the appendix.
Revised: 6, in the appendix.

Original: The formulation of Eq.
Revised: The formulation of Eq.

Original: 1 has two keybeneﬁts.
Revised: 1 has two keybeneﬁts.

Original: First, it allows a simple and intuitive design of t to obtain novel transformations.
Revised: First, it allows a simple and intuitive design of t to obtain novel transformations.

Original: Second, itis computationally efﬁcient as the coordinate-space of images are 2 / 3 -dimensional.
Revised: Second, itis computationally efﬁcient as the coordinate-space of images are 2 / 3 -dimensional.

Original: Those beneﬁtshave led to e.g. the Spatial Transformer Network [16].
Revised: Those beneﬁtshave led to e.g. the Spatial Transformer Network [16].

Original: On the other hand, Eq. 1 has one majordrawback for our purpose: the exact moments of the transformed image under random θ parametersare not tractable due to the composition of t with the nonlinear mapping I .
Revised: On the other hand, Eq. 1 has one majordrawback for our purpose: the exact moments of the transformed image under random θ parametersare not tractable due to the composition of t with the nonlinear mapping I .

Original: And as it will becomeclear in Section 3.1, computing such quantities is crucial for our study.
Revised: And as it will becomeclear in Section 3.1, computing such quantities is crucial for our study.

Original:  Analytical Moments of Transformed Images Enable Inﬁnite Dataset
Revised:  Analytical Moments of Transformed Images Enable Inﬁnite Dataset

Original: Training
Revised: Training

Original: We ﬁrst motivate this study by formulating the training process under DA sampling as doing a MonteCarlo estimate of the true (unknown) expected loss under that DA distribution (Section 3.1).
Revised: We ﬁrst motivate this study by formulating the training process under DA sampling as doing a MonteCarlo estimate of the true (unknown) expected loss under that DA distribution (Section 3.1).

Original: Goingbeyond this sampling/estimation procedures requires knowledge of the expectation and varianceof a transformed sample, motivating the construction of a novel Data-Space Transformation (DTS)(Section 3.2) that allow for a close-form formula of those moments (Section 3.3).
Revised: Goingbeyond this sampling/estimation procedures requires knowledge of the expectation and varianceof a transformed sample, motivating the construction of a novel Data-Space Transformation (DTS)(Section 3.2) that allow for a close-form formula of those moments (Section 3.3).

Original: From those results,we will be able to remove the need to sample transformed images to train a model, by obtaining the
Revised: From those results,we will be able to remove the need to sample transformed images to train a model, by obtaining the

Original: [Figure]
Revised: [Figure]

Original: close-form expected loss Section 3.4, under which training can be done directly, in close-form forlinear model and with the usual gradient-based learning in general.
Revised: close-form expected loss Section 3.4, under which training can be done directly, in close-form forlinear model and with the usual gradient-based learning in general.

Original: 3.1 Motivation: Current Data-Augmentation Training Performs Monte-Carlo Estimation
Revised: 3.1 Motivation: Current Data-Augmentation Training Performs Monte-Carlo Estimation

Original: Training a model with DA consists in (i) sampling transformed images at each training iteration foreach sample x n as in T θ n ( x n ) with θ n ∼ θ a randomly sampled DA parameter
Revised: Training a model with DA consists in (i) sampling transformed images at each training iteration foreach sample x n as in T θ n ( x n ) with θ n ∼ θ a randomly sampled DA parameter

Original: e.g. the amount oftranslation to apply, (ii) evaluating the loss L on the transformed sample/mini-batch/dataset, and (iii)using some ﬂavor of gradient descent to update the parameters γ of the model f γ .
Revised: e.g. the amount oftranslation to apply, (ii) evaluating the loss L on the transformed sample/mini-batch/dataset, and (iii)using some ﬂavor of gradient descent to update the parameters γ of the model f γ .

Original: This training procedure corresponds to a one-sample Monte-Carlo (MC) estimate [23, 10] of theexpected loss
Revised: This training procedure corresponds to a one-sample Monte-Carlo (MC) estimate [23, 10] of theexpected loss

Original: [Equation]
Revised: [Equation]

Original: with θ n i.i.d samples of θ .
Revised: with θ n i.i.d samples of θ .

Original: In a supervised setting, the loss would also receive a per-sample target y nas input.
Revised: In a supervised setting, the loss would also receive a per-sample target y nas input.

Original: Although a one-sample estimate might be insufﬁcient to apply the central limit theorem[33] and guarantee training convergence, the combination of multiple samples in mini-batch trainingdoes provide convergence in most cases.
Revised: Although a one-sample estimate might be insufﬁcient to apply the central limit theorem[33] and guarantee training convergence, the combination of multiple samples in mini-batch trainingdoes provide convergence in most cases.

Original: Yet, methods such as Self-Supervised Learning thatheavily rely on DA, tend to be more sensitive and unstable [40].
Revised: Yet, methods such as Self-Supervised Learning thatheavily rely on DA, tend to be more sensitive and unstable [40].

Original: To avoid such instabilities, andto incorporate all the DA transformations of x n into the model’s parameter update, one would betempted to compute the model parameters’ gradient on the expected loss (left-hand side of Eq.
Revised: To avoid such instabilities, andto incorporate all the DA transformations of x n into the model’s parameter update, one would betempted to compute the model parameters’ gradient on the expected loss (left-hand side of Eq.

Original: (2))(see Section 3.4).
Revised: (2))(see Section 3.4).

Original: Knowledge of the expected loss would also prove useful to measure the quality ofthe MC estimate (see Section 4.1).
Revised: Knowledge of the expected loss would also prove useful to measure the quality ofthe MC estimate (see Section 4.1).

Original: Note that even in the least-square setting, such approximationis employed as the expected loss under DAs is not know.
Revised: Note that even in the least-square setting, such approximationis employed as the expected loss under DAs is not know.

Original: Hence, one commonly sample many DAsamples, and then apply the least-square formula on this augmented training set, which is highlyinefﬁcient.
Revised: Hence, one commonly sample many DAsamples, and then apply the least-square formula on this augmented training set, which is highlyinefﬁcient.

Original: Alternatively, this study will obtain the optimum least-square parameters under DAsampling (Theorem 3.4) i.e. only using the given training set, we directly obtain the parametersthat would correspond to sampling inﬁnitely many DA samples and applying the usual least-squareformula on this inﬁnite training set.
Revised: Alternatively, this study will obtain the optimum least-square parameters under DAsampling (Theorem 3.4) i.e. only using the given training set, we directly obtain the parametersthat would correspond to sampling inﬁnitely many DA samples and applying the usual least-squareformula on this inﬁnite training set.

Original: The close-form expected loss requires knowledge of the expectation and variance of the transformedsample T θ ( x n ) , hence, we ﬁrst propose to formulate a novel and tractable augmentation model(Section 3.2) that will allow us to obtain those moments analytically in Section 3.3.
Revised: The close-form expected loss requires knowledge of the expectation and variance of the transformedsample T θ ( x n ) , hence, we ﬁrst propose to formulate a novel and tractable augmentation model(Section 3.2) that will allow us to obtain those moments analytically in Section 3.3.

Original: 3.2 Proposed Data-Space Transformation
Revised: 3.2 Proposed Data-Space Transformation

Original: Instead of altering the coordinate positions of an image, as done in the coordinate-space transformationof Eq.
Revised: Instead of altering the coordinate positions of an image, as done in the coordinate-space transformationof Eq.

Original: (1), we propose to alter the image basis functions.
Revised: (1), we propose to alter the image basis functions.

Original: Going back to the construction of functions, one easily recalls that any (image) function can beexpanded into a basis as in I ( u, v ) =
Revised: Going back to the construction of functions, one easily recalls that any (image) function can beexpanded into a basis as in I ( u, v ) =

Original: (cid:82) I ( x, y )
Revised: (cid:82) I ( x, y )

Original: δ ( u − x, v − y ) dxdy with δ the usual Dirac distributions.
Revised: δ ( u − x, v − y ) dxdy with δ the usual Dirac distributions.

Original: Suppose for now that we consider a horizontal translation by a constant θ .
Revised: Suppose for now that we consider a horizontal translation by a constant θ .

Original: Then, one can obtain thattranslated image via
Revised: Then, one can obtain thattranslated image via

Original: [Equation]
Revised: [Equation]

Original: hence by moving the basis functions onto which the image is evaluated, rather than by moving theimage’s coordinate.
Revised: hence by moving the basis functions onto which the image is evaluated, rather than by moving theimage’s coordinate.

Original: As the image is now constant with respect to the transformation parameter θ , andas the basis functions have some speciﬁc analytical forms, the derivation of the transformed imagesexpectation and variance, under random θ , will become straightforward.
Revised: As the image is now constant with respect to the transformation parameter θ , andas the basis functions have some speciﬁc analytical forms, the derivation of the transformed imagesexpectation and variance, under random θ , will become straightforward.

Original: Because the transformedimage T is now obtained by combining its pixel values (recall Eq.
Revised: Because the transformedimage T is now obtained by combining its pixel values (recall Eq.

Original: we coin this transform as
Revised: we coin this transform as

Original: Data-Space Transform (DST) and we formally deﬁne it below.
Revised: Data-Space Transform (DST) and we formally deﬁne it below.

Original: Deﬁnition 3.1 (Data-Space Transform) .
Revised: Deﬁnition 3.1 (Data-Space Transform) .

Original: We deﬁne the data-space transformation of an image I ∈
Revised: We deﬁne the data-space transformation of an image I ∈

Original: L 2 ( R 2 ) producing the transformed image T θ ( I )
Revised: L 2 ( R 2 ) producing the transformed image T θ ( I )

Original: ∈
Revised: ∈

Original: L 2 ( R 2 ) as
Revised: L 2 ( R 2 ) as

Original: [Equation]
Revised: [Equation]

Original: with h θ ( u, v, ., . )
Revised: with h θ ( u, v, ., . )

Original: ∈ C ∞ 

Revised: ∈ C ∞

Original: ( R 2 ) encoding the transformation.
Revised: ( R 2 ) encoding the transformation.

Original: In Deﬁnition 3.1, we only impose for h θ ( u, v, ., . ) to be with compact support.
Revised: In Deﬁnition 3.1, we only impose for h θ ( u, v, ., . ) to be with compact support.

Original: In fact, one shouldinterpret h θ ( u, v, ., . )
Revised: In fact, one shouldinterpret h θ ( u, v, ., . )

Original: as a distribution whose purpose is to evaluate I at a desired (coordinate) positionon its domain.
Revised: as a distribution whose purpose is to evaluate I at a desired (coordinate) positionon its domain.

Original: This evaluation -depending on the form of h θ ( u, v, ., . ) - can return a single pixel-valueof the image I at a desired location (as in Eq.
Revised: This evaluation -depending on the form of h θ ( u, v, ., . ) - can return a single pixel-valueof the image I at a desired location (as in Eq.

Original: (3)), or it can combine multiple values e.g. withh θ ( u, v, ., . )
Revised: (3)), or it can combine multiple values e.g. withh θ ( u, v, ., . )

Original: being a bump function.
Revised: being a bump function.

Original: Additionally, we do not impose any restriction on the invertibilityof this mapping i.e. T θ ( I )( u, v ) can disregard parts of the original image.
Revised: Additionally, we do not impose any restriction on the invertibilityof this mapping i.e. T θ ( I )( u, v ) can disregard parts of the original image.

Original: Coordinate-space transformations as DSTs.
Revised: Coordinate-space transformations as DSTs.

Original: The coordinate-space transformation and the proposed DST (Deﬁnition 3.1) act in different spaces: the image coordinates and the image pixel values,respectively.
Revised: The coordinate-space transformation and the proposed DST (Deﬁnition 3.1) act in different spaces: the image coordinates and the image pixel values,respectively.

Original: Nevertheless, this does not limit the range of transformations that can be applied to an image.
Revised: Nevertheless, this does not limit the range of transformations that can be applied to an image.

Original: The following statement provides a simple recipe to turn any already employed coordinate-spacetransformation into a data-space one.
Revised: The following statement provides a simple recipe to turn any already employed coordinate-spacetransformation into a data-space one.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Any coordinate-space transformation (1) using t θ : R 2 (cid:55)→ R 2 can be expressed asa data-space transformation (4) by setting h θ ( u, v, x, y )
Revised: Any coordinate-space transformation (1) using t θ : R 2 (cid:55)→ R 2 can be expressed asa data-space transformation (4) by setting h θ ( u, v, x, y )

Original: =
Revised: =

Original: δ ( t θ ( x, y ) − [ u, v ] T ) .
Revised: δ ( t θ ( x, y ) − [ u, v ] T ) .

Original: We will focus in the main text on zoom/rotation/translation/shearing, and we propose in Appendix Cthe case of Gaussian blur, color jittering, random crop, along with a discussion on extending our resultsto CutMix [46] and MixUp [50].
Revised: We will focus in the main text on zoom/rotation/translation/shearing, and we propose in Appendix Dthe case of Gaussian blur, color jittering, random crop, along with a discussion on extending our resultsto CutMix [46] and MixUp [50].

Original: Using Proposition 3.2, we obtain the DST operators h θ ( u, v, x, y )to be
Revised: Using Proposition 3.2, we obtain the DST operators h θ ( u, v, x, y )to be

Original: [Equation]
Revised: [Equation]

Original: for the vertical/horizontal translation, vertical/horizontal shearing, zoom and rotation respectively.
Revised: for the vertical/horizontal translation, vertical/horizontal shearing, zoom and rotation respectively.

Original: Before focusing on the analytical moments of the data-space transformed samples, we describe howthose operators are applied in a discrete setting.
Revised: Before focusing on the analytical moments of the data-space transformed samples, we describe howthose operators are applied in a discrete setting.

Original: Discretized version.
Revised: Discretized version.

Original: We now describe how any DST of a discrete image, ﬂattened as a vector, canbe expressed as a matrix-vector product with the matrix entries depending on the employed DAand its parameter θ , as in Figs.
Revised: We now describe how any DST of a discrete image, ﬂattened as a vector, canbe expressed as a matrix-vector product with the matrix entries depending on the employed DAand its parameter θ , as in Figs.

Original: 1 and 2.
Revised: 1 and 2.

Original: The functional form of the proposed transform in Eq.
Revised: The functional form of the proposed transform in Eq.

Original: (4)producing the target image at a speciﬁc position T θ ( I )( u, v ) is linear in the image I .
Revised: (4)producing the target image at a speciﬁc position T θ ( I )( u, v ) is linear in the image I .

Original: The continuousintegral over the image domain is replaced with a summation with indices based on the desired sampling/resolution of I .
Revised: The continuousintegral over the image domain is replaced with a summation with indices based on the desired sampling/resolution of I .

Original: Expressing linear operators as matrix-vector products will greatly easeour development, we will denote by x ∈ R hw the ﬂattened ( h × w ) discrete images I .
Revised: Expressing linear operators as matrix-vector products will greatly easeour development, we will denote by x ∈ R hw the ﬂattened ( h × w ) discrete images I .

Original: Hence, ourdata-space transformation, given some parameters θ , takes the form of
Revised: Hence, ourdata-space transformation, given some parameters θ , takes the form of

Original: [Equation]
Revised: [Equation]

Original: (6) with t ( θ ) ∈ R hw the ﬂattened transformed image (which can be reshaped as desired) and M ( θ ) ∈
Revised: (6) with t ( θ ) ∈ R hw the ﬂattened transformed image (which can be reshaped as desired) and M ( θ ) ∈

Original: R hw × hw the matrix whose rows encode the discrete and ﬂattened h θ ( u, v, ., . ) .
Revised: R hw × hw the matrix whose rows encode the discrete and ﬂattened h θ ( u, v, ., . ) .

Original: For example, andemploying a uniform grid sampling for illustration, M ( θ ) i,j = h θ ( i//w, i % w, j//w, j % w ) with //representing the ﬂoor division and % the modulo operation.
Revised: For example, andemploying a uniform grid sampling for illustration, M ( θ ) i,j = h θ ( i//w, i % w, j//w, j % w ) with //representing the ﬂoor division and % the modulo operation.

Original: For the case of multi-channel images, weconsider without loss of generality the application of the matrix-vector operation on each channelseparately.
Revised: For the case of multi-channel images, weconsider without loss of generality the application of the matrix-vector operation on each channelseparately.

Original: We depict this operation along with the exact form of M ( θ ) for the case of the zoomtransformation in Fig.
Revised: We depict this operation along with the exact form of M ( θ ) for the case of the zoomtransformation in Fig.

Original: 3.3 Analytical Expectation and Variance of Transformed Images
Revised: 3.3 Analytical Expectation and Variance of Transformed Images

Original: The above construction (Deﬁnition 3.1) turns out to make the analytical form of the ﬁrst two momentsof an augmented sample much simpler to derive.
Revised: The above construction (Deﬁnition 3.1) turns out to make the analytical form of the ﬁrst two momentsof an augmented sample much simpler to derive.

Original: As this derivation is at the core of our maincontribution, we propose a step-by-step derivation of E θ
Revised: As this derivation is at the core of our maincontribution, we propose a step-by-step derivation of E θ

Original: [ T θ ( I )] for the case of horizontal translation(as in Eq.
Revised: [ T θ ( I )] for the case of horizontal translation(as in Eq.

Original: Let’s consider again the continuous model (the discrete version is provided after Theorem 3.3).
Revised: Let’s consider again the continuous model (the discrete version is provided after Theorem 3.3).

Original: Using
Revised: Using

Original: Fubini’s theorem to switch the order of integration and recalling Eq.
Revised: Fubini’s theorem to switch the order of integration and recalling Eq.

Original: (4), we have that
Revised: (4), we have that

Original: [Equation]
Revised: [Equation]

Original: Using the deﬁnition for h θ from Eq.
Revised: Using the deﬁnition for h θ from Eq.

Original: (5) for horizontal translation, E θ
Revised: (5) for horizontal translation, E θ

Original: [ h θ ( u, v, x, y )] becomes
Revised: [ h θ ( u, v, x, y )] becomes

Original: E θ [
Revised: E θ [

Original: [Equation]
Revised: [Equation]

Original: with p the density function of θ prescribing how the translation parameter is distributed.
Revised: with p the density function of θ prescribing how the translation parameter is distributed.

Original: As a result,in this univariate translation case, the expected augmented image at coordinate ( u, v ) is given by
Revised: As a result,in this univariate translation case, the expected augmented image at coordinate ( u, v ) is given by

Original: [Equation]
Revised: [Equation]

Original: which can be further simpliﬁed into E θ
Revised: which can be further simpliﬁed into E θ

Original: [ T θ ( I )( ., v )]
Revised: [ T θ ( I )( ., v )]

Original: = I ( ., v ) (cid:63) p .
Revised: = I ( ., v ) (cid:63) p .

Original: Hence, the expected x -translatedimage is the convolution (on the x -axis only) between the original image I and the univariate densityfunction p .
Revised: Hence, the expected x -translatedimage is the convolution (on the x -axis only) between the original image I and the univariate densityfunction p .

Original: We formalize this for the transformations of Eq.
Revised: We formalize this for the transformations of Eq.

Original: (5) below.
Revised: (5) below.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: The analytical form of E θ [ h θ ( u, v, x, y )] , used to obtain the expected transformedimage (recall Eq.
Revised: The analytical form of E θ [ h θ ( u, v, x, y )] , used to obtain the expected transformedimage (recall Eq.

Original: (7) ) is given by
Revised: (7) ) is given by

Original: [Equation]
Revised: [Equation]

Original: translation)
Revised: translation)

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: other cases and E θ
Revised: other cases and E θ

Original: [ T θ ( x n ) T θ ( x n ) T ] are deferred to the proof.
Revised: [ T θ ( x n ) T θ ( x n ) T ] are deferred to the proof.

Original: (Proof in Appendix F.)
Revised: (Proof in Appendix G.)

Original: Notice for example how the expected image under random 2 -dimensional translations with ( 2dimensional) density p is given by the convolution I (cid:63) p , providing a new portal to study and interpretconvolutions with nonnegative, sum-to-one ﬁlters p .
Revised: Notice for example how the expected image under random 2 -dimensional translations with ( 2dimensional) density p is given by the convolution I (cid:63) p , providing a new portal to study and interpretconvolutions with nonnegative, sum-to-one ﬁlters p .

Original: Again, and as per Eq.
Revised: Again, and as per Eq.

Original: (6), the discretized versionof the expected image takes the form of E θ
Revised: (6), the discretized versionof the expected image takes the form of E θ

Original: [ t ( θ )]
Revised: [ t ( θ )]

Original: =
Revised: =

Original: E θ [ M ( θ )]
Revised: E θ [ M ( θ )]

Original: x with the entries of E θ [ M ( θ )] givenby discretizing Eq.
Revised: x with the entries of E θ [ M ( θ )] givenby discretizing Eq.

Original: We depict this expected matrix for various transformations as well as theirapplication onto two different discrete images in Fig. 2, and we now proceed on deriving the left-handside of Eq.
Revised: We depict this expected matrix for various transformations as well as theirapplication onto two different discrete images in Fig. 2, and we now proceed on deriving the left-handside of Eq.

Original: (2) i.e. the explicit DA regularizer.
Revised: (2) i.e. the explicit DA regularizer.

Original: 3.4 The Explicit Regularizer of Data-Augmentations
Revised: 3.4 The Explicit Regularizer of Data-Augmentations

Original: To keep notations as light as possible, we ﬁrst consider a linear regression model with Mean Squared
Revised: To keep notations as light as possible, we ﬁrst consider a linear regression model with Mean Squared

Original: Error (MSE), the nonlinear case will be studied in Section 4.
Revised: Error (MSE), the nonlinear case will be studied in Section 4.

Original: In that setting, the expected loss under
Revised: In that setting, the expected loss under

Original: DA sampling (recall Eq.
Revised: DA sampling (recall Eq.

Original: (2)) becomes
Revised: (2)) becomes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t h i r d s ec ond ﬁ r s t rotation shear r a nk o f V [ T ( x ) ] transformation ampl.
Revised: t h i r d s ec ond ﬁ r s t rotation shear r a nk o f V [ T ( x ) ] transformation ampl.

Original: with x n ∈ R D , n = 1 , . . .
Revised: with x n ∈ R D , n = 1 , . . .

Original: , N the input (ﬂattened)
Revised: , N the input (ﬂattened)

Original: n th image I n , y n ∈ R K the n th target vector, and
Revised: n th image I n , y n ∈ R K the n th target vector, and

Original: W ∈ R K × D , b ∈ R K the model’s parameters.
Revised: W ∈ R K × D , b ∈ R K the model’s parameters.

Original: Recall from Section 3.1 that to learn W under DA,the current method consists in performing a MC estimation.
Revised: Recall from Section 3.1 that to learn W under DA,the current method consists in performing a MC estimation.

Original: Instead, let’s derive (detailed derivationin Appendix D) the exact loss of Eq.
Revised: Instead, let’s derive (detailed derivationin Appendix E) the exact loss of Eq.

Original: (10) as a function of the sample mean and variance under theconsider DA.
Revised: (10) as a function of the sample mean and variance under theconsider DA.

Original: We will drop the θ subscript for clarity to obtain
Revised: We will drop the θ subscript for clarity to obtain

Original: [Equation]
Revised: [Equation]

Original: with the spectral decomposition Q ( x )Λ( x ) Q ( x ) T = V [ T ( x )] .
Revised: with the spectral decomposition Q ( x )Λ( x ) Q ( x ) T = V [ T ( x )] .

Original: The right term in Eq.
Revised: The right term in Eq.

Original: (11) is theexplicit DA regularizer.
Revised: (11) is theexplicit DA regularizer.

Original: It pushes the kernel space of W to align with the largest principal directionsof the data manifold tangent space, as modeled by the DA.
Revised: It pushes the kernel space of W to align with the largest principal directionsof the data manifold tangent space, as modeled by the DA.

Original: In fact, the largest eigenvectors in Q ( x )represent the principal directions of the data manifold tangent space at x , as encoded via V [ T ( x )] .
Revised: In fact, the largest eigenvectors in Q ( x )represent the principal directions of the data manifold tangent space at x , as encoded via V [ T ( x )] .

Original: We propose in Fig.
Revised: We propose in Fig.

Original: 3 visualization of Q and Λ for different DAs, illustrating how each DA policyimpacts the model’s parameter W through the regularization of Eq.
Revised: 3 visualization of Q and Λ for different DAs, illustrating how each DA policyimpacts the model’s parameter W through the regularization of Eq.

Original: The knowledge of E [ T ( x )]and V [ T ( x )] from Theorem 3.3 ﬁnally enables to train a (linear) model on the true expected loss(Eq.
Revised: The knowledge of E [ T ( x )]and V [ T ( x )] from Theorem 3.3 ﬁnally enables to train a (linear) model on the true expected loss(Eq.

Original: as we formalize below.
Revised: as we formalize below.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: Training a linear model with MSE and inﬁnite DA sampling is equivalent to minimizing
Revised: Training a linear model with MSE and inﬁnite DA sampling is equivalent to minimizing

Original: Eq.
Revised: Eq.

Original: (11) and produces the optimal W ∗ model’s parameter
Revised: (11) and produces the optimal W ∗ model’s parameter

Original: [Equation]
Revised: [Equation]

Original: Not surprisingly, whenever the DA is identity, E [ T ( x n )] = x n , V [ T ( x n )] = 0 and thus Eq.
Revised: Not surprisingly, whenever the DA is identity, E [ T ( x n )] = x n , V [ T ( x n )] = 0 and thus Eq.

Original: (12)recovers exactly the standard least-square solution.
Revised: (12)recovers exactly the standard least-square solution.

Original: We visualize V [ T ( x n )] for the translation DA in
Revised: We visualize V [ T ( x n )] for the translation DA in

Original: Fig.
Revised: Fig.

Original: The same line of result can be derived in the nonlinear setting by assuming that the DA isrestricted to small transformations.
Revised: The same line of result can be derived in the nonlinear setting by assuming that the DA isrestricted to small transformations.

Original: In that case, one leverages a truncated Taylor approximation ofthe nonlinear model 1 and recovers that (local) DA applies the same regularization as in Eq.
Revised: In that case, one leverages a truncated Taylor approximation ofthe nonlinear model 1 and recovers that (local) DA applies the same regularization as in Eq.

Original: (11) butwith the model’s Jacobian matrix J f γ ( x n ) in-place of W (more details in Section 4.3).
Revised: (11) butwith the model’s Jacobian matrix J f γ ( x n ) in-place of W (more details in Section 4.3).

Original: Given the above derivation of the exact expected loss, we can now turn to precisely measure howaccurate is the MC estimate commonly used to train models under DA sampling.
Revised: Given the above derivation of the exact expected loss, we can now turn to precisely measure howaccurate is the MC estimate commonly used to train models under DA sampling.

Original:  Data-Augmentation Sampling Efﬁciency and Loss Sensitivity
Revised:  Data-Augmentation Sampling Efﬁciency and Loss Sensitivity

Original: In this section we study the convergence of the MC estimate (Section 4.1), and provide varianceanalysis of that estimate as a function of the model’s Jacobian matrix and the sample variance
Revised: In this section we study the convergence of the MC estimate (Section 4.1), and provide varianceanalysis of that estimate as a function of the model’s Jacobian matrix and the sample variance

Original: [Figure]
Revised: [Figure]

Original: eigenvectors (Section 4.2), concluding by the (re-)discovery of TangentProp from ﬁrst principles(Section 4.3).
Revised: eigenvectors (Section 4.2), concluding by the (re-)discovery of TangentProp from ﬁrst principles(Section 4.3).

Original: Those results hold regardless of the type of DA employed.
Revised: Those results hold regardless of the type of DA employed.

Original: Additionally, the analysis isdone on a single sample to ease notation, in the i.i.d.
Revised: Additionally, the analysis isdone on a single sample to ease notation, in the i.i.d.

Original: setting, simply sum over all the training samplesto analyze the training set.
Revised: setting, simply sum over all the training samplesto analyze the training set.

Original: 4.1 Empirical Monte-Carlo Convergence of Transformed Images
Revised: 4.1 Empirical Monte-Carlo Convergence of Transformed Images

Original: Given the close-form average image and average loss , we empirically measure how efﬁcient is the
Revised: Given the close-form average image and average loss , we empirically measure how efﬁcient is the

Original: MC estimation from Eq.
Revised: MC estimation from Eq.

Original: We ﬁrst propose in Fig.
Revised: We ﬁrst propose in Fig.

Original: 4 a constructed (64 × 64) image for which we compute the expected loss(Eq.
Revised: 4 a constructed (64 × 64) image for which we compute the expected loss(Eq.

Original: and the MC estimate (right-hand side of Eq.
Revised: and the MC estimate (right-hand side of Eq.

Original: Surprisingly, we obtain that even forsuch a simple image and translation DA, between 1000 and 10000 samples are required to correctlyestimate the MSE loss from the augmented samples.
Revised: Surprisingly, we obtain that even forsuch a simple image and translation DA, between 1000 and 10000 samples are required to correctlyestimate the MSE loss from the augmented samples.

Original: In a more practical scenario, one could rightfullyargue that the combination of the DA samples from different images allows to obtain a better estimatewith a smaller amount of augmentations per sample.
Revised: In a more practical scenario, one could rightfullyargue that the combination of the DA samples from different images allows to obtain a better estimatewith a smaller amount of augmentations per sample.

Original: Hence, we provide in Fig.
Revised: Hence, we provide in Fig.

Original: 5 that experimentusing a linear model on MNIST [19] with varying train set size.
Revised: 5 that experimentusing a linear model on MNIST [19] with varying train set size.

Original: We observe that as the numberof samples grows as the required number of augmentation per sample reduces.
Revised: We observe that as the numberof samples grows as the required number of augmentation per sample reduces.

Original: Nevertheless, evenwith thousands of samples, at least 50 augmentations per sample are required to provide an accurateestimate (reproduction with nonlinear models given in Fig. 7).
Revised: Nevertheless, evenwith thousands of samples, at least 50 augmentations per sample are required to provide an accurateestimate (reproduction with nonlinear models given in Fig. 7).

Original: We now propose to speciﬁcally quantify the sensitivity of the MC estimate to DA sampling.
Revised: We now propose to speciﬁcally quantify the sensitivity of the MC estimate to DA sampling.

Original: 4.2 Loss Sensitivity Under Data-Augmentation Sampling in the Linear and NonlinearRegime
Revised: 4.2 Loss Sensitivity Under Data-Augmentation Sampling in the Linear and NonlinearRegime

Original: Recalling Section 3.1, current DA training is performed on a MC estimate of the loss.
Revised: Recalling Section 3.1, current DA training is performed on a MC estimate of the loss.

Original: The estimator’svariance [26] is proportional to the variance of the quantity being estimated:
Revised: The estimator’svariance [26] is proportional to the variance of the quantity being estimated:

Original: V [( L ◦ f )( T ( x ))] .
Revised: V [( L ◦ f )( T ( x ))] .

Original: Wenow characterize when, and why, would an MC estimator converge for a given model.
Revised: Wenow characterize when, and why, would an MC estimator converge for a given model.

Original: By leveraging the delta method [6, 29] i.e. a truncated Taylor expansion of the model and lossfunction mapping as L ◦ f , we have
Revised: By leveraging the delta method [6, 29] i.e. a truncated Taylor expansion of the model and lossfunction mapping as L ◦ f , we have

Original: [Equation]
Revised: [Equation]

Original: (13) with (cid:107) u (cid:107) 2 A (cid:44) u T Au .
Revised: (13) with (cid:107) u (cid:107) 2 A (cid:44) u T Au .

Original: Noticing that V [ T ( x )]
Revised: Noticing that V [ T ( x )]

Original: =
Revised: =

Original: E [ T ( x ) T ( x ) T ] − E
Revised: E [ T ( x ) T ( x ) T ] − E

Original: [ T ( x )] E
Revised: [ T ( x )] E

Original: [ T ( x )]
Revised: [ T ( x )]

Original: T and using theclosed-form moments of augmented samples from Theorem 3.3, it is possible to write out Eq.
Revised: T and using theclosed-form moments of augmented samples from Theorem 3.3, it is possible to write out Eq.

Original: (13)explicitly for model and DA speciﬁc analysis.
Revised: (13)explicitly for model and DA speciﬁc analysis.

Original: We visualize V
Revised: We visualize V

Original: [ T ( x )] in Fig. 3.
Revised: [ T ( x )] in Fig. 3.

Original: Linear regression case.
Revised: Linear regression case.

Original: To gain some insight into Eq.
Revised: To gain some insight into Eq.

Original: (13), let’s ﬁrst consider the linear regressioncase leading to V
Revised: (13), let’s ﬁrst consider the linear regressioncase leading to V

Original: [( L ◦ f )( T ( x ))]
Revised: [( L ◦ f )( T ( x ))]

Original: ≈ (cid:107)
Revised: ≈ (cid:107)

Original: y − W E
Revised: y − W E

Original: [ T ( x )] − b (cid:107) 2 W V [ T ( x )] W T .
Revised: [ T ( x )] − b (cid:107) 2 W V [ T ( x )] W T .

Original: Hence, the estimated lossvariance depends on the model predicting the correct output when observing the expected sample 282 E
Revised: Hence, the estimated lossvariance depends on the model predicting the correct output when observing the expected sample 282 E

Original: [ T ( x )] , and on the smallest right singular vectors of W to align with the largest eigenvectors of 283 V [ T ( x )] , echoing our observation below Eq.
Revised: [ T ( x )] , and on the smallest right singular vectors of W to align with the largest eigenvectors of 283 V [ T ( x )] , echoing our observation below Eq.

Original: General case.
Revised: General case.

Original: Since the result from Eq.
Revised: Since the result from Eq.

Original: (13) holds in the general setting, we can generalize the aboveobservation to the more general setting.
Revised: (13) holds in the general setting, we can generalize the aboveobservation to the more general setting.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: The variance of the loss’s MC estimate (recall Eq.
Revised: The variance of the loss’s MC estimate (recall Eq.

Original: (2) ) for an input x n goes to 0if the loss gradient ∇L ( f ( E [ T ( x )])) goes to 0 , or if the kernel of the model’s Jacobian matrix
Revised: (2) ) for an input x n goes to 0if the loss gradient ∇L ( f ( E [ T ( x )])) goes to 0 , or if the kernel of the model’s Jacobian matrix

Original: J f ( E [ T ( x )]) aligns with the largest eigenvectors of the sample variance V [ T ( x )] .
Revised: J f ( E [ T ( x )]) aligns with the largest eigenvectors of the sample variance V [ T ( x )] .

Original: The proof of the above statement simply starts from Eq.
Revised: The proof of the above statement simply starts from Eq.

Original: and applies the Cauchy-Schwarzinequality two times to obtain
Revised: and applies the Cauchy-Schwarzinequality two times to obtain

Original: [Equation]
Revised: [Equation]

Original: with V [ T ( x )] = Q Λ Q T .
Revised: with V [ T ( x )] = Q Λ Q T .

Original: From Eq.
Revised: From Eq.

Original: (14), it is quite realistic to assume that the model’s Jacobian doesnot collapse to 0 (this would not minimize the training loss in general).
Revised: (14), it is quite realistic to assume that the model’s Jacobian doesnot collapse to 0 (this would not minimize the training loss in general).

Original: Hence, the most realistic caseof Theorem 4.1 concerns the alignment between the model’s
Revised: Hence, the most realistic caseof Theorem 4.1 concerns the alignment between the model’s

Original: Jacobian matrix kernel space and thetop eigenvectors of V [ T ( x )] .
Revised: Jacobian matrix kernel space and thetop eigenvectors of V [ T ( x )] .

Original: However, for models like ResNet DNs [11], the loss variance can onlyreduce to 0 if the model minimizes the loss at the expected input since the Jacobian matrix is alwaysfull-rank.
Revised: However, for models like ResNet DNs [11], the loss variance can onlyreduce to 0 if the model minimizes the loss at the expected input since the Jacobian matrix is alwaysfull-rank.

Original: We recall that the above relied on the delta method to approximate the intractable LHS of
Revised: We recall that the above relied on the delta method to approximate the intractable LHS of

Original: Eq.
Revised: Eq.

Original: (13) (see footnote on Page 7) .
Revised: (13) (see footnote on Page 7) .

Original: 4.3 Explicit Loss Sensitivity Minimization Provably Recovers TangentProp
Revised: 4.3 Explicit Loss Sensitivity Minimization Provably Recovers TangentProp

Original: The expected loss (Eq.
Revised: The expected loss (Eq.

Original: (2)) has been derived in the linear regression setting (Eq.
Revised: (2)) has been derived in the linear regression setting (Eq.

Original: But in a moregeneral scenario, and as discussed below Theorem 3.4, this expectation might not be tractable andthus needs to be approximated e.g. based on a Taylor expansion of the loss and model.
Revised: But in a moregeneral scenario, and as discussed below Theorem 3.4, this expectation might not be tractable andthus needs to be approximated e.g. based on a Taylor expansion of the loss and model.

Original: Alternativelyto using the approximated expectation, one could employ the usual MC estimate of the expectation(right-hand side of Eq.
Revised: Alternativelyto using the approximated expectation, one could employ the usual MC estimate of the expectation(right-hand side of Eq.

Original: (2)), and leverage the MC estimator variance obtained in Eq.
Revised: (2)), and leverage the MC estimator variance obtained in Eq.

Original: (14) as aregularizer.
Revised: (14) as aregularizer.

Original: We show here that both approaches are equivalent, and recover a popular regularizerknown as TangentProp [36].
Revised: We show here that both approaches are equivalent, and recover a popular regularizerknown as TangentProp [36].

Original: Although various extensions of TangentProp have been introduced[3, 32] no principled derivation of it has yet been proposed.
Revised: Although various extensions of TangentProp have been introduced[3, 32] no principled derivation of it has yet been proposed.

Original: Using the same argument as in Section 4.2,the expectation of a nonlinear transformation of a random variable ( T ( x n ) ) can be approximatedfrom the expectation of the Taylor expansion (detailed derivations in Appendix E) from which anupper-bound is obtained by applying the Cauchy-Schwarz inequality
Revised: Using the same argument as in Section 4.2,the expectation of a nonlinear transformation of a random variable ( T ( x n ) ) can be approximatedfrom the expectation of the Taylor expansion (detailed derivations in Appendix F) from which anupper-bound is obtained by applying the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: with κ ( x ) ≥ 0 and the notations from Eq.
Revised: with κ ( x ) ≥ 0 and the notations from Eq.

Original: As a result, the TangentProp regularizer naturallyappears when using a Taylor approximation of the expected loss, and it corresponds to adding anexplicit loss variance regularization term (compare the TangentProp with Eq.
Revised: As a result, the TangentProp regularizer naturallyappears when using a Taylor approximation of the expected loss, and it corresponds to adding anexplicit loss variance regularization term (compare the TangentProp with Eq.

Original: We thus obtainedfrom ﬁrst principles that TangentProp emerges naturally when considering the second order Taylorapproximation of the expected loss given a DA.
Revised: We thus obtainedfrom ﬁrst principles that TangentProp emerges naturally when considering the second order Taylorapproximation of the expected loss given a DA.

Original:  Conclusions and Limitations
Revised:  Conclusions and Limitations

Original: In this paper, we proposed a novel set of mathematical tools (Sections 3.2 and 3.3) under which it ispossible to study DA and to provably answer some of the open questions around the efﬁciency andimpact of DA to train a model.
Revised: In this paper, we proposed a novel set of mathematical tools (Sections 3.2 and 3.3) under which it ispossible to study DA and to provably answer some of the open questions around the efﬁciency andimpact of DA to train a model.

Original: We ﬁrst obtained the explicit regularizer produced by different DAs in
Revised: We ﬁrst obtained the explicit regularizer produced by different DAs in

Original: Section 3.4.
Revised: Section 3.4.

Original: This led to the following observation: the kernel space of the W matrix is pushed toalign with the largest eigenvectors of the sample covariance matrix.
Revised: This led to the following observation: the kernel space of the W matrix is pushed toalign with the largest eigenvectors of the sample covariance matrix.

Original: This was then studied in a moregeneral setting in Section 4.2 for nonlinear models when characterizing the loss variance under DAsampling.
Revised: This was then studied in a moregeneral setting in Section 4.2 for nonlinear models when characterizing the loss variance under DAsampling.

Original: We also observed that Monte-Carlo sampling of transformed images is highly inefﬁcient(Section 4.1), even if similar training samples combine their underlying information within a dataset.
Revised: We also observed that Monte-Carlo sampling of transformed images is highly inefﬁcient(Section 4.1), even if similar training samples combine their underlying information within a dataset.

Original: Lastly, those derivations led us to provably derive a known regularizer —TangentProp— as being thenatural minimizer of a model’s loss variance.
Revised: Lastly, those derivations led us to provably derive a known regularizer —TangentProp— as being thenatural minimizer of a model’s loss variance.

Original: The main limitation of the current analysis concernsnonlinear models for which we rely on a third-order Taylor expansion.
Revised: The main limitation of the current analysis concernsnonlinear models for which we rely on a third-order Taylor expansion.

Original: We hope to explore in moredetails the impact of DA on nonlinear models, possibly with a per-architecture analysis in futurework.
Revised: We hope to explore in moredetails the impact of DA on nonlinear models, possibly with a per-architecture analysis in futurework.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contri-
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contri-

Original: butions and scope?
Revised: butions and scope?

Original: [Yes] We have precisely given the scope of our results in the end of theintroduction.
Revised: [Yes] We have precisely given the scope of our results in the end of theintroduction.

Original: Additionally, we recall the limitations and assumptions of our results within eachsection carefully.
Revised: Additionally, we recall the limitations and assumptions of our results within eachsection carefully.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] The main limitations comes from the approx-
Revised: [Yes] The main limitations comes from the approx-

Original: imation done for nonlinear models which we have repeatedly highlight in both the introductionsand the corresponding sections.
Revised: imation done for nonlinear models which we have repeatedly highlight in both the introductionsand the corresponding sections.

Original: We also highlight those in the conclusion along with future work.
Revised: We also highlight those in the conclusion along with future work.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We do not see anypotential negative societal impact for this work
Revised: [N/A] We do not see anypotential negative societal impact for this work

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] we have carefully given theassumptions each result relies on.
Revised: [Yes] we have carefully given theassumptions each result relies on.

Original: Especially, at the end of the introduction, we have highlighted a
Revised: Especially, at the end of the introduction, we have highlighted a

Original: reference paper where it was shown that our assumptions are common in the literature, and that
Revised: reference paper where it was shown that our assumptions are common in the literature, and that

Original: they are good enough to gain insights into nonlinear models.
Revised: they are good enough to gain insights into nonlinear models.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Each formal statement andderivation is linked to a proof in the appendix.
Revised: [Yes] Each formal statement andderivation is linked to a proof in the appendix.

Original: We also provide details derivations in the appendix
Revised: We also provide details derivations in the appendix

Original: of all the equations we have obtained in the main paper, even if they are not part of a formal result.
Revised: of all the equations we have obtained in the main paper, even if they are not part of a formal result.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental

Original: results (either in the supplemental material or as a URL)?
Revised: results (either in the supplemental material or as a URL)?

Original: [Yes] We provide samples of the
Revised: [Yes] We provide samples of the

Original: code to reproduce ﬁgures.
Revised: code to reproduce ﬁgures.

Original: The entire codebase will be released publicly upon completion of the
Revised: The entire codebase will be released publicly upon completion of the

Original: review-process
Revised: review-process

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We only explore MNIST and provided the training details in the caption of the correspondingﬁgure(c)
Revised: [Yes] We only explore MNIST and provided the training details in the caption of the correspondingﬁgure(c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multipletimes)?
Revised: Did you report error bars (e.g., with respect to the random seed after running experiments multipletimes)?

Original: [No] We did not provide error bars but multiple runs with various settings(d)
Revised: [No] We did not provide error bars but multiple runs with various settings(d)

Original: Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?

Original: [No] The employed experiment is small scale and can beperformed without GPU
Revised: [No] The employed experiment is small scale and can beperformed without GPU

Original: [Table]
Revised: [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a)
Revised: (a)

Original: If your work uses existing assets, did you cite the creators?
Revised: If your work uses existing assets, did you cite the creators?

Original: [Yes] MNIST only(b) Did you mention the license of the assets?
Revised: [Yes] MNIST only(b) Did you mention the license of the assets?

Original: [No](c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [No](c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No](d) Did you discuss whether
Revised: [No](d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A](e)
Revised: [N/A](e)

Original: Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)

Original: approvals, if applicable?
Revised: approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?

Original: [N/A]
Revised: [N/A]