Original: A Data-Augmentation Is Worth A Thousand Samples Author(s)
Revised: A Data-Augmentation Is Worth A Thousand Samples: Analytical Moments And Sampling-Free Training

Original: Abstract
Revised: Abstract

Original: Data-Augmentation (DA) is known to improve performance across tasks anddatasets.
Revised: Data-Augmentation (DA) is known to improve performance across tasks and datasets.

Original: We propose a method to theoretically analyze the effect of DA and studyquestions such as: how many augmented samples are needed to correctly estimatethe information encoded by that DA?
Revised: We propose a method to theoretically analyze the effect of DA and study questions such as: how many augmented samples are needed to correctly estimate the information encoded by that DA?

Original: How does the augmentation policy impactthe ﬁnal parameters of a model?
Revised: How does the augmentation policy impact the ﬁnal parameters of a model?

Original: We derive several quantities in close-form, suchas the expectation and variance of an image, loss, and model’s output under a given
Revised: We derive several quantities in close-form, such as the expectation and variance of an image, loss, and model’s output under a given

Original: DA distribution.
Revised: DA distribution.

Original: Up to our knowledge, we obtain the ﬁrst explicit regularizer thatcorresponds to using DA during training for non-trivial transformations such asafﬁne transformations, color jittering, or Gaussian blur.
Revised: Up to our knowledge, we obtain the ﬁrst explicit regularizer that corresponds to using DA during training for non-trivial transformations such as afﬁne transformations, color jittering, or Gaussian blur.

Original: Those derivations opennew avenues to quantify the beneﬁts and limitations of DA.
Revised: Those derivations open new avenues to quantify the beneﬁts and limitations of DA.

Original: For example, we showthat common DAs require tens of thousands of samples for the lossat hand to becorrectly estimated and for the model training to converge.
Revised: For example, we show given a loss at hand, common DAs require tens of thousands of samples for the loss to be correctly estimated and for the model training to converge.

Original: We  show that for atraining loss to be stable under DA sampling, the model’s saliency map (gradient ofthe loss with respect to the model’s input) must align with the smallest eigenvectorof the sample variance under the considered DA augmentation, hinting at a possibleexplanation on why models tend to shift their focus from edges to textures.
Revised: We then show that for a training loss to have reduced variance under DA sampling, the model’s saliency map (gradient of the loss with respect to the model’s input) must align with the smallest eigenvector of the sample’s covariance matrix under the considered DA augmentation; this is exactly the quantity estimated and regularized by TangentProp. Those ﬁndings also hint at a possible explanation on why models tend to shift their focus from edges to textures when speciﬁc DAs are employed.

Original:  Introduction
Revised:  Introduction

Original: Data Augmentation (DA) is a prevalent technique in training deep learning models [8, 18].
Revised: Data Augmentation (DA) is a prevalent technique in training deep learning models [1, 2, 3].

Original: These
Revised: These

Original: Deep Networks (DNs) models f γ , governed by some parameters γ ∈ Γ , are trained on the train setand expected to generalize to unseen samples (test set).
Revised: Deep Networks (DNs) models f γ , governed by some parameters γ ∈ Γ , are trained on the train set and expected to generalize to unseen samples (test set).

Original: To combat the tendency of DNs to overﬁt[48], producing a large train-test performance gap, regularization and in particular DA is heavilyemployed, amongst other mechanisms such as weight-decay [28, 21].
Revised: To combat the tendency of DNs to overﬁt [4], producing a large train-test performance gap, regularization and in particular DA is heavily employed, amongst other mechanisms such as weight-decay [5, 6].

Original: The beneﬁt of DA over thosealternatives is that deﬁning input transformations that preserve the semantics of their inputs is arelatively simple task, at least in computer vision or acoustic processing [17, 5, 35].
Revised: The beneﬁt of DA over those alternatives is that deﬁning input transformations that preserve the semantics of their inputs is a relatively simple task, at least in computer vision or acoustic processing [7, 8, 9].

Original: Furthermore, ifthe DA is well designed and rich enough, it can effectively bring the number of  training samples closeto the theoretical limits ensuring that any performance gap between the train and test set vanishes[42] as empirically observed in various scenarios [30, 39].
Revised: Furthermore, if the DA is well designed and rich enough, it can effectively bring the number of non-trivial training samples close to the theoretical limits ensuring that any performance gap between the train and test set vanishes [10] as empirically observed in various scenarios [11, 12].

Original: In fact, DA has proved so useful thatnovel training methods such as self-supervised learning (SSL) entirely rely on DA [24, 47] to learn ameaningful data representation.
Revised: In fact, DA has proven so useful that novel training methods such as self-supervised learning (SSL) entirely rely on DA [13, 14] to learn meaningful data representations.

Original: Despite its empirical effectiveness, our understanding of DA has many open questions, three of whichwe propose to study: (a) how do different DAs impact the model’s parameters during training?;(b) how sample-efﬁcient is the DA sampling, i.e., how many DA samples a model must observeto converge?; and (c) how sensitive is a loss/model to the DA sampling and how this varianceevolves during training as a function of the model’s ability to minimize the loss at hand, and as afunction of the model’s parameters?
Revised: Despite its empirical effectiveness, our understanding of DA has many open questions, three of which we propose to study: (a) how do different DAs impact the model’s parameters during training? ; (b) how sample-efﬁcient is the DA sampling, i.e., how many DA samples a model must observe to converge?; and (c) how sensitive is a loss/model to the DA sampling and how this variance evolves during training as a function of the model’s ability to minimize the loss at hand, and as a [Figure]

Original: Our goal is to analytically derive some preliminary answers andinsights into those three areas enabled by a novel image transformation operator that we introducein Section 3.2 coined Data-Space Transform (DST). DST employs a mathematical formulation ofinput transformations akin to the one of Chen et al. [4] although slightly more general as we do not original image
Revised: image

Original: I
Revised: I

Original: [Figure]
Revised: [Figure]

Original: transformed image 
Revised: im function of the model’s parameters? Our goal is to analytically derive some preliminary answers and insights into those three areas enabled by a novel image transformation operator that we introduce in Section 3.2 coined Data-Space Transform (DST).

Original: [Figure] [Figure] im require the DA to form a group.
Revised: DST employs a mathematical formulation of input transformations akin to the one of [15] although slightly more general as we do not require the DA to form a group.

Original: We study the zoom/translation/shearing/rotation DAs in the maintext and Gaussian blur, grayscale, color jitter and random crop in Appendix D.
Revised: We study the zoom/translation/shearing/rotation DAs in the main text and Gaussian blur, grayscale, color jitter and random crop in Appendix C.

Original: We summarize ourcontributions below:
Revised: We summarize our contributions below:

Original: (a) we derive the analytical ﬁrst order moments of augmented samples, and of the losses employingaugmented samples (Section 3.3), effectively providing us with the explicit regularizer inducedby each DA (Section 3.4)(b) we quantify the number of DA samples that are required for a model/loss to obtain a correctestimate of the information conveyed by that DA (Section 4.1)(c) we derive the sensitivity,
Revised: (a) we derive the analytical ﬁrst order moments of augmented samples, and of the losses employing augmented samples (Section 3.3), effectively providing us with the explicit regularizer induced by each DA (Section 3.4) (b) we quantify the number of DA samples that are required for a model/loss to obtain a correct estimate of the information conveyed by that DA (Section 4.1) (c) we derive the sensitivity,

Original: i
Revised: i

Original: e. variance, of a given loss and model under a DA policy (Section 4.2)leading us to rediscover from ﬁrst principles a popular deep network regularizer: TangentProp,as being the natural regularization to employ to minimize the loss variance (Section 4.3).
Revised: e. variance, of a given loss and model under a DA policy (Section 4.2) leading us to rediscover from ﬁrst principles a popular deep network regularizer: TangentProp, as being the natural regularization to employ to minimize the loss variance (Section 4.3).

Original: Upshot of gained insights: (a) shows us that the explicit DA regularizer corresponds to a generalized Tikhonov regularizer that depends on each sample’s covariance matrix largest eigenvectors; the kernelspace of the model’s Jacobian matrix aligns with the data manifold tangent space as modeled by the DA (b) shows us that the number of augmented samples required for a loss/model to correctly estimatethe information provided by a DA of a single sample is on the order of 10 4 .
Revised: Upshot of gained insights: (a) will show us that the explicit DA regularizer corresponds to a generalized Tikhonov regularizer that depends on each sample’s covariance matrix largest eigenvectors; the kernel space of the model’s Jacobian matrix aligns with the data manifold tangent space as modeled by the DA; (b) will show us that the number of augmented samples required for a loss/model to correctly estimate the information provided by a DA of a single sample is on the order of 10 4 .

Original: Even when combining theinformation of thousands of samples, we ﬁnd that the entire train set must be augmented at least 50 ×for a DA policy to be correctly learned by a model (c) quantiﬁes the loss variance as a simple functionof the model’s Jacobian matrix, and the eigenvectors of the augmented sample variance matrix.
Revised: Even when combining the information of thousands of samples, we ﬁnd that the entire train set must be augmented at least 50 × for a DA policy to be correctly learned by a model; lastly (c) will quantify the loss variance as a simple function of the model’s Jacobian matrix, and the eigenvectors of the augmented sample variance matrix.

Original: Regardless of the model or task-at-hand, the loss sensitivity to random DA sampling goes down asthe kernel of a model’s Jacobian matrix aligns with the principal directions of the data manifoldtangent space; as a by-product, we recover TangentProp as the natural regularizer minimizing the losssensitivity to DA sampling.
Revised: Regardless of the model or task-at-hand, the loss sensitivity to random DA sampling goes down as the kernel of a model’s Jacobian matrix aligns with the principal directions of the data manifold tangent space; as a by-product, we recover TangentProp as the natural regularizer minimizing the loss sensitivity to DA sampling.

Original: All of our image-space results e.g. expectation and variance of images under DA are exact, noapproximation is employed.
Revised: Limitations: all of our image-space results e.g. expectation and variance of images under DA are exact, no approximation is employed.

Original: Results on a model’s output/loss are provided both in the case of alinear model and of a nonlinear model.
Revised: Results on a model’s output/loss are provided both in the case of a linear model and of a nonlinear model.

Original: For the former, results are exact, for the latter a third-order Taylor approximations is employed which is common and has been shown to provide accurate enoughapproximations to safely rely on the insights/quantities obtained e.g. see Sec.
Revised: For the former, results are exact, for the latter a third-order Taylor approximations is employed which is common and has been shown to provide accurate enough approximations to safely rely on the insights/quantities obtained e.g. see Sec.

Original: A.2 of Wei et al.
Revised: A.2 of [16].

Original: All the proofs and implementation details are provided in the appendix.
Revised: All the proofs and implementation details are provided in the appendix.

Original: Full codebase will be publiclyreleased upon completion of the review process, a sample is provided in the supplementary material. We  emphasize that we are not proposing a tractable alternative to DA but a theoretical analysis onthe type of regularization that DA implicitly puts on your model, andhow to  visualize that regularizer.
Revised: We also emphasize that our primary goal is to provide a theoretical understanding of DA, to quantify its efﬁciency and impact onto learned models and to derive and visualize its explicit regularizer; we leave the search for tractable alternatives to DA based on those insights for future work.

Original:  Background
Revised:  Background

Original: Existing Explicit Regularizers From Data-Augmentation.
Revised: Existing Explicit Regularizers From Data-Augmentation.

Original: It is widely accepted that dataaugmentation (DA) regularizes a model towards the transformations that are modeled [28, 27],and this  impacts performances signiﬁcantly, possibly as much as the regularization offered by thechoice of DN architecture [9] and optimizer [37].
Revised: It is widely accepted that dataaugmentation (DA) regularizes a model towards the transformations that are modeled [5, 17], and that this regularization impacts performances signiﬁcantly and positively, possibly as much as the regularization offered by the choice of DN architecture [18] and optimizer [19].

Original: To gain precious insights into the impact of DA onto the learned functional f γ , the most common strategy is to derive the explicit regularizer that directly acts upon f γ in the same manner as if one were to use DA during training.
Revised: To gain precious insights into the impact of DA onto the learned functional f γ , the most common strategy is to derive the explicit regularizer that directly acts upon f γ in the same manner as if one were to use DA during training.

Original: This explicit derivation is however challenging and so far has been limited to DA strategies such as additive white noise or multiplicative binary noise applied identically and independently throughout the image, as with dropout[38, 2].
Revised: This explicit derivation is however challenging and so far has been limited to DA strategies such as additive white noise or multiplicative binary noise applied identically and independently throughout the image and/or feature maps, as with dropout [20, 21].

Original: In those settings, various works have studied in the linear regime the relation betweensuch data-augmentation and its equivalence to using Tikhonov [41] or weight decay [49] as inmin W (cid:80)Nn = [Equation] E (cid:15)
Revised: In those settings, various works have studied in the linear regime the relation between such DA and its equivalence to using Tikhonov regularization[22] or weight decay [23] as in min W (cid:80)

Original: [Equation] [Equation] ) (cid:2) (cid:107) y n − W ( xn + (cid:15) ) (cid:107) 22 (cid:3) = min
Revised: Nn =1 E (cid:15) ∼N (0 ,σ ) (cid:2) (cid:107) y n − W ( x

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: +

Original: [Equation]
Revised: [Equation]

Original:  (cid:107) y [Equation]
Revised: ) (cid:107) 22 (cid:3) =

Original:  W [Equation] (cid:107) 

Revised: min W (cid:80) Nn =1 (cid:107) y n − W x n (cid:107)

Original: + λ ( σ ) (cid:107) W(cid:107)
Revised: + λ ( σ ) (cid:107) W

Original: [Equation]
Revised: [Equation]

Original: F[1].
Revised: 2 F [24].

Original: More recently, Wang et al. [43], LeJeune et al. [20] extended the case of additive white noiseto nonlinear models and concluded that this DA corresponds to adding an explicit Frobenius normregularization onto the Jacobian of f θ evaluated at each data sample.
Revised: More recently, [25, 26] extended the case of additive white noise to nonlinear models and concluded that additive white noise DA corresponds to adding an explicit Frobenius norm regularization onto the Jacobian of f θ evaluated at each data sample.

Original: Going to more involved DA strategies e.g. translations or zooms of the input images is challengingand has so far only been studied from an empirical perspective.
Revised: Going to more involved DA strategies e.g. translations or zooms of the input images is challenging and has so far only been studied from an empirical perspective.

Original: For example, Hernández-Garcíaand König [14, 13, 15] performed thorough ablation studies on the interplay between DAs and acollection of known explicit regularizers to ﬁnd correlations between them.
Revised: For example, [27, 28, 29] performed thorough ablation studies on the interplay between DAs and a collection of known explicit regularizers to ﬁnd correlations between them.

Original: It was concluded thatweight-decay (the explicit regularizer of additive white noise) does not relate to those more advanced DAs.
Revised: It was concluded that weight-decay (the explicit regularizer of additive white noise) does not relate to those more advanced DAs.

Original: This also led other studies to suggest that norm-based regularization might be insufﬁcientto describe the implicit regularization of DAs involving advanced image transformations [31].
Revised: This also led other studies to suggest that norm-based regularization might be insufﬁcient to describe the implicit regularization of DAs involving advanced image transformations [30].

Original: Wedebunk this last claim in Sec. 3.3.
Revised: We debunk this last claim in Sec. 3.3.

Original: Coordinate Space Transformation.
Revised: Coordinate Space Transformation.

Original: Throughout this paper, we will consider a two-dimensionalimage I ( x, y ) to be at least square-integrable
Revised: Throughout this paper, we will consider a two-dimensional image I ( x, y ) to be at least square-integrable

Original: I ∈ L 2 ( R 2 ) [22].
Revised: I ∈ L 2 ( R 2 ) [31].

Original: Multi-channel images are dealtwith by treating each channel as its own single-channel image, as commonly done in practice [8].
Revised: Multi-channel images are dealt with by treating each channel as its own single-channel image, as commonly done in practice [1].

Original: As we are interested in practical cases, we will often assume that I has compact support e.g. hasnonzero values only within a bounded domain such as [0 , 1] 2 .
Revised: As we are interested in practical cases, we will often assume that I has compact support e.g. has nonzero values only within a bounded domain such as [0 , 1] 2 .

Original: Visualizing this image thus correspondsto displaying the sampled values of I on a regular grid (pixel positions) of [0 , 1] 2 [12].
Revised: Visualizing this image thus corresponds to displaying the sampled values of I on a regular grid (pixel positions) of [0 , 1] 2 [32].

Original: The mostcommon formulation to apply a transformation on the image I to obtain the transformed image Tis to transform the image coordinates [34, 45, 25].
Revised: The most common formulation to apply a transformation on the image I to obtain the transformed image T is to transform the image coordinates [33, 34, 35].

Original: That is, a mapping t : R 2 (cid:55)→ R 2 describes whatcoordinate t ( u, v ) of  I maps to the coordinate u, v of  T as in
Revised: That is, a mapping t : R 2 (cid:55)→ R 2 describes what coordinate t ( u, v ) of the original image I maps to the coordinate u, v of the transformed image T as in

Original: [Equation]
Revised: [Equation]

Original: This function t often has some parameter θ governing the underlying transformation as int θ ( x, y ) = [ x −
Revised: This mapping t often comes with some parameters θ governing the underlying transformation as in

Original: [Equation]
Revised: [Equation]

Original: 1 , y −

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: for translation, rotation and zoom respectively.
Revised: for translation, rotation and zoom respectively.

Original: We provide a visual depiction of the zoom transformation applied in coordinate space in Fig.
Revised: We provide a visual depiction of the zoom transformation applied in coordinate space in Fig.

Original: 6, in the appendix.
Revised: 6, in the appendix.

Original: The formulation of Eq.
Revised: The formulation of Eq.

Original: 1 has two keybeneﬁts.
Revised: 1 has two key beneﬁts.

Original: First, it allows a simple and intuitive design of t to obtain novel transformations.
Revised: First, it allows a simple and intuitive design of t to obtain novel transformations.

Original: Second, itis computationally efﬁcient as the coordinate-space of images are 2 / 3 -dimensional.
Revised: Second, it is computationally efﬁcient as the coordinate-space of images are 2 / 3 -dimensional.

Original: Those beneﬁtshave led to e.g. the Spatial Transformer Network [16].
Revised: Those beneﬁts have led to e.g. the Spatial Transformer Network [36].

Original: On the other hand, Eq. 1 has one majordrawback for our purpose: the exact moments of the transformed image under random θ parametersare not tractable due to the composition of t with the nonlinear mapping I .
Revised: On the other hand, Eq. 1 has one major drawback for our purpose: the exact moments of the transformed image under random θ parameters are not tractable due to the composition of t with the nonlinear mapping I .

Original: And as it will becomeclear in Section 3.1, computing such quantities is crucial for our study.
Revised: And as it will become clear in Section 3.1, those quantities are needed to derive DAs’ explicit regularizers. [Figure]

Original:  Analytical Moments of Transformed Images Enable Inﬁnite Dataset 
Revised:  Analytical Moments of Transformed Images Enable Inﬁnite Dataset Training

Original: Training

Original: We ﬁrst motivate this study by formulating the training process under DA sampling as doing a MonteCarlo estimate of the true (unknown) expected loss under that DA distribution (Section 3.1).
Revised: We ﬁrst motivate this study by formulating the training process under DA sampling as doing a Monte-Carlo estimate of the true (unknown) expected loss under that DA distribution (Section 3.1).

Original: Goingbeyond this sampling/estimation procedures requires knowledge of the expectation and varianceof a transformed sample, motivating the construction of a novel Data-Space Transformation (DTS)(Section 3.2) that allow for a close-form formula of those moments (Section 3.3).
Revised: Going beyond this sampling/estimation procedures requires knowledge of the average and variance of a transformed sample, i.e. the ﬁrst two centered-moments. This is our motivation to construct a novel Data-Space Transformation (DST) (Section 3.2) that will enable close-form formula for those moments (Section 3.3).

Original: From those results,we will be able to remove the need to sample transformed images to train a model, by obtaining the[Figure] close-form expected loss Section 3.4, under which training can be done directly, in close-form forlinear model and with the usual gradient-based learning in general.
Revised: From those, we will be able to remove the need to sample transformed images to train a model, by obtaining the close-form expected loss Section 3.4, under which training can be performed sampling-free.

Original: 3.1 Motivation: Current Data-Augmentation Training Performs Monte-Carlo Estimation
Revised: 3.1 Motivation: Current Data-Augmentation Training Performs Monte-Carlo Estimation

Original: Training a model with DA consists in (i) sampling transformed images at each training iteration foreach sample x n as in T θ n ( x n ) with θ n ∼ θ a randomly sampled DA parameter e.g. the amount oftranslation to apply, (ii) evaluating the loss L on the transformed sample/mini-batch/dataset, and (iii)using some ﬂavor of gradient descent to update the parameters γ of the model f γ .
Revised: Training a model with DA consists in (i) sampling transformed images at each training iteration for each sample x n as in T θ n ( x n ) with θ n ∼ θ a randomly sampled DA parameter e.g. the amount of translation to apply, (ii) evaluating the loss L on the transformed sample/mini-batch/dataset, and (iii) using some ﬂavor of gradient descent to update the parameters γ of the model f γ .

Original: This training procedure corresponds to a one-sample Monte-Carlo (MC) estimate [23, 10] of theexpected loss
Revised: This training procedure corresponds to a one-sample Monte-Carlo (MC) estimate [37, 38] of the expected loss

Original: [Equation]
Revised: [Equation]

Original: with θ n i.i.d samples of θ .
Revised: with θ n i.i.d samples of θ repeatedly sampled at each epoch.

Original: In a supervised setting, the loss would also receive a per-sample target y nas input.
Revised: In a supervised setting, the loss would also receive a per-sample target y n as input.

Original: Although a one-sample estimate might be insufﬁcient to apply the central limit theorem[33] and guarantee training convergence, the combination of multiple samples in mini-batch trainingdoes provide convergence in most cases.
Revised: Although a one-sample estimate might be insufﬁcient to apply the central limit theorem [39] and guarantee training convergence, the combination of multiple samples in mini-batch training does provide convergence in most cases.

Original: Yet, methods such as Self-Supervised Learning thatheavily rely on DA, tend to be more sensitive and unstable [40].
Revised: Although, methods such as Self-Supervised Learning that heavily rely on DA, tend to be more sensitive and unstable [40].

Original: To avoid such instabilities, andto incorporate all the DA transformations of x n into the model’s parameter update, one would betempted to compute the model parameters’ gradient on the expected loss (left-hand side of Eq.
Revised: To avoid such instabilities, and to incorporate all the DA transformations of x n into the model’s parameter update, one would be tempted to compute the model parameters’ gradient on the expected loss (left-hand side of Eq.

Original: (2))(see Section 3.4).
Revised: (see Section 3.4).

Original: Knowledge of the expected loss would also prove useful to measure the quality ofthe MC estimate (see Section 4.1).
Revised: Knowledge of the expected loss would also prove useful to measure the quality of the MC estimate (see Section 4.1).

Original: Note that even in the least-square setting, such approximationis employed as the expected loss under DAs is not know.
Revised: Note that even in the least-square setting, such approximation is employed as the expected loss under DAs is not know.

Original: Hence, one commonly sample many DAsamples, and then apply the least-square formula on this augmented training set, which is highlyinefﬁcient.
Revised: Hence, one commonly obtain many DA samples, and then apply the least-square formula on this augmented training set, which is highly inefﬁcient.

Original: Alternatively, this study will obtain the optimum least-square parameters under DAsampling (Theorem 3.4) i.e. only using the given training set, we directly obtain the parametersthat would correspond to sampling inﬁnitely many DA samples and applying the usual least-squareformula on this inﬁnite training set.
Revised: Alternatively, this study will obtain the optimum least-square parameters under DA sampling (Theorem 3.4) i.e. only using the given training set, we directly obtain the parameters that would correspond to sampling inﬁnitely many DA samples and applying the usual least-square formula on this inﬁnite training set.

Original: The close-form expected loss requires knowledge of the expectation and variance of the transformedsample T θ ( x n ) , hence, we ﬁrst propose to formulate a novel and tractable augmentation model(Section 3.2) that will allow us to obtain those moments analytically in Section 3.3.
Revised: The close-form expected loss requires knowledge of the average and variance of the transformed sample T θ ( x n ) taken with respect to the random variable θ . Hence, we ﬁrst propose to formulate a novel and tractable augmentation model (Section 3.2) that will allow us to obtain those moments analytically in Section 3.3.

Original: 3.2 Proposed Data-Space Transformation 
Revised: 3.2 Proposed Data-Space Transformation (DST)

Original: Instead of altering the coordinate positions of an image, as done in the coordinate-space transformationof Eq.
Revised: Instead of altering the coordinate positions of an image, as done in the coordinate-space transformation of Eq.

Original: (1), we propose to alter the image basis functions.
Revised: (1), we propose to alter the image basis functions.

Original: Going back to the construction of functions, one easily recalls that any (image) function can beexpanded into a basis as in I ( u, v ) =
Revised: Going back to the construction of functions, one easily recalls that any (image) function can be expanded into a basis as in I ( u, v ) =

Original: (cid:82) I ( x, y )
Revised: (cid:82) I ( x, y )

Original: δ ( u − x, v − y ) dxdy with δ the usual Dirac distributions.
Revised: δ ( u − x, v − y ) dxdy with δ the usual Dirac distribution.

Original: Suppose for now that we consider a horizontal translation by a constant θ .
Revised: Suppose for now that we consider a horizontal translation by a constant θ .

Original: Then, one can obtain thattranslated image via
Revised: Then, one can obtain that translated image via

Original: [Equation]
Revised: [Equation]

Original: hence by moving the basis functions onto which the image is evaluated, rather than by moving theimage’s coordinate.
Revised: hence, and crucial to our study, we apply the transformation onto the basis functions onto which the image is evaluated, rather than onto the original image itself .

Original: As the image is now constant with respect to the transformation parameter θ , andas the basis functions have some speciﬁc analytical forms, the derivation of the transformed imagesexpectation and variance, underrandom θ , will become straightforward.
Revised: As the image is now constant with respect to the transformation parameter θ , and as the basis functions have some convenient analytical forms, the derivation of the transformed images moments, under θ , will become straightforward.

Original: Because the transformedimage T is now obtained by combining its pixel values (recall Eq.
Revised: Because the transformed image T is now obtained by combining its pixel values (recall Eq.

Original: we coin this transform as Data-Space Transform (DST) and we formally deﬁne it below.
Revised: we coin this transform as Data-Space Transform (DST), formally deﬁned below.

Original: Deﬁnition 3.1 (Data-Space Transform) .
Revised: Deﬁnition 3.1 (Data-Space Transform) .

Original: We deﬁne the data-space transformation of an image I ∈
Revised: We deﬁne the DST of an image I ∈

Original: L 2 ( R 2 ) producing the transformed image T θ ( I )
Revised: L 2 ( R 2 ) producing the transformed image T θ ( I )

Original: ∈
Revised: ∈

Original: L 2 ( R 2 ) as
Revised: L 2 ( R 2 ) as

Original: [Equation]
Revised: [Equation]

Original:  with h θ ( u, v, ., . )
Revised: 5) with h θ ( u, v, ., . )

Original: ∈ C ∞ 

Revised: ∈ C ∞

Original: ( R 2 ) encoding the transformation.
Revised: ( R 2 ) encoding the transformation.

Original: In Deﬁnition 3.1, we only impose for h θ ( u, v, ., . ) to be with compact support.
Revised: In Deﬁnition 3.1, we only impose for h θ ( u, v, ., . ) to be with compact support.

Original: In fact, one shouldinterpret h θ ( u, v, ., . )
Revised: In fact, one should interpret h θ ( u, v, ., . )

Original: as a distribution whose purpose is to evaluate I at a desired (coordinate) positionon its domain.
Revised: as a distribution whose purpose is to evaluate I at a desired (coordinate) position on its domain.

Original: This evaluation -depending on the form of h θ ( u, v, ., . ) - can return a single pixel-valueof the image I at a desired location (as in Eq.
Revised: This evaluation –depending on the form of h θ ( u, v, ., . ) – can extract a single pixelvalue of the image I at a desired location (as in Eq.

Original: (3)), orit can combine multiple values e.g. withh θ ( u, v, ., . )
Revised: (4)), or can combine multiple values e.g. with h θ ( u, v, ., . )

Original: being a bump function.
Revised: being a bump function.

Original: Additionally, we do not impose any restriction on the invertibilityof this mapping i.e. T θ ( I )( u, v ) can disregard parts of the original image.
Revised: Additionally, we do not impose any restriction on the invertibility of this mapping i.e. T θ ( I )( u, v ) can disregard parts of the original image.

Original: Coordinate-space transformations as DSTs.
Revised: Coordinate-space transformations as DSTs.

Original: The coordinate-space transformation and the proposed DST (Deﬁnition 3.1) act in different spaces: the image coordinates and the image pixel values,respectively.
Revised: The coordinate-space transformation and the proposed DST (Deﬁnition 3.1) act in different spaces: the image coordinates and the image pixel values, respectively.

Original: Nevertheless, this does not limit the range of transformations that can be applied to an image.
Revised: Nevertheless, this does not limit the range of transformations that can be applied to an image.

Original: The following statement provides a simple recipe to turn any already employed coordinate-spacetransformation into a data-space one.
Revised: The following statement provides a simple recipe to turn any already employed coordinate-space transformation into a DST.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Any coordinate-space transformation (1) using t θ : R 2 (cid:55)→ R 2 can be expressed asa data-space transformation (4) by setting h θ ( u, v, x, y )
Revised: Any coordinate-space transformation (1) using t θ : R 2 (cid:55)→ R 2 can be expressed as a data-space transformation (5) by setting h θ ( u, v, x, y )

Original: =
Revised: =

Original: δ ( t θ ( x, y ) − [ u, v ] T ) .
Revised: δ ( t θ ( x, y ) − [ u, v ] T ) .

Original: We will focus in the main text on zoom/rotation/translation/shearing, and we propose in Appendix Dthe case of Gaussian blur, color jittering, random crop, along with a discussion on extending our resultsto CutMix [46] and MixUp [50].
Revised: We will focus in the main text on zoom/rotation/translation/shearing, and we propose in Appendix C the case of Gaussian blur, color jittering, random crop, along with a discussion on extending our results to CutMix [41] and MixUp [42].

Original: Using Proposition 3.2, we obtain the DST operators h θ ( u, v, x, y )to be
Revised: Using Proposition 3.2, we obtain the DST operators h θ ( u, v, x, y ) to be

Original: [Equation]
Revised: [Equation]

Original: for the vertical/horizontal translation, vertical/horizontal shearing, zoom and rotation respectively.
Revised: for the vertical/horizontal translation, vertical/horizontal shearing, zoom and rotation respectively (compare with Eq. 2).

Original: Before focusing on the analytical moments of the data-space transformed samples, we describe howthose operators are applied in a discrete setting.
Revised: Before focusing on the analytical moments of the DST samples, we describe how those operators are applied in a discrete setting.

Original: Discretized version.
Revised: Discretized version.

Original: We now describe how any DST of a discrete image, ﬂattened as a vector, canbe expressed as a matrix-vector product with the matrix entries depending on the employed DAand its parameter θ , as  in Figs.
Revised: We now describe how any DST of a discrete image, ﬂattened as a vector, can be expressed as a matrix-vector product with the matrix entries depending on the employed DA and its parameter θ , as we display in Figs.

Original: 1 and 2.
Revised: 1 and 2.

Original: The functional form of the proposed transform in Eq.
Revised: The functional form of the DST from Eq.

Original: (4)producing the target image at a speciﬁc position T θ ( I )( u, v ) is linear in the  imageI .
Revised: (5) producing the target image at a speciﬁc position T θ ( I )( u, v ) is linear in the original image

Original: The continuousintegral over the image domain is replaced with a summation with indices based on the desired sampling/resolution of I .
Revised: I . Hence, in the discrete setting, the continuous integral over the image domain is replaced with a summation with indices based on the desired sampling/resolution of I , i.e. the position of the pixel spatial positions.

Original: Expressing linear operators as matrix-vector products will greatly easeour development, we will denote by x ∈ R hw the ﬂattened ( h × w ) discrete images I .
Revised: Expressing linear operators as matrix-vector products will greatly ease our development, we will denote by x ∈ R hw the ﬂattened ( h × w ) discrete images I .

Original: Hence, ourdata-space transformation, given some parameters θ , takes the form of
Revised: Hence, our data-space transformation, given some parameters θ , takes the form of

Original: [Equation]
Revised: [Equation]

Original: (6) with t ( θ ) ∈ R hw the ﬂattened transformed image (which can be reshaped as desired) and M ( θ ) ∈
Revised: with t ( θ ) ∈ R hw the ﬂattened transformed image (which can be reshaped as desired) and M ( θ ) ∈

Original: R hw × hw the matrix whose rows encode the discrete and ﬂattened h θ ( u, v, ., . ) .
Revised: R hw × hw the matrix whose rows encode the discrete and ﬂattened h θ ( u, v, ., . ) .

Original: For example, andemploying a uniform grid sampling for illustration, M ( θ ) i,j = h θ ( i//w, i % w, j//w, j % w ) with //representing the ﬂoor division and % the modulo operation.
Revised: For example, and employing a uniform grid sampling for illustration, M ( θ ) i,j = h θ ( i//w, i % w, j//w, j % w ) with // representing the ﬂoor division and % the modulo operation.

Original: For the case of multi-channel images, weconsider without loss of generality the application of the matrix-vector operation on each channelseparately.
Revised: For the case of multi-channel images, we consider without loss of generality the application of the matrix-vector operation on each channel separately.

Original: We depict this operation along with the exact form of M ( θ ) for the case of the zoomtransformation in Fig.
Revised: We depict this operation along with the exact form of M ( θ ) for the case of the zoom transformation in Fig.

Revised: A crucial property of the matrix M ( θ ) lies in its sparsity, in fact, for most

Revised: DAs the transformations rely on displacing pixels rather than combining them.

Revised: Hence, although

Revised: M ( θ ) ’s total number of entries grow quadratically with the number of pixels in I , the number of nonzero entries only grows linearly with it.

Original: 3.3 Analytical Expectation and Variance of Transformed Images
Revised: 3.3 Analytical Expectation and Variance of Transformed Images

Original: The above construction (Deﬁnition 3.1) turns out to make the analytical form of the ﬁrst two momentsof an augmented sample much simpler to derive.
Revised: The above construction (Deﬁnition 3.1) turns out to make the analytical form of the ﬁrst two moments of an augmented sample much simpler to derive.

Original: As this derivation is at the core of our maincontribution, we propose a step-by-step derivation of E θ
Revised: As this derivation is at the core of our main contribution, we propose a step-by-step derivation of E θ

Original: [ T θ ( I )] for the case of horizontal translation(as in Eq.
Revised: [ T θ ( I )] for the case of horizontal translation (recall Eq.

Original: Let’s consider again the continuous model (the discrete version is provided after Theorem 3.3).
Revised: Let’s consider again the continuous model (the discrete version is provided after Theorem 3.3).

Original: Using
Revised: Using

Original: Fubini’s theorem to switch the order of integration and recalling Eq.
Revised: Fubini’s theorem to switch the order of integration and recalling Eq.

Original: (4), we have that
Revised: (5), we have that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using the deﬁnition for h θ from Eq.
Revised: Using the deﬁnition for h θ from Eq.

Original: (5) for horizontal translation, E θ [ h θ ( u, v, x, y )] becomesE θ [
Revised: (6) for horizontal translation, E θ [ h θ ( u, v, x, y )] becomes

Original: [Equation]
Revised: [Equation]

Original: with p the density function of θ prescribing how the translation parameter is distributed.
Revised: with p the density function of θ prescribing how the translation parameter is distributed.

Original: As a result,in this univariate translation case, the expected augmented image at coordinate ( u, v ) is given by
Revised: As a result, in this univariate translation case, the expected augmented image at coordinate ( u, v ) is given by

Original: [Equation]
Revised: [Equation]

Original:  which can be further simpliﬁed into E θ
Revised: (9) which can be further simpliﬁed into E θ

Original: [ T θ ( I )( ., v )]
Revised: [ T θ ( I )( ., v )]

Original: = I ( ., v ) (cid:63) p .
Revised: = I ( ., v ) (cid:63) p .

Original: Hence, the expected x -translatedimage is the convolution (on the x -axis only) between the original image I and the univariate densityfunction p .
Revised: Hence, the expected horizontallytranslated image is the convolution (on the x -axis only) between the original image I and the univariate density function p .

Original: We formalize this for the transformations of Eq.
Revised: We formalize this for the transformations of Eq.

Original: (5) below.
Revised: (6) below.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: The analytical form of E θ [ h θ ( u, v, x, y )] , used to obtain the expected transformedimage (recall Eq.
Revised: The analytical form of E θ [ h θ ( u, v, x, y )] , used to obtain the expected transformed image (recall Eq.

Original: (7) ) is given by
Revised: (8) ) is given by

Original: [Equation]

Original: translation)

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: other cases and  E θ
Revised: other cases and second-order moment E θ

Original: [ T θ ( x n ) T θ ( x n ) T ] are deferred to the proof.
Revised: [ T θ [Equation] x n ) T θ ( x n ) T ] are deferred to the proof.

Original: (Proof in Appendix G.)
Revised: (Proof in Appendix F.) t h i r d s ec ond ﬁ r s t rotation shear r a nk o f V [ T ( x ) ] transformation ampl.

Original: Notice for example how the expected image under random 2 -dimensional translations with ( 2dimensional) density p is given by the convolution I (cid:63) p , providing a new portal to study and interpretconvolutions with nonnegative, sum-to-one ﬁlters p .
Revised: An interesting observation obtained from the above ties the expected image under random 2dimensional translations with ( 2 -dimensional) density p to the convolution I (cid:63) p ; this provides a new portal to study and interpret convolutions with nonnegative, sum-to-one ﬁlters p .

Original: Again, and as per Eq.
Revised: Again, and as per Eq.

Original: (6), the discretized versionof the expected image takes the form of E θ [ t ( θ )]
Revised: (7), the discretized version of the expected image takes the form of E θ [ t ( θ )]

Original: =
Revised: =

Original: E θ [ M ( θ )]
Revised: E θ [ M ( θ )]

Original: x with the entries of E θ [ M ( θ )] givenby discretizing Eq.
Revised: x with the entries of E θ [ M ( θ )] given by discretizing Eq.

Original: We depict this expected matrix for various transformations as well as theirapplication onto two different discrete images in Fig. 2, and we now proceed on deriving the left-handside of Eq.
Revised: We depict this expected matrix for various transformations as well as their application onto two different discrete images in Fig. 2, and we now proceed on deriving the left-hand side of Eq.

Original: (2) i.e. the explicit DA regularizer.
Revised: (3) i.e. the explicit DA regularizer.

Original: 3.4 The Explicit Regularizer of Data-Augmentations
Revised: 3.4 The Explicit Regularizer of Data-Augmentations

Original: To keep notations as light as possible, we ﬁrst consider a linear regression model with Mean Squared
Revised: To keep notations as light as possible, we ﬁrst consider a linear regression model with Mean Squared

Original: Error (MSE), the nonlinear case will be studied in Section 4.
Revised: Error (MSE), the nonlinear case will be studied in Section 4.

Original: In that setting, the expected loss under
Revised: In that setting, the expected loss under

Original: DA sampling (recall Eq.
Revised: DA sampling (recall Eq.

Original: (2)) becomes
Revised: (3)) becomes

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: t h i r d s ec ond ﬁ r s t rotation shear r a nk o f V [ T ( x ) ] transformation ampl.

Original: with x n ∈ R D , n = 1 , . . .
Revised: with x n ∈ R D , n = 1 , . . .

Original: , N the input (ﬂattened)
Revised: , N the input (ﬂattened)

Original: n th image I n , y n ∈ R K the n th target vector, and W ∈ R K × D , b ∈ R K the model’s parameters.
Revised: n th image I n , y n ∈ R K the n th target vector, and W ∈ R K × D , b ∈ R K the model’s parameters.

Original: Recall from Section 3.1 that to learn W under DA,the current method consists in performing a MC estimation.
Revised: Recall from Section 3.1 that to learn W under DA, the current strategy consists in performing a Monte-Carlo (MC) estimation.

Original: Instead, let’s derive (detailed derivationin Appendix E) the exact loss of Eq.
Revised: Instead, let’s derive (detailed derivation in Appendix D) the exact loss of Eq.

Original: (10) as a function of the sample mean and variance under theconsider DA.
Revised: (11) as a function of the sample mean and variance under the consider DA.

Original: We will drop the θ subscript for clarity to obtain
Revised: We will drop the θ subscript for clarity to obtain

Original: [Equation]
Revised: [Equation]

Original:  with the spectral decomposition Q ( x )Λ( x ) Q ( x ) T = V [ T ( x )] .
Revised: 12) with the spectral decomposition Q ( x )Λ( x ) Q ( x ) T = V [ T ( x )] .

Original: The right term in Eq.
Revised: The right term in Eq.

Original: (11) is theexplicit DA regularizer.
Revised: (12) is the explicit DA regularizer.

Original: It pushes the kernel space of W to align with the largest principal directionsof the data manifold tangent space, as modeled by the DA.
Revised: It pushes the kernel space of W to align with the largest principal directions of the data manifold tangent space, as modeled by the DA.

Original: In fact, the largest eigenvectors in Q ( x )represent the principal directions of the data manifold tangent space at x , as encoded via V [ T ( x )] .
Revised: In fact, the largest eigenvectors in Q ( x ) represent the principal directions of the data manifold tangent space at x , as encoded via V [ T ( x )] .

Original: We propose in Fig.
Revised: We propose in Fig.

Original: 3 visualization of Q and Λ for different DAs, illustrating how each DA policyimpacts the model’s parameter W through the regularization of Eq.
Revised: 3 visualization of Q and Λ for different DAs, illustrating how each DA policy impacts the model’s parameter W through the regularization of Eq.

Original: The knowledge of E [ T ( x )]and V [ T ( x )] from Theorem 3.3 ﬁnally enables to train a (linear) model on the true expected loss(Eq.
Revised: The knowledge of E [ T ( x )] and V [ T ( x )] from Theorem 3.3 ﬁnally enables to train a (linear) model on the true expected loss (Eq.

Original: as we formalize below.
Revised: as we formalize below.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: Training a linear model with MSE and inﬁnite DA sampling is equivalent to minimizing
Revised: Training a linear model with MSE and inﬁnite DA sampling is equivalent to minimizing

Original: Eq.
Revised: Eq.

Original: (11) and produces the optimal W ∗ model’s parameter
Revised: (12) and produces the optimal W ∗ model ’s parameter

Original: [Equation]
Revised: [Equation]

Original: Not surprisingly, whenever the DA is identity, E [ T ( x n )] = x n , V [ T ( x n )] = 0 and thus Eq.
Revised: Not surprisingly, whenever the DA is identity, E [ T ( x n )] = x n , V [ T ( x n )] = 0 and thus Eq.

Original: (12)recovers exactly the standard least-square solution.
Revised: (13) recovers exactly the standard least-square solution.

Original: We visualize V [ T ( x n )] for the translation DA in
Revised: We visualize V [ T ( x n )] for the translation DA in

Original: Fig.
Revised: Fig.

Original: The same line of result can be derived in the nonlinear setting by assuming that the DA isrestricted to small transformations.
Revised: The same line of result can be derived in the nonlinear setting by assuming that the DA is restricted to small transformations.

Original: In that case, one leverages a truncated Taylor approximation ofthe nonlinear model 1 and recovers that (local) DA applies the same regularization as in Eq.
Revised: In that case, one leverages a truncated Taylor approximation of the nonlinear model 1 and recovers that (local) DA applies the same regularization as in Eq.

Original: (11) butwith the model’s Jacobian matrix J f γ ( x n ) in-place of W (more details in Section 4.3).
Revised: (12) but with the model’s Jacobian matrix J f γ ( x n ) in-place of W (more details in Section 4.3).

Original: Given the above derivation of the exact expected loss, we can now turn to precisely measure howaccurate is the MC estimate commonly used to train models under DA sampling.
Revised: Given the above derivation of the exact expected loss, we can now turn to precisely measure how accurate is the MC estimate commonly used to train models under DA sampling.

Original:  Data-Augmentation Sampling Efﬁciency and Loss Sensitivity
Revised:  Data-Augmentation Sampling Efﬁciency and Loss Sensitivity

Original: In this section we study the convergence of the MC estimate (Section 4.1), and provide varianceanalysis of that estimate as a function of the model’s Jacobian matrix and the sample variance[Figure] eigenvectors (Section 4.2), concluding by the (re-)discovery of TangentProp from ﬁrst principles(Section 4.3).
Revised: In this section we study the convergence of the MC estimate (Section 4.1), and provide variance analysis of that estimate as a function of the model’s Jacobian matrix and the sample variance eigenvectors (Section 4.2), concluding by the (re-)discovery of TangentProp from ﬁrst principles (Section 4.3).

Original: Those results hold regardless of the type of DA employed.
Revised: Those results hold regardless of the type of DA employed.

Original: Additionally, the analysis isdone on a single sample to ease notation, in the i.i.d.
Revised: Additionally, the analysis is done on a single sample to ease notation, in the i.i.d.

Original: setting, simply sum over all the training samplesto analyze the training set.
Revised: setting, simply sum over all the training samples to analyze the training set.

Original: 4.1 Empirical Monte-Carlo Convergence of Transformed Images
Revised: 4.1 Empirical Monte-Carlo Convergence of Transformed Images

Original: Given the close-form average image and average loss , we empirically measure how efﬁcient is the
Revised: Given the close-form average image and average loss , we empirically measure how efﬁcient is the

Original: MC estimation from Eq.
Revised: MC estimation from Eq.

Original: We ﬁrst propose in Fig.
Revised: We ﬁrst propose in Fig.

Original: 4 a constructed (64 × 64) image for which we compute the expected loss(Eq.
Revised: 4 a constructed (64 × 64) image for which we compute the expected loss (Eq.

Original: and the MC estimate (right-hand side of Eq.
Revised: and the MC estimate (right-hand side of Eq.

Original: Surprisingly, we obtain that even forsuch a simple image and translation DA, between 1000 and 10000 samples are required to correctlyestimate the MSE loss from the augmented samples.
Revised: Surprisingly, we obtain that even for such a simple image and translation DA, between 1000 and 10000 samples are required to correctly estimate the MSE loss from the augmented samples.

Original: In a more practical scenario, one could rightfullyargue that the combination of the DA samples from different images allows to obtain a better estimatewith a smaller amount of augmentations per sample.
Revised: In a more practical scenario, one could rightfully argue that the combination of the DA samples from different images allows to obtain a better estimate with a smaller amount of augmentations per sample.

Original: Hence, we provide in Fig.
Revised: Hence, we provide in Fig.

Original: 5 that experimentusing a linear model on MNIST [19] with varying train set size.
Revised: 5 that experiment using a linear model on MNIST [45] with varying train set size.

Original: We observe that as the numberof samples grows as the required number of augmentation per sample reduces.
Revised: We observe that as the number of samples grows as the required number of augmentation per sample reduces.

Original: Nevertheless, evenwith thousands of samples, at least 50 augmentations per sample are required to provide an accurateestimate (reproduction with nonlinear models given in Fig. 7).
Revised: Nevertheless, even with thousands of samples, at least 50 augmentations per sample are required to provide an accurate estimate (reproduction with nonlinear models given in Fig. 7).

Original: We now propose to speciﬁcally quantify the sensitivity of the MC estimate to DA sampling.
Revised: We now propose to speciﬁcally quantify the sensitivity of the MC estimate to DA sampling.

Original: 4.2 Loss Sensitivity Under Data-Augmentation Sampling in the Linear and NonlinearRegime
Revised: 4.2 Loss Sensitivity Under Data-Augmentation Sampling in the Linear and Nonlinear Regime

Original: Recalling Section 3.1, current DA training is performed on a MC estimate of the loss.
Revised: Recalling Section 3.1, current DA training is performed on a MC estimate of the loss.

Original: The estimator’svariance [26] is proportional to the variance of the quantity being estimated:
Revised: The estimator’s variance [46] is proportional to the variance of the quantity being estimated:

Original: V [( L ◦ f )( T ( x ))] .
Revised: V [( L ◦ f )( T ( x ))] .

Original: Wenow characterize when, and why, would an MC estimator converge for a given model.
Revised: We now characterize when, and why, would an MC estimator converge for a given model.

Original: By leveraging the delta method [6, 29] i.e. a truncated Taylor expansion of the model and lossfunction mapping as L ◦ f , we have
Revised: By leveraging the delta method [47, 48] i.e. a truncated Taylor expansion of the model and loss function mapping as L ◦ f , we have

Original: [Equation]
Revised: [Equation]

Original: (13) with (cid:107) u (cid:107) 2 A (cid:44) u T Au .
Revised: (14) with (cid:107) u (cid:107) 2 A (cid:44) u T Au .

Original: Noticing that V [ T ( x )]
Revised: Noticing that V [ T ( x )]

Original: =
Revised: =

Original: E [ T ( x ) T ( x ) T ] − E
Revised: E [ T ( x ) T ( x ) T ] − E

Original: [ T ( x )] E
Revised: [ T ( x )] E

Original: [ T ( x )]
Revised: [ T ( x )]

Original: T and using theclosed-form moments of augmented samples from Theorem 3.3, it is possible to write out Eq.
Revised: T and using the close-form moments of DST samples from Theorem 3.3, it is possible to write out Eq.

Original: (13)explicitly for model and DA speciﬁc analysis.
Revised: (14) explicitly for model and DA speciﬁc analysis.

Original: We visualize V
Revised: We visualize V

Original: [ T ( x )] in Fig. 3.
Revised: [ T ( x )] in Fig. 3.

Original: Linear regression case.
Revised: Linear regression case.

Original: To gain some insight into Eq.
Revised: To gain some insight into Eq.

Original: (13), let’s ﬁrst consider the linear regressioncase leading to V
Revised: (14), let’s ﬁrst consider the linear regression case leading to V

Original: [( L ◦ f )( T ( x ))]
Revised: [( L ◦ f )( T ( x ))]

Original: ≈ (cid:107)
Revised: ≈ (cid:107)

Original: y − W E
Revised: y − W E

Original: [ T ( x )] − b (cid:107) 2 W V [ T ( x )] W T .
Revised: [ T ( x )] − b (cid:107) 2 W V [ T ( x )] W T .

Original: Hence, the estimated lossvariance depends  on the model predicting the correct output when observing the expected sample282 E [ T ( x )] , and  on the smallest right singular vectors of W to align with the largest eigenvectors of283 V [ T ( x )] , echoing our observation below Eq. 
Revised: Hence, the estimated loss variance depends (i) on the model predicting the correct output when observing the expected sample E [ T ( x )] , and (ii) on the smallest right singular vector of W to align with the largest eigenvectors of V [ T ( x )] , echoing our observation below Eq. [Figure] See

Revised: 2 to 5 for

Original: General case.
Revised: General case.

Original: Since the result from Eq.
Revised: Since the result from Eq.

Original: (13) holds in the general setting, we can generalize the aboveobservation to the more general setting.
Revised: (14) holds in the general setting, we can generalize the above observation and formalize it into the following statement.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: The variance of the loss’s MC estimate (recall Eq.
Revised: The variance of the loss’s MC estimate (recall Eq.

Original: (2) ) for an input x n goes to 0if the loss gradient ∇L ( f ( E [ T ( x )])) goes to 0 , or if the kernel of the model’s Jacobian matrix
Revised: (3) ) for an input x n goes toif the loss gradient ∇L ( f ( E [ T ( x )])) goes to 0 , or if the kernel of the model’s Jacobian matrix

Original: J f ( E [ T ( x )]) aligns with the largest eigenvectors of the sample variance V [ T ( x )] .
Revised: J f ( E [ T ( x )]) aligns with the largest eigenvectors of the sample variance V [ T ( x )] .

Original: The proof of the above statement simply starts from Eq.
Revised: The proof of the above statement simply starts from Eq.

Original:  and applies the Cauchy-Schwarzinequality two times to obtain
Revised: (14) and applies the Cauchy-Schwarz inequality two times to obtain

Original: [Equation]
Revised: [Equation]

Original: with  V [ T ( x )] = Q Λ Q  T .
Revised: with decomposition V [ T ( x )] = Q ( x )Λ( x ) Q ( x ) T .

Original: From Eq.
Revised: From Eq.

Original: (14), it is quite realistic to assume that the model’s Jacobian doesnot collapse to 0 (this would not minimize the training loss in general).
Revised: (15), it is quite realistic to assume that the model’s Jacobian does not collapse to 0 (this would not minimize the training loss in general).

Original: Hence, the most realistic caseof Theorem 4.1 concerns the alignment between the model’s Jacobian matrix kernel space and thetop eigenvectors of V [ T ( x )] .
Revised: Hence, the most realistic case of Theorem 4.1 concerns the alignment between the model’s Jacobian matrix kernel space and the top eigenvectors of V [ T ( x )] .

Original: However, for models like ResNet DNs [11], the loss variance can onlyreduce to 0 if the model minimizes the loss at the expected input since the Jacobian matrix is alwaysfull-rank.
Revised: However, for models like ResNet DNs [49], the loss variance can only reduce to 0 if the model minimizes the loss at the expected input since the Jacobian matrix is always full-rank.

Original: We recall that the above relied on the delta method to approximate the intractable LHS of Eq.
Revised: We recall that the above relied on the delta method to approximate the intractable LHS of Eq.

Original: (13) (see footnote on Page 7) .
Revised: (14) (see footnote on Page 7) .

Original: 4.3 Explicit Loss Sensitivity Minimization Provably Recovers TangentProp
Revised: 4.3 Explicit Loss Sensitivity Minimization Provably Recovers TangentProp

Original: The expected loss (Eq. (2)) has been derived in the linear regression setting (Eq.
Revised: The expected loss (Eq. (3)) has been derived in the linear regression setting (Eq.

Original: But in a moregeneral scenario, and as discussed below Theorem 3.4, this expectation might not be tractable andthus needs to be approximated e.g. based on a Taylor expansion of the loss and model.
Revised: But in a more general scenario, and as discussed below Theorem 3.4, this expectation might not be tractable and thus needs to be approximated e.g. based on a Taylor expansion of the loss and model.

Original: Alternativelyto using the approximated expectation, one could employ the usual MC estimate of the expectation(right-hand side of Eq.
Revised: Alternatively to using the approximated expectation, one could employ the usual MC estimate of the expectation (right-hand side of Eq.

Original: (2)), and leverage the MC estimator variance obtained in Eq.
Revised: (3)), and leverage the MC estimator variance obtained in Eq.

Original: (14) as aregularizer.
Revised: (15) as a regularizer.

Original: We show here that both approaches are equivalent, and recover a popular regularizerknown as TangentProp [36].
Revised: We show here that both approaches are equivalent, and recover a popular regularizer known as TangentProp [50].

Original: Although various extensions of TangentProp have been introduced[3, 32] no principled derivation of it has yet been proposed.
Revised: Although various extensions of TangentProp have been introduced [51, 52] no principled derivation of it has yet been proposed.

Revised: Using the same argument as in

Original: Using the same argument as in Section 4.2,the expectation of a nonlinear transformation of a random variable ( T ( x n ) ) can be approximatedfrom the expectation of the Taylor expansion (detailed derivations in Appendix F) from which anupper-bound is obtained by applying the Cauchy-Schwarz inequality
Revised: Section 4.2, the expectation of a nonlinear transformation of a random variable ( T ( x n ) ) can be [Table] expected as the introduced DA variations become redundant with the training set samples. This marginal gap can be slightly increased by employing stronger DA (compare top half and bottom half of the table).

Revised: However, in the small dataset size regime , there is an important gain provided by employing the regularizer even when compared to observing many DA samples (#DA=10).

Original:  [Equation]
Revised: This indicates a potentially crucial regime in which introducing the analytical regularizer helps. More surprisingly, this is also the case if part of DA policy is misaligned with the task at hand (e.g. rotation for FashionMNIST). Additional empirical results with more training set sizes ( N ∈ { 100 , 200 , 500 , 1000 , 3000 , 10000 , 20000 } ) and DA policies (one per table) are available in Tables 2 to 5 for MNIST, in Tables 6 to 9 for EMNIST, and in Tables 10 to 13 for FashionMNIST approximated from the expectation of the Taylor expansion (detailed derivations in Appendix E) from which an upper-bound is obtained by applying the Cauchy-Schwarz inequality [Equation]

Original: with κ ( x ) ≥ 0 and the notations from Eq.
Revised: with κ ( x ) ≥ 0 and the notations from Eq.

Original: As a result, the TangentProp regularizer naturallyappears when using a Taylor approximation of the expected loss, and it corresponds to adding anexplicit loss variance regularization term (compare the TangentProp with Eq.
Revised: As a result, the TangentProp regularizer naturally appears when using a Taylor approximation of the expected loss, and it corresponds to adding an explicit loss variance regularization term (compare the TangentProp with Eq.

Original: We thus obtainedfrom ﬁrst principles that TangentProp emerges naturally when considering the second order Taylorapproximation of the expected loss given a DA.
Revised: We thus obtained from ﬁrst principles that TangentProp emerges naturally when considering the second order Taylor approximation of the expected loss given a DA.

Original:  Conclusions and Limitations
Revised:  Conclusions and Limitations

Original: In this paper, we proposed a novel set of mathematical tools (Sections 3.2 and 3.3) under which it ispossible to study DA and to provably answer some of the open questions around the efﬁciency andimpact of DA to train a model.
Revised: In this paper, we proposed a novel set of mathematical tools (Sections 3.2 and 3.3) under which it is possible to study DA and to provably answer some of the open questions around the efﬁciency and impact of DA to train a model.

Original: We ﬁrst obtained the explicit regularizer produced by different DAs in
Revised: We ﬁrst obtained the explicit regularizer produced by different DAs in

Original: Section 3.4.
Revised: Section 3.4.

Original: This led to the following observation: the kernel space of the W matrix is pushed toalign with the largest eigenvectors of the sample covariance matrix.
Revised: This led to the following observation: the kernel space of the W matrix is pushed to align with the largest eigenvectors of the sample covariance matrix.

Original: This was then studied in a moregeneral setting in Section 4.2 for nonlinear models when characterizing the loss variance under DAsampling.
Revised: This was then studied in a more general setting in Section 4.2 for nonlinear models when characterizing the loss variance under DA sampling.

Original: We also observed that Monte-Carlo sampling of transformed images is highly inefﬁcient(Section 4.1), even if similar training samples combine their underlying information within a dataset.
Revised: We also observed that Monte-Carlo sampling of transformed images is highly inefﬁcient (Section 4.1), even if similar training samples combine their underlying information within a dataset.

Original: Lastly, those derivations led us to provably derive a known regularizer —TangentProp— as being thenatural minimizer of a model’s loss variance.
Revised: Lastly, those derivations led us to provably derive a known regularizer —TangentProp— as being the natural minimizer of a model’s loss variance.

Original: The main limitation of the current analysis concernsnonlinear models for which we rely on a third-order Taylor expansion.
Revised: The main limitation of the current analysis concerns nonlinear models for which we rely on a third-order Taylor expansion.

Original: We hope to explore in moredetails the impact of DA on nonlinear models, possibly with a per-architecture analysis in futurework.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contri- butions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] We have precisely given the scope of our results in the end of theintroduction.
Revised: [Yes] We have precisely given the scope of our results in the end of the introduction.

Original: Additionally, we recall the limitations and assumptions of our results within eachsection carefully.
Revised: Additionally, we recall the limitations and assumptions of our results within each section carefully.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] The main limitations comes from the approx- imation done for nonlinear models which we have repeatedly highlight in both the introductionsand the corresponding sections.
Revised: [Yes] The main limitations comes from the approximation done for nonlinear models which we have repeatedly highlight in both the introductions and the corresponding sections.

Original: We also highlight those in the conclusion along with future work.
Revised: We also highlight those in the conclusion along with future work.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We do not see anypotential negative societal impact for this work (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] We do not see any potential negative societal impact for this work (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] we have carefully given theassumptions each result relies on.
Revised: [Yes] we have carefully given the assumptions each result relies on.

Original: Especially, at the end of the introduction, we have highlighted a reference paper where it was shown that our assumptions are common in the literature, and that they are good enough to gain insights into nonlinear models.
Revised: Especially, at the end of the introduction, we have highlighted a reference paper where it was shown that our assumptions are common in the literature, and that they are good enough to gain insights into nonlinear models.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Each formal statement andderivation is linked to a proof in the appendix.
Revised: [Yes] Each formal statement and derivation is linked to a proof in the appendix.

Original: We also provide details derivations in the appendix of all the equations we have obtained in the main paper, even if they are not part of a formal result.
Revised: We also provide details derivations in the appendix of all the equations we have obtained in the main paper, even if they are not part of a formal result.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide samples of the code to reproduce ﬁgures.
Revised: [Yes] We provide samples of the code to reproduce ﬁgures.

Original: The entire codebase will be released publicly upon completion of the review-process (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: The entire codebase will be released publicly upon completion of the review-process (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We only explore MNIST and provided the training details in the caption of the correspondingﬁgure(c) Did you report error bars (e.g., with respect to the random seed after running experiments multipletimes)?
Revised: [Yes] We only explore MNIST and provided the training details in the caption of the corresponding ﬁgure (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] We did not provide error bars but multiple runs with various settings(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
Revised: [No] We did not provide error bars but multiple runs with various settings (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] The employed experiment is small scale and can beperformed without GPU [Table]
Revised: [No] The employed experiment is small scale and can be performed without GPU 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] MNIST only(b) Did you mention the license of the assets?
Revised: [Yes] MNIST only (b) Did you mention the license of the assets?

Original: [No](c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [No] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No](d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A](e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]