Original: A Data-Augmentation Is Worth A Thousand Samples: Analytical Moments And Sampling-Free Training
Revised: A Data-Augmentation Is Worth A Thousand Samples: Analytical Moments And Sampling-Free Training

Original: Abstract
Revised: Abstract

Original: Data-Augmentation (DA) is known to improve performance across tasks and datasets.
Revised: Data-Augmentation (DA) is known to improve performance across tasks and datasets.

Original: We propose a method to theoretically analyze the effect of DA and study questions such as: how many augmented samples are needed to correctly estimate the information encoded by that DA?
Revised: We propose a method to theoretically analyze the effect of DA and study questions such as: how many augmented samples are needed to correctly estimate the information encoded by that DA?

Original: How does the augmentation policy impact the ﬁnal parameters of a model?
Revised: How does the augmentation policy impact the ﬁnal parameters of a model?

Original: We derive several quantities in close-form, such as the expectation and variance of an image, loss, and model’s output under a given
Revised: We derive several quantities in close-form, such as the expectation and variance of an image, loss, and model’s output under a given

Original: DA distribution.
Revised: DA distribution.

Original: Up to our knowledge, we obtain the ﬁrst explicit regularizer that corresponds to using DA during training for non-trivial transformations such as afﬁne transformations, color jittering, or Gaussian blur.
Revised: Up to our knowledge, we obtain the ﬁrst explicit regularizer that corresponds to using DA during training for non-trivial transformations such as afﬁne transformations, color jittering, or Gaussian blur.

Original: Those derivations open new avenues to quantify the beneﬁts and limitations of DA.
Revised: Those derivations open new avenues to quantify the beneﬁts and limitations of DA.

Original: For example,we show given a loss at hand,  common DAs require tens of thousands of samples for the loss to be correctly estimated and for the model training to converge.
Revised: For example, given a loss at hand, we ﬁnd that common DAs require tens of thousands of samples for the loss to be correctly estimated and for the model training to converge.

Original: We then show that for a training loss to have reduced variance under DA sampling, the model’s saliency map (gradient of the loss with respect to the model’s input) must align with the smallest eigenvector of the sample’s covariance matrix under the considered DA augmentation; this is exactly the quantity estimated and regularized by TangentProp.
Revised: We then show that for a training loss to have reduced variance under DA sampling, the model’s saliency map (gradient of the loss with respect to the model’s input) must align with the smallest eigenvector of the sample’s covariance matrix under the considered DA augmentation; this is exactly the quantity estimated and regularized by TangentProp.

Original: Those ﬁndings also hint at a possible explanation on why models tend to shift their focus from edges to textures when speciﬁc DAs are employed.
Revised: Those ﬁndings also hint at a possible explanation on why models tend to shift their focus from edges to textures when speciﬁc DAs are employed.

Original:  Introduction
Revised:  Introduction

Original: Data Augmentation (DA) is a prevalent technique in training deep learning models [1, 2, 3].
Revised: Data Augmentation (DA) is a prevalent technique in training deep learning models [1, 2, 3].

Original: These
Revised: These

Original: Deep Networks (DNs) models f γ , governed by some parameters γ ∈ Γ , are trained on the train set and expected to generalize to unseen samples (test set).
Revised: Deep Networks (DNs) models f γ , governed by some parameters γ ∈ Γ , are trained on the train set and expected to generalize to unseen samples (test set).

Original: To combat the tendency of DNs to overﬁt [4], producing a large train-test performance gap, regularization and in particular DA is heavily employed, amongst other mechanisms such as weight-decay [5, 6].
Revised: To combat the tendency of DNs to overﬁt [4], producing a large train-test performance gap, regularization and in particular DA is heavily employed, amongst other mechanisms such as weight-decay [5, 6].

Original: The beneﬁt of DA over those alternatives is that deﬁning input transformations that preserve the semantics of their inputs is a relatively simple task, at least in computer vision or acoustic processing [7, 8, 9].
Revised: The beneﬁt of DA over those alternatives is that deﬁning input transformations that preserve the semantics of their inputs is a relatively simple task, at least in computer vision or acoustic processing [7, 8, 9].

Original: Furthermore, if the DA is well designed and rich enough, it can effectively bring the number of non-trivial training samples close to the theoretical limits ensuring that any performance gap between the train and test set vanishes [10] as empirically observed in various scenarios [11, 12].
Revised: Furthermore, if the DA is well designed and rich enough, it can effectively bring the number of non-trivial training samples close to the theoretical limits ensuring that any performance gap between the train and test set vanishes [10] as empirically observed in various scenarios [11, 12].

Original: In fact, DA has proven so useful that novel training methods such as self-supervised learning (SSL) entirely rely on DA [13, 14] to learn meaningful data representations.
Revised: In fact, DA has proven so useful that novel training methods such as self-supervised learning (SSL) entirely rely on DA [13, 14] to learn meaningful data representations.

Original: Despite its empirical effectiveness, our understanding of DA has many open questions, three of which we propose to study: (a) how do different DAs impact the model’s parameters during training?
Revised: Despite its empirical effectiveness, our understanding of DA has many open questions, three of which we propose to study: (a) how do different DAs impact the model’s parameters during training?

Original: ; (b) how sample-efﬁcient is the DA sampling, i.e., how many DA samples a model must observe to converge?; and (c) how sensitive is a loss/model to the DA sampling and how this variance evolves during training as a function of the model’s ability to minimize the loss at hand, and as a
Revised: ; (b) how sample-efﬁcient is the DA sampling, i.e., how many DA samples a model must observe to converge?; and (c) how sensitive is a loss/model to the DA sampling and how this variance evolves during training as a function of the model’s ability to minimize the loss at hand, and as a

Original: [Figure]
Revised: [Figure]

Original: image
Revised: image

Original: I
Revised: I

Revised: [Figure]

Revised: image

Original: [Figure]
Revised: [Figure]

Original: im function of the model’s parameters?
Revised: im function of the model’s parameters?

Original: Our goal is to analytically derive some preliminary answers and insights into those three areas enabled by a novel image transformation operator that we introduce in
Revised: Our goal is to analytically derive some preliminary answers and insights into those three areas enabled by a novel image transformation operator that we introduce in

Original: Section 3.2 coined Data-Space Transform (DST).
Revised: Section 3.2 coined Data-Space Transform (DST).

Original: DST employs a mathematical formulation of input transformations akin to the one of [15] although slightly more general as we do not require the DA to form a group.
Revised: DST employs a mathematical formulation of input transformations akin to the one of [15] although slightly more general as we do not require the DA to form a group.

Original: We study the zoom/translation/shearing/rotation DAs in the main text and Gaussian blur, grayscale, color jitter and random crop in Appendix C.
Revised: We study the zoom/translation/shearing/rotation DAs in the main text and Gaussian blur, grayscale, color jitter and random crop in Appendix C; code is available 1 .

Original: We summarize our contributions below:
Revised: We summarize our contributions below:

Original: (a) we derive the analytical ﬁrst order moments of augmented samples, and of the losses employing augmented samples (Section 3.3), effectively providing us with the explicit regularizer induced by each DA (Section 3.4) (b) we quantify the number of DA samples that are required for a model/loss to obtain a correct estimate of the information conveyed by that DA (Section 4.1) (c) we derive the sensitivity, i e. variance, of a given loss and model under a DA policy (Section 4.2) leading us to rediscover from ﬁrst principles a popular deep network regularizer: TangentProp, as being the natural regularization to employ to minimize the loss variance (Section 4.3).
Revised: (a) we derive the analytical ﬁrst order moments of augmented samples, and of the losses employing augmented samples (Section 3.3), effectively providing us with the explicit regularizer induced by each DA (Section 3.4) (b) we quantify the number of DA samples that are required for a model/loss to obtain a correct estimate of the information conveyed by that DA (Section 4.1) (c) we derive the sensitivity, i.e. variance, of a given loss and model under a DA policy (Section 4.2) leading us to rediscover from ﬁrst principles a popular deep network regularizer: TangentProp, as being the natural regularization to employ to minimize the loss variance (Section 4.3).

Original: Upshot of gained insights: (a) will show us that the explicit DA regularizer corresponds to a generalized Tikhonov regularizer that depends on each sample’s covariance matrix largest eigenvectors; the kernel space of the model’s Jacobian matrix aligns with the data manifold tangent space as modeled by the DA; (b) will show us that the number of augmented samples required for a loss/model to correctly estimate the information provided by a DA of a single sample is on the order of 10 4 .
Revised: Upshot of gained insights: (a) will show us that the explicit DA regularizer corresponds to a generalized Tikhonov regularizer that depends on each sample’s covariance matrix largest eigenvectors; the kernel space of the model’s Jacobian matrix aligns with the data manifold tangent space as modeled by the DA; (b) will show us that the number of augmented samples required for a loss/model to correctly estimate the information provided by a DA of a single sample is on the order of 10 4 .

Original: Even when combining the information of thousands of samples, we ﬁnd that the entire train set must be augmented at least 50 × for a DA policy to be correctly learned by a model; lastly (c) will quantify the loss variance as a simple function of the model’s Jacobian matrix, and the eigenvectors of the augmented sample variance matrix.
Revised: Even when combining the information of thousands of samples, we ﬁnd that the entire train set must be augmented at least 50 × for a DA policy to be correctly learned by a model; lastly (c) will quantify the loss variance as a simple function of the model’s Jacobian matrix, and the eigenvectors of the augmented sample variance matrix.

Original: Regardless of the model or task-at-hand, the loss sensitivity to random DA sampling goes down as the kernel of a model’s
Revised: Regardless of the model or task-at-hand, the loss sensitivity to random DA sampling goes down as the kernel of a model’s

Original: Jacobian matrix aligns with the principal directions of the data manifold tangent space; as a by-product, we recover TangentProp as the natural regularizer minimizing the loss sensitivity to DA sampling.
Revised: Jacobian matrix aligns with the principal directions of the data manifold tangent space; as a by-product, we recover TangentProp as the natural regularizer minimizing the loss sensitivity to DA sampling.

Original: Limitations: all of our image-space results e.g. expectation and variance of images under DA are exact, no approximation is employed.
Revised: Limitations: all of our image-space results e.g. expectation and variance of images under DA are exact, no approximation is employed.

Original: Results on a model’s output/loss are provided both in the case of a linear model and of a nonlinear model.
Revised: Results on a model’s output/loss are provided both in the case of a linear model and of a nonlinear model.

Original: For the former, results are exact, for the latter a third-order Taylor approximations is employed which is common and has been shown to provide accurate enough approximations to safely rely on the insights/quantities obtained e.g. see Sec.
Revised: For the former, results are exact, for the latter a third-order Taylor approximations is employed which is common and has been shown to provide accurate enough approximations to safely rely on the insights/quantities obtained e.g. see Sec.

Original: A.2 of [16].
Revised: A.2 of [16].

Original: All the proofs and implementation details are provided in the appendix.
Revised: All the proofs and implementation details are provided in the appendix.

Original: We also emphasize that our primary goal is to provide a theoretical understanding of DA, to quantify its efﬁciency and impact onto learned models and to derive and visualize its explicit regularizer; we leave the search for tractable alternatives to DA based on those insights for future work.
Revised: We also emphasize that our primary goal is to provide a theoretical understanding of DA, to quantify its efﬁciency and impact onto learned models and to derive and visualize its explicit regularizer; we leave the search for tractable alternatives to DA based on those insights for future work.

Original:  Background
Revised:  Background

Original: Existing Explicit Regularizers From Data-Augmentation.
Revised: Existing Explicit Regularizers From Data-Augmentation.

Original: It is widely accepted that dataaugmentation (DA) regularizes a model towards the transformations that are modeled [5, 17], and that this regularization impacts performances signiﬁcantly and positively, possibly as much as the regularization offered by the choice of DN architecture [18] and optimizer [19].
Revised: It is widely accepted that dataaugmentation (DA) regularizes a model towards the transformations that are modeled [5, 17], and that this regularization impacts performances signiﬁcantly and positively, possibly as much as the regularization offered by the choice of DN architecture [18] and optimizer [19].

Original: To gain precious insights into the impact of DA onto the learned functional f γ , the most common strategy is to derive the explicit regularizer that directly acts upon f γ in the same manner as if one were to use DA during training.
Revised: To gain precious insights into the impact of DA onto the learned functional f γ , the most common strategy is to derive the explicit regularizer that directly acts upon f γ in the same manner as if one were to use DA during training.

Original: This explicit derivation is however challenging and so far has been limited to DA strategies such as additive white noise or multiplicative binary noise applied identically and independently throughout the image and/or feature maps, as with dropout [20, 21].
Revised: This explicit derivation is however challenging and so far has been limited to DA strategies such as additive white noise or multiplicative binary noise applied identically and independently throughout the image and/or feature maps, as with dropout [20, 21].

Original: In those settings, various works have studied in the linear regime the relation between such DA and its equivalence to using Tikhonov regularization[22] or weight decay [23] as in minW (cid:80) Nn =1 E (cid:15) ∼N (0 ,σ ) (cid:2) (cid:107) y n − W ( x [Equation] +
Revised: In those settings, various works have studied in the linear regime the relation between such DA and its equivalence to using Tikhonov regularization[22] or weight decay [23] as in min

Original: [Equation]
Revised: [Equation]

Original: ) (cid:107) 22[-(cid:3) = min W (cid:80) Nn =1 (cid:107) y n − W x n (cid:107) 
-] +λ ( σ ) (cid:107) W
Revised: 22[-(cid:3) = min W (cid:80) Nn =1 (cid:107) y n − W x n (cid:107) 
-] +

Original: [Equation]
Revised: [Equation]

Original: 2 F [24]. More recently, [25, 26] extended the case of additive white noise to nonlinear models and concluded that additive white noise DA corresponds to adding an explicit Frobenius norm regularization onto the Jacobian of f θ evaluated at each data sample.
Revised: More recently, [25, 26] extended the case of additive white noise to nonlinear models and concluded that additive white noise DA corresponds to adding an explicit Frobenius norm regularization onto the Jacobian of f θ evaluated at each data sample.

Original: Going to more involved DA strategies e.g. translations or zooms of the input images is challenging and has so far only been studied from an empirical perspective.
Revised: Going to more involved DA strategies e.g. translations or zooms of the input images is challenging and has so far only been studied from an empirical perspective.

Original: For example, [27, 28, 29] performed thorough ablation studies on the interplay between DAs and a collection of known explicit regularizers to ﬁnd correlations between them.
Revised: For example, [27, 28, 29] performed thorough ablation studies on the interplay between DAs and a collection of known explicit regularizers to ﬁnd correlations between them.

Original: It was concluded that weight-decay (the explicit regularizer of additive white noise) does not relate to those more advanced DAs.
Revised: It was concluded that weight-decay (the explicit regularizer of additive white noise) does not relate to those more advanced DAs.

Original: This also led other studies to suggest that norm-based regularization might be insufﬁcient to describe the implicit regularization of
Revised: This also led other studies to suggest that norm-based regularization might be insufﬁcient to describe the implicit regularization of

Original: DAs involving advanced image transformations [30].
Revised: DAs involving advanced image transformations [30].

Original: We debunk this last claim in Sec. 3.3.
Revised: We debunk this last claim in Sec. 3.3.

Original: Coordinate Space Transformation.
Revised: Coordinate Space Transformation.

Original: Throughout this paper, we will consider a two-dimensional image
Revised: Throughout this paper, we will consider a two-dimensional image

Original: I ( x, y ) to be at least square-integrable
Revised: I ( x, y ) to be at least square-integrable

Original: I ∈ L 2 ( R 2 ) [31].
Revised: I ∈ L 2 ( R 2 ) [31].

Original: Multi-channel images are dealt with by treating each channel as its own single-channel image, as commonly done in practice [1].
Revised: Multi-channel images are dealt with by treating each channel as its own single-channel image, as commonly done in practice [1].

Original: As we are interested in practical cases, we will often assume that I has compact support e.g. has nonzero values only within a bounded domain such as [0 , 1] 2 .
Revised: As we are interested in practical cases, we will often assume that I has compact support e.g. has nonzero values only within a bounded domain such as [0 , 1] 2 .

Original: Visualizing this image thus corresponds to displaying the sampled values of I on a regular grid (pixel positions) of [0 , 1] 2 [32].
Revised: Visualizing this image thus corresponds to displaying the sampled values of I on a regular grid (pixel positions) of [0 , 1] 2 [32].

Original: The most common formulation to apply a transformation on the image I to obtain the transformed image T is to transform the image coordinates [33, 34, 35].
Revised: The most common formulation to apply a transformation on the image I to obtain the transformed image T is to transform the image coordinates [33, 34, 35].

Original: That is, a mapping t : R 2 (cid:55)→ R 2 describes what coordinate t ( u, v ) of the original image I maps to the coordinate u, v of the transformed image T as in
Revised: That is, a mapping t : R 2 (cid:55)→ R 2 describes what coordinate t ( u, v ) of the original image I maps to the coordinate u, v of the transformed image T as in

Original: [Equation]
Revised: [Equation]

Original: This mapping t often comes with some parameters θ governing the underlying transformation as in 
Revised: This mapping t often comes with some parameters θ governing the underlying transformation as in t

Original: [Equation]
Revised: [Equation]

Original: for translation, rotation and zoom respectively.
Revised: for translation, rotation and zoom respectively.

Original: We provide a visual depiction of the zoom transformation applied in coordinate space in Fig.
Revised: We provide a visual depiction of the zoom transformation applied in coordinate space in Fig.

Original: 6, in the appendix.
Revised: 6, in the appendix.

Original: The formulation of Eq.
Revised: The formulation of Eq.

Original: 1 has two key beneﬁts.
Revised: 1 has two key beneﬁts.

Original: First, it allows a simple and intuitive design of t to obtain novel transformations.
Revised: First, it allows a simple and intuitive design of t to obtain novel transformations.

Original: Second, it is computationally efﬁcient as the coordinate-space of images are 2 / 3 -dimensional.
Revised: Second, it is computationally efﬁcient as the coordinate-space of images are 2 / 3 -dimensional.

Original: Those beneﬁts have led to e.g. the Spatial Transformer Network [36].
Revised: Those beneﬁts have led to e.g. the Spatial Transformer Network [36].

Original: On the other hand, Eq. 1 has one major drawback for our purpose: the exact moments of the transformed image under random θ parameters are not tractable due to the composition of t with the nonlinear mapping I .
Revised: On the other hand, Eq. 1 has one major drawback for our purpose: the exact moments of the transformed image under random θ parameters are not tractable due to the composition of t with the nonlinear mapping I .

Original: And as it will become clear in Section 3.1, those quantities are needed to derive
Revised: And as it will become clear in Section 3.1, those quantities are needed to derive

Original: DAs’ explicit regularizers.
Revised: DAs’ explicit regularizers.

Original: [Figure]
Revised: [Figure]

Original:  Analytical Moments of Transformed Images Enable Inﬁnite Dataset Training
Revised:  Analytical Moments of Transformed Images Enable Inﬁnite Dataset Training

Original: We ﬁrst motivate this study by formulating the training process under DA sampling as doing a
Revised: We ﬁrst motivate this study by formulating the training process under DA sampling as doing a

Original: Monte-Carlo estimate of the true (unknown) expected loss under that DA distribution (Section 3.1).
Revised: Monte-Carlo estimate of the true (unknown) expected loss under that DA distribution (Section 3.1).

Original: Going beyond this sampling/estimation procedures requires knowledge of the average and variance of a transformed sample, i.e. the ﬁrst two centered-moments.
Revised: Going beyond this sampling/estimation procedures requires knowledge of the average and variance of a transformed sample, i.e. the ﬁrst two centered-moments.

Original: This is our motivation to construct a novel Data-Space Transformation (DST) (Section 3.2) that will enable close-form formula for those moments (Section 3.3).
Revised: This is our motivation to construct a novel Data-Space Transformation (DST) (Section 3.2) that will enable close-form formula for those moments (Section 3.3).

Original: From those, we will be able to remove the need to sample transformed images to train a model, by obtaining the close-form expected loss Section 3.4, under which training can be performed sampling-free.
Revised: From those, we will be able to remove the need to sample transformed images to train a model, by obtaining the close-form expected loss Section 3.4, under which training can be performed sampling-free.

Original: 3.1 Motivation: Current Data-Augmentation Training Performs Monte-Carlo Estimation
Revised: 3.1 Motivation: Current Data-Augmentation Training Performs Monte-Carlo Estimation

Original: Training a model with DA consists in (i) sampling transformed images at each training iteration for each sample x n as in T θ n ( x n ) with θ n ∼ θ a randomly sampled DA parameter e.g. the amount of translation to apply, (ii) evaluating the loss L on the transformed sample/mini-batch/dataset, and (iii) using some ﬂavor of gradient descent to update the parameters γ of the model f γ .
Revised: Training a model with DA consists in (i) sampling transformed images at each training iteration for each sample x n as in T θ n ( x n ) with θ n ∼ θ a randomly sampled DA parameter e.g. the amount of translation to apply, (ii) evaluating the loss L on the transformed sample/mini-batch/dataset, and (iii) using some ﬂavor of gradient descent to update the parameters γ of the model f γ .

Original: This training procedure corresponds to a one-sample Monte-Carlo (MC) estimate [37, 38] of the expected loss
Revised: This training procedure corresponds to a one-sample Monte-Carlo (MC) estimate [37, 38] of the expected loss

Original: [Equation]
Revised: [Equation]

Original: with θ n i.i.d samples of θ repeatedly sampled at each epoch.
Revised: with θ n i.i.d samples of θ repeatedly sampled at each epoch.

Original: In a supervised setting, the loss would also receive a per-sample target y n as input.
Revised: In a supervised setting, the loss would also receive a per-sample target y n as input.

Original: Although a one-sample estimate might be insufﬁcient to apply the central limit theorem [39] and guarantee training convergence, the combination of multiple samples in mini-batch training does provide convergence in most cases.
Revised: Although a one-sample estimate might be insufﬁcient to apply the central limit theorem [39] and guarantee training convergence, the combination of multiple samples in mini-batch training does provide convergence in most cases.

Original: Although, methods such as Self-Supervised Learning that heavily rely on DA, tend to be more sensitive and unstable [40].
Revised: Although, methods such as Self-Supervised Learning that heavily rely on DA, tend to be more sensitive and unstable [40].

Original: To avoid such instabilities, and to incorporate all the DA transformations of x n into the model’s parameter update, one would be tempted to compute the model parameters’ gradient on the expected loss (left-hand side of Eq.
Revised: To avoid such instabilities, and to incorporate all the DA transformations of x n into the model’s parameter update, one would be tempted to compute the model parameters’ gradient on the expected loss (left-hand side of Eq.

Original: (see Section 3.4).
Revised: (see Section 3.4).

Original: Knowledge of the expected loss would also prove useful to measure the quality of the MC estimate (see Section 4.1).
Revised: Knowledge of the expected loss would also prove useful to measure the quality of the MC estimate (see Section 4.1).

Original: Note that even in the least-square setting, such approximation is employed as the expected loss under DAs is not know.
Revised: Note that even in the least-square setting, such approximation is employed as the expected loss under DAs is not know.

Original: Hence, one commonly obtain many DA samples, and then apply the least-square formula on this augmented training set, which is highly inefﬁcient.
Revised: Hence, one commonly obtain many DA samples, and then apply the least-square formula on this augmented training set, which is highly inefﬁcient.

Original: Alternatively, this study will obtain the optimum least-square parameters under DA sampling (Theorem 3.4) i.e. only using the given training set, we directly obtain the parameters that would correspond to sampling inﬁnitely many DA samples and applying the usual least-square formula on this inﬁnite training set.
Revised: Alternatively, this study will obtain the optimum least-square parameters under DA sampling (Theorem 3.4) i.e. only using the given training set, we directly obtain the parameters that would correspond to sampling inﬁnitely many DA samples and applying the usual least-square formula on this inﬁnite training set.

Original: The close-form expected loss requires knowledge of the average and variance of the transformed sample T θ ( x n ) taken with respect to the random variable θ .
Revised: The close-form expected loss requires knowledge of the average and variance of the transformed sample T θ ( x n ) taken with respect to the random variable θ .

Original: Hence, we ﬁrst propose to formulate a novel and tractable augmentation model (Section 3.2) that will allow us to obtain those moments analytically in Section 3.3.
Revised: Hence, we ﬁrst propose to formulate a novel and tractable augmentation model (Section 3.2) that will allow us to obtain those moments analytically in Section 3.3.

Original: 3.2 Proposed Data-Space Transformation (DST)
Revised: 3.2 Proposed Data-Space Transformation (DST)

Original: Instead of altering the coordinate positions of an image, as done in the coordinate-space transformation of Eq.
Revised: Instead of altering the coordinate positions of an image, as done in the coordinate-space transformation of Eq.

Original: (1), we propose to alter the image basis functions.
Revised: (1), we propose to alter the image basis functions.

Original: Going back to the construction of functions, one easily recalls that any (image) function can be expanded into a basis as in I ( u, v )
Revised: Going back to the construction of functions, one easily recalls that any (image) function can be expanded into a basis as in I ( u, v )

Original: =
Revised: =

Original: (cid:82) I ( x, y )
Revised: (cid:82) I ( x, y )

Original: δ ( u − x, v − y ) dxdy with δ the usual Dirac distribution.
Revised: δ ( u − x, v − y ) dxdy with δ the usual Dirac distribution.

Original: Suppose for now that we consider a horizontal translation by a constant θ .
Revised: Suppose for now that we consider a horizontal translation by a constant θ .

Original: Then, one can obtain that translated image via
Revised: Then, one can obtain that translated image via

Original: [Equation]
Revised: [Equation]

Original: hence, and crucial to our study, we apply the transformation onto the basis functions onto which the image is evaluated, rather than onto the original image itself .
Revised: hence, and crucial to our study, we apply the transformation onto the basis functions onto which the image is evaluated, rather than onto the original image itself .

Original: As the image is now constant with respect to the transformation parameter θ , and as the basis functions have some convenient analytical forms, the derivation of the transformed images moments, under θ , will become straightforward.
Revised: As the image is now constant with respect to the transformation parameter θ , and as the basis functions have some convenient analytical forms, the derivation of the transformed images moments, under θ , will become straightforward.

Original: Because the transformed image T is now obtained by combining its pixel values (recall Eq.
Revised: Because the transformed image T is now obtained by combining its pixel values (recall Eq.

Original: we coin this transform as Data-Space Transform (DST), formally deﬁned below.
Revised: we coin this transform as Data-Space Transform (DST), formally deﬁned below.

Original: Deﬁnition 3.1 (Data-Space Transform) .
Revised: Deﬁnition 3.1 (Data-Space Transform) .

Original: We deﬁne the DST of an image I ∈
Revised: We deﬁne the DST of an image I ∈

Original: L 2 ( R 2 ) producing the transformed image T θ ( I )
Revised: L 2 ( R 2 ) producing the transformed image T θ ( I )

Original: ∈
Revised: ∈

Original: L 2 ( R 2 ) as
Revised: L 2 ( R 2 ) as

Original: [Equation]
Revised: [Equation]

Original: 5) with h θ ( u, v, ., . )
Revised: 5) with h θ ( u, v, ., . )

Original: ∈ C ∞ 

Revised: ∈ C ∞

Original: ( R 2 ) encoding the transformation.
Revised: ( R 2 ) encoding the transformation.

Original: In Deﬁnition 3.1, we only impose for h θ ( u, v, ., . ) to be with compact support.
Revised: In Deﬁnition 3.1, we only impose for h θ ( u, v, ., . ) to be with compact support.

Original: In fact, one should interpret h θ ( u, v, ., . )
Revised: In fact, one should interpret h θ ( u, v, ., . )

Original: as a distribution whose purpose is to evaluate I at a desired (coordinate) position on its domain.
Revised: as a distribution whose purpose is to evaluate I at a desired (coordinate) position on its domain.

Original: This evaluation –depending on the form of h θ ( u, v, ., . ) – can extract a single pixelvalue of the image I at a desired location (as in Eq.
Revised: This evaluation –depending on the form of h θ ( u, v, ., . ) – can extract a single pixelvalue of the image I at a desired location (as in Eq.

Original: (4)), or can combine multiple values e.g. with h θ ( u, v, ., . )
Revised: (4)), or can combine multiple values e.g. with h θ ( u, v, ., . )

Original: being a bump function.
Revised: being a bump function.

Original: Additionally, we do not impose any restriction on the invertibility of this mapping i.e. T θ ( I )( u, v ) can disregard parts of the original image.
Revised: Additionally, we do not impose any restriction on the invertibility of this mapping i.e. T θ ( I )( u, v ) can disregard parts of the original image.

Original: Coordinate-space transformations as DSTs.
Revised: Coordinate-space transformations as DSTs.

Original: The coordinate-space transformation and the proposed DST (Deﬁnition 3.1) act in different spaces: the image coordinates and the image pixel values, respectively.
Revised: The coordinate-space transformation and the proposed DST (Deﬁnition 3.1) act in different spaces: the image coordinates and the image pixel values, respectively.

Original: Nevertheless, this does not limit the range of transformations that can be applied to an image.
Revised: Nevertheless, this does not limit the range of transformations that can be applied to an image.

Original: The following statement provides a simple recipe to turn any already employed coordinate-space transformation into a DST.
Revised: The following statement provides a simple recipe to turn any already employed coordinate-space transformation into a DST.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Any coordinate-space transformation (1) using t θ : R 2 (cid:55)→ R 2 can be expressed as a data-space transformation (5) by setting h θ ( u, v, x, y )
Revised: Any coordinate-space transformation (1) using t θ : R 2 (cid:55)→ R 2 can be expressed as a data-space transformation (5) by setting h θ ( u, v, x, y )

Original: =
Revised: =

Original: δ ( t θ ( x, y ) − [ u, v ] T ) .
Revised: δ ( t θ ( x, y ) − [ u, v ] T ) .

Original: We will focus in the main text on zoom/rotation/translation/shearing, and we propose in Appendix C the case of Gaussian blur, color jittering, random crop, along with a discussion on extending our results to CutMix [41] and MixUp [42].
Revised: We will focus in the main text on zoom/rotation/translation/shearing, and we propose in Appendix C the case of Gaussian blur, color jittering, random crop, along with a discussion on extending our results to CutMix [41] and MixUp [42].

Original: Using Proposition 3.2, we obtain the DST operators h θ ( u, v, x, y ) to be
Revised: Using Proposition 3.2, we obtain the DST operators h θ ( u, v, x, y ) to be

Original: [Equation]
Revised: [Equation]

Original: for the vertical/horizontal translation, vertical/horizontal shearing, zoom and rotation respectively (compare with Eq. 2).
Revised: for the vertical/horizontal translation, vertical/horizontal shearing, zoom and rotation respectively (compare with Eq. 2).

Original: Before focusing on the analytical moments of the DST samples, we describe how those operators are applied in a discrete setting.
Revised: Before focusing on the analytical moments of the DST samples, we describe how those operators are applied in a discrete setting.

Original: Discretized version.
Revised: Discretized version.

Original: We now describe how any DST of a discrete image, ﬂattened as a vector, can be expressed as a matrix-vector product with the matrix entries depending on the employed DA and its parameter θ , as we display in Figs.
Revised: We now describe how any DST of a discrete image, ﬂattened as a vector, can be expressed as a matrix-vector product with the matrix entries depending on the employed DA and its parameter θ , as we display in Figs.

Original: 1 and 2.
Revised: 1 and 2.

Original: The functional form of the DST from Eq.
Revised: The functional form of the DST from Eq.

Original: (5) producing the target image at a speciﬁc position T θ ( I )( u, v ) is linear in the original image
Revised: (5) producing the target image at a speciﬁc position T θ ( I )( u, v ) is linear in the original image

Original: I .
Revised: I .

Original: Hence, in the discrete setting, the continuous integral over the image domain is replaced with a summation with indices based on the desired sampling/resolution of I , i.e. the position of the pixel spatial positions.
Revised: Hence, in the discrete setting, the continuous integral over the image domain is replaced with a summation with indices based on the desired sampling/resolution of I , i.e. the position of the pixel spatial positions.

Original: Expressing linear operators as matrix-vector products will greatly ease our development, we will denote by x ∈ R hw the ﬂattened ( h × w ) discrete images I .
Revised: Expressing linear operators as matrix-vector products will greatly ease our development, we will denote by x ∈ R hw the ﬂattened ( h × w ) discrete images I .

Original: Hence, our data-space transformation, given some parameters θ , takes the form of
Revised: Hence, our data-space transformation, given some parameters θ , takes the form of

Original: [Equation]
Revised: [Equation]

Original: with t ( θ ) ∈ R hw the ﬂattened transformed image (which can be reshaped as desired) and M ( θ ) ∈
Revised: with t ( θ ) ∈ R hw the ﬂattened transformed image (which can be reshaped as desired) and M ( θ ) ∈

Original: R hw × hw the matrix whose rows encode the discrete and ﬂattened h θ ( u, v, ., . ) .
Revised: R hw × hw the matrix whose rows encode the discrete and ﬂattened h θ ( u, v, ., . ) .

Original: For example, and employing a uniform grid sampling for illustration, M ( θ ) i,j = h θ ( i//w, i % w, j//w, j % w ) with // representing the ﬂoor division and % the modulo operation.
Revised: For example, and employing a uniform grid sampling for illustration, M ( θ ) i,j = h θ ( i//w, i % w, j//w, j % w ) with // representing the ﬂoor division and % the modulo operation.

Original: For the case of multi-channel images, we consider without loss of generality the application of the matrix-vector operation on each channel separately.
Revised: For the case of multi-channel images, we consider without loss of generality the application of the matrix-vector operation on each channel separately.

Original: We depict this operation along with the exact form of M ( θ ) for the case of the zoom transformation in Fig.
Revised: We depict this operation along with the exact form of M ( θ ) for the case of the zoom transformation in Fig.

Original: A crucial property of the matrix M ( θ ) lies in its sparsity, in fact, for most
Revised: A crucial property of the matrix M ( θ ) lies in its sparsity, in fact, for most

Original: DAs the transformations rely on displacing pixels rather than combining them.
Revised: DAs the transformations rely on displacing pixels rather than combining them.

Original: Hence, although
Revised: Hence, although

Original: M ( θ ) ’s total number of entries grow quadratically with the number of pixels in I , the number of nonzero entries only grows linearly with it.
Revised: M ( θ ) ’s total number of entries grow quadratically with the number of pixels in I , the number of nonzero entries only grows linearly with it.

Original: 3.3 Analytical Expectation and Variance of Transformed Images
Revised: 3.3 Analytical Expectation and Variance of Transformed Images

Original: The above construction (Deﬁnition 3.1) turns out to make the analytical form of the ﬁrst two moments of an augmented sample much simpler to derive.
Revised: The above construction (Deﬁnition 3.1) turns out to make the analytical form of the ﬁrst two moments of an augmented sample much simpler to derive.

Original: As this derivation is at the core of our main contribution, we propose a step-by-step derivation of E θ
Revised: As this derivation is at the core of our main contribution, we propose a step-by-step derivation of E θ

Original: [ T θ ( I )] for the case of horizontal translation (recall Eq.
Revised: [ T θ ( I )] for the case of horizontal translation (recall Eq.

Original: Let’s consider again the continuous model (the discrete version is provided after Theorem 3.3).
Revised: Let’s consider again the continuous model (the discrete version is provided after Theorem 3.3).

Original: Using
Revised: Using

Original: Fubini’s theorem to switch the order of integration and recalling Eq.
Revised: Fubini’s theorem to switch the order of integration and recalling Eq.

Original: (5), we have that
Revised: (5), we have that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using the deﬁnition for h θ from Eq.
Revised: Using the deﬁnition for h θ from Eq.

Original: (6) for horizontal translation, E θ [ h θ ( u, v, x, y )] becomes
Revised: (6) for horizontal translation, E θ [ h θ ( u, v, x, y )] becomes

Original: [Equation]
Revised: [Equation]

Original: with p the density function of θ prescribing how the translation parameter is distributed.
Revised: with p the density function of θ prescribing how the translation parameter is distributed.

Original: As a result, in this univariate translation case, the expected augmented image at coordinate ( u, v ) is given by
Revised: As a result, in this univariate translation case, the expected augmented image at coordinate ( u, v ) is given by

Original: [Equation]
Revised: [Equation]

Original: (9) which can be further simpliﬁed into E θ
Revised: (9) which can be further simpliﬁed into E θ

Original: [ T θ ( I )( ., v )]
Revised: [ T θ ( I )( ., v )]

Original: = I ( ., v ) (cid:63) p .
Revised: = I ( ., v ) (cid:63) p .

Original: Hence, the expected horizontallytranslated image is the convolution (on the x -axis only) between the original image I and the univariate density function p .
Revised: Hence, the expected horizontallytranslated image is the convolution (on the x -axis only) between the original image I and the univariate density function p .

Original: We formalize this for the transformations of Eq.
Revised: We formalize this for the transformations of Eq.

Original: (6) below.
Revised: (6) below.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: The analytical form of E θ [ h θ ( u, v, x, y )] , used to obtain the expected transformed image (recall Eq.
Revised: The analytical form of E θ [ h θ ( u, v, x, y )] , used to obtain the expected transformed image (recall Eq.

Original: (8) ) is given by
Revised: (8) ) is given by

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: other cases and second-order moment E θ
Revised: other cases and second-order moment E θ

Original: [ T θ
Revised: [ T θ

Original: [Equation]
Revised: [Equation]

Original: x n ) T θ ( x n ) T ] are deferred to the proof.
Revised: x n ) T θ ( x n ) T ] are deferred to the proof.

Original: (Proof in
Revised: (Proof in

Original: Appendix F.) t h i r d s ec ond ﬁ r s t rotation shear r a nk o f V [ T ( x ) ] transformation ampl.
Revised: Appendix F.) t h i r d s ec ond ﬁ r s t rotation shear r a nk o f V [ T ( x ) ] transformation ampl.

Original: An interesting observation obtained from the above ties the expected image under random 2dimensional translations with ( 2 -dimensional) density p to the convolution I (cid:63) p ; this provides a new portal to study and interpret convolutions with nonnegative, sum-to-one ﬁlters p .
Revised: An interesting observation obtained from the above ties the expected image under random 2dimensional translations with ( 2 -dimensional) density p to the convolution I (cid:63) p ; this provides a new portal to study and interpret convolutions with nonnegative, sum-to-one ﬁlters p .

Original: Again, and as per Eq.
Revised: Again, and as per Eq.

Original: (7), the discretized version of the expected image takes the form of E θ [ t ( θ )]
Revised: (7), the discretized version of the expected image takes the form of E θ [ t ( θ )]

Original: =
Revised: =

Original: E θ [ M ( θ )]
Revised: E θ [ M ( θ )]

Original: x with the entries of E θ [ M ( θ )] given by discretizing Eq.
Revised: x with the entries of E θ [ M ( θ )] given by discretizing Eq.

Original: We depict this expected matrix for various transformations as well as their application onto two different discrete images in Fig. 2, and we now proceed on deriving the left-hand side of Eq.
Revised: We depict this expected matrix for various transformations as well as their application onto two different discrete images in Fig. 2, and we now proceed on deriving the left-hand side of Eq.

Original: (3) i.e. the explicit DA regularizer.
Revised: (3) i.e. the explicit DA regularizer.

Original: 3.4 The Explicit Regularizer of Data-Augmentations
Revised: 3.4 The Explicit Regularizer of Data-Augmentations

Original: To keep notations as light as possible, we ﬁrst consider a linear regression model with Mean Squared
Revised: To keep notations as light as possible, we ﬁrst consider a linear regression model with Mean Squared

Original: Error (MSE), the nonlinear case will be studied in Section 4.
Revised: Error (MSE), the nonlinear case will be studied in Section 4.

Original: In that setting, the expected loss under
Revised: In that setting, the expected loss under

Original: DA sampling (recall Eq.
Revised: DA sampling (recall Eq.

Original: (3)) becomes
Revised: (3)) becomes

Original: [Equation]
Revised: [Equation]

Original: with x n ∈ R D , n = 1 , . . .
Revised: with x n ∈ R D , n = 1 , . . .

Original: , N the input (ﬂattened)
Revised: , N the input (ﬂattened)

Original: n th image I n , y n ∈ R K the n th target vector, and W ∈ R K × D , b ∈ R K the model’s parameters.
Revised: n th image I n , y n ∈ R K the n th target vector, and W ∈ R K × D , b ∈ R K the model’s parameters.

Original: Recall from Section 3.1 that to learn W under
Revised: Recall from Section 3.1 that to learn W under

Original: DA, the current strategy consists in performing a Monte-Carlo (MC) estimation.
Revised: DA, the current strategy consists in performing a Monte-Carlo (MC) estimation.

Original: Instead, let’s derive (detailed derivation in Appendix D) the exact loss of Eq.
Revised: Instead, let’s derive (detailed derivation in Appendix D) the exact loss of Eq.

Original: (11) as a function of the sample mean and variance under the consider DA.
Revised: (11) as a function of the sample mean and variance under the consider DA.

Original: We will drop the θ subscript for clarity to obtain
Revised: We will drop the θ subscript for clarity to obtain

Original: [Equation]
Revised: [Equation]

Original: 12) with the spectral decomposition Q ( x )Λ( x ) Q ( x ) T =
Revised: 12) with the spectral decomposition Q ( x )Λ( x ) Q ( x ) T =

Original: V [ T ( x )] .
Revised: V [ T ( x )] .

Original: The right term in Eq.
Revised: The right term in Eq.

Original: (12) is the explicit DA regularizer.
Revised: (12) is the explicit DA regularizer.

Original: It pushes the kernel space of W to align with the largest principal directions of the data manifold tangent space, as modeled by the DA.
Revised: It pushes the kernel space of W to align with the largest principal directions of the data manifold tangent space, as modeled by the DA.

Original: In fact, the largest eigenvectors in Q ( x ) represent the principal directions of the data manifold tangent space at x , as encoded via V [ T ( x )] .
Revised: In fact, the largest eigenvectors in Q ( x ) represent the principal directions of the data manifold tangent space at x , as encoded via V [ T ( x )] .

Original: We propose in Fig.
Revised: We propose in Fig.

Original: 3 visualization of Q and Λ for different DAs, illustrating how each DA policy impacts the model’s parameter W through the regularization of Eq.
Revised: 3 visualization of Q and Λ for different DAs, illustrating how each DA policy impacts the model’s parameter W through the regularization of Eq.

Original: The knowledge of E [ T ( x )] and V [ T ( x )] from Theorem 3.3 ﬁnally enables to train a (linear) model on the true expected loss (Eq.
Revised: The knowledge of E [ T ( x )] and V [ T ( x )] from Theorem 3.3 ﬁnally enables to train a (linear) model on the true expected loss (Eq.

Original: as we formalize below.
Revised: as we formalize below.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: Training a linear model with MSE and inﬁnite DA sampling is equivalent to minimizing
Revised: Training a linear model with MSE and inﬁnite DA sampling is equivalent to minimizing

Original: Eq.
Revised: Eq.

Original: (12) and produces the optimal W ∗ model
Revised: (12) and produces the optimal W ∗ model

Original: ’s parameter
Revised: ’s parameter

Original: [Equation]
Revised: [Equation]

Original: Not surprisingly, whenever the DA is identity, E [ T ( x n )] = x n , V [ T ( x n )] = 0 and thus Eq.
Revised: Not surprisingly, whenever the DA is identity, E [ T ( x n )] = x n , V [ T ( x n )] = 0 and thus Eq.

Original: (13) recovers exactly the standard least-square solution.
Revised: (13) recovers exactly the standard least-square solution.

Original: We visualize V [ T ( x n )] for the translation DA in
Revised: We visualize V [ T ( x n )] for the translation DA in

Original: Fig.
Revised: Fig.

Original: The same line of result can be derived in the nonlinear setting by assuming that the DA is restricted to small transformations.
Revised: The same line of result can be derived in the nonlinear setting by assuming that the DA is restricted to small transformations.

Original: In that case, one leverages a truncated Taylor approximation of the nonlinear model 1 and recovers that (local) DA applies the same regularization as in Eq.
Revised: In that case, one leverages a truncated Taylor approximation of the nonlinear model 2 and recovers that (local) DA applies the same regularization as in Eq.

Original: (12) but with the model’s
Revised: (12) but with the model’s

Original: Jacobian matrix J f γ ( x n ) in-place of W (more details in Section 4.3).
Revised: Jacobian matrix J f γ ( x n ) in-place of W (more details in Section 4.3).

Original: Given the above derivation of the exact expected loss, we can now turn to precisely measure how accurate is the MC estimate commonly used to train models under DA sampling.
Revised: Given the above derivation of the exact expected loss, we can now turn to precisely measure how accurate is the MC estimate commonly used to train models under DA sampling.

Original:  Data-Augmentation Sampling Efﬁciency and Loss Sensitivity
Revised:  Data-Augmentation Sampling Efﬁciency and Loss Sensitivity

Original: In this section we study the convergence of the MC estimate (Section 4.1), and provide variance analysis of that estimate as a function of the model’s Jacobian matrix and the sample variance eigenvectors (Section 4.2), concluding by the (re-)discovery of TangentProp from ﬁrst principles (Section 4.3).
Revised: In this section we study the convergence of the MC estimate (Section 4.1), and provide variance analysis of that estimate as a function of the model’s Jacobian matrix and the sample variance eigenvectors (Section 4.2), concluding by the (re-)discovery of TangentProp from ﬁrst principles (Section 4.3).

Original: Those results hold regardless of the type of DA employed.
Revised: Those results hold regardless of the type of DA employed.

Original: Additionally, the analysis is done on a single sample to ease notation, in the i.i.d.
Revised: Additionally, the analysis is done on a single sample to ease notation, in the i.i.d.

Original: setting, simply sum over all the training samples to analyze the training set.
Revised: setting, simply sum over all the training samples to analyze the training set.

Original: 4.1 Empirical Monte-Carlo Convergence of Transformed Images
Revised: 4.1 Empirical Monte-Carlo Convergence of Transformed Images

Original: Given the close-form average image and average loss , we empirically measure how efﬁcient is the
Revised: Given the close-form average image and average loss , we empirically measure how efﬁcient is the

Original: MC estimation from Eq.
Revised: MC estimation from Eq.

Original: We ﬁrst propose in Fig.
Revised: We ﬁrst propose in Fig.

Original: 4 a constructed (64 × 64) image for which we compute the expected loss (Eq.
Revised: 4 a constructed (64 × 64) image for which we compute the expected loss (Eq.

Original: and the MC estimate (right-hand side of Eq.
Revised: and the MC estimate (right-hand side of Eq.

Original: Surprisingly, we obtain that even for such a simple image and translation DA, between 1000 and 10000 samples are required to correctly estimate the MSE loss from the augmented samples.
Revised: Surprisingly, we obtain that even for such a simple image and translation DA, between 1000 and 10000 samples are required to correctly estimate the MSE loss from the augmented samples.

Original: In a more practical scenario, one could rightfully argue that the combination of the DA samples from different images allows to obtain a better estimate with a smaller amount of augmentations per sample.
Revised: In a more practical scenario, one could rightfully argue that the combination of the DA samples from different images allows to obtain a better estimate with a smaller amount of augmentations per sample.

Original: Hence, we provide in Fig.
Revised: Hence, we provide in Fig.

Original: 5 that experiment using a linear model on MNIST [45] with varying train set size.
Revised: 5 that experiment using a linear model on MNIST [45] with varying train set size.

Original: We observe that as the number of samples grows as the required number of augmentation per sample reduces.
Revised: We observe that as the number of samples grows as the required number of augmentation per sample reduces.

Original: Nevertheless, even with thousands of samples, at least 50 augmentations per sample are required to provide an accurate estimate (reproduction with nonlinear models given in Fig. 7).
Revised: Nevertheless, even with thousands of samples, at least 50 augmentations per sample are required to provide an accurate estimate (reproduction with nonlinear models given in Fig. 7).

Original: We now propose to speciﬁcally quantify the sensitivity of the MC estimate to DA sampling.
Revised: We now propose to speciﬁcally quantify the sensitivity of the MC estimate to DA sampling.

Original: 4.2 Loss Sensitivity Under Data-Augmentation Sampling in the Linear and Nonlinear Regime
Revised: 4.2 Loss Sensitivity Under Data-Augmentation Sampling in the Linear and Nonlinear Regime

Original: Recalling Section 3.1, current DA training is performed on a MC estimate of the loss.
Revised: Recalling Section 3.1, current DA training is performed on a MC estimate of the loss.

Original: The estimator’s variance [46] is proportional to the variance of the quantity being estimated:
Revised: The estimator’s variance [46] is proportional to the variance of the quantity being estimated:

Original: V [( L ◦ f )( T ( x ))] .
Revised: V [( L ◦ f )( T ( x ))] .

Original: We now characterize when, and why, would an MC estimator converge for a given model.
Revised: We now characterize when, and why, would an MC estimator converge for a given model.

Original: By leveraging the delta method [47, 48] i.e. a truncated Taylor expansion of the model and loss function mapping as L ◦ f , we have
Revised: By leveraging the delta method [47, 48] i.e. a truncated Taylor expansion of the model and loss function mapping as L ◦ f , we have

Original: [Equation]
Revised: [Equation]

Original: (14) with (cid:107) u (cid:107) 2 A (cid:44) u T Au .
Revised: (14) with (cid:107) u (cid:107) 2 A (cid:44) u T Au .

Original: Noticing that V [ T ( x )]
Revised: Noticing that V [ T ( x )]

Original: =
Revised: =

Original: E [ T ( x ) T ( x ) T ] − E
Revised: E [ T ( x ) T ( x ) T ] − E

Original: [ T ( x )] E
Revised: [ T ( x )] E

Original: [ T ( x )]
Revised: [ T ( x )]

Original: T and using the close-form moments of DST samples from Theorem 3.3, it is possible to write out Eq.
Revised: T and using the close-form moments of DST samples from Theorem 3.3, it is possible to write out Eq.

Original: (14) explicitly for model and DA speciﬁc analysis.
Revised: (14) explicitly for model and DA speciﬁc analysis.

Original: We visualize V
Revised: We visualize V

Original: [ T ( x )] in Fig. 3.
Revised: [ T ( x )] in Fig. 3.

Original: Linear regression case.
Revised: Linear regression case.

Original: To gain some insight into Eq.
Revised: To gain some insight into Eq.

Original: (14), let’s ﬁrst consider the linear regression case leading to V
Revised: (14), let’s ﬁrst consider the linear regression case leading to V

Original: [( L ◦ f )( T ( x ))]
Revised: [( L ◦ f )( T ( x ))]

Original: ≈ (cid:107)
Revised: ≈ (cid:107)

Original: y − W E
Revised: y − W E

Original: [ T ( x )] − b (cid:107) 2 W V [ T ( x )] W T .
Revised: [ T ( x )] − b (cid:107) 2 W V [ T ( x )] W T .

Original: Hence, the estimated loss variance depends (i) on the model predicting the correct output when observing the expected sample E [ T ( x )] , and (ii) on the smallest right singular vector of W to align with the largest eigenvectors of V [ T ( x )] , echoing our observation below Eq.
Revised: Hence, the estimated loss variance depends (i) on the model predicting the correct output when observing the expected sample E [ T ( x )] , and (ii) on the smallest right singular vector of W to align with the largest eigenvectors of V [ T ( x )] , echoing our observation below Eq.

Original: [Figure]
Revised: [Figure]

Original: See
Revised: See

Original: 2 to 5 for
Revised: 2 to 5 for

Original: General case.
Revised: General case.

Original: Since the result from Eq.
Revised: Since the result from Eq.

Original: (14) holds in the general setting, we can generalize the above observation and formalize it into the following statement.
Revised: (14) holds in the general setting, we can generalize the above observation and formalize it into the following statement.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: The variance of the loss’s MC estimate (recall Eq.
Revised: The variance of the loss’s MC estimate (recall Eq.

Original: (3) ) for an input x n goes toif the loss gradient ∇L ( f ( E [ T ( x )])) goes to 0 , or if the kernel of the model’s Jacobian matrix
Revised: (3) ) for an input x n goes toif the loss gradient ∇L ( f ( E [ T ( x )])) goes to 0 , or if the kernel of the model’s Jacobian matrix

Original: J f ( E [ T ( x )]) aligns with the largest eigenvectors of the sample variance V [ T ( x )] .
Revised: J f ( E [ T ( x )]) aligns with the largest eigenvectors of the sample variance V [ T ( x )] .

Original: The proof of the above statement simply starts from Eq.
Revised: The proof of the above statement simply starts from Eq.

Original: (14) and applies the Cauchy-Schwarz inequality two times to obtain
Revised: (14) and applies the Cauchy-Schwarz inequality two times to obtain

Original: [Equation]
Revised: [Equation]

Original: with decomposition V [ T ( x )]
Revised: with decomposition V [ T ( x )]

Original: = Q ( x )Λ( x ) Q ( x ) T .
Revised: = Q ( x )Λ( x ) Q ( x ) T .

Original: From Eq.
Revised: From Eq.

Original: (15), it is quite realistic to assume that the model’s
Revised: (15), it is quite realistic to assume that the model’s

Original: Jacobian does not collapse to 0 (this would not minimize the training loss in general).
Revised: Jacobian does not collapse to 0 (this would not minimize the training loss in general).

Original: Hence, the most realistic case of Theorem 4.1 concerns the alignment between the model’s Jacobian matrix kernel space and the top eigenvectors of V [ T ( x )] .
Revised: Hence, the most realistic case of Theorem 4.1 concerns the alignment between the model’s Jacobian matrix kernel space and the top eigenvectors of V [ T ( x )] .

Original: However, for models like ResNet DNs [49], the loss variance can only reduce to 0 if the model minimizes the loss at the expected input since the Jacobian matrix is always full-rank.
Revised: However, for models like ResNet DNs [49], the loss variance can only reduce to 0 if the model minimizes the loss at the expected input since the Jacobian matrix is always full-rank.

Original: We recall that the above relied on the delta method to approximate the intractable LHS of Eq.
Revised: We recall that the above relied on the delta method to approximate the intractable LHS of Eq.

Original: (14) (see footnote on Page 7) .
Revised: (14) (see footnote on Page 7) .

Original: 4.3 Explicit Loss Sensitivity Minimization Provably Recovers TangentProp
Revised: 4.3 Explicit Loss Sensitivity Minimization Provably Recovers TangentProp

Original: The expected loss (Eq. (3)) has been derived in the linear regression setting (Eq.
Revised: The expected loss (Eq. (3)) has been derived in the linear regression setting (Eq.

Original: But in a more general scenario, and as discussed below Theorem 3.4, this expectation might not be tractable and thus needs to be approximated e.g. based on a Taylor expansion of the loss and model.
Revised: But in a more general scenario, and as discussed below Theorem 3.4, this expectation might not be tractable and thus needs to be approximated e.g. based on a Taylor expansion of the loss and model.

Original: Alternatively to using the approximated expectation, one could employ the usual MC estimate of the expectation (right-hand side of Eq.
Revised: Alternatively to using the approximated expectation, one could employ the usual MC estimate of the expectation (right-hand side of Eq.

Original: (3)), and leverage the MC estimator variance obtained in Eq.
Revised: (3)), and leverage the MC estimator variance obtained in Eq.

Original: (15) as a regularizer.
Revised: (15) as a regularizer.

Original: We show here that both approaches are equivalent, and recover a popular regularizer known as TangentProp [50].
Revised: We show here that both approaches are equivalent, and recover a popular regularizer known as TangentProp [50].

Original: Although various extensions of TangentProp have been introduced [51, 52] no principled derivation of it has yet been proposed.
Revised: Although various extensions of TangentProp have been introduced [51, 52] no principled derivation of it has yet been proposed.

Original: Using the same argument as in
Revised: Using the same argument as in

Original: Section 4.2, the expectation of a nonlinear transformation of a random variable ( T ( x n ) ) can be
Revised: Section 4.2, the expectation of a nonlinear transformation of a random variable ( T ( x n ) ) can be

Original: [Table]
Revised: [Table]

Original: expected as the introduced DA variations become redundant with the training set samples.
Revised: expected as the introduced DA variations become redundant with the training set samples.

Original: This marginal gap can be slightly increased by employing stronger DA (compare top half and bottom half of the table).
Revised: This marginal gap can be slightly increased by employing stronger DA (compare top half and bottom half of the table).

Original: However, in the small dataset size regime , there is an important gain provided by employing the regularizer even when compared to observing many DA samples (#DA=10).
Revised: However, in the small dataset size regime , there is an important gain provided by employing the regularizer even when compared to observing many DA samples (#DA=10).

Original: This indicates a potentially crucial regime in which introducing the analytical regularizer helps.
Revised: This indicates a potentially crucial regime in which introducing the analytical regularizer helps.

Original: More surprisingly, this is also the case if part of DA policy is misaligned with the task at hand (e.g. rotation for FashionMNIST).
Revised: More surprisingly, this is also the case if part of DA policy is misaligned with the task at hand (e.g. rotation for FashionMNIST).

Original: Additional empirical results with more training set sizes ( N ∈ { 100 , 200 , 500 , 1000 , 3000 , 10000 , 20000 } ) and DA policies (one per table) are available in Tables 2 to 5 for MNIST, in Tables 6 to 9 for EMNIST, and in Tables 10 to 13 for FashionMNIST approximated from the expectation of the Taylor expansion (detailed derivations in Appendix E) from which an upper-bound is obtained by applying the Cauchy-Schwarz inequality
Revised: Additional empirical results with more training set sizes ( N ∈ { 100 , 200 , 500 , 1000 , 3000 , 10000 , 20000 } ) and DA policies (one per table) are available in Tables 2 to 5 for MNIST, in Tables 6 to 9 for EMNIST, and in Tables 10 to 13 for FashionMNIST approximated from the expectation of the Taylor expansion (detailed derivations in Appendix E) from which an upper-bound is obtained by applying the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: with κ ( x ) ≥ 0 and the notations from Eq.
Revised: with κ ( x ) ≥ 0 and the notations from Eq.

Original: As a result, the TangentProp regularizer naturally appears when using a Taylor approximation of the expected loss, and it corresponds to adding an explicit loss variance regularization term (compare the TangentProp with Eq.
Revised: As a result, the TangentProp regularizer naturally appears when using a Taylor approximation of the expected loss, and it corresponds to adding an explicit loss variance regularization term (compare the TangentProp with Eq.

Original: We thus obtained from ﬁrst principles that TangentProp emerges naturally when considering the second order Taylor approximation of the expected loss given a DA.
Revised: We thus obtained from ﬁrst principles that TangentProp emerges naturally when considering the second order Taylor approximation of the expected loss given a DA.

Original:  Conclusions and Limitations
Revised:  Conclusions and Limitations

Original: In this paper, we proposed a novel set of mathematical tools (Sections 3.2 and 3.3) under which it is possible to study DA and to provably answer some of the open questions around the efﬁciency and impact of DA to train a model.
Revised: In this paper, we proposed a novel set of mathematical tools (Sections 3.2 and 3.3) under which it is possible to study DA and to provably answer some of the open questions around the efﬁciency and impact of DA to train a model.

Original: We ﬁrst obtained the explicit regularizer produced by different DAs in
Revised: We ﬁrst obtained the explicit regularizer produced by different DAs in

Original: Section 3.4.
Revised: Section 3.4.

Original: This led to the following observation: the kernel space of the W matrix is pushed to align with the largest eigenvectors of the sample covariance matrix.
Revised: This led to the following observation: the kernel space of the W matrix is pushed to align with the largest eigenvectors of the sample covariance matrix.

Original: This was then studied in a more general setting in Section 4.2 for nonlinear models when characterizing the loss variance under DA sampling.
Revised: This was then studied in a more general setting in Section 4.2 for nonlinear models when characterizing the loss variance under DA sampling.

Original: We also observed that Monte-Carlo sampling of transformed images is highly inefﬁcient (Section 4.1), even if similar training samples combine their underlying information within a dataset.
Revised: We also observed that Monte-Carlo sampling of transformed images is highly inefﬁcient (Section 4.1), even if similar training samples combine their underlying information within a dataset.

Original: Lastly, those derivations led us to provably derive a known regularizer —TangentProp— as being the natural minimizer of a model’s loss variance.
Revised: Lastly, those derivations led us to provably derive a known regularizer —TangentProp— as being the natural minimizer of a model’s loss variance.

Original: The main limitation of the current analysis concerns nonlinear models for which we rely on a third-order Taylor expansion.
Revised: The main limitation of the current analysis concerns nonlinear models for which we rely on a third-order Taylor expansion.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] We have precisely given the scope of our results in the end of the introduction.
Revised: [Yes] We have precisely given the scope of our results in the end of the introduction.

Original: Additionally, we recall the limitations and assumptions of our results within each section carefully.
Revised: Additionally, we recall the limitations and assumptions of our results within each section carefully.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] The main limitations comes from the approximation done for nonlinear models which we have repeatedly highlight in both the introductions and the corresponding sections.
Revised: [Yes] The main limitations comes from the approximation done for nonlinear models which we have repeatedly highlight in both the introductions and the corresponding sections.

Original: We also highlight those in the conclusion along with future work.
Revised: We also highlight those in the conclusion along with future work.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We do not see any potential negative societal impact for this work (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] We do not see any potential negative societal impact for this work (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] we have carefully given the assumptions each result relies on.
Revised: [Yes] we have carefully given the assumptions each result relies on.

Original: Especially, at the end of the introduction, we have highlighted a reference paper where it was shown that our assumptions are common in the literature, and that they are good enough to gain insights into nonlinear models.
Revised: Especially, at the end of the introduction, we have highlighted a reference paper where it was shown that our assumptions are common in the literature, and that they are good enough to gain insights into nonlinear models.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Each formal statement and derivation is linked to a proof in the appendix.
Revised: [Yes] Each formal statement and derivation is linked to a proof in the appendix.

Original: We also provide details derivations in the appendix of all the equations we have obtained in the main paper, even if they are not part of a formal result.
Revised: We also provide details derivations in the appendix of all the equations we have obtained in the main paper, even if they are not part of a formal result.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide samples of the code to reproduce ﬁgures.
Revised: [Yes] We provide samples of the code to reproduce ﬁgures.

Original: The entire codebase will be released publicly upon completion of the review-process (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: The entire codebase will be released publicly upon completion of the review-process (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We only explore MNIST and provided the training details in the caption of the corresponding ﬁgure (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] We only explore MNIST and provided the training details in the caption of the corresponding ﬁgure (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] We did not provide error bars but multiple runs with various settings (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [No] We did not provide error bars but multiple runs with various settings (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] The employed experiment is small scale and can be performed without GPU 4.
Revised: [No] The employed experiment is small scale and can be performed without GPU 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] MNIST only (b) Did you mention the license of the assets?
Revised: [Yes] MNIST only (b) Did you mention the license of the assets?

Original: [No] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [No] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]