Original: Mixed Samples Data Augmentation with Replacing Latent Vector Components in Normalizing FlowAnonymous Author(s)
Revised: Mixed Samples Data Augmentation with Replacing Latent Vector Components in Normalizing Flow

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Data augmentation mixing two samples has been acknowledged as an effectiveregularization method for various deep neural network models.
Revised: Data augmentation mixing two samples has been acknowledged as an effective regularization method for various deep neural network models.

Original: Given that imagesmixed by popular methods (e.g., MixUp and CutMix) are unnatural to the humaneye, we hypothesized that generating more natural images could achieve betterperformance as data augmentation.
Revised: Given that images mixed by popular methods (e.g., MixUp and CutMix) are unnatural to the human eye, we hypothesized that generating more natural images could achieve better performance as data augmentation.

Original: To verify this, we propose a new mixing methodthat synthesizes images in which two source images coexist naturally.
Revised: To verify this, we propose a new mixing method that synthesizes images in which two source images coexist naturally.

Original: Our methodperforms a mixing operation in latent space through a normalizing ﬂow and thekey is how to mix two latent vectors.
Revised: Our method performs a mixing operation in latent space through a normalizing ﬂow, and the key is how to mix two latent vectors.

Original: We preliminarily observed that there existsa dependency between the dimensions in input space and those in latent space intransformation with normalizing ﬂows.
Revised: We preliminarily observed that there exists a dependency between the dimensions in input space and those in latent space in transformation with normalizing ﬂows.

Original: Based on this observation, we designed ourmixing scheme in latent space.
Revised: Based on this observation, we designed our mixing scheme in latent space.

Original: We show that our method yields visually naturalaugmented images and improves classiﬁcation performance
Revised: We show that our method yields visually natural augmented images and improves classiﬁcation performance.

Original:  Introduction
Revised:  Introduction

Original: [Figure]
Revised: [Figure]

Original: Unlike MixUp and CutMix, our method, LS-Mix, generates natural mixed images.
Revised: MixUp and CutMix, our method, LS-Mix, generates natural mixed images.

Original: Data augmentation methods that generate newtraining data by mixing two source samples havebeen increasingly popular to regularize deep neural network models.
Revised: Data augmentation methods that generate new training data by mixing two source samples have been increasingly popular to regularize deep neural network models.

Original: We refer to those methodsas mixed samples data augmentation (MDA).
Revised: We refer to those methods as mixed samples data augmentation (MDA).

Original: Even basic methods such as MixUp [1, 2] and
Revised: Even basic methods such as MixUp [1, 2] and

Original: CutMix [3] are widely acknowledged for theireffectiveness.
Revised: CutMix [3] are widely acknowledged for their effectiveness.

Original: However, their success is somewhat surprising, given that the images generatedby these methods are unnatural to the human eye.(see Fig. 1.)
Revised: However, their success is somewhat surprising, given that the images generated by these methods are unnatural to the human eye. (see Fig. 1.)

Original: Considering that input images atthe test time do not contain such unnaturalness,we hypothesize that if we can generate mixedsamples more naturally, it would become more effective as augmented data.
Revised: Considering that input images at the test time do not contain such unnaturalness, we hypothesize that if we can generate mixed samples more naturally, it would become more effective as augmented data.

Original: We thus propose an MDA method that synthesizes images, in which two source images coexistnaturally by being stitched together without producing artifacts.
Revised: We thus propose an MDA method that synthesizes images, in which two source images coexist naturally by being stitched together without producing artifacts.

Original: We perform mixing operations inlatent space instead of input data space using lossless invertible transformation with normalizingﬂows (NFs)
Revised: We perform mixing operations in latent space instead of input data space using lossless invertible transformation with normalizing ﬂows (NFs)

Original: How to mix two latent vectors is the key to generating natural images.
Revised: How to mix two latent vectors is the key to generating natural images.

Original: As wewill describe in Section 3, we found a dependency between the dimensions of a latent vector andthe pixels of its corresponding image.
Revised: As we will describe in Section 3, we found a dependency between the dimensions of a latent vector and the pixels of its corresponding image.

Original: By utilizing that dependency, we design a mixing schemethat naturally stitches part of two source images while preserving their original appearance to themaximum extent.
Revised: By utilizing that dependency, we design a mixing scheme that naturally stitches part of two source images while preserving their original appearance to the maximum extent.

Original: We call our method Latent space Sequential Mix (LS-Mix).
Revised: We call our method Latent space Sequential Mix (LS-Mix).

Original: The overview of
Revised: The overview of

Original: LS-Mix is shown in Fig. 2, and the images generated by LS-Mix are shown in Fig.
Revised: LS-Mix is shown in Fig. 2, and the images generated by LS-Mix are shown in Fig.

Original: In this short paper, we perform the evaluation on SVHN, CIFAR-10, CIFAR-100, and TinyImageNet, and showthat LS-Mix outperforms MixUp, CutMix and other methods that perform mixing in latent space.
Revised: In this short paper, we perform the evaluation on SVHN, CIFAR-10, CIFAR- and TinyImageNet and show that LS-Mix outperforms MixUp, CutMix and other methods that perform mixing in latent space.

Original:  Preliminary
Revised:  Preliminary

Original: We use normalizing ﬂows (NFs), g ( · ) , for the invertible transformation between the input space Xand the latent space Z .
Revised: We use normalizing ﬂows (NFs), g ( · ) , for the invertible transformation between the input space X and the latent space Z .

Revised: An NF produces clear reconstructed images because of its invertibility, unlike

Revised: VAE [7, 8], whose reconstructed images are known to become blurred [9, 10].

Original: See Appendix A for NFs.
Revised: See Appendix A for NFs.

Original: The architecture we use is Glow [7].
Revised: The architecture we use is Glow [11].

Original: Our proposedmethod utilizes the mechanism called squeezing , which is one of the building blocks of NFs.
Revised: Our proposed method utilizes the mechanism called squeezing , which is one of the building blocks of NFs.

Original: Squeezing.
Revised: Squeezing.

Original: While the shape of images x is [ h, w, c ] with spatial dimensions h × w and channeldimension c , the latent vectors z = g ( x ) mapped with a NF are D dimensional ﬂat vectors with nospatial structure, where D = h × w × c. NFs perform the conversion of shapes between x and z by anoperation called squeezing .
Revised: While the shape of images x is [ h, w, c ] with spatial dimensions h × w and channel dimension c , the latent vectors z = g ( x ) mapped with a NF are D dimensional ﬂat vectors with no spatial structure, where D = h × w × c. NFs perform the conversion of shapes between x and z by an operation called squeezing .

Original: The squeezing is typically done with iterative use of space _ to _ depthoperation which converts the shape of inputs from [ h, w, c ] to [ h 2 , w 2 , 4 c ] until it becomes [1 , 1 , D ] ,which are treated as D (= 1 × 1 × D ) dimensional ﬂat vectors in the implementation.
Revised: The squeezing is typically done with iterative use of space _ to _ depth operation which converts the shape of inputs from [ h, w, c ] to [ h 2 , w 2 , 4 c ] until it becomes [1 , 1 , D ] , which are treated as D (= 1 × 1 × D ) dimensional ﬂat vectors in the implementation.

Original: In the inverse transformation g − 1 that transforms z back to x , the opposite conversion, depth _ to _ spaceoperation is performed.
Revised: In the inverse transformation g − 1 that transforms z back to x , the opposite conversion, depth _ to _ space operation is performed.

Original: How exactly the space _ to _ depth and depth _ to _ space operations perform the folding of the spatial dimensions into the channel dimension depends on the implementation, but typically it is implemented to work in a manner equal to tf .
Revised: How exactly the space _ to _ depth and depth _ to _ space operations perform the folding of the spatial dimensions into the channel dimension depends on the implementation, but typically it is implemented to work in a manner equal to tf .

Original: nn . space _ to _ depth andtf .
Revised: nn . space _ to _ depth and tf .

Original: nn . depth _ to _ space API in TensorFlow, which we use in our experiments.
Revised: nn . depth _ to _ space API in TensorFlow, which we use in our experiments.

Original: We depict the processof space _ to _ depth and depth _ to _ space in Fig.
Revised: We depict the process of space _ to _ depth and depth _ to _ space in Fig.

Original:  Our Method
Revised:  Our Method

Original: Motivating observation: squeezing dependency.
Revised: Motivating observation: squeezing dependency.

Original: The key to synthesizing natural images is howto mix two latent vectors.
Revised: The key to synthesizing natural images is how to mix two latent vectors.

Original: We preliminarily observed that in the transformation with NFs, thereexists a dependency between the dimensions in X and Z , which we call squeezing dependency .
Revised: We preliminarily observed that in the transformation with NFs, there exists a dependency between the dimensions in X and Z , which we call squeezing dependency .

Original: Weexperimentally observed how the generated image x = g − 1 ( z ) changes when some consecutivecomponents of latent vector z are replaced with 0-value.
Revised: We experimentally observed how the generated image x = g − 1 ( z ) changes when some consecutive components of latent vector z are replaced with 0-value.

Original: Fig.
Revised: Fig.

Original: 3 shows that as the number of consecutive components replaced with 0 gradually increases, the generated image becomes closer tobeing all black.
Revised: 3 shows that as the number of consecutive components replaced with 0 gradually increases, the generated image becomes closer to being all black.

Original: We can see that the transition to the black image follows not the way that the entireimage gradually becomes darker but instead the way that a small black area initially appearing inthe lower right corner expands to cover the entire image.
Revised: We can see that the transition to the black image follows not the way that the entire image gradually becomes darker but instead the way that a small black area initially appearing in the lower right corner expands to cover the entire image.

Original: We identiﬁed that the order of the blackarea expansion is in accordance with the alignment order of the depth _ to _ space conversion in thesqueezing operation described in Section 2.
Revised: We identiﬁed that the order of the black area expansion is in accordance with the alignment order of the depth _ to _ space conversion in the squeezing operation described in Section 2.

Original: We thus interpreted that this dependency comes fromthe order of squeezing applied during the training of the NF model.
Revised: We thus interpreted that this dependency comes from the order of squeezing applied during the training of the NF model.

Original: We note that what we did inthis experiment corresponds to the case where all elements of z 1 in Fig.
Revised: We note that what we did in this experiment corresponds to the case where all elements of z 1 in Fig.

Original: 2 were replaced by 0.
Revised: 2 were replaced by 0.

Original: Fromthis observation, it is shown that consecutive components in z correspond to consecutive areas inx = g − 1 ( z ) .
Revised: From this observation, it is shown that consecutive components in z correspond to consecutive areas in x = g − 1 ( z ) .

Original: It suggests that manipulating the latent vectors following the squeezing dependencyallows us to generate an image preserving a speciﬁc region of the source images.
Revised: It suggests that manipulating the latent vectors following the squeezing dependency allows us to generate an image preserving a speciﬁc region of the source images.

Original: Based on thisobservation, we designed our mixing method.
Revised: Based on this observation, we designed our mixing method.

Original: Method.
Revised: Method.

Original: We call our method Latent space Sequential Mix (LS-Mix).
Revised: We call our method Latent space Sequential Mix (LS-Mix).

Original: LS-Mix works as follows:mapping two source inputs x 1 and x 2 to the latent space
Revised: LS-Mix works as follows: mapping two source inputs x 1 and x 2 to the latent space

Original: Z , we obtain the corresponding latentvectors, z 1 = g ( x 1 ) and z 2 = g ( x 2 ) .
Revised: Z , we obtain the corresponding latent vectors, z 1 = g ( x 1 ) and z 2 = g ( x 2 ) .

Original: The mixing operation is done as
Revised: The mixing operation is done as

Original: [Equation]
Revised: [Equation]

Original: where M ∈ { 0 , 1 } D is a binary mask, D is the dimensionality of Z (and X ), 1 is a D -dimensionalvector in which all elements are 1, and ⊗ is an element-wise product.
Revised: where M ∈ { 0 , 1 } D is a binary mask, D is the dimensionality of Z (and X ), 1 is a D -dimensional vector in which all elements are 1, and ⊗ is an element-wise product.

Original: The key of our method is howto create M , and following the squeezing dependency, we simply set the ﬁrst (cid:98) λD (cid:101) components of
Revised: The key of our method is how to create M , and following the squeezing dependency, we simply set the ﬁrst (cid:98) λD (cid:101) components of

Original: M to 0 and 1 otherwise, according to the mixing rate λ .
Revised: M to 0 and 1 otherwise, according to the mixing rate λ .

Original: As the value of λ increases from 0 to 1,the size of the area that comes from x 2 via z 2 increases, following the order of depth _ to _ spaceoperation.
Revised: As the value of λ increases from 0 to 1, the size of the area that comes from x 2 via z 2 increases, following the order of depth _ to _ space operation.

Original: After mixing, mapping z m back to X by g − 1 , we obtain synthesized image x m = g − 1 ( z m ) .
Revised: After mixing, mapping z m back to X by g − 1 , we obtain synthesized image x m = g − 1 ( z m ) .

Original: Fig.
Revised: Fig.

Original: 2 illustrates the case of D = 16 and λ = 516 .
Revised: 2 illustrates the case of D = 16 and λ = 5 16 .

Original: Like the blue and green regions on x m in Fig. 2,
Revised: Like the blue and green regions on x m in Fig. 2,

Original: LS-Mix performs mixing in such a way that the parts of each source image are embedded spatiallycontinuously in the generated x m .
Revised: LS-Mix performs mixing in such a way that the parts of each source image are embedded spatially continuously in the generated x m .

Original: Indeed, other mixing ways can be thought of, and we evaluate fouralternatives, including Linear interpolation and Bernoulli mixup (Bern-Mix) [8], in Sections 4 and 5.
Revised: Indeed, other mixing ways can be thought of, and we evaluate four alternatives, including Linear interpolation and Bernoulli mixup (Bern-Mix) [12], in Sections 4 and 5.

Original:  Experiments
Revised:  Experiments

Original: Preparation.
Revised: Preparation.

Original: We evaluated LS-Mix in classiﬁcation accuracies, comparing to the baseline model(trained without MDA), MixUp, and CutMix.
Revised: We evaluated LS-Mix in classiﬁcation accuracies, comparing to the baseline model (trained without MDA), MixUp, and CutMix.

Original: The datasets we used are SVHN [9], CIFAR10/100 [10], and TinyImageNet [11], the details of which are shown in Appendix B.1.
Revised: The datasets we used are SVHN [13], CIFAR10/100 [14], and TinyImageNet [15], the details of which are shown in Appendix B.1.

Original: Following theprevious works such as [12], we used two architectures for the classiﬁer, the 13-layer CNN (CNN-13)and Wide-ResNet-28-10 (WRN-28-10) 1 [13].
Revised: Following the previous works such as [16], we used two architectures for the classiﬁer, the 13-layer CNN (CNN-13) and Wide-ResNet-28-10 (WRN-28-10) 1 [17].

Original: The CNN-13 has been used in the literature suchas [14, 15, 16, 17, 18, 19], whose architecture is described in Appendix B.2.
Revised: The CNN-13 has been used in the literature such as [18, 19, 20, 21, 22, 10], whose architecture is described in Appendix B.2.

Original: We ﬁrst trained only the
Revised: We ﬁrst trained only the

Original: Glow model separately from the classiﬁer for each dataset, and we use the same model throughout allthe experiments of classiﬁer training.
Revised: Glow model separately from the classiﬁer for each dataset, and we use the same model throughout all the experiments of classiﬁer training.

Revised: For the model parameters and the training settings, we would

Revised: [Table]

Original: For the model parameters and the training settings, we wouldlike to refer the reader to Appendix B. MDA methods, including LS-Mix, have a hyper-parameterα ∈ (0 , 1) , based on which the mixing rate λ is sampled as λ ∼ Beta ( α, α ) where Beta ( · ) is a betadistribution.
Revised: [Table] [Table] [Table] like to refer the reader to Appendix B. MDA methods, including LS-Mix, have a hyper-parameter α ∈ (0 , 1) , based on which the mixing rate λ is sampled as λ ∼ Beta ( α, α ) where Beta ( · ) is a beta distribution.

Original: We tested α ∈ (0 . 1 , 1 . 0) for MixUp, the full results of which are shown in Table 6 in[Table] [Table] [Table]
Revised: We tested α ∈ (0 . 1 , 1 . 0) for MixUp, the full results of which are shown in Table 6 in

Original: [Table] Appendix C, and we picked the best ones, α = 0 .
Revised: Appendix C, and we picked the best ones, α = 0 .

Original: 7 , through the experiments.
Revised: 7 , through the experiments.

Original: For CutMix, we usedα = 1 . 

Revised: For CutMix, we used α = 1 .

Original: according to [3].
Revised: according to [3].

Original: We run the experiments on a single NVIDIA Quadro P5000 GPU.
Revised: We run the experiments on a single NVIDIA Quadro P5000 GPU.

Original: Results.
Revised: Results.

Original: We ﬁrst evaluated LS-Mix with different α using CNN-13 classiﬁer on SVHN and CIFAR10.
Revised: We ﬁrst evaluated LS-Mix with different α using CNN-13 classiﬁer on SVHN and CIFAR10.

Original: To compare the mixing scheme employed in LS-Mix with different mixing ways in Z , wealso evaluated two other methods that perform mixing in Z , Linear interpolation ( Linear Intrpl )and Bernoulli mixup ( Bern-Mix ) [8], Linear Intrpl performs linear interpolation in Z .
Revised: To compare the mixing scheme employed in LS-Mix with different mixing ways in Z , we also evaluated two other methods that perform mixing in Z , Linear interpolation ( Linear Intrpl ) and Bernoulli mixup ( Bern-Mix ) [12], Linear Intrpl performs linear interpolation in Z .

Original: Mixing by
Revised: Mixing by

Original: Bern-Mix is written as Eq. 1, but unlike LS-Mix, the binary mask M in Bern-Mix is made at randomby sampling from Bernoulli ( λ ) distribution.
Revised: Bern-Mix is written as Eq. 1, but unlike LS-Mix, the binary mask M in Bern-Mix is made at random by sampling from Bernoulli ( λ ) distribution.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: The LS-Mix achievedthe best results regardless of the value of α .
Revised: The LS-Mix achieved the best results regardless of the value of α .

Original: Linear Intrpl and Bern-Mix were less effective than
Revised: Linear Intrpl and Bern-Mix were less effective than

Original: MixUp and CutMix on CIFAR-10.
Revised: MixUp and CutMix on CIFAR-10.

Original: We present the synthesized images in Appendix E. We found thatthe images produced by Linear Intrpl are almost totally darkened entirely when λ is in a neighborhoodaround 0.5.
Revised: We present the synthesized images in Appendix E. We found that the images produced by Linear Intrpl are almost totally darkened entirely when λ is in a neighborhood around 0.5.

Original: The phenomenon of all-gray images concentrated in the center of Z has been reportedin [20], and the result of Linear Intrpl is considered to be the same phenomenon.
Revised: The phenomenon of all-gray images concentrated in the center of Z has been reported in [23], and the result of Linear Intrpl is considered to be the same phenomenon.

Original: Also, we saw that
Revised: Also, we saw that

Original: Bern-Mix produces unnatural images, as described in [8].
Revised: Bern-Mix produces unnatural images, as described in [12].

Original: Next, we evaluated the methods with WRN-28-10 classiﬁer on CIFAR-10, CIFAR-100, and TinyImameNet.
Revised: Next, we evaluated the methods with WRN-28-10 classiﬁer on CIFAR-10, CIFAR-100, and TinyImameNet.

Original: We set α to 0 .
Revised: We set α to 0 .

Original: 4 for LS-Mix, 0 . 7 for MixUp, and 1 . 

Revised: 4 for LS-Mix, 0 . 7 for MixUp, and 1 .

Original: for CutMix on all datasets.
Revised: for CutMix on all datasets.

Original: Theresults in Table 2 show that LS-Mix also outperformed other methods.
Revised: The results in Table 2 show that LS-Mix also outperformed other methods.

Original: We also tested the performance of the combined use of LS-Mix, MixUp, and CutMix.
Revised: We also tested the performance of the combined use of LS-Mix, MixUp, and CutMix.

Original: As shownin Table 3, the combined use always improves the performance.
Revised: As shown in Table 3, the combined use always improves the performance.

Original: In particular, LS-Mix + CutMixachieved remarkable improvement.
Revised: In particular, LS-Mix + CutMix achieved remarkable improvement.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: [Table]
Revised: [Table]

Original: Aside from Linear Intrpl and Bern-Mix, alternative mixingschemes in the latent space are possible.
Revised: Aside from Linear Intrpl and Bern-Mix, alternative mixing schemes in the latent space are possible.

Original: We introduced twomore alternatives, Swap-Mix and Reverse-Mix .
Revised: We introduced two more alternatives, Swap-Mix and Reverse-Mix .

Original: The mixingschemes and synthesized images are presented in Appendix
Revised: The mixing schemes and synthesized images are presented in Appendix

Original: D. We found that those two yield images drastically deformedfrom the source image and that their performance as MDA isfar worse than LS-Mix, as shown in Table 4.
Revised: D. We found that those two yield images drastically deformed from the source image and that their performance as MDA is far worse than LS-Mix, as shown in Table 4.

Original: This ablationstudy implies that complying with the squeezing dependency,as in LS-Mix, is important when performing mixes that replacecomponents of the latent vectors.
Revised: This ablation study implies that complying with the squeezing dependency, as in LS-Mix, is important when performing mixes that replace components of the latent vectors.

Original: Unlike other mixing methods in latent space (i.e., Linear Intrpl, Bern-Mix [8], Swap-Mix, and
Revised: Unlike other mixing methods in latent space (i.e., Linear Intrpl, Bern-Mix [12], Swap-Mix, and

Original: Reverse-Mix), LS-Mix is designed to preserve the original structure of both source images as much aspossible and to stitch them naturally.
Revised: Reverse-Mix), LS-Mix is designed to preserve the original structure of both source images as much as possible and to stitch them naturally.

Original: That enables LS-Mix to yield natural images, and we empiricallyshowed that such images work better as data augmentation. Although it is necessary to evaluate itseffectiveness for large-size images such as ImageNet [11] in the future, this short paper demonstratedthat the LS-Mix is promising as an MDA method. Checklist For all authors...(
Revised: That enables LS-Mix to yield natural images, and we empirically

Original: a

Original: ) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes](

Original: b

Original: ) Did you describe the limitations of your work?

Original: [Yes] As we mentioned in the end of

Original: Section 5, we left the evaluation with large-size images for future work..(

Original: c

Original: ) Did you discuss any potential negative societal impacts of your work?

Original: [N A]( d)
Revised: [Table]

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: Yes

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b Did you include complete proofs of all theoretical results?
Revised: showed that such images work better as data augmentation.

Original: [N

Original: /A]3.

Original: If

Original: you

Original: ran

Original: experiments..

Original: .(a

Original: Did

Original: you

Original: include the code,

Original: data, and instructions needed to reproduce the main experimental results either in the supplemental material or as a URL)? [Yes ] We proviedethe experimental code in the supplemental material.(b)
Revised: Although it is necessary to evaluate its effectiveness for large-size images such as ImageNet [15] in the future, this short paper demonstrated that the LS-Mix is promising as an MDA method.

Original: Did you specify all the training details (e.g

Original: data

Original: splits, hyperparameters, how theywere chosen)?

Original: [Yes] We described them
Revised: Acknowledgements

Original: mainly in the Appendix.(c)

Original: Did

Original: you

Original: report error bars (e.g.,

Original: with

Original: respect to the random seed after running experiments multiple times)?

Original: [Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)

Original: [Yes

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a If your work uses existing assets did you cite the creators? Yes ](b) Did you mention the license of the assets?
Revised: The authors would like to thank Eric Jang for insightful discussions.

Original: [N/A](c)

Original: Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A](e)

Original: Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?

Original: [N/A]5.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N

Original: A](b)

Original: Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N

Original: A](c

Original: Did you

Original: include

Original: the estimated hourly wage paid to participants and the total amountspent on participant compensation

Original: [N/A

Original: A Normalizing Flow
Revised: A Normalizing Flow

Original: Let g ( · ) be an invertible function and h 0 and h 1 be random variables of equal dimensionality.
Revised: Let g ( · ) be an invertible function and h 0 and h 1 be random variables of equal dimensionality.

Original: With the change of variables rule, a transformation h 1 =
Revised: With the change of variables rule, a transformation h 1 =

Original: g ( h 0 ) can be written as the change inthe probability density function (pdf): p ( h 0 ) =
Revised: g ( h 0 ) can be written as the change in the probability density function (pdf): p ( h 0 ) =

Original: p ( h 1 ) | det ( d h 1 /d h 0 ) | .
Revised: p ( h 1 ) | det ( d h 1 /d h 0 ) | .

Original: Deﬁning h 0 :
Revised: Deﬁning h 0 :

Original: = x andh T :=
Revised: = x and h T :

Original:  z , T -times repetition of this transformation, h 1 , h 2 , . .
Revised: = z , T -times repetition of this transformation, h 1 , h 2 , . .

Original: . , h T , yields log p ( x ) = log p ( z ) +262 (cid:80)
Revised: . , h T , yields log p ( x ) = log p ( z ) + (cid:80)

Original: Tt =1 log | det ( d h t /d h t − 1 ) | , which gives us a invertible map between an image x and a correspondentlatent vector z .
Revised: Tt =1 log | det ( d h t /d h t − 1 ) | , which gives us a invertible map between an image x and a correspondent latent vector z .

Original: The target distribution p ( z ) , i.e., the latent space Z , can be set to arbitrary distribution,but we choose a standard Gaussian N (0 , I ) .
Revised: The target distribution p ( z ) , i.e., the latent space Z , can be set to arbitrary distribution, but we choose a standard Gaussian N (0 , I ) .

Original: Due to an invertible mapping, the dimensionality of Zis equal to that of the input space X , which we denote as D .
Revised: Due to an invertible mapping, the dimensionality of Z is equal to that of the input space X , which we denote as D .

Original: Through the training the ﬂow modellearns g () so that it transforms the distribution of training images p ( x ) into N (0 , I ) .
Revised: Through the training the ﬂow model learns g () so that it transforms the distribution of training images p ( x ) into N (0 , I ) .

Original: There are severaltypes of ﬂow models, but we use Glow [7], which is the most popular and has been used in manyapplications such as [21, 19].
Revised: There are several types of ﬂow models, but we use Glow [11], which is the most popular and has been used in many applications such as [24, 10].

Original: [Figure]
Revised: A.1 Factor Out

Original: To reduce computation cost and memory usage, ﬂow models, including Glow, typically employ themechanism called factor out [6, 7].
Revised: To reduce computation cost and memory usage, ﬂow models, including Glow, typically employ the mechanism called factor out [6, 11].

Original: Fig.
Revised: Fig.

Original: 5 shows the illustration cited from [6], and it is formalized as( z ( i +1) , h ( i + [Equation] ( i ) ( h ( i
Revised: 5 shows the illustration cited from [6], and it is formalized as

Original: [Equation]
Revised: [Equation]

Original: (2) z ( L

Original: [Equation]

Original: [Equation]

Original: = g ( L ) ( h ( L ) ) (3)

Original: z

Original: [Equation] [Equation] [Equation] (4) where L is the number of factor out operations.
Revised: where L is the number of factor out operations.

Original: Also, z ( i ) (and h ( i ) ) and g ( i ) indicate the outputs ofi -th operation and i -th transform function, respectively.
Revised: Also, z ( i ) (and h ( i ) ) and g ( i ) indicate the outputs of i -th operation and i -th transform function, respectively.

Original: For i <
Revised: For i <

Original: L + 1 , the dimension of the outputsof g ( i ) are split in half (Eq.
Revised: L + 1 , the dimension of the outputs of g ( i ) are split in half (Eq.

Original: The splitting is not applied for i =
Revised: The splitting is not applied for i =

Original: L (Eq.
Revised: L (Eq.

Original: All z ( i ) which havebeen factored out at different timing are concatenated to obtain the ﬁnal output
Revised: All z ( i ) which have been factored out at different timing are concatenated to obtain the ﬁnal output

Original: z (Eq. (4)).
Revised: z (Eq. (4)).

Original: Thus, thecomponents of z are gradually stacked in L + 1 times.
Revised: Thus, the components of z are gradually stacked in L + 1 times.

Original: For example, when L = 4 for 64 × 64 size
Revised: For example, when L = 4 for 64 × 64 size

Original: RGB image datasets, z is composed of 5 levels as [ z 1 , z 2 , z 3 , z 4 , z 5 ]
Revised: RGB image datasets, z is composed of 5 levels as [ z 1 , z 2 , z 3 , z 4 , z 5 ]

Original: =
Revised: =

Original: z , with dimensions of 6144,3072, 1536, 768, and 768, respectively.
Revised: z , with dimensions of 6144, 3072, 1536, 768, and 768, respectively.

Original: B Experimental Setup 
Revised: B Experimental Setup B.1 Datasets

Original: B.1 Datasets

Original: The SVHN dataset [9] consists of 32 × 32 pixel RGB images of real-world house numbers, having10 classes.
Revised: The SVHN dataset [13] consists of 32 × 32 pixel RGB images of real-world house numbers, having 10 classes.

Original: The CIFAR-10 dataset [10] also consists of 32 × 32 pixel RGB natural images in 10different classes.
Revised: The CIFAR-10 dataset [14] also consists of 32 × 32 pixel RGB natural images indifferent classes.

Original: Similarly, CIFAR-100 [10] has 100 classes.
Revised: Similarly, CIFAR-100 [14] has 100 classes.

Original: The TinyImageNet dataset [11]consists of 64 × 64 pixel RGB natural images of 200 classes.
Revised: The TinyImageNet dataset [15] consists of 64 × 64 pixel RGB natural images of 200 classes.

Original: The numbers of training/test imagesare 73 , 257 / 26 , 032 for SVHN, 50 , 000 / 10 , 000 for CIFAR-10 and CIFAR-100, 100 , 000 / 10 , 000 for
Revised: The numbers of training/test images are 73 , 257 / 26 , 032 for SVHN, 50 , 000 / 10 , 000 for CIFAR-10 and CIFAR-100, 100 , 000 / 10 , 000 for

Original: TinyImageNet, respectively.
Revised: TinyImageNet, respectively.

Original: We adopt the standard data-augmentation: random 2 × 2 translation toboth datasets and horizontal ﬂips to CIFAR-10/100 and TinyImageNet.
Revised: We adopt the standard data-augmentation: random 2 × 2 translation to both datasets and horizontal ﬂips to CIFAR-10/100 and TinyImageNet.

Original: The same augmentation isapplied to the training of Glow.
Revised: The same augmentation is applied to the training of Glow.

Original: [Table]
Revised: [Table]

Original: The architecture of CNN-13 is shown in Table
Revised: The architecture of CNN-13 is shown in Table

Original: [Table]
Revised: [Table]

Original: B.3 Hyper-Parameters
Revised: B.3 Hyper-Parameters

Original: Classiﬁers.
Revised: Classiﬁers.

Original: We used the Adam optimizer [22] for the CNN-13 with the momentum parametersβ 1 = 0 .
Revised: We used the Adam optimizer [25] for the CNN-13 with the momentum parameters β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: For WRN-28-10, we use stochastic gradient descent with Nesterovmomentum of 0 .
Revised: For WRN-28-10, we use stochastic gradient descent with Nesterov momentum of 0 .

Original: We trained with 200, 300, and 120 epochs for SVHN, CIFAR-10/100, and
Revised: We trained with 200, 300, and 120 epochs for SVHN, CIFAR-10/100, and

Original: TinyImageNet, respectively.
Revised: TinyImageNet, respectively.

Original: For CNN-13, the learning rate starts with 0.001 and exponentiallydecays with a rate 0.97 at every 2 epochs after the ﬁrst 60,000 and 184,000 updates for SVHN and
Revised: For CNN-13, the learning rate starts with 0.001 and exponentially decays with a rate 0.97 at every 2 epochs after the ﬁrst 60,000 and 184,000 updates for SVHN and

Original: CIFAR-10, respectively.
Revised: CIFAR-10, respectively.

Original: For WRN-28-10, we use a cosine learning decay, which is used in [23],starting with 0.1.
Revised: For WRN-28-10, we use a cosine learning decay, which is used in [26], starting with 0.1.

Original: Thesize of a mini-batch is 128 for CNN-13 and 50 for WRN-28-10.
Revised: The a mini batch

Original: Glow.
Revised: Glow.

Original: We used the Adam optimizer with the momentum parameters β 1 = 0 .
Revised: We used the Adam optimizer with the momentum parameters β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: The learning rate starts with 0 . 0001 , and we trained 8 , 200 iterations for both datasets, with batchsize 256 for SVHN and CIFAR-10/100 and 16 for TinyImageNet, respectively.
Revised: The learning rate starts with 0 . 0001 , and we trained 8 , 200 iterations for both datasets, with batch size 256 for SVHN and CIFAR-10/100 and 16 for TinyImageNet, respectively.

Original: There are two majorparameters to design the architecture: the depth of ﬂow K and the number of factoring out operations
Revised: There are two major parameters to design the architecture: the depth of ﬂow K and the number of factoring out operations

Original: L .
Revised: L .

Original: For SVHN, CIFAR-10 and CIFAR-100, we chose K = 32 and L = 3 .
Revised: For SVHN, CIFAR-10 and CIFAR-100, we chose K = 32 and L = 3 .

Original: For TinyImageNet, wechose K = 48 and L = 4 .
Revised: For TinyImageNet, we chose K = 48 and L = 4 .

Original: The channel width of convolutions is 128 for all datasets.
Revised: The channel width of convolutions is 128 for all datasets.

Original: In accordancewith [7], we train the Glow model on 5 -bit images converted from the original 8-bit for high ﬁdelity.
Revised: In accordance with [11], we train the Glow model on 5 -bit images converted from the original 8-bit for high ﬁdelity.

Original: We also would like to refer to our experimental code for the details.
Revised: We also would like to refer to our experimental code for the details.

Original: C Results of MixUp
Revised: C Results of MixUp

Original: [Table]
Revised: [Table]

Original: We tested MixUp with different α ∈ (0 . 1 , 1 . 0) , and the results are shown in Table 6.
Revised: We tested MixUp with different α ∈ (0 . 1 , 1 . 0) , and the results are shown in Table 6.

Original: For both datasets,the setting α = 0 .
Revised: For both datasets, the setting α = 0 .

Original: 7 was the best,
Revised: 7 was the best,

Original: although α = 1 . 0 achieved the same result on CIFAR-10.
Revised: although α = 1 . 0 achieved the same result on CIFAR-10.

Original: D Alternative Mixture Methods in Latent Space
Revised: D Alternative Mixture Methods in Latent Space

Original: We introduce two alternative mixing schemes in the latent space, Swap-Mix and Reverse-Mix .
Revised: We introduce two alternative mixing schemes in the latent space, Swap-Mix and Reverse-Mix .

Original: Thetwo alternative methods are depicted in Fig.
Revised: The two alternative methods are depicted in Fig.

Original: 6 in correspondence with Fig.
Revised: 6 in correspondence with Fig.

Original: The Swap-Mix createsthe mask M in the same way as LS-Mix, but the components taken from each z 1 and z 2 are combinedin the reverse order from LS-Mix.
Revised: The Swap-Mix creates the mask M in the same way as LS-Mix, but the components taken from each z 1 and z 2 are combined in the reverse order from LS-Mix.

Original: The Reverse-Mix sorts the components of z 1 in reverse orderbefore the selection of components is done.
Revised: The Reverse-Mix sorts the components of z 1 in reverse order before the selection of components is done.

Original: The images mixed by them are shown in Fig.
Revised: The images mixed by them are shown in Fig.

Original: 7, which
Revised: 7, which

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: are drastically deformed while retaining the remnants of their source images.
Revised: are drastically deformed while retaining the remnants of their source images.

Original: The results showedthat Swap-Mix and Reverse-Mix are much less effective than LS-Mix as shown in Table 4.
Revised: The results showed that Swap-Mix and Reverse-Mix are much less effective than LS-Mix as shown in Table 4.

Original: In fact,their performances are often even worse than the baseline model which was trained without any
Revised: In fact, their performances are often even worse than the baseline model which was trained without any

Original: MDA.
Revised: MDA.

Original: The poor performance of Swap-Mix and Reverse-Mix is probably caused by the fact thattheir mixed images deviate too much from the source images, as shown in Fig.
Revised: The poor performance of Swap-Mix and Reverse-Mix is probably caused by the fact that their mixed images deviate too much from the source images, as shown in Fig.

Original: As Swap-Mixdisplaces the position of the source images horizontally and vertically on stitching, this effect isprobably too strong for data augmentation.
Revised: As Swap-Mix displaces the position of the source images horizontally and vertically on stitching, this effect is probably too strong for data augmentation.

Original: On the other hand, Reverse-Mix showed that replacingthe latent components in reverse order immediately results in incomprehensible images to humans,regardless of mixing, as the images with λ = { 216 , 516 , 716 } shown in Fig.
Revised: On the other hand, Reverse-Mix showed that replacing the latent components in reverse order immediately results in incomprehensible images to humans, regardless of mixing, as the images with λ = { 216 , 516 , 716 } shown in Fig.

Original: Moreover, we also foundthat, unlike LS-Mix, these methods often generated entire corrupted images as shown in Appendix
Revised: Moreover, we also found that, unlike LS-Mix, these methods often generated entire corrupted images as shown in Appendix

Original: E. The same corruptions were found in the images with Bern-Mix, and those are probably what isreferred to as inverse explosion in [26], which is caused by the numerical errors that occur mainlydue to a high Lipschitz constant of g and g − 1 [27, 28].
Revised: E. The same corruptions were found in the images with Bern-Mix, and those are probably what is referred to as inverse explosion in [29], which is caused by the numerical errors that occur mainly due to a high Lipschitz constant of g and g − 1 [30, 31].

Original: We conjecture that for images that deviatefrom the training data more than a certain level, the ﬂow model becomes unstable, i.e., Lipschitzconstants become large, which leads to the generation of corrupted images due to inverse explosion.
Revised: We conjecture that for images that deviate from the training data more than a certain level, the ﬂow model becomes unstable, i.e., Lipschitz constants become large, which leads to the generation of corrupted images due to inverse explosion.

Original: E More Samples
Revised: E More Samples

Original: We show more samples as follows.
Revised: We show more samples as follows.

Original: We see that the mixed images with LS-Mix are more natural andstable than those of other methods.
Revised: We see that the mixed images with LS-Mix are more natural and stable than those of other methods.

Original: Columns at both ends are the source images, and the images inmiddle are mixture of them with mixing rate
Revised: Columns at both ends are the source images, and the images in middle are mixture of them with mixing rate

Original: λ = { 216 , 516 , 716 , 916 , 1116 , 1416 } .
Revised: λ = { 216 , 516 , 716 , 916 , 1116 , 1416 } .

Original: L i n ea r
Revised: L i n ea r

Original: I n t r p l
Revised: I n t r p l

Original: B e r nM
Revised: B e r nM

Original: i x
Revised: i x

Original: S w a p - M i x
Revised: S w a p - M i x

Original: R e v e r s eM i x
Revised: R e v e r s eM i x

Original: L S - M i x
Revised: L S - M i x

Original: L i n ea r
Revised: L i n ea r

Original: I n t r p l
Revised: I n t r p l

Original: B e r nM
Revised: B e r nM

Original: i x
Revised: i x

Original: S w a p - M i x
Revised: S w a p - M i x

Original: R e v e r s eM i x
Revised: R e v e r s eM i x

Original: L S - M i x ( ou r s )
Revised: L S - M i x ( ou r s )

Original: L i n ea r
Revised: L i n ea r

Original: I n t r p l
Revised: I n t r p l

Original: B e r nM
Revised: B e r nM

Original: i x
Revised: i x

Original: S w a p - M i x
Revised: S w a p - M i x

Original: R e v e r s eM i x
Revised: R e v e r s eM i x

Original: L S - M i x ( ou r s )
Revised: L S - M i x ( ou r s )

Original: L i n ea r
Revised: L i n ea r

Original: I n t r p l
Revised: I n t r p l

Original: B e r nM
Revised: B e r nM

Original: i x
Revised: i x

Original: S w a p - M i x
Revised: S w a p - M i x

Original: R e v e r s eM i x
Revised: R e v e r s eM i x

Original: L S - M i x ( ou r s )
Revised: L S - M i x ( ou r s )