Original: M ODALS : M ODALITY - AGNOSTIC A UTOMATED D ATA A UGMENTATION IN THE L ATENT S PACE
Revised: M ODALS : M ODALITY - AGNOSTIC A UTOMATED D ATA A UGMENTATION IN THE L ATENT S PACE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Data augmentation is an efﬁcient way to expand a training dataset by creating additional artiﬁcial data.
Revised: Data augmentation is an efﬁcient way to expand a training dataset by creating additional artiﬁcial data.

Original: While data augmentation is found to be effective in improving the generalization capability of models for various machine learning tasks, the underlying augmentation methods are usually manually designed and carefully evaluated for each data modality separately, like image processing functions for image data and word-replacing rules for text data.
Revised: While data augmentation is found to be effective in improving the generalization capability of models for various machine learning tasks, the underlying augmentation methods are usually manually designed and carefully evaluated for each data modality separately, like image processing functions for image data and word-replacing rules for text data.

Original: In this work, we propose an automated data augmentation approach called MODALS (Modalityagnostic Automated Data Augmentation in the Latent Space) to augment data for any modality in a generic way.
Revised: In this work, we propose an automated data augmentation approach called MODALS (Modalityagnostic Automated Data Augmentation in the Latent Space) to augment data for any modality in a generic way.

Original: MODALS exploits automated data augmentation to ﬁne-tune four universal data transformation operations in the latent space to adapt the transform to data of different modalities.
Revised: MODALS exploits automated data augmentation to ﬁne-tune four universal data transformation operations in the latent space to adapt the transform to data of different modalities.

Original: Through comprehensive experiments, we demonstrate the effectiveness of MODALS on multiple datasets for text, tabular, time-series and image modalities.
Revised: Through comprehensive experiments, we demonstrate the effectiveness of MODALS on multiple datasets for text, tabular, time-series and image modalities.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning models tend to perform better with more labeled training data.
Revised: Deep learning models tend to perform better with more labeled training data.

Original: However, labeled data are usually scarce and expensive to collect.
Revised: However, labeled data are usually scarce and expensive to collect.

Original: Data augmentation is a promising means to extend the training dataset with new artiﬁcial data.
Revised: Data augmentation is a promising means to extend the training dataset with new artiﬁcial data.

Original: In image recognition, image processing functions, like randomized cropping, horizontal ﬂipping, and color shifting, are commonly adopted in modern image recognition models (Krizhevsky et al., 2012;
Revised: In image recognition, image processing functions, like randomized cropping, horizontal ﬂipping, and color shifting, are commonly adopted in modern image recognition models (Krizhevsky et al., 2012;

Original: Shorten & Khoshgoftaar, 2019).
Revised: Shorten & Khoshgoftaar, 2019).

Original: Following the success of image augmentation, it is increasingly common to apply data augmentation in natural language processing tasks, like machine translation, text classiﬁcation, and semantic parsing.
Revised: Following the success of image augmentation, it is increasingly common to apply data augmentation in natural language processing tasks, like machine translation, text classiﬁcation, and semantic parsing.

Original: Various wordbased transformations have been proposed to perturb word tokens, such as replacing similar words or phrases, swapping word orders, and inserting or dropping random words (Cheng et al., 2018;
Revised: Various wordbased transformations have been proposed to perturb word tokens, such as replacing similar words or phrases, swapping word orders, and inserting or dropping random words (Cheng et al., 2018;

Original: S¸ahin & Steedman, 2018; Wei & Zou, 2019).
Revised: S¸ahin & Steedman, 2018; Wei & Zou, 2019).

Original: Over the years, more transformation functions have been proposed to augment different datasets.
Revised: Over the years, more transformation functions have been proposed to augment different datasets.

Original: Cutout randomly occludes a part of an image to avoid overﬁtting (Devries & Taylor, 2017b).
Revised: Cutout randomly occludes a part of an image to avoid overﬁtting (Devries & Taylor, 2017b).

Original: For label-mixing methods, CutMix replaces the occluded part in Cutout by a different image (Yun et al., 2019) and Mixup interpolates two images with their corresponding one-hot encoded labels (Zhang et al., 2018).
Revised: For label-mixing methods, CutMix replaces the occluded part in Cutout by a different image (Yun et al., 2019) and Mixup interpolates two images with their corresponding one-hot encoded labels (Zhang et al., 2018).

Original: These methods have been tested and found to be effective in multiple image datasets.
Revised: These methods have been tested and found to be effective in multiple image datasets.

Original: Alternatively, new data can be created using deep generative models, for example, using GAN-based approaches to generate new images (Antoniou et al., 2017; Sandfort et al., 2019), conditional pretrained language models to generate training sentences (Kumar et al., 2020), and back-translation to paraphrase sentences by translating sentences to another language and back to the original language (Xie et al., 2019).
Revised: Alternatively, new data can be created using deep generative models, for example, using GAN-based approaches to generate new images (Antoniou et al., 2017; Sandfort et al., 2019), conditional pretrained language models to generate training sentences (Kumar et al., 2020), and back-translation to paraphrase sentences by translating sentences to another language and back to the original language (Xie et al., 2019).

Original: While these generative approaches are found to be useful, the generators or language models are often hard to implement and expensive to train.
Revised: While these generative approaches are found to be useful, the generators or language models are often hard to implement and expensive to train.

Original: Apart from advancing individual transformations, another line of research studies their optimal composition.
Revised: Apart from advancing individual transformations, another line of research studies their optimal composition.

Original: As the choice and order of the transformations are decided and tested manually, the success of an augmentation scheme in one dataset may not generalize well to other datasets.
Revised: As the choice and order of the transformations are decided and tested manually, the success of an augmentation scheme in one dataset may not generalize well to other datasets.

Original: To tackle this problem, AutoAugment as an automated data augmentation method was proposed to automate this process by learning an optimal augmentation policy, which decides the probability and magnitude to apply pre-deﬁned transformations (Cubuk et al., 2019).
Revised: To tackle this problem, AutoAugment as an automated data augmentation method was proposed to automate this process by learning an optimal augmentation policy, which decides the probability and magnitude to apply pre-deﬁned transformations (Cubuk et al., 2019).

Original: Whether it is for standard augmentation or automated augmentation, the transformation functions are often designed and tested carefully for each data modality separately.
Revised: Whether it is for standard augmentation or automated augmentation, the transformation functions are often designed and tested carefully for each data modality separately.

Original: While it may be intuitive to create new and valid images using image processing techniques, it is non-trivial to deﬁne such labelpreserving transformations on discrete data like text data.
Revised: While it may be intuitive to create new and valid images using image processing techniques, it is non-trivial to deﬁne such labelpreserving transformations on discrete data like text data.

Original: This prohibits the reuse of augmentation schemes across different data modalities.
Revised: This prohibits the reuse of augmentation schemes across different data modalities.

Original: To overcome this shortcoming, we propose MODALS to apply modality-agnostic automated data augmentation in the latent space.
Revised: To overcome this shortcoming, we propose MODALS to apply modality-agnostic automated data augmentation in the latent space.

Original: The idea of transforming latent features is inspired by representation learning.
Revised: The idea of transforming latent features is inspired by representation learning.

Original: For image generation, the work of Upchurch et al.
Revised: For image generation, the work of Upchurch et al.

Original: (2017) interpolates images along speciﬁc directions in the latent space to add new semantics without changing the class identity, such as adding facial hair to the image of a male face by translating the corresponding latent representation towards the direction of male faces with facial hair.
Revised: (2017) interpolates images along speciﬁc directions in the latent space to add new semantics without changing the class identity, such as adding facial hair to the image of a male face by translating the corresponding latent representation towards the direction of male faces with facial hair.

Original: This suggests that augmenting data in the latent space can capture diverse semantic transformations which are usually hard to deﬁne in the input space.
Revised: This suggests that augmenting data in the latent space can capture diverse semantic transformations which are usually hard to deﬁne in the input space.

Original: Augmentation in the latent space poses two main challenges: learning a latent space that is continuous for transformation, and ﬁnding the effective directions to traverse.
Revised: Augmentation in the latent space poses two main challenges: learning a latent space that is continuous for transformation, and ﬁnding the effective directions to traverse.

Original: Failing to address them properly may cause an augmented example to lose its original class identity.
Revised: Failing to address them properly may cause an augmented example to lose its original class identity.

Original: In the previous work by
Revised: In the previous work by

Original: Devries & Taylor (2017a), the latent space is learned by training an autoencoder, which encodes the input data into a latent vector and decodes it back to the original data.
Revised: Devries & Taylor (2017a), the latent space is learned by training an autoencoder, which encodes the input data into a latent vector and decodes it back to the original data.

Original: The learned latent representations are then transformed by interpolation, extrapolation, or adding Gaussian noise and decoded as synthetic examples for the downstream tasks.
Revised: The learned latent representations are then transformed by interpolation, extrapolation, or adding Gaussian noise and decoded as synthetic examples for the downstream tasks.

Original: For image data, ISDA estimates the semantic directions by inspecting the feature covariance (Wang et al., 2019).
Revised: For image data, ISDA estimates the semantic directions by inspecting the feature covariance (Wang et al., 2019).

Original: LSI and manifold mixup apply mixup to the latent feature vectors (Liu et al., 2018; Verma et al., 2019).
Revised: LSI and manifold mixup apply mixup to the latent feature vectors (Liu et al., 2018; Verma et al., 2019).

Original: We develop our framework as MODALS .
Revised: We develop our framework as MODALS .

Original: At a high level, MODALS applies latent space augmentation to address the data augmentation problem.
Revised: At a high level, MODALS applies latent space augmentation to address the data augmentation problem.

Original: It differs from the previous approaches in four ways.
Revised: It differs from the previous approaches in four ways.

Original: First,  MODALS is jointly trained with augmentation.
Revised: First, as opposed to other operation-based latent space transformation methods, MODALS is jointly trained with augmentation.

Original: As it involves no auxiliary models or additional processes to generate examples, it can be efﬁciently integrated into the popular deep learning frameworks.
Revised: As it involves no auxiliary models or additional processes to generate examples, it can be efﬁciently integrated into the popular deep learning frameworks.

Original: Second, unlike most of the automated data augmentation methods which are applicable to a speciﬁc modality such as image or text data, MODALS is not restricted to any speciﬁc modality.
Revised: Second, unlike most of the automated data augmentation methods which are applicable to a speciﬁc modality such as image or text data, MODALS is not restricted to any speciﬁc modality.

Original: Third, we observe that examples that are more uncertain to predict, or considered to be hard in the active learning literature, tend to carry richer information for model training.
Revised: Third, we observe that examples that are more uncertain to predict, or considered to be hard in the active learning literature, tend to carry richer information for model training.

Original: Therefore, we modify standard latent space transformations to create harder examples in MODALS .
Revised: Therefore, we modify standard latent space transformations to create harder examples in MODALS .

Original: Fourth, MODALS introduces additional loss terms to improve the quality of augmentation in the latent space.
Revised: Fourth, MODALS introduces additional loss terms to improve the quality of augmentation in the latent space.

Original: In summary, we make four major contributions in this paper:
Revised: In summary, we make four major contributions in this paper:

Original: • Propose a framework to apply automated augmentation in the latent space.
Revised: • Propose a framework to apply automated augmentation in the latent space.

Original: • Propose a novel and effective way to create hard examples.
Revised: • Propose a novel and effective way to create hard examples.

Original: • Study additional loss terms to improve label-preserving transformation in the latent space.
Revised: • Study additional loss terms to improve label-preserving transformation in the latent space.

Original: • Evaluate MODALS extensively on classiﬁcation datasets across multiple data modalities using various deep learning models.
Revised: • Evaluate MODALS extensively on classiﬁcation datasets across multiple data modalities using various deep learning models.

Original:  R ELATED W ORK 2. A UTOMATED D ATA A UGMENTATION
Revised:  R ELATED W ORK 2. A UTOMATED D ATA A UGMENTATION

Original: In practice, multiple transformations are used compositely to augment a dataset.
Revised: In practice, multiple transformations are used compositely to augment a dataset.

Original: The choice and strength of the transformations affect the model performance on different datasets.
Revised: The choice and strength of the transformations affect the model performance on different datasets.

Original: Motivated by neural architecture search and reinforcement learning, automated data augmentation formulates the tuning of augmentation parameters as a learning task.
Revised: Motivated by neural architecture search and reinforcement learning, automated data augmentation formulates the tuning of augmentation parameters as a learning task.

Original: AutoAugment trains a child model with the augmentation transformations generated by a controller, which is updated by reinforcement learning using the downstream validation accuracy as the reward signals (Cubuk et al., 2019).
Revised: AutoAugment trains a child model with the augmentation transformations generated by a controller, which is updated by reinforcement learning using the downstream validation accuracy as the reward signals (Cubuk et al., 2019).

Original: To reduce the search effort in AutoAugment, various approaches have been proposed.
Revised: To reduce the search effort in AutoAugment, various approaches have been proposed.

Original: Population Based Augmentation (PBA) adopts an evolution-based search strategy to simultaneously train and perturb a policy with multiple child models (Ho et al., 2019).
Revised: Population Based Augmentation (PBA) adopts an evolution-based search strategy to simultaneously train and perturb a policy with multiple child models (Ho et al., 2019).

Original: RandAugment replaces the augmentation parameters in AutoAugment by uniform values across all transformations and searches for the magnitude and the number of transformations to apply (Cubuk et al., 2020).
Revised: RandAugment replaces the augmentation parameters in AutoAugment by uniform values across all transformations and searches for the magnitude and the number of transformations to apply (Cubuk et al., 2020).

Original: Adversarial AutoAugment ﬁnds an adversarial policy that creates difﬁcult examples (Zhang et al., 2020).
Revised: Adversarial AutoAugment ﬁnds an adversarial policy that creates difﬁcult examples (Zhang et al., 2020).

Original: Luo et al.
Revised: Luo et al.

Original: (2020) consider a text image recognition task and train a model to distort the text images to create more difﬁcult examples.
Revised: (2020) consider a text image recognition task and train a model to distort the text images to create more difﬁcult examples.

Original: In our work, we adopt PBA as the search strategy due to its more efﬁcient computation than AutoAugment and a larger parameter space than RandAugment.
Revised: In our work, we adopt PBA as the search strategy due to its more efﬁcient computation than AutoAugment and a larger parameter space than RandAugment.

Original: 2. L ATENT S PACE D ATA A UGMENTATION
Revised: 2. L ATENT S PACE D ATA A UGMENTATION

Original: Interpolation is a simple but effective way to create new data from existing examples.
Revised: Interpolation is a simple but effective way to create new data from existing examples.

Original: Mixup can operate on general numerical data by interpolating two training examples and their corresponding one-hot encoded labels (Zhang et al., 2018).
Revised: Mixup can operate on general numerical data by interpolating two training examples and their corresponding one-hot encoded labels (Zhang et al., 2018).

Original: Manifold Mixup applies mixup to the outputs from different hidden layers (Verma et al., 2019); LSI applies mixup in the latent space for image classiﬁcation (Liu et al., 2018).
Revised: Manifold Mixup applies mixup to the outputs from different hidden layers (Verma et al., 2019); LSI applies mixup in the latent space for image classiﬁcation (Liu et al., 2018).

Original: Previously, Devries & Taylor (2017a) found that extrapolation between features learned by an autoencoder generates more effective synthetic examples than adding Gaussian noise and interpolation.
Revised: Previously, Devries & Taylor (2017a) found that extrapolation between features learned by an autoencoder generates more effective synthetic examples than adding Gaussian noise and interpolation.

Original: In their experiments, they suspected that interpolation tightens the class boundaries, which leads to overﬁtting and harms the performance.
Revised: In their experiments, they suspected that interpolation tightens the class boundaries, which leads to overﬁtting and harms the performance.

Revised: For text data, Kumar et al.

Revised: (2019) explored simple latent space transformations and autoencoder-based methods in few-shot intent classiﬁcation tasks.

Original: In general, it remains an open question what transformation in the latent space creates better augmented examples.
Revised: In general, it remains an open question what transformation in the latent space creates better augmented examples.

Original: We leverage automated data augmentation to ﬁnd the optimal composition of latent space transformations for a variety of datasets of multiple data modalities.
Revised: We leverage automated data augmentation to ﬁnd the optimal composition of latent space transformations for a variety of datasets of multiple data modalities.

Original: 2. H ARD E XAMPLE A UGMENTATION
Revised: 2. H ARD E XAMPLE A UGMENTATION

Original: Kuchnik & Smith (2019) show that not all training examples are equally useful for augmentation and propose methods to ﬁnd a subset of useful data to augment.
Revised: Kuchnik & Smith (2019) show that not all training examples are equally useful for augmentation and propose methods to ﬁnd a subset of useful data to augment.

Original: Similar methods are also proposed by the recent augmentation algorithms, which aim to ﬁnd the composition of transformations that create difﬁcult examples (Zhang et al., 2020; Wu et al., 2020; Luo et al., 2020).
Revised: Similar methods are also proposed by the recent augmentation algorithms, which aim to ﬁnd the composition of transformations that create difﬁcult examples (Zhang et al., 2020; Wu et al., 2020; Luo et al., 2020).

Original: In our work, we propose a novel and simple way to create hard examples using latent space transformation.
Revised: In our work, we propose a novel and simple way to create hard examples using latent space transformation.

Original:  M ETHODOLOGY 3. B RIEF O VERVIEW OF P OPULATION B ASED A UGMENTATION
Revised:  M ETHODOLOGY 3. B RIEF O VERVIEW OF P OPULATION B ASED A UGMENTATION

Original: We ﬁrst brieﬂy review the policy search procedure of PBA (Ho et al., 2019).
Revised: We ﬁrst brieﬂy review the policy search procedure of PBA (Ho et al., 2019).

Original: In PBA, each augmentation function is associated with its probability and magnitude, for example, the tuple ( op : rotation, p = 0 .
Revised: In PBA, each augmentation function is associated with its probability and magnitude, for example, the tuple ( op : rotation, p = 0 .

Original: 4 , λ = 0 .
Revised: 4 , λ = 0 .

Original: 5 ) speciﬁes that the rotation operation is applied with probability 0.4 and magnitude 0.5.
Revised: 5 ) speciﬁes that the rotation operation is applied with probability 0.4 and magnitude 0.5.

Original: A policy is a set of such operation tuples.
Revised: A policy is a set of such operation tuples.

Original: When a policy is applied to a mini-batch, 0, 1 oroperation tuples are randomly sampled from the policy and applied to the batch of data with their corresponding probability and magnitude.
Revised: When a policy is applied to a mini-batch, 0, 1 oroperation tuples are randomly sampled from the policy and applied to the batch of data with their corresponding probability and magnitude.

Original: PBA formulates the augmentation policy search as hyperparameter schedule learning.
Revised: PBA formulates the augmentation policy search as hyperparameter schedule learning.

Original: Instead of a ﬁxed policy, it learns a policy schedule that speciﬁes the augmentation parameters in different stages of training.
Revised: Instead of a ﬁxed policy, it learns a policy schedule that speciﬁes the augmentation parameters in different stages of training.

Original: In particular, PBA exploits population-based training (PBT) to optimize the model weights of multiple child models jointly with their policy parameters that maximize the performance.
Revised: In particular, PBA exploits population-based training (PBT) to optimize the model weights of multiple child models jointly with their policy parameters that maximize the performance.

Original: It starts by simultaneously training a ﬁxed number of randomly-initialized child models, each with its own policy.
Revised: It starts by simultaneously training a ﬁxed number of randomly-initialized child models, each with its own policy.

Original: After certain ﬁxed intervals, the child models are evaluated using the validation set.
Revised: After certain ﬁxed intervals, the child models are evaluated using the validation set.

Original: The worst models clone the weights and policies from the best models.
Revised: The worst models clone the weights and policies from the best models.

Original: The parameters of the cloned policies are either resampled from all possible values or perturbed from the cloned values.
Revised: The parameters of the cloned policies are either resampled from all possible values or perturbed from the cloned values.

Original: The process repeats until the training of the child models completes.
Revised: The process repeats until the training of the child models completes.

Original: The ﬁnal output of PBA is a schedule of policies unrolled from the best child model.
Revised: The ﬁnal output of PBA is a schedule of policies unrolled from the best child model.

Original: The policy schedule can be applied to augment the dataset when training with more data or larger models.
Revised: The policy schedule can be applied to augment the dataset when training with more data or larger models.

Original: As there is no need to retrain any child model, PBA is considered to be an efﬁcient automated augmentation algorithm.
Revised: As there is no need to retrain any child model, PBA is considered to be an efﬁcient automated augmentation algorithm.

Original: Our method follows the policy formulation and the search strategy used in PBA.
Revised: Our method follows the policy formulation and the search strategy used in PBA.

Original: 3. MODALS
Revised: 3. MODALS

Original: Our main assumption is that, if learned properly, the region corresponding to a class in the latent space learned by a deep neural network is mostly convex and isotropic.
Revised: Our main assumption is that, if learned properly, the region corresponding to a class in the latent space learned by a deep neural network is mostly convex and isotropic.

Original: Consequently, linear transformation in such a space can produce smooth transition from real data to artiﬁcial data without altering their class identity.
Revised: Consequently, linear transformation in such a space can produce smooth transition from real data to artiﬁcial data without altering their class identity.

Original: Among other non-trivial properties, data points are often sparsely distributed in high-dimensional spaces such that distance-based methods that work well in lower-dimensional spaces fail to take advantage of the distance metrics (Donoho, 2000).
Revised: Among other non-trivial properties, data points are often sparsely distributed in high-dimensional spaces such that distance-based methods that work well in lower-dimensional spaces fail to take advantage of the distance metrics (Donoho, 2000).

Original: In the supplemental materials, we provide empirical evidence to show that the relatively low-dimensional latent spaces used in our experiments are not anisotropic, especially within the local class regions (see Appendix A.1).
Revised: In the supplemental materials, we provide empirical evidence to show that the relatively low-dimensional latent spaces used in our experiments are not anisotropic, especially within the local class regions (see Appendix A.1).

Original: In what follows, we introduce the latent space transformations and the training 
Revised: In what follows, we introduce the latent space transformations and the training objectives of our model.

Original: 3.2. T RANSFORMATIONS
Revised: 3.2. T RANSFORMATIONS

Original: Hard example interpolation.
Revised: Hard example interpolation.

Original: Given a seed latent representation z ci , which is the latent representation of the i -th example in class c , we interpolate z ci to a hard example.
Revised: Given a seed latent representation z ci , which is the latent representation of the i -th example in class c , we interpolate z ci to a hard example.

Original: The hard example is taken as the latent representation nearest to z ci among q hard example candidates sampled according to the classiﬁcation loss.
Revised: The hard example is taken as the latent representation nearest to z ci among q hard example candidates sampled according to the classiﬁcation loss.

Original: This favours the creation of harder examples and prevents z ci from being aggressively interpolated to distant locations or regions that are close to the class center.
Revised: This favours the creation of harder examples and prevents z ci from being aggressively interpolated to distant locations or regions that are close to the class center.

Original: In our implementation, we take q as 5% of the number of examples in class c .
Revised: In our implementation, we take q as 5% of the number of examples in class c .

Original: Formally, let S = { s i } qi =
Revised: Formally, let S = { s i } qi =

Original: denote the set of hard latent representations sampled according to the magnitude of the loss in class c , s = arg min s [Equation] S 
Revised: denote the set of hard latent representations sampled according to the magnitude of the loss in class c , s = arg min s ∈ S s · z ci

Original: [Equation]
Revised: [Equation]

Original:  denote the closest hard example (red circle in Figure 1a), and λ 1 denote a scaling factor.
Revised: " denote the closest hard example (red circle in Figure 1a), and λ 1 denote a scaling factor.

Original: Hard example interpolation is expressed as:
Revised: Hard example interpolation is expressed as:

Original: [Equation]
Revised: [Equation]

Original: Hard example extrapolation.
Revised: Hard example extrapolation.

Original: The data nearer the class boundaries are more difﬁcult to be classiﬁed.
Revised: The data nearer the class boundaries are more difﬁcult to be classiﬁed.

Original: Instead of extrapolating from random examples, we extrapolate z ci from the class center µ c = [Equation] m !
Revised: Instead of extrapolating from random examples, we extrapolate z ci from the class center µ c = 1 m !

Original: mj
Revised: mj

Original: =

Revised: =

Original: z cj with λ 2 as the scaling factor.
Revised: z cj with λ 2 as the scaling factor.

Original: Hard example extrapolation is thus given by:
Revised: Hard example extrapolation is thus given by:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Gaussian Noise.
Revised: Gaussian Noise.

Original: We also perturb z ci by adding Gaussian noise with zero mean and per-element standard deviation computed across the examples in the same class.
Revised: We also perturb z ci by adding Gaussian noise with zero mean and per-element standard deviation computed across the examples in the same class.

Original: Let ! ∼ N ( 0 , σ 2 c I ) and λ 3 be the scaling factor, we have:
Revised: Let ! ∼ N ( 0 , σ 2 c I ) and λ 3 be the scaling factor, we have:

Original: [Equation]
Revised: [Equation]

Original: Difference Transform.
Revised: Difference Transform.

Original: Difference transform is an alternative way to perturb z ci by translating z ci along the direction between two random latent vectors sampled from the same class.
Revised: Difference transform is an alternative way to perturb z ci by translating z ci along the direction between two random latent vectors sampled from the same class.

Original: Denoting the scaling factor as λ 4 , the difference transform can be written as:
Revised: Denoting the scaling factor as λ 4 , the difference transform can be written as:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure] These four latent space transformations are illustrated in Figure 1.[Figure] [Figure]
Revised: These four latent space transformations are illustrated in Figure 1.

Original: [Figure]
Revised: [Figure]

Original:  M ODEL
Revised: 3.2. M ODEL

Original: Instead of training an autoencoder to learn the latent space and generate additional synthetic data for training, we train a classiﬁcation model jointly with different compositions of latent space augmentations.
Revised: Instead of training an autoencoder to learn the latent space and generate additional synthetic data for training, we train a classiﬁcation model jointly with different compositions of latent space augmentations.

Original: Given an input space X , a latent space Z , and a label space Y , MODALS trains a feature extraction model F ( x ; θ ) :
Revised: Given an input space X , a latent space Z , and a label space Y , MODALS trains a feature extraction model F ( x ; θ ) :

Original: X → Z and a dense layer to map Z → Y .
Revised: X → Z and a dense layer to map Z → Y .

Original: The i -th example x i is mapped to its latent representation
Revised: The i -th example x i is mapped to its latent representation

Original: z i =
Revised: z i =

Original: F ( x i ) and further augmented by a composition of label-preserving transform functions to obtain the augmented latent representation ˆ z
Revised: F ( x i ) and further augmented by a composition of label-preserving transform functions to obtain the augmented latent representation ˆ z

Original: i ∈ Z .
Revised: i ∈ Z .

Original: We optimize the softmax cross-entropy loss among k classes with W =
Revised: We optimize the softmax cross-entropy loss among k classes with W =

Original: [ w 1 , ..., w k ] as the weights of the ﬁnal dense layer and M as the batch size.
Revised: [ w 1 , ..., w k ] as the weights of the ﬁnal dense layer and M as the batch size.

Original: The classiﬁcation loss is given by:
Revised: The classiﬁcation loss is given by:

Original: [Equation]
Revised: [Equation]

Original: In addition to the classiﬁcation loss, we further employ two additional loss terms to improve the augmentation results.
Revised: In addition to the classiﬁcation loss, we further employ two additional loss terms to improve the augmentation results.

Original: The effect of these two losses will be discussed in Appendix A.2.
Revised: The effect of these two losses will be discussed in Appendix A.2.

Original: Adversarial Loss.
Revised: Adversarial Loss.

Original: Sampling in a high-dimensional latent space may fall into invalid regions that are not on the data manifold.
Revised: Sampling in a high-dimensional latent space may fall into invalid regions that are not on the data manifold.

Original: In order to produce smooth morphing between sampled latent codes, there are well-studied techniques to enforce a posterior distribution over the latent variables using
Revised: In order to produce smooth morphing between sampled latent codes, there are well-studied techniques to enforce a posterior distribution over the latent variables using

Original: VAE or GAN-based methods (Kingma & Welling, 2014; Goodfellow et al., 2014).
Revised: VAE or GAN-based methods (Kingma & Welling, 2014; Goodfellow et al., 2014).

Original: In MODALS , we regularize the latent space by imposing an adversarial loss similar to Adversarial Autoencoder (Makhzani et al., 2016).
Revised: In MODALS , we regularize the latent space by imposing an adversarial loss similar to Adversarial Autoencoder (Makhzani et al., 2016).

Original: In particular, we employ a discriminator D ( z ; φ ) to distinguish the latent code generated by the feature extraction model and sampled from the Gaussian distribution.
Revised: In particular, we employ a discriminator D ( z ; φ ) to distinguish the latent code generated by the feature extraction model and sampled from the Gaussian distribution.

Original: The feature extraction model has to generate latent representations that are similar to the Gaussian distribution to fool the discriminator.
Revised: The feature extraction model has to generate latent representations that are similar to the Gaussian distribution to fool the discriminator.

Original: This leads to the adversarial objective L adv ( θ ) for the feature extraction model and the discriminator loss L D ( φ ) for the discriminator:
Revised: This leads to the adversarial objective L adv ( θ ) for the feature extraction model and the discriminator loss L D ( φ ) for the discriminator:

Original: [Equation]
Revised: [Equation]

Original: Triplet Loss.
Revised: Triplet Loss.

Original: Triplet loss is often used in metric learning to learn the latent representations of different classes (Schroff et al., 2015).
Revised: Triplet loss is often used in metric learning to learn the latent representations of different classes (Schroff et al., 2015).

Original: Triplet loss pulls together the representations from the same class and repels the ones from different classes.
Revised: Triplet loss pulls together the representations from the same class and repels the ones from different classes.

Original: We argue that this characteristic facilitates labelpreserving augmentation in the latent space.
Revised: We argue that this characteristic facilitates labelpreserving augmentation in the latent space.

Original: Intuitively, the data outside the convex hull of the training samples are less likely to correspond to valid data of the same class.
Revised: Intuitively, the data outside the convex hull of the training samples are less likely to correspond to valid data of the same class.

Original: As a result of using the triplet loss, the distribution of the latent representations in the same class will be more compact.
Revised: As a result of using the triplet loss, the distribution of the latent representations in the same class will be more compact.

Original: Interpolating, extrapolating or perturbing latent representations in this denser region is more likely to result in an augmented example sharing the same class identity.
Revised: Interpolating, extrapolating or perturbing latent representations in this denser region is more likely to result in an augmented example sharing the same class identity.

Revised: In theory, alternative measures, e.g., center loss (Wen et al., 2016), large margin softmax loss (Liu et al., 2016), or other contrastive losses, can also produce similar effects.

Original: In our implementation, for each anchor latent representation z , we randomly sample a positive latent representation z + from the same class and a negative representation z − from a different class in the same mini-batch.
Revised: For simplicity, we use the triplet loss in our implementation: for each anchor latent representation z , we randomly sample a positive latent representation z + from the same class and a negative representation z − from a different class in the same mini-batch.

Original: We denote the margin as γ and the cosine distance function as d ( · ) .
Revised: We denote the margin as γ and the cosine distance function as d ( · ) .

Original: The triplet loss L tri ( θ ) is given by:
Revised: The triplet loss L tri ( θ ) is given by:

Original: [Equation]
Revised: [Equation]

Original: Feature

Original: Extractor !

Original: [Equation]

Original: Dense

Original: Layer Policy [Figure] MODALS is trained by minimizing the combined loss L = L clf + α L adv + β L tri , where α and β are predeﬁned hyperparameters.
Revised: MODALS is trained by minimizing the combined loss L = L clf + α L adv + β L tri , where α and β are predeﬁned hyperparameters.

Original: Figure 2 shows the overall architecture of MODALS :
Revised: Figure 2 shows the overall architecture of MODALS :

Original: For each mini-batch, a feature extraction model maps the input data to their latent representations.
Revised: For each mini-batch, a feature extraction model maps the input data to their latent representations.

Original: The representations are used to compute the adversarial and triplet losses, and also augmented to obtain the classiﬁcation loss and validation accuracy.
Revised: The representations are used to compute the adversarial and triplet losses, and also augmented to obtain the classiﬁcation loss and validation accuracy.

Original: The combined loss L is back-propagated to the feature extraction network while the validation accuracy is treated as the reward signal to update the policy.
Revised: The combined loss L is back-propagated to the feature extraction network while the validation accuracy is treated as the reward signal to update the policy.

Original: 3.2. P OLICY S EARCH
Revised: 3.2. P OLICY S EARCH

Original: We apply the PBA policy search strategy to the latent representations with the four proposed transformations.
Revised: We apply the PBA policy search strategy to the latent representations with the four proposed transformations.

Original: We deﬁne four latent space transformations with probability ranging from 0 to 1 and magnitude λ ranging from 0 to 0.9, both in intervals of 0.1.
Revised: We deﬁne four latent space transformations with probability ranging from 0 to 1 and magnitude λ ranging from 0 to 0.9, both in intervals of 0.1.

Original: As the same operation can be applied twice, the search space is of size (10 × 11) 8 ≈ 2 .
Revised: As the same operation can be applied twice, the search space is of size (10 × 11) 8 ≈ 2 .

Original: 14 × 10 16 .
Revised: 14 × 10 16 .

Original: The total computation cost is the cost to train a single model multiplied by the number of parallel models. 
Revised: The total computation cost is the cost to train a single model multiplied by the number of parallel models. Feature Extractor ! ! ̂

Revised: Dense

Revised: Layer

Original: E XPERIMENTS
Revised: Policy Classification Loss Validation Accuracy Transform Discriminator

Revised: Adversarial Loss

Revised: Discriminator Loss

Revised: Triplet

Revised: Selector

Original:  To demonstrate the modality-agnostic property of MODALS , we evaluate it using datasets from four different data modalities: text, tabular, time-series, and image data.
Revised: Triplet Loss [Figure]  E XPERIMENTS To demonstrate the modality-agnostic property of MODALS , we evaluate it using datasets from four different data modalities: text, tabular, time-series, and image data.

Original: For each dataset, we search for the optimal augmentation schedule using PBT with 16 parallel models perturbing every three epochs.
Revised: For each dataset, we search for the optimal augmentation schedule using PBT with 16 parallel models perturbing every three epochs.

Original: The searched policy schedule is applied to the dataset with different number of training examples.
Revised: The searched policy schedule is applied to the dataset with different number of training examples.

Revised: We implement the discriminator as a 2-layer multilayer perceptron (MLP) model with 256 hidden units in each layer.

Original: In all experiments, we set α = 1 , β = 0 .
Revised: In all experiments, we set α = 1 , β = 0 .

Original: 03 and search for the metric margin value from { 0 . 5 , 1 , 2 , 4 , 8 } .
Revised: 03 and search for the metric margin value from { 0 . 5 , 1 , 2 , 4 , 8 } .

Original: We compare our method with baseline methods that do not involve training an auxiliary model.
Revised: We compare our method with baseline methods that do not involve training an auxiliary model.

Original: In particular, we construct the baseline models trained with well-deﬁned inputspace augmentation in the data modality and label-mixing augmentation.
Revised: In particular, we construct the baseline models trained with well-deﬁned input-space augmentation in the data modality and label-mixing augmentation.

Original: We report the average classiﬁcation accuracy from three trials with the baselines.
Revised: We report the average classiﬁcation accuracy from three trials with the baselines.

Original: The detailed conﬁguration and policy of each experiment are provided in Appendix A.3.
Revised: The detailed conﬁguration and policy of each experiment are provided in Appendix A.3.

Original: 4. T EXT D ATA
Revised: 4. T EXT D ATA

Original: We test MODALS on the SST2 (Socher et al., 2013) and TREC6 (Li & Roth, 2002) datasets, which are popular benchmarks for binary and multi-class text classiﬁcation tasks.
Revised: We test MODALS on the SST2 (Socher et al., 2013) and TREC6 (Li & Roth, 2002) datasets, which are popular benchmarks for binary and multi-class text classiﬁcation tasks.

Original: SST2 is a binary sentiment classiﬁcation dataset with 6,920 training examples, while TREC6 is a question classiﬁcation dataset with 5,452 training examples from six question classes.
Revised: SST2 is a binary sentiment classiﬁcation dataset with 6,920 training examples, while TREC6 is a question classiﬁcation dataset with 5,452 training examples from six question classes.

Original: We employ a 2-layer bidirectional
Revised: We employ a 2-layer bidirectional

Original: LSTM using ﬁxed pre-trained 300-dimensional GloVe as the word embedding (Pennington et al., 2014).
Revised: LSTM using ﬁxed pre-trained 300-dimensional GloVe as the word embedding (Pennington et al., 2014).

Original: The hidden state dimensionality of the model is 256.
Revised: The hidden state dimensionality of the model is 256.

Original: The ﬁrst baseline is trained without augmentation.
Revised: The ﬁrst baseline is trained without augmentation.

Original: We construct the second baseline by applying manually designed input-space augmentations proposed as easy data augmentation (EDA) techniques in (Wei & Zou, 2019).
Revised: We construct the second baseline by applying manually designed input-space augmentations proposed as easy data augmentation (EDA) techniques in (Wei & Zou, 2019).

Original: In this baseline, seed sentences are cloned multiple times, where each cloned sentence is transformed by replacing synonyms, inserting synonyms, deleting random word, or swapping word order.
Revised: In this baseline, seed sentences are cloned multiple times, where each cloned sentence is transformed by replacing synonyms, inserting synonyms, deleting random word, or swapping word order.

Original: The baseline follows the conﬁgurations suggested by EDA for datasets of different sizes.
Revised: The baseline follows the conﬁgurations suggested by EDA for datasets of different sizes.

Original: We construct the third baseline using senMixup, which applies mixup augmentation in the last hidden layer before the softmax layer (Guo et al., 2019).
Revised: We construct the third baseline using senMixup, which applies mixup augmentation in the last hidden layer before the softmax layer (Guo et al., 2019).

Revised: We also include the fourth and ﬁfth baselines which use back-translation augmentation.

Original:  Our experiments show that MODALS outperforms  other baseline methods.
Revised: The back-translation baselines are implemented using the Google Translate API to translate the original text to German (DE) or Spanish (ES) and back to English. Then, the model is trained on the back-translated text with the original text. Our experiments show that MODALS outperforms the other baseline methods in ﬁve of the settings and is comparable with back-translation (ES) in the TREC6 dataset.

Original: While EDA and senMixup  also show improvement on the full datasets, senMix slightly underperforms the ﬁrst baseline when the dataset size is small (see Table 1). 
Revised: While EDA, senMixup and back-translation also show improvement on the full datasets, senMixup and back-translation sometimes slightly underperform the ﬁrst baseline when the dataset size is small (see Table 1). [Table]

Original: [Table]
Revised: [Table]

Original: 4. T ABULAR D ATA
Revised: T ABULAR D ATA

Original: We also perform an experiment with multiple tabular datasets from the UCI repository (Dua & Graff, 2017), including the Iris, Breast Cancer, Arcene (Guyon et al., 2005), Abalone, and HTRU2 (Lyon et al., 2016) datasets.
Revised: We also perform an experiment with multiple tabular datasets from the UCI repository (Dua & Graff, 2017), including the Iris, Breast Cancer, Arcene (Guyon et al., 2005), Abalone [Table] and HTRU (Lyon et al., 2016) datasets.

Original: The data are standardized and trained with a 2-layer multilayer perceptron (MLP) that encodes the inputs to 128-dimensional latent codes.
Revised: The data are standardized and trained with a 2-layer MLP that encodes the inputs to 128-dimensional latent codes.

Original: We compare the accuracy with baseline methods using no augmentation and using Mixup.
Revised: We compare the accuracy with baseline methods using no augmentation and using Mixup.

Original: The experiment is repeated with different number of training examples.
Revised: The experiment is repeated with different number of training examples.

Original: Table 2 shows that MODALS outperforms the baseline methods inout of 15 settings.
Revised: Table 2 shows that MODALS outperforms the baseline methods in 13 out of 15 settings.

Original: In the remaining four settings, MODALS achieves the same performance as Mixup augmentation.
Revised: In the remaining two settings, MODALS achieves the same performance as Mixup augmentation

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: SERIES D ATA
Revised: [Table] [Table] [Table] [Table]

Revised: Cancer

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Arcene (s)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Arcene (m)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Arcene

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  For time-series data, we use the HAR (Anguita et al., 2013) and Malware (Catak, 2019) datasets.
Revised: [Table] [Table] 4. T IME - SERIES D ATA For time-series data, we use the HAR (Anguita et al., 2013) and Malware (Catak, 2019) datasets.

Original: HAR and Malware belong to continuous and discrete time-series data, respectively.
Revised: HAR and Malware belong to continuous and discrete time-series data, respectively.

Original: The input of the HAR dataset consists of multiple continuous smartphone accelerometer and gyroscope readings for six human activities.
Revised: The input of the HAR dataset consists of multiple continuous smartphone accelerometer and gyroscope readings for six human activities.

Original: The input of the Malware dataset consists of sequences of discrete events, speciﬁcally, the API call sequences of eight malwares.
Revised: The input of the Malware dataset consists of sequences of discrete events, speciﬁcally, the API call sequences of eight malwares.

Original: For both datasets, we train an LSTM model to encode the time-series data as 128-dimensional feature vectors and apply MODALS .
Revised: For both datasets, we train an LSTM model to encode the time-series data as 128-dimensional feature vectors and apply MODALS .

Original: MODALS performs the best in all cases (see Table 3).[Table]
Revised: MODALS performs the best in all cases (see Table 3).

Original:  I MAGE D ATA
Revised: 4. I MAGE D ATA

Original: For image data, we apply MODALS to CIFAR-10, SVHN, and the reduced versions of these two datasets, as in (Ho et al., 2019).
Revised: For image data, we apply MODALS to CIFAR-10, SVHN, and the reduced versions of these two datasets, as in (Ho et al., 2019).

Original: CIFAR-10 and its reduced version contain 50,000 and 4,000 training images, respectively, while SVHN and its reduced version contain 73,537 and 1,000 training images, respectively.
Revised: CIFAR-10 and its reduced version contain 50,000 and 4,000 training images, respectively, while SVHN and its reduced version contain 73,537 and 1,000 training images, respectively.

Original: We use Wide-ResNet-40-2 as the feature extraction model in all experiments (Zagoruyko & Komodakis, 2016).
Revised: We use Wide-ResNet-40-2 as the feature extraction model in all experiments (Zagoruyko & Komodakis, 2016).

Original: We compare our method against three baselines.
Revised: We compare our method against three baselines.

Original: For the ﬁrst  baseline, we apply randomized cropping, horizontal ﬂipping, and color normalization.
Revised: For the ﬁrst [Table] baseline, we apply randomized cropping, horizontal ﬂipping, and color normalization.

Original: For the second baseline, Cutout randomly occludes a patch of size 16 × 16 from the image on top of the simple augmentation in the ﬁrst baseline.
Revised: For the second baseline, Cutout randomly occludes a patch of size 16 × 16 from the image on top of the simple augmentation in the ﬁrst baseline.

Original: The third baseline is PBA, which uses the same search strategy with MODALS .
Revised: The third baseline is PBA, which uses the same search strategy with MODALS .

Original: Both PBA and MODALS are deployed with their corresponding searched policy schedules on top of Cutout.
Revised: Both PBA and MODALS are deployed with their corresponding searched policy schedules on top of Cutout.

Original: On the two original and the two reduced datasets, MODALS outperforms simple augmentation and Cutout but underperforms PBA (see Table 4).
Revised: On the two original and the two reduced datasets, MODALS outperforms simple augmentation and Cutout but underperforms PBA (see Table 4).

Original: We suspect that the input-space augmentations used in PBA are able to cover most variations in these datasets due to the continuous nature of image data.
Revised: We suspect that the input-space augmentations used in PBA are able to cover most variations in these datasets due to the continuous nature of image data.

Original: As the input-space augmentations are independent of the latent space augmentations, it is possible to combine PBA with MODALS by jointly searching the optimal augmentation policy parameters to further improve the results.
Revised: As the input-space augmentations are independent of the latent space augmentations, it is possible to combine PBA with MODALS by jointly searching the optimal augmentation policy parameters to further improve the results.

Original: [Table]
Revised: [Table]

Original:  A BLATION S TUDY
Revised:  A BLATION S TUDY

Original: Additional losses.
Revised: Latent space augmentation. We study the effect of latent space augmentation techniques in MODALS . We remove the latent space augmentation to isolate the effect from applying the triplet loss and adversarial loss.

Revised: Table 5 summarizes the performance of MODALS across eight datasets from different modalities.

Revised: Our study shows that the use of latent space augmentation contributes to additional performance gain when trained with different loss objectives.

Revised: The detailed breakdown is listed in Appendix A.2.

Revised: Additional losses.

Original: We perform ablation study against the baseline models trained with different loss settings on eight datasets.
Revised: We perform ablation study against the baseline models trained with different loss settings on eight datasets.

Original: Both the triplet loss and adversarial loss improve the performance when training with latent space augmentation (see Table 5).
Revised: Both the triplet loss and adversarial loss improve the performance when training with latent space augmentation (see Table 5).

Original: In Appendix A.2.1, we provide the details and evidence showing that the triplet loss enforces more compact local class regions to preserve the class labels when applying latent space augmentation. 
Revised: In Appendix A.2.1, we provide the details and evidence showing that the triplet loss enforces more compact local class regions to preserve the class labels when applying latent space augmentation. [Table]

Original: Hard examples .
Revised: Hard examples .

Original: We study the effectiveness of our proposed hard augmentation in creating uncertain examples on eight datasets.
Revised: We study the effectiveness of our proposed hard augmentation in creating uncertain examples on eight datasets.

Original: Taking the difference of the predicted probability between the most likely and second most likely labels as a measurement of prediction certainty, our study shows that the proposed hard example interpolation and extrapolation create 7.23% and 2.34%, respectively, less certain examples on average (see Table 6).
Revised: Taking the difference of the predicted probability between the most likely and second most likely labels as a measurement of prediction certainty, our study shows that the proposed hard example interpolation and extrapolation create 7.23% and 2.34%, respectively, less certain examples on average (see Table 6).

Original: We further validate that the augmented examples improve the classiﬁcation performance on hard examples (see Appendix A.2.2, A.2.3) [Table] [Table]
Revised: We further validate that the augmented examples improve the classiﬁcation performance on hard examples (see Appendix A.2.2, A.2.3).

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce MODALS to apply automated data augmentation in the latent space using four proposed modality-agnostic transformations trained with additional loss metrics and hard example augmentation techniques.
Revised: In this paper we introduce MODALS to apply automated data augmentation in the latent space using four proposed modality-agnostic transformations trained with additional loss metrics and hard example augmentation techniques.

Original: Our method is tested on text, tabular, time-series, and image data and can be readily integrated with popular deep learning models.
Revised: Our method is tested on text, tabular, time-series and image data and can be readily integrated with popular deep learning models.

Original: Beyond the data modalities tested,
Revised: Beyond the data modalities tested,

Original: MODALS also work on all other data modalities given proper feature extraction models.
Revised: MODALS also work on all other data modalities given proper feature extraction models.

Original: We believe that MODALS shows larger improvements to data modalities in which the input space augmentation is less trivial to be deﬁned, like text data, video data or even graph data in the low-resource regime.
Revised: We believe that MODALS shows larger improvements to data modalities in which the input space augmentation is less trivial to be deﬁned, like text data, video data or even graph data in the low-resource regime.

Original: A A PPENDIX A. V ALIDITY OF I SOTROPIC A SSUMPTION IN THE L ATENT S PACE
Revised: A A PPENDIX A. V ALIDITY OF I SOTROPIC A SSUMPTION IN THE L ATENT S PACE

Original: Due to the curse of dimensionality high-dimensional vectors are usually sparsely distributed.
Revised: Due to the curse of dimensionality, high-dimensional vectors are usually sparsely distributed.

Original: Let us consider a high-dimensional hypersphere with data points evenly distributed inside the hypersphere.
Revised: Let us consider a high-dimensional hypersphere with data points evenly distributed inside the hypersphere.

Original: As the dimensionality increases it can be shown that the percentage of data points residing near the surface of the hypersphere will increase signiﬁcantly.
Revised: As the dimensionality increases, it can be shown that the percentage of data points residing near the surface of the hypersphere will increase signiﬁcantly.

Original: This contradicts our common perception for low-dimensional spaces, such as two- or three-dimensional spaces.
Revised: This contradicts our common perception for low-dimensional spaces, such as two- or three-dimensional spaces.

Original: Worse still, in case the phenomenon indeed holds the isotropic assumption about the region of a class will be violated.
Revised: Worse still, in case the phenomenon indeed holds, the isotropic assumption about the region of a class will be violated.

Original: As such we want to conduct an experiment to see if there is evidence revealing the occurrence of this unusual phenomenon.
Revised: As such, we want to conduct an experiment to see if there is evidence revealing the occurrence of this unusual phenomenon.

Original: Speciﬁcally we study the spread of data points in the latent spaces of different dimensionality used.
Revised: Speciﬁcally, we study the spread of data points in the latent spaces of different dimensionality used.

Original: For each class c of size m  we compute the normalized distances { r ci } mi =1 from the class mean µ c to the latent representations { z ci } mi =1 :
Revised: For each class c of size m , we compute the normalized distances { r ci } mi =1 from the class mean µ c to the latent representations { z ci } mi =1 :

Original: [Equation]
Revised: [Equation]

Original: We repeat the experiment on one dataset for each of the four data modalities.
Revised: We repeat the experiment on one dataset for each of the four data modalities.

Original: Table 7 reports the average standard deviation of the normalized distances for all classes.
Revised: Table 7 reports the average standard deviation of the normalized distances for all classes.

Original: The average standard deviation in all datasets shows a decreasing trend as the dimensionality increases, falling between 0.25 to 0.75 for a dimensionality of 128 or 256.
Revised: The average standard deviation in all datasets shows a decreasing trend as the dimensionality increases, falling between 0.25 to 0.75 for a dimensionality of 128 or 256.

Original: In case the unusual phenomenon as described above really holds so that most of the data points have similar distances from their class center, we would expect the standard deviation to be very small or even close to zero.
Revised: In case the unusual phenomenon as described above really holds so that most of the data points have similar distances from their class center, we would expect the standard deviation to be very small or even close to zero.

Original: Our experiment shows that this is not the case especially for lower latent space dimensionality.
Revised: Our experiment shows that this is not the case especially for lower latent space dimensionality.

Original: [Table]
Revised: [Table]

Original: A. A BLATION S TUDY A.2. L OSS
Revised: A. A BLATION S TUDY A.2. L OSS

Original: Here, we study the effect of using additional loss metrics for data augmentation in the latent space.
Revised: Here, we study the effect of using additional loss metrics for data augmentation in the latent space.

Original: We repeat our previous experiments by removing one or both of the triplet and adversarial losses.
Revised: We repeat our previous experiments by removing one or both of the triplet and adversarial losses.

Original: Table 8 shows that adding the proposed loss metrics improves the classiﬁcation performance across all datasets of different modalities.[Table]
Revised: Table 8 shows that adding the proposed loss metrics improves the classiﬁcation performance across all datasets of different modalities.

Original: We further analyze the effect of the triplet loss in the augmentation context.
Revised: We further analyze the effect of the triplet loss in the augmentation context.

Original: Using the triplet loss forces the distribution of data points in the same class to be more compact.
Revised: Using the triplet loss forces the distribution of data points in the same class to be more compact.

Original: Consequently, the local class region becomes more convex and isotropic and hence facilitates smoother label-preserving transformation.
Revised: Consequently, the local class region becomes more convex and isotropic and hence facilitates smoother label-preserving transformation.

Original: We validate our conjecture by constraining the latent space to two dimensions to facilitate visualizing the data points (see Figure 3).
Revised: We validate our conjecture by constraining the latent space to two dimensions to facilitate visualizing the data points (see Figure 3).

Original: Without the triplet loss, some class regions overlap and appear in less regular shapes.
Revised: Without the triplet loss, some class regions overlap and appear in less regular shapes.

Original: When the triplet loss is used, the class regions become more compact and appear in more circular shapes as observed from the coordinate axes and the visualization plots.
Revised: When the triplet loss is used, the class regions become more compact and appear in more circular shapes as observed from the coordinate axes and the visualization plots.

Original: This favors the convex and isotropic assumption in Section 2 of the paper when we propose to apply linear transformation on the feature vectors in local class regions.
Revised: This favors the convex and isotropic assumption in Section 2 of the paper when we propose to apply linear transformation on the feature vectors in local class regions.

Original: As the classes are better separated, the transformation can better preserve the original class labels.
Revised: As the classes are better separated, the transformation can better preserve the original class labels.

Original: [Figure]
Revised: [Table]

Revised: Average

Revised: [Table]

Revised: Change -

Revised: [Table]

Original:  In addition to visualizing the data points qualitatively, we also provide quantitative measurements in Table 9 to show the separation between and within the class regions for higher dimensionality.
Revised: [Table] [Table] [Figure] In addition to visualizing the data points qualitatively, we also provide quantitative measurements in Table 9 to show the separation between and within the class regions for higher dimensionality.

Original: Speciﬁcally, we measure the average within-class distance and between-class distance of the latent representations when training with and without the triplet loss.
Revised: Speciﬁcally, we measure the average within-class distance and between-class distance of the latent representations when training with and without the triplet loss.

Original: Our study shows that the average within-class distance with the triplet loss is smaller, resulting in a more compact class region.
Revised: Our study shows that the average within-class distance with the triplet loss is smaller, resulting in a more compact class region.

Original: The ratio of between-class and within-class distances suggests a better separation between class regions rather than solely the scaling effect.
Revised: The ratio of between-class and within-class distances suggests a better separation between class regions rather than solely the scaling effect.

Original: Similar observations are also noticed in other datasets. 
Revised: Similar observations are also noticed in other datasets. [Table]

Original: A.2. H ARD E XAMPLE A UGMENTATION
Revised: A.2. H ARD E XAMPLE A UGMENTATION

Original: Next, we study the effectiveness of our proposed transformation in creating hard examples.
Revised: Next [Table] we study the effectiveness of our proposed transformation in creating hard examples.

Original: In our experiment, we measure the uncertainty of an example by the difference in predicted probability
Revised: In our experiment

Original: [Table]
Revised: [Table]

Original:  between the most likely and second most likely labels (Scheffer et al., 2001).
Revised: we measure the uncertainty of an example by the difference in predicted probability between the most likely and second most likely labels (Scheffer et al., 2001).

Original: We call this metric as margin.
Revised: We call this metric as margin.

Original: A smaller margin implies a more uncertain prediction.
Revised: A smaller margin implies a more uncertain prediction.

Original: [Equation]
Revised: [Equation]

Original: We train the models on eight datasets and compute the margin for the original features and the augmented features.
Revised: We train the models on eight datasets and compute the margin for the original features and the augmented features.

Original: Our results show that hard interpolation is more effective in creating uncertain examples than hard extrapolation in most datasets (see Table 10).
Revised: Our results show that hard interpolation is more effective in creating uncertain examples than hard extrapolation in most datasets (see Table 10).

Original: Admittedly, this ﬁnding is a bit counter-intuitive.
Revised: Admittedly, this ﬁnding is a bit counter-intuitive.

Original: We suspect that it may be due to the different shapes and complexity of the class boundaries [Table] but more investigation is needed to understand the reasons behind.
Revised: We suspect that it may be due to the different shapes and complexity of the class boundaries, but more investigation is needed to understand the reasons behind.

Original: [Table]
Revised: [Table]

Original: A.2. H ARD E XAMPLE T RAINING
Revised: A.2. H ARD E XAMPLE T RAINING

Original: We also study whether the hard examples created using the proposed transformation beneﬁt model training.
Revised: We also study whether the hard examples created using the proposed transformation beneﬁt model training.

Original: In the ablation test, we sample the top 5% hardest examples based on the classiﬁcation loss from each class and compare the changes in the margin and accuracy of the hard examples after training with different augmentation settings (see Table 11).
Revised: In the ablation test [Table] we sample the top 5% hardest examples based on the classiﬁcation loss from each class and compare the changes in the margin and accuracy of the hard examples after training with different augmentation settings (see Table 11).

Original: We use a ﬁxed augmentation instead of a ﬁne-tuned policy schedule.
Revised: We use a ﬁxed augmentation instead of a ﬁne-tuned policy schedule.

Original: In the experiment, the models trained with hard example interpolation and extrapolation achieve a higher accuracy on the sampled hard examples.
Revised: In the experiment, the models trained with hard example interpolation and extrapolation achieve a higher accuracy on the sampled hard examples.

Original: The larger increases in margin show that the models can learn to predict the hard examples with higher certainty.
Revised: The larger increases in margin show that the models can learn to predict the hard examples with higher certainty.

Original: Finally [Table] we compare the end-to-end training with simple interpolation and extrapolation to our proposed hard augmentation methods.
Revised: Finally, we compare the end-to-end training with simple interpolation and extrapolation to our proposed hard augmentation methods.

Original: Table 12 presents the accuracy of the model trained with hard augmentation and simple interpolation and extrapolation
Revised: Table 12 presents the accuracy of the model trained with hard augmentation and simple interpolation and extrapolation. [Table] [Table] A.

Original: [Table]
Revised: [Table]

Original: A. E XPERIMENT D ETAILS
Revised: E XPERIMENT D ETAILS

Original: In this section [Table] we present the detailed conﬁguration and augmentation schedule for each of the data modalities.
Revised: In this section, we present the detailed conﬁguration and augmentation schedule for each of the data modalities.

Original: In all the experiments, the augmentation policy is searched using 50% of the data as[Table] [Table] the validation set.
Revised: In all the experiments, the augmentation policy is searched using 50% of the data as the validation set.

Original: We use 16 child models in PBA implemented using the Ray Tune framework ( https://docs.ray.io/en/latest/tune/index.html ).
Revised: We use 16 child models in PBA implemented using the Ray Tune framework ( https://docs.ray.io/en/latest/tune/index.html ).

Original: The child models are evaluated and perturbed every three epochs.
Revised: The child models are evaluated and perturbed every three epochs.

Revised: For the discriminator, we employ a 2-layer MLP model with 256 hidden units and ReLU activations.

Revised: The discriminator is trained using the Adam optimizer with learning rate 0.01.

Original: A.3. T EXT D ATA
Revised: A.3. T EXT D ATA

Original: We employ a single-layer bidirectional LSTM to encode the 300-dimensional GloVe text embedding input as a 128-dimensional latent representation.
Revised: We employ a single-layer bidirectional LSTM to encode the 300-dimensional GloVe text embedding input as a 128-dimensional latent representation.

Original: The model is trained for 100 epochs using the
Revised: The model is trained for 100 epochs using the

Original: Adam optimizer with learning rate 0.01 and batch size 100.
Revised: Adam optimizer with learning rate 0.01 and batch size 100.

Original: Figure 4(a,b) and Figure 4(c,d) illustrate the augmentation policies for the SST2 and TREC6 datasets, respectively.
Revised: Figure 4(a,b) and Figure 4(c,d) illustrate the augmentation policies for the SST2 and TREC6 datasets, respectively.

Original: A.3. T ABULAR D ATA
Revised: A.3. T ABULAR D ATA

Original: For all tabular datasets, we split 20% of the dataset as the test set unless the test set is explicitly provided in the repository.
Revised: For all tabular datasets, we split 20% of the dataset as the test set unless the test set is explicitly provided in the repository.

Original: The tabular data are ﬁrst standardized and trained with a two-layer MLP model with 128 hidden units in each hidden layer.
Revised: The tabular data are ﬁrst standardized and trained with a 2-layer MLP model with 128 hidden units in each hidden layer.

Original: We use the Adam optimizer with learning rate 0.01 and batch size 32 to train the model for 30 epochs.
Revised: We use the Adam optimizer with learning rate 0.01 and batch size 32 to train the model for 30 epochs.

Original: In some cases when the dataset size is too small, we use a batch size of 16 instead.
Revised: In some cases when the dataset size is too small, we use a batch size of 16 instead.

Original: Figure 4(e,f) and Figure 4(g,h) show the augmentation policies for the IRIS and Breast Cancer datasets, respectively.
Revised: Figure 4(e,f) and Figure 4(g,h) show the augmentation policies for the IRIS and Breast Cancer datasets, respectively.

Original: A.3. T IME - SERIES D ATA
Revised: A.3. T IME - SERIES D ATA

Original: For the HAR and Malware datasets, the time series are split into subsequences using a sliding window of size 256 with 50% overlap.
Revised: For the HAR and Malware datasets, the time series are split into subsequences using a sliding window of size 256 with 50% overlap.

Original: In the Malware dataset, we remove consecutive duplicated API calls.
Revised: In the Malware dataset, we remove consecutive duplicated API calls.

Original: We use a single-layer LSTM model to encode the time-series input as a 128-dimensional latent representation.
Revised: We use a single-layer LSTM model to encode the time-series input as a 128-dimensional latent representation.

Original: For the Malware dataset, we exploit an additional embedding layer to map the discrete API call events into a 32-dimensional input embedding.
Revised: For the Malware dataset, we exploit an additional embedding layer to map the discrete API call events into a 32-dimensional input embedding.

Original: The LSTM models are trained using the Adam optimizer with learning rate 0.01 and batch size 128 for 50 epochs.
Revised: The LSTM models are trained using the Adam optimizer with learning rate 0.01 and batch size 128 for 50 epochs.

Original: The augmen- tation policies for the HAR and Malware datasets are presented in Figure 4(i,j) and Figure 4(k,l), respectively.
Revised: The augmentation policies for the HAR and Malware datasets are presented in Figure 4(i,j) and Figure 4(k,l), respectively.

Original: A.3. I MAGE D ATA
Revised: A.3. I MAGE D ATA

Original: We use Wide-ResNet-40-2 as the feature extractor for the CIFAR-10 and SVHN datasets.
Revised: We use Wide-ResNet-40-2 as the feature extractor for the CIFAR-10 and SVHN datasets.

Original: The models are trained using the SGD optimizer with batch size 100 for 200 epochs.
Revised: The models are trained using the SGD optimizer with batch size 100 for 200 epochs.

Original: We use a weight decay of 10 − 4 , learning rate of 0.01, and a cosine learning decay with one annealing cycle.
Revised: We use a weight decay of 10 − 4 , learning rate of 0.01, and a cosine learning decay with one annealing cycle.

Original: Figure 4(m,n) and Figure 4(o,p) show the augmentation policies for the CIFAR-10 and SVHN datasets, respectively.
Revised: Figure 4(m,n) and Figure 4(o,p) show the augmentation policies for the CIFAR-10 and SVHN datasets, respectively.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]