Original: B YPASSING THE R ANDOM I NPUT M IXING IN M IXUP
Revised: B YPASSING THE R ANDOM I NPUT M IXING IN M IXUP

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Mixup and its variants have promoted a surge of interest due to their capability of boosting the accuracy of deep models.
Revised: Mixup and its variants have promoted a surge of interest due to their capability of boosting the accuracy of deep models.

Original: For a random sample pair, such approaches generate a set of synthetic samples through interpolating both the inputs and their corresponding one-hot labels.
Revised: For a random sample pair, such approaches generate a set of synthetic samples through interpolating both the inputs and their corresponding one-hot labels.

Original: Current methods either interpolate random features from an input pair or learn to mix salient features from the pair.
Revised: Current methods either interpolate random features from an input pair or learn to mix salient features from the pair.

Original: Nevertheless, the former methods can create misleading synthetic samples or remove important features from the given inputs, and the latter strategies incur signiﬁcant computation cost for selecting descriptive input regions.
Revised: Nevertheless, the former methods can create misleading synthetic samples or remove important features from the given inputs, and the latter strategies incur signiﬁcant computation cost for selecting descriptive input regions.

Original: In this paper, we show that the effort needed for the input mixing can be bypassed.
Revised: In this paper, we show that the effort needed for the input mixing can be bypassed.

Original: For a given sample pair, averaging the features from the two inputs and then assigning it with a set of soft labels can effectively regularize the training.
Revised: For a given sample pair, averaging the features from the two inputs and then assigning it with a set of soft labels can effectively regularize the training.

Original: We empirically show that the proposed approach performs on par with state-of-the-art strategies in terms of predictive accuracy.
Revised: We empirically show that the proposed approach performs on par with state-of-the-art strategies in terms of predictive accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have demonstrated their profound successes in many challenging real-world applications, including image classiﬁcation (Krizhevsky et al., 2012), speech recognition (Graves et al., 2013), and machine translation (Bahdanau et al., 2015; Sutskever et al., 2014).
Revised: Deep neural networks have demonstrated their profound successes in many challenging real-world applications, including image classiﬁcation (Krizhevsky et al., 2012), speech recognition (Graves et al., 2013), and machine translation (Bahdanau et al., 2015; Sutskever et al., 2014).

Original: One key factor attributing to such successes is the deployment of effective model regularization techniques, which empower the learning to avoid overﬁtting the training data and to generalize well to unseen samples.
Revised: One key factor attributing to such successes is the deployment of effective model regularization techniques, which empower the learning to avoid overﬁtting the training data and to generalize well to unseen samples.

Original: This is because current deep models typically embrace high modeling freedom with a very large number of parameters.
Revised: This is because current deep models typically embrace high modeling freedom with a very large number of parameters.

Original: To this end, many regularizers for deep models have been introduced, including weight decay (Hanson & Pratt, 1988), dropout (Srivastava et al., 2014), stochastic depth (Huang et al., 2016), batch normalization (Ioffe & Szegedy, 2015), and data augmentation schemes (Cubuk et al., 2019; Hendrycks et al., 2020; Inoue, 2018;
Revised: To this end, many regularizers for deep models have been introduced, including weight decay (Hanson & Pratt, 1988), dropout (Srivastava et al., 2014), stochastic depth (Huang et al., 2016), batch normalization (Ioffe & Szegedy, 2015), and data augmentation schemes (Cubuk et al., 2019; Hendrycks et al., 2020; Inoue, 2018;

Original: Lecun et al., 1998; Simard et al., 1998).
Revised: Lecun et al., 1998; Simard et al., 1998).

Original: [Figure]
Revised: [Figure]

Original: Among those effective regularizers, Mixup (Zhang et al., 2018) is a simple and yet effective, dataaugmentation based regularizer for enhancing the deep classiﬁcation models.
Revised: Among those effective regularizers, Mixup (Zhang et al., 2018) is a simple and yet effective, dataaugmentation based regularizer for enhancing the deep classiﬁcation models.

Original: Through linearly interpolating random input pairs and their training targets in one-hot representation,
Revised: Through linearly interpolating random input pairs and their training targets in one-hot representation,

Original: Mixup generates a set of synthetic examples with soft labels to regularize the training.
Revised: Mixup generates a set of synthetic examples with soft labels to regularize the training.

Original: Such pairwise, label-variant data augmentation techniques (Guo, 2020; Guo et al., 2019;
Revised: Such pairwise, label-variant data augmentation techniques (Guo, 2020; Guo et al., 2019;

Original: Kim et al., 2020; Li et al., 2020a; Tokozume et al., 2018a;b; Verma et al., 2019; Yun et al., 2019; Zhang et al., 2018) have attracted a surge of interest and shown their effectiveness on boosting the accuracy of deep networks.
Revised: Kim et al., 2020; Li et al., 2020a; Tokozume et al., 2018a;b; Verma et al., 2019; Yun et al., 2019; Zhang et al., 2018) have attracted a surge of interest and shown their effectiveness on boosting the accuracy of deep networks.

Original: Nevertheless, unlike label-preserving data augmentation such as rotation, ﬂip, and crop, there is still limited knowledge about how to design better mixing policies for sample pairs for effective label-variant regularization.
Revised: Nevertheless, unlike label-preserving data augmentation such as rotation, ﬂip, and crop, there is still limited knowledge about how to design better mixing policies for sample pairs for effective label-variant regularization.

Original: Current Mixup-based approaches either mix a pair of inputs using random mixing coefﬁcients (Guo, 2020; Guo et al., 2019; Summers & Dinneen, 2019; Tokozume et al., 2018b; Verma et al., 2019;
Revised: Current Mixup-based approaches either mix a pair of inputs using random mixing coefﬁcients (Guo, 2020; Guo et al., 2019; Summers & Dinneen, 2019; Tokozume et al., 2018b; Verma et al., 2019;

Original: Yun et al., 2019; Zhang et al., 2018) or learn to mix salient features from the given pair (Dabouei et al., 2020;
Revised: Yun et al., 2019; Zhang et al., 2018) or learn to mix salient features from the given pair (Dabouei et al., 2020;

Original: Kim et al., 2020; Li et al., 2020b; Walawalkar et al., 2020) to create a set of synthetic samples.
Revised: Kim et al., 2020; Li et al., 2020b; Walawalkar et al., 2020) to create a set of synthetic samples.

Original: Nonetheless, the former methods may create misleading synthetic samples (Guo et al., 2019) or remove important features from the given inputs (Kim et al., 2020) and the latter strategies can incur signiﬁcant computation cost for identifying and selecting the most descriptive input regions (Kim et al., 2020; Walawalkar et al., 2020).
Revised: Nonetheless, the former methods may create misleading synthetic samples (Guo et al., 2019) or remove important features from the given inputs (Kim et al., 2020) and the latter strategies can incur signiﬁcant computation cost for identifying and selecting the most descriptive input regions (Kim et al., 2020; Walawalkar et al., 2020).

Original: In this paper, we show that the effort needed for the input mixing with a range of mixing ratios can be bypassed.
Revised: In this paper, we show that the effort needed for the input mixing with a range of mixing ratios can be bypassed.

Original: For a given input pair, one can average the features from the two inputs, and then assign it with a set of soft labels.
Revised: For a given input pair, one can average the features from the two inputs, and then assign it with a set of soft labels.

Original: These soft labels are adaptively learned during training to incorporate class information beyond the provided label pair as well as the evolving states of the training.
Revised: These soft labels are adaptively learned during training to incorporate class information beyond the provided label pair as well as the evolving states of the training.

Original: The method is illustrated in Figure 1, where the mixed input is the pixel-wise average of features from the two inputs and its training target is the combination of the local soft label, which is the average of the two one-hot targets, and the global soft label, which is generated by the networks during training.
Revised: The method is illustrated in Figure 1, where the mixed input is the pixel-wise average of features from the two inputs and its training target is the combination of the local soft label, which is the average of the two one-hot targets, and the global soft label, which is generated by the networks during training.

Original: We empirically show that the proposed approach performs on par with state-of-the-art methods with random input mixing policy or learning to mixing strategy, in terms of predictive accuracy.
Revised: We empirically show that the proposed approach performs on par with state-of-the-art methods with random input mixing policy or learning to mixing strategy, in terms of predictive accuracy.

Original: We also demonstrate that the synthetic samples created by our method keep tuning the networks long after the training error on the original training set is minimal, encouraging the learning to generate, for each class of the training samples, tight representation clusters.
Revised: We also demonstrate that the synthetic samples created by our method keep tuning the networks long after the training error on the original training set is minimal, encouraging the learning to generate, for each class of the training samples, tight representation clusters.

Original: Also, the Class Activation Mapping (CAM) (Zhou et al., 2016) visualization suggests that our method tends to focus on narrower regions of an image for classiﬁcation.
Revised: Also, the Class Activation Mapping (CAM) (Zhou et al., 2016) visualization suggests that our method tends to focus on narrower regions of an image for classiﬁcation.

Original:  B YPASSING THE I NPUT M IXING IN M IXUP 2. M IXUP -B ASED D ATA A UGMENTATION
Revised:  B YPASSING THE I NPUT M IXING IN M IXUP 2. M IXUP -B ASED D ATA A UGMENTATION

Original: For a standard classiﬁcation setting with a training data set ( X ; Y ) , the objective of the task is to develop a classiﬁer which assigns every input x ∈ X a label y ∈ Y .
Revised: For a standard classiﬁcation setting with a training data set ( X ; Y ) , the objective of the task is to develop a classiﬁer which assigns every input x ∈ X a label y ∈ Y .

Original: Instead of using the provided training set ( X ; Y ) , Mixup (Zhang et al., 2018) generates synthetic samples with soft labels for training.
Revised: Instead of using the provided training set ( X ; Y ) , Mixup (Zhang et al., 2018) generates synthetic samples with soft labels for training.

Original: For a pair of random training samples ( x i ; y i ) and ( x j ; y j ) , where x is the input and y the one-hot encoding of the corresponding class, Mixup creates a synthetic sample as follows.
Revised: For a pair of random training samples ( x i ; y i ) and ( x j ; y j ) , where x is the input and y the one-hot encoding of the corresponding class, Mixup creates a synthetic sample as follows.

Original: [Equation]
Revised: [Equation]

Original: where λ is a scalar mixing policy for mixing both the inputs and the modeling targets of the sample pair.
Revised: where λ is a scalar mixing policy for mixing both the inputs and the modeling targets of the sample pair.

Original: λ is sampled from a Beta( α, α ) distribution with a hyper-parameter α .
Revised: λ is sampled from a Beta( α, α ) distribution with a hyper-parameter α .

Original: The generated samples ( (cid:101) x ij λ , (cid:101) y ij λ ) are then fed into the model for training to minimize the cross-entropy loss function.
Revised: The generated samples ( (cid:101) x ij λ , (cid:101) y ij λ ) are then fed into the model for training to minimize the cross-entropy loss function.

Original: Current variants of Mixup focus on the introduction of a representation function ψ for input mixing:
Revised: Current variants of Mixup focus on the introduction of a representation function ψ for input mixing:

Original: [Equation]
Revised: [Equation]

Original: The state-of-the-art random input mixing variant CutMix (Yun et al., 2019) deﬁnes the ψ function to form a binary rectangular mask applying to a randomly chosen rectangle covering λ proportion of the input image.
Revised: The state-of-the-art random input mixing variant CutMix (Yun et al., 2019) deﬁnes the ψ function to form a binary rectangular mask applying to a randomly chosen rectangle covering λ proportion of the input image.

Original: Puzzle Mix (Kim et al., 2020) is the state-of-the-art learning to mixing variant.
Revised: PuzzleMix (Kim et al., 2020) is the state-of-the-art learning to mixing variant.

Original: This method deﬁnes the ψ function to compute the saliency map of the input pair, ﬁnd the optimal mask, and optimize the transport plans for generating the mixed example.
Revised: This method deﬁnes the ψ function to compute the saliency map of the input pair, ﬁnd the optimal mask, and optimize the transport plans for generating the mixed example.

Original: This is to ensure that the mixed image contain sufﬁcient target class information corresponding to the mixing ratio λ while preserving the local statistics of each input.
Revised: This is to ensure that the mixed image contain sufﬁcient target class information corresponding to the mixing ratio λ while preserving the local statistics of each input.

Original: 2. F ORMING THE T RAINING I NPUT
Revised: 2. F ORMING THE T RAINING I NPUT

Original: Similar to Mixup and its variants, our method also creates mixed samples from a random sample pair, but for a given sample pair our method does not generate a set of inputs with different features.
Revised: Similar to Mixup and its variants, our method also creates mixed samples from a random sample pair, but for a given sample pair our method does not generate a set of inputs with different features.

Original: Instead, for a range of mixing ratios λ s, our approach uses the same mixed input, invariant to the mixing ratios provided.
Revised: Instead, for a range of mixing ratios λ s, our approach uses the same mixed input, invariant to the mixing ratios provided.

Original: We term our method Label driven Mixup (denoted as LaMix).
Revised: We term our method Label driven Mixup (denoted as LaMix).

Original: To this end, the mixed input (cid:101) x ij in LaMix has tow forms.
Revised: To this end, the mixed input (cid:101) x ij in LaMix has two forms.

Original: For the random input mixing methods such as Mixup and CutMix, it is formed by pixel-wise average of the raw features of the input pair ( x i , x j ):(cid:101) [Equation] ij
Revised: For the random input mixing methods such as Mixup and CutMix, it is formed by pixel-wise average of the raw features of the input pair ( x i , x j ):

Original: [Equation]
Revised: [Equation]

Original: For the learning to mixing methods such as Puzzle Mix, the mixed input is formed by pixel-wise average of the saliency features computed with λ value as 0.5:
Revised: For the learning to mixing methods such as PuzzleMix, the mixed input is formed by pixel-wise average of the saliency features computed with λ value as 0.5:

Original: [Equation]
Revised: [Equation]

Original: The newly resulting input (cid:101) x ij is then used for training, by passing through a neural network model f ϕ (parameterized with ϕ ) to generate the m -dimensional input embedding S ij ∈ R m :
Revised: The newly resulting input (cid:101) x ij is then used for training, by passing through a neural network model f ϕ (parameterized with ϕ ) to generate the m -dimensional input embedding S ij ∈ R m :

Original: [Equation]
Revised: [Equation]

Original: Next, the resulting S ij is then fed into a linear fullyconnected layer W l ∈ R c × m to produce the predicted classiﬁcation distribution over the c classiﬁcation target classes:
Revised: Next, the resulting S ij is then fed into a linear fullyconnected layer W l ∈ R c × m to produce the predicted classiﬁcation distribution over the c classiﬁcation target classes:

Original: [Equation]
Revised: [Equation]

Original: The error of the training is computed by comparing the prediction y ij and the training target of the input (cid:101) x ij , which is discussed next.
Revised: The error of the training is computed by comparing the prediction y ij and the training target of the input (cid:101) x ij , which is discussed next.

Original: 2. F ORMING THE T RAINING T ARGET
Revised: 2. F ORMING THE T RAINING T ARGET

Original: For the same mixed input (cid:101) x ij created from a random input pair ( x i , x j ), LaMix associates it with various soft target label ¨ y ij , which is learned by the networks with two steps: 1) obtaining global soft label and 2) integrating with local soft label.
Revised: For the same mixed input (cid:101) x ij created from a random input pair ( x i , x j ), LaMix associates it with various soft target label ¨ y ij , which is learned by the networks with two steps: 1) obtaining global soft label and 2) integrating with local soft label.

Original: In detail, the dynamically assigned soft target label ¨ y ij is a function of ( x i , x j , y i , y j , λ ), as follows.
Revised: In detail, the dynamically assigned soft target label ¨ y ij is a function of ( x i , x j , y i , y j , λ ), as follows.

Original: [Equation]
Revised: [Equation]

Original: As illustrated in Figure 1, LaMix implements the τ ( · ) function through adding an additional fullyconnected layer W t ∈ R c
Revised: As illustrated in Figure 1, LaMix implements the τ ( · ) function through adding an additional fullyconnected layer W t ∈ R c

Original: × m to the original networks.
Revised: × m to the original networks.

Original: That is, for the given input (cid:101) x ij with λ , LaMix ﬁrst computes its probabilities over the c classiﬁcation targets, denoted as p ijλ :
Revised: That is, for the given input (cid:101) x ij with λ , LaMix ﬁrst computes its probabilities over the c classiﬁcation targets, denoted as p ijλ :

Original: [Equation]
Revised: [Equation]

Original: where σ denotes the Sigmoid function, and S ij is the same input embedding as that in Equation 7.
Revised: where σ denotes the Sigmoid function, and S ij is the same input embedding as that in Equation 7.

Original: In other words, the two predictions (i.e., Equations 7 and 9) share the same networks except for the last layer.
Revised: In other words, the two predictions (i.e., Equations 7 and 9) share the same networks except for the last layer.

Original: Because the predictions p ijλ are about all the c target classes of the given data, such predictions thus reﬂect the class information beyond the provided local label pair, namely ( y i , y j ).
Revised: Because the predictions p ijλ are about all the c target classes of the given data, such predictions thus reﬂect the class information beyond the provided local label pair, namely ( y i , y j ).

Original: For description purpose, we term p ijλ global soft label.
Revised: For description purpose, we term p ijλ global soft label.

Revised: The Sigmoid function here will provide the probability of associating the input to a particular label.

Original: After having the global soft label p ijλ , we then integrate it with the local label of the given sample pair, i.e., (cid:101) y ijλ as described in Equation 2, as follows: 
Revised: After having the global soft label p ijλ , we then integrate it with the local label of the given sample pair, i.e., (cid:101) y ij λ as described in Equation 2, as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where β is a scalar coefﬁcient between [0,1].
Revised: where β is a scalar coefﬁcient between [0,1].

Original: Doing so, we enable the synthetic sample ( (cid:101) x ij , ¨ y ijλ ) to have information from both the local labels of the provided sample pair (i.e., (cid:101) y ijλ ) and the class information beyond the labels of the given pair (i.e., p ijλ ).
Revised: Doing so, we enable the synthetic sample ( (cid:101) x ij , ¨ y ijλ ) to have information from both the local labels of the provided sample pair (i.e., (cid:101) y ijλ ) and the class information beyond the labels of the given pair (i.e., p ijλ ).

Original: This process is also illustrated in Figure 1.
Revised: This process is also illustrated in Figure 1.

Original: 2. P ARAMETER O PTIMIZATION
Revised: 2. P ARAMETER O PTIMIZATION

Original: For training, LaMix minimize, with gradient descent, the cross entropy loss E between the assigned training soft target ¨ y ijλ and the network’s predicted classiﬁcation distribution y ij : 
Revised: For training, LaMix minimize, with gradient descent, the cross entropy loss E between the assigned training soft target ¨ y ijλ and the network’s predicted classiﬁcation distribution y ij : [Equation]

Original: [Equation]
Revised: [Equation]

Original: During training, we also need to prevent LaMix from assigning target soft labels too far away from the gold labels in the original training set.
Revised: During training, we also need to prevent LaMix from assigning target soft labels too far away from the gold labels in the original training set.

Original: To attain this goal, we alternatively feed samples to the networks with either a mini-batch with original inputs, i.e., (cid:101) x ii , or a mini-batch from the mixed inputs, i.e., (cid:101) x ij .
Revised: To attain this goal, we alternatively feed samples to the networks with either a mini-batch with original inputs, i.e., (cid:101) x ii , or a mini-batch from the mixed inputs, i.e., (cid:101) x ij .

Original: Note that, when training with the former, the networks still need to learn to assign the training target ¨ y iiλ to the sample (cid:101) x ii .
Revised: Note that, when training with the former, the networks still need to learn to assign the training target ¨ y iiλ to the sample (cid:101) x ii .

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compare our method with both random input mixing and learning to mixing Mixup variants.
Revised: We compare our method with both random input mixing and learning to mixing Mixup variants.

Original: 3. C OMPARE WITH R ANDOM I NPUT M IXING M ETHODS
Revised: 3. C OMPARE WITH R ANDOM I NPUT M IXING M ETHODS

Original: We use ﬁve image classiﬁcation tasks.
Revised: We use ﬁve image classiﬁcation tasks.

Original: MNIST is a digit (1-10) recognition dataset with 60, training and 10,000 test gray-level, 28x28-dimensional images.
Revised: MNIST is a digit (1-10) recognition dataset with 60, training and 10,000 test gray-level, 28x28-dimensional images.

Original: Fashion is an image recognition dataset having the same scale as MNIST, containing 10 classes of fashion product pictures.
Revised: Fashion is an image recognition dataset having the same scale as MNIST, containing 10 classes of fashion product pictures.

Original: SVHN is the Google street view house numbers recognition data set with 73,257 digits, 32x32 color images for training, 26,032 for testing, and 531,131 additional, easier samples.
Revised: SVHN is the Google street view house numbers recognition data set with 73,257 digits, 32x32 color images for training, 26,032 for testing, and 531,131 additional, easier samples.

Original: We did not use the additional images.
Revised: We did not use the additional images.

Original: Cifar10 is an image classiﬁcation task with 10 classes with 50,000 training and 10,000 test samples.
Revised: Cifar10 is an image classiﬁcation task with 10 classes with 50,000 training and 10,000 test samples.

Original: Cifar100 is similar to Cifar10 but with 100 classes and 600 images each.
Revised: Cifar100 is similar to Cifar10 but with 100 classes and 600 images each.

Original: We experimented using networks PreAct ResNet-18 (He et al., 2016) and ResNet-50 (He et al., 2016) (denoted as Vanilla model).
Revised: We experimented using networks PreAct ResNet-18 (He et al., 2016) and ResNet-50 (He et al., 2016) (denoted as Vanilla model).

Original: To ﬁt the MNIST and Fashion data into the same convolutional network architecture, we just simply resize the 28x28 image to 32x32 as that of Cifar10, Cifar100, and SVHN.
Revised: To ﬁt the MNIST and Fashion data into the same convolutional network architecture, we just simply resize the 28x28 image to 32x32 as that of Cifar10, Cifar100, and SVHN.

Original: Each reported value (accuracy or error rate) is the mean of ﬁve runs, on a NVIDIA GTX
Revised: Each reported value (accuracy or error rate) is the mean of ﬁve runs, on a NVIDIA GTX

Original: TitanX GPU with 12GB memory.
Revised: TitanX GPU with 12GB memory.

Original: The β as discussed in Equation 10 was set as 0.5 unless otherwise speciﬁed.
Revised: The β as discussed in Equation 10 was set as 0.5 unless otherwise speciﬁed.

Original: We compare with Mixup (Zhang et al., 2018) and the state-of-the-art random input mixing method
Revised: We compare with Mixup (Zhang et al., 2018) and the state-of-the-art random input mixing method

Original: CutMix (Yun et al., 2019).
Revised: CutMix (Yun et al., 2019).

Original: For Mixup, we use the code from Facebook at 1 .
Revised: For Mixup, we use the code from Facebook at 1 .

Original: For CutMix, we use the authors’ implementation in 2 .
Revised: For CutMix, we use the authors’ implementation in 2 .

Original: For LaMix, the added fully connected layer is just a copy of the fullyconnected layer of the original network with a Softmax function on the top.
Revised: For LaMix, the added fully connected layer is the same as the fullyconnected layer of the original network with a Softmax function on the top.

Original: For all the comparison baselines, the mixing ratio λ is sampled uniformly between [0,1].
Revised: For all the comparison baselines, the mixing ratio λ is sampled uniformly between [0,1].

Original: All models are trained using mini-batched (128 examples) backprop, as speciﬁed in (Zhang et al., 2018) and with the exact settings as in the Facebook codes, for 400 epochs.
Revised: All models are trained using mini-batched (128 examples) backprop, as speciﬁed in (Zhang et al., 2018) and with the exact settings as in the Facebook codes, for 400 epochs.

Original: 3.1. P REDICTIVE A CCURACY
Revised: 3.1. P REDICTIVE A CCURACY

Original: and Cifar10 (red bars).
Revised: Cifar10 (red bars).

Original: The predictive error rates obtained by the Vanilla,
Revised: The predictive error rates obtained by the Vanilla,

Original: Mixup, CutMix, and LaMix with PreAct ResNet18 and ResNet-50 as baselines are in Table 1.
Revised: Mixup, CutMix, and LaMix with PreAct ResNet18 and ResNet-50 as baselines are in Table 1.

Original: Table 1 shows that LaMix performed on par with or outperformed the comparison models on all the ﬁve datasets.
Revised: Table 1 shows that LaMix performed on par with or outperformed the comparison models on all the ﬁve datasets.

Original: Results also indicate that the performance of CutMix and Mixup was sensitive to the datasets used: CutMix outperformed Mixup on some cases, but loss on some other datasets.
Revised: Results also indicate that the performance of CutMix and Mixup was sensitive to the datasets used: CutMix outperformed Mixup on some cases, but loss on some other datasets.

Original: This may due to the noisy mixed samples these two methods generated during training.
Revised: This may due to the noisy mixed samples these two methods generated during training.

Original: For example, on Cifar and MNIST with PreAct ResNet-18 or ResNet-50,
Revised: For example, on Cifar and MNIST with PreAct ResNet-18 or ResNet-50,

Original: CutMix obtained lower accuracy than Mixup.
Revised: CutMix obtained lower accuracy than Mixup.

Original: But on Cifar100 and SVHN with PreAct ResNet-18, CutMix outperformed Mixup with a large margin.
Revised: But on Cifar100 and SVHN with PreAct ResNet-18, CutMix outperformed Mixup with a large margin.

Original: In contrast, LaMix performed consistently better than CutMix and Mixup on all the cases.
Revised: In contrast, LaMix performed consistently better than CutMix and Mixup on all the cases.

Original: These results suggest that, although using the same mixed image for a wide range of mixing ratios,
Revised: These results suggest that, although using the same mixed image for a wide range of mixing ratios,

Original: LaMix is able to assign a set of appropriate soft labels to the same image, so that they can effectively regularize the training to achieve regularization effect as effective as that of random input mixing methods, which rely on mixing different features from a given image pair to form a set of images.
Revised: LaMix is able to assign a set of appropriate soft labels to the same image, so that they can effectively regularize the training to achieve regularization effect as effective as that of random input mixing methods, which rely on mixing different features from a given image pair to form a set of images.

Original: 3.1. E VOLVING S OFT T RAINING T ARGETS
Revised: 3.1. E VOLVING S OFT T RAINING T ARGETS

Original: In Figure 2, we visualize the top ﬁve largest training targets used by
Revised: In Figure 2, we visualize the top ﬁve largest training targets used by

Original: LaMix.
Revised: LaMix.

Original: We provide the average values of the top targets during training for both Cifar100 (left) and Cifar10 (right).
Revised: We provide the average values of the top targets during training for both Cifar100 (left) and Cifar10 (right).

Original: Results in Figure 2 indicate that
Revised: Results in Figure 2 indicate that

Original: LaMix re-distributes a large portion of the onehot distribution to the other classes.
Revised: LaMix re-distributes a large portion of the onehot distribution to the other classes.

Original: For both
Revised: For both

Original: Cifar100 and Cifar10, half of the one-hot distribution mass is re-assigned to the other targets.
Revised: Cifar100 and Cifar10, half of the one-hot distribution mass is re-assigned to the other targets.

Original: For example, as shown in Figure 2, the second largest targets for the Cifar100 and Cifar10 are 0.14 and 0.25, respectively.
Revised: For example, as shown in Figure 2, the second largest targets for the Cifar100 and Cifar10 are 0.14 and 0.25, respectively.

Original: In Figure 3, we also visualize the assigned soft labels of a random mini-batch of training samples in LaMix with PreAct ResNet-18 on Cifar10 across the 400 epochs of training.
Revised: In Figure 3, we also visualize the assigned soft labels of a random mini-batch of training samples in LaMix with PreAct ResNet-18 on Cifar10 across the 400 epochs of training.

Original: The left depicts the soft targets used for training by LaMix, and the right shows the learned global soft labels.
Revised: The left depicts the soft targets used for training by LaMix, and the right shows the learned global soft labels.

Original: In these ﬁgures, the x-axis depicts the soft target values and the y-axis is the distribution mass over different training epochs (from top to bottom).
Revised: In these ﬁgures, the x-axis depicts the soft target values and the y-axis is the distribution mass over different training epochs (from top to bottom).

Original: Results in the left subﬁgure of Figure 3 indicate that the soft target labels assigned to the same batch of training samples keep changing during the training over the 400 epochs (from top to bottom), with the largest soft target training signals laying between 0.55 and 0.65.
Revised: Results in the left subﬁgure of Figure 3 indicate that the soft target labels assigned to the same batch of training samples keep changing during the training over the 400 epochs (from top to bottom), with the largest soft target training signals laying between 0.55 and 0.65.

Original: The right subﬁgure suggests that the learned global targets for the batch of training samples tend to have small values, containing a large group with values between 0.14 to 0.18, during the early phases of training (top) and then having more values laying between 0.18 and 0.20 as the training progresses (bottom of the ﬁgure).
Revised: The right subﬁgure suggests that the learned global targets for the batch of training samples tend to have small values, containing a large group with values between 0.14 to 0.18, during the early phases of training (top) and then having more values laying between 0.18 and 0.20 as the training progresses (bottom of the ﬁgure).

Original: These results imply that for the same mixed image, the LaMix is able to assign it with different soft targets, thus increasing the size of the training set but bypassing the efforts needed for input mixing.
Revised: These results imply that for the same mixed image, the LaMix is able to assign it with different soft targets, thus increasing the size of the training set but bypassing the efforts needed for input mixing.

Original: 3.1. R E - WEIGHT THE L OCAL AND G LOBAL S OFT L ABELS
Revised: 3.1. R E - WEIGHT THE L OCAL AND G LOBAL S OFT L ABELS

Original: We here evaluate the impact of different weight factors for β as discussed in Equation 10 for the two soft labels, by varying it from 0.1 to 0.9.
Revised: We here evaluate the impact of different weight factors for β as discussed in Equation 10 for the two soft labels, by varying it from 0.1 to 0.9.

Original: The results obtained by LaMix using PreAct ResNeton Cifar100 are in Figure 4.
Revised: The results obtained by LaMix using PreAct ResNeton Cifar100 are in Figure 4.

Original: The results in Figure 4 suggest that averaging the two sets of soft labels provides better accuracy than other weighting ratios for this dataset.
Revised: The results in Figure 4 suggest that averaging the two sets of soft labels provides better accuracy than other weighting ratios for this dataset.

Original: Also, the
Revised: Also, the

Original: LaMix obtained similar accuracy when the β fell into the region between [0.3, 0.8].
Revised: LaMix obtained similar accuracy when the β fell into the region between [0.3, 0.8].

Original: [Table] [Table]
Revised: [Table] [Table]

Original: In Figure 5, we plot the training loss and validation accuracy across an extended training period, namely 2k training epochs, of LaMix, Mixup, and CutMix with PreAct ResNet-18 on Cifar100.
Revised: In Figure 5, we plot the training loss and validation accuracy across an extended training period, namely 2k training epochs, of LaMix, Mixup, and CutMix with PreAct ResNet-18 on Cifar100.

Original: Figure 5 shows that the training loss of LaMix (thin red curve) maintains a relatively higher level than the other two methods, allowing the model to keep tuning the networks.
Revised: Figure 5 shows that the training loss of LaMix (thin red curve) maintains a relatively higher level than the other two methods, allowing the model to keep tuning the networks.

Original: Promisingly, as shown by the accuracy curve (wide red), even training for a long time, LaMix is not overﬁtting the training set.
Revised: Promisingly, as shown by the accuracy curve (wide red), even training for a long time, LaMix is not overﬁtting the training set.

Original: 3.1. P ERFORMANCE ON S HALLOWER , D EEPER , AND W IDER N ETWORKS
Revised: 3.1. P ERFORMANCE ON S HALLOWER , D EEPER , AND W IDER N ETWORKS

Original: We also test LaMix using different network architectures on Cifar100, including a shallow network LeNet (Lecun et al., 1998), a wider network
Revised: We also test LaMix using different network architectures on Cifar100, including a shallow network LeNet (Lecun et al., 1998), a wider network

Original: WideResNet-28-10 (Zagoruyko & Komodakis, 2016), and the 121 layers deep network DenseNet121 (Huang et al., 2017).
Revised: WideResNet-28-10 (Zagoruyko & Komodakis, 2016), and the 121 layers deep network DenseNet121 (Huang et al., 2017).

Original: For these networks, we use the implementation from Facebook 3 .
Revised: For these networks, we use the implementation from Facebook 3 .

Original: Similar to that of using PreAct ResNet-18 and ResNet-50, LaMix performed consistently on par with or superior to the comparison models on these three additional network architectures.
Revised: Similar to that of using PreAct ResNet-18 and ResNet-50, LaMix performed consistently on par with or superior to the comparison models on these three additional network architectures.

Original: 3.1. C OMPARISON WITH L ABEL S MOOTHING T ECHNIQUE
Revised: 3.1. C OMPARISON WITH L ABEL S MOOTHING T ECHNIQUE

Original: We also compare our method LaMix with label smoothing techniques as presented in (Müller et al., 2019; Pereyra et al., 2017; Szegedy et al.
Revised: We also compare our method LaMix with label smoothing techniques as presented in (Müller et al., 2019; Pereyra et al., 2017; Szegedy et al.

Original: These techniques train models with soft targets that are a weighted average of the hard targets and the uniform distribution over labels with a smoothing factor, aiming at preventing the network from becoming over-conﬁdent.
Revised: These techniques train models with soft targets that are a weighted average of the hard targets and the uniform distribution over labels with a smoothing factor, aiming at preventing the network from becoming over-conﬁdent.

Original: We varies the smoothing factor from 0.1 to 0.9 on
Revised: We varies the smoothing factor from 0.1 to 0.9 on

Original: PreAct ResNet-18 for Cifar100.
Revised: PreAct ResNet-18 for Cifar100.

Original: Accuracy results are presented in Figure 6.
Revised: Accuracy results are presented in Figure 6.

Original: Results in Figure 6 show that label smoothing can improve the predictive accuracy of PreAct ResNet-18 if the smoothing factor being tuned properly, but even the best accuracy obtained is still far below that achieved by LaMix.
Revised: Results in Figure 6 show that label smoothing can improve the predictive accuracy of PreAct ResNet-18 if the smoothing factor being tuned properly, but even the best accuracy obtained is still far below that achieved by LaMix.

Original: For example, the best accuracy 78.47% obtained by PreAct ResNet-18 is with label smoothing factor of 0.2, which is still below the 80.62% achieved by LaMix.
Revised: For example, the best accuracy 78.47% obtained by PreAct ResNet-18 is with label smoothing factor of 0.2, which is still below the 80.62% achieved by LaMix.

Original: 3.1. C LASS A CTIVATION M APPING (CAM) V ISUALIZATION
Revised: 3.1. C LASS A CTIVATION M APPING (CAM) V ISUALIZATION

Original: After the 400 training epochs, we, in Figure 7, depict the CAM maps (Zhou et al., 2016) of two classes ( bird and frog ) using testing samples from Cifar10.
Revised: After the 400 training epochs, we, in Figure 7, depict the CAM maps (Zhou et al., 2016) of two classes ( bird and frog ) using testing samples from Cifar10.

Original: The test images are on the left, and the visualization CAMs created by LaMix, Mixup, and CutMix are depicted on the second left, second right, and the right, respectively.
Revised: The test images are on the left, and the visualization CAMs created by LaMix, Mixup, and CutMix are depicted on the second left, second right, and the right, respectively.

Original: One can see from Figure 7 that, LaMix tends to focus on narrower discriminative regions of the images than Mixup and CutMix.
Revised: One can see from Figure 7 that, LaMix tends to focus on narrower discriminative regions of the images than Mixup and CutMix.

Original: 3.1. R EGULARIZATION E FFECT OF THE S YNTHETIC S AMPLES
Revised: 3.1. R EGULARIZATION E FFECT OF THE S YNTHETIC S AMPLES

Original: We also visualize the impact of the synthetic samples created by LaMix.
Revised: We also visualize the impact of the synthetic samples created by LaMix.

Original: We present the results of three classes (i.e., Cat, Dog, and Deer) in Cifar10 with a 2D bottleneck hidden representation in the middle of the network.
Revised: We present the results of three classes (i.e., Cat, Dog, and Deer) in Cifar10 with a 2D bottleneck hidden representation in the middle of the network.

Original: In speciﬁc, following the 512 ﬁlters generated by the PreAct ResNet- (namely the layer before the Softmax layer), we use a fully-connected Tanh layer of 2 units as a bottleneck layer, followed by two fully-connected Tanh layers with 100 and 512 units respectively.
Revised: In speciﬁc, following the 512 ﬁlters generated by the PreAct ResNet- (namely the layer before the Softmax layer), we use a fully-connected Tanh layer of 2 units as a bottleneck layer, followed by two fully-connected Tanh layers with 100 and 512 units respectively.

Original: We visualize the 2D bottleneck representation for the original input samples at the 100 th , 400 th , 700 th , and 1k th training epoch.
Revised: We visualize the 2D bottleneck representation for the original input samples at the 100 th , 400 th , 700 th , and 1k th training epoch.

Original: For the last three, the training errors were 100%.
Revised: For the last three, the training errors were 100%.

Original: Figure 8 depicts the results, with different colors for the three classes.
Revised: Figure 8 depicts the results, with different colors for the three classes.

Original: Results in Figure 8 show that, as the learning progresses, the embeddings for the training samples are tuned gradually towards classes and model
Revised: Results in Figure 8 show that, as the learning progresses, the embeddings for the training samples are tuned gradually towards classes and model

Original: These results suggest that the synthetic samples help LaMix create tight representation cluster for training samples in and widen the gap between these clusters.
Revised: These results suggest that the synthetic samples help LaMix create tight representation cluster for training samples in and widen the gap between these clusters.

Original: These turnings happened long after the training accuracy on the original training set was not able to further increase.
Revised: These turnings happened long after the training accuracy on the original training set was not able to further increase.

Original: 3. C OMPARE WITH L EARNING TO M IXING M ETHOD
Revised: 3. C OMPARE WITH L EARNING TO M IXING M ETHOD

Original: In this section, we compared LaMix with the state-of-the-art learning to mixing variant of Mixup, namely the Puzzle Mix (Kim et al., 2020).
Revised: In this section, we compared LaMix with the state-of-the-art learning to mixing variant of Mixup, namely the PuzzleMix (Kim et al., 2020).

Original: We use the authors’ code published at 4 .
Revised: We use the authors’ code published at 4 .

Original: We trained the networks for 600 epochs as implemented in the provided source codes
Revised: We trained the networks for 600 epochs as implemented in the provided source codes

Original: [Table]
Revised: [Table]

Original: and kept all the settings in the codes.
Revised: and kept all the settings in the codes.

Original: Each reported error is the mean of ﬁve runs, on a NVIDIA GTX TitanX
Revised: Each reported error is the mean of ﬁve runs, on a NVIDIA GTX TitanX

Original: GPU with 12GB memory.
Revised: GPU with 12GB memory.

Original: In this set of experiments, the β in LaMix was set as 1.0 unless otherwise speciﬁed.
Revised: In this set of experiments, the β in LaMix was set as 1.0 unless otherwise speciﬁed.

Original: [Table]
Revised: [Table]

Original: We report mean scores over  with  (denoted 
Revised: We report mean scores over 5 runs with standard deviations (denoted ± ).

Original: Results on the ﬁve testing datasets as used to evaluate the random input mixing methods in Section 3.
Revised: Results on the ﬁve testing datasets as used to evaluate the random input mixing methods in Section 3.

Original: are presented in Tables 3.
Revised: are presented in Tables 3.

Original: Tables 3 shows that LaMix performed on par with the Puzzle Mix in terms of accuracy obtained.
Revised: Tables 3 shows that LaMix performed on par with the PuzzleMix in terms of accuracy obtained.

Original: As shown in the table, LaMix obtained very similar accuracy on all the testing cases as the Puzzle Mix method, except for the ResNet-50 on the Cifar10 dataset, which Puzzle Mix slightly outperformed LaMix with 0.25%.
Revised: As shown in the table, LaMix obtained very similar accuracy on all the testing cases as the PuzzleMix method, except for the ResNet-50 on the Cifar10 dataset, which PuzzleMix slightly outperformed LaMix with 0.25%.

Original: These results suggest that mixing the input features with a range of mixing ratios λ s in Puzzle Mix can be bypassed without degrading the predictive accuracy
Revised: These results suggest that mixing the input features with a range of mixing ratios λ s in PuzzleMix can be bypassed without degrading the predictive accuracy.

Original: 3.2. E XPERIMENTS ON T INY -I MAGE N ET
Revised: 3.2. E XPERIMENTS ON T INY -I MAGE N ET

Original: [Table]
Revised: [Table]

Original: The authors’ codes for Puzzle Mix also include settings for running Puzzle Mix on Tiny-ImageNet (Chrabaszcz et al., 2017), so here we also evaluated LaMix using this dataset.
Revised: The authors’ codes for PuzzleMix also include settings for running PuzzleMix on Tiny-ImageNet (Chrabaszcz et al., 2017), so here we also evaluated LaMix using this dataset.

Original: This dataset hasclasses, each with 500 training andtest 64x64x3 images.
Revised: This dataset hasclasses, each with 500 training andtest 64x64x3 images.

Original: We presented the results in Table 4.
Revised: We presented the results in Table 4.

Original: Results in Table 4 show that LaMix obtained similar error rates when compared with Puzzle Mix on this dataset.
Revised: Results in Table 4 show that LaMix obtained similar error rates when compared with PuzzleMix on this dataset.

Original: For both testing network architectures, namely PreAct ResNetand ResNet-50, the accuracy difference between LaMix and Puzzle Mix is marginal.
Revised: For both testing network architectures, namely PreAct ResNetand ResNet-50, the accuracy difference between LaMix and PuzzleMix is marginal.

Original: These results further indicate that the input mixing effort with a range of mixing ratios λ s in the learning to mixing variant Puzzle Mix can be bypassed to conduct effective model regularization.
Revised: These results further indicate that the input mixing effort with a range of mixing ratios λ s in the learning to mixing variant PuzzleMix can be bypassed to conduct effective model regularization.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Various variants of the Mixup (Zhang et al., 2018) method have been introduced, devoting to creating a set of mixed inputs by either randomly mixing the inputs (Guo, 2020; Guo et al., 2019; Tokozume et al., 2018a;b; Verma et al., 2019; Yun et al., 2019; Zhang et al., 2018) or learning to mixing the informative features from an input pair (Dabouei et al., 2020;
Revised: Various variants of the Mixup (Zhang et al., 2018) method have been introduced, devoting to creating a set of mixed inputs by either randomly mixing the inputs (Guo, 2020; Guo et al., 2019; Tokozume et al., 2018a;b; Verma et al., 2019; Yun et al., 2019; Zhang et al., 2018) or learning to mixing the informative features from an input pair (Dabouei et al., 2020;

Original: Kim et al., 2020; Li et al., 2020a;b;
Revised: Kim et al., 2020; Li et al., 2020a;b;

Original: Walawalkar et al., 2020).
Revised: Walawalkar et al., 2020).

Original: In the former category, Mixup (Zhang et al., 2018) mixes all the input features with a random mixing ratio.
Revised: In the former category, Mixup (Zhang et al., 2018) mixes all the input features with a random mixing ratio.

Original: CutMix (Yun et al., 2019) creates a mixed input by copying a box of pixels from one image and pasted it to the other image.
Revised: CutMix (Yun et al., 2019) creates a mixed input by copying a box of pixels from one image and pasted it to the other image.

Original: The second family of approaches aim to learn to mixing important input features (Dabouei et al., 2020; Kim et al., 2020; Li et al., 2020a;b; Walawalkar et al., 2020).
Revised: The second family of approaches aim to learn to mixing important input features (Dabouei et al., 2020; Kim et al., 2020; Li et al., 2020a;b; Walawalkar et al., 2020).

Original: For example, the recent method Puzzle Mix (Kim et al., 2020) leverages the optimal transport to combine the identiﬁed salient features from the inputs.
Revised: For example, the recent method PuzzleMix (Kim et al., 2020) leverages the optimal transport to combine the identiﬁed salient features from the inputs.

Original: There is also a method called Manifold Mixup (Verma et al., 2019), which conducts feature interpolation in addition to mixing the input pair.
Revised: There is also a method called Manifold Mixup (Verma et al., 2019), which conducts feature interpolation in addition to mixing the input pair.

Original: This approach performs feature mixing on a randomly selected layer of the networks.
Revised: This approach performs feature mixing on a randomly selected layer of the networks.

Original: Unlike these methods which generate a set of mixed inputs from a sample pair, our strategy creates only one mixed input by averaging the features from a given sample pair.
Revised: Unlike these methods which generate a set of mixed inputs from a sample pair, our strategy creates only one mixed input by averaging the features from a given sample pair.

Original: Our method also relates to the label smoothing techniques, which regularize the output distribution of neural models to penalize high-conﬁdence softmax distributions (Müller et al., 2019;
Revised: Our method also relates to the label smoothing techniques, which regularize the output distribution of neural models to penalize high-conﬁdence softmax distributions (Müller et al., 2019;

Original: Pereyra et al., 2017; Szegedy et al., 2016).
Revised: Pereyra et al., 2017; Szegedy et al., 2016).

Original: These techniques train models with soft targets that are a weighted average of the hard targets and the uniform distribution over labels with a smoothing factor.
Revised: These techniques train models with soft targets that are a weighted average of the hard targets and the uniform distribution over labels with a smoothing factor.

Original: These smoothing methods, however, do not relate the smooth labels with the associated features but utilize uniform label smoothing.
Revised: These smoothing methods, however, do not relate the smooth labels with the associated features but utilize uniform label smoothing.

Original: Our approach learns to form soft targets based on the features of the input pair and the states of the evolving networks.
Revised: Our approach learns to form soft targets based on the features of the input pair and the states of the evolving networks.

Original: The generation of the global soft labels in LaMix is also related to how the training targets generated by self-distillation methods (Ahn et al., 2019; Furlanello et al., 2018; Hinton et al., 2015; Mobahi et al., 2020;
Revised: The generation of the global soft labels in LaMix is also related to how the training targets generated by self-distillation methods (Ahn et al., 2019; Furlanello et al., 2018; Hinton et al., 2015; Mobahi et al., 2020;

Original: Yang et al., 2019).
Revised: Yang et al., 2019).

Original: However, self-distillation treats the ﬁnal predictions as target labels for a new round of training, and the teacher and student architectures are identical (Mobahi et al., 2020).
Revised: However, self-distillation treats the ﬁnal predictions as target labels for a new round of training, and the teacher and student architectures are identical (Mobahi et al., 2020).

Original: In LaMix, the classiﬁer network and the network for the global soft label have different architectures, and the training targets for the classiﬁcation model are augmented by the global soft labels.
Revised: In LaMix, the classiﬁer network and the network for the global soft label have different architectures, and the training targets for the classiﬁcation model are augmented by the global soft labels.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We contributed a ﬁnding that one can bypass the effort needed for input mixing with a range of mixing ratios in Mixup-based approaches to conduct effective pairwise, label-variant model regularization.
Revised: We contributed a ﬁnding that one can bypass the effort needed for input mixing with a range of mixing ratios in Mixup-based approaches to conduct effective pairwise, label-variant model regularization.

Original: Consequently, one can mitigate creating noisy synthetic samples from the given inputs or reduce the computation cost required for selecting salient regions from the inputs.
Revised: Consequently, one can mitigate creating noisy synthetic samples from the given inputs or reduce the computation cost required for selecting salient regions from the inputs.

Original: Our observation may shed light on how one may conduct label-variant data augmentation in domains that are expensive to perform input mixing with a range of mixing ratios.
Revised: Our observation may shed light on how one may conduct label-variant data augmentation in domains that are expensive to perform input mixing with a range of mixing ratios.

Revised:  A DDITIONAL E XPERIMENTS 6. M IXUP PLUS LABEL SMOOTHING

Revised: We conducted experiments of stacking uniform label smoothing (with coefﬁcients of 0.1, 0.2, 0.

Revised: and 0.4) on top of Mixup, using PreAct ResNet-18 and ResNet-50 on Tiny ImageNet, Cifar100 and

Revised: Cifar10.

Revised: We presented the results in Table

Revised: [Table]

Revised: We report mean scores over 5 runs with standard deviations (denoted ± ).

Revised: These results show that label smoothing degraded the accuracy of vanilla Mixup in the cases of

Revised: Cifar100, Cifar

Revised: [Table]

Revised: and Tiny ImageNet.

Revised: Such accuracy degradation is due to the two layers of label smoothing in Mixup.

Revised: Research has shown that the mixing label in the original Mixup has the effect of label smoothing (Carratino et al., 2020).

Revised: In this sense, adding another label smoothing regularizer on top of the vanilla Mixup means that the Mixup will have two layers of label smoothing regularizer, which may mess up the regularization effect, thus degrading the accuracy as shown in the results above.

Revised: 6. R ANDOM I NPUT M IXING IN T INY I MAGE N ET

Revised: We conducted additional experiments of random input mixing on Tiny ImageNet, and present the results in Table 6.

Revised: Similar to that in Table 1, results here again suggest that, although using the same mixed image for a wide range of mixing ratios, LaMix is able to assign a set of appropriate soft labels to the same image, so that they can effectively regularize the training to achieve regularization effect as effective as that of random input mixing methods, which rely on mixing different features from a given image pair to form a set of images

Revised: [Table]

Revised: 6. W ITHOUT ORIGINAL TRAINING SAMPLES

Revised: Without the original samples

Revised: [Table]

Revised: the LaMix method signiﬁcantly degraded its accuracy.

Revised: We conducted the experiments and provided the results in Table 7.

Revised: [Table]

Revised: The reason here is that, without the original training samples, the training samples could be very different from the testing samples.

Revised: Also, it is interesting to see that ResNet50 obtained inferior results than PreAct ResNet18 in some cases when removing the original training samples.

Revised: This indeed suggests that Resnet50 ﬁts the mixed training samples better due to its expressive power but those mixed training samples may be far from the testing samples due to the lack of the original training samples.

Revised: 6. W ITHOUT GLOBAL SOFT LABEL

Revised: Without the global soft label, namely p ij , the LaMix model signiﬁcantly degraded the accuracy.

Revised: We conducted the experiments and provided the results in Table

Revised: [Table]