M IX RL: D ATA M IXING A UGMENTATION FOR R EGRESSION USING R EINFORCEMENT L EARNING Anonymous authors
Paper under double-blind review
A BSTRACT
Data augmentation is becoming essential for improving regression accuracy in critical applications including manufacturing and ﬁnance.
Existing techniques for data augmentation largely focus on classiﬁcation tasks and do not readily apply to regression tasks.
In particular, the recent Mixup techniques for classiﬁcation rely on the key assumption that linearity holds among training examples, which is reasonable if the label space is discrete, but has limitations when the label space is continuous as in regression.
We show that mixing examples that either have a large data or label distance may have an increasingly-negative effect on model performance.
Hence, we use the stricter assumption that linearity only holds within certain data or label distances for regression where the degree may vary by each example.
We then propose MixRL, a data augmentation meta learning framework for regression that learns for each example how many nearest neighbors it should be mixed with for the best model performance using a small validation set.
MixRL achieves these objectives using Monte Carlo policy gradient reinforcement learning.
Our experiments conducted both on synthetic and real datasets show that MixRL signiﬁcantly outperforms state-of-the-art data augmentation baselines.
MixRL can also be integrated with other classiﬁcation Mixup techniques for better results
I NTRODUCTION
As machine learning (ML) becomes widely used in critical applications including manufacturing and ﬁnance, data augmentation for regression becomes essential as it provides an opportunity to improve model performance without additional data collection.
In comparison to classiﬁcation tasks like object detection in images, the goal of regression is to predict one or more real numbers.
To emphasize the importance of data augmentation in regression, we provide a case study of semiconductor manufacturing.
Here a common quality check is to measure the layer thicknesses of a 3-dimensional semiconductor and see if they are even.
However, directly measuring each thickness results in destroying the semiconductor itself, so a recently-common approach is to take an indirect measurement by applying light waves on the semiconductor, measuring the spectrum of wavelengths that bounce back from all the layers, and use ML to predict the layer thicknesses from the spectrum data (see Fig.
4 in Sec.
4 for an illustration).
With enough spectrum data and thickness information, ML models can be trained to accurately predict thicknesses from a spectrum.
The main challenge is that there is not enough training data, and the only cost-effective solution is to augment small amounts of data that exist.
Even a small improvement in model performance from the data augmentation has signiﬁcant impact in this industry.
In general, any regression task that predicts real values like emissions, stock prices, or even someone’s salary can also beneﬁt from data augmentation.
Most data augmentation techniques are designed for image classiﬁcation.
In particular,
Mixup (Zhang et al., 2018; Berthelot et al., 2019; Yun et al., 2019) is a popular data augmentation technique that is widely used for classiﬁcation tasks, but is seldom used for regression because it assumes a distinct label space.
The idea of Mixup is to mix pairs of examples using the key assumption that taking a linear interpolation between examples can be used to estimate the label of any examples in between.
Mixup is known to effectively regularize the model.
More recently, Manifold
Mixup (Verma et al., 2019) has been proposed to improve the hidden representation and decision boundaries where two examples are mixed in multiple hidden layers of neural networks. [Figure]
However the label continuous data labels
Fig a result the linearity assumption only holds to a certain extent degree may for example for including or
and et & Welling even (see
We propose MixRL, a data mixing augmentation framework that is the ﬁrst to tailor Mixup for regression tasks using reinforcement learning.
MixRL uses a stricter linearity assumption where it only holds within a certain data or label distance.
These distance limits may vary by example, and we formulate the problem of learning for each example how many nearest neighbors it should be mixed with.
MixRL employs a meta learning framework that estimates how valuable mixing an example is for reducing the model loss on a small validation set using Monte Carlo policy gradient reinforcement learning.
MixRL’s framework is inspired by the recent Data Valuation using Reinforcement Learning (DVRL) framework (Yoon et al., 2020), which solves the different problem of measuring how individual examples contribute to model performance without any mixing involved.
Fig.
1b shows how limiting the nearest neighbors to mix is better than mixing with all neighbors as in classiﬁcation.
To see if the augmentation is useful, we train simple models on the original four examples (i.e., no augmentation), the augmented data in Fig. 1a, and the augmented data in Fig.
1b. Evaluating the models on 20 random test examples results in Root Mean Square Error (RMSE; see Sec. 4) values of 0.2967, 0.5615, and 0.1834, respectively, where a lower RMSE is better. We thus conclude that carefully mixing examples is important for improving regression performance.
Experiments conducted on real and synthetic datasets show that MixRL shows better model performances relative to baselines, especially when the linearity is limited, and the mixing must be done selectively.
In addition, MixRL only requires small validation sets and scales to large training sets.
L IMITED L INEARITY IN D ATA AND L ABEL S PACE FOR R EGRESSION
We explain why the key linearity assumption used for Mixup in classiﬁcation has limitations in a regression setting.
In classiﬁcation, the labels are discrete where many examples may have the same label.
The original version of Mixup (Zhang et al., 2018) is to take a linear interpolation between any pair of examples x
i and x j with the labels y i and y j to produce the new example λ x i +(1 − λ ) x j with the label λ
y i + (1 − λ )
y j where λ ∼ Beta ( α, α ) .
The linearity assumption turns out to be reasonable because the label difference between examples is only 0 or 1 and thus not that sensitive to the data difference.
In contrast, the labels in regression are in a continuous space.
Although there is still a many-to-one relationship where multiple examples may have the same label, the degree is much smaller than in classiﬁcation.
As a result, when two examples are mixed, the interpolated label can be arbitrarily different than the actual label, e.g., mixing the points a and d in Fig. 1a results in a [Figure]
a [Figure] e l D i s t a n c e (c) label nowhere near the actual label.
In Sec. 4.1, we also show empirical results where the label error increases for larger data or label distances.
Furthermore, mixing examples with larger data or label distances tend to have increasingly-negative effects on the model trained on the augmented training set.
Figs.
2a and 2b show the model accuracies using RMSE for the Product dataset (described in Sec. 4) when mixing examples with different ranges of distances.
Regardless of adjusting the label or data distance, there are diminishing beneﬁts for larger distances.
How do we limit the data and label distances to improve Mixup for regression?
One approach is to only limit the data distance, which limits the label distance as well.
Suppose that the regression function f is continuous where lim x → c f ( x ) =
f ( c ) for any x and c within the domain of f .
We show that a short-enough data distance sufﬁciently reduces the label distance as well.
Given f ’s domain
D and lim x
→ c f ( x ) =
L , the following is known to hold: ∀ (cid:15), ∃ δ s.t.
∀ x ∈ D , if | x − c | < δ , then | f ( x ) − L | < (cid:15) .
We can use this result to prove that ∀ (cid:15), ∃ δ s.t.
∀ x i , x j ∈ D , if α x i + (1 − α )
x j = c, 0 ≤ α ≤ 1 , and | x i − x j | <
δ then the absolute difference between the mixed example’s x j value
and L is small where | αf ( x i )
+ (1 − α )
f ( x j ) − L | = | α ( f ( x i ) − L )
+ (1 − α )
( f ( x j ) − L ) | ≤ α | f ( x i ) − L | + (1 − α )
| f
( x j ) − L | < α(cid:15)
+ (1 − α ) (cid:15) =
(cid:15) .
We cannot do the converse and limit the data distance by limiting label distance because there is a many-to-one mapping from data to labels, which means that two different examples may have identical labels (e.g., Fig.
1a’s a and d ).
However, limiting the data distance to also limit the label distance may be too restrictive because there is not much correlation between the data and label distances in real data.
Fig.
2c shows how the data distance relates to the label distance for the Product dataset.
Even for small data distances, the label distance has a large range, which means that the data distance would have to be extremely limited.
Hence, our solution is to limit the data and/or label distance as needed instead of just the data distance.
This approach turns out to be more practical as we demonstrate in Sec. 4.4.
M IX RL
The goal of MixRL is to identify which examples to mix with which nearest neighbors.
Instead of ﬁnding the actual distance limits themselves, we solve the identical problem of ﬁnding the number of nearest neighbors to mix per example for convenience.
We use reinforcement learning because there is no training data on how mixing each example affects the model performance, and the reward function is non-differentiable as we explain below.
MixRL’s framework is inspired by DVRL (Yoon et al., 2020), but we solve the different problem of mixing examples and address new issues.
3. P OLICY O PTIMIZATION R EINFORCEMENT L EARNING
Finding the optimal policy involves taking the gradient of the objective function J ( θ
[Equation] where π is the policy, θ is its parameters, and R is the reward function.
For MixRL, we would like to minimize the regression model loss on a small validation set when mixing a batch of examples.
However, the validation loss is computed using the regression model, which does not involve θ .
Hence we cannot analytically compute the differential of the reward function with respect to θ .
[Figure] Instead the gradient algorithm a Monte Carlo method that is also widely used for data searching (Zoph & Le, 2017).
It is known that ∇ θ J ( θ ) can be approximated (see Sec. A.1) as:
[Equation]
where τ i is the i th state-action trajectory under policy π θ , m is the number of sample trajectories, T is the number of actions taken in a path, s it is a state at time t , and a it is an action at time t .
In our setting of minimizing loss, REINFORCE performs gradient descent for each example decreasing θ by α ∇ θ J ( θ ) where α is a learning rate.
Although the estimated gradient is unbiased, it is known to have a high variance, which we reduce using baseline techniques (Sutton & Barto, 2018).
3. F RAMEWORK
We deﬁne notations used in MixRL’s framework shown in Fig.
Let D =
( x i , y i ) }
Si =1 ∼ P be the training set where x i ∈
X is a d -dimensional input example, and y i ∈ Y is an e -dimensional label.
Let D v = { ( x vi , y vi ) } Vi =1 ∼ P t be the validation set, where P t is the distribution of the test set, which is not necessarily the same as P .
Let f φ be a regression model, and L the loss function that returns a performance score comparing f φ ( x i ) with the true label y i using Mean Square Error (MSE).
We assume a list N of possible data and label nearest neighbors (NNs) that can be mixed with an example.
For instance, N could contain the options “1 data NN”, “2 data NNs”, and “ label NNs”.
The more ﬁne-grained the NN options are, the more precisely MixRL can determine the optimal number of NNs to mix per example.
We do not add a “0 NN” option because selecting and not selecting it have identical effects, which makes the policy network training unstable because nothing can be learned.
Instead, we support this option separately by excluding examples that are not worth mixing as we explain later in this section.
The possible NNs can be represented as a one-hot encoding vector of | N | values.
We now deﬁne the states and actions, which is an important design choice of MixRL.
A state s t is a batch of examples D b = { (( x i , y i ) , k i ) }
Bi
=1 where { ( x i , y i ) }
Bi =1 ⊆ D and each k
i is an index of an
NN option in N that speciﬁes how many data or label NNs x
i should be mixed with.
An action a t is then choosing D m ⊆
D b where each (x i , y i ) in D m is mixed with its N [ k i ] NNs in D .
A policy π θ ( D m |D b ) returns the probability of selecting D m at state D b .
For each episode, MixRL selects a batch (state) once and chooses a subset of the batch to obtain a reward (action) once.
Since there is only one time step, the transition function does not play a role.
A na¨ıve implementation of the policy network would have an input dimension of B × ( d + e + | N | ) and an output dimension of 2 B , which may be too large to train for large batch sizes.
For example, in a typical setting of B = 1000 , d = 100 , e = 1 , and | N | = 10 , the input and output dimensions become 111 , 000 and 2 1000 , respectively.
Instead, we can signiﬁcantly reduce the network size by assuming independence among examples and decomposing the policy network’s prediction into two phases.
First, a Mixup value network h θ ( x , y , k ) is used to estimate the probability of an example being mixed with k NNs.
Next, we randomly select which examples to mix according to the probabilities.
The value of π θ ( D m |D b ) is thus (cid:81) ( x , y , k ) ∈D m h
θ ( x , y , k ) (cid:81) ( x , y , k ) ∈D
b \D m [1 − h θ ( x , y , k )] .
Also, h θ has input and output dimensions of only d + e + | N | and 1, respectively, and is practical.
The objective function J ( θ ) is the validation set loss l θ of f φ trained on D b ∪
Mix ( D m , D , N ) (either from scratch or from a pre-trained state) where Mix ( D m , D , N ) mixes each (x i , y i ) in D m with its N [ k i ] NNs in D :
[Equation]
Using Eq. 1, the gradient of Eq.
2 can be approximated using the validation set D v as:
[Equation]
where ∇ θ log π θ ( D m , D b ) = ∇ φ (cid:80) ( x , y , k ) ∈D m log h θ ( x , y , k )
+ (cid:80) ( x , y , k ) ∈D
b \D m [1 − h θ ( x , y , k )] .
We also use reward scaling (Henderson et al., 2018) to further improve the training.
Algorithm 1 shows the pseudo code for training h θ .
The computational complexity is not directly related to the training set size, but depends on the number of iterations needed to train h θ and how long each iteration takes.
Another factor is the number of possible NNs | N | where a higher number may result in slower training.
We show in Sec.
4.2 that MixRL scales to large training sets.
Once we train h θ , we choose the ( x , y , k ) ’s with the highest h θ ( x , y , k ) values of at least a threshold
T .
We optimize T ’s value using the validation set (see Sec. B.1).
We ignore any ( x , y , k ) whose ( x , y ) pair has already been chosen with a different k value for the same data or label distance.
We apply T in order to exclude ( x , y , k ) ’s that are not worth mixing (i.e., make them mix with 0 NNs).
Although actor-critic methods (Mnih et al., 2016; Schulman et al., 2017) improve on REINFORCE by training both value and policy networks, using the two networks may not be practical in our setting due to the high input dimensions.
For a policy network, we can reduce its dimension using the fact that the log probabilities can be calculated by multiplying selection probabilities.
Unfortunately, we cannot use the same trick for a value network because its output values are not probabilities.
E XPERIMENTS
We provide experimental results for MixRL.
We evaluate the regression models trained on the augmented training sets on separate test sets and repeat all model trainings 5 times.
We use PyTorch (Paszke et al., 2017), and all experiments are performed using Intel Xeon Silver CPUs and
NVidia RTX GPUs.
More experimental setting details are in Sec. B.1. [Figure] [Figure]
[Table]
[Figure]
Measures We use two accuracy metrics.
RMSE =
(cid:113) 1 n (cid:80) ni
=
( y i − ˆ y i ) 2 is the absolute difference between predicted labels and true labels where a lower value is better.
R 2 = 1 − (cid:80) ni
=
( y i − ˆ y i ) 2 (cid:80) ni
=
( y i − ¯ y ) 2 is the relative improvement compared to when returning the label average as a prediction.
The value is in the range [0, 1] where a higher value is better.
The two measures complement each other.
Datasets We use three real and one synthetic datasets.
The NO2 emissions dataset (Aldrin, 2004) contains trafﬁc and meteorological information around roads.
The features include cars per hour, wind speed, temperature, and others.
The label is the NO2 concentration.
The Product dataset obtained through a collaboration with a company contains spectrum data that is generated using the procedure described in Sec. 1 and illustrated in Fig.
4 on 20-layer 3-d semiconductors.
The Product dataset is proprietary, so for clarity we also experiment on a public synthetic dataset ( Synthetic ) provided by the DACON challenge (DACON Co., Ltd., 2020) where a simulator generates spectrum data using the same procedure as Product assuming 4-layer 3-d semiconductors.
Finally, the Airfoil UCI dataset (Dua & Graff, 2017) contains aerodynamic and acoustic test results for airfoil blade sections in a wind tunnel where the features include frequency, angle of attack, and chord length.
Table 1 compares the four datasets in more detail.
Note that Product and Synthetic have high dimensions, while Airfoil has the largest training set size.
MixRL Settings For the regression model, we use a multi-layer perceptron (MLP) with 2 tohidden layers and set the number of nodes per hidden layer to be [512, 256] for NO2 , [2048, 1024, 512, 256] for Product , and [2048, 1024, 512, 256] for Synthetic .
We employ layer normalization, early stopping for regularization, and reward scaling and baseline techniques that improve the performance and stability of reinforcement learning.
We use the Adam optimizer for all MLP trainings.
When mixing examples, we set the Mixup ratio λ to 0.5 unless speciﬁed otherwise.
This value empirically results in the best Mixup performance in a regression setting, and we give an explanation in Sec.
B.3. For the possible numbers of NNs for NO2 , our default setting is the series of integers { [-i } 16 i =
 = { 1 , 2 , ...-]4 , 16 , 64 } for both data and label distances, i.e., | N | = 6 .
For Product , Synthetic , and Airfoil , the default series [-is { 10 i } 27 i =
 =-]are { 100 , 150 , 200 , 250 } , { 4 , 16 , 64 } , and { 1 , 2 , 4 } , respectively.
Baselines We employ four baselines.
First, we train a regression model on the labeled data without any data augmentation (“No Augmentation”).
Next, we faithfully implement the original Mixup algorithm (Zhang et al., 2018) where all example pairs can be mixed (“Original Mixup”).
We also implement Manifold Mixup (Verma et al., 2019), which is a state-of-the-art Mixup technique for classiﬁcation that is also applicable to regression.
Finally, we compare with a simpliﬁed version of MixRL where we use a single k NN value to mix all examples with their data and label neighbors (“Global kNN”).
This k parameter is tuned using Bayesian Optimization (Mockus, 1974).
4. L IMITED L INEARITY IN R EGRESSION
We empirically show that the linearity assumption does not hold from certain data or label distances using the following simple experiment.
Given a dataset, we ﬁrst train a regression model. We then [Figure] [Table]
Manifold Mixup
[Table]
[Table] Global kNN ( [Table] =64)
[Table]
MixRL
[Table]
MixRL + Manifold Mixup
[Table]
Product
No Augmentation 1 .
[Table]
Original
[Table]
[Table]
Manifold
[Table]
[Table]
Global kNN (
[Table]
=267) 1 .
[Table]
MixRL
[Table]
[Table]
MixRL + Manifold Mixup 1 .
[Table]
Airfoil
No Augmentation 1 .
[Table]
[Table]
Original Mixup 2 .
[Table]
[Table]
Manifold Mixup 1 .
[Table]
[Table]
Global kNN (k=1) 1 .
[Table]
[Table]
MixRL 1 .
[Table]
[Table]
MixRL + Manifold Mixup 1.
[Table]
[Table]
generate mixed examples and compute the RMSE between their interpolated labels and the labels predicted by the regression model.
While the regression model is not perfect, the relative trend of the error gives sufﬁcient insight.
Figs.
5a and 5b indeed show increasing label errors for larger label and data distances for the NO2 and Product datasets, respectively.
Figs. 2c and 5c also show how label and data distances do not necessarily have a linear relationship for the two datasets.
We also train regression models on top of augmented training sets where we only mix examples within certain distance ranges.
As a result, mixing examples with larger data or label distances has increasinglynegative effects on the model accuracy as shown in Figs.
2a, 2b, 5d, and 5e.
4. A CCURACY AND R UNTIME R ESULTS
We compare MixRL with the four baselines using the settings in Table 1.
Table 2 shows the results for the three real datasets (the Synthetic dataset results are in Sec.
B.2). MixRL signiﬁcantly outperforms all the baselines in terms of regression model accuracy, which demonstrates that it is able to pinpoint which examples should be mixed with how many kNNs.
Figs.
6a and 6b show which kNN options are frequently used for NO2 and Product (the kNN options for Synthetic and Airfoil are in Sec. B.2).
Although the runtime is slower than the baselines due to the reinforcement learning, the gained accuracy can be worth the effort in real applications because data augmentation is a one-time cost that can run in batch mode.
Another observation is that Original Mixup performs relatively bet-
[Figure]
[Table]
ter for Product than other datasets because linearity holds more as evidenced by the large k =value of Global kNN. In general, MixRL outperforms Original Mixup by a larger margin the more limited the linearity.
We also integrate MixRL with Manifold Mixup where examples are mixed in multiple layers of the MLP with their NNs (see Sec. B.4 for more details).
The results are mixed where the integrated method performs the best on the Airfoil dataset, but not on the other datasets.
Validation Set and Scalability Figs.
6c and 6d show the validation set size impact on MixRL’s accuracy for the NO2 and Product datasets.
The Airfoil dataset results are similar and shown in
Sec. B.2. We observe that a small validation set is sufﬁcient for MixRL to be effective.
Also MixRL scales well against increasing training set sizes as shown in Fig.
6e.
4. K N EAREST N EIGHBOR O PTIONS
We investigate how changing the kNN options N affects MixRL’s performance in Table 3 using the
NO2 and Product datasets. The other dataset results are similar and shown in Sec. B.2.
For each dataset, we compare its default series with shorter and longer series.
We observe that providing more kNN options improves MixRL’s performance, but only to a certain extent.
Hence, MixRL performs well even with a small number of options.
4. A BLATION S TUDY
We perform an ablation study to investigate the effect of limiting label and data distances in Tableusing the NO2 and Product datasets.
The other dataset results are similar and shown in Sec.
B.2. We compare MixRL with three variants: (1) MixRL that does not exploit any distances and selects examples that have the highest h θ values when fully mixed with all the other examples; (2) MixRL that does not use label distances; and (3) MixRL that does not use data distances.
We observe that using both distances is necessary for the best accuracy results as we suggested in Sec. 2.
R ELATED W ORK
Data Augmentation for Regression There are largely two branches of work for data augmentation in regression.
One is semi-supervised regression (Kostopoulos et al., 2018;
Zhou & Li, 2005;
Kang et al., 2016;
Kim et al., 2020) where the goal is to utilize unlabeled data for training.
In comparison, MixRL does not assume the availability of unlabeled data.
Another branch is data augmentation when there is no unlabeled data, which is our research focus.
To the best of our knowledge, there is no technique tailored to regression, so we instead cover data augmentation techniques for classiﬁcation that can be extended to regression.
[Table]
We explore data augmentation techniques for classiﬁcation and investigate if they can be used in a regression setting.
There are largely three approaches: generative models, policies [Table] and Mixup techniques.
Generative models including GANs (Goodfellow et al., 2014) and VAEs (Kingma &
Welling, 2014) are popular in classiﬁcation where the idea is to generate realistic data that cannot be distinguished from the real data by a discriminator.
However, a major assumption is that the labels of the generated examples are the same, which does not necessarily hold in a regression setting where most examples may have different labels.
Another approach is to use policies (Cubuk et al., 2019), which specify ﬁxing rules for transforming the data while maintaining the label value.
For example, image processing policies may ﬂip, rotate, or adjust the brightness of images to generate new valid images.
However, in a regression setting, the same transformed examples may now have different unknown labels, making them unsuitable for training.
The recently-proposed Mixup (Zhang et al., 2018; Berthelot et al., 2019; Yun et al., 2019; Berthelot et al., 2020; Hendrycks et al., 2020; Verma et al., 2019) takes the alternative approach of generating both data and labels together by mixing existing examples with different labels assuming linearity between training examples (Chapelle et al., 2000; Wu et al., 2020).
Although the Mixup paper mentions that its techniques can easily be extended to regression, we make the non-trivial observation that the linearity assumption does not necessarily hold where mixing examples with far-away examples may not be beneﬁcial and even detrimental to model performance.
Instead, MixRL shows good performance by carefully choosing which nearest neighbors to mix with, a technique that only works in a regression setting.
Policy Optimization Reinforcement Learning Within reinforcement learning (RL) (Sutton &
Barto, 2018; Kaelbling et al., 1996), we are interested in policy searching methods (Neumann &
Peters, 2015) where our goal is to ﬁnd the best policy for mixing examples.
In particular, MixRL’s framework is inspired by the recent Data Valuation using Reinforcement Learning (DVRL) framework (Yoon et al., 2020), which uses REINFORCE (Williams, 1992) to estimate the value of data in terms of how much it improves the model performance.
In comparison, we solve the completely different problem of data augmentation for regression where we extend Mixup using REINFORCE.
Some data augmentation techniques use actor-critic methods like A3C (Mnih et al., 2016) and PPO (Schulman et al., 2017) that train value networks along with policy networks.
For example, AutoAugment (Cubuk et al., 2019) uses PPO to search for data augmentation policies that dictate how to modify existing examples to generate additional training data for image classiﬁcation.
However, using actor-critic methods is practically challenging in our setting due to the large value networks that need to be trained as we discussed in Sec. 3.2.
C ONCLUSION
We proposed MixRL, which is to our knowledge the ﬁrst Mixup-based data augmentation meta learning framework tailored to regression tasks.
We observe that linearity only holds up to a certain data or label distance per example in a regression setting.
Hence, MixRL extends Mixup by determining the k nearest neighbors to mix per example using Monte Carlo policy gradient reinforcement learning where the objective is to minimize the regression model’s loss on a small validation set.
Since MixRL only limits the examples to mix, it can be integrated with any existing Mixup technique for classiﬁcation.
In our experiments, we showed that MixRL outperforms various data augmentation baselines for regression by effectively selecting the k nearest neighbor options.
Ethics Statement MixRL demonstrates that data augmentation can signiﬁcantly enhance the model accuracy for regression tasks.
However, if not controlled, data augmentation may also introduce new bias in the data that makes the model more discriminative.
Thus an interesting future work is to augment data while ensuring model fairness.
Reproducibility Statement We add MixRL’s source code in the supplementary.
We also provide more details on Eq.
1, experimental settings, and experimental results in the appendix.
A A PPENDIX – F RAMEWORK A. G RADIENT OF J ( θ )
We continue from Sec.
3.1 and derive Eq.
1 (Brunskill, 2021):
[Equation]
where τ i is a state-action trajectory under policy π θ , m is the number of sample trajectories, T is the number of actions taken in a path, s t is a state at time t [Equation] and a t is an action at time t .
Suppose that P ( τ ; θ ) is the probability of a trajectory τ when using the policy π θ , and R ( τ ) is the reward of τ .
Then J ( θ ) can be expressed as (cid:80) τ P ( τ ; θ ) R ( τ ) .
In addition, let us specify s 0 as the initial state, µ ( s 0 ) as the initial state’s distribution [Equation] and P ( s t +1 | s t , a t ) as the dynamics model that speciﬁes which predicts the next state s t +1 when taking action a t from state s t .
We can then take the gradient of J ( θ ) as follows:
[Equation]
(cid:88) i =
R ( τ i ) ∇ θ log (cid:34) µ ( s 0 )
T − 1 (cid:89) t = π θ ( a it | s it ) P ( s it +1 | s it , a it ) (cid:35)
=m m (cid:88) i =
R ( τ i ) ∇ θ (cid:34) log µ ( s i 0 )
T − 1 (cid:88) t = log π θ ( a it | s it )
+ log P
( s t +1 | s it , a it ) (cid:35)
=m m (cid:88) i =
R ( τ i )
T − 1 (cid:88) t = ∇ θ log π θ ( a it | s it )
B A PPENDIX – E XPERIMENTS B. M ORE E XPERIMENTAL S ETTINGS
We continue describing our experimental settings from Sec. 4.
For the regression model, we use layer normalization (Ba et al., 2016) for the Product , Synthetic , and Airfoil datasets.
We also use early stopping to prevent overﬁtting on the validation set for all the datasets.
For the mixup value network h θ , we use reward scaling and baseline techniques that improve the performance and stability of reinforcement learning.
For the mixup value network h θ , we use a multi-layer perceptron with 4 hidden layers and set the number of nodes per hidden layer to be [100, 100, 100, 100] for the four datasets.
For h θ , we use a learning rate of α = 0 .
001 for all datasets. For the regression model f φ , we use a learning rate of 0 . 0001 for Product and 0 . 001 for the other datasets.
When training h θ , we set the batch size B to be the training data’s size.
When training f φ , we use a batch size offor NO2 , 64 for Product , 32 for Synthetic , and 64 for Airfoil .
We use a reward scaling (Henderson et al., 2018) constant of C = 5 for NO2 , C = 1 for Product , C = 0 .
1 for Synthetic , and C = 1 for Airfoil .
We use a moving average window of size W = 20 for the three datasets.
When setting the threshold T for limiting examples to mix, we choose a value that results in a high model accuracy (low RMSE) on the validation set.
When constructing the Synthetic dataset, we take a subset of the entire DACON challenge dataset (DACON Co., Ltd., 2020).
While the the range of thicknesses for the full datasets is [10, 300] for all four layers, we only use examples with thicknesses within the range [10, 50].
The purpose is to reduce the training time while still making a clear comparison between MixRL and the other methods.
To improve the runtime of MixRL, we use pre-trained regression models for the Product , Synthetic , and Airfoil datasets.
First, we train the base regression model on the training data.
Then, we ﬁne-tune the pre-trained model to obtain the validation losses.
B. M ORE Synthetic AND Airfoil D ATASET E XPERIMENTS
Accuracy and Runtime Results We continue from Sec.
and show the accuracy and runtime results for the Synthetic dataset in Table 5.
The results are similar to the Product dataset in Table 2.
[Table]
Mixup [Table] [Figure] kNN Options Frequency Histogram
We continue from Sec.
4.2 and show which kNN options are frequently used in our default settings for the Synthetic and Airfoil datasets in Figs.
7a and 7b, respectively.
Compared to the Product dataset, Synthetic has no cases where k = 0 because mixing is more beneﬁcial to the model’s accuracy according to the validation set. [Figure]
Validation Set Size and Accuracy
We continue from Sec.
and show how the validation set size impacts MixRL’s accuracy for the Airfoil dataset in Fig. 8.
The observations are similar to those of the other datasets where a small validation set size is sufﬁcient for MixRL to perform well.
kNN Options Comparison We continue from Sec.
4.3 and provide the kNN options experiments for the Synthetic and Airfoil datasets in Table 6.
The observations are similar to those of Tablewhere a small number of options is sufﬁcient for good model accuracy. [Figure]
[Table]
Ablation Study We continue from Sec.
4.4 and provide the ablation study for the Synthetic and Airfoil datasets in Table 7.
The results are similar to those in Table 4 where using both label and data distances results in the best model accuracies.
[Table]
B. J USTIFICATION FOR U SING λ = 0 .
We continue from Sec. 4 and provide more details on why setting λ = 0 .
5 empirically results in the best Mixup performance in a regression setting.
Figure 9 shows MixRL’s accuracy when we vary λ using the same setting in Table 1 for the NO2 and Product datasets.
As a result, the accuracy peaks when λ = 0 .
We suspect that generating examples in the sparsest areas helps the regularization the most.
In a classiﬁcation setting, however, this strategy may not be ideal especially if the generated examples are too close to a decision boundary that is in the middle of the mixed examples.
B. I NTEGRATING M IX RL WITH M ANIFOLD M IX U P
We continue from Sec.
and provide more details on integrating MixRL with Manifold Mixup (Verma et al., 2019).
We ﬁrst run MixRL and determine which examples to mix. Next, instead of mixing examples in the training data before model training, we mix examples on multiple layers in the MLP during model training.
For the second mixing step, we use Manifold Mixup as is
[Figure]
where we start with a set of eligible layers for mixing.
(In our experiments, we always set the ﬁrst three layers as eligible.)
For each training batch, we randomly select a mixing layer and mix the features and labels of the examples that MixRL decided to mix.