mixReg: A Simple Way to Improve Generalization in Regression for Deep Neural Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Improving the generalization of deep networks is an important open challenge,particularly in domains without plentiful data.
The mixup algorithm improvesgeneralization by linearly interpolating the input features of a pair of examplesand their corresponding labels.
These interpolated examples augment the originaltraining dataset.
It has shown promising results in various classification tasks,but systematic analysis of mixup in regression remains underexplored.
Usingmixup directly on regression labels could result in arbitrarily wrong labels sincethe linearity assumption behind mixup may not hold.
In this paper, we propose asimple yet powerful algorithm, mixReg, to improve generalization on regressiontasks.
In contrast with vanilla mixup, which uniformly samples mixing pairs fromthe training set, mixReg adjusts the sampling probability based on the similarityof labels.
Our theoretical analysis further confirms that mixReg with label similarity obtains a smaller mean square error than vanilla mixup and using featuresimilarity.
Another benefit of mixReg is that it can improve out-of-distributionrobustness, where the test distribution is different from the training distribution.
Byselectively interpolating examples with similar labels, it mitigates the effects ofdomain-associated information and pushes invariant representations.
We evaluatemixReg on eleven datasets, ranging from tabular to video data.
Compared to othermethods, mixReg achieves 6.56%, 4.76%, 5.14% improvements in in-distributiongeneralization, task generalization, and out-of-distribution robustness, respectively.
Introduction
Training deep neural networks faces the challenge of mitigating overfitting and improving generalization.
In order to accomplish this, a number of techniques, including data augmentation [2, 8, 10, 71, 72] and explicit regularization [12, 33, 53] have been proposed.
Representatively,mixup [72] densifies the data distribution and regularizes the model by linearly interpolating thefeatures between randomly sampled pairs of original examples and applying the same interpolationon the corresponding labels.
Despite mixup having demonstrated promising results in improvinggeneralization in classification problems, it has rarely been studied in the context of regression withcontinuous responses, on which we focus in this paper.
In contrast to classification, the goal of regression is to predict a continuous response from each input.
Directly applying mixup to input features and labels to regression tasks may yield arbitrarily incorrectlabels.
For example, as shown in Figure 1(a), ShapeNet1D pose prediction [15] aims to predict thecurrent orientation of the object relative to its canonical orientation.
We randomly select three mixingpairs and show the mixed images and labels in Figure 1(b), where only pair 1 exhibits reasonablemixing results.
Thus, sampling mixing pairs uniformly from the dataset introduces a number of noisypairs.
Alternatively, Manifold Mixup [60] proposes to linearly interpolate hidden representationsrather than input features.
Although it is possible to improve the original mixup in regression to some
[Figure]
𝑦 !
[Figure]
∘ 𝑦 # : 150 ∘ 𝑦 # : 150 ∘ 𝑦 $ : 346 ∘ )𝑦 : 160.5 ∘ )𝑦 : 248 ∘
Pair 2 (P2) 𝑦 $ : 346 ∘ 𝑦 ! : 171 ∘ )𝑦 : 258.5 ∘
Pair 3 (P3)
[Figure]
[Figure] [Figure]
[Figure] [Figure]
😄 ☹ 😐
[Figure]
extent, empirical evidence suggests that it cannot achieve consistent improvements over a wide rangeof scenarios [24].
Instead of applying mixup or Manifold Mixup to all examples, MixRL [24] selectsthe mixing pairs in regression by using a well-trained policy network.
The training of a complexpolicy network, however, requires considerable computing resources, which are not suitable forhigh-dimensional real-world data.
Instead of training an external policy network, we aim to directly adjust the sampling probability ofmixing pairs according to the similarity of examples, resulting in a simple training technique named asmixReg ( mix up for Reg ression).
Specifically, we employ a Gaussian kernel to calculate the samplingprobability of drawing another example for mixing, where closer examples are more likely to besampled.
Here, the core question is: how to measure the similarity between two examples?
The moststraightforward solution is to compute input feature similarity.
Yet, using input similarity has twomajor downsides when dealing with high-dimensional data such as images or time-series: substantialcomputational costs and lack of good distance metrics.
Specifically, first, it takes considerable timeto compute pairwise similarities across all samples.
Second, directly applying classical distancemetrics (e.g., Euclidean distance, cosine distance) does not reflect the high-level relation betweeninput features.
In the ShapeNet1D rotation prediction example (Figure 1(a)), pair 1 and 3 have closeinput similarities, which is not expected since only pair 1 shows the reasonable scenario after mixing.
To overcome these drawbacks, mixReg instead uses label similarity, which is easy to computeand reduces the computation cost since the label space is usually low dimensional.
In addition tothe obvious computational advantages, mixReg benefits three kinds of regression problems.
First ,it empirically improves in-distribution generalization in supervised learning compared with thevanilla mixup and using input feature similarity.
Second , we extend mixReg to gradient-basedmeta-learning and particularly incorporated it with MetaMix, a mixup-based task augmentationmethod [65].
Compared with vanilla MetaMix, mixReg further improves task generalization; Third ,mixReg is well-suited for improving out-of-distribution robustness, particularly to covariate shift(see the corresponding example of ShapeNet1D rotation prediction in Appendix A.1).
Concretely,we aim to learn a regression model that is robust to the covariate shift between training and testdistributions.
By performing mixup on examples with close continuous labels, mixReg encouragesthe regression model to rely on the invariant features to make prediction and ignore unrelated orspurious correlations, making the model more robust to covariate shift.
The primary contribution of this paper is: (1) We systematically investigate how to adapt mixupin regression, leading to a simple and scalable algorithm – mixReg; (2) We theoretically analyzethe advantages of mixReg.
Compared with vanilla mixup and using feature similarity, mixReg hasthe lower mean squared error and improves in-distribution generalization, task generalization androbustness to covariate shift; (3) Empirically, we conduct comprehensive experiments to evaluatemixReg on eleven datasets, including many large-scale real-world applications like drug-targetinteraction prediction [23], ejection fraction estimation with annotated echocardiogram videos [44],poverty estimation with satellite imagery [68].
Compared to the best baseline, the results demonstratethe promise of mixReg with 6.56%, 4.76%, 5.14% improvements in in-distribution generalization,task generalization, and out-of-distribution robustness, respectively.
Preliminaries
In this section, we define notation and describe the background of ERM and mixup in the supervisedlearning setting, and MetaMix in the meta-learning setting for task generalization.
ERM.
Assume a machine learning model f with parameter space Θ .
In this paper, we consider thesetting where one predicts the continuous label y ∈ Y according to the input feature x ∈ X .
Given aloss function ℓ , we train a model f θ under the empirical training distribution P tr with the followingobjective, and get the optimized parameter θ ∗ ∈ Θ :
[Equation]
Typically, we expect the model to perform well on unseen examples drawn from the test distribution
P ts .
We are interested in both in-distribution ( P tr = P ts ) and out-of-distribution ( P tr ̸ = P ts ) settings.
Mixup.
The mixup algorithm samples a pair of instances ( x i , y i ) and ( x j , y j ) , sampled uniformly atrandom from the training dataset, and generates new examples by performing linear interpolation onthe input features and corresponding labels as:
[Equation]
where the interpolation ratio λ ∈ [0 , 1] is drawn from a Beta distribution, i.e., λ ∼ Beta( α, α ) .
Theinterpolated examples are then used to optimize the model as follows:
[Equation]
Task Generalization and MetaMix.
In this paper, we also investigate few-shot task generalizationunder the gradient-based meta-regression setting.
Given a task distribution p ( T ) , we assume each task
T m is sampled from p ( T ) and is associated with a dataset D m .
A support set D sm =
{ ( X sm , Y sm ) }
={ ( x sm,i , y sm,i ) } N s i =1 and a query set D qm =
{ ( X sm , Y sm ) }
= { ( x qm,j , y qm,j ) } N q
i
=1 are sampled from D m .
Representatively, in model-agnostic meta-learning (MAML) [11], given a predictive model f withparameter θ , it aims to learn an initialization θ ∗ from meta-training tasks {T m } | M | m =1 .
Specifically,at the meta-training phase, MAML obtained the task-specific parameter ϕ m for each task T m byperforming a few gradient steps starting from θ .
Then, the corresponding query set D qm is used toevaluate the performance of the task-specific model and optimize the model initialization as:
[Equation]
ϕ m = θ −
[Equation]
∇
[Equation]
L
[Equation]
D sm )
[Equation]
At the meta-testing phase, for each meta-testing task T t , MAML fine-tunes the learned initializationθ ∗ on the support set D st and evaluates the performance on the corresponding query set D qt .
To improve task generalization, MetaMix [65] adapts mixup (Eqn. (3)) to meta-learning, whichlinearly interpolates the support set and query set and uses the interpolated set to replace the originalquery set D qm in Eqn.
Specifically, the interpolated query set is formulated as:
[Equation]
where λ ∼ Beta( α, α ) .
Mixup for Regression (mixReg)
This section first introduces the main idea of mixReg.
Then, we theoretically explain how mixRegbenefits in-distribution generalization, task generalization, and out-of-distribution robustness.
3.1 mixReg
For continuous labels, the example in Figure 1(b) illustrates that applying vanilla mixup to theentire distribution is likely to produce arbitrary labels.
To resolve this issue, mixReg proposes tosample closer pairs of examples with higher probability.
Specifically, given an example ( x i , y i ) ,mixReg introduces a symmetric Gaussian kernel in order to calculate the sampling probability
P (( x j , y j ) | ( x i , y i )) for another ( x j , y j ) example to be mixed as follows:
[Equation]
6) where d ( i, j ) represents the distance between the examples ( x i , y i ) and ( x j , y j ) , and σ describes thebandwidth.
In the case of example ( x i , y i ) , the set { P (( x j , y j ) | ( x i , y i ))
|∀ j } is then normalized to aprobability mass function that sum up to one.
Require:
Learning rates η ; Shape parameter α
Require:
Training data D :
= { ( x i , y i ) }
Ni = 1: Randomly initialize model parameters θ 2: Calculate pairwise distance matrix P via
Eqn.
3: while not converge do 4:
Sample a batch of examples B ∼ D 5: for each example ( x i , y i )
∈ B do 6:
Sample example ( x j , y j ) according to the probability P ( · | ( x i , y i )) 7:
Linearly interpolate ( x i , y i ) and ( x j , y j ) to get ( ˜ x, ˜ y ) 8:
Use interpolated examples to update the model via Eqn.
The most straightforward way to compute the distance is using the input feature x , i.e., d ( i, j )
=
d ( x i , x j ) .
However, when dealing with the highdimensional data such as images or videos, welack good distance metrics to capture structuredfeature information and the distances can be easilyinfluenced by feature noise.
Additionally, computing feature distances for high-dimensional datais time-consuming.
Instead, mixReg leveragesthe labels with d ( i, j ) =
d ( y i , y j ) =
∥
y i − y j ∥ 22 ,where y i and y j are vectors with continuous values.
The dimension of label is typically much smallerthan that of the input feature, therefore reducingcomputational costs (see more discussions aboutcompuatational efficiency in Appendix A.3).
Theoverall algorithm of mixReg is described in Alg.
1and we detail the difference between mixReg andmixup in Appendix A.4.
According to Alg. 1, given an example ( x i , y i ), all examples conceptuallyhave probabilities to be sampled as ( x j , y j ) in mixReg, but label closer examples have higher probabilities.
In addition to its computational benefits, mixReg improves generalization on three distinctkinds of regression problems, which is theoretically shown in the following sections.
3.2 mixReg for Improving In-distribution Generalization
In this section, we show that mixReg provably improves in-distribution generalization when thefeatures are observed with noise, and the response depends on a small fraction of the features in amonotonic way.
Specifically, we consider the following single index model with measurement error,
[Equation]
) where θ ∈ R p and ϵ is a sub-Gaussian random variable and g is a monotonic transformation.
Sinceimages are often inaccurately observed in practice, we assume the feature z is observed or measuredwith noise, and denote the observed value by x : x = z
+ ξ with ξ being a random vector withmean 0 and covariance matrix σ 2 ξ I .
We assume g to be monotonic to model the nearly one-to-onecorrespondence between causal features (e.g., the car pose in Figure 1(a)) and labels (rotation) in thein-distribution setting.
The out-of-distribution setting will be discussed in Section 3.4.
We would liketo also comment that the single index model has been commonly used in econometrics, statistics, anddeep learning theory [16, 22, 41, 47, 64].
Suppose we have { ( x i , y i ) } Ni =1 i.i.d.
drawn from the above model.
We first follow the single indexmodel literature (e.g. [64]) and estimate θ by minimizing the square error (cid:80) ni
=
(˜ y i − ˜ x
⊤ i θ ) 2 , wherethe (˜ x i , ˜ y i ) ′
s are the augmented data by either vanilla mixup, mixup with input feature similarity,and mixReg.
We denote the solution by θ ∗ mixup , θ ∗ feat , and θ ∗ mixReg respectively.
Given an estimateθ ∗ , we estimate g by ˆ g via the standard nonparametric kernel estimator [56] (we specify this in detailin Appendix B.1 for completeness) using the augmented data.
We consider the mean square errormetric as MSE( θ ) =
E [( y − ˆ g ( θ ⊤ x )) 2 ] , and then have the following theorem (proof: Appendix B.1):
Theorem 1.
Suppose θ ∈ R p is sparse with sparsity s =
o (min { p, σ 2 ξ } ) , p =
o ( N ) and g is smoothwith c 0 < g ′ <
c 1 , c 2 < g ′′ <
c 3 for some universal constants c 0 , c 1 , c 2 , c 3 > 0 .
There exists adistribution on x with a kernel function, such that when the sample size N is sufficiently large, withprobability 1 − o (1) ,
[Equation] [Equation]
The high-level intuition of why mixReg helps is that the vanilla mixup imposes linearity regularizationon the relationship between the feature and response.
When the relationship is strongly nonlinear andone-to-one, such a regularization hurts the generalization, but could be mitigated by mixReg.
3.3 mixReg for Improving Task Generalization
The second benefit of mixReg is improving task generalization in meta-learning when the data fromeach task follows the model discussed in the last section.
Concretely, we apply mixReg to MetaMix[65].
For each query example, the support example with a more similar label will have a higher probability of being mixed.
The algorithm of mixReg on MetaMix is summarized in Appendix A.2.
Similar to in-distribution generalization analysis, we consider the following data generative model:for the m -th task ( m ∈ [ M ] ), we have ( x ( m ) , y ( m ) ) ∼ T m withy ( m ) =
g m ( θ ⊤ z ( m ) )
+ ϵ and x ( m )
=
z ( m ) + ϵ ( m ) .
Here, θ denotes the globally-shared representation, and g m ’s are the task-specific transformations.
Note that, the formulation is close to [48] and wildly applied to theoretical analysis of metalearning [65, 67].
Following similar spirit of [54] and last section, we obtain the estimation of θ byθ ∗ =
1 M (cid:80) Mm =
( E (˜ x ( m ) , ˜ y ( m ) )
∈
ˆ D m [˜ y ( m ) − θ ⊤ ˜ x ( m ) ]) .
Here, ˆ D m denotes the generic dataset augmentedby different approaches, including the vanilla MetaMix, MetaMix with input feature similarity, andmixReg.
We denote these approaches by θ ∗ MetaMix , θ ∗ Meta − feat , and θ ∗ Meta − mixReg respectively.
For anew task T t , we again use the standard nonparametric kernel estimator to estimate g t via the augmentedtarget data.
We then consider the following error metric MSE Target ( θ ∗ )
=
E ( x,y ) ∼T t
[( y − ˆ g t ( θ ∗⊤ x )) 2 ] .
Based on this metric, we get the following theorem to show the promise of mixReg in improving taskgeneralization (see Appendix B.2 for detailed proof).
Here, mixReg achieves smaller MSE Targetcompared to vanilla MetaMix and MetaMix with input feature similarity.
Theorem 2.
Let N = (cid:80)
Mm =1 N m
and N m is the number of examples of T m .
Suppose θ k is sparsewith sparsity s =
o (min { d, σ 2 ξ } ) , p = o ( N ) and g m ’s are smooth with 0 <
g ′ m < c 1 , c 2 < g ′′ m < c 3for some universal constants c 1 , c 2 , c 3 > 0 and m ∈
[ M ] ∪ { t } .
There exists a distribution on x witha kernel function, such that when the sample size N is sufficiently large, with probability 1 − o (1) ,
[Equation]
[Equation]
3.4 mixReg for Improving Out-of-distribution Robustness
Finally, we show that mixReg improves OOD robustness in the covariate shift setting where the someunrelated features vary across different domains.
In this setting, we regard the entire data distributionconsisting of E = { 1 , . . .
, E } domains, where each domain is associated with a data distribution P efor e ∈ E .
Given a set of training domains E tr ⊆ E , we aim to make the trained model generalizewell to an unseen test domain E ts that is not necessarily in E tr .
Here, we focus on covariate shift, i.e.,the change of P e among domains is only caused by the change of marginal distribution P e ( X ) , whilethe conditional distribution P e ( Y | X ) is fixed across different domains.
To overcome covariate shift, mixing examples with close labels without considering domain information can effectively average out domain-changeable correlations and make the predicted valuesrely on the invariant causal features.
To further understand how mixReg improves the robustness tocovariate shift, we provide the following theoretical analysis.
We assume the training data ( x i , y i ) ni
=1 follows x i =
( z i ; a
i ) ∈
R p 1 + p
and y i =
θ
⊤ x i + ϵ i ,where z i ∈ R p 1 and a i ∈ R p 2 are regarded as invariant and domain-changeable unrelated features,respectively, and the last p 2 coordinates of θ ∈ R p 1 + p 2 are 0.
Now we consider the case wherethe training data consists of a pair of domains with almost identical invariant features and oppositedomain-changeable features, i.e., x i =
( z i , a i ) , x ′ i =
( z ′ i , a ′ i ) , where z i ∼ N p 1 (0 , σ 2 x I p 1 ) , z
′ i
=
z i + ϵ
i ,a i ∼ N p 2 (0 , σ 2 a I p 2 ) , a ′ i = − a
i + ϵ ′′ i .
ϵ
i , ϵ ′
i , ϵ i ′′ are noise terms with mean 0 and sub-Gaussian normbounded by σ ϵ .
We use ridge estimator θ ∗ ( k ) =
arg min θ ( (cid:80) i ∥
y i − θ ⊤ x i ∥ 2 + k ∥ θ ∥ 2 ) to reflect theimplicit regularization effect of deep neural networks [42].
The covariant shift happens when the testdomains have different a i distributions compared with training domains.
We then have the followingtheorem to show that mixReg can improve robustness to covariant shift (proof: Appendix B.3):
Theorem 3.
Supposed for some max(exp( − n 1 − o (1) ) , exp( − p 21 2 n )) <
δ ≪ 1 , we have variance constraints:
σ a = c 1 σ x , σ x ≥ c 2 max( n 5 / 2 ∥ θ ∥ δ σ ϵ , √ p 2 ∥ θ ∥ √ np 1 ) and σ 2 ϵ ≤ c 3 pn 3 / 2 .
Then for any penalty k satisfiesc 4 (cid:113) p 2 p 1 n 1 / 4+ o (1) <
k < c 5 min( σ x ∥ θ ∥ (cid:112) p 1 n 1 − o (1) , n ) and bandwidth h satisfies 0 < h ≤ cl √ log
[Equation]
n 2 /p 1 )in mixReg, when n is sufficiently large, with probability at least 1 − o (1) , we have
MSE
[Equation]
(11) where c 1 ≥ 1 , c 2 , c 3 , c 4 , c 5 , c 6 >
Experiments
In this section, we evaluate the performance of mixReg, aiming to answer the following questions:
Q1:
Compared to prior approaches, can mixReg improve the in-distribution, task generalization, and
[Table]
out-of-distribution robustness on regression?
Q2:
How does mixReg perform compared to using
[Table]
other distance metrics?
Q3:
Is mixReg sensitive to the choice of bandwidth σ in Eqn.
In ourexperiments, we apply cross-validation to tune all hyperparameters with grid search.
4.1 In-Distribution Generalization
Datasets.
The following five datasets are used to evaluate the performance of in-distribution generalization (see Appendix C.1 for detailed data statistics).
(1)&(2) Airfoil Self-Noise (Airfoil) and
NO2 [30] are both are tabular datasets, where airfoil contains aerodynamic and acoustic test resultsof airfoil blade sections and NO2 aims to predict the mount of air pollution at a particular location.(3)&(4): Exchange-Rate, and Electricity [35] are two time-series datasets, where Exchange-Ratereports the collection of the daily exchange rates and Electricity is used to predict the hourly electricityconsumption.
(5) Echocardiogram Videos (Echo): [44] is a ejection fraction prediction dataset,which consists of a series of videos illustrating the heart from different aspects.
Comparisons and Experimental Setups.
We first compare mixReg with mixup and its variants(Manifold mixup [60], k-Mixup [17] and Local Mixup [4]) that can be easily to adapted to regressiontasks.
Second, MixRL, a recent reinforcement learning framework to select mixup pairs in regression,is also used for comparison.
Note that, for k-Mixup, Local Mixup, MixRL, and mixReg, we applythem to both mixup and Manifold Mixup and report the best-performing combination.
For Airfoil and NO2, we employ a three-layer fully connected network as the backbone model.
Weuse LST-Attn [35] for the Exchange-Rate and Electricity, and EchoNet-Dynamic [44] for predictingthe ejection fraction.
We use Root Mean Square Error (RMSE) and Mean Averaged Percentage Error(MAPE) as evaluation metrics.
Detailed experimental setups are in Appendix C.2.
Results.
We report the results in Table 1 and have the following observations.
First , vanilla mixup andmanifold mixup are typically less performant than ERM when they are applied directly to regressiontasks.
These results support our hypothesis that random selection of example pairs for mixingmay produce arbitrary inaccurate virtual labels.
Second , while limiting the scope of interpolation(e.g.
, k-mixup, manifold k-mixup, Local Mixup) helps to improve generalization in most cases, theperformance of these approaches is inconsistent across different datasets.
As an example, Local
Mixup outperforms ERM in NO2 and Exchange-Rate, but fails to benefit results in Electricity.
Third ,even in more complicated datasets, such as Electricity and Echo, MixRL performs worse than Mixupand ERM, indicating that it is non-trivial to train a policy network that works well.
Finally , mixRegconsistently outperforms mixup and its variants, ERM, and MixRL, demonstrating its capability toimprove in-distribution generalization on regression problems.
[Figure]
Analysis of Overfitting.
In Figure 2, we visualize the test loss (RMSE) and the generalizationgap between training loss and test loss of ERM,
Manifold Mixup, and mixReg with respect tothe training epoch for Exchange-Rate.
Moreresults are illustrated in Appendix C.3.
Compared with ERM and mixReg, the better testperformance and smaller generalization gap ofmixReg demonstrates its ability to improve indistribution generalization and mitigate overfitting.
[Table]
Task Generalization
Dataset and Experimental Setups.
To evaluate task generalization in meta-learning settings, weuse two rotation prediction datasets named as ShapeNet1D [15] and Pascal1D [62, 69], aiming topredict an object’s rotation relative to the canonical orientation.
Each task is rotation regression forone object, where the model takes a 128×128 grey-scale image as the input, and the output is anazimuth angle normalized between [0 , 10] [69].
We detail the description in Appendix D.1.
Since MetaMix outperforms most other methods in PASCAL3D in Yao et al.
[65], we only selecttwo other representative approaches – Meta-Aug [49] and MR-MAML [69] for comparison.
Wefurther apply k-mixup and Local Mixup to MetaMix, which are called as k-MetaMix and Local
MetaMix, respectively.
Following Yin et al.
[69], the base model consists of an encoder with threeconvolutional blocks and a decoder with four convolutional blocks.
All baselines use the same basemodel.
Hyperparameters are listed in Appendix D.2.
[Table]
Results.
Table 2 shows the average MSE with a95% confidence interval over 6000 meta-testingtasks.
Our results corroborate the findingsof Yao et al.
[65] that MetaMix improves performance compared with non-mixup approaches(i.e.
, MAML, MR-MAML, Meta-Aug).
Similar to our previous in-distribution generalization findings, the better performance of Local
MetaMix over MetaMix demonstrates the effectiveness of interpolating nearby examples.
Bymixing query examples with support examplesthat have similar labels, mixReg outperforms allof the other approaches on both datasets, verifying its effectiveness of improving task generalization.
4.3 Out-of-Distribution Robustness
Datasets.
In the following, we briefly discuss the datasets used to evaluate out-of-distributionrobustness; the detailed data descriptions in Appendix E.1.
(1) PovertyMap [29] is a satellite imageregression dataset, aiming to estimate asset wealth in countries that are not shown in the training set.(2) Communities and Crime (Crime) [50] is a tabular dataset, where the problem is to predict totalnumber of violent crimes per 100K population and we aim to generalize the model to unseen states.(3)
SkillCraft1 Master Table (SkillCraft) [5] is a tabular dataset, aiming to predict the mean latencyfrom the onset of a perception action cycles to their first action in milliseconds.
Here, “LeagueIndex"is treated as domain information.
(4) Drug-target Interactions (DTI) [23] is aiming to predictout-of-distribution drug-target interactions in 2019-2020 after training on 2013-2018.
Comparisons and Experimental Setups.
To evaluate the out-of-distribution robustness of mixReg,we compare it with nine invariant learning approaches that can be adapted to regression tasks,including ERM, IRM [3], IB-IRM [1], V-REx [34], CORAL [36], DRNN [13], GroupDRO [51],
Fish [52], and mixup.
All approaches use the same backbone model, where we adopt ResNet-50,three-layer full connected network, and DeepDTA [45] for Poverty, Crime and SkillCraft, and DTI,respectively.
Following the original papers of PovertyMap [68] and DTI [23], we use R -value toevaluate the performance.
For Crime and SkillCraft, we use RMSE as the evaluation metric.
Detailedsetups and hyperparameters are listed in Appendix E.
Results.
We report both average performance and worst-domain performance in Table 3.
Accordingto the results, we can see that the performance of prior invariant learning approaches is not stableacross datasets.
For example, IRM and CORAL outperform ERM on DTI, but fail to improve theperformance on PovertyMap.
mixReg instead consistently shows the best performance regardless ofthe data types, indicating its effectiveness in improving robustness to covariate shift.
[Table]
Analysis of Learned Invariance.
Following Yao et al.
[66], weanalyze the domain invariance of mixReg.
In regression, we measuredomain invariance as pairwise divergence of the last hidden representation.
Since the labels are continuous, we first evenly split examplesinto C = { 1 , . .
. , C } bins, where C = 10 .
After splitting, we performkernel density estimation to estimate the probability density function
P ( h ce ) of the last hidden representation.
We calculate the pairwisedivergence as
[Equation]
e ′ )) .
The results on Crime and SkillCraft are reported in Table 4,indicating that mixReg learns better invariant representation thanprior invariant learning methods.
4.4 Analysis of mixReg
In this section, we conduct three analyses including the compatibility of mixReg, alternative distancemetrics, and sensitivity of bandwidth.
In Appendix F.3.2 and F.4, we analyze the sensitivity ofhyperparameter α in Beta distribution and the robustness of mixReg to label noise, respectively.
[Table]
I. Compatibility of mixReg.
mixReg is a complementary approach to vanilla mixup and its variants,where it changes the probabilities of sampling mixing pairs instead of changing the way to mixing.
Wefurther conduct compatibility analysis of mixReg byintegrating it to three representative mixup variants– PuzzleMix [27], CutMix [71], AutoMix [ ? ].
Weevaluate the performance on three real-world datasets(Exchange-Rate, Echo, PovertyMap).
The reportedresults in Table 5 indicate the compatibility and efficacy of mixReg in regression.
[Table]
II.
Analysis of Distance Metrics.
Besides the theoretical analysis, here we empirically analyze theeffectiveness of using label distance.
Here, we used ( a, b ) to denote the distance between objects a andb , e.g., d ( y i , y j ) in our case.
We consider four substitute distance metrics, including: (1) feature distance :d ( x i , x j ) ; (2) feature and label distance : we concatenate the input feature x and label y to compute thedistance, i.e., d ( x i ⊕ y i , x j ⊕ y j ) ; (3) representationdistance : since using feature distance may fail tocapture high-level feature relations, we propose touse the model’s hidden representations to measurethe distance d ( h i , h j ) , detailed in Appendix F.2.
(4)representation and label distance : d ( h i ⊕ y
i , h j ⊕ y j ) .
The results are reported in Table 6.
We find that (1) feature distance d ( x i , x j ) does benefit performancecompared with ERM and vanilla mixup in all cases since it selects more reasonable example pairs formixing; (2) representation distance d ( h i , h j ) outperforms feature distance d ( x i , x j ) since it captureshigh-level features; (3) though involving label in representation distance is likely to overwhelm theeffect of labels, the better performance of d ( h i ⊕ y
i , h j ⊕ y j ) over d ( h i , h j ) indicates the effectiveness of regression labels in measuring example distance, which is further verified by the superiority ofmixReg.
(4) Our discussion of computational efficiency in Appendix A.3 indicates that mixReg ismuch more computationally efficient than other distance measurements.
[Figure] [Figure]
III.
How does the choice of bandwidth affects theperformance?
Finally, we analyze the effects ofthe bandwidth σ in Eqn.
The performance withrespect to bandwidth of Exchange-Rate and DTI arevisualized in Figure 3, respectively.
We point outthat if the bandwidth is too large, the results are closeto vanilla mixup or Manifold Mixup, depending onwhich one is used.
Otherwise, the results are closeto ERM if the bandwidth is too small.
According tothe results, we see that the improvements of mixRegis somewhat stable over different bandwidths.
Mostimportantly, mixReg yields a good model for a widerange of bandwidths, which reduces the efforts to tune the bandwidth for every specific dataset.
Additionally, we provide empirical guidance about how to pick suitable bandwidth in Appendix F.3.1.
Related Work
Data Augmentation and mixup.
Various data augmentation strategies have been proposed to improvethe generalization of deep neural networks, including directly augmenting images with manuallydesigned strategies (e.g., whitening, cropping) [32], generating more examples with generativemodels [2, 6, 76, 74], and automatically finding augmentation strategies [8, 9, 37].
Mixup [72] andits variants [7, 10, 17, 20, 21, 27, 28, 38, 39, 46, 58–60, 71] propose to improve generalization bylinear interpolating input features of a pair of examples and their corresponding labels.
Thoughmixup and its variants have demonstrated their power in classification [72], sequence labeling [73],and reinforcement learning [61], systematic analysis of mixup on different regression tasks is stillunderexplored.
The recent MixRL [24] learns a policy network to select nearby example pairs formixing, which requires substantial computational resources and is not suitable to high-dimensionalreal-world data.
Unlike this method, mixReg instead adjusts the sampling probability of mixing pairsbased on label similarity, which makes it much more efficient.
Furthermore, mixReg, which focuseson how to select mixing examples, is a complementary method over mixup and its representativevariants (see more discussion in Appendix A.4).
Empirically, our experiments show the effectivenessand compatibility of mixReg on multiple regression tasks in Section 4.1.
Task Generalization.
Our experiments extended mixReg to gradient-based meta-learning, aiming toimprove task generalization.
In the literature, there are two lines of related works.
The first line ofresearch directly imposes regularization on meta-learning algorithms [18, 25, 55, 69].
The secondline of approaches introduces task augmentation to produce more tasks for meta-training, includingimposing label noise [49], mixing support and query sets in the outer-loop optimization [43, 65], anddirectly interpolating tasks to densify the entire task distribution [67].
mixReg is complimentary tothe latter mixup-based task augmentation methods.
Furthermore, Section 4.2 indicates that mixRegempirically outperforms multiple representative prior approaches [69, 49].
Out-of-distribution Robustness.
Many recent methods aim to build machine learning models thatare robust to distribution shift, including learning invariant representations with domain alignment [14,36, 40, 57, 70, 75, 63] or using explicit regularizers to finding a invariant predictors that performswell over all domains [1, 3, 19, 26, 31, 34].
Recently, LISA [66] cancels out domain-associatedcorrelations and learns invariant predictors by mixing examples either with the same label but differentdomains or with the same domain but different labels.
While related, mixReg can be considered as amore general version of LISA that can be used for regression tasks.
Besides, unlike LISA, mixRegdo not use domain annotations, which are often expensive to obtain.
Conclusion
In this paper, we propose mixReg, a simple yet effective variant of mixup that is well-suited toregression tasks in deep neural networks.
Specifically, mixReg adjusts the sampling probabilityof mixing example pairs by assigning higher probability to pairs with closer label values.
Boththeoretical and empirical results demonstrate the promise of mixReg in improving in-distributiongeneralization, task generalization, and out-of-distribution robustness.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Appendix G(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] mixReg is a general machine learning method
and we believe that
it has no potential negative societal impacts. (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3and Appendix B(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Our code will be released once the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix C.2, D.2, and E.2.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix C, D, E, F.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.2, D.2, and E.2.4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board
) approvals, if applicable
N/A]
[Table]
you include the estimated hourly wage paid to participants and the total amountspent on participant compensation
[N/A] [Table]