mixReg: A Simple Way to Improve Generalization in Regression for Deep Neural Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Improving the generalization of deep networks is an important open challenge,particularly in domains without plentiful data.
The mixup algorithm improvesgeneralization by linearly interpolating the input features of a pair of examplesand their corresponding labels.
These interpolated examples augment the originaltraining dataset.
It has shown promising results in various classification tasks,but systematic analysis of mixup in regression remains underexplored.
Usingmixup directly on regression labels could result in arbitrarily wrong labels since thelinearity assumption behind mixup may not hold.
In this paper, we propose a simpleyet powerful algorithm, mixReg, to improve generalization on regression tasks.
In contrast with the vanilla mixup, which uses the same sampling probability forexample pairs, mixReg adjusts the sampling probability based on the similarity oflabels.
Our theoretical analysis further confirms that mixReg with label similarityobtains a smaller mean square error than vanilla mixup and using feature similarity.
Another benefit of mixReg is that it can improve out-of-distribution robustness,where the test distribution is different from the training distribution.
By selectivelyinterpolating examples with similar labels, it mitigates the effects of domainassociated information and pushes invariant representations.
We evaluate mixRegon eleven datasets, ranging from tabular to video data.
Compared to the best priorapproach, mixReg achieves 6.56%, 4.76%, 5.14% improvements in in-distributiongeneralization, task generalization, and out-of-distribution robustness, respectively.
Introduction
Training deep neural networks faces the challenge of mitigating overfitting and improving generalization.
In order to accomplish this, a number of techniques, including data augmentation [2, 6, 8, 61, 62]and explicit regularization [10, 28, 45] have been proposed.
Representatively, mixup [62] densifiesthe data distribution and regularizes the model by linearly interpolating the features between randomlysampled pairs of original examples and applying the same interpolation on the corresponding labels.
Despite mixup having demonstrated promising results in improving generalization in classificationproblems, it has rarely been studied in the context of regression with continuous responses, on whichwe focus in this paper.
In contrast to classification, which formalizes the label as a one-hot vector, the goal of regressionis to predict a continuous response from each input.
Directly applying mixup to input features andlabels to regression tasks may yield arbitrarily incorrect labels.
For example, as shown in Figure 1(a), Shapelet1D pose prediction [13] aims to predict the current orientation of the object relative to itscanonical orientation.
We randomly select three mixing pairs and show the mixed images and labelsin Figure 1(b), where only pair 1 exhibits reasonable mixing results.
Thus, sampling mixing pairsuniformly from the dataset introduces a number of noisy pairs.
Alternatively, Manifold Mixup [50]
[Figure]
[Figure]
∘ 𝑦 # : 150 ∘ 𝑦 # : 150 ∘ 𝑦 $ : 346 ∘ )𝑦 : 160. [Figure] ∘ )𝑦 : 248 ∘
Pair 2 (P2) 𝑦 $ : 346 ∘ 𝑦 ! : 171 ∘ )𝑦 :
[Figure]
∘
[Figure]
P
[Figure]
😄 ☹ 😐
[Figure]
mixReg is able to assign higher sampling probability for more reasonable mixing pairs proposes to linearly interpolate hidden representations rather than input features.
Although it ispossible to improve the original mixup in regression to some extent, empirical evidence suggests thatit cannot achieve consistent improvements over a wide range of scenarios [21].
Instead of applyingmixup or Manifold Mixup to all examples, MixRL [21] selects the mixing pairs in regression byusing a well-trained policy network.
The training of a complex policy network, however, requiresconsiderable computing resources, which are not suitable for high-dimensional real-world data.
Instead of training an external policy network, we aim to directly adjust the sampling probability ofmixing pairs according to the similarity of examples, resulting in a simple training technique named asmixReg ( mix up for Reg ression).
Specifically, we employ a Gaussian kernel to calculate the samplingprobability of drawing another example for mixing, where closer examples are more likely to besampled.
Here, the core question is: how to measure the similarity between two examples?
The moststraightforward solution is to compute input feature similarity.
Yet, using input similarity has twomajor downsides when dealing with high-dimensional data such as images or time-series: substantialcomputational costs and lack of good distance metrics.
Specifically, first, it takes considerable timeto compute pairwise similarities across all samples.
Second, directly applying classical distancemetrics (e.g., Euclidean distance, cosine distance) does not reflect the high-level relation betweeninput features.
In the Shapelet1D rotation prediction example (Figure 1(a)), pair 1 and pair 3 haveclose input similarities, which is not expected since only pair 1 shows the reasonable scenario aftermixing.
Besides, regression task is more sensitive to noise than classification task, where any featurenoise is likely to result in inaccurate label prediction.
To overcome these drawbacks, mixReg instead uses label similarity, which is easy to compute andreduces the computation cost since the label space is usually low dimensional.
In addition to theobvious computational advantages, mixReg benefits three kinds of regression problems.
First , itimproves in-distribution generalization in supervised learning compared with the vanilla mixupand using input feature similarity.
Second , we extend mixReg to gradient-based meta-learning andparticularly incorporated it with MetaMix, a mixup-based task augmentation method [55].
Comparedwith vanilla MetaMix, mixReg further improves task generalization; Third , mixReg is well-suitedfor improving out-of-distribution robustness, particularly to covariate shift (see the correspondingexample of Shapelet1D rotation prediction in Appendix A.1).
Concretely, we aim to learn a regressionmodel that is robust to the covariate shift between training and test distributions.
By performingmixup on examples with close continuous labels, mixReg encourages the regression model to rely onthe invariant features to make prediction and ignore unrelated or spurious correlations, making themodel more robust to covariate shift.
The primary contribution of this paper is: (1) We systematically investigate how to adapt mixupin regression, leading to a simple and scalable algorithm – mixReg; (2) We theoretically analyzethe advantages of mixReg.
Compared with vanilla mixup and using feature similarity, mixReg hasthe lower mean squared error and improves in-distribution generalization, task generalization androbustness to covariate shift; (3) Empirically, we conduct comprehensive experiments to evaluatemixReg on eleven datasets, ranging from tabular data to video data.
Compared to the best baseline, theresults demonstrate the promise of mixReg with 6.56%, 4.76%, 5.14% improvements in in-distributiongeneralization, task generalization, and out-of-distribution robustness, respectively.
Preliminaries
In this section, we define notation and describe the background of ERM and mixup in the supervisedlearning setting, and MetaMix in the meta-learning setting for task generalization.
ERM.
Assume a machine learning model f with parameter space Θ .
In this paper, we consider thesetting where one predicts the continuous label y ∈ Y according to the input feature x ∈ X .
Given aloss function ℓ , we train a model f θ under the empirical training distribution P tr with the followingobjective, and get the optimized parameter θ ∗ ∈ Θ : [Equation]
[Equation]
Typically, we expect the model to perform well on unseen examples drawn from the test distribution
P ts .
Here, we are interested in both in-distribution generalization ( P tr = P ts ) and out-of-distributionrobustness ( P tr ̸ = P ts ).
Mixup.
The mixup algorithm samples a pair of instances ( x i , y i ) and ( x j , y j ) , sampled uniformly atrandom from the training dataset, and generates new examples by performing linear interpolation onthe input features and corresponding labels as:
[Equation]
where the interpolation ratio λ ∈ [0 , 1] is drawn from a Beta distribution, i.e., λ ∼ Beta( α, α ) .
Theinterpolated examples are then used to optimize the model as follows:
[Equation]
Task Generalization and MetaMix.
In this paper, we also investigate few-shot task generalization under the gradient-based meta-learning setting, where each task is a regression task.
Given a task distribution p ( T ) , we assume each task T m is sampled from p ( T ) and is associated with a dataset D m .
A support set D sm =
{ ( X sm , Y sm ) }
= { ( x sm,i , y sm,i ) } N s i =1 and a query set
D qm = { ( X sm , Y sm ) }
= { ( x qm,j , y qm,j ) } N q
i
=1 are sampled from D m , where N s and N q represent thenumber of samples of support set and query set, respectively.
Representatively, in model-agnosticmeta-learning (MAML) [9], given a predictive model f with parameter θ , it aims to learn an initialization θ ∗ from meta-training tasks {T m } | M | m =1 , such that the model can be used or adapted to the eachmeta-test task T t with a small support set.
Specifically, at the meta-training phase, MAML obtained the task-specific parameter ϕ m for eachtask T m by performing a few gradient steps starting from the model initialization θ (i.e., ϕ m =θ − α ∇ θ L ( f θ ; D sm ) ).
Then, the corresponding query set D qm is used to evaluate the performance ofthe task-specific model and optimize the model initialization as:
[Equation]
D qm ) ,
[Equation]
ϕ m = θ − α ∇ θ L ( f θ
[Equation]
D sm )
[Equation]
At the meta-testing phase, for each meta-testing task T t , MAML fine-tunes the learned initializationθ ∗ on the support set D st and evaluates the performance on the corresponding query set D q t .
To improve task generalization, MetaMix [55] adapts mixup to meta-learning, which linearly interpolates the support set and query set and uses the interpolated set to replace the original query set D qm in Eqn.
Specifically, the interpolated query set is formulated as:
[Equation]
where λ ∼ Beta( α, α ) .
Mixup for Regression (mixReg)
This section first introduces the main idea of mixReg.
Then, we theoretically explain how mixRegbenefits in-distribution generalization, task generalization, and out-of-distribution robustness.
3.1 mixReg
For continuous labels, the example in Figure 1(b) illustrates that applying vanilla mixup to theentire distribution is likely to produce arbitrary labels.
To resolve this issue, mixReg proposes to sample closer pairs of examples with higher probability.
Specifically, given an example ( x i , y i ) ,mixReg introduces a symmetric Gaussian kernel in order to calculate the sampling probability
P (( x j , y j ) | ( x i , y i )) for another ( x j , y j ) example to be mixed as follows:
[Equation]
(6) where d ( i, j ) represents the distance between the examples ( x i , y i ) and ( x j , y j ) , and σ describes thebandwidth.
In the case of example ( x i , y i ) , the set { P (( x j , y j ) | ( x i , y i ))
|∀ j } is then normalized to aprobability mass function that sum up to one.
Require:
Learning rates η ; Shape parameter α
Require:
Training data D :
= { ( x i , y i ) }
Ni = 1: Randomly initialize model parameters θ 2: Calculate pairwise distance matrix P via
Eqn.
3: while not converge do 4:
Sample a batch of examples B ∼ D 5: for each example ( x i , y i )
∈ B do 6:
Sample example ( x j , y j ) according to the probability P ( · | ( x i , y i )) 7:
Linearly interpolate ( x i , y i ) and ( x j , y j ) to get ( ˜ x, ˜ y ) 8:
Use interpolated examples to update the model via Eqn.
The most straightforward way to compute the distance is using the input feature x , i.e., d ( i, j )
=
d ( x i , x j ) .
However, when dealing with the highdimensional data such as images or videos, welack good distance metrics to capture structuredfeature information and the distances can be easilyinfluenced by feature noise.
Additionally, computing feature distances for high-dimensional datais time-consuming.
Instead, mixReg leveragesthe labels with d ( i, j ) =
d ( y i , y j ) =
∥
y i − y j ∥ 2 2 .
The dimension of label is typically much smallerthan that of the input feature, therefore reducingcomputational costs (see more discussions aboutcompuatational efficiency in Appendix A.3).
Theoverall algorithm of mixReg is described in Alg. 1.
In addition to its computational benefits, mixRegimproves generalization on three distinct kinds ofregression problems, which is theoretically shown in the following sections.
3.2 mixReg for Improving In-distribution Generalization
In this section, we show that mixReg provably improves in-distribution generalization when thefeatures are observed with noise, and the response depends on a small fraction of the features in amonotonic way.
Specifically, we consider the following single index model with measurement error,
[Equation]
where θ ∈ R p and ϵ is a sub-Gaussian random variable and g is a monotonic transformation.
Sinceimages are often inaccurately observed in practice, we assume the feature z is observed or measuredwith noise, and denote the observed value by x : x = z
+ ξ with ξ being a random vector withmean 0 and covariance matrix σ 2 ξ I .
We assume g to be monotonic to model the nearly one-to-onecorrespondence between causal features (e.g., the car pose in Figure 1(a)) and labels (rotation) in thein-distribution setting.
The out-of-distribution setting will be discussed in Section 3.4.
We would liketo also comment that the single index model has been commonly used in econometrics, statistics, anddeep learning theory [14, 19, 34, 39, 54].
Suppose we have { ( x i , y i ) } Ni =1 i.i.d.
drawn from the above model.
We first follow the single indexmodel literature (e.g. [54]) and estimate θ by minimizing the square error (cid:80) ni
=
(˜ y i − ˜ x
⊤ i θ ) 2 , wherethe (˜ x i , ˜ y i ) ′
s are the augmented data by either vanilla mixup, mixup with input ( x )-similarity, andmixReg.
We denote the solution by θ ∗ mixup , θ ∗ feat , and θ ∗ mixReg respectively.
Given an estimate θ ∗ ,we estimate g by ˆ g via the standard nonparametric kernel estimator [48] (we specify this in detailin Appendix B.1 for completeness) using the augmented data.
We consider the mean square errormetric as MSE( θ ) =
E [( y − ˆ g ( θ ⊤ x )) 2 ] , and then have the following theorem (proof: Appendix B.1):
Theorem 1.
Suppose θ ∈ R p is sparse with sparsity s =
o (min { p, σ 2 ξ } ) , p =
o ( N ) and g is smoothwith 0 < g ′ <
c 1 , c 2 < g ′′ <
c 3 for some universal constants c 1 , c 2 , c 3 > 0 .
There exists adistribution on x with a kernel function, such that when the sample size N is sufficiently large, withprobability 1 − o (1) ,
[Equation]
The high-level intuition of why mixReg helps is that the vanilla mixup imposes linearity regularizationon the relationship between the feature and response.
When the relationship is strongly nonlinear andone-to-one, such a regularization hurts the generalization, but could be mitigated by mixReg.
3.3 mixReg for Improving Task Generalization
The second benefit of mixReg is improving task generalization in meta-learning when the data fromeach task follows the model discussed in the last section.
Concretely, we apply mixReg to MetaMix[55].
For each query example, the support example with a more similar label will have a higherprobability of being mixed.
The algorithm of mixReg on MetaMix is summarized in Appendix A.2.
Similar to in-distribution generalization analysis, we consider the following data generative model:for the m -th task ( m ∈ [ M ] ), we have ( x ( m ) , y ( m ) ) ∼ T m with [Equation]
and
[Equation]
Here, θ denotes the globally-shared representation, and g m ’s are the task-specific transformations.
Note that, the formulation is close to [40] and wildly applied to theoretical analysis of metalearning [55, 57].
Following similar spirit of [46] and last section, we obtain the estimation of θ byθ ∗ =
1 M (cid:80) Mm =
( E (˜ x ( m ) , ˜ y ( m ) )
∈
ˆ D m [˜ y ( m ) − θ ⊤ ˜ x ( m ) ]) .
Here, ˆ D m denotes the generic dataset augmentedby different approaches, including the vanilla MetaMix, MetaMix with input feature similarity, andmixReg.
We denote these approaches by θ ∗ MetaMix , θ ∗ Meta − feat , and θ ∗ Meta − mixReg respectively.
For anew task T t , we again use the standard nonparametric kernel estimator to estimate g t via the augmentedtarget data.
We then consider the following error metric MSE Target ( θ ∗ )
=
E ( x,y ) ∼T t
[( y − ˆ g t ( θ ∗⊤ x )) 2 ] .
Based on this metric, we get the following theorem to show the promise of mixReg in improving taskgeneralization (see Appendix B.2 for detailed proof).
Here, mixReg achieves smaller MSE Targetcompared to vanilla MetaMix and MetaMix with input feature similarity.
Theorem 2.
Let N = (cid:80)
Mm =1 N m
and N m is the number of examples of T m .
Suppose θ k is sparsewith sparsity s =
o (min { d, σ 2 ξ } ) , p = o ( N ) and g m ’s are smooth with 0 <
g ′ m < c 1 , c 2 < g ′′ m < c 3for some universal constants c 1 , c 2 , c 3 > 0 and m ∈
[ M ] ∪ { t } .
There exists a distribution on x witha kernel function, such that when the sample size N is sufficiently large, with probability 1 − o (1) ,
[Equation]
3.4 mixReg for Improving Out-of-distribution Robustness
Finally, we show that mixReg improves OOD robustness in the covariate shift setting where the someunrelated features vary across different domains.
In this setting, we regard the entire data distributionconsisting of E = { 1 , . . .
, E } domains, where each domain is associated with a data distribution P efor e ∈ E .
Given a set of training domains E tr ⊆ E , we aim to make the trained model generalizewell to an unseen test domain E ts that is not necessarily in E tr .
Here, we focus on covariate shift, i.e.,the change of P e among domains is only caused by the change of marginal distribution P e ( X ) , whilethe conditional distribution P e ( Y | X ) is fixed across different domains.
To overcome covariate shift, mixing examples with close labels without considering domain information can effectively average out domain-changeable correlations and make the predicted valuesrely on the invariant causal features.
To further understand how mixReg improves the robustness tocovariate shift, we provide the following theoretical analysis.
We assume the training data ( x i , y i ) n i =1 follows x i =
( z i ; a
i ) ∈
R p 1 + p
and y i =
θ
⊤ x i + ϵ i ,where z i ∈ R p 1 and a i ∈ R p 2 are regarded as invariant and domain-changeable unrelated features,respectively, and the last p 2 coordinates of θ ∈ R p 1 + p 2 are 0.
Now we consider the case wherethe training data consists of a pair of domains with almost identical invariant features and oppositedomain-changeable features, i.e., x i =
( z i , a i ) , x ′ i =
( z ′ i , a ′ i ) , where z i ∼ N p 1 (0 , σ 2 x I p 1 ) , z
′ i
=
z i + ϵ
i ,a i ∼ N p 2 (0 , σ 2 a I p 2 ) , a ′ i = − a
i + ϵ ′′ i .
ϵ
i , ϵ ′
i , ϵ i ′′ are noise terms with mean 0 and sub-Gaussian normbounded by σ ϵ .
We use ridge estimator θ ∗ ( k ) =
arg min θ ( (cid:80) i ∥
y i − θ ⊤ x i ∥ 2 + k ∥ θ ∥ 2 ) to reflect theimplicit regularization effect of deep neural networks [35].
The covariant shift happens when the testdomains have different a i distributions compared with training domains.
We then have the followingtheorem to show that mixReg can improve robustness to covariant shift (proof: Appendix B.3):
Theorem 3.
Supposed for some e − n < δ ≪ 1 , we have variance constraints:
σ a = c 1 σ x , σ x ≥c 2 max( n 5 / 2 ∥ θ ∥ δ σ ϵ , √ p 2 ∥ θ ∥ √ np 1 ) and σ ϵ ≤ c 3 √ log nn 1 / 4 .
Then for any penalty k satisfies c 4 (cid:113) p 2 p 1 n 1 / 4 < k <c 5 min( σ x √ p 1 n ∥ θ ∥ , n ) and bandwidth h satisfies 0 < h ≤ cl √ log( n 2 /p 1 ) in mix-y, when n is sufficientlylarge, with probability at least 1 − o (1) , we have
[Equation]
where c 1 , c 2 , c 3 , c 4 , c 5 , c 6 > 0 are universal constants and l = min i ̸ = j | y i − y j | .
Experiments
In this section, we evaluate the performance of mixReg, aiming to answer the following questions:
Q1:
Compared to corresponding prior approaches, can mixReg improve the in-distribution, taskgeneralization, and out-of-distribution robustness on regression?
Q2:
How does mixReg performcompared to using other distance metrics?
Q3:
Is mixReg sensitive to the choice of bandwidth σ inthe Gaussian kernel of Eqn.
In our experiments, we tune all hyperparameters via cross-validation.4.1 In-Distribution Generalization
Datasets.
The following five datasets are used to evaluate the performance of in-distribution generalization (see Appendix C.1 for detailed data statistics).
(1)&(2) Airfoil Self-Noise (Airfoil) and
NO2 [25] are both are tabular datasets, where airfoil contains aerodynamic and acoustic test resultsof airfoil blade sections and NO2 aims to predict the mount of air pollution at a particular location.(3)&(4): Exchange-Rate, and Electricity [30] are two time-series datasets, where Exchange-Ratereports the collection of the daily exchange rates and Electricity is used to predict the hourly electricityconsumption.
(5) Echocardiogram Videos (Echo): [37] is a ejection fraction prediction dataset,which consists of a series of videos illustrating the heart from different aspects.
Comparisons and Experimental Setups.
We first compare mixReg with mixup and its variants(Manifold mixup, k-mixup and Local Mixup) that can be easily to adapted to regression tasks.
Second, MixRL, a recent reinforcement learning framework to select mixup pairs in regression, is also usedfor comparison.
Note that, for k-mixup, Local Mixup, MixRL, and mixReg, we apply them to bothmixup and Manifold Mixup and report the best-performing combination.
For Airfoil and NO2, we employ a three-layer fully connected network as the backbone model.
Weuse LST-Attn [30] for the Exchange-Rate and Electricity, and EchoNet-Dynamic [37] for predictingthe ejection fraction.
We use Root Mean Square Error (RMSE) and Mean Averaged Percentage Error(MAPE) as evaluation metrics.
Detailed experimental setups are in Appendix C.2.
Results.
We report the results in Table 1 and have the following observations.
First , vanilla mixup andmanifold mixup are typically less performant than ERM when they are applied directly to regressiontasks.
These results support our hypothesis that random selection of example pairs for mixingmay produce arbitrary inaccurate virtual labels.
Second , while limiting the scope of interpolation(e.g.
, k-mixup, manifold k-mixup, Local Mixup) helps to improve generalization in most cases, theperformance of these approaches is inconsistent across different datasets.
As an example, Local
Mixup outperforms ERM in NO2 and Exchange-Rate, but fails to benefit results in Electricity.
Third ,even in more complicated datasets, such as Electricity and Echo, MixRL performs worse than Mixupand ERM, indicating that it is non-trivial to train a policy network that works well.
Finally , mixRegconsistently outperforms mixup and its variants, ERM, and MixRL, demonstrating its capability toimprove in-distribution generalization on regression problems.
[Figure]
Analysis of Overfitting.
In Figure 2, we visualize the test loss (RMSE) and the generalizationgap between training loss and test loss of ERM,
Manifold Mixup, and mixReg with respect tothe training epoch for Exchange-Rate.
Moreresults are illustrated in Appendix C.3.
Compared with ERM and mixReg, the better testperformance and smaller generalization gap ofmixReg demonstrates its ability to improve indistribution generalization and mitigate overfitting.
4.2 Task Generalization
Dataset and Experimental Setups.
To evaluate task generalization in meta-learning settings, we usetwo rotation prediction datasets named as ShapeNet1D [13] and Pascal1D [52, 59], the goal of bothdatasets is to predict an object’s rotation relative to the canonical orientation. Each task is definedas the rotation regression for one object. ShapeNet1D includes 27 categories with 60 objects percategory, where 50 and 10 objects are used for meta-training and meta-testing, respectively. Pascal3Dhas 65 objects, where 50 and 15 objects are selected for meta-training and meta-testing, respectively.
For each dataset, the model takes a 128×128 grey-scale image as the input, and the output is anazimuth angle normalized between [0 , 10] .
We detail the description in Appendix D.1.
[Table]
Since MetaMix outperforms most other methods in PASCAL3D in Yao et al.
[Table] we only selecttwo other representative approaches – Meta-Aug [41] and MR-MAML [59] for comparison.
Wefurther apply k-mixup and Local Mixup to MetaMix, which are called as k-MetaMix and Local
MetaMix, respectively.
Following Yin et al.
[59], the base model consists of an encoder with threeconvolutional blocks and a decoder with four convolutional blocks.
All baselines use the same basemodel.
Hyperparameters are listed in Appendix D.2.
[Table]
Results.
Table 2 shows the average MSE with a95% confidence interval over 6000 meta-testingtasks.
Our results corroborate the findingsof Yao et al.
[55] that MetaMix improves performance compared with non-mixup approaches(i.e.
, MAML, MR-MAML, Meta-Aug).
Similar to our previous in-distribution generalization findings, the better performance of Local
MetaMix over MetaMix demonstrates the effectiveness of interpolating nearby examples.
Bymixing query examples with support examplesthat have similar labels, mixReg outperforms allof the other approaches on both datasets, verifying its effectiveness of improving task generalization.
Out
-of-Distribution Robustness
Datasets.
In the following, we briefly discuss the datasets used to evaluate out-of-distributionrobustness; the detailed data descriptions in Appendix E.1.
(1) PovertyMap [24] is a satellite imageregression dataset, aiming to estimate asset wealth in countries that are not shown in the training set.(2) Communities and Crime (Crime) [42] is a tabular dataset, where the problem is to predict totalnumber of violent crimes per 100K population and we aim to generalize the model to unseen states.(3)
SkillCraft1 Master Table (SkillCraft) [4] is a tabular dataset, aiming to predict the mean latencyfrom the onset of a perception action cycles to their first action in milliseconds.
Here, “LeagueIndex"is treated as domain information.
(4) Drug-target Interactions (DTI) [20] is aiming to predictout-of-distribution drug-target interactions in 2019-2020 after training on 2013-2018.
Comparisons and Experimental Setups.
To evaluate the out-of-distribution robustness of mixReg,we compare it with nine invariant learning approaches that can be adapted to regression tasks,including ERM, IRM [3], IB-IRM [1], V-REx [29], CORAL [31], DRNN [11], GroupDRO [43],
Fish [44], and mixup.
All approaches use the same backbone model, where we adopt ResNet-50,three-layer full connected network, and DeepDTA [38] for Poverty, Crime and SkillCraft, and DTI,respectively.
Following the original papers of PovertyMap [58] and DTI [20], we use R -value toevaluate the performance.
For Crime and SkillCraft, we use RMSE as the evaluation metric.
Detailedsetups and hyperparameters are listed in Appendix E.
Results.
We report both average performance and worst-domain performance in Table 4.
Accordingto the results, we can see that the performance of prior invariant learning approaches is not stableacross datasets.
For example, IRM and CORAL outperform ERM on Camelyon17, but fail to improvethe performance on PovertyMap.
mixReg instead consistently shows the best performance regardlessof the data types, indicating its effectiveness in improving robustness to covariate shift.
[Table]
10 4 ) of invariance analysis.
Smaller pairwise divergence value represents stronger invariance.
[Table]
Analysis of Learned Invariance.
Following Yao et al.
[56],we analyze the domain invariance of the model learned bymixReg.
For regression, we measure domain invariance aspairwise divergence of the last hidden representation.
Sincethe labels are continuous, we first evenly split examples into C = { 1 , . . .
, C } bins according to their labels, where C = 10 .
After splitting, we perform kernel density estimation to estimate the probability density function P ( h ce ) of the last hidden representation.
We calculate the pairwise divergence as
[Equation]
P
[Equation] g cE [Equation] E
[Equation] e ) [Equation]
P
[Equation]
g cE | E = e ′ )) .
Theresults on Crime and SkillCraft are reported in Table 3, wheresmaller number denotes stronger invariance. We observe thatmixReg learns better invariant representation compared to prior invariant learning approaches. 4.4 Analysis of Distance Metrics
We report the RMSE, MSE, Avg. R for ExchangeRate (Ex.-Rate), ShapeNet1D (Shape1D) and DTI, respectively.
x , y , h represent input feature, label, and hidden representation, respectively. The full results are listed in Appendix F.
[Table]
[Table] Besides the theoretical analysis in Section 3.2,here we empirically analyze the effectivenessof using label distance.
Here, we use d ( a, b )to denote the distance between objects a and b ,e.g., d ( y i , y j ) in our case.
We specifically consider four substitute distance metrics, including:(1) feature distance : d ( x i , x j ) ; (2) feature andlabel distance : we concatenate the input feature x and label y to compute the distance, i.e.,d ( x i ⊕ y i , x j ⊕ y j ) , where ⊕ represents theconcatenation operator; (3) representation distance : since only using input feature distancemay fail to capture high-level feature relations,we propose to use the model’s hidden representations to measure the example distance.
Concretely, we adopt a two-stage training processfor each iteration. In the first stage, we feedthe data into the current backbone and get hidden representations h , which is used to calculate theexample distance, i.e., d ( h i , h j ) . In the second stage, we apply mixReg with representation distance.(4) representation and label distance : d ( h i ⊕ y
i , h j ⊕ y j ) .
The results are reported in Table 5.
We find that (1) feature distance d ( x i , x j ) does benefit performancecompared with ERM and vanilla mixup in all cases since it selects more reasonable example pairs formixing; (2) representation distance d ( h i , h j ) outperforms feature distance d ( x i , x j ) since it captureshigh-level features; (3) though involving label in representation distance is likely to overwhelm theeffect of labels, the better performance of d ( h i ⊕ y
i , h j ⊕ y j ) over d ( h i , h j ) indicates the effectivenessof regression labels in measuring example distance, which is further verified by the superiority of mixReg.
(4) Our discussion of computational efficiency in Appendix A.3 indicates that mixReg ismuch more computationally efficient than other distance {+measurements.4.
+}
How does the choice of bandwidth affects the performance?
[Figure]
Finally, we analyze the effects of the bandwidthσ in Eqn.
In Appendix G, we discuss morecomprehensive hyperparameter analysis, includingbandwidth σ and α in Beta distribution.
The performance with respect to bandwidth of Exchange-Rateand DTI are visualized in Figure 3, respectively.
Wepoint out that if the bandwidth is too large, the results are close to vanilla mixup or Manifold Mixup,depending on which one is used.
Otherwise, theresults are close to ERM if the bandwidth is toosmall.
According to the results, we see that theimprovements of mixReg is somewhat stable overdifferent bandwidths.
Most importantly, mixReg yields a good model for a wide range of bandwidths,which substantially reduces the efforts to tune the bandwidth.
Related Work
Data Augmentation and mixup.
Various data augmentation strategies have been proposed to improvethe generalization of deep neural networks, including directly augmenting images with manuallydesigned strategies (e.g., whitening, cropping) [27], generating more examples with generativemodels [2, 5, 66, 64], and automatically finding data augmentation strategies [6, 7, 32].
Mixup [62]and its variants [8, 15, 18, 50, 61] propose to improve generalization by linear interpolating inputfeatures of a pair of examples and their corresponding labels.
Though mixup and its variantshave demonstrated their power in classification [62], sequence labeling [63], and reinforcementlearning [51], systematic analysis of mixup on different kinds of regression tasks is still underexplored.
The recent MixRL method [21] learns a policy network to select nearby example pairs for mixing,which requires substantial computational resources and is not suitable for high-dimensional real-worlddata.
Unlike this method, mixReg instead adjusts the sampling probability of mixing pairs based onlabel similarity, which makes it much more efficient.
Empirically, mixReg outperforms mixup andother representative variants on multiple regression tasks in Section 4.1.
Task Generalization.
Our experiments extended mixReg to gradient-based meta-learning, aiming toimprove task generalization.
In the literature, there are two lines of related works.
The first line ofresearch directly imposes regularization on meta-learning algorithms [16, 22, 47, 59].
The secondline of approaches introduces task augmentation to produce more tasks for meta-training, includingimposing label noise [41], mixing support and query sets in the outer-loop optimization [36, 55], anddirectly interpolating tasks to densify the entire task distribution [57].
mixReg is complimentary tothe latter mixup-based task augmentation methods.
Furthermore, Section 4.2 indicates that mixRegempirically outperforms multiple representative prior approaches [59, 41].
Out-of-distribution Robustness.
Many recent methods aim to build machine learning models thatare robust to distribution shift, including learning invariant representations with domain alignment [12,31, 33, 49, 60, 65, 53] or using explicit regularizers to finding a invariant predictors that performswell over all domains [1, 3, 17, 23, 26, 29].
Recently, LISA [56] cancels out domain-associatedcorrelations and learns invariant predictors by mixing examples either with the same label but differentdomains or with the same domain but different labels.
While related, mixReg can be considered as amore general version of LISA that can be used for regression tasks.
Besides, unlike LISA, mixRegdo not use domain annotations, which are often expensive to obtain.
Conclusion
In this paper, we propose mixReg, a simple yet effective variant of mixup that is well-suited toregression tasks in deep neural networks.
Specifically, mixReg adjusts the sampling probabilityof mixing example pairs by assigning higher probability to pairs with closer label values.
Boththeoretical and empirical results demonstrate the promise of mixReg in improving in-distributiongeneralization, task generalization, and out-of-distribution robustness.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix H
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] mixReg
is a general machine learning method and we believe that it has no potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section and Appendix B (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[No] Our code will
be released once the paper is accepted.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix C.2, D.2, and E.2.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See Appendix C, D, E, F.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.2, D.2, and E.2.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]