V ARIANCE B ASED S AMPLE W EIGHTING FOR S UPERVISED D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In the context of supervised learning of a function by a Neural Network (NN), we claim and empirically justify that a NN yields better results when the distribution of the data set focuses on regions where the function to learn is steeper.
We ﬁrst traduce this assumption in a mathematically workable way using Taylor expansion.
Then, theoretical derivations allow to construct a methodology that we call
Variance Based Samples Weighting (VBSW).
VBSW uses local variance of the labels to weight the training points.
This methodology is general, scalable, cost effective, and signiﬁcantly increases the performances of a large class of NNs for various classiﬁcation and regression tasks on image, text and multivariate data.
We highlight its beneﬁts with experiments involving NNs from shallow linear NN to ResNet (He et al., 2015) or Bert (Devlin et al., 2019).
I NTRODUCTION
When a Machine Learning (ML) model is used to learn from data, the distribution of the training data set can have a strong impact on its performances.
More speciﬁcally, in the context of Deep Learning (DL), several works have hinted at the importance of the training set.
In Bengio et al.
Matiisen et al.
, the authors exploit the observation that a human will beneﬁt more from easy examples than from harder ones at the beginning of a learning task.
They construct a curriculum, inducing a change in the distribution of the training data set that makes a Neural Network (NN) achieve better results in an ML problem.
With a different approach, Active Learning (Settles, 2012) modiﬁes dynamically the distribution of the training data, by selecting the data points that will make the training more efﬁcient.
Finally, in Reinforcement Learning, the distribution of experiments is crucial for the agent to learn efﬁciently.
Nonetheless, the challenge of ﬁnding a good distribution is not speciﬁc to ML.
Indeed, in the context of Monte Carlo estimation of a quantity of interest based on a random variable X ∼ d P X , Importance Sampling owes its efﬁciency to the construction of a second random variable, ¯ X ∼ d P ¯ X that will be used instead of X to improve the estimation of this quantity.
Jie & Abbeel (2010) even make a connection between the success of likelihood ratio policy gradients and importance sampling, which shows that ML and Monte Carlo estimation, both distribution based methods, are closely linked.
In this paper, we leverage the importance of the training set distribution to improve performances of NNs in supervised DL.
This task can be formalized as approximating a function f with a model f θ parametrized by θ .
We build a new distribution from the training points and their labels, based on the observation that f θ needs more data points to approximate f on the regions where it is steep.
We use Taylor expansion of a function f , which links the local behaviour of f to its derivatives, to build this distribution.
We show that up to a certain order and locally, variance is an estimator of Taylor expansion.
It allows constructing a methodology called Variance Based Sample Weighting (VBSW) that weights each training data points using the local variance of their neighbor labels to simulate the new distribution.
Sample weighting has already been explored in many works and for various goals.
Kumar et al.
; Jiang et al.
(2015) use it to prioritize easier samples for the training, Shrivastava et al.
for hard example mining, Cui et al.
(2019) to avoid class imbalance, or (Liu & Tao, 2016) to solve noisy label problem.
In this work, the weights’ construction relies on a more general claim that can be applied to any data set and whose goal is to improve the performances of the model.
VBSW is general, because it can be applied to any supervised ML problem based on a loss function.
In this work we speciﬁcally investigate VBSW application to DL. In that case, VBSW is applied within the feature space of a pre-trained NN.
We validate VBSW for DL by obtaining performance improvement on various tasks like classiﬁcation and regression of text, from Glue benchmark (Wang et al., 2019), image, from MNIST (LeCun & Cortes, 2010) and Cifar10 (Krizhevsky et al.) and multivariate data, from UCI ML repository (Dua & Graff, 2017), for several models ranging from linear regression to Bert (Devlin et al., 2019) or ResNet20 (He et al., 2015).
As a highlight, we obtain up to 1 .
65% classiﬁcation improvement on Cifar10 with a ResNet.
Finally, we conduct analyses on the complementarity of VBSW with other weighting techniques and its robustness.
Contributions : (i) We present and investigate a new approach of the learning problem, based on the variations of the function f to learn.
(ii)
We construct a new simple, scalable, versatile and cost effective methodology, VBSW, that exploits these ﬁndings in order to boost the performances of a
NN.
(iii)
We validate VBSW on various ML tasks.
R ELATED W ORKS
Active Learning - Our methodology is based on the consideration that not every sample bring the same amount of information.
Active learning (AL) exploits the same idea, in the sense that it adapts the training strategy to the problem by introducing a data point selection rule.
In (Gal et al., 2017), the authors introduce a methodology based on Bayesian Neural Networks (BNN) to adapt the selection of points used for the training.
Using the variational properties of BNN, they design a rule to focus the training on points that will reduce the prediction uncertainty of the NN.
In (Konyushkova et al., 2017), the construction of the selection rule is taken as a ML problem itself.
See (Settles, 2012) for a review of more classical AL methods.
While AL selects the data points, so modiﬁes the distribution of the initial training data set, VBSW is applied independently of the training so the distribution of the weights can not change throughout the training.
Examples Weighting - VBSW can be categorized as an example weighting algorithm.
The idea of weighting the data set has already been explored in different ways and for different purposes.
While curriculum learning (Bengio et al., 2009; Matiisen et al., 2017) starts the training with easier examples, Self paced learning (Kumar et al., 2010; Jiang et al., 2015) downscales harder examples.
However, some works have proven that focusing on harder examples at the beginning of the learning could accelerate it.
In (Shrivastava et al., 2016), hard example mining is performed to give more importance to harder examples by selecting them primarily.
Example weighting is used in (Cui et al., 2019) to tackle the class imbalance problem by weighting rarer, so harder examples.
At the contrary, in (Liu & Tao, 2016) it is used to solve the noisy label problem by focusing on cleaner, so easier examples.
All these ideas show that depending on the application, example weighting can be performed in an opposed manner.
Some works aim at going beyond this opposition by proposing more general methodologies.
In (Chang et al., 2017), the authors use the variance of the prediction of each point throughout the training to decide whether it should be weighted or not.
A meta learning approach is proposed in (Ren et al., 2018), where the authors choose the weights after an optimization loop included in the training.
VBSW stands out from the previously mentioned example weighting methods because it is built on a more general assumption that a model would simply need more points to learn more complicated functions.
Its effect is to improve the performances of a NN, without solving data set speciﬁc problems like class imbalance or noisy labels.
Importance Sampling - Some of the previously mentioned methods use importance sampling to design the weights of the data set or to correct the bias induced by the sample selection (Katharopoulos & Fleuret, 2018).
Here, we construct a new distribution that could be interpreted as an importance distribution.
However, we weight the data points to simulate this distribution, not to correct a bias induced by this distribution.
Generalization Bound - Generalization bound for the learning theory of NN have motivated many works, most of which are reviewed in (Jakubovitz et al., 2018).
In Bartlett et al.
, Bartlett et al.
(2019), the authors focus on VC-dimension, a measure which depends on the number of parameters of NNs.
Arora et al.
(2018) introduces a compression approach that aims at reducing the number of model parameters to investigate its generalization capacities.
PAC-Bayes analysis constructs generalization bounds using a priori and a posteriori distributions over the possible models.
It is investigated for example in Neyshabur et al.
; Bartlett et al.
(2017), and Neyshabur et al.
; Xu & Mannor (2012) links PAC-Bayes theory to the notion of sharpness of a NN, i.e. its robustness to small perturbation.
While sharpness of the model is often mentioned in the previous works, our bound includes the derivatives of f , which can be seen as an indicator of the sharpness of the function to learn.
Even if it uses elements of previous works, like the Lipschitz constant of f θ , our work does not pretend to tighten and improve the already existing generalization bounds, but only emphasizes the intuition that the NN would need more points to capture sharper functions.
In a sense, it investigates the robustness to perturbations in the input space, not in the parameter space.
A N EW T RAINING D ISTRIBUTION BASED ON T AYLOR E XPANSION
In this section, we ﬁrst illustrate why a NN may need more points where f is steep by deriving a generalization bound that involves the derivatives of f .
Then, using Taylor expansion, we build a new training distribution that improves the performances of a NN on simple functions.
3. P ROBLEM F ORMULATION
We formalize the supervised ML task as approximating a function f : S ⊂ R n i → R n o with an ML model f θ parametrized by θ , where S is a measured sub-space of R n i depending on the application.
To this end, we are given a training data set of N points, { X 1 , ..., X N } ∈ S , drawn from X ∼ d P X and their point-wise values, or labels { f ( X 1 ) , ..., f ( X N ) } .
Parameters θ have to be found in order to minimize an integrated loss function J X ( θ ) =
E X
[ L ( f θ ( X ) , f ( X ))] , with L the loss function,
L : R n o × R n o → R .
The data allow estimating J X ( θ ) by (cid:99) J X
( θ )
=
1 N (cid:80)
[Equation] Then, an optimization algorithm is used to ﬁnd a minimum of (cid:99) J X
( θ ) w.r.t.
θ .
3. I NTUITION BEHIND T AYLOR E XPANSION
In the following, we illustrate the intuition with a Generalization Bound (GB) that include the derivatives of f , provided that these derivatives exist.
The goal of the approximation problem is to be able to generalize to points not seen during the training.
The generalization error
J X ( θ ) =
J X ( θ ) − (cid:99)
J X ( θ ) thus needs to be as small as possible.
Let S i , i ∈ { 1 , ..., N } be some sub-spaces of S such that S
= (cid:83) Ni
=1 S i , (cid:84) Ni
=1 S i
=
Ø, and X i ∈ S
i .
Suppose that L is the squared
L 2 error, n i = 1 , f is differentiable and f θ is K θ -Lipschitz.
Provided that | S i | < 1 , we show that
[Equation]
where | S i | is the volume of S i .
The proof can be found in Appendix B .
We see that on the regions where f (cid:48)
( X i ) is higher, quantity | S
i | has a stronger impact on the GB.
This idea is illustrated on
Figure 1.
Since | S i | can be seen as a metric for the local density of the data set (the smaller | S i | is, the denser the data set is), the GB can be reduced more efﬁciently by adding more points around X i in these regions.
This bound also involves K θ , the Lipschitz constant of the NN, which has the same impact than f (cid:48)
( X i ) .
It also illustrates the link between the Lipschitz constant and the generalization error, which has been pointed out by several works like (Gouk et al., 2018), (Bartlett et al., 2017) and (Qian & Wegman, 2019).
Note that equation 1 only gives indications about n = 1 .
Indeed, this GB only has illustration purposes. Its goal is to motivate the metric described in the next section, which is based on Taylor expansion and therefore involves derivatives of order n > 1 .
3. A T AYLOR EXPANSION BASED METRIC
In this paragraph, we build a metric involving the derivatives of f .
Using Taylor expansion at order n on f and supposing that f is n times differentiable (multi index notation):
[Equation]
Quantity f ( x + (cid:15) ) − f ( x ) gives an indication on how much f changes around x .
By neglecting the orders above (cid:15) n , it is then possible to ﬁnd the regions of interest by focusing on Df n (cid:15) , given by equation 2, where Vect( X ) denotes the vectorization of a tensor X and (cid:107) .
(cid:107) the squared L 2 norm.
Note that Df n (cid:15) is evaluated using (cid:107) ∂
k f ( x ) (cid:107) instead of ∂ k f ( x ) for derivatives not to cancel each other.
f will be steeper and more irregular in the regions where x → Df n (cid:15) ( x ) is higher.
To focus the training set on these regions, one can use { Df n (cid:15) ( X 1 ) , ..., Df n (cid:15) ( X N ) } to construct a probability density function (pdf) and sample new data points from it.
This sampling is evaluated and validated in Appendix A for conciseness.
Based on these experiments, we choose n = 2 , i.e. we use { Df 2 (cid:15) ( X 1 ) , ..., Df 2 (cid:15) ( X N ) } .
The good results obtained, presented in Appendix A conﬁrm our observation and motivate its application to complex DL problems.
V ARIANCE B ASED S AMPLES W EIGHTING (VBSW) 4. P RELIMINARIES
The new distribution cannot always be applied as is, because we do not have access to f .
Problem 1: { Df 2 (cid:15) ( X 1 ) , ..., Df 2 (cid:15) ( X N ) } cannot be evaluated since it requires to compute the derivatives of f .
Moreover, it assumes that f is differentiable, which is often not true.
Problem 2: even if { Df 2 (cid:15) ( X 1 ) , ..., Df 2 (cid:15) ( X N ) } could be computed and new points sampled, we could not obtain their labels to complete the training data set.
Problem 1: Unavailability of derivatives To overcome problem 1 , we construct a new metric based on statistical estimation.
In this paragraph, n i > 1 but n o = 1 .
The following derivations can be extended to n o > 1 by applying it to f element-wise and then taking the sum across the n o dimensions.
Let (cid:15) ∼ N (0 , (cid:15) I n i ) with (cid:15) ∈ R + and I n i the identity matrix of dimension n i .
We claim that
[Equation]
The demonstration can be found in Appendix B .
Using the unbiased estimator of variance, we thus deﬁne new indices (cid:100)
Df 2 (cid:15) ( x ) by
[Equation]
with { (cid:15) 1 , ..., (cid:15) k } k samples of (cid:15) .
The metric (cid:100) Df 2 (cid:15) ( x ) → k →∞ V ar ( f ( x + (cid:15) )) and
V ar ( f ( x + (cid:15) ))
=
Df 2 (cid:15) ( x ) + O ( (cid:107) (cid:15) (cid:107) 32 ) , so (cid:100) Df 2 (cid:15) ( x ) is a biased estimator of Df 2 (cid:15) ( x ) , with bias O ( (cid:107) (cid:15) (cid:107) 32 ) .
Hence, when (cid:15) → 0 , (cid:100) Df 2 (cid:15) ( x ) becomes an unbiased estimator of Df 2 (cid:15) ( x ) .
It is possible to compute (cid:100)
Df 2 ( x ) from any set of points centered around x .
Therefore, we compute (cid:100) Df 2 ( X i ) for each i ∈ { 1 , ..., N } using the set S k ( X ) of k -nearest neighbors of X i .
We obtain (cid:100) Df 2 ( X i ) using
[Equation]
The advantages of this formulation are twofold.
First, (cid:100) Df 2 can even be applied to non-differentiable functions.
Second, all we need is { f ( X 1 ) , ..., f ( X N ) } .
In other words, the points used by (cid:100) Df 2 ( X i ) are those used for the training of the NN.
Finally, while the deﬁnition of Df 2 (cid:15) ( x ) is local, the deﬁnition of (cid:100) Df 2 (cid:15) ( x ) holds for any (cid:15) .
Note that equation 4 can even be applied when the data points are too sparse for the nearest neighbors of X i to be considered as close to X i .
It can thus be seen as a generalization of Df 2 (cid:15) ( x ) , which tends towards Df 2 (cid:15) ( x ) locally.
Problem 2: Unavailability of new labels To tackle problem 2 , recall that the goal of the training is to ﬁnd θ ∗ = argmin θ (cid:99) J X
( θ ) , with (cid:99)
J X
( θ )
=
1 N (cid:80) i
L ( f ( X i ) , f θ ( X i )) .
With the new distribution based on previous derivations, the procedure is different.
Since the training points are sampled using (cid:100) Df 2 (cid:15) , we no longer minimize (cid:99) J X
( θ ) , but (cid:99)
J ¯ X ( θ ) =
1 N (cid:80) i
L ( f ( ¯ X i ) , f θ ( ¯ X i )) , with ¯ X ∼ d P ¯ X the new distribution.
However, (cid:99)
J ¯ X ( θ ) estimates
[Equation]
Let p X ( x ) dx =
d P X , p ¯ X ( x ) dx =
d P ¯ X be the pdfs of X and ¯ X (note that Df 2 (cid:15)
∝ p ¯ X ).
Then,
[Equation]
The straightforward Monte Carlo estimator for this expression of J ¯ X ( θ ) is
[Equation]
Thus, J ¯ X ( θ ) can be estimated with the same points as J X ( θ ) by weighting them with w i =
(cid:100) Df 2 ( X i )
p X
( X i ) .
4. H YPERPARAMETERS OF VBSW
The expression of w i involves Df 2 (cid:15)
( X i ) , whose estimation has been the goal of the previous sections.
However, it also involves p X , the distribution of the data.
Just like for f , we do not have access to p X .
The estimation of p X is a challenging task by itself, and standard density estimation techniques such as K-nearest neighbors or Gaussian Mixture density estimation led to extreme estimated values of p X ( X i ) in our experiments.
Therefore, we decided to only apply ω i =
(cid:100) Df 2 ( X i ) as a ﬁrst order approximation.
In practice, we re-scale the weighting points to be between 1 and m , a hyperparameter.
As a result, VBSW has two hyperparameters: m and k .
Their effects and interactions are studied and discussed in Sections 5.1 and 5.4.
4. VBSW FOR D EEP L EARNING
We speciﬁed that local variance could be computed using already existing points.
This statement implies to ﬁnd the nearest neighbors of each point.
In extremely high dimension spaces like image spaces the curse of dimensionality makes nearest neighbors spurious.
In addition, the structure of the data may be highly irregular, and the concept of nearest neighbor misleading.
Thus, it may be irrelevant to evaluate (cid:91) D 2 f (cid:15) directly on this data.
One of the strength of DL is to construct good representations of the data, embedded in lower dimensional latent spaces.
For instance, in Computer Vision, Convolutional Neural Networks (CNN)’s deeper layers represent more abstract features.
We could leverage this representational power of
NNs, and simply apply our methodology within this latent feature space.
[Table]
The ﬁnal
Variance Based Samples Weighting (VBSW) for DL is recapitulated in Algorithm 1.
Here, M is the initial NN whose feature space will be used to project the training data set and apply VBSW.
Line 1: m and k are hyperparameters that can be chosen jointly with all other hyperparameters, e.g. using a random search.
Line 2: The initial NN, M , is trained as usual.
Notations { ( 1 N , X 1 ) , ..., ( 1 N , X N ) } is equivalent to { X 1 , ..., X N } , because all the weights are the same ( 1 N ).
Line 3:
The last fully connected layer is discarded, resulting in a new model M ∗ , and the training data set is projected in the feature space.
Line 4-5: equation 4 is applied to compute the weights w i that are used to weight the projected data set.
To perform nearest neighbors search, we use KD-Tree (Bentley, 1975).
Line The last layer is re-trained (which is often equivalent to ﬁtting a linear model) using the weighted data set and added to M ∗ to obtain the ﬁnal model M f .
As a result, M f is a composition of the already trained model M ∗ and f θ trained using the weighted data set.
E XPERIMENTS
We ﬁrst test this methodology on toy datasets with linear models and small NNs.
Then, to illustrate how general VBSW can be, we consider various tasks in image classiﬁcation, text regression and classiﬁcation. Finally, we study the robustness of VBSW and its complementarity with other sample weighting techniques.
5. T OY EXPERIMENTS
VBSW is studied on a Double Moon (DM) classiﬁcation, in the Boston Housing (BH) regression and Breast Cancer (BC) classiﬁcation data sets.
VBSW baseline DM [Table] BC 99.12 ,
[Table]
For DM, Figure 2 (c) shows that the points with highest w i (in red) are close to the boundary between the two classes.
Indeed, in classiﬁcation, VBSW can be interpreted as local label agreement.
We train a Multi Layer Perceptron of 1 layer of 4 units, using Stochastic Gradient
Descent (SGD) and binary cross-entropy loss function, on a 300 points training data set for 50 random seeds.
In this experiment, VBSW, i.e. weighting the data set with w i is compared to baseline where no weights are applied.
Figure 2 (b) and (d) displays the decision boundary of best ﬁt for each method.
VBSW provides a cleaner decision boundary than baseline.
These pictures as well as the results of Table 1 show the improvement obtained with VBSW.
For BH data set, a linear model is trained and for BC data set, a MLP of 1 layer and 30 units, with a train-validation split of 80% − 20% .
Both models are trained with ADAM (Kingma & Ba, 2014).
Since these data sets are small and the models are light, we study the effects of the choice of m and k on the performances.
Moreover, BH is a regression task and BC a classiﬁcation task, so it allows studying the effect of hyperparameters more extensively.
We train the models for a grid of 20 × 20 different values of m and k .
These hyperparameters seem to have a different impact on performances for classiﬁcation and regression.
In both cases, low values for m yields better results, but in classiﬁcation, low values of k are better, unlike in regression.
Details and visualization of this experiment can be found in Appendix C .
The best results obtained with this study are compared to the best result of the same models trained without VBSW in Table 1.
5. MNIST AND CIFAR
For MNIST, we train 40 LeNet 5, i.e. with 40 different random seeds, and then apply VBSW fordifferent random seeds, with ADAM optimizer and categorical cross-entropy loss.
Note that in the following, ADAM is used with the default parameters of its keras implementation.
We record the best value obtained from the 10 VBSW training.
The same procedure is followed for Cifar10, except that we train a ResNet20 for 50 random seeds and with data augmentation and learning rate decay.
The networks have been trained on 4 Nvidia K80 GPUs.
The values of the hyperparameters used can be found in Appendix C .
We compare the test accuracy between LeNet 5 + VBSW, ResNet + VBSW and the initial test accuracies of LeNet 5 and ResNet20 (baseline) for each of the initial networks. [Table]
[Table]
The results statistics are gathered in Table 2, which also displays statistics about the gain due to
VBSW for each model.
The results on MNIST, for all statistics and for the gain are signiﬁcantly better than forVBSW than for baseline.
For Cifar10, we get a 0 .
3% accuracy improvement for the best model and up to 1 .
65% accuracy gain, meaning that among the 50 ResNet20s, there is one whose accuracy has been improved by 1 .
65% by VBSW.
Note that applying VBSW took less than 15 minutes on a laptop with an i7-7700HQ CPU.
A visualization of the samples that were weighted by the highest w i is given in Figure 3.
5. RTE, STS-B AND MRPC
For this application, we do not pre-train Bert NN, like in the previous experiments, since it has been originally built for Transfer Learning purposes.
Therefore, its purpose is to be used as is and then ﬁne-tuned on any NLP data set see (Devlin et al., 2019).
However, because of the small size of the dataset and the high number of model parameters we chose not to ﬁne-tune the Bert model, and only to use the representations of the data sets in its feature space to apply VBSW.
More speciﬁcally, we use tiny-bert (Turc et al., 2019), which is a lighter version of the initial Bert NN.
We train the linear model with tensorﬂow, to be able to add the trained model on top of the Bert model and obtain a uniﬁed model.
RTE and MRPC are classiﬁcation tasks, so we use binary cross-entropy loss function to train our model.
STS-B is a regression task so the model is trained with Mean Squared Error.
All the models are trained with ADAM optimizer.
For each task, we compare the training of the linear model with VBSW, and without VBSW (baseline).
The results obtained with VBSW are better overall, except for Pearson Correlation in STS-B, which is slightly worse than baseline (Table 3)
[Table]
5. R OBUSTNESS OF VBSW
VBSW relies on statistical estimation: the weights are based on local empirical variance, evaluated using k points.
In addition, they are rescaled using hyperparameter m .
Section 5.1 and Appendix
C show that many different combinations of m and k and therefore many different values for the weights improve the error.
This behavior suggests that VBSW is quite robust to weights approximation error.
We also assess the robustness of VBSW to label noise.
To that end, we train a ResNet20 on Cifar with four different noise levels.
We randomly change the label of p % training points for four different values of p ( 10 , 20 , 30 and 40 ).
We then apply VBSW 30 times and evaluate the performances of the obtained NNs on a clean test set.
The results are gathered in Table 4.
[Table]
The results show that VBSW is still effective despite label noise, which could be explained by two observations.
First, the weights of VBSW rely on statistical estimation, so perturbations in the labels might have a limited impact on weights’ value.
Second, as mentioned previously, VBSW is robust to weights approximation error, so perturbation of the weights due to label noise may not critically hurt the method.
Although VBSW is robust to label noise, note that the goal of VBSW is not to address noisy label problem, like discussed in Section 2.
It may be more effective to use a sampling technique speciﬁcally tailored for this situation - possibly jointly with VBSW, like in Section 5.5.
5. C OMPLEMENTARITY OF VBSW
Existing sample weighting techniques can be used jointly with VBSW by training the initial NN
M with the ﬁrst sample weighting algorithm, and then applying VBSW on its feature space.
To illustrate this feature, we compare VBSW with the recently introduced Active Bias (AB) (Chang et al., 2017).
AB dynamically weights the samples based on the variance of the probability of prediction of each points throughout the training.
Here, we study the combined effects of AB and
VBSW for the training of a ResNet20 on Cifar10.
Table 5 gathers the results of experiments for different baselines: vanilla, for a regular training with Adam optimizer, AB for a training with AB,
VBSW for the application of VBSW on top of a regular training and VBSW + AB for a training with AB and the application of VBSW.
Unlike in section 5.2, we do not use data augmentation or learning rate decay, to simplify the experiments (which explains the accuracy loss compared to previous experiments).
These results demonstrate the competitiveness of VBSW compared with AB and their complementarity.
Indeed, the accuracy obtained with VBSW is quite similar to AB and the best mean and max accuracy is obtained for a NN trained with AB + VBSW.
Note that in this experiment, the gain
[Table]
per model is lower for AB + VBSW than for VBSW alone.
An explanation might be that AB is already improving the NN performances compared to vanilla, so there is less room for accuracy improvement by VBSW in that case.
D ISCUSSION & F UTURE W ORK
Previous experiments demonstrate the performances improvement that VBSW can bring in practice.
In addition to these results, several advantages can be pointed out.
• VBSW is validated on several different tasks, which makes it quite versatile.
Moreover, the problem of high dimensionality and irregularity of f , which often arises in DL problems, is alleviated by focusing on the latent space of NNs.
This makes VBSW scalable.
As a result, VBSW can be applied to complex NNs such as ResNet, a complex CNN or Bert, for various ML tasks. Its application to more diverse ML models is a perspective for future works.
• The validation presented in this paper supports an original view of the learning problem, that involves the local variations of f .
The studies of Appendix A , that use the derivatives of the function to learn to sample a more efﬁcient training data set, support this approach as well.
• VBSW allows to extend this original view to problems where the derivatives of f are not accessible, and sometimes not deﬁned.
Indeed, VBSW comes from Taylor expansion, which is speciﬁc to derivable functions, but in the end can be applied regardless of the properties of f . • Finally, this method is cost effective.
In most cases, it allows to quickly improve the performances of a NN using a regular CPU.
In terms of energy consumption, it is better than carrying on a whole new training with a wider and deeper NN.
We ﬁrst approximated p X to be uniform, because we could not approximate it correctly.
This approximation still led to a an efﬁcient methodology, but VBSW may beneﬁt from a ﬁner approximation of p X .
Improving the approximation of p X is among our perspectives.
Finally, the KD-tree and even Approximate Nearest Neighbors algorithms struggle when the data set is too big.
One possibility to overcome this problem would be to parallelize their execution.
We only considered the cases where we have no access to f .
However, there are ML applications where we do.
For instance, in numerical simulations, for physical sciences, computational economics or climatology, ML can be used for various reasons, e.g. sensitivity analysis, inverse problems or to speed up computer codes (Zhu et al., 2019; Winovich et al., 2019; Feng et al., 2018).
In this context data comes from numerical models, so the derivatives of f are accessible and could be directly used. Appendix A contains examples of such possible applications.
C ONCLUSION
Our work is based on the observation that, in supervised learning, a function f is more difﬁcult to approximate by a NN in the regions where it is steeper.
We mathematically traduced this intuition and derived a generalization bound to illustrate it.
Then, we constructed an original method, Variance Based Samples Weighting (VBSW), that uses the variance of the training samples to weight the training data set and boosts the model’s performances.
VBSW is simple to use and to implement, because it only requires to compute statistics on the input space.
In Deep Learning, applying VBSW on the data set projected in the feature space of an already trained NN allows to reduce its error by simply training its last layer.
Although speciﬁcally investigated in Deep Learning, this method is applicable to any loss function based supervised learning problem, scalable, cost effective, robust and versatile. It is validated on several applications such as glue benchmark with Bert, for text classiﬁcation and regression and Cifar10 with a ResNet20, for image classiﬁcation.
A PPENDIX A T AYLOR B ASED S AMPLING
In this part, we empirically verify that using Taylor expansion to construct a new training distribution has a beneﬁcial impact on the performances of a NN.
To this end, we construct a methodology, that we call Taylor Based Sampling (TBS), that generates a new training data set based on the metric introduced in Section 3.3 .
First, we recall the formula for { Df n (cid:15) ( X 1 ) , ..., Df n (cid:15) ( X N ) } .
[Equation]
To focus the training set on the regions of interest, i.e. regions of high { Df n (cid:15) ( X 1 ) , ..., Df n (cid:15) ( X N ) } , we use this metric to construct a probability density function (pdf).
This is possible since Df n (cid:15) ( x ) ≥
for all x ∈ S .
It remains to normalize it but in practice it is enough considering a distribution d ∝ Df n (cid:15) .
Here, to approximate d we use a Gaussian Mixture Model (GMM) with pdf d GMM that we ﬁt to { Df n (cid:15) ( X 1 ) , ..., Df n (cid:15) ( X N ) } using the Expectation-Maximization (EM) algorithm.
N (cid:48) new data points { ¯ X 1 , ..., ¯ X N (cid:48) } , can be sampled, with ¯ X ∼ d GMM .
Finally, we obtain { f ( ¯ X 1 ) , ..., f ( ¯ X N (cid:48) ) } , add it to { f ( X 1 ) , ..., f ( X N ) } and train our NN on the whole data set.
[Equation]
S AMPLING (TBS)
TBS is described in Algorithm 2.
Line 1: The choice criterion of (cid:15) , the number of Gaussian distribution n GMM and N (cid:48) is to avoid sparsity of { ¯ X 1 , ..., ¯ X N (cid:48) } over S .
Line 2: Without a priori information on f , we sample the ﬁrst points uniformly in a subspace S .
Line 3-7: We construct { Df n (cid:15) ( X 1 ) , ..., Df n (cid:15) ( X N ) } , and then d to be able to sample points accordingly.
Line 8: Because the support of a GMM is not bounded, some points can be sampled outside S .
We discard these points and sample until all points are inside S .
This rejection method is equivalent to sampling points from a truncated GMM.
Line 9-10: We construct the labels and add the new points to the initial data set.
Algorithm
Sampling
L 2 error
L ∞ f : Runge ( ×
[Table]
In order to illustrate the beneﬁts of TBS compared to a uniform, basic sampling (BS), we apply it to two simple functions: hyperbolic tangent and Runge function.
We chose these functions because they are differentiable and have a clear distinction between ﬂat and steep regions.
These functions are displayed in Figure 4, as well as the map x → Df 2 (cid:15) ( x ) .
All NN have been implemented in Python , with Tensorflow Abadi et al.
We use the Python package scikit-learn Pedregosa et al.
(2011), and more speciﬁcally the GaussianMixture class to construct d GMM .
The network chosen for this experiment is a Multi Layer Perceptron (MLP) with one layer of 8 neurons, with Relu activation function, that we trained alternatively with BS and TBS using
1.0 x 0.
f ( x ) 0.
D f 2 ( x ) 0.
1.0 x 0.
f ( x ) 0.
D f 2 ( x ) Adam optimizer Kingma & Ba (2014) with the defaults tensorﬂow implementation hyperparameters, and Mean Squared Error loss function.
We ﬁrst sample
{ X 1 , ..., X N } according to a regular grid.
To compare the two methods, we add N (cid:48) additional points sampled using BS to create the BS data set, and then N (cid:48) other points sampled with TBS to construct the TBS data set.
As a result, each data set have the same number of points ( N + N (cid:48) ) .
We repeated the method for several values of n , n GMM and (cid:15) , and ﬁnally selected n = 2 , n GMM = 3 and (cid:15) = 10 − 3 .
Table 6 summarizes the L 2 and the L ∞ norm of the error of f θ , obtained at the end of the training phase for N + N (cid:48)
= 16 , with N = 8 .
Those norms are estimated using a same test data set of 1000 points.
The values are the means of the 40 independent experiments displayed with a 95% conﬁdence interval.
These results illustrate the beneﬁts of TBS over BS.
Table 6 shows that TBS slightly degrades the L 2 error of the NN, but improves its L ∞ error.
This may explain the good results of VBSW for classiﬁcation.
Indeed, for a classiﬁcation task, the accuracy will not be very sensitive to small output variations, since the output is rounded to 0 or 1.
However, a high error can induce a misclassiﬁcation, and the reduction in L ∞ norm limits this risk.
A PPLICATION TO AN ODE SYSTEM
We apply TBS to a more realistic case: the approximation of the resolution of the Bateman equations, which is an ODE system :
[Equation]
with initial conditions
[Equation]
with u ∈ R + , η ∈ ( R + ) M , σ Ta ∈ R M , Σ r ∈ R M × M .
Here, f : ( u 0 , η 0 , t ) → ( u ( t ) , η ( t )) .
For physical applications, M ranges from tens to thousands.
We consider the particular case M =so that f : R 3 → R 2 , with f ( u 0 , η 0 , t ) =
( u ( t ) , η ( t )) .
The advantage of M = 1 is that we have access to an analytic, cheap to compute solution for f .
Of course, this particular case can also be solved using a classical ODE solver, which allows us to test it end to end.
It can thus be generalized to higher dimensions ( M > 1 ).
All NN trainings have been performed in Python , with Tensorflow Abadi et al.
We used a fully connected NN with hyperparameters chosen using a simple grid search.
The ﬁnal values are: 2 hidden layers, ”ReLU” activation function, and 32 units for each layer, trained with the Mean Squared Error (MSE) loss function using Adam optimization algorithm with a batch size of 50000, for 40000 epochs and on N + N (cid:48)
= 50000 points, with N = N (cid:48) .
We ﬁrst trained the model for ( u ( t ) , η ( t ))
∈ R , with an uniform sampling (BS) ( N (cid:48) = 0 ), and then with TBS for several values of n , n GMM and (cid:15) = (cid:15) (1 , 1 , 1) , to be able to ﬁnd good values.
We ﬁnally select (cid:15) =
5 × 10 − 4 , n = 2 and n GMM = 10 .
The data points used in this case have been sampled with an explicit Euler scheme.
This experiment has been repeated 50 times to ensure statistical signiﬁcance of the results.
Table 7 summarizes the MSE, i.e. the L 2 norm of the error of f θ and L ∞ norm, with L ∞ ( θ ) = max X ∈ S ( | f ( X ) − f θ ( X ) | ) obtained at the end of the training phase.
This last metric is important because the goal in computational physics is not only to be averagely accurate, which is measured with MSE, but to be accurate over the whole input space S .
Those norms are estimated using a same test data set of N test = 50000 points.
The values are the means of the 50 independent experiments displayed with a 95% conﬁdence interval.
These results reﬂect an error reduction of 6.6% for Land of 45.3% for L ∞ , which means that TBS mostly improves the L ∞ error of f θ .
Moreover, the
L ∞ error conﬁdence intervals do not intersect so the gain is statistically signiﬁcant for this norm.
[Table]
[Figure] [Figure]
[Figure]
Figure 2a
[Figure]
Figure 2b
[Figure]
r ga i n
Figure 1a shows how the NN can perform for an average prediction.
Figure 1b illustrates the beneﬁts of TBS relative to BS on the L ∞ error (Figure 2b).
These 2 ﬁgures conﬁrm the previous observation about the gain in L ∞ error.
Finally, Figure 2a displays u 0 , η 0 → max 0 ≤ t ≤ 10 D n (cid:15) ( u 0 , η 0 , t ) w.r.t.
( u 0 , η 0 ) and shows that D n (cid:15) increases when U 0 → 0 .
TBS hence focuses on this region.
Note that for the readability of these plots, the values are capped to 0 .
Otherwise only few points with high D n (cid:15) are visible.
Figure 2b displays u 0 , η
→ g θ BS ( u 0 , η 0 ) − g θ TBS ( u 0 , η 0 ) , with g θ : u
[Table]
where θ BS and θ TBS denote the parameters obtained after a training with BS and TBS, respectively.
It can be interpreted as the error reduction achieved with TBS.
The highest error reduction occurs in the expected region.
Indeed, more points are sampled where
D n (cid:15) is higher.
The error is slightly increased in the rest of S , which could be explained by a sparser sampling on this region.
However, as summarized in Table 1 , the average error loss (AEL) of TBS is around six times lower than the average error gain (AEG), with AEG = E u 0 ,η 0 ( Z 1 Z> 0 ) and
AEL = E u 0 ,η
( Z 1 Z< 0 )
where Z ( u 0 , η 0 ) =
g θ BS ( u 0 , η 0 ) − g θ TBS ( u 0 , η 0 ) .
In practice, AEG and AEL are estimated using uniform grid integration, and averaged on the 50 experiments.
A PPENDIX B: D EMONSTRATIONS I NTUITION BEHIND T AYLOR E XPANSION (S ECTION 3.2)
We want to approximate f : x → f ( x ) , x ∈ R n i , f ( x ) ∈ R n o with a NN f θ .
The goal of the approximation problem can be seen as being able to generalize to points not seen during the training.
We thus want the generalization error J X ( θ ) to be as small as possible.
Given an initial data set { X 1 , ..., X N } drawn from X ∼ d P X and { f ( X 1 ) , ..., f ( X N ) } , and the loss function L being the squared L 2 error, recall that the integrated error J X ( θ ) , its estimation (cid:99) J X ( θ ) and the generalization error J X ( θ ) can be written:
[Equation]
where (cid:107) .
(cid:107) denotes the squared L 2 norm.
In the following, we ﬁnd an upper bound for J X ( θ ) .
We start by ﬁnding an upper bound for J
X ( θ ) and then for J X ( θ ) using equation 7.
Let S i , i ∈ { 1 , ..., N } be some sub-spaces of a bounded space S such that S
= (cid:83) Ni
=1 S i , (cid:84) Ni
=1 S i
=
Ø, and X i ∈ S
i .
Then,
[Equation]
X
[Equation]
Suppose that n i = n o = 1 and f twice differentiable.
Let | S | = (cid:82) S d P X .
The volume | S | = 1 since d P X is a probability measure, and therefore | S i | < 1 for all i ∈ { 1 , ..., N } .
Using Taylor expansion at order 2, and since | S i | < 1 for all i ∈ { 1 , ..., N } J X ( θ ) =
N
[Equation]
To ﬁnd an upper bound for J ( θ ) , we can ﬁrst ﬁnd an upper bound for |
A
i ( x ) | , with A i ( x ) =
f ( X i ) + f (cid:48)
( X i )( x − X i )
+ 12 f (cid:48)(cid:48)
[Table]
x
[Table]
NN f θ is K θ − Lipschitz, so since S is bounded (so are S i ), for all x ∈ S i ,
| f θ ( x ) − f θ ( X i ) | ≤ K θ | x − X i | .
Hence,
[Equation]
And ﬁnally, using triangular inequality,
[Equation]
Now, (cid:107)
[Equation]
being the squared L 2 norm:
[Equation]
Hornik’s theorem (Hornik et al., 1989) states that given a norm (cid:107) .
(cid:107) p,µ = such that (cid:107) f [-θ ( X i ) |+ 
-](cid:107) pp,µ
=
(cid:82)
S | f ( x )[-|+ 
-] | p dµ ( x ) , with dµ a probability measure, for any (cid:15) , there exists θ such that for a Multi
[-− X i | (cid:0) 1 
-]
[-f θ ( X i ) |+ 
-]
Layer Perceptron, f θ , (cid:107) f ( x ) − f θ ( x ) (cid:107) pp,µ < (cid:15) ,
This theorem grants that for any (cid:15) , with dµ = (cid:80) Ni
=
N δ ( x − X i ) , there exists θ such that [Equation]
Let’s consider i ∗ such that i ∗ = argmin | S i | . For any i ∈ { 1 , ..., N } , O ( | S ∗ i | 4 )
=
O ( | S i | 4 ) .
Now, let’s choose (cid:15) such that (cid:15)
= O ( | S ∗ i | 4 ) . Then, for any i ∈ { 1 , ..., N } , O ( | S ∗ i | 4 ) =
O ( | S i | 4 ) so equation 8 implies that
[Equation]
Thus, J X ( θ ) = J X ( θ ) − (cid:99)
J X ( θ ) =
J X ( θ ) + O ( | S i | 4 ) and therefore,
[Equation]
Finally,
[Equation]
We see that on the regions where f (cid:48)
( X i ) + K θ is higher, quantity | S i | (the volume of S i ) has a stronger impact on the GB.
Then, since | S i | can be seen as a metric for the local density of the data set (the smaller | S i | is, the denser the data set is), the Generalization Bound (GB) can be reduced more efﬁciently by adding more points around X i in these regions.
This bound also involves K θ , the Lipschitz constant of the NN, which has the same impact as f (cid:48)
( X i ) .
It also illustrates the link between the Lipschitz constant and the generalization error, which has been pointed out by several works like, for instance, Gouk et al.
, Bartlett et al.
and Qian & Wegman (2019).
P ROBLEM 1: U NAVAILABILITY OF DERIVATIVES (S ECTION 4.1)
In this paragraph, we consider n i > 1 but n o = 1 .
The following derivations can be extended to n o > 1 by applying it to f element-wise.
Let (cid:15) ∼ N (0 , (cid:15) 1 n i ) with (cid:15) ∈ R + and (cid:15) =
( (cid:15) 1 , ..., (cid:15) n i ) , i.e. (cid:15) i ∼ N (0 , (cid:15) ) .
Using Taylor expansion on f at order 2 gives
[Equation]
With ∇ x f and H x f ( x ) the gradient and the Hessian of f w.r.t.
x .
We now compute V ar ( f ( X + (cid:15) )) and make Df 2 (cid:15) ( x ) = (cid:15) (cid:107)∇ x f ( x ) (cid:107) 2 F + 12 (cid:15) 2 (cid:107) H x f ( x ) (cid:107) 2 F appear in its expression to establish a link between these two quantities:
[Equation]
Since (cid:15) i ∼ N (0 , (cid:15) ) , x = ( x 1 , ..., x n i ) and with ∂ 2 f ∂x i x j ( X ) the cross derivatives of f w.r.t.
x i and x j ,
[Equation]
[Table]
1 , (cid:15) i 2 (cid:17) + n i (cid:88)
j 1 = n i (cid:88) k
[Equation]
n i (cid:88)
j
[Equation]
= n i (cid:88)
[Equation]
=
[Equation]
f
[Equation]
x )
[Equation]
f
[Equation]
+
[Equation]
f
[Equation]
(cid:16) (cid:15)
[Equation]
(cid:15)
[Equation]
(cid:15)
[Equation]
(cid:17) .
In this expression, we have to assess three quantities:
Cov ( (cid:15) i 1 , (cid:15) i 2 ) , Cov ( (cid:15) i , (cid:15) j (cid:15) k ) and
Cov ( (cid:15) j 1 (cid:15) k 1 , (cid:15)
j 2 (cid:15) k 2 ) .
First, since ( (cid:15) 1 , ..., (cid:15) n i ) are i.i.d.,
Cov (cid:16) (cid:15) i 1 , (cid:15) i 2 (cid:17) = (cid:26) V ar ( (cid:15) i ) =
(cid:15) if i 1 =
i
= i, 0 otherwise.
To assess Cov ( (cid:15) i , (cid:15) j (cid:15) k ) , three cases have to be considered.
• If i = j = k , because E [ (cid:15) 3 i ] = Cov [Equation] = Cov
[Equation]
• If i = j or i =
k (we consider i =
k , and the result holds for i =
j by commutativity),
[Equation]
• If i (cid:54) =
j and i (cid:54)
=
k , (cid:15) i and (cid:15) j (cid:15) k are independent and so Cov ( (cid:15)
i , (cid:15) j (cid:15) k )
= 0.
Finally, to assess Cov ( (cid:15) j 1 (cid:15) k 1 , (cid:15) j 2 (cid:15) k 2 ) , four cases have to be considered:
• If j 1 = j 2 = k 1 = k 2 =
i ,
[Equation]
ar
[Equation] [Equation] [Equation] • If j 1 = k {+
+}
= i and j 2 = k 2 =
j , Cov ( (cid:15)
j 1 (cid:15) k 1 , (cid:15) j 2 (cid:15) k 2 ) =
Cov ( (cid:15) 2 i , (cid:15) 2 j ) = 0 since (cid:15) 2 i and (cid:15) 2 j are independent.
• If j 1 = j 2 = j and k [Equation] k
[Equation]
k
[Equation]
• If j 1 (cid:54) =
k 1 , j 2 and k 2 ,
[Equation]
All other possible cases can be assessed using the previous results, commutativity and symmetry of
Cov operator.
Hence,
[Equation]
And ﬁnally,
[Equation]
If we consider (cid:100) Df 2 (cid:15) ( x ) as deﬁned in equation 2, on section 3.3 of the main document, (cid:100) Df 2 (cid:15) ( x ) → k →∞ V ar ( f ( x + (cid:15) )) .
Since V ar ( f ( x + (cid:15) ))
=
Df 2 (cid:15) ( x ) + O ( (cid:107) (cid:15) (cid:107) 32 ) , (cid:100) Df 2 (cid:15) ( x ) is a biased estimator of Df 2 (cid:15) ( x ) , with bias O ( (cid:107) (cid:15) (cid:107) 32 ) .
Hence, when (cid:15) → 0 , (cid:100) Df 2 (cid:15) ( x ) becomes an unbiased estimator of Df 2 (cid:15)
A PPENDIX C: H YPERPARAMETERS
This appendix Section is split in two parts.
First, we describe the results of the experiments on the hyperparameters search of Boston Housing (BH) and Breast Cancer (BC) data sets (Section 5.1).
The second part is a list of ﬁnal hyperparameters values chosen for the experiments of the main paper.
E XPERIMENTS ON B OSTON H OUSING AND B REAST C ANCER DATA SETS
For BH and BC experiments, we conduct a grid search for VBSW on the values of m and k .
As a reminder, m is the ratio between the highest and the lowest weights, and k is the number of neighbor points used to compute the local variance.
We train a linear model for BH and a MLP with 30 units for BC with VBSW on a grid of 20 values of m equally distributed between 2 and 100 and 20 values of k equally distributed between 10 and 50 .
As a result, we train the model on 400 pairs of ( m, k ) values, and with 10 different random seeds for each pair.
These experiments, illustrated in Figure 6 shows that the inﬂuence of m and k on the performances of the model can be different.
For BH data set, low values of k clearly lead to poorer performances
Hyperparameter m seems to have less impact, although it should be chosen not too far from its lowest value, 2 .
For BC data set, at the contrary, the best performances are obtained for low values of k , while m could be chosen in high values.
These experiments highlight that the impact of m and k can be different between classiﬁcation and regression, but it could also be different depending on the data set.
Hence, we recommend considering these hyperparameters like many other involved in
DL, and to select their values using classical hyperparameters optimization techniques.
This also shows that many different ( m, k ) pairs lead to error improvement.
This suggests that the weights approximation
P APER HYPERPARAMETERS VALUES
The values chosen for the hyperparameters of the paper experiments are gathered in Table 8.
For
ADAM optimizer hyperparameters, we kept the default values of Keras implementation.
We chose these hyperparameters after simple grid searches
[Table]