Not too close and not too far: enforcing monotonicity requires penalizing the right points
Abstract
In this work, we propose a practical scheme to enforce monotonicity in neural networks with respect to a given subset of the dimensions of the input space.
The proposed approach focuses on the setting where point-wise gradient penalties are used as a soft constraint alongside the empirical risk during training.
Our results indicate that the choice of the points employed for computing such a penalty deﬁnes the regions of the input space where the desired property is satisﬁed.
As such, previous methods result in models that are monotonic either only at the boundaries of the input space or in the small volume where training data lies.
Given this, we propose an alternative approach that uses pairs of training instances and random points to create mixtures of points that lie inside and outside of the convex hull of the training sample.
Empirical evaluation carried out using different datasets show that the proposed approach yields predictors that are monotonic in a larger volume of the space compared to previous methods.
Our approach does not introduce relevant computational overhead leading to an efﬁcient procedure that consistently achieves the best performance amongst all alternatives.
Introduction
Highly expressive model classes such as artiﬁcial neural networks, trained under the empirical risk minimization framework, have achieved impressive prediction performance across a broad range of supervised learning tasks and domains such as object recognition [12], speech recognition [9], machine translation [2], among many other examples.
However, ﬁnding predictors attaining low risk on unseen data is often not enough to enable the use of such models in practice.
In fact, practical applications usually have more requirements beside the prediction accuracy.
As such, devising approaches that search risk minimizers satisfying practical needs led to several research threads seeking to enable the use of neural networks in real-life scenarios.
Examples of such requirements include:
• Robustness , where low risk is expected even if the model is evaluated under distribution shifts or adversarial perturbations.
• Fairness , where the performance of the model is expected to not signiﬁcantly change across different sub-populations of the data.
• Explainability/Interpretability , where models are expected to indicate how the features of the data affect the predictions of the model.
In addition to the requirements mentioned above, a property commonly expected in trained models in certain applications is monotonicity with respect to some subset of the input dimensions, i.e., an increase (or decrease) along some particular dimensions strictly imply the function value will not decrease (or will not increase), provided that all other dimensions are kept ﬁxed.
As a result, the behavior of monotonic models will be more aligned with properties that the data under consideration is believed to satisfy.
For example, in the case of models used to accept/reject job applications, we expect acceptance scores to be monotonically non-decreasing with respect to features such as past years of experience of a candidate.
Thus, given two applicants with exactly the same features, if one of them has more years of experience, then they will have higher, or at least the same, chance of getting accepted.
Similar observations can be made when applying machine learning models to other areas such as loan applications.
For applications where monotonicity is expected, models failing to satisfy this requirement would damage the user’s conﬁdence.
In addition, monotonicity was also observed to be a good inductive bias in order to improve generalization in cases where prior knowledge indicates the data generation process satisﬁes such property [5].
Given the importance of enforcing monotonicity properties in numerous applications, different strategies have been devised in order to enable learning of monotonic predictors.
These approaches can be divided into two main categories:
• Monotonicity by construction: This approach focuses on deﬁning a model class that guarantees monotonicity in all of its elements [3, 17, 14, 18, 7, 1].
However, this approach can not be used with general architectures in the case of neural networks.
Additionally, the model class might be so constrained that it might affect the prediction performance.
Encouraging monotonicity during model training: This approach searches for monotonic candidates within a general class of models [13, 15, 10].
Such group of methods is more generally applicable and can be used, for instance, with any neural network architecture.
However, they are not guaranteed to yield monotonic predictors unless the monotonicity of the trained model is veriﬁed/certiﬁed, which can be computationally expensive.
Moreover, several retraining cycles might be required to enforce monotonicity.
In this work, we focus on the latter and tackle the problem of performing empirical risk minimization in large classes of models such as neural networks, while simultaneously searching for monotonic predictors within the set of risk minimizing solutions.
Speciﬁcally, we revisit methods that introduce monotonicity-enforcing regularization penalties.
Our contributions can be summarized as follows:
We show that previous methods can only satisfy monotonicity either near the training data or near the boundaries of the input domain.
To the best of our knowledge, this is the ﬁrst work that raises the issue of the effect of the points used in calculating the monotonicity constraint and proposes an efﬁcient algorithm that tackles the problem.
We propose an efﬁcient approach that, given the same budget, can enforce monotonicity in a larger volume of the input space compared to the previous methods in literature.
The remainder of this paper is organized as follows: in Section 2, we provide background and formally introduce the notion of partial monotonicity under consideration.
The related literature is also discussed in this section.
The proposed approach is deﬁned in Section 3, where we also discuss and explain the issues in the prior methods upon which we improve.
Empirical evaluation is performed in Section 4, while conclusions are summarized in Section 5 along with a discussion on follow-up work.
Background and related work
We start by deﬁning the notion of partial monotonicity used throughout the paper.
Consider the standard supervised learning setting where data instances are observed in pairs x, y ∼ X × Y , where
X ⊂ R d and Y ⊂ R correspond to the input and output spaces, respectively.
Further consider the function f : X (cid:55)→ Y , and let M indicate some subset of the input dimensions, i.e. M ⊂ { 1 , ...
d } , such that x = concat ( x M , x ¯ M ) , where ¯ M = { 1 , ..., d } \ M .
We further overload the notation indicating function calls to f such that f ( x ) =
f
( x M , x ¯ M ) .
Deﬁnition Partially monotonic functions relative to M : We say f is monotonically non-decreasing relative to M 2 , denoted f M , if f ( x M , x ¯ M ) ≤ f ( x (cid:48)
M , x ¯ M ) , ∀ x M ≤ x (cid:48)
M , ∀ x ¯ M , where the comparison x M ≤ x (cid:48)
M is performed for every dimension.
The above deﬁnition covers functions that do not decrease in value given increasing changes along a subset of the input dimensions, provided that all other dimensions are kept unchanged.
Several approaches were introduced for deﬁning model classes that have such property.
The simplest approach restricts the weights of the network to be non-negative [1].
However, doing so affects the prediction performance.
Another approach corresponds to using the lattice regression models proposed in [7] and recently extended in [3, 18].
In such case, models are given by interpolations in a grid deﬁned by training data.
While such class of models can be made monotonic via the choice of the interpolation strategy, it scales poorly with the dimension of the input space.
For neural networks, approaches such as [14] reparameterizes fully connected layers such that gradients with respect to parameters can only be non-negative.
In [17], on the other hand, the class of predictors H : X (cid:55)→ Y of the form H ( x ) = (cid:82) x 0 h ( t ) dt + H (0) is deﬁned, where h ( t ) is parameterized by a neural network.
Given that h ( t ) is strictly positive, which can be easily enforced via the choice of the activation function, the resulting H ( x ) will be montonically non-decreasing.
While such approaches guarantee monotonicity by design, they can be too restrictive or render overly complicated learning procedures.
For example, the approach in [17] requires performing backpropagation through the integration operator.
An alternative approach to that of constraining the model class is to search over general classes of models while assigning higher importance to candidate predictors that are observed to be monotonic in certain parts of the space.
Similar to the case of adversarial training [8], Sivaraman et al.
[15] proposed an approach that ﬁnds counterexamples, i.e. pair of points where the monotonicity constraint is violated.
The counterexamples are then included in the training data with adjustments to their target values so as to enforce next iterates of the model will satisfy the monotonicity conditions for them.
However, this approach only supports fully-connected ReLU networks.
Moreover, the procedure for ﬁnding the counterexamples is costly.
Alternatively, a regularization penalty is introduced in both
Liu et al.
[13] and Gupta et al.
[10] so that monotonicity is point-wise enforced during training.
This requires selecting the points where the penalty is computed during training.
While Liu et al.
[13] use random draws from a uniform distribution over X , Gupta et al.
[10] apply the regularization penalty over the training instances only.
As will be further discussed, both approaches have shortcomings that we seek to address in this contribution.
Proposed approach
Given the standard supervised learning setting where (cid:96) :
Y 2 (cid:55)→ R + is a loss function indicating the goodness of the predictions relative to ground truth targets, the goal is to ﬁnd a predictor h ∈ H such that its expected loss – or the so-called risk – over the input space is minimized.
Such an approach yields the empirical risk minimization framework once a ﬁnite sample is used to estimate the risk.
However, given the extra monotonicity requirement, we consider an augmented framework where such property is further enforced.
We thus seek the optimal monotonic predictors relative to M – denoted h ∗ M – such that:
[Equation]
where γ is a hyperparameter weighing the importance of the penalty Ω( h, M ) which, in turn, is a measure of how monotonic the predictor h is relative to the dimensions indicated by M .
Ω( h, M ) can be deﬁned by the following gradient penalty [10, 13]:
[Equation]
where ∂h ( x ) ∂x i indicates the gradients of h relative to the input dimensions i ∈ M , which are constrained to be non-negative, rendering h monotonically non-decreasing relative to M .
At this point, the only missing ingredient to deﬁne a practical algorithm to estimate h ∗ M in Eq.
1 is how to deﬁne the distribution D over which the expectation in Eq.
2 is computed.
In the following, ﬁrst we discuss the choices of D in prior work and the consequential issues of such choices.
Next, we propose alternatives to alleviate these issues.
3.1 Choosing distributions over which to compute Ω
Past choices for D are such that:
Deﬁne D as the empirical distribution of the training sample : In [10], given a training dataset of size N , in addition to using the observed data to estimate the risk, the same data is used to compute the monotonicity penalty so that:
[Equation]
where x k indicates the k -th instance within the training sample.
While this choice seems natural and can be easily implemented, it only enforces monotonicity in the region where the training samples lie, which can be problematic.
For example, in case of covariate-shift, the test data might lie in parts of the space different from that of the training data so monotonicity cannot be guaranteed.
We thus argue that one needs to enforce the monotonicity property in a region larger than what is deﬁned by the training data.
In Section 4.1, we conduct an evaluation under domain shift and show the issue to become more and more relevant with the increase in the dimension d of the input space X . 2.
Deﬁne D = Uniform ( X )
: In [13], a simple strategy is deﬁned so that Ω is computed over the random points drawn uniformly across the entire input space X ; i e.:
[Equation]
Despite its simplicity and ease of use this approach has some
ﬂaws. In high-dimensional spaces, random draws from any distribution of bounded variance will likely lie in the boundaries of the space, hence far from the regions where data actually lie.
Moreover, it is commonly observed that naturally occurring high-dimensional data is structured in lowerdimensional manifolds (c.f.
[6] for an in-depth discussion on the manifold hypothesis).
It is thus likely that random draws from the uniform distribution will lie nowhere near regions of space where training/testing data will be observed.
We further illustrate the issue with an example in Section 3.2, which can be summarized as follows: consider the cases of uniform distributions over the unit n -sphere.
In such case, the probability of a random draw lying closer to the sphere’s surface than to its center is P ( || x || 2 > 12 ) =
2 n − 1 2 n , as given by the volume ratio of the two regions of interest.
Note that P ( || x || 2 > 12 ) → 1 as n → ∞ , which suggests the approach in [13] will only enforce monotonicity at the boundaries of the space.
In summary, the previous approaches are either too focused in enforcing monotonicity where the training data lie, or too loose such that the monotonicity property is uniformly enforced across a large space, and the actual data manifold may be neglected.
We thus propose an alternative to such approaches where we can have some control over the volume of the input space where the monotonicity property will be enforced.
Our approach uses the idea of data mixup [19], where auxiliary data is created via interpolations of pairs of data points. 2 Mixtures of points as a means to enlarge the region where Ω is computed
Mixup was introduced in [19] with the goal of training classiﬁers whose outputs are smooth across trajectories in the input space from instances of different classes.
Given a pair of data points ( x (cid:48) , y (cid:48) ) , ( x (cid:48)(cid:48) , y (cid:48)(cid:48) ) , the method augments the training data using interpolations given by ( λx (cid:48)
+ (1 − λ )
x (cid:48)(cid:48) , λy (cid:48)
+ (1 − λ )
y (cid:48)(cid:48) ) , where λ ∼ Uniform ([0 , 1]) .
The approach was later extended in [16] so that the smoothness property is further imposed in high-level learned representations.
Moreover, mixup is used to enforce fairness constraints in [4] by forcing training algorithms to give preference to predictors that behave smoothly in the space in between sub-populations of the data.
[Figure]
In this work, we propose using mixup to generate points where the monotonicity penalty Ω can be computed and highlight the following motivations for doing so:
Interpolation: this is obtained by mixing-up pairs of data instances (i.e. when 0 < λ < 1 ), which more densely populates the convex hull of the training data.
We argue doing so is beneﬁcial in helping to achieve monotonicity in the entirety of the hull rather than only on the training data itself as in [10]. 2.
Extrapolation : mixing up data points with boundary instances obtained at random from Uniform ( X ) enables extrapolation in the sense that resulting mixtures of points can lie anywhere between the data manifold and the boundaries of the space.
This encourages monotonicity on points that are outside the hull made of the training sample.
We thus claim – and report empirical evidence – that performing mixup enables the computation of Ω on parts of the space that are disregarded if one focus only on either observed data or random draws from uninformed choices of distributions such as the uniform.
To further illustrate the issue discussed in the item 2 of Section 3.1 as well the effect of our proposal, we discuss a simple example considering random draws from the unit n -sphere, shown in Figure 1, i.e., the set of points B = { x ∈ R n : || x
|| 2 < 1 } .
We further consider a concentric sphere of radius 0 < r < 1 given by B r = { x ∈ R n : || x || 2 < r } .
We are interested in the probability of a random draw from B to lie outside of B r , i.e.: P ( || x || 2 > r ) , x ∼ D ( B ) , for some distribution D .
We start by deﬁning D as the Uniform ( B ) , which results in P ( || x || 2 > r )
= 1 − r n .
In Figure 2a, we can see that for growing n , P ( || x || 2 > r ) is very large even if r ≈ 1 , which suggests most random draws will lie close to B ’s boundary.
We now evaluate the case where mixup is applied and random draws are taken in two steps: we ﬁrst observe y ∼ Uniform ( B ) , and then we perform mixup between y and the origin 3 , i.e. x = λy , λ ∼ Uniform ([0 , 1]) .
In this case, P ( || x || 2 > r ) =
(1 − r n )(1 − r ) , which is shown in Figure 2b as a function of r for increasing n .
We can then observe that even for large n , P ( || x || 2 > r ) decays linearly with r , i.e., we populate the interior of B and x in this case follows a non-uniform distribution such that its norms histogram is uniform.
In the next section, we perform an empirical evaluation where we compare the predictors resulting from different choices for computing Ω during training.
Moreover, we evaluate the strategy described above and observe that combining random and data points yields monotonicity in a larger volume of
X compared to existing algorithms.
Evaluation
In order to evaluate the effect of the choices of Ω as discussed in Section 3, we carry out an empirical study which is divided into two parts: we start by analyzing proof-of-concept experiments using synthetic data, generated in such a way that one can control the dimension of X as well as the number of dimensions with respect to which monotonicity is expected.
After that, we report results oncommonly used datasets covering classiﬁcation and regression settings with input spaces of different
1.0 r 0.
p ( | x | > r ) n=2 n= n=18 n= n=34 n= n=50 n= n=66 n= n [Figure]
1.0 r 0.
p
_ l a m bd a ( | x | > r ) n=2 n=
n=18 n= n=34 n= n=50 n= n=66 n= n
[Figure]
[Figure]
[Figure]
dimensions
In both are implemented using the in whose weights are kept separate in early layers for the input dimensions with respect to which monotonicity is to be enforced.
We set the depth of all networks to and use a bottleneck of size 10 for two datasets (Compas and Loan Lending Club) and 100 for the case of the Blog
Feedback dataset the experiments on generated data
Training is carried out with the Adam optimizer [11] with a global learning rate of 5e − 3 , and γ is set to 1e4 .
The training batch size is set to 256 throughout experiments.
For all evaluation cases, we consider the baseline where training is carried out without any monotonicity enforcing penalty.
For the regularized cases, the different approaches used for computing Ω are as follows:
Ω random ([13]): uses random points drawn from Uniform ( X ) .
In this case, the sample observed at each training iteration is set to a size of 1024 throughout all experiments.
Ω train ([10]): uses the actual data observed at each training iteration;
i
e., the observed mini-batch itself is used to compute Ω . 3.
Ω mixup : in this case, the penalty is computed on points generated by mixing-up points from the training data and random points.
In further details, for each mini-batch of size N > 1 , we augment it with complementary random data and obtain a ﬁnal mini-batch of size 2 N .
Out of the 2 N (2 N − 1)possible pairs of points, we take a random subsample of 1024 pairs to compute mixtures of instances.
In this case, we use λ ∼ Uniform([0,1]) and λ is independently drawn for each pair of points.
Results are reported in terms of both prediction performance and level of monotonicity.
The latter is assessed via the fraction ρ of points within a sample where the monotonicity constraint is violated; i.e., given a set of N data points, we compute:
[Equation]
such that ρ = 0 corresponds to monotonic models over the considered points.
In order to quantify the degree of monotonicity in different parts of the space, we compute ρ for 3 different sets of points:
ρ random : computed on a sample drawn according to Uniform ( X ) .
We used a sample of 10,000 points throughout the experiments.
ρ train : computed on the training data.
ρ test : computed on the test data. [Table] [Table]
of non-monotonic points ρ
for models trained on generated data in spaces of growing dimension ( D ) and number of monotonic dimensions ( | M | ).
Different regularization strategies is effective on only one of ρ random or ρ test , while Ω mixup seems effective throughout conditions.
4.1 Proof-of-concept evaluation
We start by describing the approach we employ to generate data containing the properties required by our evaluation.
Denote a design matrix by X N × D such that each of its N rows corresponds to a feature vector within R D .
In order to ensure the data lies in some manifold, we ﬁrst obtain a low-dimensional synthetic design matrix given by X (cid:48)
N × d , where each entry is sampled randomly from Uniform ([ − 10 , 10]) .
We then expand it to R D by applying the following transformation:
[Equation]
where the expansion matrix given by A d × D is such that each of its entries are independently drawn from Uniform ([0 , 1]) .
Throughout our experiments, d = (cid:98) 0 .
3 D (cid:99) was employed.
Target values for the function f to be approximated are deﬁned as sums of functions of scalar arguments applied independently over each dimension.
We thus select a set of dimensions M ∈ [ D ] with respect to which f is to be monotonic, i.e.:
[Equation]
and every g i : R (cid:55)→ R is increasing monotonic, while every h i : R (cid:55)→ R is not monotonic.
We then create two evaluation datasets.
One of them, referred to as the validation set, is identically distributed with respect to X since it is obtained following the same procedure discussed above.
In order to simulate covariate-shift, we create a test set by changing the expansion matrix A to a different one.
[Equation]
where A test will be given by entry-wise linear interpolations between A , used to generate the training data, and a newly sampled expansion matrix A (cid:48) :
A test = αA (cid:48) + (1 − α )
A .
The parameter α ∈ [0 , 1] , set to 0 .
8 in the reported evaluation, controls the shift between A and A test in terms of the Frobenius norm, which, in turn, enables the control of how much the test set shifts relative to the training data.
We thus trained models to approximate f for spaces of increasing dimensions as well as for an increasing number of dimensions with respect to which f is monotonic.
Results are reported in Table 1 in terms of RMSE on the two evaluation datasets, and in terms of monotonicity in Table 2 where ρ is computed both on random points and on the shifted test set.
Entries in the tables correspond to the centers of 95% conﬁdence intervals resulting from 20 independent training runs.
Throughout experiments, models follow the architecture in [13] using dense layers whose weights are kept separate in early layers for the input dimensions with respect to which monotonicity is to [Table] [Table] enforced.
We set the depth of all networks
[Table]
and bottleneck of size proof of concept and Loan Lending
[Table]
case the experiments on generated data
Training is carried out the Adam optimizer [ learning rate
5e is
We highlight the two following observations regarding the prediction performances shown in table 1: different models present consistent performances across evaluations, which suggests different monotonicity-enforcing penalties do not signiﬁcantly affect prediction accuracy.
Moreover, the proposed approach used to generate test data under covariate-shift is effective given the gap in performance consistently observed between the validation and the test partitions.
In terms of monotonicity, results in Table 2 suggest that Ω random and Ω train are only effective on either random or data points, which seems to aggravate when the dimension D grows.
Ω mixup , on the other hand, is effective on both sets of points, and continues to work well for growing D .
Furthermore, covariate-shift affects Ω train for higher-dimensional cases, while Ω mixup performs well in such case.
4.2 Benchmarks
We now proceed and evaluate our models on real data.
Three often used benchmarks are employed for this set of experiments, and the datasets are described in further details in Table 3.
Results are summarized in Tables 4, 5, and 6 in terms of both prediction performance along with the metric indicating the degree of monotonicity of the predictor for each regularization strategy.
Prediction performance is measured in terms of accuracy for classiﬁcation tasks, and RMSE for the case of regression.
Monotonicity, on the other hand, is quantiﬁed via the fraction ρ of points within a sample where the monotonicity constraint is violated.
Results reported in the tables correspond to 95% conﬁdence intervals corresponding to multiple independent training runs.
We start the analysis of the results by comparing the prediction performances of the models obtained under the different regularization strategies with the unregularized baseline.
In this case, we observe that across all datasets, the different penalties do not result in signiﬁcant variations in the performance of the ﬁnal predictors.
This indicates that the class of predictors corresponding to the subset of H that is monotonic relative to M , denoted by H M , has enough capacity so as to be able to match the performance of the best canditates within H .
In terms of monotonicity, we observe a clear pattern leading to the following intuition: monotonicity is achieved in the regions where it is enforced .
This is evidenced by the observation that ρ random is consistently lower for Ω random relative to Ω train and Ω mixup while, on the other hand, ρ train and ρ test are consistently lower for Ω train and Ω mixup compared to Ω random .
Moreover, a comparison between Ω train and Ω mixup shows what we anticipated: enforcing monotonicity in points resulting from mixup yields predictors that are as
[Table]
[Table] training runs. Results correspond to the checkpoint that obtained the best prediction performance on validation data throughout training monotonic as those given by the use of Ω train in actual data, but signiﬁcantly better at the boundaries of X .
Finally, the results demonstrate that our proposed approach Ω mixup − TR achieves the best results in terms of monotonicity for all the sets of points that we considered.
Moreover, our approach introduces no signiﬁcant computation overhead.
Conclusion
We studied the supervised learning setting where the outputs of the model are expected to be monotonic with respect to some subset of the input space dimensions.
Speciﬁcally, we focused on the case where general classes of models are used but the learning scheme is expected to yield risk minimizers satisfying monotonicity requirements.
We thus focused on point-wise regularization penalties that bias the learning algorithm towards predictors that have non-negative gradients relative to the dimensions with respect to which monotonicity is desired.
Our empirical evaluation shows that the monotonicity property is achieved in the parts of the space where it is imposed.
Such ﬁnding suggests that past approaches will only enforce monotonicity too far from the data [13] or too close to the data [10].
In order to alleviate this issue, we propose a novel approach that computes the regularization penalty on mixup points.
Our method generates the penalty data via linear combinations between training instances and random points.
In doing so, we can interpolate the data points and populate the convex hull of the training sample.
At the same time, we can extrapolate and create points further away from the manifold where data lies.
Empirical evaluation conﬁrmed that our proposed gradient penalty Ω mixup , using combinations between random samples and data instances, results in the most effective regularization strategy across all evaluated schemes, yielding models observed to satisfy the monotonicity requirement in a larger volume of the input space when compared to alternative strategies.
Given that we observed point-wise gradient penalties to be effective in yielding monotonicity, we envision extensions of this work towards larger neural networks applied in structured data such as image and text.
In that case, however, we hypothesize monotonicity might be enforced between the model’s outputs and high-level representations so as to enable interpretability of predictions.