Constrained Monotonic Neural Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep neural networks are becoming increasingly popular in approximating arbitraryfunctions from noisy data.
But wider adoption is being hindered by the need toexplain such models and to impose additional constraints on them.
Monotonicityconstraint is one of the most requested properties in real-world scenarios and is thefocus of this paper.
One of the oldest ways to construct a monotonic fully connectedneural network is to constrain its weights to be non-negative while employing amonotonic activation function.
Unfortunately, this construction does not workwith popular non-saturated activation functions such as ReLU, ELU, SELU etc,as it can only approximate convex functions.
We show this shortcoming can befixed by employing the original activation function for a part of the neurons inthe layer, and employing its point reflection for the other part.
Our experimentsshow this approach of building monotonic deep neural networks have matchingor better accuracy when compared to other state-of-the-art methods such as deeplattice networks or monotonic networks obtained by heuristic regularization.
Thismethod is the simplest one in the sense of having the least number of parameters,not requiring any modifications to the learning procedure or post-learning steps.
Finally, we give a proof it can approximate any continuous monotone function on acompact subset of R n .
Introduction
Deep Learning has witnessed widespread adoption in many critical real-world domains such asfinance, healthcare etc [21].
Predictive models built using deep neural networks have been shown tohave high accuracy.
Incorporating prior knowledge such as monotonicity in trained models help inimproving the performance and generalization ability of the trained models [25, 9].
The introductionof structural biases such as monotonicity makes models also more data-efficient, enabling a leap inpredictive power on smaller datasets [42].
Apart from the requirements of having models with highaccuracy, there is also a need for transparency and interpretability and monotonicity helps in partiallyachieving the above requirements [14].
Due to legal, ethical and/or safety concerns, monotonicityof predictive models with respect to some input or all the inputs is required in numerous domainssuch as financial (house pricing, credit scoring, insurance risk), healthcare (medical diagnosis, patientmedication) and legal (criminal sentencing) to list just a few.
All other things being equal, a largerhouse should be deemed more valuable, a bank’s clients with higher income should be eligible for alarger loan [36], and an offender with a longer crime history should be predicted as more likely tocommit another crime [32], etc.
A model without such a monotonic property would not, and certainlyshould not, be trusted by society to provide a basis for such important decisions.
However, the of deep learning models is not a guaranteed property even when trained on monotonic data, let alone when training on noisy data typically used in practice.
Although monotonicity is an important and increasingly often even required property, there is nosimple or easy method to enforce it.
It has been an active area of research and the existing methods on the subject can be broadly categorized into two types
Monotonic architectures by construction: This line of research focuses on neural architectures guaranteeing monotonicity by construction [2, 37, 8, 24, 43].2.
Monotonicity by regularization: This line of research focuses on enforcing monotonicityin neural networks during training by employing a modified loss function or a heuristicregularization term [38, 13].
We give a more detailed account of the existing methods in the next section.
The simplest method to achieve monotonicity by construction is to constrain the weights of the fullyconnected neural network to have only non-negative (for non-decreasing variables) or only nonpositive values (for non-ascending) variables when used in conjunction with a monotonic activationfunction, a technique known for almost 30 years [2].
However, this method does not work well inpractice.
When used in conjunction with saturated (bounded) activation functions such as the sigmoidand hyperbolic tangent, these models are difficult to train, i.e. they do not converge to a good solution.
On the other hand, when used with non-saturated (unbounded) convex activation functions such as
ReLU [26], the resulting models are always convex [22], severely limiting the applicability of themethod in practice.
The main contribution of this paper is a simple modification of the method above which, in conjunctionwith non-saturated activation functions, is capable of approximating non-convex functions as well: ifboth concave and convex monotone activation functions are used in a neural network with constrainedweights, it regains the ability to approximate monotone continuous functions that are either convex,concave or nothing of the above.
There are several possible ways to satisfy this condition, with the simplest one using both an activation function ˘ ρ and its point reflection with respect to point (0 , 0) defined as:
[Equation]
Assuming the original activation function ˘ ρ is both monotonic and convex, the properties holding for
ReLU, ELU, SELU, and Softplus, the proposed modification uses the original activation function ˘ ρon a part of the neurons in the network and its point reflection ˆ ρ on the rest of the neurons.
The resulting model is guaranteed to be monotonic, can be used in conjunction with any convexmonotonic non-saturated activation function, doesn’t have any additional parameters compared to anon-monotonic fully-connected network for the same task, and can be trained without any additionalrequirements on the learning procedure.
Experimental results show it is exceeding the performanceof all other state-of-the-art methods, all while being both simpler (in the number of parameters) andeasier to train.
Our contributions can be summarized as follows:
We propose a modification to an existing constricted neural network layer enabling it tomodel non-convex functions when used with non-saturated monotone convex activation
functions such as ReLU, ELU, SELU, and alike.2.
We perform comparisons with other recent works and show that our proposed novel neuralnetwork block can yield comparable and in some cases better results than the previousstate-of-the-art and with significantly fewer parameters.3.
We prove the universal approximation property for the ReLU activation function, showingthat the proposed architecture can approximate any monotone continuous function on acompact subset of R n .
Related work
Before dwelling on the methods employed to build monotonic models, we give an overview ofactivation functions as they form a crucial part of our work.
2.1 Activation functions
Right from its inception in perceptron [31], non-linear activation functions have historically beenone of the most important components of neural networks.
Previously, the saturated functions such as the sigmoid [33], the hyperbolic tangent [27], and its variants were the most common choiceof activation functions.
Currently, one of the most important factors for state-of-the-art resultsaccomplished by modern neural networks is the use of non-saturated activation functions.
The useof Rectified Linear Unit (ReLU) [26, 12] as activation function was instrumental in achieving goodperformance in newer architectures.
The ReLU has since become a defacto choice of activation inmost practical implementations and continues to be widely used because of its advantages such assimple computation, representational sparsity, and linearity.
Later, a number of activation functionswere proposed to deal with solving problems of dead neurons and aid in faster convergence [23], [6][15], [45], [16], [30], [20].
The idea of using both the original activation function and its point reflection in the same layer hasbeen proposed in [35] where both outputs of ReLU and the negative value of its point reflectionwere used in the construction of concatenated ReLU (CReLU) activation function.
The proposedmodification outputs two values instead of one and therefore increases the number of parameters.
In[44], the authors propose negative concatenated ReLU (NCReLU) flip the sign and use the pointreflection directly.
Notice that the proposed architectural change could be applied to other nonsaturated, monotonic activation functions as well, but with an unknown impact on their performance.
In [10], the authors propose bipolar ReLU which consists of using ReLU on the half of the neuronsin the layer and the point reflection of ReLU on the other half.
The same construction could beused with other ReLU family activation functions as well.
However, the focus of their work was toalleviate the need of normalizing layers and improve the performance of deep vanilla recurrent neuralnetworks (RNNs) by using bipolar ReLU .
Both NCReLU and bipolar ReLU could have been used in the construction of a constrained neuralnetwork capable of representing non-convex functions, but to the best of our knowledge, they havenot been so far.
2.2 Monotonicity by construction
Apart from the approaches mentioned in the introduction (section 1), another approach to buildingmonotonic neural architecture is Min-Max networks where monotonic linear embedding and maxmin-pooling are used [37].
In [8], authors generalized this approach to handle functions that arepartially monotonic and proved that the resulting networks have the universal approximation property.
However, such networks are very difficult to train and not used in practice.
Their construction doesnot allow for replacement with other activation functions.
Deep lattice networks (DLN) [43] use a combination of linear calibrators and lattices [24] for learningmonotonic functions.
This is the most widely used method in practice today, but not without itslimits.
Lattices are structurally rigid thereby restricting the hypothesis space significantly.
Also, DLNrequires a very large number of parameters to obtain good performance.
Given a model with a convex output function, it is possible to use backpropagation [34] to make amonotonic model by computing the derivation of the output function.
One simple way to construct aconvex function is to use an unsaturated monotonic activation function in a fully connected layer asmentioned above, but we could also use a more elaborate architecture such as the input convex neuralnetworks [1].
Although possible, these kinds of constructions are computationally more complexthan the simple solution proposed here.
2.3 Monotonicity by regularization
In the second category, the research works focus on enforcing monotonicity during the trainingprocess by modifying the loss function or by adding a regularization term.
In [38], the authors propose a modified loss function that penalizes the non-monotonicity of themodel.
The algorithm models the input distribution as a joint Gaussian estimated from the trainingdata and samples random pairs of monotonic points that are added to the training data.
In [13], theauthors propose a point-wise loss function that acts as a soft monotonicity constraint.
These methodsare straightforward to implement and can be used with any neural network architecture.
However,these methods do not guarantee the monotonicity of the trained model. [Figure]
Recently, there is an increasing number of proposed methods to certify or verify monotonicityobtained by regularization methods.
In [22], the authors propose an optimization-based techniquefor mathematically verifying, or rejecting, the monotonicity of an arbitrary piece-wise linear (e.g.,
ReLU) neural network.
The method consists of transforming the monotonicity verification probleminto a mixed integer linear programming (MILP) problem that can be solved using an off-the-shelf
MILP solver.
In [39], the authors propose an approach that finds counterexamples (defined as the pair of pointswhere the monotonicity constraint is violated) by employing satisfiability modulo theories (SMT)solver [3].
To satisfy the monotonicity constraints, these counterexamples are included in the trainingdata with adjustments to their target values to enforce the next iterations of the model to be monotonic.
Both methods [22, 39] have been shown to support ReLU as the activation function only and thereis no obvious way how to extend them to other activation functions.
More precisely, they rely onpiece-wise linearity of ReLU to work, the property not satisfied by other variants such as ELU, SELU,
GELU, etc.
Last but not least, the procedure for certifying/verifying using MILP or SMT solvers iscomputationally very costly.
These approaches also require multiple reruns or iterations to arrive atcertified/verified monotonic networks.
Constrained neural networks
Most of the commonly used activation functions such as ReLU, ELU, SELU, etc. are monotonicallyincreasing, convex, non-polynomial functions.
When used in a fully-connected, feed-forward neuralnetwork with at least one hidden layer and with unconstrained weights, they can approximateany continuous function on a compact subset.
The simplest way to construct a monotonic neuralnetwork is to constrain its weights when used in conjunction with a monotone activation function.
However, when the activation function is convex as well, the constrained neural network is not ableto approximate non-convex functions.
To better illustrate this, and to propose a simple solution in this particular example, we refer thereaders to Figure 1 where the goal is to approximate a simple cubic function x 3 using a neural networkwith a single hidden layer with either 2 or 32 neurons and with ReLU activation.
A cubic function isapt for our illustration since it is concave in the considered interval [ − 1 , 0] and convex in the interval[0 , − 1] :
non-convex segments such as the cubic function on [ − 1 , 0] in the figure, and increasingthe number of neurons from 2 to 32 does not yield any significant improvement in theapproximation.
[Equation]
[Figure]
Fig solution convex concave activation functions in the hidden in this case, bipolar ReLU, to gain the ability to model non-convex, monotone continuousfunctions.
Notice that increasing the number of neurons increases the number of piecewiselinear segments to approximate the cubic function.
The resulting network is monotone byconstruction even when trained on noisy data.
The schematic block diagram of our proposed solution (which we refer to as Constrained Monotone
Fully Connected Layer or Monotonic Dense Unit interchangeably) is shown in the figure Fig.
The individual components of the proposed solution are defined and described in the subsequentsubsection.
3.1 Constrained monotone fully connected layer
A function f is partially monotone with respect to its parameter x i if ∂f ∂x i ( x 1 , . . .
, x n ) is non-negativeor non-negative for all x 1 , . . .
, x n .
A set S ⊆ R is called compact if every sequence in S has asubsequence that converges to a point in S .
One can easily show that closed intervals [ a, b ] arecompact, and compact sets can be thought of as generalizations of such closed bounded intervals.
Our construction is preconditioned on a priori knowledge of (partial) monotonicity of a multivariate,multidimensional function f .
Let f : K
(cid:55)→ R m be defined on a compact segment K ⊆ R n .
Then wedefine its n -dimensional monotonicity indicator vector t element wise as follows:
[Equation]
Given an ( n × m ) -dimensional matrix W and n -dimensional monotonicity indicator vector t , wedefine the operation | .
| t assigning an ( n × m ) -dimensional
matrix W ′ =
| W | t to W as follows:
[Equation]
Definition 1 (Constrained linear layer) .
Let W ∈ R n × m , t ∈ R n , x ∈ R n and b ∈ R n .
The outputh ∈ R m of the constrained linear layer with monotonicity indicator vector t , weights W , biases band input x is:
[Equation]
Lemma 1.
For each i ∈ { 1 , . . . , n } and j ∈ { 1 , . . .
, m } we have:
• if t i = 1 , then ∂h j ∂x i ≥ 0 , and
• if t i = − 1 , then ∂h j ∂x i ≤ 0 .
Let ˘ ρ be a monotonically increasing convex function such as ReLU, ELU, etc.
Then its point reflectionˆ ρ around the origin is:
[Equation]
Notice that ˆ ρ is a monotonically increasing concave function.
Definition 2 (Combined activation function) .
Given a monotonically increasing convex function˘ ρ , its point reflection ˆ ρ and m -dimensional activation selector vector s ∈ [0 , 1] m , the output of thecombined activation function ρ s is a weighted sum of ˘ ρ and ˆ ρ :
[Equation]
where ⊙ is the element-wise (Hadamard) product and 1 is m -dimensional vector with all elementsequal to 1.
Lemma 2.
Let y
=
ρ s ( h ) .
Then for each j ∈ { 1 , . .
. , m } we have ∂y j ∂h j ≥ 0 .
Moreover
• if s = 1 , then ρ s j is convex; and
• if s = 0 , then ρ s j is concave.
Definition 3 (Monotone constrained fully connected layer) .
Let n and m be positive natural numbers,˘ ρ a monotonically increasing convex function, t an n -dimensional monotonicity selector vector ands an m -dimensional activation selector vector.
Then the monotone constrained fully connected layerwith m neurons is a tuple (˘ ρ, t , s ) , denoted MFC ˘ ρ, s , t .
Moreover, let weights W be an ( n × m ) -dimensional matrix, biases b an m -dimensional vector andinput x an n -dimensional vector.
The output function of the monotone constrained fully connectedlayer, denoted MFC W , b ˘ ρ, s , t ( x ) , assigns an m -dimensional vector y to ( W , b , x ) as follows:
[Equation]
Notice that MFC ˘ ρ, s , t determines an architecture of the layer, while MFC W , b ˘ ρ, s , t is a fully instantiatedlayer with all of its internal parameters W and b defining a function from input to output values.
From Lemma 1 and 2 directly follows:
Corollary 3.
For each i ∈ { 1 , . . . , n } and each j ∈ { 1 , . . .
, m } we have:
if t i = 1 , then ∂y j ∂x i ≥ 0 ; if s = 1 , then y j is convex; if t i = − 1 , then ∂y j ∂x i ≤ 0 ; if s = 0 , then y j is concave.
On the layer level, we can control both monotonicity, convexity and concavity of the output withrespect to chosen input variables.
The following section discuss how we can use such layers to buildpractical neural networks with the same properties.
3.2 Composing monotonic constrained dense layers
As mentioned before, the main advantage of our proposed monotonic dense unit is its simplicity.
Wecan build deep neural nets with different architectures by plugging in our monotonic dense blocks.
The figures Fig 3 and 4 show two examples of neural architectures that can be built using the proposedmonotonic dense block.
The first example shown in the figure Fig 3, corresponds to the standard MLP type of neural networkarchitecture used in general, where each of the input features is concatenated to form one single input
[Figure]
t
[Figure]
feature vector x and fed into the network, with the only difference being that instead of standard fullyconnected or dense layers, we employ monotonic dense units thorughout.
For the first (or input layer)layer, the indicator vector t , is used to identify the monotonicity property of the input feature withrespect to the output.
Specifically, t is set to 1 for those components in the input feature vector thatare monotonically increasing and is set to − 1 for those components that are monotonically decreasingand set to 0 if the feature is non-monotonic.
For the subsequent hidden layers, monotonic dense unitswith the indicator vector t always being set to 1 are used in order to preserve monotonicity.
Finally,depending on whether the problem at hand is a regression problem or a classification problem (oreven a multi-task problem), an appropriate activation function (such as linear activation or sigmoid orsoftmax) to obtain the final output.
Figure Fig.
4 shows another example of a neural network architecture that can be built employingproposed monotonic dense blocks. The difference when compared to the architecture described abovelies in the way input features are fed into the hidden layers of neural network architecture. Insteadof concatenating the features directly, this architecture provides flexibility to employ any form ofcomplex feature extractors for the non-monotonic features and use the extracted feature vectors asinputs. Another difference is that each monotonic input is passed through separate monotonic denseunits.
This provides an advantage since depending on whether the input is completely concave orconvex or both, we can adjust the activation selection vector s appropriately along with an appropriatevalue for the indicator vector t .
Thus, each of the monotonic input features has a separate monotonicdense layer associated with it.
Thus as the major difference to the above-mentioned architecture, weconcatenate the feature vectors instead of concatenating the inputs directly.
The subsequent partsof the network are similar to the architecture described above wherein for the rest of the hiddenmonotonic dense units, the indicator vector t is always set to 1 to preserve monotonicity.
The classical Universal Approximation Theorem [7, 17, 28] states that any continuous function on aclosed interval can be approximated with a feed-forward neural network with one hidden layer if andonly if its activation function is nonpolynomial. In [19], authors prove the approximation propertyholds for arbitrary deep neural networks with bounded number of neurons in each layer holds if theactivation function is nonaffine and differential at at least one point.
In [8], authors shows the universal approximation property for constrained multivariate neuralnetworks using sigmoid as the activation functions: any multivariate continuous monotone functionon a compact subset of R k can be approximated with a constrained neural network with the sigmoidactivation function of at most k layers (Theorem 3.1). However, the proof of the theorem uses onlythe fact that the Heavyside function H defined as
{+[Equation] can be approximated with the sigmoid function on a closed interval. It is important to note that thisprovides an upper bound on the number of monotonic layers in the general case and the number oflayers for a particular function should be determined experimentally in practice (in our experimentsin Section 4, we got the best performing networks using 2-3 layers). Monotonic Dense Block xt 1s 
+}
MonotonicInputs
Non-monotonicInputs
Monotonic Dense Block x mt 1 ms 1 mArbitrary Neural Network
C on ca t e n a t e x m x nMonotonic Dense Block
Final Monotonic Dense Block y Final Activation t 2 =t k =ss k
We provide a simple condition for an arbitrary activation function for the universal approximationproperty to hold:
Theorem 4.
Let ˘
ρ be a monotone activation function.
If a constrained neural network using ˘ ρwith a single hidden layer can approximate the Heavyside function on a closed interval, then anymultivariate continuous monotone function on a compact subset of R k can be approximated with aconstrained neural network of at most k layers using activation functions ˘ ρ .
Since we can approximate any continuous monotonic function with monotonic piecewise linearsegments, we have:
Corollary 5.
A constrained neural network of at most k layers using ReLU as the activation functioncan approximate any multivariate continuous monotone function on a compact subset of R k .
The same property can be shown using in the same way for activation functions other than ReLU.
In order to analyze the practical utility of the proposed method, we experiment with various datasetsand compare them with the recent state-of-the-art.
For the first set of experiments, we use the datasetsemployed by authors in [22] and use the exact train and test split for proper comparison.
We performexperiments on 3 datasets: COMPAS [18], which is a classification dataset with 13 features of which4 are monotonic; Blog Feedback Regression [4], which is a regression dataset with 276 features ofwhich 8 are monotonic; Loan Defaulter 1 , which is a classification dataset with 28 features of which 5are monotonic.
The dataset contains half a million data points.
For comparison with other methods,we compare with Certified monotonic networks (Certified) [22] and other methods described in it.
For the second set of experiments, we use 2 datasets: Auto MPG (which is a regression dataset with 3monotonic features) and Heart Disease (which is a classification dataset with 2 monotonic features)as employed in the work [39] and once again use the exact train and test split for proper comparison.
We compare with the method COMET described in [39] along with Min-Max Net [8] and Deep
Lattice Network (DLN) [43] as described in [39].
For the classification tasks, we use cross-entropy and for the regression tasks, we use mean-squarederror as loss functions.
We employ gridsearch to find the optimal number of neurons, network depthor layers, batch size, activation function and the number of epochs.
For training, we first find theoptimal learning rate using learning rate finder [40] and then train with one cycle policy [41].
4.1 Results
The results on the dataset above are summarized in Tables 1, 2.
It shows that our method tends eithermatch or surpass the other methods in terms of test accuracy for classification tasks and root mean squared error (RMSE) for regression tasks.
For each of the datasets, we run the experiments ten timesafter finding the optimal hyperparameters and report the mean and standard deviation of the best fiveresults.
Experiment results show that networks learned by our method can achieve better results withfewer parameters, than the best-known algorithms for monotonic neural networks, such as Min-Max Network [8] and Deep Lattice Network [43].
It should be noted that the recent state-of-the-art worksCertified [22] and COMET [39] require multiple runs in order to even satisfy monotonic constraintswhereas monotonicity is guaranteed by simply employing the proposed monotonic dense units. The most important advantage of our solution is simplicity and computational complexity. Ourmodels have slightly better performance (accuracy or RMSE) on all datasets we tested them on, but itis important to note they have significantly fewer parameters and the simplest training procedure. Assuch, they reduce the carbon footprint when used in data centers and also aid in the easier adoption ofedge computing applications.
[Table] [Table]
Conclusion
In this paper, we proposed a simple and elegant solution to build constrained monotonic networkswhich can approximate convex as well as concave functions.
Specifically, we introduced a constrainedmonotone fully connected layer which can be used as a drop-in replacement for a fully connectedlayer to enforce monotonicity.
We then employed our constrained monotone fully connected layerto build neural network models and showed that we can achieve either similar or better results tothe recent state-of-the-art ([39, 22] in addition to the well-known works such as Min-Max networks[8] and DLNs [43]. However, the main advantage of the proposed solution is not higher accuracybut its computational and memory complexity: we use orders of magnitude fewer parameters andcomputation which makes the resulting neural networks more energy efficient.
One drawback of our proposed method is that we are restricted by the choice of activation functionsi.e. , we can only employ monotonic activation functions.
In the future, we would like to buildsimple monotonic counterparts for other standard neural layers such as convolutional neural networks,recurrent neural networks and their variants, and attention models. Last but not least, we proved suchnetworks can approximate any multivariate monotonic function.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] In Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Provided in
supplementary
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Provided
as part of code in supplementary.
Also, it should benoted that we build our experiments based on two previous state-of-the-arts - Certified
Monotonic Network [22] and COMET [39], therefore for fair comparison, we employthe exact splits the authors employed in their repective works
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See 1,
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Our code can be run on Google’s
Collaboratory and with default settings.
A GPU instance uses a NVIDIA Tesla T
GPU with 16GB RAM
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] Our work has also been applied aspatent.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] The code is included in the supplemental material and will be released on github
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
[Table] [Table] [Table]
A Detailed proofs
We restate all lemmas from the main text here are give detailed proofs of them.
The following is well known result, proved here for completeness only:
Lemma 1.
For each i ∈ { 1 , . . . , n } and j ∈ { 1 , . . .
, m } we have:
• if t i = 1 , then ∂h j ∂x i ≥ 0 , and
• if t i = − 1 , then ∂h j ∂x i ≤ 0 .
Proof.
From h = | W | t · x + b we get h j = (cid:80) i w
′ i,j x i + b j .
Hence ∂h j ∂x i = w ′
i,j .
Finally, fromequation 2 we have
[Equation]
Lemma 2.
Let y
=
ρ s ( h ) .
Then for each j ∈ { 1 , . .
. , m } we have ∂y j ∂h j ≥ 0 .
Moreover
• if s = 1 , then ρ s j is convex; and
• if s = 0 , then ρ s j is concave.
Proof.
From equation
[Equation]
we have:
[Equation]
Since both ˘ ρ and ˆ ρ are monotonically increasing and s j ∈ [0 , 1] we have:
[Equation]
if s = 1 or s = 0 , we have ρ s ( h ) = ˘ ρ ( h ) or ρ s ( h ) =
ˆ ρ ( h ) , which is a convex or a concave functionin all components of the output, respectively.
For completeness, we repeat the Theorem 3.1 from [8] and its proof here: Theorem 3.1 For any continuous monotone nondecreasing function f : K −→ R , where K is acompact subset of R k , there exists a feedforward neural network with at most k hidden layers, positiveweights, and output O such that | O x − f ( x ) < ϵ , for any x ∈ K and ϵ > 0 . Proof.
The proof is derived by induction on the number of input variables k . Without loss ofgenerality, we may assume that f > 0 (otherwise, we add a constant C and approximate f + C withthe network output O , then modify O with a negative bias at the output node). First, we assume thatf is strictly increasing and C ∞ . In case of k = 1 , we write
[Equation]
where H is the Heavyside function:
[Equation]
Since f is continuous and increasing, it is invertible and therefore the right-hand side of 7 can bewritten as
[Equation]
The integral can be approximated arbitrarily well by a Riemann sum
[Equation]
where [ v i ] Ni =1 is a partition of the interval [ f ( a ) , f ( b )] .
This expression corresponds to a neuralnetwork with input x , one hidden layer with N neurons all connected to the input with weight of 1,bias term in the hidden neurons f − 1 ( v i ) , and the weights connecting the hidden layer with the outputv i +1 − vi > 0 .
Note that the Heavyside function H can be replaced by a sigmoid activation functionusing a standard approximation argument.
Assume that Theorem 3.1 holds for k − 1 input variables.
We now combine the integral representation in 7 with the induction assumption.
For a given v , we may solve the equation of the level set corresponding to to v for x k f
[Equation]
[Equation]
x k
[Equation]
= v.
By the implicit function theorem, there exists a function g v such that
[Equation]
[Equation]
Note that g v is decreasing in all arguments x i .
This can be seen by taking the partial derivative ofwith respect to x i .
We will now show that
[Equation]
analogously to 8 for the 1-D case.
Note that it is sufficient to show that
[Equation]
only
[Equation]
x
[Equation]
[Equation]
. , x k
−
1 ) and f ( x ) > v
if and only if x k > g v ( x 1 , . .
, x k − 1 ) .
But this follows easily from 10 and the fact that f is increasing in all its arguments.
We nowapproximate the integral in 7 with a Riemann sum, leading to the following equation analogously to9:
[Equation]
Since g v i is decreasing in all its arguments − g v
i is increasing.
By the induction assumption, we can approximate − g v i with a feedforward neural network O i with x 1 , . . .
, x k − 1 as inputs, k − 1 hidden layers, and nonnegative weights, such that
[Equation]
because the sum is finite.
Expression 11 corresponds to a feedforward neural network with k inputsand k hidden layers.
Here k − 1 hidden layers are needed to represent − g v i and the k -th hiddenlayer is needed to combine N neural networks with outputs
O i and the input x k .
The weights on theconnections between the last hidden layer and the final output are ( v i +1 − v i ) > 0 .
The input x k isdirectly (skip-layer) connected to the k -th hidden layer.
We can now easily generalize the proof to continuous nondecreasing functions.
For continuous functions, we define the convolution of f with a mollifier K δ by
[Equation]
Then, f δ is C infty and f δ −→ f as δ ↓ 0 uniform on compact subsets.
Furthermore, f δ is alsoincreasing since K δ > 0 .
Now choose δ such that | f − f δ | < ϵ 2 and approximate f δ with a feedforward 534 neural network O such that | f δ − O | < ϵ 2 .
Then, | f − O | < ϵ .
If f is nondecreasing, then approximate f by f δ f δ =
f
+ δ ( x 1 + · · · + x k ) which is strictly increasing and let δ ↓ 0 .
B Datasets Description
The descriptions of datasets used for comparison are detailed below.
As mentioned in the section 4,the datasets are chosen from [22] and [39] for proper evaluation.
The train-test splits of 80% − 20%are used for all comparison experiments.
COMPAS [18] is a binary classification dataset, where the task is to predict risk score
of an individual committing crime again two years,
based on the criminal records of
individuals arrested in Florida.
The risk score needs to be monotonically increasing with
respect to the following attributes number of prior adult convictions, number
of juvenile felony, number of juvenile misdemeanor , and number of other
convictions .
It should be noted that there have been ethical concerns with the dataset
Blog Feedback [4] is a regression dataset where the task is to predict the number of comments
in the upcoming 24 hours from a feature set containing 276 features of which 8 (A51, A52,
A53, A54, A56, A57, A58, A59) are monotonic features.
The readers are suggested to
refer to link 2 for more details.
As mentioned by the authors of [22], only the data points
with targets smaller than the 90th percentile are used since the outliers could dominate
the
mean-squared-error metric.
Lending club loan data 3 is a classification dataset, where the task is to predict whether the
individual would default on loan, from a feature set having 28 features containing data
such as the current loan status, latest payment information etc,.
The probability of default
should be non-decreasing with respect to number of public record bankruptcies,
Debt-to-Income ratio , and non-increasing with respect to credit score, lengthof employment, annual income .
Auto MPG 4 [29] is a regression dataset where the task is to predict city-cycle fuel con-
sumption in miles per gallon (MPG) from a feature set containing 7 features of which
the
monotonic features are weight (W) , displacement (D) , and horse-power (HP)
Heart Disease 5 [11] is a classification dataset, where the task is to predict the presence of
heart disease from a feature set containing 13 features of which the risk associated with
heart disease needs to be monotonically increasing with respect to the features trestbps
(T), cholestrol (C))
C Additional Experiments and Results Details
For all our experiments, we adopt simple architectures of the types depicted in Figure.
3 or Figure. 4.
More often than not, we have seen from our experiments that the neural architecture type 2 (Figure.4) performs better than the neural architecture type 1 (Figure. 3).
The number of hidden layers (apartfrom the input layer) is either set to be 1 or 2 .
The number of neurons in the hidden layer is selectedfrom 4, 8, 16, 32, 64.
The activation function is set to be either exponential linear unit (ELU) or
Rectified Linear Unit (ReLU).
− −   2 x − −    y
Ground Truth −3.2 −2.
−1.6 −0.
(a) − −   2 x − −    y
Unconstrained ELU −3.2 −2.
−1.6 −0.
(b) − −   2 x − −    y
Constrained ELU −1.2 −0.
(c) − −   2 x − −    y
Ours −4 − −2 − 02 (d)
Apart from the experimental results in the main part of the paper, here we highlight the usefulnessof the our monotonic dense unit.
In a scenario where the dataset is noisy and very small and alsocontains monotonic features, our neural network constructed using our proposed monotonic denseunit tends to perform better because of the inductive bias.
To illustrate this we consider a syntheticdataset generated by function f
( x, y ) =
sgn ( ax ) x 3 + b sin( cy ) , a, b, c ∈ { 0 . 5 , 0 .
Wesample points uniformly in the range ( − 2 .
5 , 2 . 5) for both x and y , and add zero-centered Gaussiannoise.
We then sample only 100 points for training and train three neural networks having thesame kind of architecture but having difference in constraints on weights as mentioned in section 3,but activation function used is ELU instead of ReLU.
We test the three networks - Unconstrained,
Constrained ELU and Constrained bipolar ELU, to on the task of fitting the aforementioned function.
The results be seen in the Figure.
It is evident that the unconstrained neural network does notpreserve monotonicity and although constrained neural network does preserve monotonicity, it cannotfaithfully reproduce the concave part of the function, whereas ours (constrained bipolar ELU) can fitboth concave and convex part of the function.