M ONOTONICITY AS A REQUIREMENT AND AS A REGULARIZER : EFFICIENT METHODS AND APPLICATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the setting where risk minimization is performed over general classes of models and consider two cases where monotonicity is treated as either a requirement to be satisﬁed everywhere or a useful property.
We speciﬁcally consider cases where point-wise gradient penalties are used alongside the empirical risk during training.
In our ﬁrst contribution, we show that different choices of penalties deﬁne the regions of the input space where the property is observed.
As such, previous methods result in models that are monotonic only in a small volume of the input space.
We thus propose an approach that uses mixtures of training instances and random points to populate the space and enforce the penalty in a much larger region.
As a second contribution, we introduce the notion of monotonicity as a regularization bias for convolutional models.
In this case, we consider applications, such as image classiﬁcation and generative modeling where monotonicity is not a hard constraint but can help improve some aspects of the model.
Namely, we show that using group monotonicity can be beneﬁcial in several applications such as: (1) deﬁning strategies to detect anomalous data, (2) allowing for controllable data generation and (3) generating explanations for predictions.
Our proposed approaches do not introduce relevant computational overhead while leading to efﬁcient procedures that provide extra beneﬁts over baseline models
I NTRODUCTION
Highly expressive model classes such as artiﬁcial neural networks have achieved impressive prediction performance across a broad range of supervised learning tasks and domains (Krizhevsky et al., 2012; Graves & Jaitly, 2014; Bahdanau et al., 2014).
However, ﬁnding predictors attaining low risk on unseen data is often not enough to enable the use of such models in practice.
In fact, practical applications usually have more requirements other than prediction accuracy.
Hence, devising approaches that search risk minimizers satisfying practical needs led to several research threads seeking to enable the use of neural networks in real-life scenarios.
Examples of such requirements include: (1) Robustness , where low risk is expected even if the model is evaluated under distribution shifts, (2) Fairness , where the performance of the model is expected to not signiﬁcantly change across different sub-populations of the data, and (3) Explainability/Interpretability , where models are expected to indicate how the features of the data affect their predictions.
In addition to the requirements mentioned above, a property commonly expected in trained models in certain applications is monotonicity with respect to some subset of the input dimensions, i.e., an increase (or decrease) along some particular dimensions strictly imply the function value will not decrease (or will not increase), provided that all other dimensions are kept ﬁxed.
As a result, the behavior of monotonic models will be more aligned with the properties that the data under consideration is believed to satisfy.
For example, in the models used to accept/reject job applications, we expect acceptance scores to be monotonically non-decreasing with respect to features such as past years of experience of a candidate.
Thus, given two applicants with exactly the same features except their years of experience, the more experienced candidate should be assigned an equal or higher chance of getting accepted.
For applications where monotonicity is expected, having a predictor failing to satisfy this requirement would damage the user’s conﬁdence.
As such, different strategies have been devised in order to enable training monotonic predictors.
These approaches can be divided into two main categories.
The ﬁrst one is monotonicity by construction , where the focus lies on deﬁning a model class that guarantees monotonicity in all of its elements Bakst et al.
Wehenkel & Louppe (2019); Nguyen & Mart´ınez (2019); You et al.
(2017); Garcia & Gupta (2009);
Archer & Wang (1993).
However, this approach can not be used with general architectures in the case of neural networks.
Additionally, the model class might be so constrained that it might affect the prediction performance.
Alternatively, a second approach is based on encouraging monotonicity during training , i.e., searching for monotonic candidates within a general class of models (Liu et al., 2020; Sivaraman et al., 2020; Gupta et al., 2019).
Such a group of methods is more generally applicable and can be used, for instance, with any neural network architecture.
However, they are not guaranteed to yield monotonic predictors unless extra veriﬁcation/certiﬁcation steps are performed, which can be computationally very expensive.
In addition to being a requirement as in the examples discussed above, monotonicity has also been observed to be a useful feature in certain cases.
For example, it can deﬁne an effective inductive bias and improve generalization in cases where prior knowledge indicates the data generation process satisﬁes such a property (Dugas et al., 2001).
In such cases, however, it is not necessary to satisfy the property everywhere, since it is enforced simply as a desirable feature of trained models rather than a design speciﬁcation.
In this work, we tackle the problem of performing empirical risk minimization over rich classes of models such as neural networks, while simultaneously searching for monotonic predictors within the set of risk minimizing solutions.
In summary, our contributions are two-fold:
monotonicity as a requirement : We show
that previous methods can only
satisfy monotonicity either near the training data or near the boundaries of the
input domain.
Then, we propose
an efﬁcient algorithm that tackles this problem.
In short, we apply Mixup
(Zhang et al., 2018)
between the data and random points to populate the input space, which is shown to enforce monotonicity in a larger volume relative to previous methods in literature.
To the best of our knowledge, this is the ﬁrst work that studies
the effect of the sample points used in calculating the monotonicity constraints.
monotonicity as a regularizer : We deﬁne a new notion of monotonicity which is shown to be useful when enforced for applications such as object recognition or generative modeling of images.
In these cases, it is not necessarily required to enforce the property everywhere and, as such, the constraints that focus only on the actual data points are discussed (i.e., mixup is not needed).
Models satisfying the property are compared with standard predictors across three applications, where it is shown the property is beneﬁcial without compromising the original performance.
B ACKGROUND AND RELATED WORK
We start by deﬁning the notion of partial monotonicity used throughout the paper.
Consider the standard supervised learning setting where data instances are observed in pairs x, y ∼ X ×Y , where
X ⊂ R d and Y ⊂ R correspond to the input and output spaces, respectively.
Further, consider the function f : X (cid:55)→ Y , and let M indicate some subset of the input dimensions, i.e., M ⊂ { 1 , ...
d } , such that x = concat ( x M , x ¯ M ) , where ¯ M = { 1 , ..., d } \ M .
We further overload the notation of function calls to f such that f ( x ) =
f
( x M , x ¯ M ) .
Deﬁnition 1 Partially monotonic functions relative to M :
We say f is monotonically nondecreasing relative to M 1 , denoted f M , if f ( x M , x ¯ M ) ≤ f ( x (cid:48)
M , x ¯ M ) , ∀ x M ≤ x (cid:48)
M , ∀ x ¯ M , where the comparison x M ≤ x (cid:48)
M is performed for every dimension.
This deﬁnition covers functions that do not decrease in value given increasing changes along a subset of the input dimensions, provided that all other dimensions are kept unchanged.
Several approaches were introduced for deﬁning model classes that have such a property.
The simplest approach restricts the weights of the network to be non-negative (Archer & Wang, 1993).
However, doing so affects the prediction performance.
Another approach corresponds to using lattice regression models proposed by Garcia & Gupta (2009);
You et al.
In this case, models are given by interpolations in a grid deﬁned by training data.
Such a class of models can be made monotonic via the choice of the interpolation strategy and recently introduced variations (Bakst et al., 2021) scale efﬁciently with the dimension of the input space, but downstream applications might still require different classes of models to satisfy this type of property.
For neural networks, approaches such as (Nguyen & Mart´ınez, 2019) reparameterize fully connected layers such that the gradients with respect to parameters can only be non-negative.
Wehenkel & Louppe (2019), on the other hand, consider the class of predictors H :
X
(cid:55)→ Y of the form H ( x ) =
(cid:82) x 0 h ( t )
dt + H (0) , where h ( t ) is a strictly positive mapping parameterized by a neural network.
While such approaches guarantee monotonicity by design, they can be too restrictive or give overly complicated learning procedures.
For example, the approach in (Wehenkel & Louppe, 2019) requires backpropagating through the integral.
An alternative approach is based on searching over general classes of models while assigning higher importance to predictors observed to be monotonic.
Similar to the case of adversarial training (Goodfellow et al., 2014), Sivaraman et al.
(2020) proposed an approach to ﬁnd counterexamples, i.e., pairs of points where the monotonicity constraint is violated, which are included in the training data to enforce monotonicity conditions in the next iterations of the model.
However, this approach only supports fully-connected ReLU networks.
Moreover, the procedure for ﬁnding the counterexamples is costly.
Alternatively, Liu et al.
; Gupta et al.
(2019) introduced point-wise regularization penalties for enforcing monotonicity, where the penalties are estimated via sampling.
While Liu et al.
(2020) use uniform random draws, Gupta et al.
(2019) apply the regularization penalty over the training instances.
Both approaches have shortcomings that we seek to address.
M ONOTONICITY AS A REQUIREMENT
Given the standard supervised learning setting where (cid:96) :
Y 2 (cid:55)→ R + is a loss function indicating the goodness of the predictions relative to ground truth targets, the goal is to ﬁnd a predictor h ∈ H such that its expected loss – or the so-called risk – over the input space is minimized.
Such an approach yields the empirical risk minimization framework once a ﬁnite sample is used to estimate the risk.
However, given the extra monotonicity requirement, we consider an augmented framework where such a property is further enforced.
We seek the optimal monotonic predictors relative to M , h ∗ M :
[Equation]
where γ is a hyperparameter weighing the importance of the penalty Ω( h, M ) which, in turn, is a measure of how monotonic the predictor h is relative to the dimensions indicated by M .
Ω( h, M ) can be deﬁned by the following gradient penalty (Gupta et al., 2019; Liu et al., 2020):
[Equation]
where ∂h ( x ) ∂x i indicates the gradients of h relative to the input dimensions i ∈ M , which are constrained to be non-negative, rendering h monotonically non-decreasing relative to M .
At this point, the only missing ingredient to deﬁne algorithms to estimate h ∗ M is how to deﬁne the distribution D over which the expectation in Eq.
2 is computed, discussed in the following sections.
3. C HOOSING DISTRIBUTIONS OVER WHICH TO COMPUTE THE MONOTONICITY PENALTY
In the following, we present and discuss two past choices for D : 1) Deﬁne D as the empirical distribution of the training sample :
In (Gupta et al., 2019), given a training dataset of size N , in addition to using the observed data to estimate the risk, the same data is used to compute the monotonicity penalty so that:
[Equation]
where x k indicates the k -th instance within the training sample.
While this choice seems natural and can be easily implemented, it only enforces monotonicity in the region where the training samples lie, which can be problematic.
For example, in case of covariate-shift, the test data might lie in parts of the space different from that of the training data so monotonicity cannot be guaranteed.
We thus argue that one needs to enforce the monotonicity property in a region larger than what is deﬁned by the training data.
In Appendix B, we conduct an evaluation under domain shift and show the issue to become more and more relevant with the increase in the dimension d of the input space X .
2) Deﬁne D = Uniform ( X ) :
In Liu et al.
(2020), a simple strategy is deﬁned so that Ω is computed over the random points drawn uniformly across the entire input space X ; i.e.:
[Equation]
Despite its simplicity and ease of use, this approach has some ﬂaws.
In high-dimensional spaces, random draws from any distribution of bounded variance will likely lie in the boundaries of the space, hence far from the regions where data actually lie.
Moreover, it is commonly observed that naturally occurring high-dimensional data is structured in lower-dimensional manifolds (c.f.
(Fefferman et al., 2016) for an in-depth discussion on the manifold hypothesis).
It is thus likely that random draws from the uniform distribution will lie nowhere near regions of space where training/testing data will be observed.
We further illustrate the issue with examples in Appendix A, which can be summarized as follows: consider the cases of uniform distributions over the unit n -sphere.
In such a case, the probability of a random draw lying closer to the sphere’s surface than to its center is
P ( || x || 2 > 12 ) = 2 n − 1 2 n , as given by the volume ratio of the two regions of interest.
Note that
P ( || x || 2 > 12 ) → 1 as n → ∞ , which suggests the approach in (Liu et al., 2020) will only enforce monotonicity at the boundaries of the space.
In summary, the previous approaches are either too focused on enforcing monotonicity where the training data lie, or too loose such that the monotonicity property is uniformly enforced across a large space, and the actual data manifold may be neglected.
We thus propose an alternative approach where we can have some control over the volume of the input space where the monotonicity property will be enforced.
Our approach uses the idea of data mixup (Zhang et al., 2018; Verma et al., 2019;
Chuang & Mroueh, 2021), where auxiliary data is created via interpolations of pairs of data points, to populate areas of the space that are otherwise disregarded.
Mixup was introduced by Zhang et al.
(2018) with the goal of training classiﬁers with smooth outputs across trajectories in the input space from instances of different classes.
Given a pair of data points ( x (cid:48) , y (cid:48) ) , ( x (cid:48)(cid:48) , y (cid:48)(cid:48) ) , the method augments the training data using interpolations given by ( λx (cid:48)
+ (1 − λ )
x (cid:48)(cid:48) , λy (cid:48)
+ (1 − λ )
y (cid:48)(cid:48) ) , where λ ∼ Uniform ([0 , 1]) .
We propose using mixup to generate points where the monotonicity penalty Ω can be computed and highlight the following motivations for doing so: (1) Interpolation of data points more densely populates the convex hull of the training data.
(2) Extrapolation cases where mixup is performed between data points and instances obtained at random results in points that lie anywhere between the data manifold and the boundaries of the space.
We thus claim that performing mixup enables the computation of Ω on parts of the space that are disregarded if one focus only on either observed data or random draws from uninformed choices of distributions such as the uniform.
3. E VALUATION
In order to evaluate the effect of different choices of Ω , we report results on three commonly used datasets covering classiﬁcation and regression settings with input spaces of different dimensions.
Namely, we report results for the following datasets: Compas , Loan Lending Club , and Blog Feedback .
Models are implemented using the same architecture as in (Liu et al., 2020).
Further details on the data, models, and training settings can be found in Appendix C. For all evaluation cases, we consider the baseline where training is carried out without any monotonicity enforcing penalty.
For the regularized cases, the different approaches used for computing Ω are as follows: (1) Ω random (Liu et al., 2020) which uses random points drawn from Uniform ( X ) .
In this case, the sample observed at each training iteration is set to a size of 1024 throughout all experiments.
(2) Ω train (Gupta et al., 2019) which uses the actual data observed at each training iteration; i.e., the observed mini-batch itself is used to compute Ω .
And (3) Ω mixup ( ours ), in which case the penalty is computed on points generated by mixing-up points from the training data and random points.
In details, for each minibatch of size N > 1 , we augment it with complementary random data and obtain a ﬁnal mini-batch of size 2 N .
Out of the 2 N (2 N − 1)possible pairs of points, we take a random subsample of 1024 pairs to compute mixtures of instances.
In this case, we use λ ∼ Uniform([0,1]) and λ is independently drawn for each pair of points.
Results are reported in terms of both prediction performance and level of monotonicity.
The latter is assessed via the fraction ρ of points within a sample where the monotonicity constraint is violated; i.e., given a set of N data points, we compute: ρ =
[Equation]
, such that ρ =
[Table]
corresponds to monotonic models over the considered points.
Moreover, in order to quantify the degree of monotonicity in different parts of the space, we compute ρ for 3 different sets of points:
(1) ρ random , computed on a sample drawn according to Uniform ( X ) .
We used a sample of 10, points throughout the experiments.
(2) ρ train , computed on the training data.
And (3) ρ test : computed on the test data.
Results are summarized in Table 1 in terms of both prediction performance along with the metric indicating the degree of monotonicity of the predictor for each regularization strategy.
Prediction performance is measured in terms of accuracy for classiﬁcation tasks, and
RMSE for the case of regression.
Monotonicity, on the other hand, is quantiﬁed via the fraction ρ of points within a sample where the monotonicity constraint is violated.
Results reported in the tables represent 95% conﬁdence intervals corresponding to multiple independent training runs.
Across all datasets, the different penalties do not result in signiﬁcant variations in the performance of the ﬁnal predictors.
This indicates that the class of predictors corresponding to the subset of H that is monotonic relative to M , denoted H M , has enough capacity so as to be able to match the performance of the best canditates within H .
In terms of monotonicity, we observe a clear pattern leading to the following intuition: monotonicity is achieved in the regions where it is enforced .
This is evidenced by the observation that ρ random is consistently lower for Ω random relative to Ω train and Ω mixup while, on the other hand, ρ train and ρ test are consistently lower for Ω train and Ω mixup compared to Ω random .
A comparison between Ω train and Ω mixup shows what we anticipated: enforcing monotonicity in points resulting from mixup yields predictors that are as monotonic as those given by the use of Ω train in actual data, but signiﬁcantly better at the boundaries of X .
Finally, the results demonstrate that our proposed approach Ω mixup achieves the best results in terms of monotonicity for all the sets of points that we considered.
Moreover, our approach introduces no signiﬁcant computation overhead.
Algorithm 1 in Appendix C details on how to compute Ω mixup .
M ONOTONICITY AS A REGULARIZER
In Section 3, we presented an efﬁcient approach to enforce monotonicity when it is a requirement.
We now consider a different perspective and show that adding monotonicity constraints during training can yield extra beneﬁts to trained models.
In these cases monotonicity is not a requirement, and hence it is not necessary for it to be satisﬁed everywhere.
As such, the penalties we discuss from now on are computed considering only data points, and no random draws are utilized.
In the following sections, we introduce notions of monotonicity that will be enforced in our models, and discuss advantages of using monotonicity for different applications such as for the detection of anomalous data and controllable generative modelling.
In Appendix F, we consider a further application for cases where one’s interest is to obtain explanations from observed predictions.
4. G ROUP M ONOTONICITY
First, we consider the case of K -way classiﬁers realized through convolutional neural networks.
In this case, data examples correspond to pairs x, y ∼ X × Y , but Y = { 1 , 2 , 3 , ..., K } , K ∈ N .
Models parameterize a data-conditional categorical distribution over Y , i.e., for a given model h , h ( x ) Y will yield likelihoods for each class indexed in Y .
Under this setting, we introduce the notion of Group Monotonicity : we aim to ﬁnd the models h such that the outputs corresponding to each class satisfy a monotonic relationship with a speciﬁc subset of high-level representations given by some inner convolutional layer.
Let the outputs of a speciﬁc layer within a convolutional model be represented by a w , w ∈ [1 , 2 , 3 , ..., W ] , where W indicates the width of the chosen layer given by its number of output feature maps.
For simplicity of exposition, we consider the rather common case of convolutional layers where each feature map a w is 2-dimensional.
We then partition such a set of representations into disjoint subsets, or slices , of uniform sizes.
Each subset is then paired with a particular output or class, and hence denoted by S k , k ∈ Y .
An illustration is provided in Figure 1, where a generic convolutional model has the outputs of a speciﬁc layer partitioned into slices S k which are paired with the corresponding output units.
Deﬁnition 2 Group monotonic convolutional classiﬁers: We say h is group monotonic for input x and its corresponding class label
y if h ( x ) y is partially monotonic relative to all elements in S y .
We highlight that in this case, unlike the discussion in Section 3, monotonicity is not an application requirement, and it does not need to be satisﬁed everywhere.
Intuitively, our goal is to “reserve” groups of high-level features to be monotonic with respect to different classes.
Such an structure can add extra beneﬁts to models, e.g., more accurate anomaly detection.
For training, we perform monotonic risk minimization as described in Eq. 1, and the risk is given by the negative log-likelihood over training points.
Moreover, we design a penalty Ω that focuses only on observed data points during training and penalizes the slices of the Jacobian corresponding to a given class, i.e., a crossentropy criterion enforces larger gradients on the class slice.
In order to formally introduce such a penalty, denoted by Ω group , we ﬁrst deﬁne the total gradient O k , k ∈ Y , of a slice S k as follows:
O y ( x ) = (cid:80) a w
[Equation]
S y (cid:80) i,j ∂h ( x ) y ∂a w,i,j , where the inner sum accounts for spatial dimensions of a w .
Given the set of total gradients, a batch of size m , and inverse temperature µ , Ω group will be:
[Equation]
4. E VALUATION 4.2. A SSESSING PERFORMANCE OF GROUP MONOTONIC CLASSIFIERS
We start our evaluation by verifying whether the group monotonicity property can be effectively enforced into classiﬁers trained on standard object recognition benchmarks.
In order to do so, we verify the performance of the total activation classiﬁer , as deﬁned by: arg max k ∈Y T k
( x ) , where T k indicates the total activation on slice S k : T k ( x )
=
(cid:80) a w ∈ S k (cid:80) i,j a w,i,j ( x ) .
A good prediction performance of such a classiﬁer serves as evidence that the group monotonicity property is satisﬁed by the model over the test data under consideration since it indicates the slice relative to the underlying class of test instances has the highest total activation.
We thus run evaluations for both CIFARand ImageNet, and classiﬁers in each case correspond to WideResNets (Zagoruyko & Komodakis, 2016) and ResNet-50 (He et al., 2016), respectively.
Training details are presented in Appendix D.
Results are reported in Table 2 and compared based on the top-1 prediction accuracy measured on the test data.
We use standard classiﬁers as the baselines where no monotonicity penalty is applied in order to isolate the effect of the penalty.
In both datasets, the total activation classiﬁers for group monotonic models (indicated by the preﬁx mono ) are able to closely match the performance of
[Table]
the classiﬁer deﬁned at the output layer, arg max k ∈Y h ( x ) k .
This suggests that the higher total activation generally matches the predicted class for group monotonic models, which indicates the property is successfully enforced.
Moreover, the prediction performance obtained at the output layer was not affected by enforcing the monotonicity property, i.e., one can match the accuracy of unconstrained models while focusing on the set of group monotonic candidates.
Additional experiments are reported on Table 8 on Appendix E for cases with small sample sizes, where we show that the performance of the classiﬁer deﬁned at the output layer upper bounds that of the total activation classiﬁer, i.e., the better the underlying classiﬁer the more group monotonic it can be made .
4.2. U SING GROUP MONOTONICITY TO DETECT ANOMALIES
After showing that group monotonicity can be enforced successfully without affecting the prediction performance, we discuss approaches to leverage it and introduce applications of the models satisfying such a property.
In particular, we consider the application of detecting anomalous instances, i.e., those where the model may have made a mistake.
For example, consider the case where a classiﬁer is deployed to production and, due to some problem external to the model, it is queried to do prediction for an input consisting of white noise.
Standard classiﬁers would provide a prediction even for such a clearly anomalous input.
However, a more desirable behavior is to somehow indicate that the instance is problematic.
We claim that imposing structure in the features, e.g., by enforcing group monotonicity, can help in deciding when not to predict.
To evaluate the proposed method, we implement anomalous test instances using adversarial perturbations.
Namely, we create L ∞ PGD attackers (Madry et al., 2017) and detect anomalies based on simple statistics of the features.
In details, for a given input x , we compute the normalized entropy H ∗ ( x ) of the categorical distribution deﬁned by the application of the softmax operator over the set of total activations
T Y ( x ) : H
[Equation]
, where K = |Y| and the set p Y ( x ) corresponds to the parameters of a categorical distribution deﬁned by: p Y ( x ) = softmax ( T Y ( x )) .
Decisions can then be made by comparing H ∗ ( x ) with a threshold τ ∈ [0 , 1] , deﬁning the detector 1 { H ∗ >τ } .
We evaluate the detection performance of this approach on both MNIST and CIFAR-10.
Training for the case of
CIFAR-10 follows the same setup discussed on Section 4.2.1.
For MNIST on the other hand, we modify the standard LeNet architecture by increasing the width of the second convolutional layer from 64 to 150.
This layer is then used to enforce the group monotonicity property.
The resulting model is referred to as WideLeNet.
Moreover, γ and µ are set to 1 e 10 and 1 , respectively.
Adversarial attacks are created under the white-box setting, i.e., by exposing the full model to the attacker.
The perturbation budget in terms of L ∞ distance is set to 0 .
3 and 8255 for the cases of MNIST and
CIFAR-10, respectively.
Detection performance is reported in Table 3 for the considered cases in terms of the area under the operating curve (AUC-ROC).
The baselines are the models for which the monotonicity penalty is not enforced.
They are trained under the same conditions and the same computation budget as the models where the penalty is enforced.
The results are as expected, i.e., for monotonic models, test examples for which the total activations are not structured very often correspond to anomalous inputs.
Finally, due to space constraints, we discuss the application of group monotonicity to explainability in appendix F.
[Table]
perturbed data instances.
[Figure]
4. D ISENTANGLED REPRESENTATION LEARNING UNDER MONOTONICITY CONSTRAINTS
We now discuss another application where using monotonicity as a regularizer can be beneﬁcial.
In particular, we consider the case of learning a disentangled set of representations.
In this case, it is often assumed that the latent variables are independent, and hence control over generative factors can be achieved, e.g., one can modify a speciﬁc aspect of the data by modifying the value of a corresponding latent variable.
However, we argue that disentanglement is necessary but not sufﬁcient to enable controllable data generation.
We need latent variables that satisfy some notion of monotonicity to be able to decide their values resulting in desired properties.
For example, assume we are interested in generating images of simple geometric forms and desire to control factors such as shape and size.
In this example, even if a disentangled set of latent variables is available, we cannot decide how to change the value of the latent variable to get a bigger or a smaller object if there is no monotonic relationship between the size and the value of the corresponding latent variable.
We address this issue and build upon the weakly supervised framework introduced by Locatello et al.
This work extends the popular β -VAE setting (Higgins et al., 2016) by introducing weak supervision such that the training instances are presented to the model in pairs ( x 1 , x 2 ) where only one or a few generative factors are changing between each pair.
Here, we propose to apply group monotonocity over the activations of the corresponding latent variables to have more controlable factors.
In the VAE setting, the data is generated according to p ( x | z ) p ( z ) given the latent variables z .
Approximation is then performed by introducing p θ ( x | z ) and q φ ( z | x ) , both parameterized by neural networks.
Our goal is to have z fully factorizable in its dimensions, i.e., p ( z ) =
(cid:81)
Dim [ z ] i = p
( z i ) , which needs to be captured by the approximate posterior distribution q φ ( z | x ) .
Training is performed by maximization of a lower-bound on the likelihood of the data, which yields the following loss:
[Equation]
where ˜ q φ (ˆ z j | x i )
=
q φ ( z j | x i ) for the latent dimensions z i that change across x 1 and x 2 , and ˜ q φ (ˆ z j | x i ) =
12 ( q φ (ˆ z j | x 1 ) + q φ (ˆ z
j | x 2 )) for those that are common (i.e., the approximate posterior of the shared latent variables are forced to be the same for x 1 and x 2 ).
The outer expectation is estimated by sampling pairs of data instances ( x 1 , x 2 ) where only a number of generative factors vary.
In our experiments, we consider the case where exactly one generative factor changes across inputs.
Moreover, we follow Locatello et al.
and assign the changing factor, denoted by y , to the dimension j of z such that y = arg max
j ∈ Dim [ z ] D KL ( z 1 j , z 2 j ) .
While the above objective enforces disentanglement, controllable generation requires some regularity in z so that users can decide values of z resulting in desired properties in the generated samples.
We introduce Ω V AE to enforce such a regularity, i.e., a monotonic relationship is enforced for the distance between pairs of images where only a particular generative factor vary and a corresponding latent variable.
In other words, an increasing trend in the value of each dimension of z should yield a greater change in the output along a generative factor.
Formally, Ω V AE is deﬁned as follows:
[Equation]
where L is given by the gradient of the mean squared error (MSE) between pairs of images that are 1-factor away with respect to the dimension y of z assigned to the changing factor, i.e.:
[Equation]
In this case, ˆ x indicates the reconstruction of x .
We evaluate such an approach by training the same 4-layered convolutional VAEs described in (Higgins et al., 2016) using the 3d-shapes dataset 2 .
The dataset is composed of images containing shapes generated from 6 independent generative factors: ﬂoor color , wall color , object color , scale , shape and orientation .
All combinations of these factors are present exactly once, resulting in m = 480000 .
We compared VAEs trained with and without the inclusion of the monotonicity penalty given by Ω V AE .
We highlight that the goal of
[Figure]
B a s e li n e : d i s e n t ang l e d bu t non - m ono t on i c
M ono t on i c m od e l : s m oo t h t r an s iti on s a l ong g e n e r a ti ve f a c t
o r
B a s e li n e
M ono t on i c m od e l
[Figure]
Orientation
[Figure]
Shape
Orientation
Size the proposed framework is not to improve over current approaches in terms of how disentangled the learned representations are.
Rather, we seek to achieve similar results in that sense, but impose extra regularity and structure in the relationship between the generated images and the values of z so that the generative process is more easily controllable .
Qualitative analysis is performed and shown in
Figure 2.
The two panels on the left represent the data generated by a linear combination of the latent code corresponding to two images that only vary in the factor object color .
The panels stacked on the right present a per-dimension traversal of the latent space starting from a common image.
It can be observed that disentanglement is indeed achieved in both cases.
The monotonic model presents much smoother transitions between colors while the base model gives long sequences of very close images followed by very sharp transitions where the colors sometimes repeat (e.g., green-yellowgreen transitions in the fourth row).
As for the results per factor, the monotonic model provides more structure in the latent space compared to the base model.
This can be observed in the shape factor.
The monotonic model provides a certain order: sphere, cylinder, and then cube.
Visually inspecting many samples, the monotonic model is following this order for the generated shapes.
This pattern is even more pronounced in the color factors.
We have found that the colors generated by the monotonic model follows the order of the colours in the HUE cycle .
So our model has ordered the latent space and we know how to navigate it to generate a desired image.
On the other hand, the baseline has no clear order of the latent space.
For example, the baseline generates cubes at different ranges of z .
Similarly, the colors generated by the baseline model do not have a clear order.
To further support the claim that Ω V AE induces regularity in the latent space, we introduce the analysis shown in Table 4.
We started by increasing z 3 (associated to ﬂoor color for both models), and recorded the sequence of the generated colors.
We observed that for a large fraction of the data, the monotonic models yield sequences of images where the color of the ﬂoor is ordered according to its corresponding
HUE angle.
Further details are available in Appendix H.
[Table]
C ONCLUSION
We proposed approaches that enable learning algorithms based on risk minimization to ﬁnd solutions that satisfy some notion of monotonicity.
First, we discussed the case where monotonicity is a design requirement that needs to be satisﬁed everywhere.
In this case, we identiﬁed limitations in prior work that resulted in models satisfying the property only in very speciﬁc parts of the space.
We then introduced an efﬁcient procedure that was observed to signiﬁcantly improve the solutions in terms of the volume of the space where the monotonicity requirement is achieved.
In addition, we further argued that, even when not required, models satisfying monotonicity present useful properties .
We studied the case of image classiﬁers and generative models and showed that imposing structure in learned representations via group monotonicity is beneﬁcial and can be done efﬁciently.
A I LLUSTRATIVE EXAMPLES ON THE SPHERE : M IXUP HELPS TO POPULATE THE SMALL VOLUME INTERIOR REGION
To further illustrate the issue discussed in the item 2 of Section 3.1 as well the effect of our proposal, we discuss a simple example considering random draws from the unit n -sphere, shown in Figure 3, i.e., the set of points B = { x ∈ R n : || x
|| 2 < 1 } .
We further consider a concentric sphere of radius 0 < r < 1 given by B r = { x ∈ R n : || x || 2 < r } .
We are interested in the probability of a random draw from B to lie outside of B r , i.e.: P ( || x || 2 > r ) , x ∼ D ( B ) , for some distribution D .
We start by deﬁning D as the Uniform ( B ) , which results in P ( || x || 2 > r )
= 1 − r n .
In Figure 4a, we can see that for growing n , P ( || x || 2 > r ) is very large even if r ≈ 1 , which suggests most random draws will lie close to B ’s boundary.
We now evaluate the case where mixup is applied and random draws are taken in two steps: we ﬁrst observe y ∼ Uniform ( B ) , and then we perform mixup between y and the origin 3 , i.e., x = λy , λ ∼ Uniform ([0 , 1]) .
In this case, P ( || x || 2 > r ) =
(1 − r n )(1 − r ) , which is shown in Figure 4b as a function of r for increasing n .
We can then observe that even for large n , P ( || x || 2 > r ) decays linearly with r , i.e., we populate the interior of B and x in this case follows a non-uniform distribution such that its norms histogram is uniform.
[Figure]
[Figure] [Figure]
[Figure]
In this case, x = λy , λ ∼ Uniform ([0 , 1]) , y ∼
Uniform ( B ) .
B P ROOF - OF - CONCEPT EVALUATION
We start by describing the approach we employ to generate data containing the properties required by our evaluation.
Denote a design matrix by X N × D such that each of its N rows corresponds to a feature vector within R D .
In order to ensure the data lies in some manifold, we ﬁrst obtain a lowdimensional synthetic design matrix given by X (cid:48)
N × d , where each entry is sampled randomly from
Uniform ([ − 10 , 10]) .
We then expand it to R D by applying the following transformation:
[Equation]
where the expansion matrix given by A d × D is such that each of its entries are independently drawn from Uniform ([0 , 1]) .
Throughout our experiments, d = (cid:98) 0 .
3 D (cid:99) was employed.
Target values for the function f to be approximated are deﬁned as sums of functions of scalar arguments applied independently over each dimension.
We thus select a set of dimensions M ∈ [ D ] with respect to which f is to be monotonic, i.e.:
[Equation]
and every g i : R (cid:55)→ R is increasing monotonic, while every h i : R (cid:55)→ R is not monotonic.
We then create two evaluation datasets.
One of them, referred to as the validation set, is identically distributed with respect to X since it is obtained following the same procedure discussed above.
In order to simulate covariate-shift, we create a test set by changing the expansion matrix A to a different one.
[Equation]
where A test will be given by entry-wise linear interpolations between A , used to generate the training data, and a newly sampled expansion matrix A (cid:48) :
A test = αA (cid:48) + (1 − α )
A .
The parameter α ∈ [0 , 1] , set to 0 .
8 in the reported evaluation, controls the shift between A test and A test in terms of the Frobenius norm, which in turn enables the control of how much the test set shifts relative to the training data.
We thus trained models to approximate f for spaces of increasing dimensions as well as for an increasing number of dimensions with respect to which f is monotonic.
Results are reported in
Table 5 in terms of RMSE on the two evaluation datasets, and in terms of monotonicity in Table 6 where ρ is computed both on random points and on the shifted test set.
Entries in the tables correspond to the centers of 95% conﬁdence intervals resulting from 20 independent training runs.
We highlight the two following observations regarding the prediction performances shown in table 5: different models present consistent performances across evaluations, which suggests different monotonicity-enforcing penalties do not signiﬁcantly affect prediction accuracy.
Moreover, the proposed approach used to generate test data under covariate-shift is effective given the gap in performance consistently observed between the validation and the test partitions.
In terms of monotonicity, results in Table 6 suggest that Ω random and Ω train are only effective on either random or data points, which seems to aggravate when the dimension D grows.
Ω mixup , on the other hand, is effective on both sets of points, and continues to work well for growing D .
Furthermore, covariate-shift signiﬁcantly affects Ω train for higher-dimensional cases, while Ω mixup performs well in such a case.
[Table] [Table]
M | /
[Table]
C D ATASETS , MODELS , AND TRAINING DETAILS FOR EXPERIMENTS REPORTED IN S ECTION 3.
Algorithm 1 describes a procedure used to compute the proposed regularization Ω mixup .
Algorithm 1 Procedure to compute Ω mixup .
Input mini-batch X [ N × d ] , model h , monotonic dimensions M
X Ω = {} # Initialize set of points used to compute regularizer.
˜ X [ N × d ] ∼ Uniform ( X N ) # Sample random mini-batch with size N .
ˆ X = concat ( X, ˜ X )
# Concatenate data and random batches.
repeat i, j ∼ Uniform ( { 1 , 2 , ..., 2 N } 2 )
# Sample random pair of points.
λ ∼ Uniform ([0 , 1]) x
= λ ˆ X
i + (1 − λ )
ˆ X j # Mix random pair.
X Ω .add ( x ) # Add x to set of regularization points.
until Maximum number of pairs reached
[Equation]
return Ω [Equation]
In Table 7, we list details on the three datasets used to evaluate our proposals as reported in Section 3.2.
Dataset
[Table]
Models follow the architecture in (Liu et al., 2020) using dense layers whose weights are kept separate in early layers for the input dimensions with respect to which monotonicity is to be enforced.
We set the depth of all networks to 3, and use a bottleneck of size 10 for two datasets (Compas and Loan Lending Club), and 100 for the case of the Blog Feedback dataset and the experiments on generated data.
Training is carried out with the Adam optimizer (Kingma & Ba, 2014) with a global learning rate of 5e − 3 , and γ is set to 1e4 .
The training batch size is set to 256 throughout experiments.
[Table]
D M ODELS AND TRAINING DETAILS FOR EXPERIMENTS REPORTED IN S ECTION
For the case of CIFAR-10, WideResNets (Zagoruyko & Komodakis, 2016) are used.
The models are initialized randomly and trained both with and without the monotonicity penalty.
Standard stochastic gradient descent (SGD) implements the parameters update rule with a learning rate starting at 0.1, being decreased by a factor of 10 on epochs 10, 150, 250, and 350.
Training is carried out for a total of 600 epochs with a batch size of 64.
For ImageNet, on the other, training consists of ﬁne tuning a pre-trained ResNet-50, where the ﬁne-tuning phase included the monotonicity penalty.
We do so by training the model for 30 epochs on the full ImageNet training partition.
In this case, given that the label set Y is relatively large, using the standard ResNet-50 would result in small slices
S k .
To avoid that, we add an extra ﬁnal convolution layer with W = 15 K .
Training is once more carried out with SGD using a learning rate set to 0.001 in this case, and reduced by a factor of 5 at epoch 20.
In both cases, the group monotonicity property is enforced at the last convolutional layer.
Other hyperparameters such as the strength γ of the monotonicity penalty as well as the inverse temperature µ used to compute Ω group are set to 1 and 50 for the case of CIFAR-10, and to 5 and 10 for the case of ImageNet.
Both momentum and weight decay are further employed and their corresponding parameters are set to 0.9 and 0.0001.
For MNIST classiﬁers, training is performed for 20 epochs using a batch size of 64 and the Adadelta optimizer (Zeiler, 2012) with a learning rate of 1.
E E NFORCING GROUP MONOTONICITY UNDER SMALL SAMPLES
Using CIFAR-10, we further evaluate how the proposed group monotonicity penalty behaves in dataconstrained settings, i.e., we check whether or not the property can be enforced under small sample regimes.
We do so by sub-sampling the original training data by randomly selecting a fraction of the training images uniformly across classes.
We then train the same WideResNet for the same computation budget in terms of number of iterations as the models trained in the complete set of images.
The learning rate schedule also matches that of the training on the full dataset in that the learning rate is reduced at exactly the same iterations across all training cases.
Results are reported in
Table 8 for sub-samples corresponding to 10%, 30%, and 60% of CIFAR-10.
Results are consistent across the three sets of results in showing that predictions obtained from the total activation of feature slices approximate the prediction performance of the underlying model for the case of group monotonic predictors, i.e., the extent to which the underlying model is able to accurately predict correct classes upper bound the resulting “level of monotonicity”.
In simple terms, the better the classiﬁer, the more group monotonic it can be made.
F S ELECTING FEATURE MAPS TO COMPUTE VISUAL EXPLANATIONS
Approaches based on Class Activation Maps (CAM) such as Grad-CAM and its variations (Selvaraju et al., 2017; Chattopadhay et al., 2018) seek to extract explanations from convolutional models.
By explanation we mean to refer to indications of properties of the data implying the predictions of a given model.
Under such a framework, one can obtain so-called explanation heat-maps through the following steps: (1) Compute a weighted sum of activations of feature maps in a chosen layer; (2)
Upscale the results in order to match the dimensions of the input data; (3) Superimpose results onto the input data.
Speciﬁcally for the case of applications to image data, following those steps results in highlighting the patches of the input that were deemed relevant to yield the observed predictions.
Different approaches were then introduced in order to deﬁne the weights used in the ﬁrst step.
A very common choice is to use the total gradient of the output corresponding to the prediction with respect to activations of each feature map.
For the case of group monotonic classiﬁers, we are interested in verifying whether one can deﬁne useful explanation heat-maps by considering only the feature slices corresponding to the predicted class, i.e., for a given input pair ( x, y ) , we compute explanation heat-maps considering only its corresponding feature activation slice S y ( x ) .
We thus design an experiment to evaluate the effectiveness of such an approach by using external auxiliary classiﬁers to perform predictions from test data that was occluded using explanation heat-maps obtained using different models and sets of representations.
In other words, we use the explanation maps to remove from the data the parts that were not indicated as relevant.
We then assume that good explanation maps will be such that classiﬁers are able to correctly classify occluded data since relevant patches are conserved.
In further details, occlusions are computed by ﬁrst applying a CAM operator given a model h and data x , which results in a heat-map with entries in [0 , 1] .
We then use such a heat-map as a multiplicative mask to get an occluded version of x , denoted x (cid:48) , i.e.:
[Equation]
where the operator ◦ indicates element-wise multiplication.
An example of such a procedure is shown in Figure 5.
We apply the above procedure to all of the validation data, and use resulting points to then assess the prediction performance of auxiliary classiﬁers.
Explanation maps are computed using the same models discussed in Section 4.2.1 for ImageNet.
The CAM operator corresponds to a variation of Grad-CAM++ (Chattopadhay et al., 2018) where the model activations are directly employed for weighing feature maps rather than the gradients.
We consider 4 auxiliary pre-trained classiﬁers corresponding to ResNext-50 (Xie et al., 2017),
MobileNet-v3 (Howard et al., 2019), VGG-16 (Simonyan & Zisserman, 2014), and SqueezeNet (Iandola et al., 2016).
Results are reported in Table 9 which also include the reference performance of the auxiliary classiﬁers on the standard validation set in order to provide an idea of the gap in performance resulting from removing parts of test images via occlusion.
We highlight the performance reported in the last row of the Table.
In that case, explanation maps for the group monotonic model are computed from only the features of the class slice, which is enough to match the performance of a standard ResNet-50 with full access to the features.
This suggests that representations learned
[Table]
by group monotonic models are such that all the information required to explain a given class is contained in the slice reserved for that class.
G E XAMPLES OF EXPLANATION HEAT - MAPS AND OCCLUDED DATA
H A NALYSIS OF COLOR SEQUENCES FOR GENERATED DATA
We performed a set of experiments in order to evaluate whether some kind of ordering could be observed once we generate data for increasing values of z , speciﬁcally on dimensions that correspond to colors.
To do that, we created an increasing sequence of values by deﬁning a uniform grid in [0 , 1] with 50 steps.
We then encoded a particular image, but decoded latent vectors after substituting the z value in the dimension corresponding to ﬂoor color by the values in the sequence.
Generated sequences of images are shown in Figures 9 and 10 for the base and monotonic models, respectively.
In each such a case, we plot the images on the left, and bottom-left patches of size 10x10 so as to highlight the color sequences that we observe with such an approach.
Surprisingly, we observed that monotonic models tend to generate colors in a sequence that matches the HUE circle for RGB images, represented in Figure 8 for reference.
Besides visually verifying that to be the case across a number of generated examples, in Table 4 in Section 4.3 we check the fraction of the dataset where such sequences of patches are sorted in terms of their HUE angles.
[Figure] [Figure]
I E XAMPLES OF DATA GENERATED WITH STANDARD AND MONOTONIC MODELS
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]