Fast Implicit Constrained Optimization of Non-decomposable Objectives for Deep Networks
Abstract
We consider a popular family of constrained optimization problems in machine learning that involve optimizing a non-decomposable objective while constraining another.
Different from the previous approach that expresses the classiﬁer thresholds as a function of all model parameters, we consider an alternative strategy where the thresholds are expressed as a function of only a subset of the model parameters, i.e., the last layer of the neural network.
We propose new training procedures that optimize for the bottom and last layers separately, and solve them using standard gradient-based methods.
Experiments on a benchmark dataset demonstrate our proposed method achieves performance comparable to the existing approach while being computationally efﬁcient.
Introduction
Many machine learning applications involve optimizing for non-decomposable metrics , such as false-positive rate (FPR) or precision, while constraining another, such as false-negative rate (FNR) or recall.
For these metrics, the loss on a set of data points cannot be expressed as the sum of losses of individual data points (e g., FPR involves a ratio that depends on the total false-positive and true-negative classiﬁed examples).
Such problems arise in several applications.
For example, many fairness objectives can be expressed as rate constraints, including the popular equal opportunity and equalized odds fairness criteria [1, 4, 12, 13].
The typical baseline for solving these problems is ﬁrst to train the model using regular cross-entropy (CE) loss, followed by tuning the thresholds imposed on the model predictions to classify between positive versus negative classes while the constraint is satisﬁed.
However, the post hoc approach often yields sub-optimal results as the model parameters are trained on a different objective than the evaluation metric.
To improve upon the baseline approach, Implicit Constrained Optimization (ICO) [6] reformulates the prediction threshold as a function of the model parameters and transforms the constrained optimization problem into an unconstrained one to jointly optimize for the model parameters and the thresholds.
ICO solves the problem by decomposing the gradient into a sum of disjoint terms, which consist of the gradient of the loss function and the constraint with respect to the model parameters and the thresholds.
Such a decomposition is computationally more expensive than standard training since it requires additional backward passes to compute the gradients with respect to each component.
In particular, ICO becomes more expensive when there are multiple constraints, e.g., when optimizing for the area under the ROC
Curve (AUC).
To reduce the computational cost, ICO resorts to approximations, e.g., intermittently updating the thresholds – Section 3.3 in [6].
Thus, it is desirable to close this computation gap while retaining the simplicity of the method as well as potentially improved performance on the metrics.
Motivation of This Work: Constrained Optimization of Only the Last Layer Standard neural network architectures can be decoupled into a feature extractor sub-network at the bottom, followed by a single fully-connected classiﬁer layer to predict the class probabilities at the top (see Figure 1).
Several recent studies have shown that the feature extractor network can be trained reasonably well even when the training data is noisy [7] or imbalanced [10, 5].
These studies show the signiﬁcance of retraining the last layer weights on a clean or balanced validation set in obtaining good performance based on the feature representation extracted by the bottom sub-network.
While it is natural to impose the constraint on all the model parameters in our problem, the results from the earlier work raise the question of whether constraining the last layer weights is sufﬁcient for obtaining desired performance [6].
To this end, we propose to make the prediction thresholds a function of only the last layer parameters.
This simpliﬁcation leaves the parameters of the earlier layers free to be optimized with any surrogate loss in an unconstrained manner.
In the meantime, the constraint is imposed on the last layer parameters.
The major beneﬁt of our approach is avoiding additional backward passes to calculate the gradient of the constraint(s) with respect to the weights of the bottom sub-network.
[Figure] [Figure]
Problem Formulation
Basic Setup In this paper, we consider a binary classiﬁcation setting with feature space X and binary label { 0 , 1 } .
Our goal is to learn a binary threshold scoring rule s θ | λ : X → { 0 , 1 } m parameterized by model parameter θ ∈ R p and threshold values λ ∈ R m , such that for any given input x ∈ X , [ s θ | λ ( x )]
i = 1 ( s θ ( x ) ≥ λ i ) where s θ ( x ) : X → R is a scoring model parameterized by the model parameters θ ∈ R p and 1 ( · ) is the indicator function.
We assume the scoring model s θ is an L -layer neural network whose parameters can be divided into two disjoint parts θ = [ θ (cid:57) L , θ L ] ∈ R p : the parameters of the last dense layer θ L ∈ R p L , and the weights from the rest of the network, which we will also refer to as the bottom layers: θ (cid:57) L ∈ R p (cid:57) L , where p = p (cid:57) L + p L .
The bottom layers transform the input x into a feature representation φ ( x ) .
The last dense layer then sets s θ ( x ) = θ (cid:62) L φ ( x ) .
In this work, we explore ideas involving faster training of deep networks using non-decomposable objectives by applying different strategies for training the last layer weights θ L and the rest of the weights θ (cid:57) L .
Objective Function
In this paper, we consider a general form of a constrained optimization problem with m constraints:
[Equation]
where f : R p × R m → R and g : R p × R m →
R m are non-decomposable functions that cannot be expressed as the sum of losses on individual data points, θ ∈ R p are the model parameters, and λ ∈ R m are the thresholds and chosen such that the constraint is satisﬁed.
Similar to [6], throughout the training process, we assume that θ stay in the feasible region, namely ∀ θ , ∃ λ s.t.
g ( θ, λ ) = 0 .
Here, we provide one example that satisﬁes this general constraint optimization form, which is also going to be the main task of our experimental demonstration in Section 4:
Example 1 (Minimizing FNR subject to a ﬁxed FPR) .
Consider a setting where we want to minimize the FNR at the threshold λ ∈ R at which the FPR is a ﬁxed value β ∈ [0 , 1] : [Equation] = [Equation]
TN
[Equation]
where FN ( s θ | λ ) , TP ( s θ | λ ) , and TN ( s θ | λ ) denote the true-positive, false-positive, and false-negative values, respectively, for the threshold scoring rule.
Surrogate Losses It is possible that the optimization objectives and constraints described above are non-differentiable.
To make the training viable using gradient descent-based methods, we replace f and g with their corresponding smooth surrogate losses , ˜ f and ˜ g , and relax Eq.
(1) into:
[Equation]
Methods
We ﬁrst brieﬂy discuss the previous methods and introduce our efﬁcient alternative.
3.1 Previous Approach: Implicit Constraint Optimization (ICO) on All Weights
Different from traditional Lagrangian based methods [3], ICO [6] avoids explicitly solving the constrained problem in Eq.
(2) by formulating an equivalent unconstrained problem in which the thresholds λ are expressed as an implicit function of the model parameters θ (within a neighborhood around θ ).
Speciﬁcally, the implicit function theorem [11] implies: suppose ( θ 0 , λ 0 ) satisﬁes the constraints, namely ˜ g ( θ 0 , λ 0 )
= 0 . Then, we can express the thresholds as {+λ 
+}
= ˜
h ( θ 0 ) in the neighborhood of θ 0 where ˜ h is some implicit function that depends on problem setup and the local structure of ( θ 0 , λ 0 ) .
See the detailed description of the theorem (Theorem 1) in the appendix.
The beneﬁt of using Theorem 1 is that we can change the constrained optimization problem to be an unconstrained optimization problem with only one free variable θ :
[Equation]
A differentiable function ˜ h ( θ ) that provides us ˜ g ( θ, ˜ h ( θ ))
=
for a particular threshold that we care about might not always exist.
However, [6] identiﬁes some conditions under which the resulting
Eq.
(3) is convex in θ .
3 We also provide detailed derivations in the appendix.
In this work, we propose making the thresholds λ a function of only the last layer’s weights θ L , and applying different training procedures for the bottom layer weights θ (cid:57) L and last layer weights θ L separately.
Implicit function theorem on θ L
First, we propose to make the thresholds λ in the constraint ˜ g ( θ, λ ) =
a function of only θ L , which gives us the following modiﬁed form of the implicit function theorem (see the appendix).
The modiﬁed theorem implies that given φ ( x ) , the classiﬁer weights θ L control the value of the thresholds λ in a feasible region.
New objective form: decoupled training for θ (cid:57) L and θ L
We propose a new optimization procedure for training a neural network that optimizes its bottom layer weights θ (cid:57) L and last layer weights θ L separately using different training objectives:
[Equation]
(4) where ˜ r : R p (cid:57) L → R and ˜ (cid:96) : R p
L × R m → R are differential objective function for θ (cid:57) L and θ L , respectively.
For the objective function ˜ r , the notation ( θ (cid:57) L | θ L , λ ) means we optimize over θ (cid:57) L while ﬁxing θ L and λ .
The notation holds similarly for ˜ (cid:96) and ˜ g .
Using this convention, we can write ˜ f ( θ, λ ) =
˜ r ( θ (cid:57) L | θ L , λ ) + ˜ (cid:96) ( θ L , λ | θ (cid:57) L ) .
We assume the conditions in Theorem 2 hold, which means that we can re-write the thresholds λ as a function of the last layer weights θ L (i.e., λ = ˜
h L ( θ L ) ).
The speciﬁcation of the function form ˜ h L ( · ) holds with respect to the modiﬁed setting. [Table]
The main focus of this work is to explore different training objectives and training strategies that take the form of Eq.
(4) to speed up the training procedure and get better prediction outcomes [Table]
• Method 1: minimizing CE for θ (cid:57) L and constrained FNR for θ L
Our ﬁrst proposed method is to use cross-entropy (CE) loss as the objective function for the bottom network θ (cid:57) L , and use constrained false-negative rate (FNR) only for updating the gradient for θ L .
This corresponds to specifying ˜ r , ˜ (cid:96) , and ˜ g in Eq.
(4) as follows:
[Equation]
= [Equation] where β ∈ [0 , 1] is the target FPR rate for the constraint.
• Method 2: minimizing FNR with regularization Our second proposed method involves minimizing FNR for the whole network while imposing FPR
constraints on the last layer weight. In addition , we impose
a
constraint regularizer on θ (cid:57) L .
This corresponds to specify ˜ r , ˜
(cid:96) , and ˜ g in Eq.
as follows:
[Equation]
where β ∈ [0 , 1] is the target FPR rate for the constraint and η > 0 balances between minimizing the FNR loss and the regularization term.
The intuition behind adding this regularization term is to avoid trivially minimizing FNR by always predicting one at the output for any given input.
Also, the regularized objective form in Eq.
(5) resembles the Lagrangian method proposed in [3, 2, 9] for solving constrained optimization problems.
• Ablation: regularization on all layers’ weights For comparison, we also try imposing the constraint regularizer on all weights.
This is equivalent of modifying ˜ (cid:96) , the objective function for θ L in Eq.
(5) by adding a similar regularization term as in ˜ r :
[Equation]
Interestingly, we observe that applying ICO on the last layer weights improves the performance over using the same regularized loss for training all weights.
We provide a detailed comparison in Eq.
(4) (see Table 1).
Experiments
We evaluate our approach on an image classiﬁcation task.
In particular, we use the celebA dataset [8].
We consider the task of minimizing the false-negative rate (FNR) at a given false-positive rate (FPR) of β .
Please refer to the appendix for more details.
Experimental Results We present our experimental results in Table 1.
Overall our proposed method 2 (which we refer to as “FPR Regularizer on θ L + Constrained FNR on θ L ”) achieves comparable performance compared to ICO, and we observe a consistent improvement of both our proposed Method 2 and Ablation (which we refer to as “FPR Regularizer on θ ”) compared to the other two baseline methods (CE and TFCO [2, 9]).
Our method involves training θ L and θ (cid:57) L separately while ﬁxing the other parameters and making the thresholds as a function of only θ L . Thus, our approach is signiﬁcantly cheaper compared to the baseline method ICO , which trains all parameters together, making the thresholds a function of all the parameters.
This observation demonstrates that our proposed method can successfully preserve performance while reducing the cost of training compared to previous methods.
Interestingly, among the two proposed methods, imposing ICO on θ L while using FPR-regularizer on θ (cid:57) L leads to better performance compared to imposing FPRregularizer on all weights θ .
This supports our hypothesis that θ (cid:57) L and θ L take up different roles during training, and the last layer weights θ L play a more signiﬁcant role in overall performance.
We propose new training techniques for optimizing a popular constrained optimization problem that involves non-decomposable rate metrics.
Compared to the previous approach, where the classiﬁer thresholds are expressed as a function of all model parameters, we consider an alternative technique where the thresholds are expressed as a function of only the last layer weights of a neural network.
Our empirical results show that among all proposed methods, adding a regularizer on the bottom layers while solving the constrained optimization problem for the top layer achieves a comparable performance to the previous approach, supporting our hypothesis on the important role of last layer training in deep neural networks. Appendix An Example of Surrogate Loss for
FNR For our particular example involving minimizing f = FNR =
FN total positives , recall that FN ( s θ | λ )
=
(cid:80) i : y i =1 1 ( p i ( θ ) ≤ λ ) where p i is the predicted probability for the i-th positive example.
We replace the indicator function with its smooth surrogate and get (cid:102)
FN ( s θ | λ ) =
(cid:80) i : y i =1 σ τ ( − ( p i − λ )) ,
where σ τ ( u ) = 1 / (1 + exp( − τ u )) denotes a temperature scaled sigmoid function.
This gives us ˜ f
= (cid:102)
FN total positives as the surrogate loss.
Implicit Function Theorem on θ
Here, we provide the statement for the implicit function theorem and its modiﬁcation for the last layer weights:
Theorem 1 (Implicit Function Theorem on θ [11], informal ) .
For any ( θ 0 , λ 0 ) ∈ U ⊆
R p × R m pair that satisﬁes ˜ g
( θ 0 , λ 0 )
= 0 , if the determinant of the Jacobian matrix is nonzero, i.e., det [ ∂
˜ g
i ∂θ j
( θ 0 , λ 0 )] (cid:54) = 0 , then there exists a neighborhood Θ × Λ of ( θ 0 , λ 0 ) in U and a unique function ˜
h : Θ ⇒ Λ :
[Equation]
[Equation]
Theorem 2 (Implicit Function Theorem on θ L ) .
For any ( θ 0 , λ 0 ) ∈ U ⊆
R p × R m pair that satisﬁes ˜ g
( θ 0 , λ 0 )
= 0 , if the determinant of the Jacobian matrix w.r.t θ L is nonzero, i.e. det [ ∂ ˜ g
i ∂θ L j ( θ 0 , λ 0 )]
(cid:54) = 0 , then there exists a neighborhood Θ × Λ of ( θ 0 , λ 0 ) in U and a unique function ˜
h L : Θ ⇒ Λ such that
[Equation]
Gradient Update Rule for Eq.
(3) To compute a local derivative for ˜ f
( θ, ˜ h
( θ )) within the neighborhood of θ 0 using Theorem 1, we have the following update rule:
[Equation]
We will further need the derivative of the implicit function ˜
h w.r.t.
θ , i.e., ∇ θ ˜ h
( θ ) .
{+Since ˜ g ( θ, ˜ h ( θ )) = 
+}
in the neighborhood of θ 0 , we have:
[Equation]
Plugging Eq.
(9) back to Eq.
(8), we can get the ﬁnal gradient for the model parameter θ .
See section 3.3 of [6] for a more detailed derivation for the update rule of θ and λ .
Gradient Update Rule for Eq.
Furthermore, we can break the gradient of Eq.
(4) into two parts: the gradient w.r.t θ (cid:57) L and the gradient w.r.t θ L .
We thus obtain the following gradients:
[Equation]
[Equation]
Experimental Details Following [6], we choose three binary attributes (i.e., High-cheekbones,
Smiling, and Wearing-lipsticks) as target attributes for our experiments and train a binary classiﬁer for each attribute.
Similar to [6], we use a 6-layer neural network with 5 convolutional layers with 128 , 256 , 512 , 512 ﬁlters, respectively, and we use ReLU as our activation functions and apply batch normalization layers in the networks.
We conduct 5 random trials for each experiment and report the average values of the metric.